Presidents, Vice Presidents, and Washington's Most Notorious Slave Pen

In 1836, a striving, twenty-four-year-old New England shoemaker took an excursion southward to recuperate from a bout with ill health. By May, Henry Wilson found himself in Washington, D.C., on a transformative journey that permanently altered the trajectory of his life. What the naïve, wide-eyed Wilson saw in the nation’s capital made a profound impression upon him. While there, he gazed upon “the infamous Williams slave pen and . . . the poor people marched down to the riverside and shipped off to the ‘far South.’” He witnessed “men and women, in chains, put upon the auction block for the crime of possessing ‘a skin darker than his own,’ and sold to hopeless slavery.” As a consequence of witnessing these spectacles, Wilson stated, “I pledged myself to liberty.” He turned to a life of politics “to make war upon slavery.” The humble shoemaker entered the Massachusetts legislature at age twenty-eight, occupying a seat in the statehouse from 1841 to 1852. Beginning in 1855, Wilson served in the United States Senate, where he remained until his resignation in 1873 to become Ulysses S. Grant’s vice president.¹ Click here to learn more about the enslaved households of President Ulysses Grant.

William H. Williams’ infamous private slave jail, dubbed the Yellow House, was instrumental in the education of Henry Wilson. The Yellow House was located just south of the National Mall, on the block bounded by Seventh and Eighth streets, and by B Street and Maryland Avenue. The three-story brick structure, plastered and painted its distinctive color, sat where today’s passersby find the Federal Aviation Administration’s Orville Wright Federal Building, across the street from the Smithsonian’s Hirshhorn Museum. Williams’ slave pen was readily visible as one gazed westward from the Capitol Building. During his single term in the U.S. House of Representatives, Abraham Lincoln remarked that he could see, “from the windows of the capitol, a sort of negro-livery stable, where droves of negroes were collected, temporarily kept, and finally taken to Southern markets, precisely like droves of horses.” With only mild exaggeration, Henry Wilson placed the Yellow House slave jail “beneath the shadow of the flag that waved over the Capitol.”²

The most famous prisoner ever held captive in the Yellow House was the abducted northern free black man Solomon Northup, whose story was chronicled in Twelve Years a Slave. After being drugged and kidnapped from the National Hotel on Pennsylvania Avenue, Northup regained consciousness enchained in the Yellow House’s basement dungeon. The twelve-foot square cell had walls of “solid masonry” and a “floor . . . of heavy plank.” A “small window, crossed with great iron bars” might have admitted some little light from the outside, but it was covered “with an outside shutter, securely fastened.” Crude accommodations consisted of a “wooden bench” and “dirty box stove” but no bed or blanket.³

From the street, the Yellow House sat back from the road and was partially obscured by poplar trees. Looking like “a quiet private residence” from the outside, its innocuous façade disguised the trauma experienced by those imprisoned within its walls. According to Solomon Northup’s description from 1841, an outdoor “yard extended rearward from the house about thirty feet.” This central space was “paved with brick” and “surrounded by a brick wall ten or twelve feet high.” As Northup further explained, “The top of the wall supported” the lower “end of a roof, which ascended inwards,” toward the courtyard, “forming a kind of open shed. Underneath the roof there was a crazy loft all around, where slaves, if so disposed, might sleep at night, or in inclement weather seek shelter from the storm.” Northup likened the place to “a farmer’s barnyard in most respects.” Abutting the courtyard, an outbuilding constructed of brick served as the slave quarters. It was “two stories high, fronting . . . one of the public streets of Washington,” with “[w]indows grated and prison-like.” Enslaved captives were imprisoned either in this outbuilding, “in a room adjoining the kitchen” of the dwelling house, or in three basement “apartments,” each “about 25 feet square.” Fearsome, ferocious dogs patrolled the facility.⁴

By the 1830s, Washington, D.C., had emerged as a premier depot in the domestic slave trade. Several slave dealers in the District and their agents scoured the Chesapeake countryside to buy enslaved people from owners in Maryland and Virginia. After making their purchases, traders imprisoned their enslaved captives in Williams’ Yellow House or another slave jail in the District until they had accumulated enough to compose a shipment for sale in the Deep South.

The administration of Andrew Jackson (1829-1837) brought with it the “flush times” on the cotton frontier of the Old Southwest. The business of slave trading was booming. Jackson himself was the only president who once trafficked in enslaved people professionally, although he was not the only commander-in-chief to buy, own, or sell property in human beings. Most pre-Civil War presidents did. In 1848, William H. Williams and his brother and business partner Thomas Williams sold Mexican-American War hero Zachary Taylor – “Old Rough and Ready” – two enslaved men, named Henry and Daniel, for $1,500, for use on the general’s East Baton Rouge Parish, Louisiana, plantation. Taylor’s son Richard served as his father’s agent in the purchase, transacted at the Williamses’ New Orleans office. A week later, General Taylor became the Whig Party’s nominee for president. He went on to win the election.⁵ Click here to learn more about the enslaved households of President Andrew Jackson and President Zachary Taylor.

With the domestic slave trade gaining momentum, abolitionists in the 1830s launched a petition drive to convince Congress to end the trade, if not slavery altogether, in the District of Columbia. Memorials, sometimes bearing thousands of signatures, poured into Congress. Women sympathetic to the plight of the enslaved gathered many of the signatures and affixed their own names to these petitions. In January 1836, future president James Buchanan, then serving as a U.S. senator from Pennsylvania, observed that one particular “memorial from the softer sex contains as much matter of offence, insult, and vituperation, applicable to all the slaveholding portions of their fellow-citizens, as could possibly have been put into a paper of the same compass.” Buchanan found it unbecoming. He disapproved of the women’s bold language and criticized their “fanaticism,” remarking that their political activism had made them too masculine. “They have unsexed themselves,” he scoffed.⁶

Facing an onslaught of antislavery petitions, proslavery forces in the House of Representatives implemented the so-called “gag rule” in 1836, which received but then automatically tabled any memorial on the subject of slavery. Antislavery petitions could not be read, discussed, or printed. Watching congressional proceedings from the gallery, convalescing shoemaker Henry Wilson lamented seeing “the slave masters ‘laying on the table’ the petition[s] of the Christian men and women of the country against the abominable traffic in human beings.” The House would not vote to overturn the gag rule until December 1844, one month after the fifteenth presidential election.⁷

Unexpectedly, William H. Williams’ slave pen injected itself into the national campaign for president in 1844. The election that year featured a contest between Democrat James Knox Polk and Whig candidate Henry Clay. Polk and his running mate, George M. Dallas, championed the annexation of the independent slaveholding republic of Texas into the United States. Their program embraced the geographic expansion of the nation and rejected federal interference with slavery. Henry Clay, by contrast, opposed the annexation of Texas, which he calculated could earn him antislavery votes in the North despite his status as a slaveowner himself. Click here to learn more about the enslaved households of President James K. Polk.

In mid-October 1844, only weeks before voting began, William H. Williams’ brother Thomas, then presiding over the Yellow House, sparked controversy when he hoisted a Polk and Dallas flag above the slave jail. With the banner of the Democratic ticket rippling in the wind above it, the Yellow House became a touchstone for political division, setting off a newspaper war between the District’s Democratic mouthpiece, the Washington Globe, and its cross-town rival, the National Intelligencer, the leading organ of the Whig Party.

Editor John C. Rives of the Democratic-leaning Globe sought to distance his party’s presidential ticket from the operators of the city’s leading slave pen. Although, individually, slave dealers such as the Williams brothers were prosperous and respectable men rather than social outcasts, collectively they nevertheless attracted broad censure in public conversation. One-term Illinois congressman Abraham Lincoln captured the popular sensibility when he described to an audience in 1854 the “sneaking individual, of the class of native tyrants, known as the ‘Slave-Dealer.’” As Lincoln explained, “He . . . crawls up to buy your slave, at a speculating price . . . You despise him utterly. You do not recognize him as a friend, or even as an honest man.” Slave traders did not merit so much as the traditional courtesy of a handshake. “If you are obliged to deal with him, you try to get through the job without so much as touching him,” Lincoln continued. “It is common with you to join hands with the men you meet; but with the slave dealer you avoid the ceremony – instinctively shrinking from the snaky contact.” Even after the slave dealer “grows rich and retires from business, you still remember him, and still keep up the ban of non-intercourse upon him and his family.” In reality, slave traders were highly respected in southern society, sometimes even running successfully for political office. But their reviled reputation in public discourse struck fear in the hearts of political partisans like John C. Rives of the Globe. Any association in voters’ minds between his Democrats and the Washington slave jailer flying the Polk and Dallas flag could damage the party’s prospects in the upcoming election.⁸

Rives thus concocted a grand conspiracy theory that he peddled in his correspondence and throughout the pages of the Globe. As Rives would have his readers believe, slave trader Thomas Williams was actually a Whig. He offered as evidence Williams’ advertisements for enslaved people in the District’s Whig newspaper and the slave dealer’s purported participation in a Whig parade held in the city two months earlier. Rives also asserted that area slave traders were “betting on Clay.” Affluent men of business routinely wagered on the outcomes of elections, typically putting money – sometimes thousands of dollars – on their preferred candidates. Given the assembled evidence, Rives concluded that flying the Polk and Dallas flag above the Yellow House was a clever Whig hoax designed to undermine the Democratic Party in advance of the election, by associating it with the taint of slavery. Elevating the Polk and Dallas flag above Williams’ slave pen was part of a deceptive plot to trick northern antislavery voters into casting ballots for the Whig Party’s slaveholding presidential candidate. “This appears to me . . . the deepest laid, the most diabolical and nefarious scheme that I have yet seen concocted,” Rives carped.⁹

Whereas Rives depicted Thomas Williams as a covert Whig operative engaged in subterfuge, Washington’s Whig press offered a simpler rationale for why the Democratic Polk and Dallas banner fluttered above the Yellow House: The slave-trading brothers were, in fact, Democrats. As shrewd businessmen seeking even greater riches by selling enslaved human beings, voting Democrat was in their best economic interests. A Democratic victory in the presidential election of 1844 heralded the annexation of Texas, which would open a whole new market for domestic slave traders. Texas was cotton country, and it required laborers the Williams brothers would be happy to supply. It was no coincidence, then, that reports had William H. Williams wagering heavily on a Polk victory. Indeed Polk won the election and secured vast new territories for the United States.¹⁰

After the election of 1844, the Yellow House remained in operation for another six years. At that point, the Compromise of 1850, among its other provisions, terminated the D.C. slave trade. Soon thereafter, the Yellow House slave jail was torn down. In 1854, former shoemaker turned abolitionist Henry Wilson returned to Washington, D.C., prior to assuming his first term as U.S. senator from Massachusetts. “I went to the spot where stood Williams’s slave pen,” he reflected, the very place “where, eighteen years ago, I saw men, women and children chained together for the Southern market,” a scene so emotionally overwhelming that it had propelled him into the world of antislavery politics. But now, the prison was gone. The “pen had passed away,” recorded the future vice president, “and flowers were blooming where the slave once sighed,” an auspicious sign for Wilson and his fellow abolitionists. In April 1862, amid the throes of Civil War, Congress passed a law emancipating people enslaved in the District of Columbia. It granted owners an average of $300 in compensation for each individual liberated from bondage. The complete eradication of slavery throughout the country would take longer and come at the cost of hundreds of thousands of lives, but nowhere else in the United States were owners reimbursed for the loss of their enslaved labor forces.¹¹

Jeff Forret is Professor and Distinguished Faculty Research Fellow at Lamar University. His latest book is Williams’ Gang: A Notorious Slave Trader and His Cargo of Black Convicts (Cambridge University Press, 2020).