The Enslaved Household of President James Madison

In a single week in early 1801, James Madison experienced two major life events. On February 27, his father James Madison Sr. died. He bequeathed his estate Montpelier in Orange County, Virginia, and more than 100 enslaved people to his son. On March 5, President Thomas Jefferson named Madison secretary of state, and he prepared to move his family to Washington, D.C., for the first time.¹ Throughout the family’s time in the city, including Madison’s term as secretary of state, his presidency, and Dolley Madison’s widowhood, they would rely on enslaved labor to run their household. This approach was common among the elite households of the new capital city. Washington society was maintained on the backs of enslaved people. Click here to learn more about the enslaved households of President Thomas Jefferson.

The Madisons brought a number of enslaved people with them from Montpelier, but they also hired out enslaved laborers from other slave owners in D.C., paying wages directly to the slave owners rather than to the people actually doing the work. In 1801, Madison entered into an agreement with Benjamin Orr “that Plato the slave of the said Orr is to serve the said Madison for five years,” and that during that time Plato was to be “under the direction in every respect of the said Madison, as fully & completely as if he was his own Slave & property.”² Five years was an unusually long term of hire, but otherwise this type of arrangement was rather common. Hiring out enslaved workers provided flexibility in the labor market, especially in urban areas, allowing slave owners to temporarily expand their labor force or rent out enslaved people as a revenue source as needed. In Washington, D.C., where each election cycle brought new residents and new labor demands to the city, such arrangements were particularly essential. James Madison, like many of his contemporaries, continued to make use of this system throughout his lifetime.³

James Madison was, according to historian Elizabeth Dowling Taylor, a “garden-variety slaveholder.” He adhered to the established social norms of Virginia society when it came to the treatment and living conditions of his enslaved household. Enslaved people worked from dawn to dusk, six days a week, with the customary Sunday off. Madison maintained control, but avoided the kind of excessive cruelty that might have drawn judgment from his peers.⁴ Like many in his time, he was concerned about the possibility that slaves might revolt. An attempted revolt in Richmond in 1800 stoked fears of mass slave uprisings, and the British willingness to take in fugitives during the War of 1812 only heightened those fears.⁵ Otherwise, he generally accepted slavery as a way of life. His wife, Dolley Madison, had been raised by a Quaker father who emancipated his own enslaved people after the Revolution, but she does not seem to have shared his convictions regarding the immorality of slavery.

A letter to his friend and former secretary Edward Coles offers some insight on Madison’s attitudes toward slavery. Coles had been a slave owner himself, but after leaving Madison’s employ he moved to Illinois, freed his enslaved people, and bought enough land to give each freed family a farm. Madison praised this effort as “a fair experiment for their happiness,” but wrote that unless Coles could change “their colour as well as their legal condition,” the freedmen would lack the “moral rank” and “social blessings” to truly take advantage of their newfound freedom.⁶ Coles later confided to his sister that he believed Madison would similarly free his own enslaved workforce when he died, as President George Washington had done.⁷ He, however, was mistaken. Madison specified in his will that “none of [the enslaved people] should be sold without his or her consent,” to keep the families together, but he left them to his wife instead of freeing them.⁸ His instruction not to sell enslaved people without consent was not legally binding, and Dolley Madison would go on to sell most of those enslaved people to alleviate her financial troubles later in life.

While most of enslaved individuals remained at Montpelier during his presidency, President Madison brought several with him to the White House to serve as household staff. One enslaved man, John Freeman, was already at the White House when the Madisons arrived. Freeman, who worked primarily as a dining room servant, had been hired and later purchased by Thomas Jefferson during his presidency. When Jefferson’s second term ended, Freeman resisted returning to Virginia because it would have meant leaving his family behind. Jefferson agreed to sell Freeman to the incoming president, James Madison, so that he could stay.⁹ He was freed in 1815 according to the terms of his original sale contract. He went on to purchase a house, raise eight children, and become a pillar of D.C.’s free black community.

Like John Freeman, Joseph Bolden, an enslaved man brought to the White House in bondage left a free man. Bolden cared for the Madison family’s horses and carriages. Mary Cutts, a niece of Dolley Madison’s who lived with them for a time, noted that “with his own wages he soon freed himself.”¹⁰ Cutts gave no specific detail regarding these wages, but presumably either the Madisons valued him highly enough to pay him a small stipend, or Bolden worked for wages for other families during his limited free time. Although Joseph Bolden earned his freedom, his wife, Milley, remained enslaved. She belonged to Francis Scott Key, the man who would go on to write what became the national anthem. “Your Servant Joe has been anxious to purchase the freedom of his wife,” Key wrote to Dolley Madison in 1810. Mrs. Madison agreed to advance the couple $200 to buy the freedom of Milley and her child, on the condition that they work for the Madisons to pay off that debt.¹¹ They struck a deal, and Joseph and Milley Bolden continued to work at the White House as free hired servants for the rest of Madison’s presidency.

The best-documented member of President Madison’s enslaved household was Paul Jennings. Jennings was ten-years-old when Madison became president and brought him to the White House to act as a footman. In D.C., Jennings encountered a substantial free black community for the first time. He witnessed historic events like the British burning of the White House and Capitol Building in 1814. When Madison’s presidency ended, Jennings returned to Montpelier, where he served as Madison’s valet. He married his wife Fanny, an enslaved woman who lived at a neighboring plantation, and despite their separation they raised a family. When James Madison died, however, Dolley Madison returned to Washington, bringing Jennings with her. When it became clear that financial troubles would require the liquidation of the enslaved population owned by Mrs. Madison, Jennings used his contacts in the free black community to get in touch with Massachusetts Senator Daniel Webster. Webster agreed to help purchase Jennings’ freedom in 1847. Jennings went on to write A Colored Man’s Reminiscences of James Madison, the first published memoir about life in the White House.¹²

Paul Jennings worked in close proximity to the Madisons for decades. “I was always with Mr. Madison till he died, and shaved him every other day for sixteen years,” he recalled in his memoir.¹³ The family knew Jennings well and clearly valued his service, but that did not stop them from exploiting his labor. When President Madison died, Edward Coles lamented that he had “died without having freed one – no not even Paul.”¹⁴ Previous presidents, including George Washington and Thomas Jefferson, had freed their manservants upon their deaths, and Coles had expected Madison to do the same.¹⁵ Jennings undoubtedly harbored the same hope. Mary Cutts described how Jennings “sighed for freedom” and attempted to run away to New York.¹⁶ Family oral histories also suggest that he used his ability to read and write to forge freedom papers for other enslaved people seeking to escape. These talents were particularly rare as most slave owners resisted the idea of educating enslaved communities, fearing that they could use these skills to escape or organize an uprising.¹⁷ After he secured his own freedom, Jennings likely helped orchestrate the attempted escape of nearly eighty enslaved individuals on board the schooner Pearl, which was thwarted by bad winds and a tip made to local slave owners.¹⁸

One of Jennings’ documented attempts at freedom reveals two additional members of the Madisons’ enslaved household – Jim and Abram. In early 1817, James Madison’s nephew, Robert Lewis Madison, added the following note in a letter to his uncle: “Capt. Eddins thinks that you ought to be apprised that when you were in Orange, your Servants Jim, Abram & Paul observed in the presence of Warrell that they never intended to return to Va. Upon being asked what they meant to do, they replyed that their were Captains of Vessels who wanted Cooks & that they would enter into their service.”¹⁹ The three men must have understood that they had a better chance of escaping from metropolitan Washington, with its close proximity to water and free territory and well-connected free black community, than they ever would back in Orange County, Virginia. With President Madison’s term about to end, they had to seize that opportunity or lose it forever.

Unfortunately for them, someone tipped off Abraham Eddins, an overseer on the Montpelier plantation.²⁰ There is no record of how President Madison dealt with this attempted escape plan, but we know Paul Jennings went back to Virginia at the end of Madison’s presidency, so the plot was likely foiled. Jim and Abram disappear after this point. Like many of the enslaved people who served in the White House, they only appeared in the written record when they were resisting a slave owner’s authority. Once that had been dealt with, they stopped appearing in Madison’s letters. Of these three, only Jennings worked closely enough with the family to be regularly mentioned in their papers. We can assume that Jim and Abram were reprimanded or punished in some way, which could have meant additional labor, physical punishment, or even sale, although there are no records to suggest that they were sold.

Sukey (possibly short for Susan), Dolley Madison’s lady’s maid, was one of Paul Jennings’ contemporaries at the White House. Like Paul, she was a teenager during her White House years. She assisted Mrs. Madison with every aspect of daily life, from bathing to dressing to styling her hair.²¹ Mrs. Madison’s letters suggest a growing tension between her personal clashes with Sukey and her outright dependence on Sukey’s labor. In an 1818 letter to her sister Anna Payne Cutts, Dolley Madison wrote that Sukey “has made so many depridations on every thing, in every part of the house that, I sent her to black Meadow last week but find it terribly inconvenient to do without her, & suppose I shall take her again.”²² She seemingly believed Sukey was stealing from her, and tried to punish her by sending her to one of the outlying quarter farms a few miles from Montpelier, but after only a week found that she could not manage without her. She acknowledged her own dependence, and how little she could do without Sukey. “I must even let her steal from me, to keep from labor myself,” she told her sister.²³

After spending her teenage years in the White House, Sukey returned to Montpelier with the Madisons and raised five children. She came back to Washington, D.C. with Dolley Madison after James Madison’s death, but the former first lady’s financial troubles soon threatened Sukey’s family. Her eighteen-year-old son Ben was sold by Madison and sent to Georgia in 1843.²⁴ The others soon followed. By 1848, all of Sukey’s children except the youngest, fifteen-year-old Ellen, had died or been sold. When Ellen found out she was going to be sold as well, she attempted to escape on the Pearl, probably with Paul Jennings’ aid. Dolley Madison, furious that Ellen had disappeared, sold Sukey to a local Washington family. Ellen was captured with the rest of the Pearl fugitives, but abolitionists raised the funds to buy her freedom and find her employment in Boston.²⁵

A few months later, Ben, who had been in Georgia for five years, penned a heartbreaking letter to Dolley Madison, encouraging her to buy him back or find another buyer in Virginia so that he could come home. “Should you be Kind enough to take me back to Virginia I Shall I can Say to You be a dutiful & Faithful Servant as long as you may live,” Ben wrote. He asked her to “Consider my unfortunate Situation away from my Relatives, Who are very near & very dear to me.”²⁶ Dolley Madison never responded. Of course, Ben did not know the rest of his family had already been sold, so the hoped-for reunion would have been impossible either way. In the end, Ben did not return to Washington until after the Civil War, when he found a job as a tour guide at the U.S. Capitol and made a living telling tourists stories about the Madisons.²⁷ He eventually bought a house on L Street, just one block away from Paul Jennings’ home.²⁸

With any research into the history of enslaved people, the greatest obstacle is the lack of definitive and comprehensive sources. President Madison undoubtedly utilized more enslaved labor working for him at the White House than those previously mentioned, but in many cases the White House connection is difficult to prove. An enslaved man named Gabriel, born in 1792, worked for the Madisons as a domestic servant and courier. Benjamin McDaniel was one of the few confirmed literate, enslaved individuals owned by the Madisons. He was about the same age as Paul Jennings.²⁹ Their ages and assigned tasks suggest that they could have been among the enslaved servants brought from Montpelier to the White House, but none of the extant records prove a definitive link. Ralph Philip Taylor, another enslaved domestic servant, was born during James Madison’s presidency. His mother worked in domestic service as well, so if she was working at the White House then Taylor may have spent the first years of his life there.³⁰

Whether or not Ralph Taylor spent his childhood in the White House, he certainly served in the President’s Neighborhood. Dolley Madison brought him to Washington to work in her house on Lafayette Square during her retirement, and he became her most trusted servant after Paul Jennings left. In fact, because of James Madison’s tenure as secretary of state before he became president and Dolley Madison’s retirement on Lafayette Square, the Madisons have deeper connections to slavery in the President’s Neighborhood than any other first family.

Because the Madison household was divided between Montpelier and Washington for so long, letters were an essential means of communication for both free and enslaved people. A few of the enslaved individuals owned by the Madisons were literate. Some letters written to Dolley Madison and even between enslaved people have survived, mostly from the last years of Mrs. Madison’s life. Sarah Stewart, an enslaved woman who remained at Montpelier when Dolley Madison retired to Washington, sent Mrs. Madison updates about marriages, children, and illnesses among the plantation’s enslaved community. When the local sheriff seized individuals enslaved at Montpelier because of court cases over Mrs. Madison’s debts, it was Sarah Stewart who conveyed the fears of those around her, many of whom were worried that they would be separated from their families. She pleaded with Madison to “make some bargain with some body by which we could be kept together.”³¹ Instead, the estate was sold to Henry Moncure shortly thereafter in 1844. Moncure purchased several of the enslaved people living at Montpelier, but others were retained by Dolley Madison or given to Payne Todd, her son from her first marriage. Many of those were later sold to a variety of buyers. The plantation’s enslaved community was permanently fractured.³²

Paul Jennings, in the last years before obtaining his freedom, wrote to Mrs. Madison as well, mostly when he was away from Washington to visit his ailing wife Fanny. Fanny died in 1844, with Paul by her side.³³ Jennings’ most remarkable letter, though, is one written directly to Sukey around the same time. It is a rare instance of surviving correspondence between two enslaved people. Jennings’ letter, addressed to “Sister sukey,” illuminates the depth and importance of relational connections within Montpelier’s enslaved community. Jennings was at Montpelier with his wife Fanny, expecting “every day to see the last of her,” but even during this personal crisis he was careful to ensure that news and greetings were passed between the enslaved community at Montpelier and those back in Washington with Mrs. Madison, many of whom had been separated from their families for months. Jennings sent his blessings to “Beckey Ellen Ralph and sister jane Bell” back in Washington and informs Sukey that “Cattey an the Boys & peater is well.”³⁴ “Cattey an the Boys” were Catharine Taylor and her sons, who were separated from Ralph Taylor while he served Dolley Madison in Washington. Similarly, “Beckey,” or Rebecca Walker, must have appreciated receiving the news that her husband Peter was doing well back at Montpelier. Because so few enslaved people were literate, it was difficult for separated families to communicate with one another, but it is clear from this letter that Paul Jennings purposely sent word to inform those separated from their loved ones.

Before the end of her life, Dolley Madison was in significant debt due to downturns in the Virginia economy and the spending of her son, Payne Todd. She sold her husband’s political papers, the Montpelier plantation, and most of the enslaved community there, and her son inherited the remainder when she died. In his will, he attempted to free those that remained in bondage after his death in 1852. However, he was in such deep debt that those enslaved people were likely sold to pay his creditors.³⁵ Beyond a few well-documented individuals like Paul Jennings and John Freeman, we know little about what happened to most of the household. As this research initiative continues, we hope to uncover additional stories about the enslaved people who worked under James and Dolley Madison during their multiple residencies in Washington, D.C.

Thank you to Dr. Elizabeth Chew, Executive Vice President and Chief Curator at James Madison’s Montpelier, and Christian Cotz, Director of Education & Visitor Engagement, for their contributions to this article.