Unraveling the Dolley Myths

The spirit of Dolley Madison is everywhere in the blocks surrounding the White House. She lived at 1333 F Street, in the White House, in the Octagon House, and in a former row house in the 2000 block of Pennsylvania Avenue. But nowhere in Washington do visitors better imagine Dolley than at her residence across from Saint John’s Church.

Here is where the widowed Mrs. Madison came to spend her final years. One imagines her sitting on the porch or waving from a window. One can almost see the White House, or friends arriving to escort her to a session of Congress. The house, with its yellow exterior and shiny painted shutters, seems to echo Dolley Madison’s legendary charm.

However, the house is not what it seems. It was probably not yellow when Dolley Madison lived there, and the entrance faced the park, not H Street, as it does today. The home Dolley Madison knew was only two stories high and stood alone, and the first story windows did not reach the floor.¹ Yet onlookers seem content to imagine that the house is exactly as Dolley Madison saw it, willing to ignore the “what isn’t” in order to create a mythical “what is.” The same can be said of the home’s famous mistress. No historical figure has been more cloaked in “what isn’t” than Dolley Payne Todd Madison.

Dolley Payne was born on May 20, 1768, in an America on the brink of change. Her Quaker mother, Mary Coles Payne, had a Virginia genealogy that included such notables as Patrick Henry. Her Virginian father, John Payne, who converted to Quakerism after his marriage, came from more common stock. The family struggled at plantation farming both in Virginia and in Dolley’s birth colony of North Carolina. Finally, in keeping with the newest Quaker mandates, he sold his slaves and moved his wife and eight children to Philadelphia, where he entered the starch-making trade. Dolley was 15.

In 1790, Dolley Payne wed a good-looking, successful young Quaker lawyer named John Todd. Three years later, a yellow-fever epidemic devastated Philadelphia, taking with it the lives of both her husband and one of her two young sons. Eleven months a widow, Dolley Payne Todd remarried, this time to the distinguished Virginian congressman, James Madison, seventeen years her senior, small and quiet spoken, but by her own account, “the Man who of all other’s I most admire.”² From 1797 into 1801, with her husband out of public office, Dolley Madison made her home at Montpelier, the Madison family plantation in Orange County, Virginia. There she might have remained in obscurity among pastures and country pleasures, but in 1801 her husband accepted the position of secretary of state under newly elected president, Thomas Jefferson. With that, the Madisons moved to Washington City, where Dolley Madison, with no public mandate to guide her, no official office to propel her, but instead through love of husband, strength, personality, and grace, would come to reign.

The myths about Dolley Madison started early, while she was still alive, and they were not always flattering. “Your friend Mrs. Madison,” Senator Samuel latham Mitchill wrote his wife in 1808, “is shockingly and unfeelingly traduced in the Virginia papers. The attack grows out of the approaching election, in which her husband is so prominent a character.”³ rumor had it, among other things, that Dolley Madison and her sisters had slept their way through the city, and that a young Dolley “served out the liquor at the bar of her father’s [nonexistent] tavern.”⁴ Those tales faded quickly into the background. More durable, however, has been the creation of myths with only good intentions—inconsequential myths about cupcakes and ice cream, a bigger myth that exaggerated feats, and more seriously the myth of Dolley Madison was Thomas Jefferson’s official first lady.

A Montpelier docent once remarked to this author that visitor misconceptions about Mrs. Madison center mostly on cupcakes and ice cream. The cupcake myth began in 1937 with the founding of the Dolly Madison Bakery, a national brand that aspired to create snacks, including cupcakes, “fit for a socialite like Madison yet affordable for everyone.”⁵ The ice cream myth, usually in the form of Mrs. Madison having been the first to serve ice cream in the White House, is partially true. Dolley Madison did serve ice cream in the White House, quite frequently, but her doing so was not the cultural landmark that firsts usually indicate. Thomas Jefferson also served ice cream at his presidential dinners, and for all we know, so did Abigail Adams during her short residency at the executive mansion in the winter of 1800–1801.

One of the more steadfast myths about Dolley Madison has a wartime first lady “fleeing the Executive Mansion clutching the Gilbert Stuart portrait of George Washington.”⁶ In a letter she wrote her sister amid the turmoil and panic of the British invasion, August 24, 1814, Dolley Madison gave a different version of events. “Our kind friend, Mr. Carroll, has come to hasten my departure, and is in a very bad humor with me because I insist on waiting until the large picture of General Washington is secured, and it requires to be unscrewed from the wall. This process was found too tedious for these perilous moments; I have ordered the frame to be broken, and the canvass taken out; it is done—and the precious portrait placed in the hands of two gentlemen of New York, for safe keeping.”⁷

By 1831, though, the fable was cemented. Wrote one New York visitor that year, “We drove to the palace, entered . . . a drawing-room, in which is a fine full-length picture of General Washington. When the British came here in the last war, the President was obliged to fly. His wife, Mrs. Madison, cut the picture from the frame and took it with her—the only article she took!”⁸

Paul Jennings, a former Madison slave, attempted to correct the myth in his 1865 published reminiscences. “It has often been stated in print, that when Mrs. Madison escaped from the White House, she cut out from the frame the large portrait of Washington . . . and carried it off. This is totally false. She had no time for doing it. It would have required a ladder to get it down . . . . John Susé [Jean-Pierre Sioussat, the doorkeeper] . . . and Magraw, the President’s gardener, took it down and sent it off on a wagon, with some large silver urns and such other valuables as could be hastily got hold of.”⁹ However, the image of a brave Mrs. Madison fleeing the White House, the George Washington portrait secure in her arms, was too delightful to die. Firsthand accounts notwithstanding, the myth has persisted into the twenty-first century.¹⁰

Mrs. Madison’s letter describing the events of 1814 became part of a ten-page biographical sketch written by Margaret Bayard Smith for the National Portrait Gallery and published in 1836. That biography contains the seeds of the most significant Dolley myth, the belief that Dolley Madison was Thomas Jefferson’s surrogate first lady. In discussing Dolley Madison’s years as wife of the secretary of state, Smith wrote that “the president’s house was the seat of hospitality, where Mrs. Madison always presided, (in the absence of Mr. Jefferson’s daughters,) when there were female guests.” Although Smith quickly added, “After the president’s, the house of the secretary of state was the resort of most company,” her initial statement presumed a constant presence by Dolley Madison in the Jefferson White House, and it had that affect on Madison family lore.¹¹

The National Portrait Gallery biography received little press, but in 1886, a small volume of letters appeared that further validated the surrogate first lady myth. Memoirs and Letters of Dolly Madison: Wife of James Madison, President of the United States was edited by Lucia Beverly Cutts, a grandniece of Dolley Madison, born two years after her aunt’s death. Among other flatteries, Cutts told her readers that “Mrs. Madison, aided by her sister, usually presided at the White House and was much depended upon,” and further in the book that Dolley Madison “was obliged then and always, during [the Jefferson] administration, to take the head of the table.”¹² As evidence of this obligation, Cutts related the story of a single Jeffersonian dinner at which Dolley Madison was present and further produced two communications from Jefferson. Written during the first months of his presidency, the notes requested that either Dolley Madison or her sister dine with him in order “to take care of female friends expected.”¹³ Despite the limits of the Cutts evidence, her book quickly became the standard citation among both historians and popular writers for confirming that Dolley Madison served as President Jefferson’s surrogate first lady. Thomas Jefferson, however, left his own set of records, and from these we get a different picture.

During the last five years of his presidency, Jefferson kept a running list of everyone invited to his Washington table, excluding houseguests and those who resided at the Executive Mansion. For each dinner, the president recorded the date, sometimes the day of the week, and a list of guests by surname. His technique was meticulous. Guests invited, but who declined, were listed with a strikethrough (for example, “Johnston”). He indicated the inclusion of women guests occasionally with the term “aux” (auxiliary), occasionally with the moniker “Mrs.” or “Miss,” but more often he indicated companion guests, female or not, before or after the surname, for example “2 Gallatin” or “Madison 3.”¹⁴

The guest records show that of the nearly four hundred dinners Jefferson gave between November 1804 and March 1809, less than 20 percent included women guests. For example, during the 1804–5 congressional session, the president gave sixty-three dinners, fifty-six of which were attended solely by men. At those dinners, no hostess was necessary. Jefferson also recognized his role in Washington City’s social circle, however, and his obligation to dine the few wives who followed their legislative husbands to the capital. Thus, during this same 1804–5 congressional session he invited ladies to his table seven times. Of those seven dinners, the Madisons attended three. For another three of the seven dinners, other cabinet wives, but not Dolley Madison, attended.¹⁵ This tabulation lends credence to Jefferson’s 1804 statement that, when needed, he asked “one of the ladies of the 4. secretaries to come & take care of my company.”¹⁶ At the seventh dinner, with no cabinet wife in attendance, Jane Gibbon Duvall, as wife of the comptroller of the treasury, most likely performed what Jefferson called “the honors of the table.”¹⁷

During the summer season following the 1804–5 congressional session, Jefferson held twenty more dinners, of which eight included women guests. Dolley Madison attended no more than three of those dinners. Other cabinet wives attended an additional three. At the remaining two dinners, and using as a guide their placement on the guest list, Margaret Tingey, wife of the superintendent of the Washington Navy Yard, most likely assisted at one dinner, and Anna Maria Thornton, wife of the superintendent of the patent office, at the second.

During the winter of 1805–6, Jefferson’s daughter, Martha Jefferson randolph, occupied the Executive Mansion, and either she or one of her teenage daughters would have presided at dinners attended by female guests. At other times, the dinner records reveal patterns similar to 1804–5. Each year shows Dolley Madison presiding over more Jefferson dinners than any other cabinet wife, with Hannah Nicholson Gallatin, the elegant New York wife of the secretary of the treasury, coming in a distant second. However, the number of dinners remains small and Dolley Madison’s presence is inconsistent.

There is no reason to think that Jefferson occasionally forgot to list Dolley Madison, or that she went unlisted when unaccompanied by her husband. Jefferson had methodical record-keeping skills and no aversion to listing unescorted women guests. In addition, period descriptions of the Jefferson dinners—and there are a number of them—do not mention Dolley Madison’s attendance except at those confirmed in the records. All indications are that President Jefferson recorded every dinner guest except for residents and houseguests at the Executive Mansion, and the Madisons were neither of those after 1801.

In summary, during Jefferson’s last five years as president, Dolley Madison attended less than 10 percent of the dinners Jefferson held at the Executive Mansion. She may have assisted at more dinners earlier in his administration, but period journals and diaries do not indicate a dramatic increase in her attendance prior to the dinner records. The numbers also show that Dolley Madison assisted at dinner more often than did other cabinet wives, but even as the preferred choice, the role was limited and nonexclusive, and it does not warrant the title of either official hostess or surrogate first lady. When Dolley Madison walked onto the presidential grounds, she did so as a guest and a friend, not because she was executing a standing obligation.

Even in 1886, without access to the Jefferson dinner records, the Cutts claim was unstable. Henry Adams, in his exhaustive history of the Jefferson administration, included detailed commentary on Jefferson’s political use of presidential hospitality, yet in more than a thousand pages of print, he never gave Dolley Madison a passing credit. Period newspapers did not routinely cover social events, but coverage of the annual New Year’s reception spoke only of Jefferson as the host. Although such lack of acknowledgment by both Henry Adams and the newspapers might be considered typical of a century largely incapable of appreciating a woman’s worth outside the home, correspondences and journals of the Jeffersonian period also fail to assign Dolley Madison the role of presidential hostess. Washington society wrote of Dolley Madison’s conversation, her charm, her wardrobe, and her enjoyable companionship. At times, they even spoke of her warm presence at a Jefferson dinner. Never did they speak, however, of her as having maintained a constant role at the Jefferson table. Even Margaret Bayard Smith, her biographical sketch aside, made no such comment in her book, The First Forty Years of Washington Society.¹⁸

Besides being mistakenly assigned the role of surrogate first lady, Dolley Madison is sometimes identified solely on the strength of her firsts, most of which tell us little of her worth to history and many of which, if not myths, are only partial truths. “First to attend a husband’s Inauguration,” may be true, but we know from letters that Martha Washington was in Philadelphia the day of her husband’s second inaugural and may well have been at the ceremony. “First president’s wife to be called First Lady” is reported to come from a eulogy President Zachary Taylor gave at Dolley Madison’s funeral, but primary sources are lacking and some insist Lucy Webb Hayes wears that title, others say Harriet Lane, and still others, Julia Dent Grant. “First inaugural ball” also needs qualifying. Dolley Madison was the first presidential spouse to organize an inaugural ball; it was the first inaugural ball held the day of the inauguration, and the first inaugural ball in Washington City, but James Madison was not the first president to have an inaugural ball. New Yorkers gave one for George Washington in 1789.¹⁹

“Firsts” have one of two good purposes: either they serve as a cultural landmark, or they indicate a societal barrier having been broken (as in “first woman president”). The Madison firsts, whether true, partially true, or mythical, are mildly interesting as cultural landmarks, but are not hard-fought barrier breakers. Moreover, they tell us little about the woman.

Dolley Payne Madison was, by all accounts, exceedingly popular in Washington society, both as a cabinet wife and as a presidential spouse. However, her Wednesday night receptions at the White House made her famous. These informal entertainments afforded the “opportunity of seeing all [those] whom fashion, fame, beauty, wealth or talents, have render’d celebrated.”²⁰ The weekly event was a republican blending of the formal gentlemen levees held by Presidents George Washington and John Adams and the less formal weekly receptions held by their wives. Washington Irving, after happily attending one of Dolley Madison’s Wednesday night squeezes, described going from the “dirt & darkness” of Washington City streets “into the blazing splendour of Mrs. Madison’s Drawing room. Here I was most graciously received—found a crowded collection of great and little men, of ugly old women, and beautiful young ones—and in ten minutes was hand and glove with half the people in the assemblage. Mrs. Madison is a fine, portly, buxom dame—who has a smile & pleasant word for every body.”²¹ In 1817, the Republican politician James C. Jewett of Maine wrote a friend that he expected to attend the next presidential levee for “the pleasure of seeing the accomplished Mrs. Madison, who presides, it is said, better than any other woman can in the country.”²²

Victorian America loved Dolley for the same reasons Irving and Jewett did. The country recognized in her a gracious woman of society who gave delightful entertainments that benefited both her husband’s career and the nation’s image. As deemed appropriate for a woman of that time, she did not promote animosity; she did not debate political issues. She loved fashion and dressed for her role in society, yet delightfully, she wore her clothes as she did her manners, a bit larger than life, slightly provincial, slightly provocative, always inviting, and never with the intent to promote jealousies. She made the White House her stage and decorated it in the same style as her wardrobe and her manners. She took the time to remember names, and when speaking with guests, made them feel welcome and special. “We remarked the ease with which she glided into the stream of conversation,” wrote one admirer. “In the art of conversation she is said to be distinguished, and it became evident. . . . We have not forgotten how admirably the air of authority was softened by the smile of gayety: and it is pleasing to recall a certain expression that must have been created by the happiest of all dispositions,—a wish to please, and a willingness to be pleased. This, indeed, is to be truly good and really great.”²³

Modern America is less taken with the art of conversation than was that admirer or those of his generation. Instead, recent historians are looking at Dolley Madison anew, reexamining her importance in terms they can better appreciate, her political acuity.²⁴

Dolley Madison certainly had the heart of a politician in the way she remembered names, dressed to be noticed, and, as with all good politicians, charmed everyone in her presence. She made her drawing rooms as inclusive as nineteenth-century propriety would allow, welcoming everyone, sniped one Federalist wife, down to “the shoemakers and their wives.”²⁵ She took an interest in her surroundings, and her surroundings were political. Louisa Catherine Adams recalled that Mrs. Madison not only exhibited “great influence in society,” but “had considerable interest in the political world which she was said to use with much discretion.”²⁶ She was skilled at keeping the conversation light and flattering to her listener. She never ruffled oppositional feathers. A visiting New Yorker agreed. “[There] is something very fascinating about her,” she wrote, “—yet I do not think it possible to know what her real opinions are[;] she is all things to all men.”²⁷

Dolley Madison also had the instincts of a politician. Although she loved society, and would have filled her house with guests whatever her husband’s profession, she recognized that her Wednesday night drawing rooms served a valuable political purpose that put Madison and his administration in their best light. As the historian Edith Mayo notes, Dolley Madison understood “that entertainment was a venue for political lobbying” and that the White House was “a stage from which to convey an image of power, cultivate political loyalties, and project dignity and authority.”²⁸

Outside of the drawing room, Dolley showed other political attributes. She was visible throughout the city. She gave and accepted gifts with delight (a rarely recognized, but particularly important skill for politicians who often rely on the pocketbooks of others, and the giving and taking of favors). like many a politician, she used her considerable influence for patronage, actively locating government positions for family and friends—among them a consul position in Tripoli for her less than reputable brother, John C. Payne, and a clerkship in Washington for a nephew by her first marriage, Samuel P. Todd.²⁹ Dolley Madison even worked to help push through legislation. In A Perfect Union, Catherine Allgor speaks of Dolley writing her sister in May 1812 that brother-in-law Richard Cutts needed to return to Washington immediately, “just in time—not a moment too soon, it is supposed, to give his vote for War.”³⁰

However, one must proceed carefully in politicizing Dolley Madison. She wanted Richard Cutts back to vote for the war because she fervently supported her husband’s politics, and because her brother-in-law’s absence from the city was beginning to reflect badly on Madison. “If Mr. Cutts does not come,” she wrote, “it will be a disadvantage to him as well as to his party—some of them have reproached him already.”³¹ As first lady, Dolley Madison used her influence to obtain positions for friends and family because they were, well, friends and family. She held drawing rooms because that is what prominent society women did in that period, for several reasons, not the least being to maintain their position and support their husbands. First ladies Martha Washington and Abigail Adams both held weekly receptions for those very reasons, as did notable women throughout the capital before, during, and after the Madison administration. However, as mistress of the White House, Dolley Madison just had a grander stage and a broader responsibility than most society women, which, combined with her faculty for entertaining, gained her national recognition. Catherine Allgor explains it this way, “Dolley and women like her . . . were the essentially conservative wives, mothers, and daughters of political families, and they saw their work as furthering their families’ interests.”³² As a politician, Dolley Madison worked for a party of one, her husband, James.

For too long Dolley Payne Madison has been wrapped in myths and half-truths meant to promote her importance in American history, but which, in actuality, have only confused her contributions. The twenty-first century is left to reexamine Dolley Madison more faithfully, to admire and assess her political acuity, to appreciate her skill at blurring the lines between politics and presidential hospitality, and most importantly to better understand the realities behind her decades-long influence on Washington City’s unique political culture.