Eliza McCardle Johnson: Conflicting Memories and Vanishing Evidence of the Enslaved Past

In 1980, Margaret Johnson Patterson Bartlett, great-granddaughter of First Lady Eliza McCardle Johnson and President Andrew Johnson, gave an oral interview at Harpers Ferry about the history of her family. Alluding that her ancestors wanted to keep certain family secrets hidden from the public, Mrs. Bartlett recalled when her father sold documents to the Library of Congress in 1904: “My daddy [Andrew Johnson Patterson] was the kind of person that said, ‘let the dead bury the dead.’ Why bring it to the front?”¹

Whether it was intentional or incidental, specific details and stories surrounding the Johnson family and the people they enslaved rarely appear in historic records. Included in the possible destruction of records were sources that may have shed light on Eliza McCardle Johnson’s participation in American slavery. However, the Papers of Andrew Johnson, family letters, and recollections from freed people allow us to construct a narrative surrounding Eliza’s involvement in enslavement.

Eliza McCardle Johnson was one of several first ladies who became an owner of enslaved individuals later in life through marriage.² Eliza knew from the moment when aspiring tailor, Andrew Johnson, first came to Greeneville that someday, “mark it,” she would marry him.³ They were married on May 17, 1827, in Greeneville.⁴ Mordecai Lincoln, a relative of Abraham Lincoln, officiated their wedding.⁵

Eliza worked hard to enhance her husband’s public speaking abilities and encouraged his political aspirations. Andrew valued his wife’s opinion and Eliza often read and advised him on his speeches. With her support, Andrew began his political career as the alderman of Greeneville and eventually became mayor.⁶ Andrew was elected to the Tennessee State Legislature in 1835.⁷ At the same time, Andrew’s newfound political and social status paved the way for his family’s involvement in American slavery.

For reasons unknown, Eliza decided early in Andrew’s political ascension to avoid public and social roles commonly associated with most politician’s wives. Instead, she concentrated her efforts on her home, supervising the family business and raising several children.⁸ Managing the bonded, or enslaved, members of her household fell within Eliza’s domestic duties.⁹

Like many southerners in the nineteenth century, the Johnsons acquired enslaved servants to work in their home. The decision to purchase enslaved people most likely originated from Andrew’s need to exhibit his social and political ascendency rather than the need to contribute to his household economy.¹⁰

Andrew first purchased a young boy named Sam in 1842 and later Sam’s half-sister, Dolly. According to Dolly’s son, William Johnson, Dolly sought out Andrew in a crowd of prospective buyers at a slave auction and asked him to purchase her.¹¹ However, Mrs. Bartlett recalled a different version of the event that was most likely retold according to her family’s oral tradition and cast Johnson’s actions in a more favorable light. Mrs. Bartlett recollected that Andrew “went to a nearby town to buy slaves…and bought a sister and brother. On the day of the sale, there were two children that cried because they wanted to come too, and to show that Andrew Johnson was kind, he took the family and brought em’ all over. That’s just a little insight into his kindness.”¹²

Sam later married a woman named Margaret. They had nine children, three of whom were born into slavery. Dolly also gave birth to three children, Liz, Florence, and William, who also assumed their mother’s enslaved status, thereby naturally increasing Johnson’s investment in human capital. The paternity of Dolly’s children remains unknown. According to the census Dolly never married; although historians speculate that Andrew Johnson or one of his sons may have fathered Dolly’s children.¹³ Click here to learn more about the formerly enslaved household of President Andrew Johnson.

Fourteen years after he purchased Dolly, Andrew purchased a thirteen-year-old child named Henry on May 6, 1857. Henry’s bill of sale illustrates the all-too-common denial of humanity associated with American slavery in that the document describes Henry as “sound, healthy, sensible and a slave for life.”¹⁴ It is estimated that Andrew enslaved up to ten individuals during his lifetime.¹⁵

Mrs. Bartlett’s interview portrays Andrew and Eliza as “benevolent” slave owners, a fictitious standard that many southern slave owning families espoused to mask their involvement in America’s brutal slave regime. In Roll, Jordan Roll, historian Eugene Genovese argues that benevolence grew in popularity among enslavers who desired to portray their status in a more favorable light during the growing abolitionist movement the 1830s.¹⁶ Andrew was among many enslavers who assumed a “paternalistic” role in managing enslaved individuals. However, Andrew never escaped the cruelty at the foundation of the institution. His contradictory position as a slaveowner serves as a prime example of what historian, Erskine Clark describes as, “a story marked by bitter irony of…benevolent impulses becoming ideological supports for deep oppression.”¹⁷ In an interview with journalist Ernie Pyle, Dolly’s son William later recalled that as a child Andrew held him inappropriately while he was sitting on one of Andrew’s knees. His sister sat on his other knee and Andrew rubbed their heads together and laughed.¹⁸ Despite Andrew’s attempts at bestowing “fair” treatment, William later reflected that “[a]ny man would rather be free than a slave.”¹⁹

What Eliza thought of her husband’s decision to purchase enslaved persons and his treatment of those individuals bonded in their household is limited due to meager records.²⁰ However, a Johnson family letter perhaps best demonstrates Eliza’s direct involvement with her enslaved household. Southern enslavers often utilized the hiring out system; a process by which they would hire out their enslaved laborers to neighboring farms and households and collect their wages for profit.²¹ On one occasion, Eliza hired out Sam to chop wood for a neighbor in Greeneville. To recoup his bodily autonomy, Sam demanded payment from Eliza for his services. Sam exclaimed that he would “be damned” to work without pay in a remarkable display of enslaved resistance. Eliza coldly responded that, “if he was ready to pay others as he was to collect, he would do better.” Eliza’s son, Charles Johnson, may have invoked further disciplinary action through intimidation in defense of his mother. In a tactic that many enslavers used to punish and threaten resistant enslaved persons, Charles proposed Sam should be “sold away.”²² Though the dilemma between the parties was eventually resolved, the correspondence illustrates that Eliza exercised some control over enslaved lives and profited from their forced labor.²³ The incident demonstrates that Eliza was not only a complicit figure in human captivity but was an active participant in preserving and perpetuating the institution of slavery in America.

Despite his contrary position on slavery, Andrew remained politically loyal to the Union. When the Civil War began in 1861, Andrew was the only southern senator and slave owner to support the Union. When President Lincoln’s Emancipation Proclamation freed all enslaved persons in areas of open rebellion in the United States on January 1, 1863, Andrew followed suit. On August 8, 1863, Andrew released all enslaved persons in his household from servitude. It is possible that he instructed Eliza to inform them of his decision. William Johnson later recalled, “one day Mrs. Johnson called us in and said we were free now. She said we were free to go, or we could stay if we wanted. We all stayed.”²⁴ On October 24, 1864, Andrew proclaimed unconditional freedom to every person in Tennessee.²⁵

That same year, the National Convention in Baltimore announced that Andrew would be President Lincoln’s running mate in his reelection campaign. Lincoln and Johnson were elected as President and Vice President of the United States that November. Although Andrew was honored with his new position, he and Eliza were hesitant to move to Washington. However, a quick turn of events hastened their move to the capital.

President Lincoln was shot at Ford’s Theatre on April 14, 1865, and died the following day. After he took the Oath of Office at the Kirkwood House Hotel, Andrew sent an urgent telegram to his son-in-law and Martha’s husband, Judge Patterson, informing the entire family that they should begin preparations to move to Washington.²⁶ In the wake of President Lincoln’s death, Eliza dreaded her husband’s assumption of the presidency. She lived in constant fear that her husband would be assassinated as well. A letter from Eliza’s daughter, Martha, described Eliza as “almost deranged” with worry, illuminating the psychological and emotional toll these traumatic events had on her mental health.²⁷

In May 1865, the president, Eliza, their two sons, Robert and Andrew, their eldest daughter, Martha Patterson and her husband, David Trotter Patterson, a senator from Tennessee, and their two children, as well as their widowed daughter, Mary Stover, and her three children all moved into the White House.²⁸ Martha Patterson described her family as, “…plain people, from the mountains of Tennessee, called here for a short time by a national calamity. I trust too much will not be expected of us.”²⁹

During President Johnson’s term, the entire extended family lived closely in the family quarters of the Second Floor. The family moved most of the furniture up from the Green Room to make a guest room their sitting room. It is here where the Johnsons, Pattersons, and Stovers spent most of their time talking, reading, and sewing. Although Eliza disliked her new role as first lady, she genuinely appreciated the time when her entire family lived together in the White House.³⁰

The enslaved past continued to pulsate within the White House walls as President Johnson brought some of his freed servants to work as paid employees.³¹ Among the freed servants to accompany the president and his family to the White House was Henry Johnson. Henry’s obituary in the Knoxville Miscellany states that he was “a well-known colored man, in the service of President Andrew Johnson at the White House.”³² Florence, Dolly’s second daughter, who the president emancipated when she was about thirteen, also worked in the Executive Mansion as a maid. The president sent Florence to culinary school to advance her domestic skills during his term.³³ Johnson hired Florence as the family cook after his presidency.³⁴ Her obituary stated, “she was a woman who made the most of her opportunities, and with the funds at her disposal helped many worthy young persons of her race in their efforts to go to college.”³⁵

Much like Eliza’s duties managing enslaved servants in her Greeneville home, the first lady’s involvement with freed people of color working in the White House is largely unknown. According to the editors of the Andrew Johnson papers, only a few family letters are extant in the collection. This includes limited correspondences between President Johnson and his wife. In the absence of what must have been a considerable exchange of letters that might have cast light on Eliza’s role in the domestic scene of the White House, her treatment of freed servants during her tenure as first lady remains shrouded in “obscurity and sentimentality.”³⁶ Though this leaves much to be desired, other scholars have suggested that Eliza was close to White House staff and treated both Black and White servants as “members of the household.”³⁷ Eliza watched over the health and happiness of the staff, and often sent notes, flowers, and delicacies when they were sick or in trouble.³⁸

Eliza’s transition into her new role as a first lady was quite difficult. Finding herself once again in the public eye, Eliza shied away from all social engagements customary of the president’s wife. For lack of evidence, we can only speculate why Eliza avoided her highly public position. Perhaps she sought to evade intense public criticism imposed on her predecessor, Mary Lincoln, or negative critiques over her husband’s controversial presidency. It is also possible that Eliza felt she lacked certain social talents required of first ladies.³⁹ Moreover, Eliza’s tuberculosis diagnosis may have influenced this decision.⁴⁰

Whatever her reasoning, Eliza willingly delegated many of her hostess responsibilities to her daughters, Martha and Mary.⁴¹ Ultimately, Martha’s familiarity with Washington from her years at a private school in Georgetown, her frequent visits with former first lady, Sarah Polk, and her soft-spoken yet sociable demeaner made Martha a natural fit for serving as hostess at the White House during her father’s term.⁴² Although Martha acted as a surrogate first lady during the White House renovations, Eliza made a few of her intentions known. For example, she requested that the paint on the East Room ceiling be of the best quality.

Although Eliza refrained from many of her first lady duties, she continued to exert a strong influence over her husband’s personal and public role as president. Eliza carefully presided over the president’s wardrobe and looked over her husband’s bedroom to make sure everything was in order the way he liked it.⁴³ As she did in his previous political positions, Eliza continued to monitor the newspapers, clipping articles for her scrap books, and informed him of current events.⁴⁴ Oral tradition from White House observers said the Johnsons “seemed as to souls and minds merged as one.” When his temper raged, Eliza cooled it by a simple hand on his shoulder saying, “Now, Andy…” A chronicler asserted, “I should not wonder if Andrew Johnson did not consult his wife…more than he did any fellow statesman.” A Greeneville resident called Eliza the president’s “counterpart,” while a Reconstruction expert said she “remained his most trusted counselor” during the congressional trials.⁴⁵ During his impeachment trial, Eliza followed the proceedings intensely and he would visit her each morning to seek her advice.⁴⁶

Despite the end of the war and federally mandated emancipation, remnants of slavery continued to permeate national politics throughout President Johnson’s term. Historians have argued that attempts to heal the nation and secure rights for newly freed African Americans during Reconstruction were largely a failure. This foundering was, in part, due to President Johnson’s racial biases, his unwillingness to address “Black Codes,” and his veto of the Civil Rights Act of 1866.⁴⁷

The extent to which Eliza advised her husband on Reconstruction era politics is unclear. Although the intimate details that may have illustrated Eliza’s influence over husband’s political decisions have now been long destroyed, Eliza’s clipping and reading of newspapers, as well as accounts of the president visiting his wife to discuss political matters, offers reasonable evidence that the first lady exerted some impact in what the future held for people of color in America.

After Johnson’s term as president, the family returned to Greeneville, Tennessee, much to Eliza’s delight. Eliza never learned to love living at the White House and once told a staff member, “It’s all very well for those who like it, but I do not like this public life at all. I often wish the time would come when we could return to where I feel we best belong.”⁴⁸ Although Eliza’s health remained poor, she outlived her husband by six months. Eliza died on January 15, 1876, and was buried next to her husband in the Johnson Family Cemetery. An article in The Tennessean on January 20, 1876, reported Eliza’s death:

“If Mrs. Johnson had been as well able to act as the adviser and guide of her husband during the last part of her life as she was during the first part of it, he might have been saved from some of the errors into which he fell. But from first to last, she was to him a wife whom he always loved…who assisted him in entering and aided him in pursuing the paths that led to eminence, who bore her sufferings in patience and resignation, and who lived a life that will make her remembered among American women.”⁴⁹

Whether or not The Tennessean was correct in reporting that Eliza’s influence over President Johnson was enough to save him from his own racial prejudices and biases throughout his tenure is true, answers to those questions lie in letters now lost to posterity. When Martha Johnson Patterson died in 1901, the personal papers of President Andrew Johnson passed to the possession of her son, Andrew Johnson Patterson. He sold the collection to the Library of Congress. However, the president’s grandson was extremely particular of which documents were selected.⁵⁰

Mrs. Bartlett described the process by which her father selected certain papers available for donation and public consumption:

“I’ll be very frank with ya and tell you that when daddy sent an awful lot of letters to the Congressional library – I can see him now – he was upstairs in a vacant room working, and things that he thought was nobody’s business, he’d burn them. And I’m glad he did. And here are some of these historians who are grabbers and like to feature that stuff – the fire has consumed them! And I’m glad of it. It’s family. Private!”⁵¹

Mrs. Bartlett’s admitted fear that historians may misconstrue certain letters in the collection leaves much to the imagination. Was she referring to racist underpinnings that appeared in correspondence written by her ancestors? Perhaps intimate letters between family members revealed the identity(ies) of the father of Dolly’s offspring. In any event, the destruction of evidence that may have illuminated Eliza’s ties to American slavery contributes to what historian David Blight refers to as “historic amnesia.”⁵² In choosing what Americans would remember and forget about his family’s memory, Andrew Johnson Patterson was intentional about protecting the Johnson family legacy.

If in fact Andrew Johnson Patterson did eliminate any sensitive information pertaining to enslaved and freed servants, this decision contributes to what historians refer to as the Lost Cause mythology in historic memory. In downplaying the tragic and brutal realities of human bondage for the sake of reconciling a painful past, the Lost Cause became a central tenant of Jim Crow era politics.⁵³ It would take America almost another hundred years to completely dismantle widespread racial segregation and discrimination institutionalized by Jim Crow laws when President Lyndon B. Johnson signed the 1964 Civil Rights Act.⁵⁴

Eliza McCardle Johnson’s life and her tenure as first lady is a reminder of the repeated challenges faced when working with records of enslavers. Available sources illuminating the lives of enslaved and freed people are sometimes just as scarce as those that explain the life of the enslaver because memories of that past were intentionally dismantled. Nevertheless, a fuller understanding of the Executive Mansion’s ties to American Slavery are achievable through constructing a narrative from fragmentary bits of evidence from a shattered past. In doing so, it is evident that the contributions and small victories of underrepresented people of color still constitute an impactful legacy in the history of the White House.

Thank you to Kendra Hinkle, museum specialist at the Andrew Johnson National Historic Site, for her research assistance with this article.