James Madison's Brother Willey

"I am much at a loss how to dispose of Willey,” James Madison wrote to his father on December 8, 1779. “I can not think it would be expedient in the present state things to send him out of the State.”¹ Then a member of Virginia’s Council of State, James Madison was referring to his eleven-years-younger brother, William Madison, and plans for William’s future education amid the chaos and confusion of the Revolutionary War. William Madison was born in Orange County, Virginia, on May 1, 1762, the youngest of four sons of James Madison, Sr. and Nelly Conway Madison.²

William Madison would always live in the shadow of his famous brother. Few people today know that President Madison even had younger brothers. William Madison possessed neither the intellectual firepower nor philosophical curiosity of his elder brother, but William was no wastrel or slouch. He established his own record of public service, first in the Virginia House of Delegates and then by achieving high rank and distinction in the War of 1812.

William Madison probably received some formal education from the schoolmaster Thomas Martin, who lived with the Madisons. Martin had prepared James Madison for the College of New Jersey (now Princeton University). When James Madison left for college, his brother William was only 7 years old. By 1773, at the age of 11, he was enrolled at the college preparatory school at Princeton. But when the Revolutionary War came to New Jersey, William Madison was removed and sent to Virginia, first to Hampden-Sydney College in 1776–77, and then to the College of William and Mary. However, because by 1779 William and Mary abolished its grammar and divinity schools in favor of higher college level courses, William Madison’s parents enrolled him at Walker Maury’s academy in Williamsburg.³

William Madison later returned to William and Mary in 1780 at age 18, probably staying in Williamsburg with his second cousin and the college president, the Reverend James Madison. However, William Madison appears not to have completed his studies, as Rev. Madison informed James Madison on November 18, 1781, that “yr brother has not returned since the first invasion.”⁴ Sometime in early 1781, William Madison enlisted as a soldier in one of Virginia’s state units but was later commissioned a lieutenant in what became Colonel Charles Dabney’s Virginia State Legion formed in 1782. William Madison was in the recruiting service but later served in the legion’s infantry and artillery units. The legion was at Yorktown, where he witnessed Lord Cornwallis’s surrender. His military experiences would prepare him for a lifelong career of service with the Virginia militia, one aspect of public service that his more famous brother avoided.⁵ In 1828, William Madison applied for and received a Revolutionary War pension based on his service in the war. His pension file includes little supporting detail about his service, but there are affidavits as to his wartime activities from prominent citizens of Orange and Culpeper. Interestingly, there are none from his brother.⁶

William Madison returned to Montpelier until the war was officially over. He remained undecided on a career. He gave some thought to the law, a study that James Madison himself had taken up for a time before dismissing it. A plan that would send young William to study law under Thomas Jefferson was apparently executed. Jefferson confirmed the plan in a letter to James Madison on March 24, 1782, when he referred to William as “your brother who is about to study the law in my neighborhood.”⁷ Whatever time William Madison may have spent with Jefferson pursuing the law, it was presumably brief, as Jefferson would soon leave and take up his responsibilities as peace negotiator in Great Britain.

Little is known about William Madison’s professional activities at this time. We do know that on December 20, 1783, he married Frances Throckmorton (1765–1832) from Culpeper County. The marriage was a fruitful one, as eleven children were born to the Madisons between 1785 and 1808. Although most of their children survived adolescence, all but two died before their parents, victims of the tuberculosis (consumption) that afflicted the Madison family.⁸ During the 1780s, William Madison probably lived and farmed on land his father owned in Culpeper County, as William’s name appeared on the personal property lists for that county during that decade. After Frances died in 1832, William co-habitated with Nancy Jerrell whom he later married in 1834, but not before fathering an illegitimate child, John “Jack” R. Madison.⁹

In 1794 James Madison Sr. provided William Madison with 1300 acres that lay in the new county of Madison, about 8 miles from Montpelier. William named his plantation Woodberry Forest.¹⁰ It is possible that Thomas Jefferson had a hand in the fundamental design of the main house. James Madison on May 9, 1793, had asked Jefferson for some ideas about the design for his brother’s house. On May 19, Jefferson returned a sketch of the proposed house along with separate detailed list of recommendations. On June 19, James Madison thanked Jefferson for his suggestions and passed them on to William Madison.¹¹ Unfortunately, neither sketch has survived. Among the many Jeffersonian architectural trademarks present in the house are the front pediment, semicircular windows, and an original floor plan centered around a living room with four rooms to the left and two rooms to the right.

From his younger brother Ambrose, James Madison learned in 1787 that William was interested in pursuing a public career. In a letter to Ambrose from New York on October 11, 1787, James Madison confessed that he was “at a loss what to say as to brother Wms adventuring into public life.” He recommended that William should not embark on such a career until he was absolutely certain of success, and that he would not stand in the way of doing so. “If he wishes to establish himself in the good will of the County,” James Madison advised, “the only durable as well as honorable plan will be to establish a character that merits it.”¹²

Four years later, in 1791, William Madison won election to the House of Delegates from Culpeper County, and he was reelected to a second term the following year. Thereafter he became ineligible to run again from Culpeper because Woodberry Forest had become part of Madison County. His first two years as delegate, however, enabled him to acquaint himself with the traditions and routine of the House of Delegates, a place where his elder brother James had made a reputation for himself only a few years before. During William Madison’s first two terms he sat on the Privileges and Grievances Committee, which considered requests for building turnpikes, ferries, and bridges, and for boundary changes, the creation of new counties, and the granting of divorces. On October 24, 1792, he voted with the majority to create the new county of Madison.¹³

In 1794 William Madison stood for election in Madison County and won a seat to the House of Delegates. He was appointed to the prestigious Privileges and Elections and later to the Claims and Courts of Justice committees. He was also assigned to smaller committees to prepare bills for establishing free schools, regulating escheators, and increasing militia pay.¹⁴ By the end of the 1794–95 session, William lost his interest in public service and declined to run for the next session. It would be nine years until William Madison once again served in the Virginia House.

Shortly after his entry into the political field, William Madison also began a military career in the Virginia militia. On May 7, 1794, he was appointed captain in the Madison County militia. No doubt his Revolutionary War experience helped him to obtain his rank. The next year, on October 1, 1795, he was appointed major, and on December 13, 1801, he became lieutenant colonel of the Madison militia. With this accomplishment and his previous legislative experience, William Madison again sought and won a seat to represent Madison County in the House of Delegates for the 1804–5 session.¹⁵

William and James’s father, Colonel James Madison Sr., died on February 27, 1801. The execution of the will’s provisions would prove extremely complicated and contentious to its executors, William and his older brother. Although Colonel Madison had updated his will in 1787, he later failed to make crucial changes in it to reflect the deaths of two of his sons, Ambrose in 1793, and Francis in 1800, both of whom died intestate. Executing the provisions of such a will soon proved to be beyond James Madison’s ability, as his time was fully occupied as secretary of state in the Jefferson administration.

James Madison described his father’s will to Jefferson as “ambiguous in some important points and will raise a variety of questions of legal opinion if not controversies.”¹⁶ As James Madison was unavailable to act as executor, the task was given to younger brother William to sort it all out. William Madison kept his brother informed of any progress that he was making on settling the will. The will’s execution proved to be as problematic as James Madison had predicted, and in the end the case had to be settled by the High Court of Chancery in Richmond. On March 3, 1803, the Court ruled that all the slaves and personal estate willed by James Sr. to his wife and children who survived him would be divided equally and that the property of those children’s descendants who died before the testator had to await the decision of a court-appointed commissioner.¹⁷ This would not be the last time that William Madison would be involved in his family’s legal proceedings.

With his father’s will settled, William Madison continued his career in the House of Delegates and remained a Madison County representative in the House until February 1811. Unlike James Madison, he never sought higher office on the national scene. He neither asked for, nor expected, any special favors from his brother, either as secretary of state or as president. William Madison, like James Madison, was a committed Jeffersonian Democratic-Republican, and his voting record in the House reflected its dominant political philosophy of small government and low taxes. His voting record also reflected a generally pro-business stance that maintained the status quo. He supported bills for allowing new bridges and ferries and supported those to establish banks and free schools.¹⁸ When a vacancy became available in the militia establishment, his fellow members on December 9, 1809, appointed him a brigadier general in the First Virginia Brigade. General Madison’s duties included supervising the training and equipping of his brigade’s regiments and commanding in the field if asked by the governor to do so.¹⁹

During the War of 1812, some of William Madison’s men were called to defend Norfolk in 1813, but most were marched to assist other counties along the Chesapeake Bay. In 1814, most were called out when the British threatened the Rappahannock and Potomac River counties. On July 29, 1814, Governor James Barbour directed General Madison to call out three hundred and fifty of his men to be stationed between Fredericksburg and the Potomac River.²⁰

When a British squadron entered the Potomac River in early August 1814, Barbour ordered Madison to coordinate with General William Winder, commander of the U.S. Tenth Military District, which included Madison’s brigade.²¹ Madison executed his orders so well that he drew the praise of Acting Secretary of War James Monroe, who stated that when “such a spirit shall have become general in our country we shall have nothing to fear from the foe.”²² Madison’s troops were not involved in the defense of Washington or Baltimore in August 1814, but they were directed to march to the Washington-Baltimore area in October 1814 to assist other Virginia militia already ordered there. At this time, Madison’s brigade consisted of about six hundred men under the command of Lieutenant Colonel Robert Crutchfield.²³

On October 29, 1814, Major General Winfield Scott, then commanding the Tenth District, ordered Madison and most of his men to march from Camp Snowden, located between Baltimore and Washington, to Herring Bay in Ann Arundel County, Maryland, to engage some British marines who were raiding along the West River area. Madison’s brigade with Colonel Crutchfield’s regiment arrived on October 31, 1814, along with some local militia and Captain John A. Bird’s U.S. Light Dragoons. Madison’s plan was to surprise the enemy, but Captain Bird’s dragoons charged ahead and immediately engaged the enemy so that the element of surprise was lost. Some of Bird’s horses became stuck in the heavy mud near the river and were captured by the enemy. The enemy saw that the rest of Madison’s men were coming up and withdrew from the field under protection of their ship’s guns.²⁴

Madison realized that a great opportunity had been lost by the impetuousness of Captain Bird’s charge. “It was a gallant and spirited charge,” Madison reported to Scott, “yet unauthorized, and no doubt was the effect of too much zeal and impudent council, emanating from not one attached to the Brigade.” Madison boasted that “every officer and soldier within my observation marched to the field of expected battle with the greatest cheerfulness and alacrity.”²⁵ The British navy commanding officer at the West River, Commander Thomas Alexander, reported the skirmish to his superior, Captain Robert Barrie. Barrie’s report to Admiral Sir George Cockburn contained the name of General William Madison, but Barrie never suspected that his forces had crossed swords in battle with the resident’s brother.²⁶

William Madison’s troops remained in Maryland until the end of November, when they were marched back home. He remained commanding general of the First Brigade until the General Assembly elected him as major general of the Second Division of the Virginia militia on December 17, 1822.²⁷ He then became one of only four generals to hold the highest rank in the Virginia militia. William Madison probably returned home sometime in December 1814, a widower, to await the end of the war, which officially came on February 17, 1815. From that time to his death in 1843, he seems to have been a gentleman farmer content to live at Woodberry Forest.

When James Madison returned to Montpelier after his presidency, the brothers no doubt visited one another from time to time, but from all accounts they were not particularly close in their retirement years. William Madison did provide beef to Montpelier and perhaps there were reciprocal agreements, but very little is known about the relationship that the two men had during the last years of their lives.²⁸ In a letter to William Madison in 1829, James Madison complained that an illness prevented him from settling disputes with William and regretted that the “distressing State of things shd continue.”²⁹ William Madison visited Montpelier on special family occasions, but the relationship appeared to have been strained. James and Dolley Madison probably did not approve of William’s living out of wedlock with Nancy Jerrell. Lingering resentments over William’s handling of his and James’ father’s will continued to poison the relationship between the two brothers and their families. In addition, William appeared to be constantly in financial straits and seeking any opportunity to obtain money from his only slightly better-off elder brother.³⁰

William Madison’s unpleasant relationship with Dolley Madison, particularly after James Madison’s death in 1836, was well known at the time. William Madison, among others, contested his brother’s will. James Madison’s will clearly stated his wish and his wife’s wish to distribute money and goods among current living nieces and nephews, or to the surviving children of any deceased niece or nephew. William’s argument was that any money or property should be given only to nieces and nephews rather than their offspring, which would reduce the number of inheritors from thirty-one to twenty-six. The Chancery Court held otherwise and supported Dolley Madison’s interpretation. Following his brother’s death, William Madison also pursued a claim against his brother’s estate alleging that James Madison owed him $2,000 in expenses William had accrued in settling their father’s will. William Madison pressed Dolley Madison for the money and was overheard saying that he would sue his brother’s family for every bit of money due to him. Dolley Madison stated in court affidavits that James Madison always denied that he owed money to his brother William for this or any other reason. The dispute continued in court long after William Madison’s death.³¹

William Madison died on July 19, 1843, at Woodberry Forest. He left 1,270 acres and thirty slaves to be divided up between his wife, Nancy, and their son Jack; his and his first wife’s surviving son and daughter, Ambrose and Rebecca; and his grandchildren. Like his brother James, William Madison allowed his slaves to choose with whom they wished to reside.³² In 1870 John S. Walker purchased much of the land from Ambrose Madison for his son, Robert S. Walker. The Walkers educated their children there and started a school on the property. Today William Madison’s home, named The Residence, is the headmaster’s house and is part of the well known Woodberry Forest School.³³

It is difficult to assess the character of William Madison because there is so little private correspondence available by which to weigh him. For instance, little is known about his views on slavery or religion, or whether they were similar to James Madison’s. His relationship with his brother James appeared to have been more cordial before the two men retired from public service than after. Many Virginia farms suffered greatly in the years following the War of 1812, and William Madison’s estate may have failed to earn good profits—hence his attempt to press his brother’s estate and Dolley Madison for money. Or perhaps he brooded for years over his relative obscurity and lack of success while watching James Madison become a very famous and important man. Whatever his reasons, William Madison alienated many of his brother’s family in his final years.³⁴