Joseph Manigault was the son of Peter and Elizabeth Wragg Manigault. Altogether Peter and Elizabeth had four children: Gabriel, Ann, Joseph and Harriet. Peter and Elizabeth’s house was located on the northwest corner of Amen and East Bay Streets. Due to Elizabeth’s ill health, the children were raised by Peter’s parents, Gabriel and Ann Ashby Manigault, who lived a few blocks away between Chalmers and Queen on Church Street. Both of Joseph’s parents had died by the time Joseph was ten. Below is one of the only anecdotes Joseph tells his own children about his childhood with his parens.