Cadet was born in Haiti to a wealthy white father and an impoverished black mother. After the death of his mother, Cadet was "given" to another Haitian family as a child servant, or restavec. Emotionally, verbally, physically, and sexually abused by his masters, he was forced to work long hours and was often loaned to neighbors and friends to work for them. Excluded from all family, cultural, civic, and religious activities, Cadet became an observer, rather than a participant, in Haitian culture and society.

At the age of 15, Cadet immigrated to the United States with his owners. When they realized that Cadet would be required to attend school alongside their own children, they expelled him from the house. Despite these obstacles, Cadet finished high school, joined the United States Army, started a family, and earned a master’s degree in French literature. In 1998, Cadet published his memoir, Restavec: From Haitian Slave Child to Middle Class American, one of the few books written by survivors of contemporary slavery. He is the founder of the Jean R. Cadet Restavek Organization, a nonprofit, nonpartisan organization dedicated to ending child slavery in Haiti.