In 1854, he was elected on the American Party ticket to the Thirty-fourth Congress as part of the Know Nothing party sweep of the Massachusetts congressional delegation that year. An abolitionist who received an honorary degree from Amherst in the same ceremony as Charles Sumner, he became a Republican and was reelected to Congress as such in 1856.
He was married to Clara Nourse (1813 – 1848) until her death in 1848.[1] They had two children: a daughter, Emma Lovetta Wilder (Chaffee) (1838 – 1910), and a son, Clemens Clifford Chaffee (1841 – 1867).
In 1850, Chaffee married Eliza Irene Emerson (née Sanford) (1815 – 1903). Irene Emerson was the widow of Dr. John Emerson, the owner of the slave Dred Scott. She had a daughter, Henrietta Sanford King (Emerson) (1843 – 1919), from her first marriage. There is speculation [2] that Chaffee advanced the Dred Scott case as a test for slavery. However, contemporary reports have him discover from the Springfield Argus that his new wife owned the most famous slave in the world in February 1857, only a month before the Supreme Court handed down the infamous Dred Scott decision. Criticized nationwide for apparent hypocrisy, Chaffee immediately arranged for the return of Scott to his original owners, the Blow family, for emancipation.
Because of negative publicity from the Scott case, Chaffee did not seek reelection in 1858 and became Librarian of the House of Representatives from 1860-1862. He then practiced medicine in Washington, D.C., until 1876, when he returned to Springfield. He died there in 1896 at age 84.