The house was constructed in 1853 for William Cozzens, a horticulturalist from New England, who made his way to California via Panama. Mr. Cozzens couldn't pay the builder and lost the house in court that same year. J. Neely Johnson, fourth Governor of California who served from 1856 until 1858, lived in the home prior to his term as governor. A press report in 1856 noted that Johnson gave a brief speech from the balcony prior to being escorted to the State Capitol for his inauguration. Johnson was a lawyer by trade who moved to California during the 1849 California Gold Rush. Selden A. McMeans, State Tresurer of California elected in 1853, purchased the home in 1854 and later sold it in January 1856 to David S. Terry, California Supreme Court Justice. Terry resided there during the time of his infamous duel with Senator David Broderick who was shot by Terry and died three days later. Johnson, McMeans and Terry were prominent members of the American Party, also called the "Know Nothings" because of the secretive nature of their dealings. It is speculated that the Party conducted some of their private meetings at this home.