Withee was born on October 26, 1834, in Somerset County, Maine.[1] He was raised on his father's farm and received a common school education.[2] Upon reaching adulthood, he was tempted by the opportunities in the new western states and moved to Wisconsin. He arrived in La Crosse, Wisconsin, in 1853 and went to work in the lumber industry.[3] By 1859, he had enough earnings to begin his own business further north, in Clark County, Wisconsin.[3]
He was an owner or shareholder in many lumber companies in western Wisconsin, but was principally involved in the company Bright & Withee. He was also invested in the Electric Light Plant, the Gas Light Company, and the La Crosse Farming Company.[3]
Levi Withee was a son of Zachariah Withee and his wife Polly (née Longley). Zachariah Withee was a veteran of the War of 1812 and received land as a reward for his service. Levi Withee was the fifth of seven children.[3] His older brother Niran Withee also worked as a lumberman in the La Crosse area and was a member of the Wisconsin Legislature.
On June 3, 1868, he married Louisa H. Smith.[1] They had one son, Abner, who was also active in the lumber business in Wisconsin and Florida, and was appointed to the Bureau of Internal Revenue in Florida.[6]
