He was the son of Adriel Ely and Sarah (Stowe) Ely. He graduated from Yale College in 1804. Then he studied medicine with Dr. Thomas Broadhead at Clermont, New York, and was licensed to practice in 1809. In 1810, he removed to Middlefield, New York, and practiced medicine there. On June 11, 1816, he married Hannah Knapp Gilbert (1791–1868), and they had five sons, among them Assemblyman William H. Ely (b. 1829).
Sumner Ely was Postmaster of Middlefield and a brigadier general of the New York State Militia.