O'Connell was born in Albany, New York and dropped out of school in the fifth grade, finding work as a bricklayer and as a bartender.[1] In 1919, he was elected County Assessor. A few years later left that position to become the chairman of the County Democratic Committee. In 1921, O'Connell was instrumental in the mayoral election victory of William Stormont Hackett; Hackett's victory was a Democratic Party victory over a Republican political machine—led by William Barnes Jr.—that had been in power in Albany for over 20 years.
During the 56 years that followed, O'Connell had undeniable power in the political life of Albany, determining patronage and essentially ruling the city during the administrations of mayors William Hackett, John Boyd Thacher II and most notably Erastus Corning 2nd.
According to author Paul Grondahl, O'Connell is suspected of having ordered the murder of gangster Legs Diamond in 1931.
William Kennedy's 1975 novel Billy Phelan's Greatest Game is based on the attempted 1933 kidnapping of O'Connell's nephew, John O'Connell Jr. Kennedy's 2002 novel Roscoe is a fictionalized account of O'Connell's life.