He married twice; firstly Hannah, the daughter of John Cookson of Whitehill, co. Durham and secondly Jane Maxwell, with whom he had 4 sons. He also had an illegitimate daughter.
Richard Ellison MP is the great-great-great grandfather of Richard Ellison (cricketer).[citation needed]