Whitbread married Lady Isabella Charlotte Pelham, youngest daughter of Henry Pelham, 3rd Earl of Chichester on 9 July 1855. They had four children together, Samuel, Maude, Henry, and Francis.
His eldest son, Samuel Howard, followed his father into politics.
Maud married her cousin Charles, son of Samuel's younger brother William.
Henry married Mary Raymond and lived at Norton Bavant, Warminster.
Francis married Ida, daughter of Charles Hanbury-Tracy, 4th Baron Sudeley, and lived at Burford House, Tenbury Wells.
^Ritchie, Berry (1992). An Uncommon Brewer, the Story of Whitbread. London: James & James. p. 63. ISBN978-0907383369.
^Craig, F. W. S. (1989) [1977]. British parliamentary election results 1832–1885 (2nd ed.). Chichester: Parliamentary Research Services. p. 39. ISBN0-900178-26-4.