On 19 May 1840 he married Fanny Lowther (1818–1890) at Saint Martin in the Fields, a few hundred yards from the homes of both. She was the "natural" (illegitimate) daughter of the unmarried Viscount Lowther, the future William Lowther, 2nd Earl of Lonsdale, also a Tory MP, who had been a friend of George IV. By a later liaison of her mother with Dr Charles Lewis Meryon, Fanny Lowther was the half-sister of the French artist Charles Meryon who she helped support to the end of his life. She was living with her father at 15 Carlton House Terrace at her marriage, and received a dowry of £10,000.[4] After Lowther's death in 1872 Fanny was left £125,000.[6]
They had two sons and a daughter, Mary, who died young (1851–66). The elder son was Arthur Broadwood (1849–1927), who retired from the army as a colonel in 1906 and who had five children. His brother was Alfred Broadwood (1856–1909).[7]
Broadwood had been left £20,000 and a country house in Essex at his father's death in 1812. Most of this was invested by his trustees in a partnership in a Lion Brewery in London (apparently not the Lion Brewery Co in Lambeth). But the business, perhaps neglected by Broadwood, was not a success,[4] and in 1848 his "fortune collapsed", and for the rest of his life he lived in Tunbridge Wells, though remaining an MP until 1852.[8]