On 4 October 1779, he married Elizabeth Drummond (1758–1818) at Quebec, Canada. She was the eldest daughter of the former Katherine Oliphant and Colin Drummond, of Megginch Castle, Perthshire, who was deputy Commissary-General and deputy Paymaster to the Forces in Canada (and son of John Drummond, 10th of Lennoch). Before his death in 1796, they were the parents of a daughter:[6]
Hon. Elizabeth Catherine Caroline Hervey (1780–1803), who married Charles Ellis, 1st Baron Seaford, the second son of John Ellis of Jamaica, a wealthy sugar farmer and slave trader.[2]
Lord Hervey died on 10 January 1796. On his death in 1796, his courtesy title was assumed by his brother, Frederick (later created Marquess of Bristol). His widow died on 4 September 1818.[2]
^G.E. Cokayne; with Vicary Gibbs, H.A. Doubleday, Geoffrey H. White, Duncan Warrand and Lord Howard de Walden, editors, The Complete Peerage of England, Scotland, Ireland, Great Britain and the United Kingdom, Extant, Extinct or Dormant, new ed., 13 volumes in 14 (1910-1959; reprint in 6 volumes, Gloucester: Alan Sutton Publishing, 2000), volume II, page 327.