1759年、キャサリン・エリザベス・ハンソン(Catherine Elizabeth Hanson、1733年ごろ – 1817年3月15日没、オズワルド・ハンソンの娘)と結婚[1]、2男4女をもうけた[2]。アバディーン伯爵がキャサリン・エリザベスを誘惑した後、彼女にピストルで結婚を強要されたという逸話がある[3]。
^"Mrs Forest, the former housekeeper of his London home, was brought to the newly acquired Cairnbulg Castle, near Fraserburgh, to deliver him a son called John; an illegitimate daughter was accommodated in the London house; yet another, unknown woman was housed at Wiscombe Park, in Devon, where she gave the Earl a son, Charles; his castle at Ellon housed ‘a lady of charm called Penelope Dering’, a friend of one of his legitimate daughters, who gave him a daughter and a son." 出典: John Doran "Haddo: The House with Outstretched ArmsArchived 15 February 2010 at the Wayback Machine., Leopard magazine, July 2002