On 16 March 1846, Hay was married to James Duff, son of General Hon. Sir Alexander Duff and Anne Stein, at the British Embassy in Paris, France.[2] Duff later inherited the earldom of Fife upon the death of his uncle in 1857. The marriage had issue:
Lady Anne Elizabeth Clementina Duff (16 August 1847 – 31 December 1925) married John Townshend, 5th Marquess Townshend (1831–1899) on 17 October 1865. They had two children.
Lady Ida Louisa Alice Duff (11 December 1848 – 29 May 1918) married Adrian Elias Hope (1845–1919) on 3 June 1867. They had one daughter and later divorced. She married, secondly, William Wilson (1839–1905), a London stockbroker, on 20 September 1880.
Lady Alexina Duff (20 Mar 1851 – 30 Apr 1882) married Henry Aubrey Coventry (1845–1909), grandson of George Coventry, 8th Earl of Coventry, on 2 July 1870.
Lady Agnes Cecil Emmeline Duff (1852–1925) married, firstly, George Robert Hay-Drummond (1849–1886), son of George Hay-Drummond, 12th Earl of Kinnoull, on 4 Oct 1871. They had issue, but later divorced. She married, secondly, Herbert Flower (1853–1881), brother of Cyril Flower, 1st Baron Battersea, on 5 August 1876. She married thirdly, Alfred Cooper (1838–1908) on 4 July 1882, whose descendants include former British prime minister David Cameron.
Lady Mary Hamilton Duff (20 February 1854 – 20 March 1854)
^Marlene A. Eilers, Queen Victoria's Descendants, page 179.
^G.E. Cokayne; with Vicary Gibbs, H.A. Doubleday, Geoffrey H. White, Duncan Warrand and Lord Howard de Walden, editors, The Complete Peerage of England, Scotland, Ireland, Great Britain and the United Kingdom, Extant, Extinct or Dormant, new ed., 13 volumes in 14 (1910–1959; reprint in 6 volumes, Gloucester, U.K.: Alan Sutton Publishing, 2000), volume V, page 379. Hereinafter cited as The Complete Peerage.