Townshend married in 1787 Georgiana Anne Fawkener, daughter of William Poyntz, and the divorced wife of William Fawkener. When his grandmother Etheldreda Townshend died in 1788, she left him much of her fortune, including Balls Park.[2] Their elder son Charles Fox Townshend was the founder of the Eton Society but died young. Their younger son John became an admiral in the Royal Navy and succeeded his first cousin as 4th Marquess Townshend in 1855. Their daughter Elizabeth Frances Townshend (2 August 1789 –10 April 1862 Nice) married 20 October 1813 Captain Augustus Clifford R.N., later Sir Augustus Clifford, 1st Bt (1788-1877), illegitimate son of the 5th Duke of Devonshire and his mistress, Lady Elizabeth Foster, née Hervey, later the Duke's second wife (1811).[3] Mrs Clifford became Lady Clifford in 1838 when her husband was raised to a baronetcy. Townshend died in February 1833, aged 76. Lady Georgiana Anne died in 1851.