He was the son of George Whitmore (1739–1794) and Mary Walls (1744 – 11 March 1808). He entered Woolwich Academy at the age of 14, and had an army commission at age 18.[1]
Whitmore married Cordelia Ainslie (1780 – 19 December 1857) on 16 January 1798. Their second daughter Cordelia Winifreda married Captain Montagu Stopford, RN, on 25 August 1827.[3] Their grandson, Sir George Stoddart Whitmore (1829–1903), was born on Malta to Lieutenant (later Major) George St Vincent Whitmore RE and to the chief justice of Malta, Sir John Stoddart's, daughter. He later became an army officer and police officer in New Zealand.