Shirley married Eliza, daughter of Arthur Stanhope, in 1810. They had several children, including Evelyn Shirley, also a politician, and Arthur Shirley (1813–1877), a Major-General in the Army. Shirley died in December 1856, aged 68. His wife survived him by three years and died in April 1859.