In 1809 in Madras (now Chennai), Erskine married Maitland Mackintosh (1792-1861), daughter of Mackintosh and his first wife Katherine Stuart. They had 14 children, one of whom, Frances, married the statistician and civil servant Thomas Farrer, 1st Baron Farrer. Another daughter, Mary, was head nurse in the Naval Hospital at Therapia during the Crimean War, and looked after Florence Nightingale while she recovered from illness.
He was removed from office in 1823 by Sir Edward West after being accused of defalcation (misappropriation of funds) and for many of his later years resided in Edinburgh, as well as Pau in South West France. He was Provost of St Andrews in 1836.[2]