He succeeded his brother George de Grey, 3rd Baron Walsingham, in 1831 when the latter was killed with his wife as the result of a house fire at his London Home, inheriting the barony and the family seat of Merton Hall, Norfolk.
He died in 1839 and was buried at Merton, Norfolk. He had married in 1802 Elizabeth North, the daughter of Rt Rev Hon Brownlow North DD, Bishop of Winchester. They had six sons and three daughters. He was succeeded by his eldest son Thomas de Grey, 5th Baron Walsingham.