Wharton died at the age of about 34. He had married Lady Philadelphia Carey, daughter of Robert Carey, 1st Earl of Monmouth on 11 April 1611. His elder brother George had been killed in a duel in 1609, and thus his eldest son young Philip inherited the barony when the 3rd Baron died in 1625. Wharton's second son Thomas was also MP for Westmorland.[5]