Drury was considered the favourite candidate to succeed George Butler as headmaster of Harrow in 1829; Charles Longley, an Oxford academic and a last-minute external candidate, was instead appointed.[4] Thereafter Drury's effectiveness as a schoolmaster declined: he became "increasingly eccentric, bad-tempered, and indolent", losing classroom control, missing morning lessons, wearing flowery dressing-gowns, eating fruit during lessons, and borrowing cigars from schoolboys. Nonetheless, he remained well-liked by the boys.[5]
In 1808 he married Caroline Tayler, daughter of Archdale Wilson Tayler, and sister of the artist John Frederick Tayler. Just two years after the marriage, Byron wrote to Francis Hodgson "Talking of marriage puts me in mind of Drury (who, I suppose, has a dozen children by this time, all fine, fretful brats); I will never forgive matrimony for having spoiled such an excellent bachelor".[6] Caroline's sister Susannah Tayler later married Hodgson.