He was named the lieutenant governor for Prince Edward Island in 1812, replacing Joseph Frederick Wallet DesBarres. He attempted to reorganize the island's militia to prepare for a possible attack from the United States but encountered resistance from the legislative assembly. Smith also dismissed a number of appointed officials who he believed were conspiring against him including James Bardin Palmer, William Johnston and John Frederick Holland and attempted to govern without convening the island's assembly. He also appointed his relatives to several public offices. Smith attempted to deal with the ongoing issue of land ownership on the island by initiating escheat proceedings against several property owners which remained largely vacant and also by attempting to collect quitrents which he believed would encourage land owners to make more productive use of their properties. His son-in-law John Edward Carmichael, by attempting to collect unpaid quitrents, triggered a series of meetings calling for Smith's recall as governor.
In 1824, a review of Smith's activities by the Colonial Office found evidence of misconduct. Lord Bathurst asked Smith to resign and he was replaced by John Ready. With the aid of his brother, Sir William Sidney Smith, he was able to secure a pension in 1829, which he continued to collect until his death at Dawlish in England in 1855.
