Dufferin, along with the barony of Kinelarty and part of Castlereagh was at one time part of the territory of the Cenél Foghartaigh (Kinelarty), ruled by the Mac Artáin (MacCartan) sept.[4][5] The Ó Labhradha (Lowry, Lavery) sept are also noted in Dufferin.[5][6]
In the reign of Henry VIII the White family, who were originally from Flemington in County Meath, became Lords of the Manor of Dufferin. Patrick White (died 1561), the first of the White family to own Dufferin, was second Baron of the Court of Exchequer (Ireland).[7] The Whites, although they were driven away for a time due to local disturbances, owned Dufferin until 1610, when they sold the barony to James Hamilton, 1st Viscount Claneboye.[8]