Empey was born in Dublin, the son of the Reverend Francis Fullerton Empey and Mildred May "Mimi" Empey (née Cox). His father was also a cleric, serving in Donoughmore and Donard, Fenagh and then in Enniscorthy. Empey was educated in local national schools in his father's parishes and then at Portora Royal School in Enniskillen. He then took a BA at Trinity College, Dublin in 1957.
Archbishop Empey married Louisa "Louie" Hall in July 1960 and they have four children; Patrick (born 1961), Karl (born 1963), Kevin (born 1967) and Sheila (born 1971). He also has ten grandchildren, the youngest born on 20 April 2013.