McCaffrey was born in Tucson, Arizona and played collegiately at Saint Louis.[2][3] Later, in 1994, he was inducted into the school's hall of fame.[4]
Aside from playing for the national team, McCaffrey played in the Amateur Athletic Union, first for the Buchan Bakers and then for the Akron Goodyear Wingfoots.[5] He was named an AAU All-American three times (1962–1964) while playing for the Wingfoots.[6] He also played 16 times for the United States.