*Club domestic league appearances and goals, correct as of 17:59, 4 March 2024 (UTC)
^ All Dover Athletic players' statistics for the 2020–21 season were expunged due to staff being furloughed because of the costs associated with the COVID-19 pandemic, Passley made 7 league appearances during this season.
Passley started his football career at Fulham and joined them at age eleven.[3] He was initially a winger but switched to right back at sixteen, as well as playing at centre-back.[3] Passley signed his first professional contract in 2013.[3][4] On 26 January 2014 he made his debut for Fulham in a 1–1 draw against Sheffield United in the FA Cup fourth round, playing the full ninety minutes.[5][6]
Passley joined Shrewsbury Town on a youth loan on 27 November 2014.[7] He made his Football League debut two days later in a 1–0 win over Burton Albion at New Meadow.[8] Despite being ever-present for the duration of his loan period, he returned to his parent club when his loan deal ended on 5 January 2015.[9]
On 29 January 2015, Passley joined Portsmouth on an initial one-month youth loan.[10] He made his Portsmouth debut two days later, playing 90 minutes in a 0–0 draw at Wycombe Wanderers.[11] During his run of first team games, he helped the club achieve three clean sheets in three games, playing in a wing-back role. This was the same position he played while at Shrewsbury Town.[12] Having made six appearances, Passley's loan spell with Portsmouth was extended for another month and again until the end of the season.[13][14] His loan spell with Pompey came to an end on 12 April 2015 when he was taken off injured in the 83rd minute after losing consciousness in a defeat at Morecambe.[15]
At the end of the 2014–15 season, Passley was among nine players to leave Fulham.[16]
Dagenham & Redbridge
Passley joined League TwoDagenham & Redbridge on a one-year deal in July 2015.[17] Despite being a virtual ever-present for The Daggers in his first season, Passley was among eleven players to leave following relegation to the National League.[18]
Whitehawk
Passley joined National League South club Whitehawk in February 2017, the team he scored against in an FA Cup second round replay on 16 December 2015 in a match televised live on BT Sport. After helping Whitehawk secure safety in the National League South Passley left the club at the end of the season.[19]
Dover Athletic
On 19 July 2017, Passley signed for Dover Athletic on a one-year deal.[20] Passley started on the opening day of the season in a 1–0 away victory to Hartlepool United.[21]
Passley signed a one-year contract extension on 21 April 2018 after making 33 appearances in his debut season for the club.[22] Passley scored his first goal for the club in the last match of the 2018–19 season, winning and scoring a last minute penalty in a 3–0 victory over Sutton United.[23] Following's Dover's decision to not play any more matches in the 2020–21 season, made in late January, and subsequent null and voiding of all results, on 5 May 2021 it was announced that Passley was out of contract and had left the club.[24]
Havant & Waterlooville
On 21 May 2021, Passley dropped down a division to join National League South side Havant & Waterlooville on a free transfer, after reportedly turning down offers to remain in the National League.[25][26]