Born in Copenhagen, Brylle started playing football with local club Frederiksberg Boldklub (FB). He played for Hvidovre IF in the secondary Danish 2nd Division, and helped the club win promotion for the top-flight Danish 1st Division in the 1978 season. Before the start of the 1979 season, he moved to 1st Division rivals Vejle Boldklub, where he got his national breakthrough. He scored 18 goals in 26 league games, before moving abroad to play professionally for Belgian club RSC Anderlecht in 1979. He stayed five years at Anderlecht, with whom he won the 1983 UEFA Cup, scoring the only goal in the final.
He moved to Dutch team PSV Eindhoven in 1984 and on to Olympique Marseille in France in 1985. In 1986, he moved back to Belgium to play for Club Brugge. He played a total of 101 matches and scored 49 goals for Club Brugge, with whom he won the 1988 Jupiler League championship. He went on to play for Belgian clubs Beerschot, Lierse S.K. and FC Knokke.