His entire professional player career was with the Grasshoppers Zürich in the Swiss league. In 24 years he won seven Swiss championships and five Swiss cups. He appeared 56 times for the Switzerland national football team. He made 3 goals and was in the team at the 1996 UEFA European Championship.
Coach
Koller's career as coach began in the Swiss club FC Wil in the season 1997/98. In January 1999 he went to FC St. Gallen in the Swiss Super League. The next year the club consecutively won the championship for almost 100 years. In the UEFA Cup the Swiss team would eliminate English top club FC Chelsea. Koller was voted Swiss manager of the year in 2000. From 2002 till 2003 he managed his former club Grasshoppers Zürich. His first club abroad was in the German Bundesliga where he managed 1. FC Köln. Being unsuccessful he resigned after seven months. At this time Lukas Podolski made his first match in the Bundesliga.
From 2005 on he was coach of the VfL Bochum in the 2.Bundesliga. They promoted to the Bundesliga. The next year he planned to stay in the league. The club finished that year in 8th place, the third-best season ever for Bochum. After poor results at the start of the 2009/10 season, Koller was dismissed.
On 4 October 2011, Koller was announced as under contract as the new coach of the Austria national team (ÖFB) until December 2013. After his nomination, there was heavy criticism, he being from abroad. During the long history of the team, he is the third coach who was not from Austria. He took over on 1 November. His first match as new coach was an unsuccessful friendly match versus Ukraine.