Halldór took over as prime minister on 15 September 2004, succeeding Independence Party leader Davíð Oddsson, while Davíð replaced Halldór as Foreign Minister.
On 5 June 2006, following poor results for his party in the 2006 Icelandic municipalities elections, Halldór announced his resignation as prime minister and stated that he intended to step down as leader of the Progressive Party in August 2006, and leave politics all together. Geir H. Haarde, the Foreign Minister of Iceland, succeeded him on 15 June 2006.
Halldór Ásgrímsson's successor as Progressive Party leader was Jón Sigurðsson, Ministry of Industry, Energy and Tourism, who was elected at the party's convention in August 2006. At the convention Halldór ended his political career with an emotional and dynamic farewell speech to the party. Halldór resigned as MP after the convention; he was the longest serving MP at the time.