In 1967, Howson was caught up in the VIP affair, which saw allegations that the government had misused the VIP aircraft fleet for ministers' private purposes. When asked to table records on the fleet's movements, Holt and Howson refused and implied that they did not exist, but Senator John Gorton later found that the records did exist and tabled them in the Senate.[5] When Gorton became Prime Minister on 10 January 1968, he retained all of the previous ministers in his ministry, but after he won a seat in the House of Representatives he carried out a Cabinet reshuffle on 28 February 1968 and dropped Howson from the ministry.[6]
Expecting to be rewarded for his support of McMahon during Gorton's ministry, Howson was disappointed when he was appointed as Australia's first Minister for the Environment, Aborigines and the Arts. He was reported as commenting: "The little bastard gave me trees, boongs and pooftas".[7] However, according to Rob Chalmers, he subsequently "showed great energy and concern to improve the lot of Aborigines".[8]
In 1973, Howson founded the Deafness Foundation Victoria.[9]
In 1984, Howson published a diary (edited by Don Aitkin) recording the events during his period as a parliamentarian and as a minister.[10] According to Rob Chalmers, it was "one of the most informative and interesting books on Australian postwar politics ever published".[3]