Born in 1921, Milne joined the Communist Party in 1939. He was a patternmaker by trade and first worked at the Hall Russell shipyard. The secretary of Aberdeen Trades Council from 1948 to 1969, he made his mark by working for safer working conditions for fishing trawler crew. But his interests and activities were always very wide; he was a member of the Regional Hospital Board, where he kept up a constant pressure for reforms in the interests of patients.
He became a member of the General Council of the STUC in 1954, the youngest person elected to that time. Deputy General Secretary in 1969 and the General Secretary in 1975, Milne was also a member of the Communist Party Executive Committee for a period and the Scottish Committee until his death.
During his period of office, the STUC was heavily involved in a range of activities of an educational and cultural nature. He spent six years as Chair of SCOTBEC and, with the Glasgow Trades Council, a residential college was established at Treesbank. Jimmy Milne spent 12 years on the board of Govan Shipbuilders and was the longest serving member of the Parole Board when he stepped down, after 15 years membership.
Married to Alice, Jimmy Milne was also a great music lover, who consciously spread the STUC's influence widely in education and the arts and received an honorary doctorate from Heriot Watt University for such work. The Scottish National Orchestra performed a specially commissioned overture, `Sunset Song', by William Sweeney on his retirement in a similar gesture. He died in 1986, only ten days before his official retirement date. Over a thousand people attended his funeral, including the then Shadow Scottish Secretary, Donald Dewar, and many other prominent individuals from all walks of life.