Withers was born in Bunbury, Western Australia. Withers was the son of Frederick Withers, a former Labor member of the Western Australian Legislative Assembly. Withers was educated at Perth Technical College. Withers served in the Royal Australian Navy from 1942 until 1946 before returning to Australia to study law at the University of Western Australia under the ex-servicemen's scheme. While at university, Withers opposed what he saw as the authoritarian stance of the Chifley Labor government and joined the Liberal Party of Australia.
Returning to Bunbury to practise law, first as a solicitor and, from 1953, a barrister, Withers was elected to Bunbury Municipal Council and began to involve himself in Liberal Party affairs, serving at various times as Liberal Party State President and Vice-President and Federal Vice-President.
Withers entered the Senate on 17 February 1966 to fill the vacancy caused by the death of Senator Sir Shane Paltridge, but lost his seat later that year, before returning to the Senate in 1968.
Described as having a "jovial manner and perpetual grin", Withers quickly gained a reputation as the Liberal numbers man and served as Senate Government Whip from 1969–71. After the defeat of the McMahon government in 1972, Withers became Opposition Leader in the Senate, where he retained a thin majority and acted to block much of the Whitlam Government's legislation. Withers was widely known as "The Toecutter" for his alleged approach to enforcing party loyalty and his role in the 1975 Australian constitutional crisis.
After the dismissal of the Whitlam government on 11 November 1975, Withers was appointed to Malcolm Fraser's first (caretaker) ministry, becoming Vice-President of the Executive Council as well as briefly holding the portfolios of Special Minister of State, Capital Territory, Media, and Tourism and Recreation during the period leading up to the December election. After the election, Withers became Minister for Administrative Services, and continued as Vice-President of the Executive Council until 7 August 1978. He was dismissed by Fraser in the wake of the findings of a Royal Commission into aspects of a redistribution of certain federal electorates in Queensland. The royal commission found that Withers had exercised his ministerial influence in an inappropriate way. At the time he commented about Fraser that "When the man who's carried the biggest knife in this country for the last ten years starts giving you a lecture about propriety, integrity and the need to resign, then he's either making a sick joke or playing you for a mug".