Shvernik was born in St. Petersburg and joined the Bolsheviks in 1905. In 1924 he became a People's Commissar in the Russian Soviet Federative Socialist Republic and became a full member of the Central Committee of the party in 1925. In 1927 he was demoted and sent to the Urals to head the local party organization. Stalin found him a loyal supporter of his policy of rapid industrialisation and moved him back to Moscow in 1929 making him chairman of the Metallurgist Trade Union. He resumed his rise in the party becoming a member of the Orgburo and the party Secretariat. He also served as first secretary of the All-Union Central Council of Trade Unions from July 1930 to March 1944. As such, Shvernik presided over the 1931 Menshevik Trial, in which fourteen Russian economists came up for trial on charges of treason.
During the Second World War Shvernik was responsible for evacuating Soviet industry away from the advancing Wehrmacht. He was Chairman of the Presidium of the Supreme Soviet of the Russian SFSR from 1943 to 1946. In 1946 he became Chairman of the Presidium of the Supreme Soviet of the USSR, succeeding Mikhail Kalinin. He only became a member of the Politburo of the CPSU Central Committee (then named the Presidium of the Party's Central Committee) in 1952 but was demoted in 1953 when the body was reduced in size.
Following the death of Stalin, Shvernik was removed as titular president of the USSR and replaced by Kliment Voroshilov on 15 March 1953. Shvernik returned to his work as the chairman of the All-Union Central Council of Trade Unions. In 1956, after his work in the Pospelov Commission, which was the basis of Khrushchev's "Secret Speech" denouncing Stalinism, Khrushchev recommended Shvernik for the post of chairman of the Party Control Committee and later put him in charge of rehabilitating the victims of Stalin's purges (Shvernik Commission). In 1957, Shvernik again became a full member of the Presidium and remained on the body until he retired in 1966.