Like many Soviet NKVD officers who conducted political repression, Yagoda himself became ultimately a victim of the Purge. He was demoted from the directorship of the NKVD in favor of Nikolai Yezhov in 1936 and arrested in 1937. Charged with the crimes of wrecking, espionage, Trotskyism and conspiracy, Yagoda was a defendant at the Trial of the Twenty-One, the last of the major Soviet show trials of the 1930s. Following his confession at the trial, Yagoda was found guilty and shot.
Yagoda was born in Rybinsk into a Jewish family. The son of a jeweller, trained as a statistician, who worked as a chemist's assistant, he claimed that he was an active revolutionary from the age of 14, when he worked as a compositor on an underground printing press in Nizhni-Novgorod, and that at the age of 15 he was a member of a fighting squad in the Sormovo district of Nizhni-Novgorod, during the violent suppression of the 1905 revolution. One of his brothers was killed during the fighting in Sormovo; the other was shot for taking part in a mutiny in a regiment during the war with Germany. He said he joined the Bolsheviks in Nizhni-Novgorod at the age of 16 or 17, and was arrested and sent into exile in 1911. Before his arrest, he married Ida Averbakh, one of whose uncles, Yakov Sverdlov, was a prominent Bolshevik, and another, Zinovy Peshkov, was the adopted son of the writer Maxim Gorky. In 1913, he moved to St Petersburg to work at the Putilov steel works. After the outbreak of war, he joined the army, and was wounded in action.
There is another version of his early career, told in the memoirs of the former NKVD officer Aleksandr M. Orlov, who alleged that Yagoda invented his early revolutionary career and did not join the Bolsheviks until 1917, and that his deputy Mikhail Trilisser was dismissed from the service for trying to expose the lie. It can be assumed that this was gossip put around by Yagoda's successors after his fall from power.
After the October Revolution of 1917, Yagoda rose rapidly through the ranks of the Cheka (the predecessor of the OGPU and NKVD) to become the second deputy of Felix Dzerzhinsky, the head of the Cheka, in September 1923. After Dzerzhinsky's appointment as chairman of the Supreme Council of National Economy in January 1924, Yagoda became the real manager of the State Political Directorate (OGPU), as the deputy chairman Vyacheslav Menzhinsky had little authority because of his serious illness. In 1924, he joined the USSR's head of government, Alexei Rykov on a ship tour of the Volga. An American journalist who was allowed to join them on the trip described Yagoda as "a spare, slightly-tanned, trim looking, youngish officer" adding that it was "difficult to associate terror with the affable and modest person". By contrast, the chemist Vladimir Ipatieff met Yagoda briefly in Moscow in 1918 and thought that "it was unusual for a young men in his early twenties to be so unpleasant. I felt then that it would be unlucky for me or anyone else ever to fall into his hands." When he saw him again in 1927, "his appearance had changed considerably: he had grown fatter and looked much older and very dignified and important."
Though Yagoda appears to have known Joseph Stalin since 1918, when they were both stationed in Tsaritsyn during the civil war, "he was never Stalin's man" When Stalin ordered that the Soviet Union entire rural population were to be forced onto collective farms, Yagoda is reputed to have sympathised with Bukharin and Rykov his opponents on the right of the communist party, Nikolai Bukharin claimed in a leaked private conversation in July 1928 that "Yagoda and Trilisser are with us", but once it became apparent that the right was losing the power struggle, Yagoda switched allegiance. In the contemptuous opinion of Bukharin's widow, Anna Larina, Yagoda "traded his personal views for the sake of his career" and degenerated into a "criminal" and a "miserable coward".