Ahmad Yani was born in Jenar, Purworejo, Central Java on 19 June 1922 to the Wongsoredjo family, a family that worked at a sugar factory run by a Dutch owner. In 1927, Yani moved with his family to Batavia, where his father now worked for a Dutch General. At Batavia, Yani worked his way through primary and secondary education. In 1940, Yani left high school to undergo compulsory military service in the Army of the Dutch East Indies colonial government. He initially trained to be a Navy Seaman. He studied military topography in Malang, East Java, but this education was interrupted by the arrival of the invading Japanese in 1942. At the same time, Yani and his family moved back to Central Java.
In 1943, he joined the Japanese sponsored Peta (Defenders of the Motherland) army, and underwent further training in Magelang as an Artillery officer. After completing this training, Yani enlisted to be trained as a Peta platoon commander and was transferred to Bogor, West Java to receive his training. Upon completion, he was sent back to Magelang as an instructor.
After Independence Yani joined the army of the fledgling republic and fought against the Dutch. During the first months after the Declaration of Independence, Yani formed a battalion with himself as Commander and led it to victory against the British at Magelang. Yani then followed this up by successfully defending Magelang against the Dutch when it tried to take over the city, earning him the nickname of the "Savior of Magelang". Another notable highlight of Yani's career during this time period was the series of guerilla offensives he launched in early 1949 to distract the Dutch whilst Lieutenant Colonel Suharto prepared for the 1 March General Offensive which was to be directed at Yogyakarta.
After Indonesia's Independence was recognized by the Dutch, Yani was transferred to Tegal, Central Java. In 1952, he was called back into action to fight the Darul Islam, a group of rebels seeking to establish a theocracy in Indonesia. To deal with this rebel group, Yani formed a special forces group called the Banteng Raiders (today the 400th Raider Infantry Battalion, Kodam IV/Diponegoro). The decision to call in Yani paid dividends and over the next 3 years, Darul Islam forces in Central Java suffered one defeat after another.
In December 1955, Yani left for the United States to study at the Command and General staff College, Fort Leavenworth, Kansas. Returning in 1956, Yani was transferred to the Army Headquarters in Jakarta where he became a staff member for General Abdul Haris Nasution. At the Army Headquarters, Yani served as Logistics Assistant to the Army Chief of Staff before becoming Deputy Army Chief of Staff for Organization and Personnel.