The origins of the borough began in 1941, when the federal government acquired 91 acres (370,000 m2) in Randolph Township as the site of a 300-unit housing project for war industry employees. The borough's name is derived from the victory gardens planted at homes and parks during World War II to provide additional supplies of fruits and vegetables. The federal government paid for all infrastructure. Streets are named for U.S. Presidents.
Randolph Township residents approved a referendum as part of a September 1951 special election in which voters were asked if the township's Victory Gardens neighborhood should be removed from the township and created as an independent municipality for its 1,300 residents covering 92 acres (37 ha). Residents of other areas of Randolph Township argued that the compensation paid by the federal government for the more than 250 students attending the Randolph Township Schools did not adequately cover the cost of their public education, that the housing and other structures in Victory Gardens was out of compliance with the Township's building and zoning ordinances and that the overwhelming Democratic Party political leanings of residents of Victory Gardens were out of sync with the largely Republican Party township.
Victory Gardens was incorporated as a borough by an act of the New Jersey Legislature on June 20, 1951, from portions of Randolph Township, based on the results of the referendum passed on September 18, 1951.
A project approved in 1973 brought the construction of 184 units of garden apartments on a site covering 12.4 acres (5.0 ha), providing additional rateables and offering permanent housing for an estimated 400 people, that would contrast with the temporary original structures built in the 1940s that had long passed their expected lifespan.