Scavullo was born in Staten Island, New York City. Early on, he used his father's camera to photograph his sisters who would model for him. After graduating from high school in 1945, Scavullo began working for a studio that produced fashion catalogs. He soon moved to Vogue. Scavullo spent three years as Horst P. Horst's assistant, studying Horst's techniques. In 1948, he created a cover for Seventeen that won him a contract with the magazine. Scavullo soon opened his own studio in Manhattan.
In 1969, Scavullo painted singer Janis Joplin, with a cigarette in her hand, a picture exhibited at the Amon Carter Museum in Fort Worth, Texas. The museum poster refers to Joplin, who died in 1970, as having a "free-spirited fervor of the counterculture revolution."
Some of Scavullo's more controversial work included a Cosmospolitan centerfold of a nude Burt Reynolds, and photographs of a young Brooke Shields that some considered overly sexual. He also befriended a young teenager from Philadelphia, future supermodel Gia Carangi, whose career he was largely responsible for launching. Later, when Carangi's heroin addiction made it impossible for her to find work, Scavullo continued to employ her and support her until her eventual death from complications relating to AIDS.
Scavullo also created shots for various movie posters, album covers and Broadway shows, including one for A Star is Born (featuring Barbra Streisand and Kris Kristofferson), Judy Collins' Hard Times for Lovers, a portrait of Julie Andrews for Blake Edwards' Victor Victoria., and the cover and in-sleeve photos of Donna Summer for her Once Upon A Time and Live and More double albums. In 1981, Scavullo was commissioned by Mikhail Baryshnikov to photograph the dancers of the American Ballet Theatre which formed the basis of an exhibition that was later shown in a nationwide tour.