Miller was born in Lebanon, Pennsylvania and grew up in Ono, Pennsylvania. He attended East Stroudsburg State College where he played on both the soccer and basketball teams. He was a midfielder who earned second team All American recognition in 1958 and 1959.
While he starred as a soccer and basketball player, his first job was as the Albright College golf coach. He gained his first soccer coaching position with New Paltz State. He spent five years with the New Paltz, winning three New York Conference Championships.
In 1967, Hartwick College hired Miller. In his six seasons at Hartwick, he took the team to a 64-12-3 record and a 1970 Final Four appearance.
Several events led to Miller moving from the collegiate to the professional coaching ranks. Tom McCloskey, a wealthy Philadelphia businessman was interested in owning a professional sports team. While attending the 1973 Super Bowl, Lamar Hunt broached the subject of McCloskey starting a North American Soccer League (NASL) franchise in Philly. The NASL awarded McCloskey a team to begin play in the 1973 season. With very little time to prepare, McCloskey, and his general manager Bob Ehlinger, hired Miller who was both a successful collegiate soccer coach and from the Philadelphia area, as the Philadelphia Atoms first coach.
Miller decided to build the Atoms around a core of native U.S. players. This was significant in that most NASL teams used U.S. players as goalkeepers or bench warmers. Miller took Bob Rigby with the first pick in the 1973 NASL college draft. Rigby went on to become one of the best goalkeepers in the NASL. In the second round, Miller selected Bobby Smith who was selected as a second team All Star his rookie year. Miller then took his team to England to train. While there noticed several players who he thought could help his team. He negotiated an on loan arrangement for Andy Provan, Jim Fryatt and Chris Dunleavy.