Carroll O'Connor, an Irish American, was the eldest of three sons. He was born on August 2, 1924, in Manhattan, New York City, to Edward Joseph O'Connor, a lawyer, and his wife, Elise Patricia O'Connor. Both of his brothers became doctors: Hugh, who died in a motorcycle accident in 1961, and Robert, a psychiatrist in New York City. O'Connor spent much of his youth in Elmhurst and Forest Hills, Queens, the same borough in which his character Archie Bunker would later live.
In 1941, Carroll O'Connor enrolled at Wake Forest University in North Carolina, but dropped out when the United States entered World War II. During the war, he was rejected by the United States Navy and enrolled in the United States Merchant Marine Academy for a short time. After leaving that institution, he became a merchant seaman and served in the United States Merchant Marine during World War II.
After the war, O'Connor attended the University of Montana-Missoula, where he met Nancy Fields, who later became his wife. He also worked at the Montana Kaimin student newspaper as an editor. At the University of Montana, he joined Sigma Phi Epsilon fraternity. O'Connor did not take any drama courses as an undergraduate at the University of Montana. He later left that university to help his younger brother Hugh get into medical school in Ireland, where Carroll completed his studies at the University College Dublin and began his acting career.
After O'Connor's fiancée, Nancy Fields, graduated from the University of Montana in 1951 with degrees in drama and English, she sailed to Ireland to meet Carroll, who was visiting his brother, Hugh. The couple married in Dublin on July 28, 1951. In 1956, O'Connor returned to Missoula to earn a master's degree in speech. O'Connor said on The Dick Cavett Show, in December 1971, that the level of education that he achieved was a master's degree from the University of Montana and that he had completed a bachelor's degree at the University of Dublin.