Bayer was born and raised in Orange, Ohio, a suburb of Cleveland, to a Reform Jewish family. She is the daughter of Carolyn (née Polster) and Todd Bayer. She has a brother, Jonah, who is a music journalist and the drummer for the punk supergroup United Nations.
At the age of 15, she was diagnosed with leukemia. Bayer stated that while battling the disease, she discovered the meaning of “comic relief”. “I don’t know if it made me funnier, but it was so amazing, how it made everything be O.K.,” she said.
While attending college, Bayer interned on the television shows Sesame Street and Late Night with Conan O'Brien and participated in the cast of Bloomers, an all-female musical and sketch comedy troupe. Bayer was part of the all-Jewish cast of Second City's stage show Jewsical: The Musical, a musical that presented a comedic take on Jewish life and culture.