Aliza Herz was born in Prague, in the Kingdom of Bohemia (a part of Austria-Hungary), to Friedrich and Sofie "Gigi" Herz. Herz's family was part of the small German-speaking minority of assimilated Jews in Prague, although Herz stated that she also spoke Czech. Her father was a merchant and her mother was highly educated and moved in circles of well-known writers. She had two sisters, including a twin sister, Mariana, and two brothers. Her parents ran a cultural salon where Herz, as a child, met writers including Franz Kafka and Franz Werfel, composers including Gustav Mahler, philosophers, and intellectuals such as Sigmund Freud. Herz once noted that "Kafka was a slightly strange man. He used to come to our house, sit and talk with my mother, mainly about his writing. He did not talk a lot, but rather loved quiet and nature. We frequently went on trips together. I remember that Kafka took us to a very nice place outside Prague. We sat on a bench and he told us stories." Herz's sister Irma was married to Felix Weltsch, was a prominent German-language Jewish philosopher, journalist, librarian, and Zionist who later worked as a librarian in Jerusalem after his emigration from Austria.
Herz's older sister Irma taught her how to play the piano, which she studied diligently, and the Austrian-Jewish pianist Artur Schnabel, a friend of the family, encouraged her to pursue a career as a classical musician; a choice she decided to make. She went on to study under the Czech pianist Václav Štěpán (1889-1944) and at the Prague German Conservatory of Music, where she was the youngest pupil. Herz married the businessman and amateur musician Leopold Sommer in 1931; the couple had a son, Stephan (later known as Raphael, 1937–2001). She began giving concerts and making a name for herself across Europe until the Nazis took over Prague, as they did not allow Jews to perform in public, join music competitions or teach non-Jewish pupils.
After the invasion of Czechoslovakia, most of Herz-Sommer's family and friends emigrated to Palestine via Romania, including Max Brod and brother-in-law Felix Weltsch, but Herz-Sommer stayed in Prague to care for her ill mother, Sofie, aged 72; both women were arrested and Sofie Herz was murdered in a concentration camp. In July 1943 Herz was sent to Theresienstadt, where she played in more than 100 concerts along with other musicians, for prisoners and guards. She commented of her performances in the camp:
We had to play because the Red Cross came three times a year. The Germans wanted to show its representatives that the situation of the Jews in Theresienstadt was good. Whenever I knew that I had a concert, I was happy. Music is magic. We performed in the council hall before an audience of 150 old, hopeless, sick and hungry people. They lived for the music. It was like food to them. If they hadn't come , they would have died long before. As we would have.
Herz-Sommer was billeted with her son during their time at the camp, he was one of only a few children to survive Theresienstadt. Her husband died of typhus in Dachau, six weeks before the camp was liberated.