Schmidt was born in Pozsony (known in German as Pressburg), in the Hungarian part of the Austro-Hungarian Empire (the city is now Bratislava, capital of Slovakia). His father was half Hungarian and his mother entirely Hungarian. He was a Roman Catholic.
His earliest teacher was his mother, Mária Ravasz, an accomplished pianist, who gave him a systematic instruction in the keyboard works of J. S. Bach. He received a thorough foundation in theory from Brother Felizian Moczik, the outstanding organist at the Franciscan church in Pressburg. He studied piano briefly with Theodor Leschetizky, with whom he clashed. He moved to Vienna with his family in 1888, and studied at the Vienna Conservatory (composition with Robert Fuchs, cello with Ferdinand Hellmesberger and theory (the counterpoint class) with Anton Bruckner), graduating "with excellence" in 1896.
He beat 13 other applicants and obtained a post as cellist with the Vienna Court Opera Orchestra, where he played until 1914, often under Gustav Mahler. Mahler habitually had Schmidt play all the cello solos, even though Friedrich Buxbaum was the principal cellist. Schmidt was also in demand as a chamber musician. Schmidt and Arnold Schoenberg maintained cordial relations despite their vast differences in style. Also a brilliant pianist, in 1914 Schmidt took up a professorship in piano at the Vienna Conservatory, which had been recently renamed Imperial Academy of Music and the Performing Arts. (Apparently, when asked who the greatest living pianist was, Leopold Godowsky replied, "The other one is Franz Schmidt.") In 1925 he became Director of the Academy, and from 1927 to 1931 its Rector.
As teacher of piano, cello and counterpoint and composition at the Academy, Schmidt trained numerous musicians, conductors and composers who later achieved fame. Among his best-known students were the pianist Friedrich Wührer and Alfred Rosé (son of Arnold Rosé, the legendary founder of the Rosé Quartet, Konzertmeister of the Vienna Philharmonic and brother-in-law of Gustav Mahler). Among the composers were Theodor Berger, Marcel Rubin and Alfred Uhl. He received many tokens of the high esteem in which he was held, notably the Franz-Josef Order, and an Honorary Doctorate from the University of Vienna.
Schmidt's private life was in stark contrast to the success of his distinguished professional career, and was overshadowed by tragedy. His first wife Karoline Perssin (c1880-1943) was confined in the Vienna mental hospital Am Steinhof in 1919, and three years after his death was murdered under the Nazi euthanasia program. Their daughter Emma Schmidt Holzschuh (1902-1932, married 1929) died unexpectedly after the birth of her first child. Schmidt experienced a spiritual and physical breakdown after this, achieved an artistic revival and resolution in his Fourth Symphony of 1933 (which he inscribed as "Requiem for my Daughter") and, especially, in his oratorio The Book With Seven Seals. His second marriage in 1923, to a successful young piano student Margarethe Jirasek (1891-1964), for the first time brought some desperately needed stability into the private life of the artist, who was plagued by many serious health problems.