Torgrim Castberg was born in Skien as the son of customs surveyor and politician Johan Christian Tandberg Castberg (1827–1899) and his wife Hanna Magdalene Frisak Ebbesen (1839–1881). He had several brothers and sisters, including the notable politician Johan.
His paternal grandfather Peter Hersleb Harboe Castberg was a priest and politician. In addition, Torgrim was a first cousin of sculptor Oscar Ambrosius Castberg. On the maternal side he was a grandson of Jørgen Tandberg Ebbesen and nephew of Just Bing Ebbesen, both of whom were involved in politics too.
He died in 1928 in Paris.