She was born at Beaulieu Abbey, Hampshire, on 26 September 1909, the second daughter of the 2nd Lord Montagu by his first wife Lady Cecil Kerr. She was a direct descendant of Charles II and Lucy Walter, via their son, the Duke of Monmouth. Her mother died in the 1918 flu pandemic. Varley attended St Margaret's School, Bushey, and attended a finishing school at Lausanne, but was not interested in society life and chose instead to train as an actress at the RADA from 1932.
She began an acting career at the Newcastle Repertory Company and appeared in a number of stage productions in the West End and on BBC Radio dramas. She was personal assistant to Arturo Toscanini while he was in London to conduct at the BBC's London Music Festivals in 1937–38. She helped Walter Legge found the Philharmonia Orchestra. After the war, she worked on film scripts for film directors, such as Alexander Korda.
In 1948 she met Graham Greene in Vienna. She showed him (and later Orson Welles) some of its less reputable night-clubs while he worked on the script for The Third Man. She introduced Greene to Peter Smolka, the eastern European correspondent for The Times. Smolka gave Greene stories about the black market in Vienna. She was a committee member of Benjamin Britten's English Opera Group and she wrote the libretti for Rolf Liebermann and Paul Burkhard.
She travelled extensively throughout Europe in 1935–39, and also studied at a piano school in Switzerland. When asked about her racial background while applying for a visa at the German consulate in St. Gallen, she angrily swept items – including a portrait of Hitler – from the desktop onto the floor. Her passport was confiscated.
She was sent to France at the start of the Second World War as an ambulance driver. After her unit retreated to Bordeaux, she refused to leave and hid at a friend's home for several months. On a train to Switzerland she noticed she was under surveillance, so she asked two hat salesmen for help at the next station. She escaped while hidden under the boxes in their car.