James Yorke Macgregor Scarlett was educated at Eton and Trinity College, Cambridge. He was a Captain in the 3rd Battery Queen's Own Cameron Highlanders, and saw active service in the Second Boer War, leaving England for South Africa in February 1900.
Scarlett was the son of William Scarlett, 3rd Baron Abinger, and Helen Magruder. His sister was Evelina Haverfield. He succeeded his father to the title in 1892, and died unmarried without male heirs. The title of Baron Abinger then went to his second cousin Shelley Scarlett, who descended from the 3rd son of the 1st Baron.