Ross was born in London, the nephew of Sir John Ross, under whom he entered the navy in 1812, accompanying him on Sir John's first Arctic voyage in search of a Northwest Passage in 1818. Between 1819 and 1827, Ross took part in four Arctic expeditions under Sir William Parry, and in 1829 to 1833, again served under his uncle on Sir John's second Arctic voyage. It was during this trip that a small party led by James Ross (including Thomas Abernethy) located the position of the North Magnetic Pole on 1 June 1831 on the Boothia Peninsula in the far north of Canada. It was on this trip, too, that Ross charted the Beaufort Islands, later renamed Clarence Islands by his uncle.
In 1834, Ross was promoted to Captain. In December 1835, he offered his services to the Admiralty to resupply 11 whaling ships which had become trapped in Baffin Bay. They accepted his offer, and he set sail in HMS Cove in January 1836. The crossing was difficult, and by the time he had reached the last known position of the whalers in June, all but one had managed to return home. Ross found no trace of this last vessel, the William Torr, which was probably crushed in the ice in December 1835. He returned to Hull in September 1836 with all his crew in good health.
From 1835–1839, except for his voyage with the Cove, he conducted a magnetic survey of Great Britain with Edward Sabine.
Between 1839 and 1843, Ross commanded an Antarctic expedition comprising the vessels HMS Erebus and HMS Terror; he charted much of the coastline of the continent. Francis Crozier was second-in-command of the expedition, commanding HMS Terror. Support for the expedition had been arranged by Francis Beaufort, hydrographer of the Navy and a member of several scientific societies. On the expedition was Joseph Dalton Hooker, who had been invited along as assistant surgeon. Erebus and Terror were bomb vessels – an unusual type of warship named after the mortar bombs they were designed to fire and constructed with extremely strong hulls, to withstand the recoil of the mortars, which were to prove of great value in thick ice.
In 1841, James Ross discovered the Ross Sea, Victoria Land, and the volcanoes Mount Erebus and Mount Terror, which were named for the expedition's vessels. They sailed for 250 nautical miles (460 km) along the edge of the low, flat-topped ice shelf they called variously the Barrier or the Great Ice Barrier, later named the Ross Ice Shelf in his honour. In the following year, he attempted to penetrate south at about 55°W, and explored the eastern side of what is now known as James Ross Island, discovering and naming Snow Hill Island and Seymour Island. Ross reported that Admiralty Sound (which he named Admiralty Inlet) appeared to him to have been blocked by glaciers at its southern end. He was awarded the Gold Medal of the Société de Géographie in 1843, knighted in 1844, and elected to the Royal Society in 1848.