Harrison was the eldest child of Benjamin Harrison IV and Anne Carter, and a grandson of Robert Carter I. The first Benjamin Harrison is said to have arrived in the colonies around 1630. British historian F. A. Inderwick contends that Benjamin IV is also descended from Thomas Harrison, a participant in the regicide of Charles I, but this is disputed. Benjamin IV and Anne built the manor house at Berkeley Plantation in Virginia and he served as a Justice of the Peace and represented Charles City County in the Virginia House of Burgesses.
Benjamin V, one of ten children, was described in his youth as "tall and powerfully built", with "features that were clearly defined, and a well-shaped mouth above a strong pointed chin". His next younger brother, Carter Henry, became a leader in Cumberland County; two other brothers, Henry and Robert, died young. Brother Nathaniel settled in Prince George County, became sheriff, and was later elected to the House of Burgesses. Another brother, Henry, fought in the French and Indian War and later settled at Hunting Quarter in Sussex County. The youngest brother Charles joined the Continental Army and rose to the rank of brigadier general.
On July 12, 1745, at the age of fifty-one, father Benjamin IV, in the midst of an afternoon thunderstorm, with daughters Hannah and Lucy a second in hand, attempted to shut an upstairs window, was struck by lightning, and he and Hannah were killed. Upon his father's death, Benjamin inherited the bulk, but not all, of his father's estate including the family home Berkeley and a number of surrounding plantations; he also assumed ownership and responsibility for the manor house's equipment, stock and numerous slaves. Benjamin's younger siblings inherited another six plantations, possessions and slaves, as the father chose to depart from the tradition of primogeniture (i.e. leaving the entire estate to the eldest son.)
While Harrison's slaves were sustained by the success of the plantations, their treatment presented a picture consistent with the period. It is known that in the division of slaves among his children, Benjamin's father specifically prohibited any splitting of slave families. The Harrisons also assumed a sense of duty to indoctrinate their slaves in Christianity. There were a number of mulattoes among the slaves Benjamin V inherited; while there is no specific information to show who fathered them, it was particularly common for younger plantation family members and guests, as well as overseers, to invade slave dwellings for carnal reasons. In this light, Benjamin IV's decision to ignore primogeniture subjected his younger children and their slaves to potentially precarious circumstances.
Benjamin V followed his father in representing the counties of Charles City and Surry in the House of Burgesses and also served as a county justice. In 1770 Benjamin was among the first signers of an agreement among Virginia lawmakers and merchants boycotting British imports until the British Parliament repealed its taxes on tea. Harrison also joined in sponsoring a bill that declared certain laws passed by Parliament affecting Virginia to be illegal without the consent of His Majesty's subjects in the colony.