Harriet was the daughter of Harriet Browne Hanson and William Hanson, a carpenter. Both parents were descended from early English settlers, but without distinguished ancestors. Her elder brother was John Wesley Hanson (1823–1901), and she had two surviving younger brothers, Benjamin and William. Harriet's father died when she was six, leaving his widow to support four young children.
Harriet's mother was determined to keep her family together, despite the difficulty in doing so. Harriet later recalled, in her autobiography Loom and Spindle (Hanson Robinson 1898), her mother's response when a neighbour offered to adopt Harriet so that her mother had one less mouth to feed: "No; while I have one meal of victuals a day, I will not part with my children." She later wrote that her mother's words on that occasion stuck with her "because of the word 'victuals'" whose meaning she wondered about for a long time thereafter.
Initially, Mrs. Hanson ran a small store in Boston, Massachusetts, which sold food, candy, and firewood. The family lived in the back room of the shop, all sharing one bed "two at the foot and three at the head" as Harriet was later to recall. At the invitation of Harriet's maternal aunt, Angeline Cudworth, also a widow, the family moved to Lowell, Massachusetts, a center of the textile industry. Lowell was a planned factory town. Under the Lowell System, the company recruited young farm girls to work in the mills, building dormitories where they could live at low rents, and hiring matrons to monitor the social conduct of the girls. The company arranged for cultural events, bible studies and other educational opportunities. Initially, the pay was appealing to the women, as the alternative employment, domestic work and teaching, paid only about one-sixth as much as the mills However, working conditions were often poor and unsafe and wages low, leading to a strike in 1834. Mrs. Hanson obtained a job running a boarding house for a textile mill company in Lowell.
At the age of ten, Harriet started to work in the Lowell mills. By her own account, she wanted to work so she could earn money for herself, and the experience was good for her. However, there may have been an element of necessity since her mother earned little money from running the boarding house. Her job, which paid $2 per week, was as a "doffer", replacing full bobbins with empty ones. The job took only a quarter of each hour, and during the free periods the children could play or read or even go home for a while.
In 1836, the Lowell Mill Girls organized their second strike, or "turn out" as they called them. The first strike had been in 1834 over a 15% cut in wages. This second strike was over an increase in board charges that was equivalent to a 12.5% cut in wages. To Harriet, aged eleven, it was her first strike. In her autobiography, she recounted it with pride: