Born on Cornwall Road in Waterloo, London, Fraser was the youngest of five children and grew up in poverty. His mother was of Irish and Norwegian ancestry; his father was half Native American. At the age of five he moved with his family to a flat on Walworth Road, Elephant and Castle. Although his parents were not criminals, Fraser turned to crime aged 10 with his sister Eva, to whom he was close. Fraser was a deserter during the Second World War, on several occasions escaping from his barracks. It was during the war that he first became involved in serious crime, with the blackout and rationing, combined with the lack of professional policemen due to conscription, providing ample opportunities for criminal activities such as stealing from houses while the occupants were in air-raid shelters.
In 1941 he was sent to Borstal for breaking into a Waterloo hosiery store, then given a 15-month prison sentence at HM Prison Wandsworth for shop-breaking. Such were the criminal opportunities during the war, Fraser later joked in a television interview many years later that he had never forgiven the Germans for surrendering. In 1942 while serving a prison sentence in HM Prison Chelmsford he came to the attention of the British Army. Although he was conscripted he later boasted that he had never once worn the uniform, preferring to ignore call-up papers, desert and resume his criminal activities.
After the war Fraser was involved in a smash-and-grab raid on a jeweller, for which he received a two-year prison sentence, served largely at HM Prison Pentonville. It was during this sentence that he was first certified insane and was sent to Cane Hill Hospital before being released in 1949. During the 1950s his main occupation was as bodyguard to well-known gangster Billy Hill. He took part in more bank robberies and spent more time in prison. He was again certified insane while at HM Prison Durham and this time sent to Broadmoor Hospital. Afraid of being heavily medicated for bad behaviour, Fraser stayed out of trouble and was released in 1955. In 1956 the British mobster Jack Spot and wife Rita were attacked, on Hill's say-so, by Fraser, Bobby Warren and at least half a dozen other men. Both Fraser and Warren were given seven years for their acts of violence.
It was in the early 1960s that Fraser first met Charlie and Eddie Richardson of the Richardson Gang, rivals to the Kray twins. According to him, it was they who helped him avoid arrest for the Great Train Robbery by bribing a policeman. Together they set up the Atlantic Machines fruit-machine enterprise, which acted as a front for the criminal activities of the gang. In 1966, he was charged with the murder of Richard Hart who was shot at Mr Smith's club in Catford while other members including Jimmy Moody were charged with affray. The witness changed his testimony and the charges were eventually dropped, though Fraser still received a five-year sentence for affray. He was also tried in court in the so-called 'Torture trial', in which members of the Richardson gang were charged with burning, electrocuting and whipping those found guilty of disloyalty by a kangaroo court. Fraser himself was accused of pulling out the teeth of victims with a pair of pliers. Following the trial at the Old Bailey in 1967 he was sentenced to 10 years' imprisonment.
Fraser's 42 years served in over 20 different prisons in the UK were often coloured by violence. He was involved in riots and frequently fought with prison officers and fellow inmates. He also attacked various governors. He was one of the ringleaders of the major Parkhurst Prison riot in 1969, spending the following six weeks in the prison hospital owing to his injuries. Involvement in such activities often led to his sentences being extended. Whilst in Strangeways, Manchester, in 1980 Fraser was 'excused boots' as he claimed he had problems with his feet because another prisoner had dropped a bucket of boiling water on them after Fraser had hit him; he was allowed to wear slippers. He was released from prison in 1985.