Johnson was born in Holly Springs, Mississippi. Several of his brothers had careers in music, among them the soul musician Syl Johnson and the bassist Mack Thompson, who played with Magic Sam. In his younger years he played piano and sang in gospel groups. He and his family moved to Chicago in 1950, where he worked as a welder and played guitar in his spare time. He began playing professionally with Slim Willis in 1959, changing his last name to Johnson, as did his brother Syl. As a guitarist he was influenced by Buddy Guy and Otis Rush. He played with Freddy King, Albert King, Magic Sam, Otis Rush, and Eddy Clearwater, among others.
In the 1960s he played music in more of an R&B style, working with Otis Clay, Denise LaSalle, and Garland Green. He had his own group from the early 1960s, and by the mid-1960s he had released his first single. By 1974, Johnson had returned to playing blues, working with Jimmy Dawkins and touring Japan with Otis Rush in 1975.
His first solo albums appeared on MCM Blues Records in 1978 and Delmark Records in 1979, when he was fifty years old. He was an award-winner at the first Blues Music Awards, held in Memphis in November 1980. His career continued to pick up until December 2, 1988, when his touring van crashed in Indiana, killing his band's keyboardist, St. James Bryant, and bassist, Larry Exum. Johnson was injured and took an extended break from the music industry. He returned to record for Verve Records in 1994. In 2002, he recorded with his brother Syl. He remained active and toured Europe in 2009, performing in the UK and at the Copenhagen Blues Festival in Denmark.
The subject of this article has sometimes been confused with another blues musician called Jimmy Johnson, who was born James Franklin Johnson in Fordyce, Arkansas, on February 23, 1926, and died in Los Angeles on November 25, 2000. James Franklin Johnson wrote the song "Don't Answer the Door", which was recorded by B. B. King.