Brown apprenticed with Mississippi Joe Callicott, who was his neighbor in Nesbit, Mississippi, from age 12 to 15, when Callicott died. He had heard Othar Turner and others in nearby Como picnics, and cited Junior Kimbrough, Johnny Winter, and Johnny Shines as influences.
Around 1971, beside working in construction, Brown began playing with two other musicians. Johnny Woods would make an occasional playing partner to his death in 1990. More steady was Brown's learning with R. L. Burnside, who claimed Brown as his "adopted son," and affectionately called him "white boy on guitar" and "my white son." Brown has noted that they had trouble to book dates, when European event organizers would hear he is a white musician playing the traditionally African American blues, and that American record producers and critics have similar reservations.
Still in the early seventies they started to perform in their region, and would keep up as a duo for twenty years. Cedric Burnside joined their tours from about 1994, as Burnside's reputation surged. In the 1990s and early 2000s Brown participated in most of Burnside's tours and recordings, including the Burnside-Jon Spencer Blues Explosion collaborations and the remixed albums.
On record, he plays second guitar on two of Junior Kimbrough's albums throughout, and on some tracks on the posthumous compilation, God Knows I Tried. He is on tracks by Asie Payton, CeDell Davis and Paul "Wine" Jones, as well as Frank Frost and Cyndi Lauper.
Brown's own debut album was Goin' Back to Mississippi (1996), produced by Dale Hawkins. He has recorded one album for Fat Possum Records, Stingray (2003). He released Cheap, Fast, and Dirty (2006) with Danish guitarist Troels Jensen, at Olufsen Records. Meet Ya In The Bottom (2008) is a CD Baby release. His double album Can't Stay Long (2011)  was released on Devil Down Records.
Brown's guitar work was featured in the 2006 film Black Snake Moan, where he provided backing for star Samuel L. Jackson's vocals. He can be seen in the film's climax as a guitarist in a blues band, playing alongside Cedric Burnside.