Brown was born in Florida, probably in Gadsden County, and graduated from the Florida Agricultural and Mechanical College. In 1934, he performed at the first National Folk Festival in St. Louis, Missouri. He was musically discovered by folklorist Zora Neale Hurston. She enlisted Alan Lomax, who recorded Brown for the Library of Congress in June 1935.
Like Ralph Willis, Alec Seward and Brownie McGhee, Brown then relocated from the South to New York City. Hurston gave Brown a part in her light opera Polk County. In 1935 Brown started a four-year tenure with the Federal Arts Theatre, initially under the direction of Orson Welles. By the late 1930s, Brown performed as a singer on Cincinnati radio and took part in the show St. Louis Woman. He found employment with the civil service, working for the Army Signal Corps in Asbury Park, New Jersey. His first full recording session was in 1943, produced by Joe Davis, and the twosome worked together until Brown's final sessions in 1952.
In 1945 Brown sang, "I can't have no luck at all, the jinx is on me."
Several of Brown's recordings were not released during his lifetime, and some of those that were issued were not promoted tastefully or accurately. In the late 1940s, various tracks were licensed to Coral Records. Eventually Davis worked in A&R for MGM Records, and Brown followed him to that label, where he was promoted as a pop singer. Compilation albums of his recordings have been released by various labels.