After stints with Elmer Snowden, Fletcher Henderson, Horace Henderson, McKinney's Cotton Pickers, and Luis Russell, Stewart joined the Ellington band in 1934, replacing Freddie Jenkins. Duke Ellington arranged many of his pieces to showcase Stewart's half-valve effects, muted sound, and forceful style.
Stewart co-wrote "Boy Meets Horn" and "Morning Glory" while with Ellington, and frequently supervised outside recording sessions by members of the Ellington band. After eleven years Stewart left to lead his own groups - " little swing bands, that were a perfect setting for his solo playing." He also toured Europe and Australia with Jazz at the Philharmonic from 1947 to 1951. He made a cameo appearance in Jacques Becker's 1949 film Rendezvous in July. From the early 1950s on he worked in radio and television and published highly regarded jazz criticism. The book Jazz Masters of the Thirties is a selection of his criticism. Stewart also wrote for Playboy, Down Beat and several other publications during his life. He lived in upstate New York after purchasing a 100 year old farmhouse. He hosted a jazz radio program in Troy, New York and owned a small restaurant for a very short time near a drag racing stadium in Vermont. While living in France, he attended the Cordon Bleu school of cooking and dedicated his life to being a fine cook.
Stewart moved to Los Angeles, California to be near his three children - Rex Jr., Helena and Regina. His other son Paul Albert Hardy lived in New York City. While in Los Angeles he re-connected to many of the Ellington side-men who lived there and played a lot of "jam" sessions in clubs in the Los Angeles area. Rex was also one of the regular studio musicians seen on the Steve Allen TV show. He hosted and coproduced (with George Cole) two radio shows about jazz music from his era, Dixieland Doings and Things Aint What They Used To Be.