Rapp was born the son of the janitor of the town-hall of Colmar, then located in the Old Customs House. He began theological studies to become a clergyman, but with his build and heated character, he was better suited to the military, which he joined in March 1788. From the rank of a regular of the chasseurs de Cévennes, he worked his way up through his courage and character to the rank of a division general and adjutant of Napoleon Bonaparte. As a lieutenant, his reputation grew through his impetuousness as well as the wounds he received in battle. He was made aide-de-camp of Louis Desaix, who named him captain and took him to Egypt, where Rapp distinguished himself at Sediman, capturing an enemy battery. For that he was given a squadron and later a brigade by Napoleon.
After the Egyptian campaign, Rapp remained under the command of Desaix until the latter's death at Marengo on 14 June 1800. He then became aide-de-camp of Napoleon, then the First Consul, a post he held until 1814. Under this title, he was charged with many confidential missions by Napoleon in the Vendée, Switzerland and Belgium. In 1803 he was promoted to brigadier general and in December 1805, he led a memorable attack at Austerlitz, when he charged at the head of two squadrons each of the Mounted Chasseurs and the Mounted Grenadiers of the Guard and the Guard Mameluks and decimated the Chevalier Guards of the Russian Imperial Guard. Promoted to division general, he later fought at Jena on 14 October 1806 and was wounded at Golymin.
Rapp stayed in the line of fire: at Essling, he led the front of his fusiliers of the Garde impériale and carried the day; during the signing of the Treaty of Schönbrunn, he averted a planned attempt on Napoleon by the young Friedrich Staps; in Russia, he was wounded by four bullets at the battle of Moscow on 5-7 September 1812. He saved Napoleon's life a second time by repelling an attack of Cossacks at the Gorodnia and was again wounded at the passage of the Berezina, fighting alongside Ney in the rear guard. As governor of Danzig, Rapp held the town for a year after the Grande Armée left Russia.
During the Hundred Days, Rapp rallied to Napoleon and was given command of V Corps, consisting of about 20,000 men. It was used to observe the border near Strasbourg, and to defend the Vosges. Ten days after the battle of Waterloo (in which his corps took no part), he met some Coalition forces near Strasbourg and defeated them at the Battle of La Suffel. After the Waterloo Campaign, he offered his resignation several times, but was reinstated. Later, Rapp became a deputy of the department of Haut-Rhin and was appointed as treasurer of Louis XVIII in 1819.
He died in Rheinweiler in Baden. His hometown of Colmar built a statue in his honor on the Champ de Mars with the inscription Ma parole est sacrée (my word of honour is sacred). Rapp's heart is kept in a shrine in the Église Saint-Matthieu, Colmar.