Pelham was the son of Thomas Pelham and his wife Annetta, daughter of wealthy merchant George Bridges (d.1714) of Pera, Constantinople by his wife Anetta, a local girl. Sir John Pelham, 3rd Baronet, was his great-grandfather and Thomas Pelham-Holles, 1st Duke of Newcastle, and Henry Pelham his first cousins once removed. He was educated at Westminster School (1740) and Clare College, Cambridge (1745) and undertook the Grand Tour through France, Switzerland, Italy and Germany between 1746 and 1750.
Pelham was elected to the House of Commons for Rye in 1749, a seat he held until 1754, and then represented Sussex until 1768. He served as a Commissioner of Trade and Plantations from 1754 to 1761, as a Lord of the Admiralty from 1761 to 1762 and as Comptroller of the Household from 1765 to 1774 and was admitted to the Privy Council in 1765.
In 1768 Pelham succeeded his cousin the Duke of Newcastle as second Baron Pelham of Stanmer according to a special remainder in the letters patent. He also inherited the Pelham baronetcy created in 1611. Pelham also served as Surveyor-General of Customs of London from 1773 to 1805 and as the last Keeper of the Great Wardrobe from 1775 to 1782. In 1801 he was created Earl of Chichester.
He succeeded his father in 1737, inheriting the Stanmer Park estate near Lewes, Sussex.
Lord Chichester married Anne Frankland, daughter of Frederick Meinhardt Frankland, in 1754. They had three sons and three daughters. All three daughters and one son predeceased him. His third son the Right Reverend the Hon. George Pelham became Bishop of Bristol, Exeter and Lincoln. Lord Chichester died in January 1805, aged 76, and was succeeded in his titles by his eldest son Thomas, who became a prominent politician. Lady Chichester died in 1813.