Sheffield was the son of John Baker-Holroyd, 1st Earl of Sheffield, by his third wife Lady Anne, daughter of Prime Minister Frederick North, 2nd Earl of Guilford, and succeeded his father in the earldom in 1821 at the age of nineteen. He was later able to take a seat in the House of Lords in right of his junior title of Baron Sheffield, which was in the Peerage of the United Kingdom. He served as a Lord-in-waiting (government whip in the House of Lords) from 1858 to 1859 in the Conservative administration of the Earl of Derby. John's second wife was Lady Lucy Pelham, 1763-1797,a daughter of the Earl of Chichester, who died without issue.
Lord Sheffield married Lady Harriet, daughter of Henry Lascelles, 2nd Earl of Harewood, in 1825. He died in April 1876, aged 74, and was succeeded in his titles by his second but eldest surviving son Henry. The Countess of Sheffield died in January 1889.