He was born John Rees at Ponterwyd in Ceredigion, to a lead miner and farmer, Hugh Rees, and his wife. Rhys was educated at schools in Pantyffynnon and Ponterwyd before moving to the British School, a recently opened institution at Penllwyn, in 1855. Here Rhys was enrolled as a pupil and teacher, and after leaving studied at Bangor Normal College from 1860 to 1861. Upon leaving Bangor Normal College, Rhys gained employment as headmaster at Rhos-y-bol, Anglesey. It was here that Rhys was introduced to Dr Charles Williams, then the Principal of Jesus College, Oxford, in 1865. This meeting eventually led to Rhys being accepted into the college, where he studied literae humaniores. In 1869, he was elected to a fellowship at Merton College.
Rhys also travelled and studied in Europe during this period, staying in Paris, Heidelberg, Leipzig, and Göttingen. He attended lectures by Georg Curtius and August Leskien whilst in Leipzig, and it was during this period that his interest in philology and linguistics developed. Rhys matriculated from Leipzig in 1871, and it was around this time that he adopted the Welsh spelling of his name. He returned to Wales as a government inspector of schools, covering Flint and Denbigh, and he settled in Rhyl. Rhys also began to write, with articles on the grammar of the Celtic language and articles on the glosses in the Luxembourg manuscript being printed, the latter in the Revue Celtique. In 1872 Rhys married Elspeth Hughes-Davies.
In 1874 Rhys delivered a series of lectures in Aberystwyth, later published as Lectures on Welsh Philology, which served to establish his reputation as a leading scholar of the Celtic language. This reputation saw him appointed as the first Professor of Celtic at Oxford University in 1877. He was also made a Fellow of Jesus College, Oxford. Rhys was elected bursar of the college in 1885, a position he held until 1895, when he succeeded Daniel Harper as principal.
Rhys served on several public bodies.
Rhys gained his knighthood in 1907, and in 1911 was appointed to the Privy Council. Rhys was one of the founding Fellows of The British Academy when it was given its Royal Charter in 1902, and after his death the Academy established an annual lecture in his name, the Sir John Rhys Memorial Lecture. The Oxford Dictionary of National Biography declares him to be "foremost among the scholars of his time" in his published fields, noting that "his pioneering studies provided a firm foundation for future Celtic scholarship and research for many decades."
John Morris-Jones and Rhys prepared an edition of The Elucidarium and other tracts in Welsh from Llyvyr agkyr Llandewivrevi A.D. 1346 (The Book of the Anchorite of Llanddewi Brefi), a collection of Medieval Welsh manuscripts in Jesus College Library, which they published in 1894.