Smith was born at Irthington, near Carlisle, Cumberland, the son of a gardener to the Gilpin family, and educated at St. Bees. The fortunate social connection allowed him to study art under the animal painter Sawrey Gilpin.
Becoming known as a skilful topographical draughtsman, he was employed on Samuel Middiman's Select Views in Great Britain, and obtained the patronage of George Greville, 2nd Earl of Warwick, which enabled him to travel to Italy between 1776 and 1781. While there he met other British artists such as Francis Towne, Thomas Hearne and William Pars. He came to be known as "Warwick" or "Italian" Smith. In his subsequent works, which were largely views in Italy, he gradually abandoned the simple tinting to which watercolour work had previously been limited for a more effective mode of colouring, the novelty and beauty of which was much admired.
He returned to England in 1781 in the company of Francis Towne, travelling through the Italian lakes and Switzerland. He settled in Warwick, making frequent visits to Wales from 1784; he also toured the Lake District in the late 1780s and early 1790s. In 1783 he married Elizabeth Gerrard – still a minor – at St Mary's church, Warwick. He visited Italy again in 1785–6, this time in the company of Lord Warwick. Lord Warwick acquired a large collection of his work, which was eventually dispersed at an auction in 1936.
On one of his tours of Wales, at some time after 1788, Smith was accompanied by Lord Warwick's brother, Robert Fulke Greville, and the artist Julius Caesar Ibbetson . They spent a considerable time at Hafod, near Aberystwyth, the home of the bibliophile Thomas Johnes. Hafod was destroyed by fire in 1807, and three years later Sir J. E. Smith published A Tour to Hafod, illustrated with fifteen aquatints by J. G. Stadler from watercolours, by " Warwick " Smith, possibly made in the course of the visit with Ibbetson.
Other books illustrated with engravings after works by Smith include Select Views in Italy (1792-6); Views of the Lakes of Cumberland, with twenty aquatints by James Merigot (1791-5); William Byrne's Britannia Depicta and William Sotheby's Tour through Wales (1794). This last book was subtitled "Odes and other Poems. With Engravings from Drawings taken on the Spot by J. Smith". In the preface, Sotheby wrote "the author of the following Poems thinks proper to signify, that the present edition is published solely for the emolument of the artist, who has stamped a value on the descriptive parts of the Welsh Tour, by the embellishments of his accurate and masterly pencil". The plates were etched in aquatint by Samuel Alken.
Smith moved to London in 1807. In that year he began exhibiting with Watercolour Society, which he had joined two years before. He remained a major contributor to its exhibitions until 1823, when he resigned his membership. He was elected president of the society in 1814, 1817, and 1818, secretary in 1816, and treasurer in 1819, 1821, and 1822.