He was the son of a ship carpenter and sparmaker. His birthplace and childhood home, Belfast, Maine, was located on the west side of Penobscot Bay, and the bay and harbor provided much opportunity for activity. His father died while Dudley was a boy, and his resulting situation in life required him to devote much of his time outside of school hours to manual labor on land and at sea, under the direction of an uncle.
During this time he joined with some other high school boys in putting up a horizontal bar and other apparatus on the high school grounds, and they started a gymnastic club. They were additionally inspired by reports of exhibitions given at Bowdoin College, 60 miles away in Brunswick, and gave similar public exhibitions themselves at the town hall, and in towns nearby.
In 1869, he was invited by Bowdoin College to direct the gymnasium there. Two years later he entered the College as a student, from which he graduated (A.B.) in 1875 and in 1878 received the degree of M.D. at Yale Medical School. During his time in the medical school, he was also an instructor in gymnastics at Yale College. After his graduation from Yale, he moved to New York City, where he conducted a private gymnasium for a year. He assessed the physical condition of clients using a physical examination and adapted an exercise regimen on various pieces of exercise equipment accordingly.
From 1879 until his retirement in 1919, Sargent was director of the Hemenway Gymnasium at Harvard University. From 1879 to 1889 was assistant professor of physical training at Harvard. His nomination in 1889 for full professor was blocked by alumni on the Board of Overseers at Harvard who were annoyed about an 1885 ban the faculty athletic committee, of which Sargent was a member, had put on football. Although Sargent was accused of being opposed to football itself, the ban stemmed from "rough play and fighting." The athletic committee was founded in 1882, Sargent being a founding member, as a pioneering effort by Harvard to regulate intercollegiate competition.
In 1881-1916 Sargent was director of the Normal School of Physical Training at Cambridge, Massachusetts, and after 1916 was president of its successor, the Sargent School of Physical Education. This school was for the training of teachers of physical education. In 1883, it found accommodations at 20 Church Street, and in 1904 in a new building on Everett Street. Initially, only women were taught; in 1904 the school was open to men as well. Sargent challenged the Victorian tradition of females prone to fainting, and encouraged freedom of dress and vigorous activity for girls and women. Sargent taught both German and Swedish gymnastics although at this time the two were competing in a "battle of systems." In 1907-16, Sargent was president of the Health Education League.
He married Ella Frazier Ledyard in 1881. They separated after a few years together, and had one child, Ledyard, who later continued his father's work.
Sargent Center for Outdoor Education, owned by Boston University in Hancock, NH is named in honor of Dudley Sargent.
He is the inventor of gymnasium apparatus, of the Sargent Anthropometric Charts, and published:
and numerous articles and papers on physical education.