He studied theology under Karl Joseph von Hefele (1809-1893) at the University of Tübingen, and in 1866 continued his studies at the University of Freiburg. During the following year he received his ordination, and in 1868 began teaching classes at the school of theology in Lucerne. During the Franco-Prussian War he served as a field minister in the Bernese Jura during the summer of 1870.
In reaction to the First Vatican Council's decision regarding papal infallibility, he expressed his opinions of opposition at the Old Catholic Congress at Cologne in September-1872. Shortly afterwards he served as an "Old Catholic" parish priest in Krefeld, and in March-1873 started serving as priest in Olten. In 1876 he became pastor at the Church of St. Peter and Paul in Bern, as well as professor at the newly established Old Catholic Faculty of the University of Bern.
In June 1876 he was appointed the first Christian Catholic Church bishop of Switzerland, and on September 18, 1876 was consecrated at Rheinfelden by Joseph Hubert Reinkens (1821-1896). Later that year, he was officially excommunicated by Pope Pius IX.
Herzog died in Bern.