The Wildenberg family was a cadet branch of the Greifenstein family of Filisur. In addition to Wildenberg Castle in Falera, they owned castles and estates throughout the valleys of Graubünden. Wildenberg Castle in Zernez was first mentioned as the home of the family in 1280/90. In 1296 the Bishop of Chur took the right to collect tithes from Wildenberg and pledged it to the Lords of Planta. It is unclear if the castle went with the tithes, but by the 14th century Wildenberg was owned by the bishop and held by the Planta family. In 1365 representatives from the communities around Chur met for the first time at the castle under the auspices of the Planta family. The representatives met because they were concerned about the growing power of Tyrol and the bishop's debts. Two years after the meeting at Wildenberg and building off the 1365 agreements, they founded the League of God's House to curtail the bishop's power.
By around 1400 the branch of the Planta family at Wildenberg was using the name von Wildenberg or Planta-Wildenberg and held the castle as a permanent fief. They became very wealthy with lands and rights from the bishop as well as the mining concession in the Lower Engadine from Emperor Henry VII. While the Planta-Wildenberg family often loaned money to the bishop, their relationship was not always good. In 1430 Conradin von Planta-Wildenberg wrote and demanded the abdication of Bishop Johann von Chur, who in turn excommunicated Conradin. Late in the 15th century the Planta-Wildenberg family supported the bishop against the emperor. In 1498 Hans von Planta-Wildenberg signed a treaty between the League of God's House and the Swiss Confederation which sought to limit the power of the emperor. According to tradition, during the Swabian War of 1499 Hans von Planta-Wildenberg killed an imperial knight at the Battle of Calven.
In 1570 the chronicler Ulrich Campell described Wildenberg as a medieval castle with a central tower and three residential wings. The owner of the castle, Balthasar von Planta-Wildenberg had been married six times and had numerous children who helped increase the wealth and power of the family.
In 1616 Rudolf von Planta-Wildenberg switched from supporting the bishop against the Habsburgs to join the pro-Habsburg party. This infuriated the local villagers and minor nobility. In 1618 when Rudolf received Archduke Leopold of Austria at Wildenberg, the people revolted, attacked and besieged the castle. During the siege, Rudolf and his family escaped but the castle was taken and burned. Four years later, during an Austrian offensive Rudolf was finally able to return to Wildenberg. He rebuilt the damaged castle beginning in 1622. After Rudolf's death in 1638, the castle passed to the a cadet line of the Planta family, the Planta-Aredz. In 1649-50 the castle was again expanded.
The castle continued to be held by the Planta family for several more centuries. In the 18th century, the castle was once again expanded. In 1856 it was sold to the Bezzola family. Then, in 1956, the municipality of Zernez acquired Wildenberg and moved the municipal administration into the building. It was repaired and restored in 1989-94.
The castle is an "L" shaped building with the central tower in the center. Stone walls enclose gardens on the inside of the "L" (to the north-west) and on the south side of the castle. The courtyard is also enclosed with walls and is on the east and north-east side of the castle. The gate into the courtyard garden features the combined coats of arms of Hartmann and Flandrina von Planta from 1650. The medieval tower was topped with a spire in the 18th century. Above the external stairs from 1760 are the coats of arms of Johann von Planta and his wife Maria Jecklin The two wings are connected by a large staircase with a richly stuccoed and painted ceiling in a classical style from the 17th century.