The castle was originally built in the 11th century in timber, with a square keep added in stone in 1107. The castle was of primarily local significance, being relatively small. During the Anarchy of the 12th century, the castle was seized in the early years of the conflict by Robert, Earl of Gloucester on behalf of the Empress Matilda. In 1142 King Stephen seized the castle in a surprise attack, setting fire to it afterwards.
Subsequent events are disputed by historians. Most 19th-century historians argued that the castle was subsequently rebuilt and held by William de la Dive, a follower of Robert, Earl of Leicester, then a supporter of the Empress. When Robert came to an agreement with Stephen at the end of the conflict, William surrendered the castle to the king. During the baronial revolt against Henry III the castle was once again garrisoned against the king; once captured by royal forces, Henry ordered it finally destroyed for good. Historian E. Fuller argued in 1890, however, that this later history was a consequence of a misreading of place names, arguing that the history of the castle concluded with its destruction in 1142.