The castle, which may have originally been constructed by a Norseman by the name of Torquil, was a stronghold of Clan Mackay that they used as a base to carry out raids on their neighbours the Clan Sutherland. The Mackays failed to appear in front of Mary of Guise, the mother of Mary, Queen of Scots, and as a result John Gordon, 11th Earl of Sutherland, chief of Clan Sutherland was ordered to destroy the castle. Guise hired a private ship, the Lion in August 1554 to attack the Castle. Sir Hew Kennedy of Girvanmains, the Earl's stepfather (or Stepbrother), embarked 50 men of war and the royal gunner Hans Cochrane with a cannon. It is also said that a cannon was dragged all the way from Edinburgh. Rory Mor Mackay, Captain of the castle was apparently dismayed to see that the enemy was armed with a cannon as the castle had withstood all previous sieges. The castle was destroyed and Rory Mor Mackay was hanged. In November Kennedy returned to Edinburgh with Iye Du Mackay, 12th of Strathnaver who was imprisoned in Dumbarton and then Edinburgh Castle.
After Borve Castle was destroyed a force from the Clan Mackay assembled under John Mor Mackay and engaged in battle with the Clan Sutherland, known as the Battle of Garbharry where the Mackays were defeated. This was the last battle between the clans Sutherland and Mackay.
Historian Sir Robert Gordon (1580-1656) (a younger son of Alexander Gordon, 12th Earl of Sutherland) wrote of the siege of Borve Castle in his book "A Genealogical History of the Earldom of Sutherland":
The Queen granted commission to John Gordon, Earl of Sutherland, against Iye MacKay and his country. The Earl of Sutherland, assisted by Huntly, and having the regent's commission, raised an overwhelming force, and for the first time that ever any Earl of Sutherland adventured to do so, invaded MacKay's country. So Earl John assembling all his forces, entered into Strathnaver, sack-ing and spoiling before him in a hostile manner, and possessed all the places of doubt, lest, as Hollinshed sayeth, any hole might be left for them to pass away from thence. But when Earl John perceived that Iye MacKay would not abide and fight him, he besieged the strong castle and fort called Borwe, the principal strength of that country, not two miles distant from Far. After a short siege, he took it by force, and hanged Rory MacKean-Voir, captain thereof.
Written from a manuscript wrote in the reign of James VI of Scotland