The first mention of the barons of Taufers dates back to 1136. At the latest in the 1920s, they built a new and large stately home.

Mastio and dongione (habitable tower) were erected towards the mountainous coast, the only part from which an attack could have come. The palace (later used as a barn) was erected on the western side, in an area protected by the nature of the land itself.

The building area, already delimited in itself by the conformation of the rocky spur on which the castle stands, was further defined with the construction of a wall, along which several residential buildings were subsequently added, especially at the beginning of the sixteenth century.