Rothiemay or "Rath-may" means the fortress in the meadow, the name suggesting a defensive site - at least from Norman times. No record of these early fortifications exists except for a first mention of Rothiemay in history from 1264, during the reign of Alexander III, when the lands were taken over by the crown.
Part of the castle remembered today, probably dated from the 15th century. This oldest part was L-shaped with walls up to 2.4m. thick, stone vaulted at first floor level, accomodating a baronial "Solar" and "Great Hall" over kitchen and servants rooms. Extensive rebuilding work in the scottish classical style was carried out in 1788 by the 2nd Earl of Fife to make a modern mansion house, and in 1901 Col. Forbes added the tall porch tower in the scottish baronial style. The castle was completely demolished in 1964.
The doocot dates from c.1753 and is 8.3m. square divided into two chambers. The original roof was in a steeply sloping swept pavillion style covered with slates and culminating in a central louvre. It appears in Williams Tomkins 1767 painting "View of Rothiemay House".