Edzell was originally the headquarters of the Stirlings or Strivelyns of Glenesk whose seat was in the Motte Castle sited to the south-west of the present Castle, near the Churchyard. The Crawford Lindsays acquired the estate in 1358. The Tower House which forms the earliest part of the present castle, was probably built towards the end of the 15th century by Sir David Lindsay. Another David Lindsay, late 9th Earl of Crawford, who inherited in 1529, added the gatehouse range, possibly in 1553. His son, yet another David, added further buildings within the courtyard, and it was he who formed the garden.
He was a greatly accomplished scholar and his knowledge and interests probably led to the unique relief carvings on the walls of The Pleasance enclosure which are dated 1604. The importance of Edzell lies in the presence of this garden next to the Castle, demonstrating an age when the desire for comfort and more settled political conditions allowed people to create recreational areas for their own enjoyment. Ochterlony's account of Forfarshire in 1682 described the gardens as 'far exceeding any new work of their times'. He also noted that, 'in the excellent outer court, large and level, they used to play football and there are four great growing trees that are the debts' (goals).
David, Lord Edzell, died in 1610. His son was involved in the murder of a kinsman, Lord Spynie, and the estate was badly affected by the troubles which followed. The last laird of Edzell sold the estate in 1715 to the 4th Earl of Panmure in order to raise funds for the 1715 uprising. The new owner, however, was also involved in the Rebellion, and his lands were confiscated by the Crown and subsequently belonged to the York Building Company. The Castle was occupied by the Argyll Highlanders in 1746, during the 2nd Rebellion. The York Building Company was declared bankrupt in 1764, and the floors and roof were stripped from the Castle at Edzell in order to pay creditors. At this time too, the Beech Avenue which linked the Castle with the parish church was cut down.
The Edzell Estate was repurchased by William Maule on whose death, in 1782, it passed to the 8th Earl of Dalhousie. In 1932, The Pleasance was placed in state care by the 14th Earl of Dalhousie and, six years later, he also placed in care the remainder of the ruins of the Castle. By that time, the original layout of the garden had gone. The new owner, now known as the Historic Buildings & Monuments Directorate of the Scottish Development Department, has consolidated the stonework and reformed the garden.