Three families have been responsible for the development of Dirleton since the first castle was built in the 12th century: the de Vaux, the Halyburtons and the Ruthvens.
In the early 12th century, the Barony of Dirleton, which at that time extended as far as the Firth of Forth, was acquired by William de Vaux. The castle which he constructed on the site is thought to have been built of timber. The first stone castle on the site is thought to have been built in the 13th century with stone from a quarry in the nearby village of Gullane. In 1298, the castle was attacked by troops of Edward I and occupied by them until 1311. During this period, some repairs and alterations are thought to have been carried out. When Scottish forces loyal to Robert the Bruce reclaimed the castle, some of the work carried out by the English was demolished. Dirleton was returned to the de Vaux family but, some time during the reign of David II (1329-71), the daughter and heiress of William de Vaux married John Halyburton, 2nd son of Sir Adam Halyburton of Halyburton. The estates then passed to the Halyburton family.
In 1363, the castle was seized by the Earl of Douglas during a revolt against David II. The Halyburtons regained control and considerable reconstruction work was carried out at the end of the 14th century and throughout the 15th century. The upper part of the earlier 13th century towers were formed whilst the curtain walls which linked them are thought to have been remodelled. A new gateway was formed in the south- east corner to replace an earlier drawbridge. James IV visited Dirleton in 1505 and is recorded as having given money to workmen employed on the construction of the rectangular enclosure at the north- east side of the castle.
The great grandson of John Halyburton who had married the de Vaux heiress was Lord High Treasurer and a hostage for the Ransom of James I (1406-1437). His son, John, was created Lord Halyburton of Dirleton. The 5th Lord Halyburton died in 1505. His daughter, Janet, married the 2nd Lord Ruthven, Provost of Perth, Extraordinary Lord of Session and Keeper of the Privy Seal.
Their son, Patrick, 3rd Lord Ruthven, was involved in the murder of Rizio at Holyrood in 1566. Patrick's son, William, who was also involved in the affair, became 4th Lord Ruthven on his father's death the same year. He was created Earl of Gowrie in 1581. Lord Gowrie was a keen arboriculturist and is thought to have planted many trees at Dirleton to ornament the gardens which he may well have laid out. He was responsible for the 'Raid of Ruthven' when James VI was held at Lord Gowrie's home, Ruthven Castle, Perthshire. Lord Gowrie and his associates had assumed power for less than a year when the King regained control. Lord Gowrie was eventually beheaded in 1585 following involvement in a plot to seize Stirling Castle.
Lady Dorothea Gowrie, his widow, gave up Dirleton and the King granted the lands to his adviser, the Earl of Arran. They were, however, returned to Lady Gowrie within the year. Misfortune returned to the family when two of her sons were hung for their part in the 'Gowrie Conspiracy'. Thereafter, Parliament ordered the family name to be wiped out and decreed that no successor should hold 'any office, honour or possessions'. Lady Dorothea's other two sons escaped to England, and Dirleton passed to Sir Thomas Erskine of Gogar. The Ruthvens had been responsible for considerable alterations to the interior of the castle and, despite the Parliamentary Order, Lady Gowrie was allowed to remain in residence at Dirleton.
In 1625 Sir Thomas Erskine was created Lord Dirleton, Viscount Fenton and Earl of Kellie. His son sold the lands of Dirleton to Sir James Douglas. He, in turn, sold to Alexander Morieson of Prestongrange who, in 1631, sold to James Maxwell, proprietor of the Innerwick estate, near Dunbar.
In 1646 Maxwell was created Earl of Dirleton and Lord Fenton of Eilbotle. In 1650 the castle was attacked by Cromwell's forces and used as a hospital for a short time. The castle was returned to the Countess of Dirleton, whose son sold it to Sir John Nisbet in 1663.
Sir John was responsible for the construction of a new house on the neighbouring estate of Archerfield. He died in 1687. Dirleton and the other Lothian estates were inherited by his cousin, William Nisbet, whose daughter Mary married the 7th Earl of Elgin. The estates passed through the female line, and their daughter, Lady Mary Nisbet Hamilton, married Robert Dundas, 6th son of Lord President Dundas of Arniston in 1828. By this time, Dirleton had become the flower garden of the designed landscape of Archerfield which was laid out around 1780 by Robert Robinson. An account of 1885 noted that the gardens were 'beautifully kept' and had many fine specimen trees. Even at that time, Dirleton was open to the public for one day per year. Their daughter, Mary, married Henry Nisbet Hamilton Ogilvy of Biel, Archerfield, Dirleton, Winton and Weilvale in 1888. After her death, Archerfield was sold but Dirleton remained in the Nisbet family ownership. It has been held in the care of the Secretary of State for Scotland for some 60 years and it is now owned by the National Trust for Scotland.
During this time, the herbaceous borders have been remodelled and other ornamental planting has been replaced but the Bowling Green and old trees remain as remnants of the earlier layout.