The designed landscape of Torosay dates largely from the period between 1830-65 when the present Castle was built. The farm square, slipway and boatman's cottage and some remaining trees date from the previous century. The gardens date from 1897-1906 and have been further developed since c.1950. Documentary evidence is provided by an estate map of 1829, the 1st edition OS map of c.1875, a plan of the policies for the sale catalogue of 1897, and the 2nd edition OS of c.1910.
The earliest records of the estate indicate that the parish of Torosay was part of the livelihood of the Abbots of Iona. Torosay, or 'Torr rasach', means 'Hill covered in shrubs'. It became the territory of the Macleans of Duart but the lands were forfeited to the Dukes of Argyll in 1688 and Duart Castle followed during the '45. A house, kitchen garden and farm square were built on the present site in the latter years of the 18th century before being sold c.1820 to the Macquaries of Ulva who sold in 1822 to Colonel Campbell of Possil, Glasgow.
In the mid-19th century the Georgian House was demolished and a new Castle in the Scots Baronial style was built. The estate then called Achnacroish was purchased in 1865 by Arbuthnot Charles Guthrie, younger son of David Charles Guthrie of Craigie and co-founder of Chalmers Guthrie, the London Merchant Bank. He and his wife, Anne, spent some months here every year for the next 32 years and, during that time, changed the name to Duart House. On the death of her husband in 1897, Anne moved to London, taking the furniture and contents of the Castle interior with her. Thus, Walter Murray Guthrie, her nephew and partner of Chalmers Guthrie, inherited an empty Castle. He immediately put it up for sale but withdrew it after a visit to the island with his wife, Olive. They realised the potential of the site and commissioned Robert Lorimer to lay out the gardens to form a link between the Victorian Castle and the Georgian walled garden. Over the next ten years, there were many important visitors to the Castle including Sir Winston Churchill, who stayed here often and indeed shot his first stag on the estate.
In 1906, the health and business interests of Walter Murray Guthrie began to falter. In 1911 he sold the ruins of Duart Castle and the Point to Sir Fitzroy Maclean, 10th Baronet, who began its restoration. One week after the sale, he died. Olive Guthrie then changed the name of the Castle back to Torosay and fought to keep up the estate but limited financial resources and the onset of two World Wars took their toll. Some 8,000 acres of land were sold to the Forestry Commission and the gardens became run down. Before World War II, she transferred the estate to her daughter, Bridget, and, on her death in 1945, left the Castle and eleven acres of gardens to her grandson, David Guthrie James, the son of Bridget by her first marriage to Sir Archibald James, KBE MC MP.
After World War II the Castle was run as a hotel for two years before David Guthrie James and his wife, the Hon Jaquetta James, daughter of the 10th Baron Digby, DSO MC, assumed residency of the Castle during breaks from his political career in London. Meanwhile Mr Guthrie-James' mother and his stepfather, Lt Colonel A.G. Miller DSO, converted the kitchen wing of the Castle for their use and assumed full- time residency enabling them to restore the gardens which had, by then, been neglected for some 35 years. The estate was transferred to a Trust for the four sons of Mr David Guthrie-James in 1964. Mr David Guthrie- James and his wife assumed permanent residency at Torosay in 1983 and devoted themselves to the development of the Castle and gardens. Torosay is said to be the only castle in the West of Scotland maintained as a private residence but open daily to the public (May- October).