Little evidence remains of the house or the village of Chillington as described in the Domesday book, but it is probable that the Norman dwelling – most likely in the form of a fortified manor – was not the first on the site.
The medieval manor was replaced in Tudor times by a building of some significance as testified by the fact that the Great Hall of that house is now encased in the shell of John Soane’s Saloon in the present house.
However prior to Soane’s arrival, changing taste and the fact that the building suffered damage by parliamentary forces during the civil war necessitated additions to the Tudor building. In 1723 a contract was drawn up between Peter Giffard and William Smith, brother of Francis Smith of Warwick. William died the following year, so it is more than likely that his brother undertook the building of the south side of the house as seen today.
Peter’s son Thomas undertook the landscaping of the park in the 1760’s. He then commissioned Robert Adam to draw up designs for an entirely new house for the site but though the drawings exist still in the archives of the Sir John Soane’s Museum in London, they were never executed, probably due to Thomas’ death and the fact that his son, also named Thomas was a minor at his father’s death.