It is probable there was a house on the site from the Middle Ages, perhaps a hunting lodge, though the earliest evidence comes from a plan of 1700, which shows a traditional manor house with extensive service accommodation. In the mid-18th century, John 2nd Visct St.John gave the house a fashionable makeover, remodelling it in the classically inspired Palladian style with a grand new southwest façade fronting a new suite of state rooms.
Dwindling family fortunes meant the house became increasingly neglected in the nineteenth century. The 4th Visct tried to supply the house with some of the amenities it lacked and built a service wing in 1830, but no improvements and probably scarcely any maintenance was undertaken thereafter.
On the death of Lady Bolingbroke in 1940, the estate was mortgaged up to the hilt with no option but to sell up. In a farsighted move, Swindon Corporation purchased the property and set about its restoration, opening it to the public in 1955. Today, Lydiard House is an Accredited Museum with much of the original contents returned.