The St.Johns originally came from a village in Normandy called St.Jean-le-Thomas and by the thirteenth century had established themselves as influential landowners in both Wales and England. The family came to prominence when Oliver St.John married Margaret Beauchamp, mother of Margaret Beaufort, and Lydiard was conveyed to the St.John family, where it remained for the next five hundred years. Margaret Beaufort, mother of Henry VII, remained close to her St.John half brothers and sisters, and the family benefitted from the royal connection. As the seat of the junior branch of the family, Lydiard grew in importance as the family became involved in county and national politics, and the family made several astute marriages during the Tudor period to continue their upward mobility.
In 1611 Sir John St.John purchased the title of baronet, further increasing the family's stature. He sat in parliament, became sheriff of the county, and was greatly concerned with perpetuating the family's pedigree, erecting the important monuments in St. Mary's church, including the exqusite polyptych. Sir John was a staunch Royalist and lost three sons in the British Civil War in the king's service.
Sir John's sixth son, Walter, a man of Puritan sympathies, married his distant cousin Johanna St.John, a relative of Oliver Cromwell, and inherited the estate in 1648. The family moved their base to the town house in Battersea, while Lydiard became a holiday home and the source of country produce. Walter also sat in several parliaments, and although found to be "backward in kissing the king's hand" at the Restoration, fared well through the rest of the century.
Lydiard was even less frequented by Sir Walter's grandson, Henry, who rose to fame as a Tory statesman and political philosopher. In 1712 Henry St.John was created Viscount Bolingbroke and as Minister of State to Queen Anne, negotiated the peace treaty with France, creating many personal enemies. On the accession of George I he was forced to flee to France, where he lived in exile until receiving a Royal Pardon in 1723.
The successive three Viscounts had little money to spend on the estate, although John St.John, the second viscount, married an heiress and remodelled the house in about 1743. The 3rd Viscount Bolingbroke lived mostly abroad, and Lydiard became increasingly neglected and was mostly let to tenants. This continued throughout the nineteenth century and on the death of the 5th Viscount, Henry Mildmay in 1899, the estate his young son inherited was heavily mortgaged. The estate was broken up at sales in 1920 and 1930. In 1943 Swindon Corporation bought Lydiard Park, shortly after the death of Mary Emily, Viscountess widow of the 5th Viscount.