The present mansion was begun by the sixth Earl of Shrewsbury and as already stated he was the fourth husband of Bess of Hardwick, herself a great builder. It was considerably enlarged and to some extent rebuilt in the second half of the seventeenth century by the fourth baronet Sir George Savile, later created Marquis of Halifax. When writing to his brother in 1680 he said " I have still left some decayed front of old building .... I have at least so much reverence for it now as I had when it was encumbered with those sanctified ruins." The firm in charge of the sale of the estate stated that the house had been built mainly on the foundations of the former abbey, and that the crypt and other portions of it and some of the abbey walls on the south and west fronts had been incorporated in the new mansion. Certain detached buildings which always formed part of all monastic layouts have disappeared. The southern portion of the then building has been altered only in a minor degree and is a fair example of Elizabethan architecture.

Only one or two members of the Savile family have earned a place in history. The first of these was Sir George Savile, the fourth baronet, created Marquis of Halifax in the late 17th century, who took an active part in politics and diplomacy in that century although nicknamed "The Trimmer" for obvious reasons. The Lord Savile who died in 1898 was British ambassador to Italy.

Various members of the royal families of the day have stayed at Rufford, apart from Edward I who spent a night there, as already mentioned, but that was largely due to the fact that there was not much choice of resting places in those early days. James I, Charles I, George IV stayed there while Prince of Wales, and Edward VII stayed several times at Rufford before and after his accession to the throne.

Like many other old houses Rufford had its ghost. In 1897 Lord Savile wrote that he personally had not seen it but added that several persons then living "and of undoubted veracity" said they had seen the "little old lady in black." He also stated that "more recently great alarm was occasioned to one of the guests by a visit from a gigantic monk with a death's head under his cowl." His Lordship expressed no opinion about this alleged apparition. It is said that in the early registers of Edwinstowe church an entry records the burial of a man who "died from fright after seeing the Rufford ghost," presumably the Abbey ghost. Another churchyard record is of a different type. In 1898 in Ollerton churchyard there was a tombstone in memory of Francis Thompson butler at Rufford Abbey. The inscription began —

Beneath the droppings of this spout
Here lies the body once so stout
of Francis Thompson.....

The Nottinghamshire County Council has given much careful thought to the problem of preserving the abbey or parts of it and has been in consultation with the appropriate government departments, and with certain societies and associations interested in its preservation. The result has been that enough land has been purchased to give access and to enable demolition to proceed and repairs to be effected. Some land has also been purchased for use as an open space. Some articles of interest, some tapestry, the great gates, outbuildings and the lake have also been purchased.

Rufford Abbey, artist's impression of controlled demolition.Rufford Abbey, artist's impression of controlled demolition.

At their meeting in November, 1954, the Council regretfully decided that there is no alternative but to proceed with the controlled demolition of the buildings for which purpose £10,000 was included in that year's estimates. It would be uneconomical to preserve the building owing to the heavy cost of repairs and the risk of mining subsidence. The County Architect was therefore instructed to obtain tenders for demolishing the abbey down to ground floor level of the most northerly and most modern section and also the majority of the centre section. The demolition of the southern section is a matter for the Ministry of Works. The crypt will be preserved.

It is interesting to recall that in 1797 when Throsby published his edition of Dr. Robert Thoroton's great history of Nottinghamshire he wrote: "Rufford as a dwelling has neither enough of religious antiquity remaining to attract or splendour as a great seat to arrest the stranger's travel.