In 1537, Mansel first leased, and then in 1540 purchased, the abbey buildings and some of its former granges. In three further transactions he secured most of the abbey’s former possessions. Margam now replaced Oxwich as the Mansels’ main residence. The house based on the domestic ranges of the abbey was added to in a variety of architectural styles by succeeding generations to form a rambling mansion with formal gardens, orchards and an extensive deer park.
Continuity of worship within the abbey church appears to have been broken for a short interval but in 1542, the church had been converted for parochial use and served by a priest. The Mansels treated the Abbey Church with scant respect. As the building was far too large to serve the limited population of Margam, over the next two centuries the eastern sections were largely allowed to deteriorate, with the church used at one time as a coach house. Meanwhile, the chapter house was used for storing coal and its vestibule for brewing beer. The present church is formed by the remaining six western-most bays of the monastic nave.
The Mansel line became extinct with the death of Bussey, fourth Lord Mansel in 1750, the vast Margam and Penrice estates passing though the female line to the Revd. Thomas Talbot of Lacock Abbey. In 1768, Thomas Mansel Talbot inherited his father’s estates but disliking Margam, decided to abandon and demolish the decaying historic home of his Mansel ancestors. He preferred Penrice where he built a villa close to the old family seat at Oxwich. At Margam he developed the pleasure gardens, and built a magnificent Orangery to house a famous collection of orange trees with statues and other antiquities collected on the Grand Tour. The church however, continued to be neglected.