The sandstone castle overlooks the town from a tor where Saxon fortifications once stood. It was built by the Norman baron, William de Mohun but, by 1374, the Mohun line was dying out and the castle was sold to Lady Elizabeth Luttrell. When George Fownes Luttrell inherited the property in 1867, he commissioned the architect, Anthony Salvin, to transform the castle and give it the appearance of a fortified Jacobean manor house.