When the Normans built Colchester Castle in the 11th century, they used stones and tiles taken from the ruined Roman town which had been abandoned over 600 years before. By the time the Member of Parliament for Colchester Charles Gray become the owner of it in the 18th century, it was in a sorry state. It was dilapidated, the roof was long gone and the top of the castle had been demolished by a previous owner to sell off the stone for reuse. Gray had been given a large collection of books which had belonged to Samuel Harsnett who had been headmaster of Colchester Royal Grammar School in the late 16th century and went on to become Archbishop of York in the early 17th century.
Gray decided to build a library in the front part of the castle. He put large windows in the front to let in light to read by and added a roof to it. As 95% of the Castle is made of recycled Roman material, Gray made the roof in an Italian style which was fashionable at the time. He also put an Italian style cupola on top of the grand staircase.