Castles were traditionally integrated with the walls. It would be a very creative and beneficial perk if, perhaps through a library skill, the castle could connect with the walls. This would increase the amount of wall pieces available and would make the castle much more historically accurate.