Having been to Portmeirion 4 or 5 years ago, and only finished watching the Prisoner this year, I feel I can recall a lot.
Basically, what they did to Portmeirion was put up quite a few easily removed things like signposts, and frilly window shades and numbers outside houses. Everything else is sets at a studio. For example, number 2's. They were really careful about the camera angles, so everything you see up until he is inside, is actually real. The copper domed round building is only about 30 foot across, if that, and functions as an art gallery in real life.
Number 6's house is also smaller than the one in the series. The same goes for the town hall, and the other buildings, and the hospital etc. It really is amazingly well done- if I hadn't been to Portmeirion, I would have had trouble spotting how they had done it all. And rover- even today would be very effective as a proper barrier to escape, and would spook some people.
I don't quite get what is so good about Ian McKellan. But remember 2 or 3 of the number 2's appeared twice, so you could bring him back for the ending. There were after all several times number 2 nearly won, and one of those involved the number 2 who was in the last 2 episodes.
Ahhh, how to talk about it without giving anything away to people who havn't seen it.
Anyway, it might do the place good if they re-used it. It needs a bit of money for the upkeep, its getting a bit run-down, and IIRC some of the plantings could do with being replaced.
Some of the humour in the series is spot on. When no.6 is being "interviewed" by one middle aged well dressed stooge, someone watching comments (roughly)
"Yes, he's ex-civil service, fitted straight in."