"I'm afraid Downton will be a place of pilgrimage for a while."