"But you can't leave Downton."