"I know Lady Mary didn't like being made a fool of."