"I had never intended to have an affair with Miss Lytton,"