"But he pulled back because of Mary and me, we were just kids."