"I do believe those first two years were the worst for him."