"As impossible as that may sound, we have to let him go."