"I'd always ask him to come, but he'd never want to."