"And it absolutely drove him crazy."