"Or worse, that he's abandoning the administration."