"We can't have the President in a tug-of-war against the Almoner."