"People don't leave the White House to become a governor."