"It's one thing to break ranks and vote against my candidate..."