"They're not abandoning the party."