"They don't see me as their legitimate Speaker."