"There's a strong feeling in the party we mustn't allow ourselves to be bullied"