"We can't tell our people they can vote yes"