"I've had three members of the Black Caucus tell me"