"But if this is going to work between us, you can't do that."