"He'd only punch me back."