"We're not saying you had an affair or anything..."