"Let's not throw out the baby with the bath water."