"If we don't leave now, this could all come tumbling down"