"He never really loved hanging out with us."