"You can't just take him. He's our grandson."