"They'd come down and visit us occasionally when Rory was a baby..."