"and it all came clear when I realized how much we could hurt Rory."