"Because we would tell him we were trying all the time"