"We knew if any of the bigots in town found out about us, there'd be a terrible scandal."