Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update synonym list from CourtListener #14

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mlissner
Copy link

@mlissner mlissner commented Sep 14, 2016

This is a big addition to our synonym list. The source for this data is mostly from the Indigo Book, which has many tables of abbreviations, such as:

  • U.S. States and Other Jurisdictions
  • Services & Publishers
  • Legislative Documents
  • Treaties
  • Arbitral Reporters
  • Intergovernmental Organizations
  • Court names
  • Titles of judges and other people
  • Case name abbreviations
  • Geographical terms
  • Document subdivistions (pargaph, section, etc)
  • Explanatory phrases
  • Institutions
  • Publishing terms
  • Month names
  • Common words in periodical names

On top of that, I added a few things:

  • Numbers from 1-20
  • Units
  • And any extra things I saw missing (like trans didn't have a mapping to transgender)

From there I did the following:

  • Remove duplicates (of which there were many)

  • Removed phrases (Solr is bad at this)

  • Removed anything with a period or apostrophe in the middle, like, U.S., because those things get split by Solr anyway.

  • Cleaned up a bunch of items that had brackets, like trans[lator, lation], trans. In the case that the expanded word lists were semantically different, I made them into mappings. Else, I made them into synonyms. For example, here's what trans maps to:

    trans => translation,translator,transgender
    

    This means that when you type trans the system doesn't know if you mean translator or transgender. OTOH, something else might just be:

    assemb,assembly,assemblyman,assemblywoman,assemblymember
    

    Because they're all essentially the same semantically.

  • Eliminated one-letter abbreviations (they're not likely to be useful)

  • Removed real words that'd cause trouble. For example, we wouldn't want every search for cat to turn up results for category.

All in all, I think she's a fine list. Definitely a conservative one, which lawyers like, but also one that should make lots of searches (especially those involving abbreviations) work better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant