Party full text search

[regetz]

What

This PR is another clone of PR 165 (which was done for plant concepts), now introducing full text search for parties via the collection GET endpoint.

This PR also sneaks in the addition of an obs_count column to returned Party records.

Why

So that API clients can request only parties whose last name, first name (with lower weight), and/or organization name match a desired search term.

How

• Added a Flyway migration that does the following:
  • Adds a new column in the party table to store a "document" (tsvector type) containing the text substrate for search
  • Creates an index on this new column
  • Defines a function for building these documents using the surname, givenname and organizationname columns
  • Defines a function for inserting the documents into the new column
  • Defines a trigger and corresponding function for updating that column whenever changes are made to any of the source columns
  • Applies the insert function to initially populate the column for all existing parties
• Updated the Party operator to include the relevant SQL snippets, namely in:
  • the WHERE clause, filtering for matching search
  • the SELECT clause, adding a new search_rank field that indicates the relative match strength
• Updated the query parameter validation method to pass along the search parameter

More details

Repeating the same details from the earlier Plant Concept FTS PR description:

Under the hood, this uses the websearch_to_tsquery() Postgresql function for converting the client's search term into the correct FTS query. As described in the Postgres documentation, this function uses search syntax "similar to the one used by web search engines", which in simple form searches on individual terms, but can also be used to denote phrases ("like this"), exclusions (-this), and "or" conditions ("this" or "that"). See here for more detail.

Note that we have not implemented any sort of fuzzy or semantic search. If you misspell "seqoia", you won't find what you're looking for.

Demo

Search for a word (here just returning the count of matching records)

$ http GET 'http://127.0.0.1/parties?search=mary&count'

{
    "count": 19
}

Search for an organization

$ http GET 'http://127.0.0.1/parties?search=nceas'  \
    | jq '.data[] | {given_name, surname, organization_name}'

{
  "given_name": "Mark",
  "surname": "Anderson",
  "organization_name": "NCEAS"
}

Show that matches on last name (surname) score higher than on first name (given_name)

$ http GET 'http://127.0.0.1/parties?search=marshall' \
    | jq '.data[] | "\(.given_name) \(.surname)  \(.search_rank)"'
```json
"Michael Marshall  0.6079271"
"Everett Marshall  0.6079271"
"Marshall Ellis    0.24317084"

[DarianGill]

Looks good to me! Thanks for adding the obs_count in as well

[RWShelton]

I like being able to search by org name as well as the person, that's a great feature. Good to go.