Elastic Search Integration

[PascalSenn]

Elastic Search Integration

We are looking into the development of an elastic search integration for HotChocolate.

The first step is to define what the integration should look like and in what direction we want to go.

Elastic Search has a broad application range, we should focus on one thing and plan this out.

Common Elastic Search use cases

1. Elastic Search as a Search Index
   Probably the most common use case is to use ElasticSearch solely for indexing data for search and use a database as the primary storage of the data.

2. Elastic Search aș a Database
   Elastic APM uses elastic search as it’s primary storage. This seems to be a more specialised use case.

We should focus in a first iteration to make the search index use case work.

GraphQL API

We have to define an interface for query input and query output. This should be the first step.

Input

We can integrate elastic search into the filtering API, and are theoretically not bound to the current format. The search API is super powerful, the question is how much of this do we want to expose to the user.

A simple HotChocolate Data query looks like this:

{
   users(where: {name: { eq: “Sam” }} ) {
      nodes  {  name }
   }
}

Directly translated to elastic search this would be something like

{
  "query": {
        "bool": {
          "must": [
           {
             "match": {
               “name": “Sam”
             }
           }
        ]
      }
    }
  }

But this is probably not what elastic search users would like to use. There many ways to query elastic search that are not available in conventional databases.
As an example have a look at this query:

{
  "query": {
    "match": {
      "phrase": {
        "query" : "heuristic roots help",
        "minimum_should_match": 3
      }
    }
  }
}

How do we expose this in a GraphQL API.
The simplest approach would be to just open up the query API:

{
   users(where: {
       match: {
            phrase: {
                query: “heuristic roots help”, 
                minimumShouldMatch: 3 
            }
        }
   } ) {
       nodes { 
            name
       }
   }
}

But this exposes a big attack surface for e.g. denial of service attacks.
It’s is also not nice, as the front-end developer would have to make the decision about what the best way to query the elastic search is.
It would be much nicer if the possible searches were defined in the backend and expose a saner query to the front-end.

This is a useful resource to find sample queries for elastic: 
42 Example Queries

Output

Like the input we must define how we want to expose the filtered data.
The easiest representation would be just as the resolved objects from the database.
But it could also be interesting to expose additional metadata.
An elastic search response looks like this:

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 0.6785374,
    "hits" : [
      {
        "_index" : "employees",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 0.6785374,
        "_source" : {…}
      },
      {
        "_index" : "employees",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 0.6257787,
        "_source" : {…}
      }
    ]
  }
}

Most of the properties are too low level to expose. But it could be interesting to expose parts of “hits” on a connection and maybe even the score on the edges:

{
   users(where: {name: { eq: “Sam” }} ) {

      hits {
         total
      }
      edges  { 
	 node { 
              name 
         }
         cursor
         score
      }
   }
}

Inspirations

• https://github.com/searchkit/searchkit
• https://github.com/graphql-compose/graphql-compose-elasticsearch (seems way to low level)

Open Questions / Next Steps

• Are there any implementations of a GraphQL interface for elastic search?
• How do people use elastic search at the moment (in combination with HC)?
• What information from the elastic search result has to be exposed to the UI?

Future Steps

• Define what C# library we want to use NEST or the low level client Elasticsearch.Net
• Define how the C# interface should look like. How do we define the filters in the annotation based, code first and schema first way.
• Implement a provider for data
• Think about sorting
• Champagne 🥂🍾

[frzmohammadali]

For common use cases

• I also believe we better leave "Elasticsearch as a Database" use case for future iteration. Having the ability to search on an index seems way more appreciated at first.
• Sharing my own experience, I worked on a real-time web app which abstracts every communication of visitors of a website in the concept of "conversations". Then we always persist the state of these conversations in a Cassandra database while keeping them indexed at Elasticsearch. Therefore, the main motivation behind using ES is to be able to "filter them" as well as "search in them" at the same time. So any sort of mutations was generally not interesting in that case since the data got indexed by automated backend services.

For the Input interface

• As far as my experience with Elasticsearch goes, HotChocolate Data queries are often used in combination with more complex and specific Elasticsearch queries (like the aggregate queries from ES). Therefore I believe we first need to provide a working HotChocolate Data query with Elasticsearch, and then think of exposing the Search API or any other approach afterwards maybe..

For the output

• Everything returned under "hits" is often found useful for the frontend in my case and the rest are considered too low level and only interesting for being logged maybe.
• However, one important thing is sorting and paging of the data, which I personally found it tricky to handle. Elasticsearch supports cursor-based sorting same as HotChocolate, but does generate cursors in a complete different way I believe.

For the C# library

• NEST is generally nice, but for this case since we are building an integration at framework level, I believe Elasticsearch.Net could be more reliable, as it gives us more control on the input/output of every interaction with Elasticsearch.

[PascalSenn]

As far as my experience with Elasticsearch goes, HotChocolate Data queries are often used in combination with more complex and specific Elasticsearch queries (like the aggregate queries from ES). Therefore I believe we first need to provide a working HotChocolate Data query with Elasticsearch, and then think of exposing the Search API or any other approach afterwards maybe..

Can you share some examples of queries you would like to write or currently use?

[daniefer]

I know the product team at my company will likely want something like highlighting and aggregates per field (total income, total expenses, average time in stage, etc) as soon as they know it is possible. 😆

[frzmohammadali]

@PascalSenn
We usually have a complex dynamic filtering builder in the client side so the elastic queries that it generates might not be the most optimized but here is how our tool currently generates a query for searching by wildcard in 4 field of an index:

{
  "size": 20,
  "sort": [
    {
      "status": {
        "order": "ASC",
        "unmapped_type": "long"
      }
    },
    {
      "id": "ASC"
    }
  ],
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "should": [
              {
                "wildcard": {
                  "contactDisplayName": "*foo*"
                }
              },
              {
                "wildcard": {
                  "teamName": "*foo*"
                }
              },
              {
                "wildcard": {
                  "userDisplayName": "*foo*"
                }
              },
              {
                "wildcard": {
                  "note": "*foo*"
                }
              }
            ]
          }
        }
      ]
    }
  }
}

We also used the data that is indexed in elastic as look up data, which means we only fetch the full list of indexed data sorted in a specified order, and then paginate through them. So a sample elastic query can look as simple as:

{
  "size": 5,
  "search_after": "<some cursor>",
  "sort": [
    {
      "status": {
        "order": "ASC"
      }
    }
  ]
}

[daniefer]

My thoughts:

Use cases

I thought I would start with our anticipated use cases. One of the reasons we want to integrate with elasticsearch (ES) is because we want to combine search results from ES with authoritative data from Sql Server. A concrete example would be "Find records that contain the string "app" on any one of 3 fields that are also assigned to the current user." To further complicate matters, we need to be able to "disabled" certain operators or fallback to Sql Server in the event our near-real-time search index becomes not-so-near-real-time. We would like to be able to hide the complexities of switching back and forth between data sources without the consumer knowing any difference.

I am still new to the internals, so forgive me if I am way off in the following:

Input

Having a default operators set such as eq, contains, in, etc. would be a good start. I agree that passing through the query language is not an approach that would scale to other providers and expose a huge testing surface area. Given the large surface area ES exposes, it makes since to me to allow customizing or extending the default operators. NEST would be the ideal interface to expose for this customization.

Output

Enriching of results with ES metadata is exactly what we would be looking for. Match document count, aggregates, score, highlighting, etc. Being able to define what these are up front in our schema is valuable in our case because we don't want to tie each of these metadata items to ES.

{
   users(where: {name: { eq: “Sam” }} ) {

      hits {
         total
         aggregations
      }
      edges  { 
	 node { 
              name
              highlight { startIndex, endIndex }
         }
         cursor
         score
      }
   }
}

Other considerations:

1. OpenSearch is something we might want to consider in place of Elasticsearch

[PascalSenn]

Cool thanks for the response :)

Use cases

The dynamic switching is not something that does not work out of the box at the moment. We attach the field handler to the fields and elastic will have different handlers than for example SQL Server. What you could have is a second parameter. e.g.

{
   users(search: {...} , where: {...}) {...}
}

Input

I think we all agree on what the first version should look like. default data operations, then add search capabilities in a second iteration.

Does NEST and ElasticSearch.NET have interop between them? or are these two different worlds?

Output

Aggregations is still a topic that we have to address in the overall data story. I guess this is a problem we should solve another day :)

[PascalSenn]

@daniefer @frzmohammadali

I think we have a general idea on how the integration should look like from a API and conceptual perspective.

The following will be addressed in the first iteration:

1. Using ElasticSearch as a search index
2. In the first iteration we will add support for common data operations (what will be supported can be defined later)
3. Exposing "hits" and "score" on the connections for elastic search.

Future iterations will include (rough draft):

1. Support for the search API
2. Support for aggregations on connections

Input & Output

In the first iteration the query interface will look like the usual data interface (plus the "hits" and "score):

{
   users(where: {name: { eq: “Sam” }} ) {

      hits {
         total
      }
      edges  { 
	 node { 
              name
         }
         cursor
         score
      }
   }
}

This will be translated to an elastic search query like the following:

{
  "query": {
        "bool": {
          "must": [
           {
             "match": {
               “name": “Sam”
             }
           }
        ]
      }
    }
  }

Driver

What driver we use in the end is up for discussion. We will only really see just before or during the implementation what will fit better.
I personally would prefer the low lever driver but if the higher level one would benefit us more (e.g. automatic query optimization on the client) then we can also go with NEST. The question about the driver was more to see where we already have experience in.

C# API

The next step is to define how the C# interface should look like for types and resolvers.

Types

Data types are a different version of InputObjectTypes. A filter type can contain operations (like eq or contains) and fields. In the implementation for databases we can infer the possible fields from the output model. users(where: => FilterInputType<User>. We have to define how this will work for elastic search. The relevant questions are:

1. Do you have Poco's that define the structure of the data in elastic search or do you indexed based on string/definitions/..
2. Does you indexed data reflect the structure of the output type, or is the search definition different from the output types

Resolvers

Once we have define how we define the filter types, we have to define how we want to expose the filtering capabilities to the user. In the database implementations we usually return a IQueryable<T> or a IExecutable<T>. I think this does not really fit this use case though.

With elastic seach we have a three phase query:

1. Translate filter input to elastic search query
2. Query data from elastic search
3. Query data from the database based on the results of elastic search.

In c# code this could look something like this.

public class Query 
{

     [UseFiltering<UserFilterInputType>]
     [UsePagination]
     public Task<User[]> GetUsersAsync(
        [Service] IElasticSearchClient es, // NEST or ElasticSearch.NET or both 
        [Service] UserService service, 
        IResolverContext context,
        CancellationToken ct) // we need this to access the inputs, we could also have something more explicit like SearchContext 
       {
            // 1 Phase is already execute by the middleware pipeline
            
           // 2 Phase
           var searchResult = await es.SearchAsync(context, ct);

           // 3 Phase
           return service.Search(searchResult) // you need to provide some examples on how you do this currently
        
       } 
}

The open questions where are:

1. What interface do you currently use to query elastic search
2. How do you query the database based on the search result

How do you query now. C# code examples
How do we build query, same models?

[tunurgitr]

Background

Similar to @daniefer, we don't keep our "authoritative" data in elastic. It has data custom-modeled specifically for being searched effectively in Elasticsearch. With our shift to graphql, we were able to lean even more heavily towards this model. In our use-case, we're doing product search in an e-commerce context. We actually return just the ids for the products, the products are then added on and we use a dataloader to load them. Our search result classes look something like this:

public class ProductSearchResult {
    public decimal Score { get; set; }
    public ProductId ProductId { get; set; }
}

We do something similar with aggregate results, we just aggregate back the ids of the entities we want, then use dataloaders to get details if relevant. This could be integrated into the Edge type of a connection, but we'd want to load the entity itself using a dataloader.

The models we have in elastic also look nothing like our normal Product model, and we have extensive translation from user's search inputs. For example, if they want to filter by a manufacturer, we actually want to target a manufacturerIds.keyword field. When they want to search for text, we need to do a multimatch across several fields and subfields with different boosts. We use Nest and our own "handwritten" code to take our search inputs and create the elastic query. Although we've started out some of our elasticsearch indexes using pocos, we usually end up with strings as we tune things.

My hunch is that most real-world elasticsearch use cases are going to be on non-authoritative versions of data and will need extensive translation of a graphql input to a nest query. We usually create the SearchDescriptor<T> object and modify it before passing it to the SearchAsync method on the nest client.

Possibilities

Pagination

One thing I think would be directly relevant would be some extensions to apply pagination to the elastic request -- an API something like:

   var query = new SearchDescriptior<ElasticProduct>()
       // other stuff
       .ApplyOffsetPaginationFrom(resolverContext)

This sort of API would make pagination possible in more complex scenarios, and might help enable pagination, searching, and filtering separately. The type of API above is too all-encompassing to be practical for any of our use cases until a lot of other stuff was added.

Filters

Filters are probably a simpler use-case to wire up, but we'd need the ability to map a filter property to a specific field via a string. There are also cases where specific operations (like "contains") may need to map to different subfields for comparison...some kind of filter like:

    descriptor
        .BindFieldsExplicitly()
        .Filter(t => t.ManufacturerName)
        .AllowEquals().MapToElasticField("manufacturerName.keyword-lower").And()
        .AllowContains().MapToElasticField("manufacturerName.ngram").And()
        // .. more stuff

And then using it like:

   var query = new SearchDescriptior<ElasticProduct>()
       // other stuff
       .ApplyFilter(resolverContext)
       .ApplyOffsetPaginationFrom(resolverContext)

Sorting

Similar to filters, we need to extend a sorting input type to map to specific elastic subfields

Aggregations

I'm not sure what's going on with aggregations for other providers, but in this case we probably need some kind of class that defines how aggregations work. We do c#-side pagination of aggregation results.

Ideally we'd want to define something like this:

public class ProductAggregationType : AggregationType<Product>
{
    public override void Configure(IAggregationDescriptor<Product> descriptor)
    {
        descriptor.Aggregation("manufacturer")
            .Field("manufacturerId")
            .Type<ObjectType<Manufacturer>>()
            .ResolveUsingDataLoader<ManufacturerDataLoader>() // custom resolver also okay
            .UseOffsetPagination();
    }
}

And then use it like this:

   var query = new SearchDescriptior<ElasticProduct>()
       // other stuff
       .ApplyFilter(resolverContext)
       .ApplyOffsetPaginationFrom(resolverContext)
       .ApplyAggregations(resolverContext)

Searching

This is the most complex for us. We don't allow clients to specify all the nuances of the query -- we want to expose a "clean" query surface with most of the elastic-specific intricacies stripped away.

[frzmohammadali]

@PascalSenn

Do you have Poco's that define the structure of the data in elastic search or do you indexed based on string/definitions/..

I'm generally not fan of having dynamic data structure so I tend to define POCO for the structure of the data in elastic search as often as I can and avoid having everything as string. Using the high level NEST library is also easier with this approach I believe.

Does you indexed data reflect the structure of the output type, or is the search definition different from the output types

For me the answer to this question is yes. The shape of my indexed data IS exactly what I want to receive after search. However, if by "output type" you mean the payload of the GraphQL query, the answer could be no sometime as I might need to search elastic, find some ids, and then finally load them from database and return them to the frontend.

public class Query 
{

     [UseFiltering<UserFilterInputType>]
     [UsePagination]
     public Task<User[]> GetUsersAsync(
        [Service] IElasticSearchClient es, // NEST or ElasticSearch.NET or both 
        [Service] UserService service, 
        IResolverContext context,
        CancellationToken ct) // we need this to access the inputs, we could also have something more explicit like SearchContext 
       {
            // 1 Phase is already execute by the middleware pipeline
            
           // 2 Phase
           var searchResult = await es.SearchAsync(context, ct);

           // 3 Phase
           return service.Search(searchResult) // you need to provide some examples on how you do this currently
        
       } 
}

Just sharing my first impression, I really liked the style of this piece of code as the final GraphQL query if we can achieve this 😜

What interface do you currently use to query elastic

How do you query now. C# code examples

Honestly, I always try to inject NEST client as a service and use NEST or the low level Elasticsearch.Net client interchangeably which is possible as NEST comes with Elasticsearch.Net inside it. I usually evaluate then based on each particular logic that I need. In which case, I think it's obvious to mention that I choose the one that feels easier at the end of the day :)
Here is a "query_string" query that I built with NEST:

var response = await _elasticClient.SearchAsync<Contact>(x
                             => x
                                .Index(Indices.Index(indexName))
                                .From(0)
                                .Size(limit)
                                .Query(q => q
                                           .QueryString(descriptor =>
                                                            descriptor.Query($"*{stringToSearch}*")
                                               )
                                    )
                    );

How do you query the database based on the search result

I think we can abstract this detail in the first iteration by assuming that there will be a service which receive elastic search response and load the data however it wants (just like how you modeled the GraphQL query), because it can be varied based on the general architecture of the software I believe. For me as an example, I build on top of a microservice architecture so I would query my data access services using REST or gRPC but still I have an internal service class in my project that takes care of this action.

[daniefer]

The open questions where are:

1. What interface do you currently use to query elastic search
2. How do you query the database based on the search result

1. NEST is our api of choice
2. we would be fetching data by primary key.

[Blue4x4Rebel]

Our Experience

We have a setup currently using ES with HotChocolate. While we talked early on about trying to setup an IQueryable of our own to help split logic between search and actual data return, we ended up with a sort of hybrid setup. We actually load far more of our data into Elastic than just the searchable fields -- enough to populate one of the "core" objects on our graph. From there, we use stitching to fill the gaps. It gives us a sort of best-of-both-worlds using the current setup. Just waiting for v13 and some of the stitching/federation improvements :)

Schema Definition

To define our ES schema we use strings. Initially this meant defining it as POCOs several times, since we had the definition for ES, the definition for our database, and the definition for HC. We've since shifted to a schema-first approach, as having to hydrate all those objects all over the place just isn't efficient, especially when you don't need the full object. For many operations, we don't actually hydrate an object until just before it goes out to the database. We've actually shifted completely away from .NET for populating the ES index just because of performance.

Driver Concerns

When you're looking at the driver, beware of NEST. Newer versions are no longer compatible with AWS/OpenSearch, and I suspect many users lean in that direction. It's an interesting time in this space, and it may not be good to hitch the wagons to one particular horse (if it can be avoided).

What we'd like to see

I love the idea of plugging in filters, pagination, and sorting, as we had to roll our own for each of these. While we do like a clean filter approach as mentioned above, having some help getting from there to the ES syntax would be fantastic. Maybe a way to specify filter options that are allowed? This seems especially relevant in a world where the schema in ES differs from the database schema. With more teams adopting NoSQL databases, this is especially relevant given there can be more than one way to retrieve things, multiple representations of the data, or even related data that isn't related in an ORM sense.

[PascalSenn]

So the whole OpenSearch story is a bit unfortunate. I also see that there is a new Package for elastic search 8 that is currently in Alpha. It's difficult to find a "one-fits-all" client at the moment.

For filtering we have to define a datastructure that we build during visitation of the input object. This can either be a custom data structure that we then apply to the driver, or we use the drivers data structure like Query. e.g. Filtering = Expressions, MongoDb = BsonDocument.

It would be nice if we have interop between NEST and the low-level client. Is something like this possible? Can you build parts of the query with nest and then "merge" it with the low-level client definition?

I imagine something like this could be possible:

public class BookFilterInputType : FilterInputType 
{
     protected override void Configure(IFilterInputTypeDescriptor<Book> descriptor) 
    {
          descriptor.Field(x => x.Title); // this adds all data operations for title (eq, neq, in, nin etc.)
          descriptor.Field(x => x.Description, descriptor => 
              {
                     descriptor.Operation(ElasticOperation.Search).Name("search").Type<StringType>();  // e.g. wildcard search
                     descriptor.Query(
                            "custom", 
                            (Query q, String v) => q.Match(m => m.Field("TextID").Query(v)  ))); // custom operation with nest interopt
              });
    }
}

Example query:

 {
    books(where: {title: {eq: "123"}, description: {search: "something", custom:"this will be in the v variable" } } ) {
       title
   }

Querstions:

What client are the OpenSearch people using?
What data structure are we going to use for building the context of the queries?
How is the interop between elasticsearch.net and NEST?

[ptdave20]

We are stuck using Nest 7.13.2 for opensearch until there is an official o.s. client .

[daniefer]

Nest 7.12.1 and we haven't started the process of preparing for the OpenSearch migration we will need to make.

Our ES indexes are very flat. We would end up doing a match with a few should statements.

We don't use elasticsearch.net directly, so I can't say much about interoperability.

[frzmohammadali]

@PascalSenn I don't have a strong opinion on OpenSearch but as far as Elasticsearch is concerned, Nest and Elasctisearch.Net have a somewhat good interoperability I would say. For instance we can build a SearchDescriptor and pass it to Nest client like following:

var query = new QueryContainerDescriptor<Contact>()
    .Match(descriptor => descriptor.Field(contact => contact.DisplayName).Query("foo bar"));

var searchDescriptor = new SearchDescriptor<Contact>()
            .Index("index_name")
            .TrackTotalHits(true)
            .Query(_ => query)
            .Size(100)
    ;

ISearchResponse<Contact> response1 = await _elasticClient.SearchAsync<Contact>(searchDescriptor);

Or we serialize it to raw query text and pass it to the Elasticsearch.Net low level client like this:

string rawQueryText = _elasticClient.RequestResponseSerializer.SerializeToString(searchDescriptor);

ISearchResponse<Contact> response2 = await _elasticLowLevelClient.SearchAsync<SearchResponse<Contact>>("index_name", rawQueryText);

The response type can be same for both clients like I demonstrated in the example above, but can be different as well. For example Elasticsearch.Net client supports raw StringResponse which might be helpful in some case. Maybe the Nest client also supports that response type but I haven't tried it honestly.

[PascalSenn]

Thanks to @A360JMaxxgamer things are moving again: #4998

[zeeshanpolaris]

Hello. I was wondering if there is plan to get this PR moving? Thank you.