Question: Filtering traced queries

[dmurat]

I'm wondering if there is a way to exclude some queries from being reported as traces.
I have a system that polls a database every second with a simple query, and I will like to exclude it from traces reported to the Jaeger.

[ttddyy]

@dmurat I believe you could use the default mechanism that Micrometer Observation API provides - ObservationPredicate.

https://micrometer.io/docs/observation#_observation_predicates_and_filters

[dmurat]

First, I'm new to micrometer-based tracing, so my observations and used terminology might be off, but I hope I can explain the idea.

After some code exploring end debugging, I think predicates will not suffice. There is no information about executed queries when predicates are evaluated. Filters are a different story, but I think they cannot easily suppress observation.

I believe in supporting query filtering, a broader change is required. I came up with the following idea. When stopped, each query observation can evaluate whether contained queries should be ignored and filtered out. If all included queries are to be ignored, observation should be abandoned. In addition, it should report the count of ignored and non-ignored queries to its parent connection observation. When connection observation is stopped, it should evaluate its query counts, and if it contains only ignored queries, connection observation should be abandoned too.

Of course, for smooth integration with Boot, all options should be configurable, including a list of ignored queries, whether query observations should be abandoned if they contain only ignored queries, and whether connection observations should be abandoned if they contain only ignored queries.

This is it. What do you think?

[ttddyy]

How about this predicate?

public class MyPredicate implements ObservationPredicate {

	@Override
	public boolean test(String name, Context context) {
		if (context instanceof QueryContext queryContext) {
			return queryContext.getQueries().stream().noneMatch(query -> query.contains("QUERY TO IGNORE"));
		}
		return true;
	}
}

[dmurat]

Maybe I don't know how to set it up properly, but I can't get any queries in the predicate. As long as I can see, predicates are evaluated immediately after an empty observation is created. There are no queries available to the predicate.

Here is my setup code in Spring Boot 3 (the code is in Groovy, but should be easy to follow):

  @Bean
  ObservationRegistryCustomizer<ObservationRegistry> myObservationRegistryCustomizer() {
    return (ObservationRegistry observationRegistry) -> {
      observationRegistry.observationConfig()
          .observationPredicate((String observationName, Observation.Context observationContext) -> {
            if (observationContext instanceof QueryContext) {
              QueryContext queryContext = observationContext
              List<String> queries = queryContext.getQueries()
              if (queries) {
                println "from predicate: ${ queries.inspect() }" // --> never prints anything
              }
              return true
            }
            return true
          })
          .observationFilter((Observation.Context observationContext) -> {
            if (observationContext instanceof QueryContext) {
              QueryContext queryContext = observationContext
              List<String> queries = queryContext.getQueries()
              if (queries) {
                println "from filter: ${ queries.inspect() }"
              }
            }

            return observationContext
          })
    }
  }

[dmurat]

After some more digging, it looks like DataSourceObservationListener.startQueryObservation() calls populateQueryContext() after createAndStartObservation(). It looks like populateQueryContext() should be before createAndStartObservation(). Then queries may be available in the predicate.

[dmurat]

After moving populateQueryContext () in front of createAndStartObservation(), the predicate started to work as expected :-)

Now, I will like to have the ability not to report connection observations where the connection was used only for ignored queries. Similarly, I will like to be able to ignore result sets of ignored queries. I'm aware of jdbc.includes configuration property, but instead of all-or-nothing, I would like to have a selective approach to ignoring connections and result sets.