Discussion: Asynchronous Tracing

[HaloFour]

In implementing Money tracing over Apache's HttpAsyncClient (#84) I've been giving a lot of thought to how money tracing could be generalized over asynchronous operations in general. Normally in TraceAspect when a method annotated with @Traced returns the span is immediately stopped.

A thought I had would be to add a new element async of type boolean with a default value of false. If that element value is set to true and the return type of the annotated method is determined to be one of the compatible types then the aspect would change how it advices the method so that the span remains open after the method returns successfully and that a callback would be registered on the return value which would be responsible for closing the span.

@Traced(value = "my-async-method", async = true)
public CompletionStage<String> myAsyncMethod() {

    // could be anything asynchronous here, not specifically executor/thread related
    CompletableFuture<String> future = new CompletableFuture<String>();
    executor.submit(() -> {
        // do some work
        future.complete("All Done");
    });

    return future;
}

The exact list of compatible return types would depend on the references available. Money currently targets JRE 1.6 which would preclude CompletableFuture<T> and CompletionStage<T> which were added in Java 8 and the current dependencies don't give us many other options. Normal Java Future could be instrumented but it would be tricky as that interface provides no mechanism for registering for completion which would require either wrapping/pointcuts and waiting on a call to Future#get or polling Future#isDone on a separate thread, neither of which are ideal.

It would require more engineering work but a mechanism could be built on Java 6 Service Loader to register classes that would be used to recognize other asynchronous types. That would allow for separate modules that could be used to instrument Java 8, Guava, Spring, etc.

[HaloFour]

I've opened #86 as a proof of concept implementation.

This PR:

1. Changes the @Trace annotation to have an async element that defaults to false.
2. Adds AsyncTraceService (awful name) trait which is used to recognize the return value as a future/promise and register a callback for completion.
3. Uses ServiceLoader to load any number of registered AsyncTraceService implementations.
4. Modifies TraceAspect so that if a method is annotated with @Trace where async is true and the return value is supported by one of the registered AsyncTraceService implementations that the span is left open until that future/promise is completed.
5. Separately logs the time of the method call itself as a timer on the span.

This is one possible implementation. I've also mulled having a second child span immediately started in the case of these "async" methods so that the parent span would be stopped when the method returns but the child span remains open until the future/promise completes.