provide a relative starting point for runtime event timestamps

[c-cube]

Currently, runtime events are annotated with a timestamp (internally, a int64 that seems to correspond to nanoseconds). I have two issues with it curently:

• there isn't that much in the documentation about the unit of the int64. It seems to be nanoseconds but I think this should be clearly documented so it can be relied on.
• there isn't an anchor for these timestamps, nor a way to call the caml_time_counter from user code. This means that these timestamps have no meaning that can be related to, say, timestamps from mtime, or values of rdtsc (even though they might come from the same source). For tracing purposes, I personally would need to be able to correlate the Mtime.Span.t timestamps that I use for most spans (elapsed since beginning of program) with the ones that come from runtime events, in order to combine the two data sources into a coherent whole that can be visualized.

so really I think there needs to be more features around timestamps: either expose caml_time_counter (as a monotonic clock with a well defined unit of measure; and then mtime could provide a compatibility layer with it); or at least provide information about the starting point of the program.

[gasche]

caml_time_counter is indeed currently placed under CAML_INTERNALS, so not supposed to be relied upon by users. It is well documented, but the specification is not so clear (I would say: best-effort monotonic counter):

/* Returns the current value of a counter that increments once per nanosecond.
   On systems that support it, this uses a monotonic timesource which ignores
   changes in the system time (so that this counter increases by 1000000 each
   millisecond, even if the system clock was set back by an hour during that
   millisecond). This makes it useful for benchmarking and timeouts, but not
   for telling the time. The units are always nanoseconds, but the achieved
   resolution may be less. The starting point is unspecified. */
extern uint64_t caml_time_counter(void);

I think that it would be reasonable to put this function out of CAML_INTERNALS (so we commit to support it), and even possibly expose it as Unix.time_counter : unit -> int64 or Unix.monotonic_time : unit -> int64 with the same documentation, and then document in the runtime events documentation that the timestamps are produced by this function.

[gasche]

(In any case I take @dbuenzli's point that we should take inspiration of his API design in mtime, instead of reinventing the wheel with slightly different conventions. I also wonder if there are thinks to learn from the implementation itself, but I have no idea what's in the runtime today and how they compare.)

[dbuenzli]

I am not sure that the "featureful" approach is best for the ecosystem and also the maintenance of the compiler.

Not exactly sure why. I would even say that providing abstract types is better for the eco-system and the maintenance of the OCaml system.

Also it's not scary "abstraction-breaking construction and deconstruction functions", it's friendly "converters" :-)

But in any case I think the representations should rather go in the stdlib so that this gives the eco-system a nice type to agree on for representing monotonic time.

[gasche]

I don't necessarily trust us upstream to take good care of this module that is there only to support what (relatively to the whole distribution) is only a niche concern. In practice I am worried about:

• people complaining that the API should be done better, and that it would be easier to improve and iterate outside the stdlib (example: Filename)
• or people pushing to add more features there, or to add code to support their favorite OS, that adds more maintenance burden for us

I am not saying that this is clearly a bad approach. It may be a good move, but it also involves a fair amount of work (now and in the future) and I am not sure we want to commit to that for the tiny usage of Mtime we have.

[c-cube]

I think these are reasonable concerns in general. However:

• timestamps are already in runtime events, and to make these more useful it makes sense to expose the monotonic clock they rely on
• clocks are pervasively useful for all sorts of performance related concerns. It's no coincidence that even rust has them in their stdlib, or that it's part of posix.
• I think mtime is fairly small already, but the minimal API the stdlib could/should provide, imho, is more or less now_ns: unit -> int64 and start_time: int64. Mtime itself could probably build on that in a compatible way, although I don't want to push work on @dbuenzli .
• supporting another OS definitely means supporting its timestamps! That also holds for runtime events and the rest of the Unix-like modules. Worst case, on x86_64, it can be done by calling the rdtsc instruction.

So just like Uchar is the minimum we need as a compatibility layer for everyone to handle unicode, a monotonic clock would be a good compatibility layer for all timing/profiling needs, including the stdlib's/runtime_events own.

[dbuenzli]

• I think mtime is fairly small already, but the minimal API the stdlib could/should provide, imho, is more or less now_ns: unit -> int64 and start_time: int64. Mtime itself could probably build on that in a compatible way, although I don't want to push work on @dbuenzli

Not sure if you have something else in mind but you already have Mtime.{to,of}_uint64_ns.

If you meant reimplementing Mtime_clock in term of these you'd need one more bit more since I expose, for those interfaces that support it, the clock frequency (expressed as its inverse to avoid introducing yet another type). Also mtime doesn't seem to use exactly the same sources as caml_time_counter but that could be evened out.

Personally I also find monotonic time to be pervasives enough to be warranted types for representing it in the stdlib and, possibly, a way to measure it. Not even for profiling, in any service, app, cli tool that runs periodic or long tasks you are interested in reporting how much time those take to run – that tiny mtime dependency "you hate on having to add to all of your projects".

[gasche]

I wonder if we could store somewhere the timestamp at the beginning of the program and then subtract whenever we ask for a timestamp. Would that end up being costly, or neglectible compared to the other stuff going on in runtime events? I would bet "neglectible" but I don't know.

[c-cube]

The simplest, imho, is to store it at the beginning and provide a function to access it (val start_time : unit -> Timestamp.t). The user can decide to subtract or not.

[c-cube]

Re-reading all that a few weeks later, I'd hate to see it just fall by the wayside.

I think a minimal solution would be a good start. The actual only thing we'd need for that, I think, is just Sys.monotonic_time_ns : unit -> int64 or equivalent, alongside a port of the more varied sources that mtime uses (mach_continuous_time, CLOCK_BOOTTIME on linux, etc.). In addition, runtime events' Timestamp.t would gain the guarantee to be compatible with these timestamps so that they're convertible/comparable.

Mtime can still be useful to get the clock frequency, provide a nice API for spans, an elapsed function, etc. as long as it's using that new standard clock as a basis for timestamps.

I'm not sure whether adding this to Sys would require a bootstrap; if it doesn't, I'm willing to open a PR to do the work. I'd hope Daniel would be game to review such a PR if it happens :)

[gasche]

I don't feel like asking you for more work right now given how succesful my previous idea was ("oh it's probably easy to add thread-local storage!"). But if you want to work on this, yes, personally I think this would be fine.

[dbuenzli]

I'm mostly fine with your idea but I'm not going to provide the clock period lookup in mtime based on assumptions on what Sys.monotonic_time_ns : unit -> int64 could be doing. So if I don't get that too I'm not going to switch to Sys.monotonic_time_ns in mtime.