config/index does not always preserve index correctly #957
Comments
jarpy
changed the title
|
For me or has always been one of the golden Riemann rules: thou shalt only use one index. It's stated in the documentation pretty clearly, don't you think? |
|
I must admit that I don't remember reading that in the documentation, but memory is not my best talent! :) Do you know where it is? The API documentation is the stuff that I always have open when I'm working with Riemann, so I might add that warning to the docstring if we need it (similar to the one that But I think we can do more, and ask whether that rule should be necessary. On my team, we have another rule. It's: Thou shalt not write an s-expression that fits not on thy 4K monitor. If we can only call Currently, the behaviour of (instrumentation {:enabled? false})
(tcp-server {:host "0.0.0.0"})
(periodically-expire 1)
(info "Let binding index.")
(let [index (index)]
(streams
(not-expired index prn)
(expired prn)))
(info "Calling (index).")
(index)works correctly if you send Riemann a A couple of ideas:
These options are presented in descending order of my personal preference, but also in descending order of invasiveness. What do folks think about them (or others)? Cheers. |
Fix the "orphaned indicies" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load.
Fix the "orphaned indicies" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load. Closes: riemann#957
Fix the "orphaned indicies" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load. Closes: riemann#957
|
I ended up going with option 1, which is implemented in #997. |
Set current index on first call of config/index Fix the "orphaned indices" problem described in riemann#957. Allows a Riemann config to make multiple calls to riemann.config/index with predictable results, both on initial load and after a reload. Without this change, multiple calls are OK after reload or explicit apply, but create orphan indices on initial load.
The
indexfunction inconfig.cljdoesn't always preserve the existing index as expected. In particular, whenindexis called multiple times within the same config, the index is replaced, which results in odd behaviour.Given this config:
and adding some debug logs to the index function, we see this output:
This causes weird behaviour with the index in any config where
indexis called (probably let-bound) multiple times. In particular, expiry gets messed up. With this config:when injecting an event with
:ttl 1, we'd expect to print an expired event within 2 seconds, but it never arrives:Is this because the reaper is attached to a different index than the one that contains the event?
Partly, I think this is related to the behaviour of
indexvs its docstring:"Set the index used by this core. Returns the index."It looks like it technically sets the index used by the next core, and that change is not made concrete until something or someone calls for a core transition. Adding some more debug, it looks like
(:index @core)remainsnileven after several calls to(index):A work-around in the config is to manually "fix" the index into the core by calling a combination of
indexandapply!before doing anything serious:This config preserves the index for future calls to
index, and expires the test event as expected:I wonder if the
indexfunction itself should ensure that the index is fixed on startup like this. Alternatively, should there be an additionalindex!function that really does set the index on the current core?So, please excuse the wall-o-text. I'd love to hear any ideas from experienced Riemann hackers on whether this looks like a real problem, and if so, what we could (safely) do to change it.
Thank you.
The text was updated successfully, but these errors were encountered: