How to add a spy to port ?

[ymadzhunkov]

Hi, I have the following scenario.
I have 4 kernels that do processing of some sort. Each of them has one input and one output ( first and last are single output/ single input).

 m += a >> b >> c >> d;

I am designing a 'Spy' type of kernels that I want to attach to specific ports and monitor the content that passes though. They will not modify anything, just read the content. Image this will be used for visualization purposes, testing, logging or data dump.

One solution I found is to create an sky kernel and include it in the production chain.

 m += a >> spy >> b >> c >> d;

However, this has the disadvantage that I have to edit the production chain code.

I would much more like to write something like:

spy.attach( a, b);

Is there any way I can do this with raftlib ? Can you provide an example. Thank in advance.

[mr-j0nes]

Hi there,
how about inheriting all your kernels from a super-kernel where you can spy all you want.
This super-kernel will have the run() method, from which you will call a subRun() method where you will put the current logic in every kernel.
The data to be spied can be sent to subRun() as parameter so afterwards the super-kernel can do what it needs...
Hope it helps,
Cheers..

[jonathan-beard]

I've been contemplating something like this. A colleague of mine when I was a new grad student built this instrumentation system for yet another streaming framework called Auto-Pipe (which in many respects is where I got the idea of building something entirely new which is RaftLib). The TimeTrial system had an annotation language for the streams. Maybe we could build a kernel as @mr-j0nes suggested and it have basic hooks to do monitoring and instrumentation. Once I add multi-node support, that would be preferable to adding your own given the run-time could move your kernels out from under you and you'd have very odd readings on your instrumentation. Any thoughts on the hooks that'd be needed? Ones that immediately come to mind for me are arrival rates on streams, streaming averages of things, and some moment estimation on the streams. I think I still have kernels in the repo for most of that which can be used to tie-in things. I'd suspect we could provide some lower-level pull-outs from the ring-buffer class that are privileged to the "spy" class. I think given the recent security scares..and for ease of text searching, lets go with "monitor" or perhaps "instrument" as a class name.

If you're willing to put a proposal together, I'm willing to help code them. It can definitely be a collaborative process.