Capnp message size caps limits one submit size

[gavento]

Capnp limits builders to 8M words (64 MB) which bounds the size of submitted nodes to cca 200 000. This limit is fixed in message.h since Builder does not accept any options and uses the default limit.

🌧  INFO 2018-03-17T14:53:11Z Starting local server (0.0.0.0:7210)
🌧  INFO 2018-03-17T14:53:11Z Dashboard: http://lomikamen:8080/
🌧  INFO 2018-03-17T14:53:11Z Server pid = 95230
🌧 ERROR 2018-03-17T14:53:11Z Process 'server' terminated with exit code exit code: 101; process outputs can be found in server.{out/err}
🌧  INFO 2018-03-17T14:53:11Z Error occurs; clean up started processes ...
Expect 16384 objects of size 32768, total 512.000 MB
Traceback (most recent call last):
  File "scalebench.py", line 61, in <module>
    session.submit()
  File "/aux/gavento/rtest/rain/python/rain/client/session.py", line 164, in submit
    self.client._submit(self._tasks, self._dataobjs)
  File "/aux/gavento/rtest/rain/python/rain/client/client.py", line 82, in _submit
    req.send().wait()
  File "capnp/lib/capnp.pyx", line 1932, in capnp.lib.capnp._RemotePromise.wait (capnp/lib/capnp.cpp:41109)
capnp.lib.capnp.KjException: src/capnp/rpc.c++:1527: context: sending RPC call; callBuilder.getInterfaceId() = 14056942509132787212; callBuilder.getMethodId() = 3
capnp/rpc-twoparty.c++:92: failed: expected size < ReaderOptions().traversalLimitInWords; size = 12047798; Trying to send Cap'n Proto message larger than the single-message size limit. The other side probably won't accept it and would abort the connection, so I won't send it.
stack: 0x7ffff66aacd6 0x7ffff66a88c0 0x7ffff66a7580 0x7ffff663d0e9 0x7ffff661e9f3 0x7ffff661eb1d 0x7ffff661ef4b 0x5555556d4411 0x5555556d493f 0x5555556d493f 0x5555556d9286 0x5555556d9f9f 0x5555557a78f2 0x5555557a9e1d 0x5555557aa5be 0x5555557d84d7 0x555555668c01 0x7ffff6cee2b1 0x5555557721ba

[gavento]

A partial solution is to call submit() with smaller chunks, but that also requires involved object ordering and keeping the boundary objects.

[spirali]

We can introduce an internal fragmentation of submits to multiple calls that is composed in the server and pushed to scheduler at once. This way allows us to avoid "keeping" boundary objects.

Btw: This has another implication, it limits the maximal allowed size of a response from the datastore. We should limit it in the datastore server implementation. The same problem also in worker <-> subworker in memory exchanges.

[gavento]

Will be most likely resolved when #11 is resolved for the client API.