Abstraction layer for switching between Theano/TensorFlow/... backends

[ibab]

By adding a layer between the framework and the underlying symbolic computation engine, the backend could be switched out.
This would be similar to what has been done for Keras: http://keras.io/backend/

[glouppe]

@fchollet tells me on Twitter that the abstraction layer is in fact very standalone, which is great :) It would be worth investigating if all the components we need have been abstracted away though, since it is very NN-oriented.

[glouppe]

Hi Igor! I see you started working on something similar based on Tensorflow. What are your impressions so far? (Tensorflow is really developed rapidly at the moment, which is nice!)

[ibab]

Hi, in many ways tensorflow feels like an improved theano. The compilation time is almost negligible and the symbolic language is more flexible. Also, you can implement your operators in C++ (and link them in dynamically), which is nicer than constructing the code for a C++ program in Python (never quite understood how that works in theano).
The maintainers are very active and pull requests get reviewed quickly (big 👍 ).

The internals are well-designed and can be understood by someone with C++ knowledge in a few days or so. A lot of the code is based on a very nice Tensor module that they've pushed upstream to eigen3.

I love that the computational graph in tensorflow is simply a protobuf message, that's a great idea.
(Allows you to edit/optimize it from both C++ and Python, can be serialized quickly, operations on it are fast).

Right now tensorflow has a few rough edges, for example you might find that an operator you want doesn't exist or is only enabled for float32 instead of float64, and so on.
Also, I found it really difficult to rewrite graphs, as the API is very low-level, and there are not enough convenience functions for doing things that should be simple.
It's also not that easy to get running at the moment (for example, the wheel for python 3 will just segfault on python 3.5).

I think that it makes sense to go with tensorflow for new projects, although theano is certainly nice as well. Using an abstraction layer to switch between the two makes a lot of sense for projects that are already based on theano.

[ibab]

Oh, and I've tried running a large fit on several GPUs at once and it was 🚀

[cranmer]

Talking with @fegin and I think it would be nice to do this so that we can start some tests. So I'd up the priority on this.

[glouppe]

@cranmer Not sure if you have followed, but @ibab started tensorprob (https://github.com/tensorprob/tensorprob). It is more or less the same thing but based on TensorFlow. Seems pretty neat so far!

[cranmer]

oh, I didn't see that. I was hoping that would wait until this other discussion converged. But I'm glad to see progress. Can we try to organize a discussion together with @fegin sometime soon.