[WIP/RFC] Initial support for modules

[vchuravy]

This is one of the bigger features that the Python frontend has that we are currently missing.
My goals with this are to support several different use-cases:

• Bucketing: Bucketing in MXNet.jl #129
• Custom operators in Julia: Julia-based definition of new layer #39, Any plans to add custom loss functions? #77, Is creation of custom operators supported? #166, Custom output such as exponential function? #167
• For optimal performance I plan on integrating this with CUDAnative.jl for Julia v0.6 so that custom operators can be executed on the GPU

So far I have been working on a straight port of the Python functionality, but I would be interested in a bit more design discussion.

PRs against this branch are very welcome.

[Arkoniak]

Thank you, it would be nice to have modules api implemented. I have couple of preliminary questions

1. Now Nullable{...} should be used instead of Union{Void, ...}?
2. May be make organization of modules in the same way as optimizers? I mean, move AbstractModule to "src/module.jl" and all implementations in "src/modules/...". it looks more natural to general structure of MXNet.jl.
3. Should SymbolModule support AbstractDataProvider? There is slight inconvenience, that AbstractDataProvider has its own functions for data/label names, so user will be forced to enter same information twice, moreover dataprovider's and module data_names can conflict. May be it would be better to infer names from DataProvider but unfortunately it is only possible during bind.
4. Are there any reasons to put modules in their own namespace? At first glance new functions shouldn't conflict with previous code, and mx.Module.SymbolModule looks redundant. But may be I am missing something important.

[vchuravy]

1. Yes, Nullable is preferred.
2. Will do that on the next push.
3. Will have to think about that.
4. Right now it is to make sure that things won't conflict and we can reexport the important bindings later once the interface has stabilised.

[vchuravy]

Rebased to include #179.

[vchuravy]

@pluskid If you have the time, it would be nice to get your opinion of the implementation so far.

[pluskid]

@vchuravy Thanks a lot! Currently I did not have time to go through it very carefully. But quickly scanning it and it looks good.

Regarding SequentialModule in another PR, maybe for output_shapes, instead of using Dict, we still keep vector of name, shape pairs?

[vchuravy]

Some thoughts I had over the last days:

• @pluskid and I had a conversation in Port Executor Manager to Julia #153 on not doing automatic data parallelism in SymbolModule e.g. not using ExecutorGroup, what we could do instead is to simplify SymbolModule and provide a DataParallelModule wraps a module and parallelises it over a set of devices. This would split the complexity of ExecutorGroup and might make the API simpler.
• For sequential modules the wiring of input to output needs to be done during bind and there should be no need to pass data into backwards and forwards.

[pluskid]

@vchuravy I agree that the current implementation is a bit complicated due to auto data par. DataParallelModule might be a good idea. I would support it if it solve the issue. Though maybe we need to inspect and design a bit more carefully. For example, data par implementation needs to make sure the gradients are aggregated via the same KVStore.

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@2a66ea3). Click here to learn what that means.

@@           Coverage Diff            @@
##             master    #173   +/-   ##
========================================
  Coverage          ?   75.8%           
========================================
  Files             ?      30           
  Lines             ?    2294           
  Branches          ?       0           
========================================
  Hits              ?    1739           
  Misses            ?     555           
  Partials          ?       0

Impacted Files	Coverage Δ
src/MXNet.jl	100% <ø> (ø)
src/module/pipeline.jl	0% <ø> (ø)
src/executor.jl	88.5% <100%> (ø)
src/module/sequential_module.jl	38.39% <38.39%> (ø)
src/symbolic-node.jl	81.27% <50%> (ø)
src/module/Module.jl	64.7% <64.7%> (ø)
src/io.jl	92.19% <72.72%> (ø)
src/module/symbol_module.jl	74.16% <74.16%> (ø)
src/executor-group.jl	81.2% <81.2%> (ø)
src/ndarray.jl	85.13% <93.33%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2a66ea3...fa45309. Read the comment docs.

[iblislin]

😱 why close this?

[vchuravy]

I don't have any time to devote to this and I don't want the illusion of progress,