Investigate gradient-based optimisation algorithms

[ben18785]

You guys have probably thought about this but I was wondering about trying the following algorithms out in the optimisation side of things,

• stochastic gradient descent with/without momentum or with Nesterov momentum,
• adagrad,
• RMSprop with/without Nesterov momentum,
• Adam.

There are also second order methods that we could look into,

• Newton's method,
• conjugate gradients,
• BFGS.

I have seen these all being used with statistical inference problems before and I wondered about their use.

[martinjrobins]

+1 for the stochastic gradient descent methods, I've been wanting to implement some of these. The second order ones would be fairly trivial, just need to wrap scipy.optimise (scipy already in dependencies). I think the main design change would be allowing models to return gradients.

[MichaelClerx]

Scipy has a bunch of them that we could wrap. None of them work for ion channel problems so I'm not super interested myself :-)

[MichaelClerx]

See also #55 and #54

[MichaelClerx]

Duplicate of #684