Support for regularization

[hkirvesl]

Is your feature request related to a problem? Please describe.

When training models, the loss function with which the network weights are updated must be a function of the input data and the response variable, $L=L(X,y)$.

Many times you would like to add regularization to the model: If there are many ways to achieve similar training error, we would like a model that is in some way as simple as possible.

In order to do so, the parameter update step must take into account some properties of the network.

Describe the solution you'd like

A straightforward way to fix this problem would be to add this logic to the trainer class. This is consistent with the literature where regularization for model M with inputs X and response y is commonly phrased as

$$L_{\textrm{total}}(X,y,M) = L_{\textrm{original_loss}}(X,y) + \lambda L_{\textrm{regression_penalty}}(M). $$

Describe alternatives you've considered

An alternative way to achieve this would be to have a wrapper that modifies the model $M$ and input data $X$ to include whatever factors are needed, i.e. $X^{}=(X,M)$ and then use custom loss $L^{}(X^{},y^{})$.

However, this approach take does not generalize well (as each problem would need their own wrapper) and may lead to unnecessary overhead (e.g. Input $X^{*}$ needs to contain $M$).

[raphaelreinauer]

Would you like to incorporate a general regularization approach? If you are specifically interested in L2 penalty, it can be directly used within the Adam optimizer.

[RaphaelLilt]

@hkirvesl Can this be closed?

[hkirvesl]

@RaphaelLilt Absolutely, this should be closed indeed for it's solved by #132