Feature/orthogonal initialization

[miquelflorensa]

Description

This PR added orthogonal initialization for Linear, Convolutional and LSTM layers.

For Linear and LSTM layers, orthogonality is achieved on matrices of size input × output. For CNN layers, orthogonality is enforced between the kernels. By initializing weight matrices with orthogonal structures, the norm of the input is preserved across layers, leading to a more stable training.

Changes Made

• Added orthogonal init function in src/param_init.cpp.
• Modified initialization of Linear, CNN and LSTM layers to accept orthogonal init in src/param_init.cpp.
• Added #include <eigen3/Eigen/Dense> in include/param_init.h.

Checklist

• I have followed the project's coding conventions and style guidelines.
• I have updated the documentation, if applicable.
• I have rebased my branch on the latest upstream code to incorporate any changes.
• I have tested the changes locally.

Notes for Reviewers

I use the external library Eigen to perform SVD efficiently on C++. Hence it is necessary to install the library running:

sudo apt install libeigen3-dev

or manually installing the library from Eigen.

To use orthogonal initialization in Python, update the init_method parameter as shown below:

TAGI_CNN_BATCHNORM = Sequential(
    Conv2d(1, 32, 4, padding=1, in_width=28, in_height=28, bias=False, init_method="orthogonal"),
    ReLU(),
    BatchNorm2d(32),
    AvgPool2d(3, 2),
    Conv2d(32, 64, 5, bias=False, init_method="orthogonal"),
    ReLU(),
    BatchNorm2d(64),
    AvgPool2d(3, 2),
    Linear(64 * 4 * 4, 256, init_method="orthogonal"),
    ReLU(),
    Linear(256, 11, init_method="orthogonal"),
)

[miquelflorensa]

@lhnguyen102 I added the orthogonal initialization by using an external C++ library. Should I somehow try to implement a simple SVD or should we add it as another external library?

[lhnguyen102]

@miquelflorensa We tend to move away from external libraries but in this case, it will take time to implement SVD. I would rather focus on the relevant work. Please make sure it works on both MACOS and Ubuntu. I would appreciate if you can update the installation instructions in the doc for both OS

[jamesgoulet]

@miquelflorensa I tried installing your branch on my Mac and it does not compile.

Regarding the dependency on external library, I did not realized that one would have to manually install the two libraries in order to pip install .. It seems limiting from an external user point of view. Can you check the feasibility of using chatgpt to create a cpp svd function that would be equivalent to the one you are using? I can assume it will not be as efficient, but this SVD is only use in this initialization step. We can discuss it further over zoom or in person when you are back.