Skip to content

awintel/lava-dl

 
 

Repository files navigation

Lava DL

lava-dl is a library of deep learning tools within Lava that support offline training, online training and inference methods for various Deep Event-Based Networks.

There are two main strategies for training Deep Event-Based Networks: direct training and ANN to SNN converison.

Directly training the network utilizes the information of precise timing of events. Direct training is very accurate and results in efficient networks. However, directly training networks take a lot of time and resources.

On the other hand, ANN to SNN conversion is especially suitable for rate coded SNNs where we can leverage fast training of ANNs. These converted SNNs, however, typically require increased latency compared to directly trained SNNs.

Lava-DL provides an improved version of SLAYER for direct training of deep event based networks and a new ANN-SNN accelerated training approach called Bootstrap to mitigate high latency issue of conventional ANN-SNN methods for training Deep Event-Based Networks.

The lava-dl training libraries are independent of the core lava library since Lava Processes cannot be trained directly at this point. Instead, lava-dl is first used to train the model which can then be converted to a network of Lava processes using the netx library using platform independent hdf5 network description.

The library presently consists of

  1. lava.lib.dl.slayer for natively training Deep Event-Based Networks.
  2. lava.lib.dl.bootstrap for training rate coded SNNs.

Coming soon to the library

  1. lava.lib.dl.netx for training and deployment of event-based deep neural networks on traditional as well as neuromorphic backends.

More tools will be added in the future.

Lava-DL Workflow

Drawing

Typical Lava-DL workflow:

  • Training: using lava.lib.dl.{slayer/bootstrap} which results in a hdf5 network description. Training usually follows an iterative cycle of architecture design, hyperparameter tuning, and backpropagation training.
  • Inference: using lava.lib.dl.netx which generates lava proces from the hdf5 network description of the trained network and enables inference on different backends.

Installation

Cloning Lava-DL and Running from Source

We highly recommend cloning the repository and using pybuilder to setup lava. You will need to install pybuilder for the same.

Note: We assume you have already setup Lava with virtual environment. Make sure PYTHONPATH contains path to Lava core library first.

  • Linux/MacOS: echo $PYTHONPATH
  • Windows: echo %PYTHONPATH%

The output should contain something like this /home/user/lava

[Linux/MacOS]

$ git clone [email protected]:lava-dl/lava.git
$ cd lava-dl
$ pip install -r build-requirements.txt
$ pip install -r requirements.txt
$ export PYTHONPATH=$PYTHONPATH:$(pwd)/src
$ pyb -E unit

[Windows]

cd %HOMEPATH%
git clone git@github.com:lava-dl/lava.git
cd lava-dl
pip install -r build-requirements.txt
pip install -r requirements.txt
set PYTHONPATH=%PYTHONPATH%;%cd%\src
pyb -E unit

You should expect the following output after running the unit tests:

PyBuilder version 0.13.3
Build started at 2021-11-05 18:44:51
------------------------------------------------------------
[INFO]  Installing or updating plugin "pypi:pybuilder_bandit, module name 'pybuilder_bandit'"
[INFO]  Processing plugin packages 'pybuilder_bandit' to be installed with {}
[INFO]  Activated environments: unit
[INFO]  Building lava-nc/lava-dl version 0.2.0
......  PyBuilder Logs ...
[INFO]  Running unit tests
[INFO]  Executing unit tests from Python modules in /home/user/lava-dl/tests
[INFO]  Executed 80 unit tests
[INFO]  All unit tests passed.
......  PyBuilder Logs ...
------------------------------------------------------------
BUILD SUCCESSFUL
------------------------------------------------------------
Build Summary
             Project: lava-nc/lava-dl
             Version: 0.2.0
      Base directory: /home/user/lava-dl
        Environments: unit
               Tasks: prepare [45089 ms] analyze [660 ms] compile_sources [0 ms] run_unit_tests [184641 ms] package [1086 ms] run_integration_tests [0 ms] verify [0 ms] publish [15128 ms]
Build finished at 2021-11-05 18:49:25
Build took 273 seconds (273800 ms)

[Alternative] Installing Lava from Binaries

If you only need the lava package in your python environment, we will publish Lava releases via GitHub Releases. Please download the package and install it.

Open a python terminal and run:

[Windows/MacOS/Linux]

$ python3 -m venv python3_venv
$ pip install -U pip
$ pip install lava-nc-0.1.0.tar.gz

Getting Started

End to end tutorials

Deep dive tutorials

lava.lib.dl.slayer

lava.lib.dl.slayer is an enhanced version of SLAYER. Most noteworthy enhancements are: support for recurrent network structures, a wider variety of neuron models and synaptic connections (a complete list of features is here). This version of SLAYER is built on top of the PyTorch deep learning framework, similar to its predecessor. For smooth integration with Lava, lava.lib.dl.slayer supports exporting trained models using the platform independent hdf5 network exchange format.

In future versions, SLAYER will get completely integrated into Lava to train Lava Processes directly. This will eliminate the need for explicitly exporting and importing the trained networks.

Example Code

Import modules

import lava.lib.dl.slayer as slayer

Network Description

# like any standard pyTorch network
class Network(torch.nn.Module):
    def __init__(self):
        ...
        self.blocks = torch.nn.ModuleList([# sequential network blocks 
                slayer.block.sigma_delta.Input(sdnn_params), 
                slayer.block.sigma_delta.Conv(sdnn_params,  3243),
                slayer.block.sigma_delta.Conv(sdnn_params24363),
                slayer.block.rf_iz.Conv(rf_params36643, delay=True),
                slayer.block.rf_iz.Conv(sdnn_cnn_params64643, delay=True),
                slayer.block.rf_iz.Flatten(),
                slayer.block.alif.Dense(alif_params64*40100, delay=True),
                slayer.block.cuba.Recurrent(cuba_params10050),
                slayer.block.cuba.KWTA(cuba_params5050, num_winners=5)
            ])

    def forward(self, x):
        for block in self.blocks: 
            # forward computation is as simple as calling the blocks in a loop
            x = block(x)
        return x

    def export_hdf5(self, filename):
        # network export to hdf5 format
        h = h5py.File(filename, 'w')
        layer = h.create_group('layer')
        for i, b in enumerate(self.blocks):
            b.export_hdf5(layer.create_group(f'{i}'))

Training

net = Network()
assistant = slayer.utils.Assistant(net, error, optimizer, stats)
...
for epoch in range(epochs):
    for i, (input, ground_truth) in enumerate(train_loader):
        output = assistant.train(input, ground_truth)
        ...
    for i, (input, ground_truth) in enumerate(test_loader):
        output = assistant.test(input, ground_truth)
        ...

Export the network

net.export_hdf5('network.net')

lava.lib.dl.bootstrap

In general ANN-SNN conversion methods for rate based SNN result in high latency of the network during inference. This is because the rate interpretation of a spiking neuron using ReLU acitvation unit breaks down for short inference times. As a result, the network requires many time steps per sample to achieve adequate inference results.

lava.lib.dl.bootstrap enables rapid training of rate based SNNs by translating them to an equivalent dynamic ANN representation which leads to SNN performance close to the equivalent ANN and low latency inference. More details here. It also supports hybrid training with a mixed ANN-SNN network to minimize the ANN to SNN performance gap. This method is independent of the SNN model being used.

It has similar API as lava.lib.dl.slayer and supports exporting trained models using the platform independent hdf5 network exchange format.

Example Code

Import modules

import lava.lib.dl.bootstrap as bootstrap

Network Description

# like any standard pyTorch network
class Network(torch.nn.Module):
    def __init__(self):
        ...
        self.blocks = torch.nn.ModuleList([# sequential network blocks 
                bootstrap.block.cuba.Input(sdnn_params), 
                bootstrap.block.cuba.Conv(sdnn_params,  3243),
                bootstrap.block.cuba.Conv(sdnn_params24363),
                bootstrap.block.cuba.Conv(rf_params36643),
                bootstrap.block.cuba.Conv(sdnn_cnn_params64643),
                bootstrap.block.cuba.Flatten(),
                bootstrap.block.cuba.Dense(alif_params64*40100),
                bootstrap.block.cuba.Dense(cuba_params10010),
            ])

    def forward(self, x, mode):
        ...
        for block, m in zip(self.blocks, mode):
            x = block(x, mode=m)

        return x

    def export_hdf5(self, filename):
        # network export to hdf5 format
        h = h5py.File(filename, 'w')
        layer = h.create_group('layer')
        for i, b in enumerate(self.blocks):
            b.export_hdf5(layer.create_group(f'{i}'))

Training

net = Network()
scheduler = bootstrap.routine.Scheduler()
...
for epoch in range(epochs):
    for i, (input, ground_truth) in enumerate(train_loader):
        mode = scheduler.mode(epoch, i, net.training)
        output = net.forward(input, mode)
        ...
        loss.backward()
    for i, (input, ground_truth) in enumerate(test_loader):
        mode = scheduler.mode(epoch, i, net.training)
        output = net.forward(input, mode)
        ...

Export the network

net.export_hdf5('network.net')

lava.lib.dl.netx 

For inference using Lava, lava.lib.dl.netx provides an automated API for loading SLAYER-trained models as Lava Processes, which can be directly run on a desired backend. lava.lib.dl.netx imports models saved via SLAYER using the hdf5 network exchange format. The details of hdf5 network description specification can be found here.

Example Code

Import modules

from lava.lib.dl.netx import hdf5

Load the trained network

# Import the model as a Lava Process
net = hdf5.Network(net_config='network.net')

Attach Processes for Input Injection and Output Readout

from lava.proc.io import InputLoader, BiasWriter, OutputReader

# Instantiate the processes
input_loader = InputLoader(dataset=testing_set)
bias_writer = BiasWriter(shape=input_shape)
output = OutputReader()

# Connect the input to the network:
input_loader.data_out.connect(bias_writer.bias_in)
bias_writer.bias_out.connect(net.in_layer.bias)

# Connect network-output to the output process
net.out_layer.neuron.s_out.connect(output.net_output_in)

Run the network

from lava.magma import run_configs as rcfg
from lava.magma import run_conditions as rcnd

net.run(condition=rcnd.RunSteps(total_run_time), run_cfg=rcfg.Loihi1SimCfg())

About

Deep Learning library for Lava

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Jupyter Notebook 60.8%
  • Python 35.7%
  • Cuda 3.5%