Possible to convert a GRU Classifier into HLS?

[IainBarkley]

Hello hls4ml team,
I'm working on implementing a GRU-based classifier for FPGA deployment using hls4ml. I've been able to successfully convert simple GRU models based on your test examples, but I'm encountering issues when adding a classification layer on top of the GRU.
My model structure is as follows:

class GRUClassifier(nn.Module):
    def __init__(self, input_size=32, hidden_size=128, output_size=10):
        super().__init__()
        self.gru = nn.GRU(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=1,
            batch_first=True,
            bias=True,
            bidirectional=False
        )
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x, h0):
        batch_size = x.shape[0]
        seq_length = x.shape[1]
        output, hidden = self.gru(x, h0)
        last_hidden = hidden.view(batch_size, -1)
        logits = self.fc(last_hidden)
                         
        return logits

I see from your tests that GRU models return the sequence output, but when I try to convert the classifier, I typically get symbolic tracing problems.
Is it currently possible to implement a GRU + classification layer using hls4ml?

If so, are there any specific modifications I should make to my model structure to improve compatibility?

Should I be using streaming (no explicit h0) or non-streaming (with h0) architecture for better results?

I couldn't see any examples of RNN-based classifiers that have been successfully converted, so I'm just wondering if this is even possible with RNN

My target is a sequence length of 500 with 32 input features and 128 hidden features, but I can adjust these parameters if needed for better FPGA implementation.

Thank for your time,
Iain Barkley

[JanFSchulte]

Hi Iain,

this should be supported. Looking at your model, I do get this error when trying to parse it:

Exception: Unsupported function getattr

This is actually not an issue of the Linear layer, but the first two lines of the forward function where you use the shape to extract the batch_size and sequence_length. If you look at the graph of the model created by torch.fx:

graph():
    %x : [num_users=3] = placeholder[target=x]
    %h0 : [num_users=1] = placeholder[target=h0]
    %getattr_1 : [num_users=1] = call_function[target=builtins.getattr](args = (%x, shape), kwargs = {})
    %getitem : [num_users=1] = call_function[target=operator.getitem](args = (%getattr_1, 0), kwargs = {})
    %getattr_2 : [num_users=1] = call_function[target=builtins.getattr](args = (%x, shape), kwargs = {})
    %getitem_1 : [num_users=0] = call_function[target=operator.getitem](args = (%getattr_2, 1), kwargs = {})
    %gru : [num_users=2] = call_module[target=gru](args = (%x, %h0), kwargs = {})
    %getitem_2 : [num_users=0] = call_function[target=operator.getitem](args = (%gru, 0), kwargs = {})
    %getitem_3 : [num_users=1] = call_function[target=operator.getitem](args = (%gru, 1), kwargs = {})
    %view : [num_users=1] = call_method[target=view](args = (%getitem_3, %getitem, -1), kwargs = {})
    %fc : [num_users=1] = call_module[target=fc](args = (%view,), kwargs = {})
    return fc

You see that these assignments get rendered as get_attr operations, which we do not support. The solution to that would be to just hard code those values. However, I see that once I do that, something else goes wrong which looks like an actual bug. I'll have a closer look on Monday.

[JanFSchulte]

Hi Iain,

taking a closer look now I realized that you are applying the Linear layer to the hidden state of the GRU. Unfortunately, returning the hidden state from the calculation is not something we support at the moment, we only return the output.