Conformer v0.1.0

[Rishit-dagli]

This is the initial release of Conformer and implements the Conformer (Convolution-augmented Transformer for Speech Recognition) as a subclassed TensorFlow model. Conformer achieves the best of both worlds (transformers for content-based global interactions and CNNs to exploit local features) by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way.

Classes

You would majorly want to follow the examples and use the ConformerConvModule or ConformerBlock classes.

• Attention: Implements attention mechanism as shown in the paper as a TensorFlow Keras layer
• Swish: The Swish activation function originally proposed by Ramachandran et al. in Searching for Activation Functions
• GLU: The GLU activation function originally proposed by Dauphin et al. in Language Modeling with Gated Convolutional Networks
• DepthwiseLayer: A modified depthwise convolutional layer
• Scale: Apply an operation and scale it by a certain value
• PreNorm: Normalize the activations of the previous layer for each given example in a batch independently and apply some function to it, implemented as a TensorFlow Keras Layer
• FeedForward: Create a FeedForward neural net with two Dense and 2 Dropout layers and Swish activation, implemented as a TensorFlow Keras Layer
• BatchNorm: Modified Conditional batchnorm layer
• ConformerConvModule: The class for a ConformerConvModule, the major contribution of this paper
• ConformerBlock: A complete ConformerBlock put together

---

This discussion was created from the release Conformer v0.1.0.