A paper implementation and tutorial from scratch combining various great resources for implementing Transformers discussesd in Attention in All You Need Paper for German to English Translation
This repository combines the explanations of BenTrevett,Jay Alamar and paper authors into one place and shows the implementation of Transformers for Machine Translation from scratch.It can be used for tutorials and self-understanding.It is a long notebook so one can take time to understand. The task for which the transformer was trained on is German to English Translation.
To view the notebook incase it shows some error in viewing directly, please copy paste the notebook link to this site .[1] Attention is All You Need, Paper https://arxiv.org/pdf/1706.03762.pdf
[2] The Illustrated Transformer by Jay Alamar, https://jalammar.github.io/illustrated-transformer/
[3] What Does BERT Look At? An Analysis of BERT’s Attention, https://arxiv.org/pdf/1906.04341.pdf
[4] Bertology, https://huggingface.co/transformers/bertology.html
[5] The Annotated Transformer (PyTorch Implementation (some portions of the code might be obsolete now), http://nlp.seas.harvard.edu/2018/04/03/attention.html
[6] BenTrevett and jadore, https://github.com/jadore801120/attention-is-all-you-need-pytorch
[7] Advanced NLP, https://people.cs.umass.edu/~miyyer/cs685/