This is just another ELECTRA re-implementation, but on pytorch-lightning. Currently, there are several ELECTRA implementation, which have helped me in considerably to make my own, in particular, the implementations made by Richard Wang, Phil Wang and of course the official implementation by Google.
Although all the implementation mentioned above have a high quality. I want to do some multi-modal experiments using this model. So I need a bit more of control. In addition, I also want to play a bit training language models and ELECTRA is the only one (for now) that allows me to train interesting models using achievable computational power.