This repository contains code for an extension to the Transformer via the integration of neuromodulation. The goal is to explore neuromodulation and its potential impacts on generalisation in QA. The architecture is coded in TensorFlow and can be found in the models folder.
Code to load all datasets is found in the load_datasets folder. We utilise the Colossol Cleaned Crawled Corpus (C4) for pre-training our model; LAMBADA, WikiText, and PTB to test our pre-trained model's language modelling capabilities; performance in QA is measured on ARC, BoolQ, CommonsenseQA, DROP, MCTest, NarrativeQA, OBQA, PIQA, Quoref, ROPES, SIQA, WG, RACE, and SQuADv2.
All training files can be found in the training folder.