Stance Classification of Tweets using Transfer Learning

Applying transfer learning (using existing neural network architectures) to perform stance classification of Tweets as per the SemEval 2016 Stance Detection Task.

The methodology is described in detail in this Medium post and compared (in detail) the transfer learning approaches used.

For subtask A, the goal is to classify Tweets in response to a particular topic into one of three classes: Favor, Against and None. The provided notebooks attempt this using a technique in deep learning called transfer learning. While transfer learning has been ubiquitous throughout computer vision applications since the success of ImageNet, it is only since 2017-18 that significant progress has been made for transfer learning in NLP applications. There have been a string of interesting papers in 2018 that discuss the power of language models in natural language understanding and how they can be used to provide pre-trained representations of a language's syntax, which can be far more useful when training a neural network for previously unseen tasks.

Analysis Notebooks

See the included Jupyter notebooks for the stance classification workflow using ULMFit and the OpenAI transformer.

Method 1: ULMFiT

ulmfit.ipynb: (LSTM-based approach)

Method 2: OpenAI Transformer

transformer.ipynb: (Transformer-based approach)

Module Installation

The below sections highlight the installation steps for each approach used. Python 3.6+ and PyTorch 1.0.0 is used for all the work shown.

Set up virtual environment:

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

Once virtual environment has been set up, activate it for further development.

source venv/bin/activate

PyTorch requirements

Install the latest version of pytorch (1.0+) as shown below:

pip3 install -r pytorch-requirements.txt

ULMFit with the fastai framework

This utilizes the fastai framework (built on top of PyTorch) to perform stance classification.

The notebook ulmfit.ipynb uses v1 of fastai, which has been refactored for efficiency and updated to move forward with future PyTorch versions (1.0+).

Install fastai as shown below:

pip3 install fastai

spaCy language model

For tokenization, fastai uses the SpaCy library's English language model. This has to be downloaded manually:

python3 -m spacy download en 

Evaluation

To evaluate the F1 score as per the SemEval 2016 Task 6 guidelines, use the perl script given in data/eval/ as shown:

perl eval.pl -u

---------------------------
Usage:
perl eval.pl goldFile guessFile

goldFile:  file containing gold standards;
guessFile: file containing your prediction.

These two files have the same format:
ID<Tab>Target<Tab>Tweet<Tab>Stance
Only stance labels may be different between them!
---------------------------