Adversarial Training Methods for Semi-Supervised Text Classification
Code for Adversarial Training Methods for Semi-Supervised Text Classification
This code reproduce the [Miyato et al., 2017] with Chainer.
Setup envirment
Please install Chainer and Cupy.
You can set up the environment easily with this Setup.md.
Download Pretrain Model
Please download pre-trained model.
$ wget http://sato-motoki.com/research/vat/imdb_pretrained_lm.model
Result
| Model | Error Rate |
|---|---|
| Baseline [Miyato et al., 2017] | 7.39 |
| Baseline (Our code) | 6.62 |
| Adversarial [Miyato et al., 2017] | 6.21 |
| Adversarial Training (Our code) | 6.35 |
| Virtual Adversarial Training [Tensorflow code] | 6.40 |
| Virtual Adversarial Training [Miyato et al., 2017] | 5.91 |
| Virtual Adversarial Training (Our code) | 5.82 |
Run
Pretrain
$ python -u pretrain.py -g 0 --layer 1 --dataset imdb --bproplen 100 --batchsize 32 --out results_imdb_adaptive --adaptive-softmax
Note that this command takes about 30 hours with single GPU.
Train (VAT: Semi-supervised setting)
$ python train.py --gpu=0 --n_epoch=30 --batchsize 32 --save_name=imdb_model_vat --lower=0 --use_adv=0 --xi_var=5.0 --use_unlabled=1 --alpha=0.001 --alpha_decay=0.9998 --min_count=1 --ignore_unk=1 --pretrained_model imdb_pretrained_lm.model --use_exp_decay=1 --clip=5.0 --batchsize_semi 96 --use_semi_data 1
Note that this command takes about 8 hours with single GPU.
Train (Adversarial Training: Supervised setting)
$ python train.py --gpu=0 --n_epoch=30 --batchsize 32 --save_name=imdb_model_adv --lower=0 --use_adv=1 --xi_var=5.0 --use_unlabled=1 --alpha=0.001 --alpha_decay=0.9998 --min_count=1 --ignore_unk=1 --pretrained_model imdb_pretrained_lm.model --use_exp_decay=1 --clip=5.0
Note that this command takes about 6 hours with single GPU.
Authors
We thank Takeru Miyato (@takerum) who suggested that we reproduce the result of a [Miyato et al., 2017].
- Code author: @aonotas
- Thanks for Adaptive Softmax implementation: @soskek Adaptive Softmax: https://github.com/soskek/efficient_softmax
Reference
[Miyato et al., 2017]: Takeru Miyato, Andrew M. Dai and Ian Goodfellow
Adversarial Training Methods for Semi-Supervised Text Classification.
International Conference on Learning Representation (ICLR), 2017