**Update**: working on a Pytorch Version!!
- Thank you for asking! I'll update a pytorch version! Hopefully within this month (Oct 2021)
**Update**: On the Interpretability of Discrete Set Representations: A Bayesian Perspective
- The discussion about the identifiability and interpretability of the LBOW model in the unsupervised setting.
- Work in progress
The Latent Bag of Words Model
Implementation of Yao Fu, Yansong Feng and John Cunningham, Paraphrase Generation with Latent Bag of Words. NeurIPS 2019. paper
As is shown in the above example, given a source sentence, our model first infers the neighbor words of each source words, then sample a bag of words from the neighbors, then generate the paraphrase based on the sampled words
For more background about deep generative models for natural language processing, see the DGM4NLP journal list.
Reproduce
mkdir models
mkdir outputs
cd src
python3 main.py
# quicker start
python3 main.py --batch_size=5 --train_print_interval=10
# Monitor training:
loss: 9.1796 -- total loss
enc_loss: 3.1693 -- BOW NLL loss
dec_loss: 6.0103 -- Decoder loss
precision_confident: 0.4794 -- BOW precision on confident = most confident word neighbors
recall_confident: 0.1727 -- BOW recall on confident = most confident word neighbors
precision_topk: 0.1186 -- BOW percision on topk = tok predicted word neighbors
recall_topk: 0.2387 -- BOW recall on topk = tok predicted word neighborsMay need to install nltk stopwords first, just follow the prompt
Data
We use the MSCOCO(17) dataset and the Quora dataset. The Quora dataset is provided in the data/ folder. The MSCOCO dataset can be downloaded from its offical website
Code Structure
The core implementation is in the following files:
- config.py
- main.py
- controller.py
- latent_bow.py
Others
There are certain codes about testing the Wikibio dataset. These part of the code is not included in the paper, its just for tesing the data-to-text task. So the published part might be incomplete. If you do want to extend the model to data-to-text, feel free to contact me.
The rouge evaluation is from here: https://pypi.org/project/py-rouge/. There is also a google implementation recently: https://github.com/google-research/google-research/tree/master/rouge.
The main_test.py is not for testing the model, it is for debugging in the ipython terminal.
Embarrassingly, I cannot replicate the results in Pytorch ... I moved to Pytorch after this project and there are still many things to understand (as a many-year tfer). It seems that the pytorch LSTM does not support residual connections, which results in degraded performance when I changed the number of layers to be 2. If you happen to find a solution, or find something close, do contact me.