Research literature notes 🤓
Notes from papers I'm reading, ordered by topic and chronologically.
NLP
- What’s Going On in Neural Constituency Parsers? An Analysis, Gaddy et al., 2018 [Paper] [Notes] #nlp
- Two Methods for Domain Adaptation of Bilingual Tasks: Delightfully Simple and Broadly Applicable, Hangya et al., 2018 [Paper] [Notes] #nlp
- What do you learn from context? Probing for sentence structure in contextualized word representations, Tenney et al., 2019 [Paper] [Notes] #nlp
- BPE-Dropout: simple and effective subword regularization, Provilkov et al., 2019 [Paper] [Notes] #nlp
- From English To Foreign Languages: Transferring Pre-trained Language Models, Tran, 2020 [Paper] [Notes] #nlp
- Evaluating NLP models via contrast sets, Gardner et al., 2020 [Paper] [Notes] #nlp
- Byte Pair Encoding is Suboptimal for Language Model Pretraining, Bostrom et al., 2020 [Paper] [Notes] #nlp
- Translation artifacts in cross-lingual transfer learning, Artetxe et al., 2020 [Paper] [Notes] #nlp
- Weight poisoning attacks on pre-trained models, Kurita et al., 2020 [Paper] [Notes] #nlp
- SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings, Sabet et al., 2020 [Paper] [Notes] #nlp
- Experience Grounds Language, Bisk et al., 2020 [Paper] [Notes] #nlp #linguistics
- Beyond accuracy: behavioral testing of NLP models with CheckList, Ribeiro et al., 2020 [Paper] [Notes] #nlp
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes, Kiela et al., 2020 [Paper] [Notes] #nlp
- The Unstoppable Rise of Computational Linguistics in Deep Learning, Henderson, 2020 [Paper] [Notes] #nlp #linguistics
- Language (Technology) is Power: A Critical Survey of "Bias" in NLP, Blodgett et al., 2020 [Paper] [Notes] #nlp
- Representation Learning for Information Extraction from Form-like Documents, Majumder et al., 2020 [Paper] [Notes] #nlp
- Learning to tag OOV tokens by integrating contextual representation and background knowledge, He et al., 2020 [Paper] [Notes] #nlp
- It's not just size that matters, small language models are also few-shot learners, Schick and Schütze, 2020 [Paper] [Notes] #nlp
- Did you read the next episode? Using textual cues for predicting podcast popularity, Joshi et al., 2020 [Paper] [Notes] #nlp
- A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios, Hedderich et al., 2020 [Paper] [Notes] #nlp
- Challenges in Deploying Machine Learning: a Survey of Case Studies, Paleyes et al., 2020 [Paper] [Notes] #nlp
- Adapting Coreference Resolution to Twitter Conversations, Aktas et al., 2020 [Paper] [Notes] #nlp
- Learning from others' mistakes: avoiding dataset biases without modeilng them, Sanh et al., 2020 [Paper] [Notes] #nlp
- CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation, Clark et al., 2021 [Paper] [Notes] #nlp
Embeddings
- Semi-supervised sequence tagging with bidirectional language models, Peters et al., 2017 [Paper] [Notes] #nlp #embeddings
- Mimicking Word Embeddings using Subword RNNs, Pinter et al., 2017 [Paper] [Notes] #nlp #embeddings
- Deep contextualized word representations, Peters et al., 2018 [Paper] [Notes] #nlp #embeddings
- Linguistic Knowledge and Transferability of Contextual Representations, Liu et al., 2019 [Paper] [Notes] #nlp #embeddings
- Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates, Kudo, 2018 [Paper] [Notes] #nlp #embeddings
- Dissecting contextual word embeddings: architecture and representation, Peters et al., 2018 [Paper] [Notes] #nlp #embeddings
- BERT: Pre-training of deep bidirectional transformers for language understanding, Devlin et al., 2018 [Paper] [Notes] #nlp #embeddings
- Learning Semantic Representations for Novel Words: Leveraging Both Form and Context, Schick et al., 2018 [Paper] [Notes] #nlp #embeddings
- Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia, Yamada et al., 2018 [Paper] [Notes] #nlp #embeddings
- Rare Words: A Major Problem for Contextualized Embeddings and How to Fix it by Attentive Mimicking, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
- Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
- BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance, Schick et al., 2019 [Paper] [Notes] #nlp #embeddings
- BERT is Not a Knowledge Base (Yet): Factual Knowledge vs. Name-Based Reasoning in Unsupervised QA, Poerner et al., 2019 [Paper] [Notes] #nlp #embeddings
Architectures
- Conditional Random Fields: probabilistic models for segmenting and labeling sequence data, Lafferty et al, 2001 [Paper] [Notes] #nlp #architectures
- Bidirectional LSTM-CRF Models for sequence tagging, Huang et al., 2015 [Paper] [Notes] #nlp #architectures
- Neural Architectures for Named Entity Recognition, Lample et al., 2016 [Paper] [Notes] #nlp #architectures #NER
- Named Entity Recognition with Bidirectional LSTM-CNNs, Chiu et al., 2016 [Paper] [Notes] #nlp #architectures
- Attention is all you need, Vaswani et al., 2018 [Paper] [Notes] #nlp #architectures
- Reasoning with Sarcasm by Reading In-between, Tay et al., 2018 [Paper] [Notes] #sarcasm-detection #architectures
- XLNet: generalized autoregressive pretraining for language understanding, Yang et al., 2019 [Paper] [Notes] #nlp #architectures
- R-Transformer: Recurrent Neural Network Enhanced Transformer, Wang et al., 2019 [Paper] [Notes] #nlp #architectures
- Generalization through Memorization: Nearest Neighbor Language Models, Khandelwal et al., 2019 [Paper] [Notes] #nlp #architectures
- Single Headed Attention RNN: Stop Thinking With Your Head, Merity, 2019 [Paper] [Notes] #nlp #architectures
- A Transformer-based approach to Irony and Sarcasm detection, Potamias et al., 2019 [Paper] [Notes] #sarcasm-detection #architecture
- ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, Qi et al., 2020 [Paper] [Notes] #nlp #architectures
- Pre-trained Models for Natural Language Processing: A Survey, Qiu et al., 2020 [Paper] [Notes] #nlp #architectures
- SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, Iandola et al., 2020 [Paper] [Notes] #nlp #architectures #computer-vision
- A comparison of LSTM and BERT for small corpus, Ezen-Can, 2020 [Paper] [Notes] #nlp #architectures
Frameworks
- Flair: an easy-to-use framework for stat-of-the-art NLP [Paper] [Notes] #nlp #frameworks
- HuggingFace's Transformers: State-of-the-art Natural Language Processing, Wolf et al., 2019 [Paper] [Notes] #nlp #frameworks
- Selective Brain Damage: Measuring the Disparate Impact of Model Pruning, Hooker et al., 2019 [Paper] [Notes] #frameworks
- Why should we add early exits to neural networks?, Scardapane et al., 2020 [Paper] [Notes] #frameworks
Datasets
- Introduction to the CoNLL-2003 shared task: language-independent named entity recognition, Sang et al., 2003 [Paper] [Notes] #nlp #datasets
- Datasheets for datasets, Gebru et al., 2018 [Paper] [Notes] #nlp #datasets
- SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference, Zellers et al., 2018 [Paper] [Notes] #nlp #datasets
- A Named Entity Recognition Shootout for German, Riedl and Padó, 2018 [Paper] [Notes] #nlp #NER #datasets
- Probing Neural Network Comprehension of Natural Language Arguments, Nivel and Kao, 2019 [Paper] [Notes] #nlp #datasets
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference., McCoy et al., 2019 [Paper] [Notes] #nlp #linguistics #datasets
- UR-FUNNY: A Multimodal Language Dataset for Understanding Humor, Hasan et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
- HellaSwag: Can a Machine Really Finish Your Sentence?, Zellers et al., 2019 [Paper] [Notes] #nlp #datasets
- Sentiment analysis is not solved! Assessing and probing sentiment classification, Barnes et al., 2019 [Paper] [Notes] #nlp #datasets
- Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model, Cai et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
- Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper), Castro et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
- iSarcasm: A Dataset of Intended Sarcasm, Oprea et al., 2019 [Paper] [Notes] #datasets #sarcasm-detection
- Lessons from archives: strategies for collecting sociocultural data in machine learning, Seo Jo and Gebru, 2019 [Paper] [Notes] #nlp #datasets
- BERTweet: A pre-trained language model for English Tweets, Nguyen et al., 2020 [Paper] [Notes] #nlp #datasets
- GAIA: a fine-grained multimedia knowlege extraction system, Li et al., 2020 [Paper, [Notes] #nlp #datasets
- It's morphin' time! Combating linguistic discrimination with inflectional perturbations, Tan et al., 2020 [Paper, [Notes] #nlp #datasets
- Reactive Supervision: A New method for Collecting Sarcasm Data, Shmueli et al, 2020 [Paper] [Notes] #datasets #sarcasm-detection
NER
- Introduction to the CoNLL-2003 shared task: language-independent named entity recognition, Sang et al., 2003 [Paper] [Notes] #nlp #datasets #NER
- Neural Architectures for Named Entity Recognition, Lample et al., 2016 [Paper] [Notes] #nlp #architectures #NER
- Named Entity Recognition with Bidirectional LSTM-CNNs, Chiu et al., 2016 [Paper] [Notes] #nlp #architectures #NER
- Towards Robust Named Entity Recognition for Historic German, Schweter et al., 2019 [Paper] [Notes] #nlp #NER
- A Named Entity Recognition Shootout for German, Riedl and Padó, 2018 [Paper] [Notes] #nlp #NER #datasets
Sarcasm detection
- Sarcasm Detection on Twitter: A Behavioral Modeling Approach, Rajadesingan et al., 2015 [Paper] [Notes] #sarcasm-detection
- Contextualized Sarcasm Detection on Twitter, Bamman and Smith, 2015 [Paper] [Notes] #sarcasm-detection
- Harnessing Context Incongruity for Sarcasm Detection, Joshi et al., 2015 [Paper] [Notes] #linguistics #sarcasm-detection
- Automatic Sarcasm Detection: A Survey, Joshi et al., 2017 [Paper] [Notes] #sarcasm-detection
- Detecting Sarcasm is Extremely Easy ;-), Parde and Nielsen, 2018 [Paper] [Notes] #sarcasm-detection
- CASCADE: Contextual Sarcasm Detection in Online Discussion Forums, Hazarika et al., 2018 [Paper] [Notes] #sarcasm-detection
- Reasoning with Sarcasm by Reading In-between, Tay et al., 2018 [Paper] [Notes] #sarcasm-detection #architectures
- Tweet Irony Detection with Densely Connected LSTM and Multi-task Learning, Wu et al., 2018 [Paper] [Notes] #sarcasm-detection
- UR-FUNNY: A Multimodal Language Dataset for Understanding Humor, Hasan et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
- Exploring Author Context for Detecting Intended vs Perceived Sarcasm, Oprea and Magdy, 2019 [Paper] [Notes] #sarcasm-detection
- Towards Multimodal Sarcasm Detection (An Obviously Perfect Paper), Castro et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
- Multi-Modal Sarcasm Detection in Twitter with Hierarchical Fusion Model, Cai et al., 2019 [Paper] [Notes] #sarcasm-detection #datasets
- A2Text-Net: A Novel Deep Neural Network for Sarcasm Detection, Liu et al., 2019 [Paper] [Notes] #sarcasm-detection
- Sarcasm detection in tweets, Rajagopalan et al., 2019 [Paper] [Notes] #sarcasm-detection
- A Transformer-based approach to Irony and Sarcasm detection, Potamias et al., 2019 [Paper] [Notes] #sarcasm-detection #architecture
- Deep and dense sarcasm detection, Pelser et al., 2019 [Paper] [Notes] #sarcasm-detection
- iSarcasm: A Dataset of Intended Sarcasm, Oprea et al., 2019 [Paper] [Notes] #datasets #sarcasm-detection
- Reactive Supervision: A New method for Collecting Sarcasm Data, Shmueli et al, 2020 [Paper] [Notes] #datasets #sarcasm-detection
Text summarization
- Evaluating the Factual Consistency of Abstractive Text Summarization, Kryscinski et al., 2019 [Paper] [Notes] #nlp #text-summarization
- TLDR: extreme summarization of scientific documents, Cachola et al, 2020 [Paper] [Notes] #nlp #text-summarization
- A survey on text simplification, Sikka and Mago, 2020 [Paper] [Notes] #nlp #text-summarization
Machine translation
- Unsupervised Tokenization for Machine Translation, Chung and Gildea, 2009 [Paper] [Notes] #nlp #machine-translation
- Neural Machine Translation of Rare Words with Subword Units, Sennrich et al., 2015 [Paper] [Notes] #nlp #machine-translation
- Unsupervised neural machine translation, Artetxe et al., 2017 [Paper] [Notes] #nlp #machine-translation
- How Much Does Tokenization Affect Neural Machine Translation? Domingo et al., 2018 [Paper] [Notes] #nlp #machine-translation
- Reusing a Pretrained Language Model on Languages with Limited Corpora for Unsupervised NMT, Chronopoulou et al., 2020 [Paper] [Notes] #nlp #machine-translation
Text generation
Reinforcement learning
- Theory of Minds: Understanding Behavior in Groups Through Inverse Planning, Shum et al., 2019 [Paper] [Notes] #reinforcement-learning #social-sciences
- The Hanabi Challenge: A New Frontier for AI Research, Bard et al., 2019 [Paper] [Notes] #reinforcement-learning
- Mastering Atari, Go, Chess and Shogi by Planning with a learned model, Schrittwieser et al., 2019 [Paper] [Notes] #reinforcement-learning
- Language as a cognitive tool to imagine goals in curiosity-driven exploration, Colas et al., 2020 [Paper] [Notes] #reinforcement-learning
- Planning to Explore via Self-Supervised World Models, Sekar et al., 2020 [Paper] [Notes] #reinforcement-learning
Computer vision
- Cubic Stylization, Derek Liu and Jacobson, 2019 [Paper] [Notes] #computer-vision
- SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, Iandola et al., 2020 [Paper] [Notes] #nlp #computer-vision
Machine learning
- Gender shades: intersectional accuracy disparities in commercial gender classification, Buolamwini and Gebru, 2018 [Paper] [Notes] #machine-learning
- Interpretable Machine Learning - A Brief History, State-of-the-Art and Challenges, Molnar et al., 2020 [Paper] [Notes] #machine-learning
Audio
- End-to-End Adversarial Text-to-Speech, Donahue et al., 2020 [Paper] [Notes] #audio
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations, Baevski et al., 2020 [Paper] [Notes] #audio
- Large-scale multilingual audio visual dubbing, Yang et al., 2020 [Paper] [Notes] #audio
Linguistics
- Moving beyond the plateau: from lower-intermediate to upper-intermediate, Richards, 2015 [Paper] [Notes] #linguistics
- Harnessing Context Incongruity for Sarcasm Detection, Joshi et al., 2015 [Paper] [Notes] #linguistics #sarcasm-detection
- A Trainable Spaced Repetition Model for Language Learning, Settles and Meeder, 2016 [Paper] [Notes] #linguistics
- Targeted synctactic evaluation of language models, Marvin and Linzen, 2018 [Paper] [Notes] #nlp #linguistics
- Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference., McCoy et al., 2019 [Paper] [Notes] #nlp #linguistics #datasets
- Language Models as Knowledge Bases?, Petroni et al., 2019 [Paper] [Notes] #nlp #linguistics
- Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche, Coupé et al., 2019 [Paper] [Notes] #linguistics #social-sciences
- My English sounds better than yours: Second language learners perceive their own accent as better than that of their peers, Mittlerer et al., 2020 [Paper] [Notes] #linguistics
- Experience Grounds Language, Bisk et al., 2020 [Paper] [Notes] #nlp #linguistics
- The Unstoppable Rise of Computational Linguistics in Deep Learning, Henderson, 2020 [Paper] [Notes] #nlp #linguistics
- Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data, Bender et al., 2020 [Paper] [Notes] #nlp #linguistics
Social sciences
- Antisocial Behavior in Online Discussion Communities, Cheng et al., 2015 [Paper] [Notes] #social-sciences
- How much does education improve intelligence? A meta-analysis, Ritchie et al., 2017 [Paper] [Notes] #social-sciences
- Theory of Minds: Understanding Behavior in Groups Through Inverse Planning, Shum et al., 2019 [Paper] [Notes] #reinforcement-learning #social-sciences
- Fake news game confers psychological resistance against online misinformation, Roozenbeek and van der Linden, 2019 [Paper] [Notes] #social-sciences #humanities
- Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche, Coupé et al., 2019 [Paper] [Notes] #linguistics #social-sciences
- Kids these days: Why the youth of today seem lacking, Protzko and Schooler, 2019 [Paper] [Notes] #social-sciences
Humanities
- Fake news game confers psychological resistance against online misinformation, Roozenbeek and van der Linden, 2019 [Paper] [Notes] #social-sciences #humanities
Economics
- Why do people stay poor? Balboni et al., 2020 [Paper] [Notes] #economics
Physics
Neuroscience
- A deep learning framework for neuroscience, Richard et al., 2019 [Paper] [Notes] #neuroscience
Algorithms
- Replace or Retrieve Keywords In Documents At Scale, Singh, 2017 [Paper] [Notes] #algorithms