hdDeepLearningStudy
Papers,code etc for deep learning study group
See group discord - https://discord.gg/HuWVmMgmqS
zoom link - On the meetup page
meeting time - 6:30 pm California time
Tuesday, Sept 5, 2023
https://arxiv.org/pdf/2308.08708.pdf - Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Tuesday, August 29, 2023
paper: https://arxiv.org/pdf/2307.15936.pdf - A Theory for Emergence of Complex Skills in Language Models and video
youtube: https://www.youtube.com/watch?v=0D23NeBjCeQ
Tuesday, August 22, 2023
Paper: https://arxiv.org/pdf/2206.04843.pdf -- Neural Laplace: Learning diverse classes of differential equations in the Laplace domain
Slides and video from ICML 2022: https://icml.cc/virtual/2022/oral/16728
Wednesday, August 16, 2023
paper: https://arxiv.org/abs/2308.03296 - Studying Large Language Model Generalization with Influence Functions
blog: https://www.anthropic.com/index/influence-functions
Wednesday, August 9, 2023
paper: Music Generations https://arxiv.org/pdf/2306.05284.pdf
blog: https://about.fb.com/news/2023/08/audiocraft-generative-ai-for-music-and-audio/
blog: https://ai.meta.com/blog/audiocraft-musicgen-audiogen-encodec-generative-ai-audio/
Wednesday, August 2, 2023
paper: https://arxiv.org/abs/2205.10343 Towards Understanding Grokking: An Effective Theory of Representation Learning
blog: https://ericjmichaud.com/grokking-squared/
blog: https://www.beren.io/2022-01-11-Grokking-Grokking/
blog: https://www.beren.io/2022-04-17-Understanding_Overparametrized_Generalization/
Wednesday, July 26, 2023
paper: Mixture of experts (similar to chatGPT4): https://arxiv.org/abs/2305.14705
blog: Mixture-of-Experts with Expert Choice Routing -
https://ai.googleblog.com/2022/11/mixture-of-experts-with-expert-choice.html
blot: Introducing Pathways: A next-generation AI architecture
https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/
Wednesday, July 19, 2023
We're going to cover Chapter 16 Deep Networks for Classification from the following book:
https://book-wright-ma.github.io/Book-WM-20210422.pdf - High dimensional Data Analysis with Low Dimensional Models
blog: https://terrytao.wordpress.com/2007/04/13/compressed-sensing-and-single-pixel-cameras/#more-25
Wednesday, July 12, 2023
We're going to cover the 4th chapter of this book.
https://book-wright-ma.github.io/Book-WM-20210422.pdf - High dimensional Data Analysis with Low Dimensional Models
Wednesday, July 5, 2023
We're going to cover the 1st chapter of this book.
https://book-wright-ma.github.io/Book-WM-20210422.pdf - High dimensional Data Analysis with Low Dimensional Models
Blog: https://terrytao.wordpress.com/2007/04/13/compressed-sensing-and-single-pixel-cameras/#more-25
Wednesday, June 28, 2023
paper: https://arxiv.org/pdf/2305.17126.pdf - Large Language Models as Tool Makers
youtube: https://www.youtube.com/watch?v=qWI1AJ2nSDY
youtube: https://www.youtube.com/watch?v=KXlPzMRTfMk
youtube: https://www.youtube.com/watch?v=srDVNbxPgZI
Wednesday, June 21, 2023
Consciousness as a Memory System https://pubmed.ncbi.nlm.nih.gov/36178498/
Wednesday, June 14, 2023
https://arxiv.org/abs/1804.08838
Blog: https://www.uber.com/blog/intrinsic-dimension/
more good stuff on intrinsic dimension:
Nature paper: https://www.nature.com/articles/s41598-017-11873-y
Wikipedia: https://en.wikipedia.org/wiki/Intrinsic_dimension
Application - Yann LeCun at 57:15 on does text fully represent world model?
https://www.youtube.com/watch?v=SGzMElJ11Cc
vs. differing view from Ilya Sutskever at 15:30
https://www.youtube.com/watch?v=SjhIlw3Iffs
Applying intrinsic dimension to scaling laws in training / loss:
https://jmlr.csail.mit.edu/papers/volume23/20-1111/20-1111.pdf
https://arxiv.org/abs/2102.06701
Wednesday, June 7, 2023
Paper: https://arxiv.org/pdf/2305.16291.pdf
Twit: Tweet with nice overview by author https://twitter.com/DrJimFan/status/1662117784023883777
Code: https://github.com/MineDojo/Voyager
website: https://voyager.minedojo.org/
Wednesday, May 31, 2023
paper: https://arxiv.org/pdf/2203.15556.pdf - Training Compute-Optimal Large Language Models
blog: https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications
blog: https://www.harmdevries.com/post/model-size-vs-compute-overhead/
google blog: https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html
Wednesday, May 24, 2023
paper: https://arxiv.org/abs/2212.09720 - The case for 4-bit precision: k-bit Inference Scaling Laws
paper: https://arxiv.org/pdf/2210.17323.pdf - GPTQ: ACCURATE POST-TRAINING QUANTIZATION FOR GENERATIVE PRE-TRAINED TRANSFORMERS
Wednesday, May 17, 2023
paper: https://arxiv.org/pdf/2106.09685.pdf - LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
Wednesday, May 10, 2023
paper: https://arxiv.org/pdf/2210.03629.pdf - REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS
paper: https://www.pinecone.io/learn/locality-sensitive-hashing/
Wednesday, May 3, 2023
paper: https://arxiv.org/pdf/2201.11903.pdf - Chain of thought prompting elicits reasoning in large language models.
paper: https://arxiv.org/pdf/2210.03629.pdf - REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS
paper: https://www.pinecone.io/learn/locality-sensitive-hashing/
Wednesday, Apr 26, 2023
https://python.langchain.com/en/latest/modules/agents.html
https://arxiv.org/pdf/2210.03629.pdf - REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS
https://www.pinecone.io/learn/locality-sensitive-hashing/
Wednesday, Apr 19, 2023
Blog: https://yoheinakajima.com/task-driven-autonomous-agent-utilizing-gpt-4-pinecone-and-langchain-for-diverse-applications/
Code: https://github.com/hwchase17/langchain
Wednesday, Apr 12, 2023
Paper: Eliciting Latent Predictions from Transformers with the Tuned Lens https://arxiv.org/abs/2303.08112
Wednesday, Apr 5, 2023
Paper: https://openreview.net/pdf?id=lMMaNf6oxKM - Recipe for a General, Powerful, Scalable Graph Transformer
youtube: https://www.youtube.com/watch?v=DiLSCReBaTg
Wednesday, Mar 29, 2023
Paper: https://proceedings.neurips.cc/paper/2021/hash/f1c1592588411002af340cbaedd6fc33-Abstract.html - Do Transformers Really Perform Badly for Graph Representation?
video: https://www.youtube.com/watch?v=FKuQpPIRjLk - review by authors
video: https://www.youtube.com/watch?v=xQ5ltOOxoFg
Wednesday, Mar 22, 2023
Paper: https://arxiv.org/abs/2212.07359 - Post-hoc Uncertainty Learning using a Dirichlet Meta-Model
youtube: https://www.youtube.com/watch?v=nE8XJ1f0zO0
Wednesday, Mar 15, 2023
Paper: https://arxiv.org/abs/2202.05262 - Locating and Editing Factual Associations in GPT
blog: https://rome.baulab.info/
Yannic video: https://www.youtube.com/watch?v=_NMQyOu2HTo
Wednesday, Mar 8, 2023
Paper: Human-Timescale Adaptation in an Open-Ended Task Space: https://arxiv.org/pdf/2301.07608.pdf
https://www.youtube.com/watch?v=A2hOWShiYoM
https://sites.google.com/view/adaptive-agent/
Wednesday, Mar 1, 2023
Paper: Toolformer: Language Models Can Teach Themselves to Use Tools: https://arxiv.org/abs/2302.04761
Wednesday, Feb 22, 2023
Paper: https://arxiv.org/pdf/2203.02155.pdf - Training language models to follow instructions with human feedback
Wednesday, Feb 15, 2023
Paper: https://arxiv.org/pdf/2111.15664.pdf - OCR-free Document Understanding Transformer
Wednesday, Feb 8, 2023
Paper: https://arxiv.org/abs/2205.06175 - A generalist agent - Gato
YouTube: Eden Mayer https://www.youtube.com/watch?v=wSQJZHfAg18
YouTube - Jay Alamar https://www.youtube.com/watch?v=kT6DYKgWNHg
YouTube - Lex Fridman and Oriol Vinyals on How Gato Works https://www.youtube.com/watch?v=vwB9zO2h9j0
Overview - main site on Gato at Deepmind https://www.deepmind.com/publications/a-generalist-agent
blog review - https://arshren.medium.com/deep-minds-generalist-agent-gato-209969e12782
Wednesday, Feb 1, 2023
Paper: https://openreview.net/pdf?id=M95oDwJXayG - ADDRESSING PARAMETER CHOICE ISSUES IN UNSUPERVISED DOMAIN ADAPTATION BY AGGREGATION
Wednesday, Jan 25, 2023
Paper: https://arxiv.org/pdf/2301.04104v1.pdf - Mastering Diverse Domains through World Models
Blog: https://danijar.com/project/dreamerv3/
YouTube: https://www.youtube.com/watch?v=vfpZu0R1s1Y
Wednesday, Jan 18, 2023
Paper: https://arxiv.org/abs/2212.04089 - Composable NN: Editing Models With Task Arithmetic
Wednesday, Jan 11, 2023
Paper: https://arxiv.org/pdf/1707.06690.pdf - DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning
Wednesday, Jan 4, 2023
Paper: https://arxiv.org/abs/2212.04458 - GENERAL-PURPOSE IN-CONTEXT LEARNING BY META-LEARNING TRANSFORMERS
Wednesday, Dec 21, 2022
paper: https://arxiv.org/pdf/2209.04836.pdf - GIT RE-BASIN: MERGING MODELS MODULO PERMUTATION SYMMETRIES
Wednesday, Dec 14, 2022
paper: https://arxiv.org/abs/2012.09855 - Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image
blog: https://infinite-nature.github.io/
Wednesday, Dec 7, 2022
Paper: https://arxiv.org/abs/2206.00364 - Elucidating the Design Space of Diffusion-Based Generative Models
video: https://www.youtube.com/watch?v=OYiQctx7kDE
Wednesday, Nov 30, 2022
paper: https://arxiv.org/pdf/2206.10991.pdf - Graph Neural Networks as Gradient Flows: understanding graph convolutions via energy
youtube (author): https://www.youtube.com/watch?v=sgTTtmwOMgE
youtube: https://www.youtube.com/watch?v=hmI4C6AodEQ
Wednesday, Nov 16, 2022
paper: https://www.pnas.org/doi/full/10.1073/pnas.2016239118
video: https://slideslive.com/38942412/biological-structure-and-function-emerge-from-scaling-unsupervised-learning-to-250-million-protein-sequences
Wednesday, Nov 9, 2022
paper: https://arxiv.org/pdf/2209.11178.pdf - Poisson Flow Generative Models
Wednesday, Nov 2, 2022
paper: https://arxiv.org/pdf/2209.12892.pdf - LEARNING TO LEARN WITH GENERATIVE MODELS OF NEURAL NETWORK CHECKPOINTS
blog: https://www.marktechpost.com/2022/10/21/latest-machine-learning-research-at-uc-berkeley-proposes-a-way-to-design-a-learned-optimizer-using-generative-models-of-neural-network-checkpoints/
author blog: https://www.wpeebles.com/Gpt.html
Wednesday, Oct 26, 2022
paper: Cellular automata as convolutional neural networks https://arxiv.org/pdf/1809.02942.pdf
survey: Collective Intelligence for Deep Learning: A Survey of Recent Developments https://arxiv.org/abs/2111.14377
demo: Self-classifying MNIST Digits https://distill.pub/2020/selforg/mnist/
Wednesday, Oct 19, 2022
paper: https://proceedings.mlr.press/v162/zhu22c/zhu22c.pdf - Neural-Symbolic Models for Logical Queries on Knowledge Graphs
Wednesday, Oct 12, 2022
paper: https://arxiv.org/pdf/2206.02768.pdf - The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization
Wednesday, Oct 5, 2022
paper: https://papers.nips.cc/paper/2019/file/952285b9b7e7a1be5aa7849f32ffff05-Paper.pdf - Legendre Memory Units: Continuous-Time
Wednesday, Sept 28, 2022
paper: https://arxiv.org/pdf/2208.01618.pdf - An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
githup.io: https://textual-inversion.github.io/
YouTube https://www.youtube.com/watch?v=f3oXa7_SYek
Wednesday, Sept 21, 2022
paper: https://arxiv.org/pdf/2205.14415.pdf - Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting
Wednesday, Sept 14, 2022
paper: https://arxiv.org/abs/2110.02402 - Language Modeling using LMUs: 10x Better Data Efficiency or Improved Scaling Compared to Transformers
youtube vid: https://www.youtube.com/watch?v=8t64QaTdBcU
Wednesday, August 31, 2022
Paper: HOW NEURAL NETWORKS EXTRAPOLATE: FROM FEEDFORWARD TO GRAPH NEURAL NETWORKS - https://arxiv.org/pdf/2009.11848.pdf
Wednesday, August 24, 2022
Paper: Masked Siamese Networks for Label-Efficient Learning - https://arxiv.org/abs/2204.07141
Wednesday, August 17, 2022
Paper: Principle of Maximal Coding Rate Reduction https://arxiv.org/abs/2006.08558
ReduNet: https://arxiv.org/pdf/2105.10446.pdf
Github: https://github.com/ryanchankh/mcr2
Wednesday, August 10, 2022
Paper: On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence https://arxiv.org/abs/2207.04630
Background: On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence https://arxiv.org/abs/2207.04630
Background: https://www.youtube.com/watch?v=OIVcfZeR1CE youtube by author
Background: https://cmsa.fas.harvard.edu/wp-content/uploads/2021/04/Deep_Networks_from_First_Principles.pdf - slides by author
Wednesday, August 3, 2022
Paper: Data Distributional Properties Drive Emergent In-Context Learning in Transformers https://arxiv.org/pdf/2205.05055.pdf
Wednesday, July 27, 2022
Paper: A Mathematical Framework for Transformer Circuits https://transformer-circuits.pub/2021/framework/index.html#model-simplifications
Wednesday, July 20, 2022
Paper: A Mathematical Framework for Transformer Circuits https://transformer-circuits.pub/2021/framework/index.html#model-simplifications
Wednesday, July 13, 2022
Paper: https://arxiv.org/abs/2001.08361 - Scaling Laws for Neural Language Models
Blog: https://medium.com/nlplanet/two-minutes-nlp-scaling-laws-for-neural-language-models-add6061aece7
Wednesday, July 6, 2022
Paper: https://arxiv.org/abs/2206.11795 - Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos
https://github.com/openai/Video-Pre-Training
Yannic Review: https://www.youtube.com/watch?v=oz5yZc9ULAc
Wednesday, June 29, 2022
Paper: https://arxiv.org/pdf/2110.00966.pdf - Translating Images into Maps
Wednesday, June 22, 2022
Paper: https://arxiv.org/abs/2205.09665 - Automated Crossword Solving
Wednesday, June 15, 2022
Paper: https://arxiv.org/pdf/2205.10824.pdf - ReLU Fields: The Little Non-linearity That Could
Wednesday, June 8, 2022
Paper: https://arxiv.org/abs/2102.06810 - Understanding Self-Supervised Learning Dynamics without Contrastive Pairs
Wednesday, June 1, 2022
Paper: https://arxiv.org/pdf/2205.06175.pdf - A Generalist Agent
Blog: https://www.deepmind.com/publications/a-generalist-agent
Wednesday, May 25, 2022
https://arxiv.org/pdf/2202.05780.pdf - A Modern Self-Referential Weight Matrix That Learns to Modify Itself
Wednesday, May 18, 2022
https://openreview.net/pdf?id=M752z9FKJP - LEARNING STRIDES IN CONVOLUTIONAL NEURAL NETWORKS
Wednesday, May 11, 2022
https://openreview.net/pdf?id=b-ny3x071E5 - BOOTSTRAPPED META-LEARNING
Wednesday, May 4, 2022
https://arxiv.org/abs/2202.06991 - Transformer Memory as a Differentiable Search Index
https://www.youtube.com/watch?v=C7mUYocWdG0 - Yannic author interview
https://www.youtube.com/watch?v=qlB0TPBQ7YY - Yannic on Transformer paper
Wednesday, April 27, 2022
https://arxiv.org/abs/2204.06125 - Hierarchical Text-Conditional Image Generation with CLIP Latents
https://openai.com/dall-e-2/ - OpenAI blog
https://www.youtube.com/watch?v=j4xgkjWlfL4 - yannic video
Wednesday, April 20, 2022
https://arxiv.org/pdf/2103.00020.pdf - Learning Transferable Visual Models From Natural Language Supervision
https://www.youtube.com/watch?v=1LUWWAnK_Ks
https://www.youtube.com/watch?v=3X3EY2Fgp3g
Wednesday, April 13, 2022
https://arxiv.org/pdf/2110.13985.pdf - Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers
Wednesday, April 6, 2022
https://arxiv.org/pdf/2202.00666.pdf - Typical Decoding for Natural Language Generation
https://www.youtube.com/watch?v=AvHLJqtmQkE
Wednesday, March 30, 2022
https://arxiv.org/pdf/2105.04906.pdf - VICREG: VARIANCE-INVARIANCE-COVARIANCE REGULARIZATION FOR SELF-SUPERVISED LEARNING
https://www.youtube.com/watch?v=MzKDNmOJ67Q
Wednesday, March 23, 2022
https://openreview.net/forum?id=4orlVaC95Bo - Task-Agnostic Undesirable Feature Deactivation Using Out-of-Distribution Data
Wednesday, March 16, 2022
https://arxiv.org/abs/2203.03466 - Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
https://www.youtube.com/watch?v=MNOJQINH-qw
Wednesday, March 9, 2022
https://arxiv.org/abs/2201.12122 - Can Wikipedia Help Offline Reinforcement Learning?
Yannic's talk on this,
https://www.youtube.com/watch?v=XHGh19Hbx48
and he also has a followon video interview with the authors
https://www.youtube.com/watch?v=FNDVy_BR8aA
Wednesday, March 2, 2022 -
https://arxiv.org/pdf/2107.03342.pdf - A Survey of Uncertainty in Deep Neural Networks
Wednesday, February 23, 2022 -
https://arxiv.org/pdf/2201.08239v2.pdf - LaMDA: Language Models for Dialog Applications
Wednesday, February 16, 2022 -
https://openreview.net/pdf?id=TrjbxzRcnf- MEMORIZING TRANSFORMERS
Wednesday, February 9, 2022 -
https://arxiv.org/pdf/2106.07644.pdf - A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip
Wednesday, February 2, 2022 -
https://arxiv.org/pdf/2108.08052.pdf - Moser Flow: Divergence-based Generative Modeling on Manifolds
Wednesday, January 26, 2022 -
https://dylandoblar.github.io/noether-networks/ - Noether Networks: meta-learning useful conserved quantities
https://www.youtube.com/watch?v=Xp3jR-ttMfo
Wednesday, January 19, 2022 -
https://arxiv.org/pdf/2010.15277.pdf - Class-incremental learning: survey and performance evaluation on image classification
Wednesday, January 12, 2022 -
https://arxiv.org/abs/2006.11287 - Discovering Symbolic Models from Deep Learning with Inductive Biases
Wednesday, January 5, 2022 -
https://arxiv.org/pdf/2006.09252.pdf - Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
Wednesday, December 29, 2021 -
https://arxiv.org/pdf/2112.04426.pdf - Improving Language Models by Retrieving from Trillions of Tokens
Wednesday, December 22, 2021 -
https://arxiv.org/abs/2106.01798 - Implicit MLE: Backpropagating Through Discrete Exponential Family Distributions
https://www.youtube.com/watch?v=W2UT8NjUqrk
Wednesday, December 15, 2021 -
https://arxiv.org/pdf/2108.01073.pdf - Image Synthesis and Editing with Stochastic Differential Equations
Wednesday, December 1, 2021 -
https://openreview.net/forum?id=HfpNVDg3ExA OpenReviewOpenReview Probabilistic Transformer For Time Series Analysis
Wednesday, November 17, 2021 -
https://arxiv.org/pdf/2110.03922.pdf - NEURAL TANGENT KERNEL EIGENVALUES ACCURATELY PREDICT GENERALIZATION
Wednesday, November 10, 2021 -
https://arxiv.org/pdf/2104.00681.pdf - NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
https://github.com/zju3dv/NeuralRecon
Wednesday, October 27, 2021 -
https://arxiv.org/pdf/2110.09485.pdf - Learning in High Dimension Always Amounts to Extrapolation
Wednesday, October 20, 2021 -
https://arxiv.org/pdf/2109.02355.pdf - A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
Wednesday, October 13, 2021 -
https://arxiv.org/pdf/2006.09011.pdf - Improved Techniques for Training Score-Based Generative Models
Wednesday, October 6, 2021 -
https://arxiv.org/abs/2006.05929 - Dataset Condensation with Gradient Matching
Wednesday, September 29, 2021 -
https://arxiv.org/abs/1811.10959 - Dataset distillation
Wednesday, September 22, 2021 -
https://arxiv.org/pdf/2003.13216.pdf - Learning to Learn Single Domain Generalization
Wednesday, September 15, 2021 -
https://arxiv.org/pdf/2108.11482.pdf - ETA Prediction with Graph Neural Networks in Google Maps
Wednesday, September 8, 2021 -
https://cascaded-diffusion.github.io/assets/cascaded_diffusion.pdf - Cascaded Diffusion Models for High Fidelity Image Generation
Wednesday, September 1, 2021 -
https://arxiv.org/pdf/2107.06277.pdf - Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
Wednesday, August 25, 2021 -
https://arxiv.org/abs/2108.07732 - Program Synthesis with Large Models
Wednesday, August 18, 2021 -
https://arxiv.org/abs/2012.13349 - Solving Mixed Integer Programs Using Neural Networks
Wednesday, August 11, 2021 -
https://www.nature.com/articles/s41586-021-03819-2 - DeepFold
Wednesday, August 4, 2021 -
Alphafold - blog https://deepmind.com/blog/article/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology paper https://www.nature.com/articles/s41586-021-03819-2 supplemental info https://static-content.springer.com/esm/art%3A10.1038%2Fs41586-021-03819-2/MediaObjects/41586_2021_3819_MOESM1_ESM.pdf
Wednesday, July 21, 2021 -
https://www.zdnet.com/article/googles-supermodel-deepmind-perceiver-is-a-step-on-the-road-to-an-ai-machine-that-could-process-everything/ https://arxiv.org/abs/2103.03206
Wednesday, July 14, 2021 -
https://arxiv.org/pdf/1503.03585.pdf (Deep Unsupervised Learning using Non equilibrium Thermodynamics) by Surya Ganguli at Stanford
Wednesday, July 7, 2021 - https://arxiv.org/pdf/2105.05233.pdf - Diffusion Models Beat GANs on Image Synthesis
Wednesday, June 30, 2021 -
https://arxiv.org/pdf/2006.11239.pdf - Denoising Diffusion Probabilistic Models
Wednesday, June 23, 2021 -
https://arxiv.org/abs/2010.03409 - Learning mesh-based simulation with graph networks
https://sites.google.com/view/learning-to-simulate
https://deepmind.com/research/publications/Learning-to-Simulate-Complex-Physics-with-Graph-Networks
Wednesday, June 16, 2021 -
https://arxiv.org/pdf/2106.01345.pdf - Decision Transformer: Reinforcement Learning via Sequence Modeling
https://www.youtube.com/watch?v=-buULmf7dec
https://sites.google.com/berkeley.edu/decision-transformer
Wednesday, June 9, 2021 -
https://arxiv.org/pdf/2103.07945.pdf - Learning One Representation to Optimize All Rewards
Wednesday, June 2, 2021 -
https://distill.pub/2021/multimodal-neurons/ - Multimodal Neurons in Artificial Neural Networks
https://openai.com/blog/clip/ - CLIP: Connecting Text and Images
Wednesday, May 26, 2021 -
https://arxiv.org/pdf/2104.14294.pdf - Emerging Properties in Self-Supervised Vision Transformers
Wednesday, May 19, 2021 -
https://arxiv.org/pdf/2104.10558.pdf - Contingencies from Observations: Tractable ContingencyPlanning with Learned Behavior Models
Wednesday, May 12, 2021 -
https://arxiv.org/pdf/1806.09055.pdf - DARTS: Differentiable Architecture Search (ICLR 2019)
Wednesday, May 5, 2021 -
https://arxiv.org/pdf/2104.06644.pdf - Masked Language Modeling and the Distributional Hypothesis:Order Word Matters Pre-training for Little
Wednesday, April 28, 2021 -
https://arxiv.org/pdf/2009.03717.pdf - Hierarchical message passing graph neural networks
Wednesday, April 14, 2021 -
https://arxiv.org/pdf/2103.03230v1.pdf - Barlow Twins: Self-Supervised Learning via Redundancy Reduction
Wednesday, April 7, 2021 -
https://arxiv.org/pdf/2103.14770.pdf - Categorical representation learning: morphism is all you need
Wednesday, March 31, 2021 -
https://arxiv.org/pdf/2102.12736v1.pdf - Time-Series Imputation with Wasserstein Interpolation for Optimal Look-Ahead-Bias and Variance Tradeoff
Wednesday, March 24, 2021 -
https://awacrl.github.io/ - Accelerating online reinforcement learning with offline datasets
Wednesday, March 17, 2021 -
https://arxiv.org/pdf/2102.12092.pdf - Zero-Shot Text-to-Image Generation
https://openai.com/blog/dall-e/
Wednesday, March 10, 2021 -
https://giotto-ai.github.io/gtda-docs/latest/notebooks/gravitational_waves_detection.html
Wednesday, March 3, 2021 -
https://arxiv.org/pdf/2102.08602.pdf - Modeling long-range interactions without attention
Wednesday, February 24, 2021 -
https://arxiv.org/pdf/2101.08692.pdf - Characterizing signal propagation to close the performance gap in unnormalized resnets
Wednesday, February 17, 2021 -
https://arxiv.org/pdf/2006.10742.pdf - Learning Invariant Representations forReinforcement Learning without Reconstruction
Wednesday, February 10, 2021 -
https://arxiv.org/pdf/2007.13544.pdf - Combining Deep Reinforcement Learning and Search for Imperfect-Information Games
Wednesday, February 3, 2021 -
https://arxiv.org/pdf/2010.11929.pdf - An image is worth 16x16 words: transformers for image recognition at scale
Wednesday, January 27, 2021 -
https://arxiv.org/abs/2003.02821 - What went wrong and when? Instance-wise feature importance for time-series black-box models
Wednesday, January 20, 2021 -
https://arxiv.org/pdf/1912.09363.pdf - Temporal Fusion Transformersfor Interpretable Multi-horizon Time Series Forecasting
Wednesday, January 13, 2021 -
https://arxiv.org/abs/1905.10403 - Neural Jump Stochastic Differential Equations
Wednesday, January 6, 2021 -
http://implicit-layers-tutorial.org/neural_odes/ - We're continuing this from last week. This week we'll cover Ch 3,4,5.
Wednesday, December 30, 2020 -
http://implicit-layers-tutorial.org/ - NeurIPS tutorial on deep implicit networks
Wednesday, December 23, 2020 - https://arxiv.org/pdf/1907.03907.pdf - Latent ODEs for Irregularly-Sampled Time Series
https://www.youtube.com/watch?v=tOkH339Wucs
Wednesday, December 16, 2020 -
https://papers.nips.cc/paper/2020/file/08425b881bcde94a383cd258cea331be-Paper.pdf - Ridge Rider: Finding Diverse Solutions by FollowingEigenvectors of the Hessian
Wednesday, December 9, 2020 -
https://proceedings.neurips.cc/paper/2020/file/28e209b61a52482a0ae1cb9f5959c792-Paper.pdf “OOD-MAML: Meta-Learning for Few-Shot Out-of-Distribution Detection and Classification"
Wednesday, December 2, 2020 -
https://arxiv.org/pdf/2011.02421.pdf - ONE-SHOT CONDITIONAL AUDIO FILTERING OF ARBITRARY SOUNDS
Wednesday, November 18, 2020 -
https://arxiv.org/pdf/2010.14498.pdf - Implicit under-parametrization inhibits data efficient deep reinforcement learning
Mar 11 - Hacker Dojo
https://arxiv.org/pdf/2002.11089.pdf - Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
Mar 4 - Hacker Dojo
https://www.osapublishing.org/DirectPDFAccess/C6D6B2C3-953C-4461-695B6E5E2F993943_415059/prj-7-8-823.pdf?da=1&id=415059&seq=0&mobile=no --Nanophotonic media for artificial neural inference
Feb 19 - Hacker Dojo
https://arxiv.org/pdf/1910.02789.pdf - Language is Power: Representing States Using Natural Language in Reinforcement Learning
Feb 12 - Hacker Dojo
https://deepmind.com/blog/article/AlphaFold-Using-AI-for-scientific-discovery - Protein folding paper.
Feb 5 - Hacker Dojo
https://arxiv.org/abs/2001.04451 Reformer, the efficient transformer
https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html
Jan 22 - Hacker Dojo
https://arxiv.org/pdf/1906.05717.pdf - Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semantics
Jan 15 - Hacker Dojo
https://arxiv.org/pdf/1912.09524.pdf - Evolving ab initio trading strategies in heterogeneous environments
Jan 8 - Hacker Dojo
https://arxiv.org/pdf/1911.05892.pdf - Reinforcement Learning for Market Making in Multi-agent Dealer Market
Dec 18 - Hacker Dojo
https://www.nature.com/articles/s41586-019-1724-z.epdf?author_access_token=lZH3nqPYtWJXfDA10W0CNNRgN0jAjWel9jnR3ZoTv0PSZcPzJFGNAZhOlk4deBCKzKm70KfinloafEF1bCCXL6IIHHgKaDkaTkBcTEv7aT-wqDoG1VeO9-wO3GEoAMF9bAOt7mJ0RWQnRVMbyfgH9A%3D%3D
https://www.gwern.net/docs/rl/2019-vinyals.pdf
https://deepmind.com/blog/article/AlphaStar-Grandmaster-level-in-StarCraft-II-using-multi-agent-reinforcement-learning
Nov 20 - Hacker Dojo
https://arxiv.org/pdf/1911.04252.pdf - Self-training with Noisy Student improves ImageNet classification
Nov 13 - Hacker Dojo
https://arxiv.org/pdf/1910.12713.pdf - Few-shot video-video synthesis
Nov 6 - Hacker Dojo
https://arxiv.org/pdf/1906.11883.pdf - Unsupervised learning of Object Keypoints for Perception and Control
Oct 30 - Hacker Dojo
https://arxiv.org/pdf/1710.03748.pdf - Emergent Complexity via Multi-Agent Competition
https://openai.com/blog/competitive-self-play/
Oct 23 - Hacker Dojo
https://arxiv.org/pdf/1703.04908.pdf - Emergence of Grounded Compositional Language in Multi-Agent Populations
Oct 16 - Hacker Dojo
https://arxiv.org/pdf/1909.07528.pdf - Emergent tool use from multi agent autocurricula
https://openai.com/blog/emergent-tool-use/
Oct 9 - Hacker Dojo
https://arxiv.org/pdf/1901.00949.pdf - Machine Teaching in Hierarchical Genetic Reinforcement Learning: Curriculum Design of Reward Functions for Swarm Shepherding
Sept 25 - Hacker Dojo
https://arxiv.org/pdf/1812.01729.pdf - Boltzman Generators - Sampling equilibrium states of many body systems with deep learning
Sept 18 - Hacker Dojo
https://arxiv.org/pdf/1907.10599.pdf - Fine Grained Spectral Perspective on Neural Networks
Sept 11 - Hacker Dojo
https://arxiv.org/pdf/1906.08237.pdf - XLNet Generalized autoregressive pretraining for language understanding
Sept 4 - Hacker Dojo
https://arxiv.org/pdf/1905.09272.pdf - Data efficient image recognition with contrastive predictive coding.
August 21 - Hacker Dojo
https://arxiv.org/pdf/1904.10509.pdf - Generating long sequences with sparse transformers
August 14 - Hacker Dojo
https://arxiv.org/pdf/1807.03748.pdf - Representation learning with contrastive predictive coding.
July 31 - Hacker Dojo
https://arxiv.org/pdf/1906.08253.pdf - When to trust your model: model-based policy optimization
July 24 - Hacker Dojo
https://arxiv.org/pdf/1901.09321.pdf - Fixup initialization - residual learning without normalization
July 17 - Hacker Dojo
http://proceedings.mlr.press/v97/mahoney19a/mahoney19a.pdf - Traditional and heavy tailed self regularization in neural net models
July 3 - Hacker Dojo
https://arxiv.org/pdf/1804.08838.pdf - Measuring intrinsic dimension of objective landscapes
June 19 - Hacker Dojo
https://arxiv.org/abs/1810.09536 - Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
June 12 - Hacker Dojo
https://arxiv.org/pdf/1812.05159.pdf - An empirical study of example forgetting during neural network training.
June 5 - Hacker Dojo
https://arxiv.org/pdf/1812.00417.pdf - Snorkel Drybell - A case study in weak supervision at industrial scale
https://arxiv.org/pdf/1905.04981.pdf - Modelling instance level annotator reliability for natural language labelling
May 29 - Hacker Dojo
https://arxiv.org/pdf/1901.09321.pdf - Fixup Initialization: Residual Learning without Normalization
May 22 - Hacker Dojo
https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf - Language Models are Unsupervised Multitask Learners.
May 15 - Hacker Dojo
https://arxiv.org/pdf/1811.00995.pdf - Invertible Residual Networks
Apr 29 - Hacker Dojo
https://arxiv.org/pdf/1904.01681.pdf - Augmented Neural ODE's
Apr 8 - Hacker Dojo
https://arxiv.org/pdf/1901.00596.pdf - Comprehensive Survey of Graph Neural Nets
https://github.com/rusty1s/pytorch_geometric
Apr 1 - Hacker Dojo
https://arxiv.org/pdf/1901.00596.pdf - Comprehensive Survey of Graph Neural Nets
Mar 25 - Hacker Dojo
https://papers.nips.cc/paper/7539-optimal-algorithms-for-non-smooth-distributed-optimization-in-networks.pdf - nips award winner
Mar 18 - Hacker Dojo
https://papers.nips.cc/paper/8200-non-delusional-q-learning-and-value-iteration.pdf - Non-delusional Q-learning and Value Iteration
Mar 11 - Hacker Dojo
https://arxiv.org/pdf/1706.03762.pdf - attention is all you need - Vaswani
https://github.com/jadore801120/attention-is-all-you-need-pytorch - easier to read code
https://www.youtube.com/watch?v=S0KakHcj_rs
https://tdls.a-i.science/events/2018-10-22/
https://tdls.a-i.science/events/2019-02-04/
http://nlp.seas.harvard.edu/2018/04/03/attention.html
Mar 4 - Hacker Dojo
https://arxiv.org/pdf/1806.02643.pdf - Re-evalating Evaluation
Feb 25 - Hacker Dojo
https://arxiv.org/pdf/1812.11951.pdf - Learning to Design RNA
Feb 11 - Hacker Dojo -
https://arxiv.org/pdf/1901.02860.pdf - Transformer XL - Attentive Language Models, Beyond a fixed length context
Feb 4 - Hacker Dojo
https://arxiv.org/pdf/1809.06646.pdf - Model Free Adaptive Optimal Control of Sequential Manufacturing Process Using Reinforcement Learning
January 28 - Hacker Dojo
https://arxiv.org/pdf/1806.07366.pdf - Neural Ordinary Differential Equations - Top paper NIPS2019
January 21 - Hacker Dojo
https://arxiv.org/pdf/1606.05312.pdf - Successor Features for Transfer in Reinforcement Learning
http://proceedings.mlr.press/v37/schaul15.pdf - Universal Value Function Approximators
http://proceedings.mlr.press/v80/barreto18a/barreto18a.pdf - Transfer in deep reinforcement learning using successor features and generalised policy improvement.
https://www.youtube.com/watch?v=YDCPHekLUI4&t=1053s - Tom Schaul
https://www.youtube.com/watch?v=OCHwXxSW70o - Tejas Kulkarni
January 14 - Hacker Dojo
https://arxiv.org/pdf/1812.07626.pdf - Universal Successor Features Approximators
January 7 - Hacker Dojo
https://arxiv.org/pdf/1810.12715.pdf - On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models
December 17 - Hacker Dojo
https://openreview.net/pdf?id=S1x4ghC9tQ - Temporal Difference Variational Autoencoder
December 10 - Hacker Dojo
https://openreview.net/pdf?id=S1JHhv6TW - Boosting Dilated Convolution with Mixed Tensor Decompositions
December 3 - Hacker Dojo
https://arxiv.org/pdf/1712.01208.pdf - The case for learned index structures
November 26 - Hacker Dojo
https://arxiv.org/abs/1809.07402 - Generalization properties of nn - Socher
https://einstein.ai/research/blog/identifying-generalization-properties-in-neural-networks - blog for above paper
November 19 - Hacker Dojo
https://arxiv.org/pdf/1802.05983.pdf - Disentangling by Factorising
https://arxiv.org/pdf/1804.00104.pdf - Learning Disentangled Joint, Discrete and Continuous Representations
https://arxiv.org/pdf/1807.05520.pdf - Deep Clustering for Unsupervised Learning of Visual Features
https://github.com/1Konny/FactorVAE
https://github.com/paruby/FactorVAE
https://github.com/nicolasigor/FactorVAE
November 12 - Hacker Dojo
https://arxiv.org/pdf/1810.12894.pdf - Exploration by Random Network Distillation - OpenAI
November 5 - Hacker Dojo
https://arxiv.org/pdf/1810.04805.pdf - Pre-trainged bi directional transformers for language translation
October 22 - Hacker Dojo
https://arxiv.org/pdf/1801.02613.pdf - Characterizing Adversarial Examples using Local Intrinsic Dimensionality
October 15 - Hacker Dojo
https://arxiv.org/pdf/1808.06670.pdf - Learning Deep Representations by Mutual Estimation Estimation and Maximization - Hjelm, Bengio
October 8 - Hacker Dojo
https://arxiv.org/pdf/1802.04364.pdf - Junction Tree Variational Auto-Encoder for Molecular Graph Generation
http://snap.stanford.edu/proj/embeddings-www/files/nrltutorial-part2-gnns.pdf
October 1 - Hacker Dojo
https://arxiv.org/pdf/1808.06601.pdf - Video to video synthesis https://github.com/NVIDIA/vid2vid - code
September 24 - Hacker Dojo
https://arxiv.org/pdf/1807.03146.pdf - Discovery of 3d keypoints from 2d image
September 17 - Hacker Dojo
https://arxiv.org/abs/1709.02371 - PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume," by Deqing Sun et al. (CVPR 2018)
Phil Ferrier will present the paper and run though his code for us. Phil's code is on his github reop:
https://github.com/philferriere/tfoptflow
September 10 - Hacker Dojo
https://arxiv.org/pdf/1807.03247.pdf - Intriguing failure (and improvement) to CNN for determining rotations.
September 3 - Hacker Dojo
https://arxiv.org/pdf/1803.03324.pdf - Learning Deep Generative Models of Graphs
August 27 - Hacker Dojo
https://arxiv.org/abs/1709.10082 - Optimally decentralized multi-robot collision avoidance w reinforcement learning.
https://github.com/TensorSwarm/TensorSwarm - Andreas Pasternak code for above
August 13 - Hacker Dojo
https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/learning-dexterity/learning-dexterity-paper.pdf -Robot doing single hand manipulations.
https://www.theverge.com/2018/7/30/17621112/openai-robot-dexterity-dactyl-artificial-intelligence
July 30 - Hacker Dojo -
https://arxiv.org/pdf/1711.03953.pdf - Breaking the softmax bottleneck
https://arxiv.org/pdf/1805.10829.pdf - SigSoftMax: Reanalyzing the softmax bottleneck
https://severelytheoretical.wordpress.com/2018/06/08/the-softmax-bottleneck-is-a-special-case-of-a-more-general-phenomenon/
July 23 - Hacker Dojo -
https://arxiv.org/pdf/1807.01281.pdf - Human level performance in first person multiplayer games with population reinforcement learning.
https://deepmind.com/blog/capture-the-flag/
https://www.youtube.com/watch?v=steioHoiEms
https://arxiv.org/abs/1711.09846v2
https://arxiv.org/pdf/1611.05397.pdf
July 16 - Hacker Dojo
https://arxiv.org/pdf/1803.10122.pdf - schmidhuber paper on RL
July 9 - Hacker Dojo
https://deepmind.com/research/publications/neural-scene-representation-and-rendering/ - Rendering 3d scene
July 2 - Hacker Dojo -
https://arxiv.org/pdf/1707.06347.pdf - Proximal Optimization Policies
June 25 - Hacker Dojo
https://openreview.net/pdf?id=BJOFETxR- - Learning to represent programs with graphs
June 18 - Hacker Dojo
https://openreview.net/pdf?id=BkisuzWRW - Zero Shot Visual Imitation - Reinforcement Learning
June 11 - Hacker Dojo
https://openreview.net/forum?id=HkL7n1-0b - Wasserstein Auto Encoders - one of ICLR top papers.
June 4 - Hacker Dojo
https://openreview.net/pdf?id=Hy7fDog0b - Ambient GAN - Generative Models from Lossy Measurements - ICLR top paper
May 21 - Hacker Dojo
https://arstechnica.com/science/2018/05/ai-trained-to-navigate-develops-brain-like-location-tracking/ - Grid representations in rat brain
https://deepmind.com/documents/200/Banino_at_al_final.pdf --
https://www.nature.com/articles/s41586-018-0102-6 --
May 14 - Hacker Dojo
https://arxiv.org/pdf/1712.06567.pdf - Deep Neuroevolution: Genetic Algorithms are a Competitive Alternative for
Training Deep Neural Networks for Reinforcement Learning
https://arxiv.org/pdf/1712.06560.pdf - Improving Exploration in Evolution Strategies for Deep Reinforcement
Learning via a Population of Novelty-Seeking Agents
https://eng.uber.com/deep-neuroevolution/ - Uber engineering blog post
May 7 - Hacker Dojo
https://arxiv.org/pdf/1801.10130.pdf - spherical CNN
Apr 30 - Hacker Dojo
https://arxiv.org/pdf/1710.07313.pdf - Using machine learning to replicate chaotic attractors
http://www.bmp.ds.mpg.de/tl_files/bmp/preprints/Zimmermann_Parlitz_preprint.pdf - paper to be published in "chaos"
https://www.quantamagazine.org/machine-learnings-amazing-ability-to-predict-chaos-20180418/ - blog post
Apr 23 - Hacker Dojo
https://arxiv.org/pdf/1711.10925.pdf - Deep Image Prior
https://dmitryulyanov.github.io/deep_image_prior - git hub from authors
https://box.skoltech.ru/index.php/s/ib52BOoV58ztuPM
http://mlexplained.com/2018/01/18/paper-dissected-deep-image-prior-explained/
http://fortune.com/2018/04/24/nvidia-artificial-intelligence-images/ - Article w video showing photo editing use
Apr 16 - Hacker Dojo
Finish Fractal AI
https://arxiv.org/pdf/1711.07971.pdf - non-local filtering
Apr 9 - Hacker Dojo
http://lanl.arxiv.org/pdf/1803.05049v1 - Fractal AI
Apr 2 - Hacker Dojo
https://arxiv.org/pdf/1803.04831.pdf - IndRNN longer deeper RNN's
Mar 26 - Hacker Dojo
https://arxiv.org/pdf/1711.10433.pdf - parallel wavenet
https://arxiv.org/pdf/1708.04552.pdf - regularizing convnet with cutout (desert paper)
http://www.cs.toronto.edu/~jmartens/docs/Deep_HessianFree.pdf - will get short presentation on this one.
Mar 19 - Hacker Dojo
https://arxiv.org/pdf/1802.03268.pdf - Efficient Neural Architecture Search via Parameter Sharing
https://github.com/carpedm20/ENAS-pytorch
some related papers and reviews.
https://arxiv.org/pdf/1708.05344.pdf - One shot architecture search
https://openreview.net/forum?id=ByQZjx-0-
and
https://openreview.net/forum?id=rydeCEhs-
Mar 12 - Hacker Dojo
https://arxiv.org/abs/1703.10135 - tacotron - end-to-end speech synthesis
https://arxiv.org/pdf/1712.05884.pdf - tacotron 2
https://research.googleblog.com/2017/12/tacotron-2-generating-human-like-speech.html -
https://github.com/A-Jacobson/tacotron2 - pytorch code
http://research.baidu.com/deep-speech-3%EF%BC%9Aexploring-neural-transducers-end-end-speech-recognition/
Feb 26 - Hacker Dojo
https://arxiv.org/pdf/1705.09792.pdf - Deep Complex Networks
Feb 19 - Hacker Dojo
https://arxiv.org/pdf/1801.10308.pdf - Nested LSTM's
https://arxiv.org/pdf/1705.10142.pdf - KRU from Fair
https://github.com/hannw/nlstm - tf code for Nested LSTM
Feb 12 - Hacker Dojo
http://openaccess.thecvf.com/content_cvpr_2017/papers/Khoreva_Simple_Does_It_CVPR_2017_paper.pdf - Weakly Supervised Instance and Semantic Segmentation
https://www.mpi-inf.mpg.de/departments/computer-vision-and-multimodal-computing/research/weakly-supervised-learning/simple-does-it-weakly-supervised-instance-and-semantic-segmentation/
https://github.com/philferriere/tfwss - Phil Ferriere's code
https://drive.google.com/file/d/1wPHMA4PqygawvIxRiy-2ZMKcpUO447cz/view?usp=sharing - mehul's notebook on segmentation
Feb 5 - Hacker Dojo
https://arxiv.org/pdf/1511.06939.pdf - using rnn for recommendation system
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46488.pdf - latest paper on rnn for recommendation
Jan 29 - Hacker Dojo
https://arxiv.org/pdf/1709.04511.pdf - Empirical study of multi-agent RL
https://github.com/geek-ai/1m-agents - code
Jan 22 - Hacker Dojo
https://arxiv.org/pdf/1704.00028.pdf - Improvements in Wasserstein GAN training
Jan 15 - Hacker Dojo
https://arxiv.org/pdf/1710.02298.pdf - Combining improvements in deep reinforcement learning
Jan 8 - Hacker Dojo
https://openreview.net/pdf?id=HJWLfGWRb - follow-on to capsule network paper
https://www.youtube.com/watch?v=pPN8d0E3900
https://www.youtube.com/watch?v=2Kawrd5szHE
https://github.com/ageron/handson-ml/blob/master/extra_capsnets.ipynb
https://github.com/naturomics/CapsNet-Tensorflow
https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks-part-ii-how-capsules-work-153b6ade9f66
Dec 11 - Hacker Dojo
https://arxiv.org/pdf/1710.09829.pdf - Dynamic routing between capsules - Hinton
Nov 27 - Hacker Dojo
https://arxiv.org/pdf/1701.01724.pdf - DeepStack: Expert-Level Artificial Intelligence in Heads-Up No-Limit Poker
Nov 13 - Hacker Dojo
https://deepmind.com/documents/119/agz_unformatted_nature.pdf - alpha zero paper
https://webdocs.cs.ualberta.ca/~mmueller/talks/2016-LeeSedol-AlphaGo.pdf - some slides
Nov 6 - Hacker Dojo
https://arxiv.org/pdf/1703.10593.pdf - cycle consistent GANs
Oct 30 - Hacker Dojo
https://arxiv.org/pdf/1503.02406.pdf Naftali Tishby and Noga Zaslavsky. information bottleneck principle.
https://www.cs.huji.ac.il/labs/learning/Papers/allerton.pdf - Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method.
Oct 23 - Hacker Dojo
Mask R-CNN
https://arxiv.org/abs/1703.06870
And these are prerequisites (read at least Fast R-CNN and Faster R-CNN)
R-CNN
https://arxiv.org/abs/1311.2524
Fast R-CNN
https://arxiv.org/pdf/1504.08083.pdf
Faster R-CNN
https://arxiv.org/abs/1506.01497 Feature Pyramid Networks
https://arxiv.org/abs/1612.03144
Oct 16 - Hacker Dojo
https://arxiv.org/pdf/1703.00810.pdf - Opening the Black Box of Neural Nets via Information
https://www.youtube.com/watch?v=ekUWO_pI2M8
https://www.youtube.com/watch?v=bLqJHjXihK8
Oct 9 - Hacker Dojo
https://arxiv.org/pdf/1501.00092.pdf - super resolution first paper
https://arxiv.org/abs/1608.00367 - super resolution second paper
Oct 2 - Hacker Dojo
https://arxiv.org/abs/1604.03901 - Single-Image Depth Perception in the Wild
Sept 25 - Hacker Dojo
https://arxiv.org/pdf/1706.08947.pdf - Exploring generalization in deep networks.
Sept 18 - Hacker Dojo
https://arxiv.org/pdf/1705.02550.pdf - nvidia drone nav
https://github.com/NVIDIA-Jetson/redtail/wiki - code
Sept 11 - Hacker Dojo
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.365.5060&rep=rep1&type=pdf - hyperneat ref
https://arxiv.org/pdf/1609.09106.pdf - Hypernet ref
http://blog.otoro.net/2016/09/28/hyper-networks/ - blog on hypernet
https://www.youtube.com/watch?v=-8oyTYViuJ4 - vid on hyperNeat
http://eplex.cs.ucf.edu/hyperNEATpage/HyperNEAT.html - blog on hyperNeat
August 28 - Hacker Dojo
https://arxiv.org/pdf/1708.05344.pdf - SMASH: One-Shot Model Architecture Search through HyperNetworks https://www.youtube.com/watch?v=79tmPL9AL48 - youtube vid on SMASH
August 21 - Hacker Dojo
https://arxiv.org/pdf/1706.02515.pdf - Self Normalizing Neural Networks - Hochreiter
August 14 - Hacker Dojo
https://arxiv.org/pdf/1606.01541.pdf - Reinforcement Learning for Dialog Generation - Jurafsky
https://github.com/liuyuemaicha/Deep-Reinforcement-Learning-for-Dialogue-Generation-in-tensorflow - tensorflow code for same
https://github.com/jiweil/ - some related code
https://arxiv.org/pdf/1612.00563.pdf - self critical training for image captioning - RL for text prob.
Some papers referenced by Jurafsky paper
[1506.05869] A Neural Conversational Model - Vinyals and Le
https://arxiv.org/abs/1604.04562 - Dialogue generation system - Wen
Aug 7 - Hacker Dojo
https://arxiv.org/pdf/1705.04304.pdf - A Deep Reinforced Model for Abstractive Summarization - socher
July 31 - Hacker Dojo
https://arxiv.org/pdf/1706.01433.pdf - visual interaction networks - deep mind
https://arxiv.org/pdf/1706.01427.pdf - neural model for relational reasoning - deep mind
July 24
Guest Speaker - Using FPGA to speed CNN.
https://arxiv.org/pdf/1703.03130.pdf - A structured self-attentive sentence embedding - Lin and Bengio
https://github.com/dennybritz/deeplearning-papernotes/blob/master/notes/self_attention_embedding.md (review)
https://github.com/yufengm/SelfAttentive code
https://github.com/Diego999/SelfSent code
July 17 - Hacker Dojo
https://arxiv.org/pdf/1706.03762.pdf - attention is all you need - Vaswani
https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/models
https://github.com/jadore801120/attention-is-all-you-need-pytorch - easier to read code
https://arxiv.org/pdf/1607.06450.pdf - layer normalization paper - hinton
https://www.youtube.com/watch?v=nR74lBO5M3s - google translate paper - youtube video
https://arxiv.org/pdf/1609.08144.pdf - google translate paper -
July 10 - Hacker Dojo
https://arxiv.org/pdf/1706.03762.pdf - attention is all you need - Vaswani
https://github.com/tensorflow/tensor2tensor/tree/master/tensor2tensor/models
https://github.com/jadore801120/attention-is-all-you-need-pytorch - easier to read code
https://arxiv.org/pdf/1607.06450.pdf - layer normalization paper - hinton
Some added references regarding positional encodings
http://www.machinelearning.org/proceedings/icml2006/047_Connectionist_Tempor.pdf - A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber
https://www.reddit.com/r/MachineLearning/comments/6jdi87/r_question_about_positional_encodings_used_in/
June 26 - Hacker Dojo
https://arxiv.org/pdf/1705.03122.pdf - convolutional sequence to sequence learning
https://arxiv.org/pdf/1706.03762.pdf - attention is all you need - Vaswani
http://www.machinelearning.org/proceedings/icml2006/047_Connectionist_Tempor.pdf - A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber
June 19 - Hacker Dojo
https://arxiv.org/pdf/1701.02720.pdf - RNN for end to end voice recognition
June 12 - Hacker Dojo
New reinforcement learning results -- Too cool for school. Watch the video and you'll be hooked.
https://www.youtube.com/watch?v=2vnLBb18MuQ&feature=em-subs_digest
http://www.cs.ubc.ca/~van/papers/2017-TOG-deepLoco/index.html - paper
May 22 - Hacker Dojo
https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/HintonDengYuEtAl-SPM2012.pdf - comparison of RNN and HMM for speech recognition
May 15 - Hacker Dojo
https://arxiv.org/pdf/1412.6572.pdf - Explaining and Harnessing Adversarial Examples
May 1 - Hacker Dojo
https://arxiv.org/abs/1704.03453 - The Space of Transferable Adversarial Examples
Apr 24 - Hacker Dojo
https://discourse-production.oss-cn-shanghai.aliyuncs.com/original/3X/1/5/15ba4cef726cab390faa180eb30fd82b693469f9.pdf - Using TPU for data center
Apr 17 - Hacker Dojo
Reservoir Computing by Felix Grezes. http://www.gc.cuny.edu/CUNY_GC/media/Computer-Science/Student%20Presentations/Felix%20Grezes/Second_Exam_Survey_Felix_Grezes_9_04_2014.pdf
Slides by Felix Grezes: Reservoir Computing for Neural Networks
http://www.gc.cuny.edu/CUNY_GC/media/Computer-Science/Student%20Presentations/Felix%20Grezes/Second_Exam_Slides_Felix_Grezes_9-14-2014.pdf
(more at: http://speech.cs.qc.cuny.edu/~felix/ )
This is a short, very useful backgrounder on randomized projections,
here used for compressed sensing, in a blog post by Terence Tao
https://terrytao.wordpress.com/2007/04/13/compressed-sensing-and-single-pixel-cameras/
and the same story told with illustrations on the Nuit Blanche blog:
http://nuit-blanche.blogspot.com/2007/07/how-does-rice-one-pixel-camera-work.html
(BTW http://nuit-blanche.blogspot.com is a tremendous website.)
If we have time, we may discuss this paper:
Information Processing Using a Single Dynamical Node as Complex System.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3195233/pdf/ncomms1476.pdf
Apr 10 - Hacker Dojo
https://arxiv.org/pdf/1603.08678.pdf - Instance-sensitive Fully Convolutional Networks
https://arxiv.org/pdf/1611.07709.pdf - Fully Convolutional Instance-aware Semantic Segmentation
Apr 3 - Hacker Dojo
https://arxiv.org/pdf/1703.03864.pdf - Sutskever paper on using evolutionary systems for optimizing RL prob
http://jmlr.csail.mit.edu/papers/volume15/wierstra14a/wierstra14a.pdf - ES paper with algo used in Sutskever paper
Mar 27 - Hacker Dojo
Aurobindo Tripathy will reprise a talk he's going to give at Embedded Summit this year. His talk will survey recent progress in object detection from RCNN to Single Shot MultiBox Detector and Yolo 9000.
Mar 20 - Hacker Dojo
https://arxiv.org/pdf/1612.05424.pdf - Unsupervised Pixel-level domain adaptation with generative adversarial networks
Mar 13 - Hacker Dojo
https://arxiv.org/pdf/1701.06547.pdf - adversarial learning for neural dialog generation
February 27 - Hacker Dojo
https://arxiv.org/pdf/1612.02699.pdf - Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
Zeeshan's slides are in the folder with his name on it. Along with his descriptions of his own ground-breaking work, he gives an excellent history of efforts to identify 3d objects from 2d images.
February 20 - Hacker Dojo
https://arxiv.org/pdf/1506.07285.pdf - Ask me anything - Socher
https://github.com/YerevaNN/Dynamic-memory-networks-in-Theano - Code and implementation notes.
https://www.youtube.com/watch?v=FCtpHt6JEI8&t=27s - Socher presentation of material
February 13 - Hacker Dojo
https://arxiv.org/pdf/1701.06538v1.pdf - Outrageously large neural networks
February 6 - Hacker Dojo
https://arxiv.org/pdf/1505.00387v2.pdf - Highway networks
https://arxiv.org/pdf/1507.06228.pdf - Also highway networks - different examples
https://arxiv.org/pdf/1607.03474v3.pdf - Recurrent Highway Networks
January 30 - Hacker Dojo
https://arxiv.org/pdf/1603.03116v2.pdf - Low-rank pass-through RNN's follow-on to unitary rnn https://github.com/Avmb/lowrank-gru - theano code
January 23 - HackerDojo
https://arxiv.org/abs/1612.03242 - Stack Gan Paper
https://github.com/hanzhanggit/StackGAN - Code
January 16 - Hacker Dojo
https://arxiv.org/pdf/1511.06464v4.pdf - Unitary Evolution RNN https://github.com/amarshah/complex_RNN - theano code
January 9 - Hacker Dojo
Cheuksan Edward Wang Talk
https://arxiv.org/pdf/1612.04642v1.pdf - rotation invariant cnn
https://github.com/deworrall92/harmonicConvolutions - tf code for harmonic cnn
http://visual.cs.ucl.ac.uk/pubs/harmonicNets/index.html - blog post by authors
January 2 - Hacker Dojo
https://arxiv.org/pdf/1602.02218v2.pdf - using typing to improve RNN behavior
http://jmlr.org/proceedings/papers/v37/jozefowicz15.pdf - exploration of alternative LSTM architectures
December 19 - Hacker Dojo
https://arxiv.org/pdf/1611.01576.pdf - Socher qRnn paper
December 12 - Hacker Dojo
https://arxiv.org/pdf/1604.02135v2.pdf - latest segmentation fair
https://github.com/MarvinTeichmann/tensorflow-fcn - code for segmenter
December 5 - Hacker Dojo
https://arxiv.org/pdf/1506.06204.pdf - Object segmentation
https://arxiv.org/pdf/1603.08695v2.pdf - refinement of above segmentation paper
https://code.facebook.com/posts/561187904071636/segmenting-and-refining-images-with-sharpmask/ - blog post
https://github.com/facebookresearch/deepmask - torch code for deepmask
November 28 - Hacker Dojo
https://arxiv.org/pdf/1506.01497v3.pdf
people.eecs.berkeley.edu/~rbg/slides/rbg-defense-slides.pdf - Girshick thesis slides
Check edge boxes and selective search
https://arxiv.org/pdf/1406.4729v4.pdf - key part of architecture
https://github.com/smallcorgi/Faster-RCNN_TF - excellent code
November 21 - Hacker Dojo
https://people.eecs.berkeley.edu/~rbg/papers/r-cnn-cvpr.pdf - RCNN
https://arxiv.org/pdf/1504.08083v2.pdf - RCNN - first in series
https://arxiv.org/pdf/1506.01497v3.pdf - Faster R-CNN
http://techtalks.tv/talks/rich-feature-hierarchies-for-accurate-object-detection-and-semantic-segmentation/60254/ - video of Girshick talk
November 14 - Hacker Dojo
https://arxiv.org/pdf/1506.02025v3.pdf - Spatial transformer networks
https://github.com/daviddao/spatial-transformer-tensorflow - tf code for above
October 31 - Hacker Dojo
https://github.com/jazzsaxmafia/show_attend_and_tell.tensorflow - tf code for attention-captioning http://cs.stanford.edu/people/karpathy/densecap/ - karpathy captioning https://arxiv.org/pdf/1412.2306v2.pdf - earlier karpathy captioning paper
October 20 - Galvanize
https://webdocs.cs.ualberta.ca/~sutton/book/the-book.html - Deep dive into reinforcement learning - Sutton and Barto - Chapters 1 and 2.
Oct 17 - Hacker Dojo
https://arxiv.org/pdf/1608.06993v1.pdf - DenseNet. New reigning champion image classifier
https://github.com/liuzhuang13/DenseNet - lua code
The DenseNet paper is straight-forward, so we're also going to start on image captioning
http://www.cs.toronto.edu/~zemel/documents/captionAttn.pdf
http://kelvinxu.github.io/projects/capgen.html
http://people.ee.duke.edu/~lcarin/Yunchen9.25.2015.pdf - slides for caption attention
collections of captioning papers.
https://github.com/kjw0612/awesome-deep-vision#image-captioning - images
https://github.com/kjw0612/awesome-deep-vision#video-captioning - video
Oct 13 - SF
http://www.mit.edu/~dimitrib/NDP_Encycl.pdf - (early) Bersekas paper on RL, policy and value iteration
http://www.nervanasys.com/demystifying-deep-reinforcement-learning/?imm_mid=0e2d7e&cmp=em-data-na-na-newsltr_20160420 - blog post on RL. Nice coverage of value iteration
Oct 10 - Hacker Dojo
https://github.com/carpedm20/pixel-rnn-tensorflow - tensorflow code for pixel rnn (and cnn)
Sept 19 - Hacker Dojo
https://arxiv.org/pdf/1606.05328v2.pdf - Conditional Image Generation with PixelCNN decoders
https://arxiv.org/pdf/1601.06759v3.pdf - Pixel RNN
https://drive.google.com/file/d/0B3cxcnOkPx9AeWpLVXhkTDJINDQ/view - wavenet Generative Audio
https://deepmind.com/blog/wavenet-generative-model-raw-audio/ - wavenet blog
Sept 15 - Galvanize SF
http://www.gitxiv.com/posts/fepYG4STYaej3KSPZ/densely-connected-convolutional-netowork-densenet
Sept 12 - Hacker Dojo
http://arxiv.org/pdf/1410.3916v11.pdf - original memory networks
https://arxiv.org/pdf/1606.03126v1.pdf - key/value memory augmented nn
http://www.thespermwhale.com/jaseweston/icml2016/icml2016-memnn-tutorial.pdf#page=87 - tutorial on memory networks in language understanding
August 29 - Hacker Dojo
https://arxiv.org/pdf/1410.5401v2.pdf - Neural Turing Machines
https://github.com/carpedm20/NTM-tensorflow
https://www.youtube.com/watch?v=_H0i0IhEO2g - Alex Graves presentation at microsoft research
http://www.robots.ox.ac.uk/~tvg/publications/talks/NeuralTuringMachines.pdf - slides for ntm
August 25 - Galvanize (SF)
http://arxiv.org/pdf/1410.3916v11.pdf - original memory networks
https://arxiv.org/pdf/1606.03126v1.pdf - key/value memory augmented nn
http://www.thespermwhale.com/jaseweston/icml2016/icml2016-memnn-tutorial.pdf#page=87 - tutorial on memory networks in language understanding
August 22 - Hacker Dojo
https://arxiv.org/pdf/1605.07648v1.pdf - fractal net - alternative to resnet for ultra-deep convolution
https://github.com/edgelord/FractalNet - tf code
http://www.gitxiv.com/posts/ibA8QEu8bvBJSDxr9/fractalnet-ultra-deep-neural-networks-without-residuals
August 18, 2016 - Galvanize (SF)
https://arxiv.org/pdf/1602.01783v2.pdf - new RL architecture - deep mind
Code:
https://github.com/Zeta36/Asynchronous-Methods-for-Deep-Reinforcement-Learning - tf
https://github.com/miyosuda/async_deep_reinforce - tf
https://github.com/coreylynch/async-rl - keras (tf)
https://github.com/muupan/async-rl - chainer (good discussion)
August 15, 2016 - Hacker Dojo
https://arxiv.org/pdf/1607.02533v1.pdf - Hardening deep networks to adversarial examples.
August 11, 2016 - Galvanize (SF)
http://www.gitxiv.com/posts/HQJ3F9YzsQZ3eJjpZ/model-free-episodic-control - deep mind gitxiv paper and code on github https://github.com/sudeepraja/Model-Free-Episodic-Control - other code https://github.com/ShibiHe/Model-Free-Episodic-Control
August 8, 2016 - Hacker Dojo
https://arxiv.org/pdf/1406.2661.pdf - originating paper on generative adversarial net (gan) - goodfellow, bengio
http://arxiv.org/pdf/1511.06434v2.pdf - deep cnn gan - radford
https://github.com/Newmu/dcgan_code - theano code for cnn gan - radford
August 4, 2016 - Galvanize (SF)
http://www.gitxiv.com/posts/HQJ3F9YzsQZ3eJjpZ/model-free-episodic-control - deep mind gitxiv paper and code on github
August 1, 2016 - Hacker Dojo
Papers -
https://drive.google.com/file/d/0B8Dg3PBX90KNWG5KQXNQOFlBLU1JWWVONkN1UFpnbUR6Y0cw/view?pref=2&pli=1 - Using Stochastic RNN for temporal anomaly detection
https://home.zhaw.ch/~dueo/bbs/files/vae.pdf - cover math
https://arxiv.org/pdf/1401.4082v3.pdf - Rezende - Other Original VAE paper
Code Review -
https://github.com/oduerr/dl_tutorial/blob/master/tensorflow/vae/vae_demo.ipynb
https://github.com/oduerr/dl_tutorial/blob/master/tensorflow/vae/vae_demo-2D.ipynb
July 28, 2016 - SF
Papers:
http://arxiv.org/pdf/1410.5401v2.pdf - Neural Turing Machines - Graves et. al.
https://arxiv.org/pdf/1605.06065v1.pdf - One Shot Learning - DeepMind
Code:
http://icml.cc/2016/reviews/839.txt
https://github.com/brendenlake/omniglot
https://github.com/tristandeleu/ntm-one-shot
https://github.com/MLWave/extremely-simple-one-shot-learning
July 25, 2016 - Hacker Dojo
Papers - Using VAE for anomaly detection
https://arxiv.org/pdf/1411.7610.pdf - Stochastic Recurrent Networks
https://drive.google.com/file/d/0B8Dg3PBX90KNWG5KQXNQOFlBLU1JWWVONkN1UFpnbUR6Y0cw/view?pref=2&pli=1 - Using Stochastic RNN for temporal anomaly detection
July 21, 2016 - SF
Papers to read:
http://www.thespermwhale.com/jaseweston/ram/papers/paper_16.pdf
http://snowedin.net/tmp/Hochreiter2001.pdf -
Comments / Code
http://icml.cc/2016/reviews/839.txt
https://github.com/brendenlake/omniglot
https://github.com/tristandeleu/ntm-one-shot
https://github.com/MLWave/extremely-simple-one-shot-learning
https://www.periscope.tv/hugo_larochelle/1ypJdnPRYEoKW
July 18, 2016 - Hacker Dojo
Papers to read:
http://arxiv.org/pdf/1312.6114v10.pdf - variational autoencoders - U of Amsterdam - Kingma and Welling
http://arxiv.org/pdf/1310.8499v2.pdf - deep autoregressive networks - deep mind
https://arxiv.org/abs/1606.05908 - tutorial on vae
Commentaries/Code
https://jmetzen.github.io/2015-11-27/vae.html - metzen - code and discussion
http://blog.keras.io/building-autoencoders-in-keras.html - chollet - discusses different autoencoders, gives keras code.
June 27, July 11 2016 - Hacker Dojo
Recurrent network for image generation - Deep Mind
https://arxiv.org/pdf/1502.04623v2.pdf
Background and some references cited
http://blog.evjang.com/2016/06/understanding-and-implementing.html - blog w. code for VAE
http://arxiv.org/pdf/1312.6114v10.pdf - Variational Auto Encoder
https://jmetzen.github.io/2015-11-27/vae.html - tf code for variational auto-encoder
https://www.youtube.com/watch?v=P78QYjWh5sM
https://arxiv.org/pdf/1401.4082.pdf - stochastic backpropagation and approx inference - deep mind
http://www.cs.toronto.edu/~fritz/absps/colt93.html - keep neural simple by minimizing descr length - hinton
https://github.com/vivanov879/draw - code
June 20, 2016 - Penninsula
Recurrent models of visual attention - Deep Mind
https://papers.nips.cc/paper/5542-recurrent-models-of-visual-attention.pdf
June 23, 29 2016 - SF
http://arxiv.org/pdf/1410.5401v2.pdf - Neural Turing Machines - Graves et. al.
https://arxiv.org/pdf/1605.06065v1.pdf - One Shot Learning - DeepMind
http://www.shortscience.org/paper?bibtexKey=journals/corr/1605.06065 - Larochell comments on One-Shot paper
https://github.com/shawntan/neural-turing-machines - Code
https://www.reddit.com/r/MachineLearning/comments/2xcyrl/i_am_j%C3%BCrgen_schmidhuber_ama/cp4ecce - schmidhuber's comments
http://www.thespermwhale.com/jaseweston/ram/papers/paper_16.pdf
http://snowedin.net/tmp/Hochreiter2001.pdf -
Reviews:
http://icml.cc/2016/reviews/839.txt
Code
https://github.com/brendenlake/omniglot
https://github.com/tristandeleu/ntm-one-shot
https://github.com/MLWave/extremely-simple-one-shot-learning
June 13, 2016 - TBD, Penninsula
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning:
http://arxiv.org/pdf/1602.07261v1.pdf
June 9, 2016 - Galvanize
Visualizing and Understanding RNN:
https://arxiv.org/pdf/1506.02078v2.pdf
June 6, 2016 - Hacker Dojo
Google inception paper - origin of 1x1 convolution layers
http://arxiv.org/pdf/1409.4842v1.pdf
June 2, May 26, 2016 - Galvanize
Image segmentation with deep encoder-decoder
https://arxiv.org/pdf/1511.00561.pdf
May 23, 2016 - Hacker Dojo
Compressed networks, reducing flops by pruning
https://arxiv.org/pdf/1510.00149.pdf
http://arxiv.org/pdf/1602.07360v3.pdf
May 16, 2016
Word2Vec meets LDA:
http://arxiv.org/pdf/1605.02019v1.pdf - Paper
https://twitter.com/chrisemoody - Chris Moody's twiter with links to slides etc.
http://qpleple.com/topic-coherence-to-evaluate-topic-models/ - writeup on topic coherence
May 9, 2016
https://arxiv.org/pdf/1603.05027v2.pdf - Update on microsoft resnet - identity mapping
http://gitxiv.com/posts/MwSDm6A4wPG7TcuPZ/recurrent-batch-normalization - batch normalization w. RNN
May 2, 2016
Go playing DQN - AlphaGo
https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf
https://m.youtube.com/watch?sns=em&v=pgX4JSv4J70 - video of slide presentation on paper
https://en.m.wikipedia.org/wiki/List_of_Go_games#Lee.27s_Broken_Ladder_Game - Handling "ladders" in alphgo
https://en.m.wikipedia.org/wiki/Ladder_(Go) - ladders in go
April 25, 2016 - Microsoft Resnet
The Paper
http://arxiv.org/pdf/1512.03385v1.pdf
References:
http://arxiv.org/pdf/1603.05027v2.pdf - Identity mapping paper
Code:
https://keunwoochoi.wordpress.com/2016/03/09/residual-networks-implementation-on-keras/ - keras code
https://github.com/ry/tensorflow-resnet/blob/master/resnet.py - tensorflow code
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/skflow/resnet.py
April 18, 2016 - Batch Normalization
The Paper
https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf
http://gitxiv.com/posts/MwSDm6A4wPG7TcuPZ/recurrent-batch-normalization - Batch Normalization for RNN
April 11, 2016 - Atari Game Playing DQN
The Paper https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf)
Related references:
This adds 'soft' and 'hard' attention and the 4 frames are replaced with an LSTM layer:
http://gitxiv.com/posts/NDepNSCBJtngkbAW6/deep-attention-recurrent-q-network
http://home.uchicago.edu/~arij/journalclub/papers/2015_Mnih_et_al.pdf - Nature Paper
http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html - videos at the bottom of the page
http://llcao.net/cu-deeplearning15/presentation/DeepMindNature-preso-w-David-Silver-RL.pdf - David Silver's slides
http://www.cogsci.ucsd.edu/~ajyu/Teaching/Cogs118A_wi09/Class0226/dayan_watkins.pdf
http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html - David Silver
Implementation Examples:
http://www.danielslater.net/2016/03/deep-q-learning-pong-with-tensorflow.html
March 3, 2016 Gated Feedback RNN
The Paper
"Gated RNN" (http://arxiv.org/pdf/1502.02367v4.pdf
-Background Material
http://arxiv.org/pdf/1506.00019v4.pdf - Lipton's excellent review of RNN
http://www.nehalemlabs.net/prototype/blog/2013/10/10/implementing-a-recurrent-neural-network-in-python/ - Discussion of RNN and theano code for Elman network - Tiago Ramalho
http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf - Hochreiter's original paper on LSTM
https://www.youtube.com/watch?v=izGl1YSH_JA - Hinton video on LSTM
-Skylar Payne's GF RNN code
https://github.com/skylarbpayne/hdDeepLearningStudy/tree/master/tensorflow
-Slides
https://docs.google.com/presentation/d/1d2keyJxRlDcD1LTl_zjS3i45xDIh2-QvPWU3Te29TuM/edit?usp=sharing
https://github.com/eadsjr/GFRNNs-nest/tree/master/diagrams/diagrams_formula
Reviews
http://www.computervisionblog.com/2016/06/deep-learning-trends-iclr-2016.html
https://indico.io/blog/iclr-2016-takeaways/