omerbsezer/LSTM_RNN_Tutorials_with_Demo omerbsezer/LSTM_RNN_Tutorials_with_Demo

  • Stars
    star
    672
  • Rank 67,180 (Top 2 %)
  • Language
    Python
  • Created over 6 years ago
  • Updated 8 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
omerbsezer/LSTM_RNN_Tutorials_with_Demo
github

omerbsezer

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

LSTM-RNN Tutorial with LSTM and RNN Tutorial with Demo with Demo Projects such as Stock/Bitcoin Time Series Prediction, Sentiment Analysis, Music Generation using Keras-Tensorflow

LSTM and RNN Tutorial with Demo (with Stock/Bitcoin Time Series Prediction, Sentiment Analysis, Music Generation)

There are many LSTM tutorials, courses, papers in the internet. This one summarizes all of them. In this tutorial, RNN Cell, RNN Forward and Backward Pass, LSTM Cell, LSTM Forward Pass, Sample LSTM Project: Prediction of Stock Prices Using LSTM network, Sample LSTM Project: Sentiment Analysis, Sample LSTM Project: Music Generation. It will continue to be updated over time.

Keywords: Deep Learning, LSTM, RNN, Stock/Bitcoin price prediction, Sentiment Analysis, Music Generation, Sample Code, Basic LSTM, Basic RNN

NOTE: This tutorial is only for education purpose. It is not academic study/paper. All related references are listed at the end of the file.

Table of Contents

What is Deep Learning (DL)?

"Deep Learning is a subfield of machine learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks." There are different types of DL models: Convolutional Neural Network, Recurrent Neural Networks (RNN), Long Short Term Memory (LSTM), Restricted Boltzmann Machine (RBM), Deep Belief Networks, etc.

In this tutorial, we are focusing on recurrent networks, especially LSTM. Basic RNN structure, Basic LSTM structures and Stock/Bitcoin Price Prediction Sample code are presented in the following sections.

What is RNN?

  • Recurrent neural network (RNN) is a type of deep learning model that is mostly used for analysis of sequential data (time series data prediction).
  • There are different application areas that are used: Language model, neural machine translation, music generation, time series prediction, financial prediction, etc.
  • The aim of this implementation is to help to learn structure of basic RNN (RNN cell forward, RNN cell backward, etc..).
  • Code is adapted from Andrew Ng's Course 'Sequential models'.

Code: https://github.com/omerbsezer/LSTM_RNN_Tutorials_with_Demo/tree/master/BasicRNN

RNN Cell

rnn_step_forward

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

RNN Forward Pass

rnn_fw

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

RNN Backward Pass

rnn_cell_backprop

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

RNN Problem

  • In theory, RNNs are absolutely capable of handling such “long-term dependencies.”
  • In practice, RNNs don’t seem to be able to learn them.
  • The problem was explored in depth by Hochreiter (1991) [German] and Bengio, et al. (1994) with LSTM

What is LSTM?

  • It is a special type of RNN, capable of learning long-term dependencies.

  • "Long short-term memory (LSTM) units are units of a recurrent neural network (RNN). An RNN composed of LSTM units is often called an LSTM network. A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell remembers values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell"

  • Long Short Term Memory (LSTM) is a type of deep learning model that is mostly used for analysis of sequential data (time series data prediction).

  • There are different application areas that are used: Language model, Neural machine translation, Music generation, Time series prediction, Financial prediction, Robot control, Time series prediction, Speech recognition, Rhythm learning, Music composition, Grammar learning, Handwriting recognition, Human action recognition, Sign Language Translation,Time series anomaly detection, Several prediction tasks in the area of business process management, Prediction in medical care pathways, Semantic parsing, Object Co-segmentation.

  • LSTM was proposed in 1997 by Sepp Hochreiter and Jürgen Schmidhuber and improved in 2000 by Felix Gers' team. Paper

Code: https://github.com/omerbsezer/LSTM_RNN_Tutorials_with_Demo/tree/master/BasicLSTM

LSTM Cell

lstm_cell

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

LSTM Forward Pass

lstm_fw

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

SAMPLE LSTM CODE: Prediction of Stock Prices Using LSTM network

Stock and ETFs prices are predicted using LSTM network (Keras-Tensorflow).

Code: https://github.com/omerbsezer/LSTM_RNN_Tutorials_with_Demo/tree/master/StockPricesPredictionProject

  • Stock prices are downloaded from finance.yahoo.com. Disneyland (DIS) Stock Price CSV file.
  • Closed value (column[5]) is used in the network, LSTM Code
  • Values are normalized in range (0,1).
  • Datasets are splitted into train and test sets, 50% test data, 50% training data.
  • Keras-Tensorflow is used for implementation.
  • LSTM network consists of 25 hidden neurons, and 1 output layer (1 dense layer).
  • LSTM network features input: 1 layer, output: 1 layer , hidden: 25 neurons, optimizer:adam, dropout:0.1, timestep:240, batchsize:240, epochs:1000 (features can be further optimized).
  • Root mean squared errors are calculated.
  • Output files: lstm_results (consists of prediction and actual values), plot file (actual and prediction values).

dis_prediction_and_actualprice

SAMPLE LSTM CODE: Sentiment Analysis

Sentiment Analysis is an analysis of the sentence, text at the document that gives us the opinion of the sentence/text. In this project, it will be implemented a model which inputs a sentence and finds the most appropriate emoji to be used with this sentence. Code is adapted from Andrew Ng's Course 'Sequential Models'.

NOTE:This project is adapted from Andrew Ng, Sequential Models Course, Deep Learning Specialization for educational purpose

Code: https://github.com/omerbsezer/LSTM_RNN_Tutorials_with_Demo/tree/master/SentimentAnalysisProject

Results

resultsemoji

DataSet

We have a tiny dataset (X, Y) where:

  • X contains 127 sentences (strings)
  • Y contains a integer label between 0 and 4 corresponding to an emoji for each sentence

data_set

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

Embeddings

Glove 50 dimension, 40000 words of dictionary file is used for word embeddings. It should be downloaded from https://www.kaggle.com/watts2/glove6b50dtxt (file size = ~168MB))

  • word_to_index: dictionary mapping from words to their indices in the vocabulary (400,001 words, with the valid indices ranging from 0 to 400,000)
  • index_to_word: dictionary mapping from indices to their corresponding words in the vocabulary
  • word_to_vec_map: dictionary mapping words to their GloVe vector representation.

LSTM Model in Sentiment Analysis

LSTM structure is used for classification.

emojifier-v2

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

Parameters:

lstm_struct

SAMPLE LSTM CODE: Music Generation

With trained DL model (LSTM), new sequences of time series data can be predicted. In this project, it will be implemented a model which inputs a sample jazz music and samples/generates a new music. Code is adapted from Andrew Ng's Course 'Sequential models'.

NOTE:This project is adapted from Andrew Ng, Sequential Models Course, Deep Learning Specialization for educational purpose

Code: https://github.com/omerbsezer/LSTM_RNN_Tutorials_with_Demo/tree/master/MusicGenerationProject

How to Run Code?

Input File and Parameters

Model is trained with "data/original_music"

  • "X, Y, n_values, indices_values = load_music_utils()"

  • Number of training examples: 60,

  • Each of training examples length of sequence:30

  • Our music generation system will use 78 unique values.

  • X: This is an (m, Tx , 78) dimensional array. We have m training examples, each of which is a snippet of Tx=30Tx=30 musical values. At each time step, the input is one of 78 different possible values, represented as a one-hot vector. Thus for example, X[i,t,:] is a one-hot vector representating the value of the i-th example at time t.

  • Y: This is essentially the same as X, but shifted one step to the left (to the past).

  • n_values: The number of unique values in this dataset. This should be 78.

  • indices_values: python dictionary mapping from 0-77 to musical values.

LSTM Model in Music Generation

LSTM model structure is:

music_generation

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

Model is implemented with "djmodel(Tx, n_a, n_values)" function.

Predicting and Sampling:

Adding model, predicting and sampling feature, model structure is:

music_gen

[Andrew Ng, Sequential Models Course, Deep Learning Specialization]

Music Inference Model is similar trained model and it is implemented with "music_inference_model(LSTM_cell, densor, n_values = 78, n_a = 64, Ty = 100)" function. Music is generated with "redict_and_sample" function. Finally, your generated music is saved in output/my_music.midi.

Resources:

References:

More Repositories

1

Fast-Kubernetes

This repo covers Kubernetes with LABs: Kubectl, Pod, Deployment, Service, PV, PVC, Rollout, Multicontainer, Daemonset, Taint-Toleration, Job, Ingress, Kubeadm, Helm, etc.
PowerShell
1,773
star
2

Reinforcement_learning_tutorial_with_demo

Reinforcement Learning Tutorial with Demo: DP (Policy and Value Iteration), Monte Carlo, TD Learning (SARSA, QLearning), Function Approximation, Policy Gradient, DQN, Imitation, Meta Learning, Papers, Courses, etc..
Jupyter Notebook
711
star
3

Fast-Ansible

This repo covers Ansible with LABs: Multipass, Commands, Modules, Playbooks, Tags, Managing Files and Servers, Users, Roles, Handlers, Host Variables, Templates and details.
Jinja
625
star
4

Fast-Docker

This repo covers containerization and Docker Environment: Docker File, Image, Container, Commands, Volumes, Networks, Swarm, Stack, Service, possible scenarios.
Dockerfile
566
star
5

Fast-Pytorch

Pytorch Tutorial, Pytorch with Google Colab, Pytorch Implementations: CNN, RNN, DCGAN, Transfer Learning, Chatbot, Pytorch Sample Codes
Jupyter Notebook
427
star
6

Generative_Models_Tutorial_with_Demo

Generative Models Tutorial with Demo: Bayesian Classifier Sampling, Variational Auto Encoder (VAE), Generative Adversial Networks (GANs), Popular GANs Architectures, Auto-Regressive Models, Important Generative Model Papers, Courses, etc..
Jupyter Notebook
326
star
7

Fast-Terraform

This repo covers Terraform (Infrastructure as Code) with LABs using AWS and AWS Sample Projects: Resources, Variables, Meta Arguments, Provisioners, Dynamic Blocks, Modules, Provisioning AWS Resources (EC2, EBS, EFS, VPC, IAM Policies, Roles, ECS, ECR, Fargate, EKS, Lambda, API-Gateway, ELB, S3, etc.
HCL
274
star
8

CNN-TA

Algorithmic Financial Trading with Deep Convolutional Neural Networks: Time Series to Image Conversion Approach: A novel algorithmic trading model CNN-TA using a 2-D convolutional neural network based on image processing properties.
Java
108
star
9

Fast-Kubeflow

This repo covers Kubeflow Environment with LABs: Kubeflow GUI, Jupyter Notebooks on pods, Kubeflow Pipelines, Experiments, KALE, KATIB (AutoML: Hyperparameter Tuning), KFServe (Model Serving), Training Operators (Distributed Training), Projects, etc.
Python
75
star
10

SparkDeepMlpGADow30

A Deep Neural-Network based (Deep MLP) Stock Trading System based on Evolutionary (Genetic Algorithm) Optimized Technical Analysis Parameters (using Apache Spark MLlib)
Java
64
star
11

TimeSeries2DBarChartImageCNN

Conversion of the time series values to 2-D stock bar chart images and prediction using CNN (using Keras-Tensorflow)
Python
38
star
12

SparkMlpDow30

A new stock trading and prediction model based on a MLP neural network utilizing technical analysis indicator values as features (using Apache Spark MLlib)
Java
35
star
13

AIMap

Map of Artificial Intelligence: Classifications, Approaches, Algorithms, Libraries, Tools, State of Art Studies, Awesome Repos, etc..
21
star
14

PolicyGradient_PongGame

Pong Game problem solving using RL - Policy Gradient with OpenAI Gym Framework and Tensorflow
Python
14
star
15

Qlearning_MountainCar

Mountain Car problem solving using RL - QLearning with OpenAI Gym Framework
Python
8
star
16

IoTSmartHomeOntologySimulator

A smart home sensor ontology (that is a specialized ontology based on the Semantic Sensor Networks (SSN) ontology) and simulation evironment of a smart home use case
Java
5
star
17

SentimentAnalysis

Sentences are classified in 5 different sentiment using LSTM (Keras). Results are expressed with emoji characters.
Python
5
star
18

NeuralStyleTransfer

Art/Painting Generation using AI (Neural Style Transfer) using Tensorflow
Python
5
star
19

BasicLSTM

The aim of this implementation is to help to learn structure of basic LSTM (LSTM cell forward, LSTM cell backward, etc..)
Python
4
star
20

IoTWeatherSensorsAnalysis

Proposed "An Extended IoT Framework" learning part is presented with a use case on weather data clustering analysis. Sensor faults and anomalies are determined using K-means clustering (using scikit-learn)
Python
4
star
21

MusicGeneration

Music generation with LSTM model (Keras)
Python
3
star
22

QLearning_CartPole

Cart Pole problem solving using RL - QLearning with OpenAI Gym Framework
Python
2
star
23

modelbuild_pipeline

Test Repo for Model Build
Python
1
star