Laurae

Advanced High Performance Data Science Toolbox for R by Laurae

me = wants download

devtools::install_github("Laurae2/Laurae")

Latest News (DD/MM/YYYY)

24/03/2017: Added Xgboard, an interactive dashboard for visualizing xgboost training, whether you are on computer, on your phone, on a tablet... by setting up a server accessible using a web browser (Google Chrome, Firefox...). Supports only Accuracy and Timing, more to come soon!

04/03/2017: Added Deep Forest implementation in R using xgboost, which may provide similar performance versus very simple Convolutional Neural Networks (CNNs), and slightly better results than boosted models. You can find the paper here. Supported: Complete-Random Tree Forest, Cascade Forest, Multi-Grained Scanning, Deep Forest. You can use Gradient Boosting to get a sort of "Deep Boosting" model.

Benchmark on MNIST 2,000 samples for training, 10,000 samples for testing, i7-4600U, 3-fold cross-validation (Cascade Forest with poor parameters for speed, Multi-Grained Scanning with poor parameters for speed):

10/02/2017: Added Partial Dependence Analysis, currently a skeleton but I will build more on it. It is fully working for the analysis of single observations against an amount of features you specify. The multiple observation version is not yet working when it comes to analyzing statistically the results.

30/01/2017: Added "Lextravagenza", a machine learning model based on xgboost ignoring past gradient/hessian for optimization, but allowing dynamic trees to outperform small boosted trees.

09/01/2017: My LightGBM PR for easy installation in R has been merged in LightGBM official repository. When I will get time to work more on it (harvest metric, harvest feature importance, save/load models), I will update this package and get rid of the old LightGBM wrapper. This way, one will be able to use the latest versions of LightGBM, instead of being stuck with the (old) PR 33 of LightGBM.

08/01/2017: I'm starting to work on an automated machine learning model / stacker.

What is Data Science

What can I do with it?

Mostly... in a nutshell:

Supervised Learning:

• Deep Forest Implementation: first implementation in R of Complete-Random Tree Forest, Cascade Forest, Multi-Grained Scanning, and Deep Forest. Read more on this paper.
• (Soon Deprecated) Use LightGBM in R (first wrapper available in R for LightGBM) tuned for maximum I/O without using in-memory dataset moves (which is both a good and bad thing! - 10GB of data takes 4 mins of travel in a HDD) and use feature importance with smart and readable plots - I recommend using official LightGBM R Package which I contribute to
• Automated Machine Learning from a set of features and hyperparameters (provide algorithm functions, features, hyperparamters, and a stochastic optimizer does the job for you with full logging if required)
• Use a repeated cross-validated xgboost (Extreme Gradient Boosting)
• Get pretty interactive feature importance tables of xgboost ready-to-use for markdown documents
• Throw supervised rules using outliers anywhere you feel it appropriate (univariate, bivariate)
• Create cross-validated and repeated cross-validated folds for supervised learning with more options for creating them (like batch creation - those ones can be fed into my LightGBM R wrapper for extensive analysis of feature behavior)
• Feature Engineering Assistant (mostly non-linear version) using automated decision trees
• Dictionary of loss functions and ready to input into xgboost (currently: Absolute Error, Squared Error, Cubic Error, Loglikelihood Error, Poisson Error, Kullback-Leibler Error)
• Symbolic Derivaton for custom loss functions (finding gradient/hessian painlessly)
• Lextravagenza model (dynamic boosted trees) which are good for small boosting iterations, bad for high boosting iterations (good for diversity)
• Partial dependency analysis for single observation: the way to get insights on why a black box made a specific decision!

Unsupervised Learning:

• Auto-tune t-SNE (t-Distributed Stochastic Neighbor Embedding), but it comes already with premade hyperparameters tuned for minimal reproduction loss!

Automated Reporting for Machine Learning:

• Generate an in-depth automated report for linear regression with interactive elements.
• Generate an in-depth automated report for xgboost regression/classification with interactive elements, with unbiased feature importance computations

Interactive Analysis:

• Discover and optimize gradient and hessian functions interactively in real-time
• Plot up to 1 dependent variable, 2 independent variables, 2 conditioning variables, and 1 weighting variable for Exploratory Data Analysis using ggplot, in real-time
• Plot up to three variables for Exploratory Data Analysis using 3djs via NVD3, in real-time
• Plot several variables for Exploratory Data Analysis using 3djs via Plotly/ggplot, in real-time
• Discover rule-based (from decision trees) non-linear relationship between variables, with rules ready to be copied and pasted for data.tables
• Visualize interactively Color Brewer palettes with unlimited colors (unlike the original palettes), with ready to copy&paste color codes as vectors
• Monitor xgboost training in real time

Optimization:

• Do feature selection & hyperparameter optimization using Cross-Entropy optimization & Elite optimization
• Do the same optimization but with any variable (continuous, ordinal, discrete) for any function using fully personalized callbacks (which is both a great thing and a hassle for the user) and a personalized training backend (by default it uses xgboost as the predictor for next steps, you can modify it by another (un)supervised machine learning model!)
• Symbolic Derivaton for custom loss functions (finding gradient/hessian painlessly)

Improvements & Extras:

• Improve data.table memory efficiency by up to 3X while keeping a large part of its performance (best of both worlds? isn't that insane?)
• Improve Cross-Entropy optimization by providing a more powerful frontend (at the expense of the user's necessary knowledge) in order to converge better on feature selection & but slower on hyperparameter optimization of black boxes
• Load sparse data directly as dgCMatrix (sparse matrix)
• Plot massive amount of data in an easily readable picture
• Add unlimited colors to the Color Brewer palettes
• Add the ability to add linear equation coefficient to ggplot facets
• Add multiplot ggplot

Sparsity SVMLight converter benchmark:

• Benchmark to convert a dgCMatrix with 2,500,000 rows and 8,500 columns (1.1GB in memory) => 5 minutes
• I think it needs minimum hours if not days for the other existing converters for such size.
• Currently not merged on this repository: see https://github.com/Laurae2/sparsity !

Nice pictures:

• Partial Dependence for single observation analysis (5-variate example):

• Partial Dependence for multiple observation analysis (univariate example):

• LightGBM Feature Importance:

• xgboost Interactive Feature Importance:

• Automated Reporting with pretty tables:

• Interactive Symbolic Derivation:

• Interactive EDA using 3djs/Plotly/ggplot2:

• Interactive Feature Engineering Assistant:

• Deep Forest example:

Installing this package? (Unproper installation)

Proper version is at the end of this page.

If you already installed this package in the past, or you want to install this package super fast because you want the functions, run in R:

devtools::install_github("Laurae2/Laurae")

Running in a Virtual Machine and/or have no proxy redirection from R? Use the following alternative:

devtools::install_git("git://github.com/Laurae2/Laurae.git")

Need all R dependencies in one shot?:

devtools:::install_github("ramnathv/rCharts")
install.packages("https://cran.r-project.org/src/contrib/Archive/tabplot/tabplot_1.1.tar.gz", repos=NULL, type="source")
install.packages(c("data.table", "foreach", "doParallel", "rpart", "rpart.plot", "partykit", "tabplot", "partykit", "ggplot2", "ggthemes", "plotluck", "grid", "gridExtra", "RColorBrewer", "lattice", "car", "CEoptim", "DT", "formattable", "rmarkdown", "shiny", "shinydashboard", "miniUI", "Matrix", "matrixStats", "R.utils", "Rtsne", "recommenderlab", "Rcpp", "RcppArmadillo", "mgcv", "Deriv", "outliers", "MASS", "stringi"))
devtools:::install_github("Laurae2/sparsity")

Getting Failed with error: 'there is no package called 'sparsity'' ? Run install_github("Laurae2/sparsity") or install_git("git://github.com/Laurae2/sparsity.git") if you wish to hide this error or if you want to use the super fast column-compressed sparse matrix (dgCMatrix) -> SVMLight converter in R.

What you need?

If I am not missing stuff (please make a pull request if something is missing that must be added):

Manual installations:

• * Mandatory version of Microsoft/LightGBM (PR 33): https://github.com/Microsoft/LightGBM/tree/9895116d9e71a91b6722ca7ef1139c946fb608bf
• ** dmlc/xgboost PR 1855: https://github.com/dmlc/xgboost/commit/b21e658a0251d692d4e6ff3b614893c54e7d0c7c
• *** Laurae2/sparsity: https://github.com/Laurae2/sparsity

Installing dependencies?

• For LightGBM (use PR 33 please), please do NOT use: git clone --recursive https://github.com/Microsoft/LightGBM for the repository. Use my stable version which is aligned with Laurae package via git clone --recursive https://github.com/Laurae2/LightGBM. Then follow the installation steps (https://github.com/Microsoft/LightGBM/wiki/Installation-Guide).
• For xgboost, refer to my documentation for installing in MinGW: https://github.com/dmlc/xgboost/tree/master/R-package - If you encounter strange issues in Windows (like permission denied, etc.), please read: https://medium.com/@Laurae2/compiling-xgboost-in-windows-for-r-d0cb826786a5. Make sure you are using MinGW.
• sparsity: You must use Laurae's sparsity package (SVMLight I/O conversion) which can be found here: https://github.com/Laurae2/sparsity/blob/master/README.md - compilation simply requires writing devtools:::install_github("Laurae2/sparsity") (and having Rtools in Windows).
• tabplot: please use: install.packages("https://cran.r-project.org/src/contrib/Archive/tabplot/tabplot_1.1.tar.gz", repos=NULL, type="source"). The 1.3 version is "junk" since they added standard deviation which makes unreadable tableplots when it is too high, even if standard deviation is disabled.

Strange errors on first run

Sometimes you will get strange errors (like a corrupted documentation database) on the first load ever on the package. Restart R to get rid of this issue. It does not show up anymore afterwards.

Printed text is missing after interrupting LightGBM / xgboost

Write in your R console sink() until you get an error.

A lot of functions that worked are giving errors.

Write in your R console sink() until you get an error.

What is inside?

TO-DO:

• Add a super fast matrix to data.table converter
• Refactor LightGBM code
• Better handling of LightGBM arguments
• Better handling of LightGBM files
• Fuse Laurae2/sparsity 's SVMLight converter/reader and Laurae2/Laurae
• Add Differential Evolution algorithm for feature selection and hyperparameter simultaneous optimization (add another backend via another interface as it typically takes a lot of time for both) (cancelled)
• (Attempt to) Add automated non-linear feature creation using decision trees (cancelled)
• Provide more for LauraeML
• Provide dynamic shrinkage for Lextravagenza for maximally overfitting validation data

To add:

• xgboost grid search (LauraeML)
• xgboost unbalanced large dataset learning (cancelled)
• large sparse matrix loader for categorical data (cancelled)
• Categorical to Numeric converter: h2o's autoencoder, mxnet's autoencoder, t-SNE, Generalized Low Rank Models, largeVis, FeatureHashing - along with testing performance using xgboost
• Logloss brute force calibration
• Prediction Analyzer (analyze any type of model predictions, currently only binary)
• Automated Feature Creator (create automatically features using linear (^1), quadratic (^2), cubic (^3), quartic (^4), mean, and standard deviation of different random features as inputs) - all with a GUI to interrupt without losing data with full verbosity of search
• Automated Feature Analyzer (analyze created features and test against randomness of improval) - all with verbosity
• Leave-one-out encoding (encodes any categorical using a continuous variable such as the label or a feature)
• AND MANY MORE...

Extra contributors:

• @fakyras for the base R code for LightGBM.

Installing this package? (Proper installation)

If you need the modeling packages, you are going to need LightGBM and xgboost compiled. Also, xgboost requires to be installed afterwards as a R package. Using drat or CRAN version is not guaranteed to work with my package.

Linux users can skip xgboost (https://github.com/dmlc/xgboost/tree/master/R-package) and LightGBM (https://github.com/Microsoft/LightGBM/wiki/Installation-Guide) installation steps, as they are straightforward (compile source).

Windows users need MinGW (architecture x86_64) and Visual Studio 2015 Community (or any working version, starting from 2013). Prepare at least 10 GB.

xgboost (~1 GB in Windows)

This applies to Windows only. Linux users can just compile "out of the box" xgboost with the gcc tool chain and install easily the package in R.

Check first if you have RTools. If not, download a proper version here: https://cran.r-project.org/bin/windows/Rtools/

Check also whether you installed Git Bash or not. If not, install Git Bash (https://git-for-windows.github.io/).

Make sure you installed MinGW (mandatory) for x86_64 architecture.

Run in R: system('gcc -v')

• If you don't see MinGW, then edit the PATH variable appropriately so MinGW is FIRST.
• If you see MinGW, open Git Bash and run:

mkdir C:/xgboost
cd C:/xgboost
git clone --recursive https://github.com/dmlc/xgboost
cd xgboost
git submodule init
git submodule update
alias make='mingw32-make'
cd dmlc-core
make
cd ../rabit
make lib/librabit_empty.a
cd ..
cp make/mingw64_min.mk config.mk
make

This should compile xgboost perfectly out of the box on Windows. If you get an error at the last "make", it means you are not using MinGW or you messed up something in the steps.

Now, fire up an R session and run this:

setwd('C:/xgboost/xgboost/R-package')
library(devtools)
install()

If you get a "permission denied" error, go to C:\xgboost\xgboost\R-package, right-click on the “src” folder, select “Properties”:

• Under the “Security” tab, click “Edit”
• Click “Full control” to all group or user names (click on each group, click Full control for each)
• Click OK twice
• Right-click on the “src” folder, select “Properties”
• Under the “Security” tab, click “Advanced”
• Check “Replace all child object permission entries with inheritable permission entries from this object” (it is the last box at the bottom left of the opened tab).
• Click OK twice
• Run again install() in the R console

And you should have now xgboost compiled in Windows.

Check quickly that xgboost works:

library(xgboost)
set.seed(11111)
n=100
ncov=4
z=matrix(replicate(n,rnorm(ncov)),nrow=n)
alpha=c(-1,0.5,-0.25,-0.1)
za=z%*%alpha
p=exp(za)/(1+exp(za))
t=rbinom(n,1,p)
xgb.train(list(objective="binary:logitraw"), xgb.DMatrix(data=z,label=t), nrounds=10)

LightGBM installation (~10 GB in Windows)

• Deprecated soon: I recommend to use the official LightGBM R package I contribute to, it is a one-liner install in R and you do not even need Visual Studio (but only Rtools). LightGBM in Laurae's package will be deprecated soon. *

This applies to Windows only. Linux users can just compile "out of the box" LightGBM with the gcc tool chain

LightGBM use Visual Studio (2013 or higher) to build in Windows. If you do not have Visual Studio, follow this: download Visual Studio 2015 Community. It is free. When installing Visual Studio Community, use the default installation method. Otherwise, you might have random errors on the UI if you try a minimal installation. Prepare at least 8GB of free drive space. Install it with the Visual C++ additions (custom install, select the first box which has 3 subboxes - it should say you will install the Windows SDK blablabla - ignore the update failure error at the end).

Once you are done installing Visual Studio 2015 Community, reboot your computer.

Now, or if you skipped the installation step, clone the latest (CLEARLY UNRECOMMENDED) LightGBM repository by doing in Git Bash:

cd C:/xgboost
git clone --recursive https://github.com/Microsoft/LightGBM

If you want the stable (RECOMMENDED) version aligned to Laurae package, use git clone --recursive https://github.com/Laurae2/LightGBM instead. You have 99%+ guarantee to have a non-working version if you use the fully bleeding edge devel version of LightGBM with this package (well, most of the things work but it is refusing to train on data most of the times, even via direct command line).

Now the steps:

• Under C:/xgboost/LightGBM/windows, double click LightGBM.sln to open it in Visual Studio.
• Accept any warning pop up about project versioning issues (Upgrade VC++ Compiler and Libraries --> OK).
• Wait one minute for the loading.
• On the Solution Explorer, click "Solution 'LightGBM' (1 project)"
• On the bottom right tab (Properties), change the "Active config" to "Release|x64" (default is "Debug_mpi|x64")
• Compile the solution by pressing Ctrl+Shift+B (or click Build > Build Solution).
• Should everything be correct, you now have LightGBM compiled under C:\xgboost\LightGBM\windows\x64\Release

If you get an error while building (Windows SDK version blabla), then you will need the correct SDK for your OS. Start Visual Studio from scratch, click "New Project", select "Visual C++" and click "Install Visual C++ 2015 Tools for Windows Desktop". Then, attempt to build LightGBM.

If Visual Studio fails to load the "project", delete LightGBM folder and clone LightGBM repository again in Git Bash. If it still does not compile in Visual Studio, try adjusting the PATH to include the appropriate Windows SDK path. Restart Visual Studio and try compiling again. Another way: uninstall Visual Studio (using the installer), reboot, and reinstall using Custom install (and select all Visual C++ things, it must be the first box with 3 subboxes to check - which will tell you it will install the SDK etc.). Then, you should be able to compile it perfectly.

Once you compiled it (and after you installed everything else you need, like the Laurae package), create a folder named "test" in "C:/" (or any appropriate folder you have), and try to run the following in R (you will get two prompts: the first for the "temporary" directory you created, and the second for the LightGBM executable to select):

# Make sure you have data.table in case
setwd(choosedir(caption = "Select the temporary folder"))
library(Laurae)
library(stringi)

DT <- data.table(Split1 = c(rep(0, 50), rep(1, 50)), Split2 = rep(c(rep(0, 25), rep(0.5, 25)), 2))
DT$Split5 <- rep(c(rep(0, 5), rep(0.05, 5), rep(0, 10), rep(0.05, 5)), 4)
label <- as.numeric((DT$Split2 == 0) & (DT$Split1 == 0) & (DT$Split3 == 0) & (DT$Split4 == 0) | ((DT$Split2 == 0.5) & (DT$Split1 == 1) & (DT$Split3 == 0.25) & (DT$Split4 == 0.1) & (DT$Split5 == 0)) | ((DT$Split1 == 0) & (DT$Split2 == 0.5)))

trained <- lgbm.train(y_train = label,
                      x_train = DT,
                      bias_train = NA,
                      application = "binary",
                      num_iterations = 1,
                      early_stopping_rounds = 1,
                      learning_rate = 1,
                      num_leaves = 16,
                      min_data_in_leaf = 1,
                      min_sum_hessian_in_leaf = 1,
                      tree_learner = "serial",
                      num_threads = 1,
                      lgbm_path = lgbm.find(),
                      workingdir = getwd(),
                      validation = FALSE,
                      files_exist = FALSE,
                      verbose = TRUE,
                      is_training_metric = TRUE,
                      save_binary = TRUE,
                      metric = "binary_logloss")

tabplot

To have "more readable" tableplots for visualizations, you will need to install an old version of the tabplot package. You can do this by running in your R console:

install.packages("https://cran.r-project.org/src/contrib/Archive/tabplot/tabplot_1.1.tar.gz", repos=NULL, type="source")

Other packages

You can install the other packages by running in your R console:

install.packages(c("data.table", "foreach", "doParallel", "rpart", "rpart.plot", "partykit", "tabplot", "partykit", "ggplot2", "ggthemes", "plotluck", "grid", "gridExtra", "RColorBrewer", "lattice", "car", "CEoptim", "DT", "formattable", "rmarkdown", "shiny", "shinydashboard", "miniUI", "Matrix", "matrixStats", "R.utils", "Rtsne", "recommenderlab", "Rcpp", "RcppArmadillo", "mgcv", "Deriv", "outliers", "MASS", "stringi"))
devtools:::install_github("ramnathv/rCharts")
devtools:::install_github("Laurae2/sparsity")

Laurae

You can now install the Laurae package and use the fully fledged version of it.

devtools::install_github("Laurae2/Laurae")

Running in a Virtual Machine and/or have no proxy redirection from R? Use the following alternative:

devtools::install_git("git://github.com/Laurae2/Laurae.git")

Getting a package error while running install_github/install_git which is not "could not connect to server"? Make sure you have the package outlined in the error, which is required by devtools.

• (Soon Deprecated) Use LightGBM in R (first wrapper available in R for LightGBM) tuned for maximum I/O without using in-memory dataset moves (which is both a good and bad thing! - 10GB of data takes 4 mins of travel in a HDD) and use feature importance with smart and readable plots - I recommend using official LightGBM R Package which I contribute to
• Automated Machine Learning from a set of features and hyperparameters (provide algorithm functions, features, hyperparamters, and a stochastic optimizer does the job for you with full logging if required)
• Use a repeated cross-validated xgboost (Extreme Gradient Boosting)
• Get pretty interactive feature importance tables of xgboost ready-to-use for markdown documents
• Throw supervised rules using outliers anywhere you feel it appropriate (univariate, bivariate)
• Create cross-validated and repeated cross-validated folds for supervised learning with more options for creating them (like batch creation - those ones can be fed into my LightGBM R wrapper for extensive analysis of feature behavior)
• Feature Engineering Assistant (mostly non-linear version) using automated decision trees
• Dictionary of loss functions and ready to input into xgboost (currently: Absolute Error, Squared Error, Cubic Error, Loglikelihood Error, Poisson Error, Kullback-Leibler Error)
• Symbolic Derivaton for custom loss functions (finding gradient/hessian painlessly)
• Lextravagenza model (dynamic boosted trees) which are good for small boosting iterations, bad for high boosting iterations (good for diversity)
• Partial dependency analysis for single observation: the way to get insights on why a black box made a specific decision!

Unsupervised Learning:

• Auto-tune t-SNE (t-Distributed Stochastic Neighbor Embedding), but it comes already with premade hyperparameters tuned for minimal reproduction loss!

Automated Reporting for Machine Learning:

• Generate an in-depth automated report for linear regression with interactive elements.
• Generate an in-depth automated report for xgboost regression/classification with interactive elements, with unbiased feature importance computations

Interactive Analysis:

• Discover and optimize gradient and hessian functions interactively in real-time
• Plot up to 1 dependent variable, 2 independent variables, 2 conditioning variables, and 1 weighting variable for Exploratory Data Analysis using ggplot, in real-time
• Plot up to three variables for Exploratory Data Analysis using 3djs via NVD3, in real-time
• Plot several variables for Exploratory Data Analysis using 3djs via Plotly/ggplot, in real-time
• Discover rule-based (from decision trees) non-linear relationship between variables, with rules ready to be copied and pasted for data.tables
• Visualize interactively Color Brewer palettes with unlimited colors (unlike the original palettes), with ready to copy&paste color codes as vectors

Optimization:

• Do feature selection & hyperparameter optimization using Cross-Entropy optimization & Elite optimization
• Do the same optimization but with any variable (continuous, ordinal, discrete) for any function using fully personalized callbacks (which is both a great thing and a hassle for the user) and a personalized training backend (by default it uses xgboost as the predictor for next steps, you can modify it by another (un)supervised machine learning model!)
• Symbolic Derivaton for custom loss functions (finding gradient/hessian painlessly)

Improvements & Extras:

• Improve data.table memory efficiency by up to 3X while keeping a large part of its performance (best of both worlds? isn't that insane?)
• Improve Cross-Entropy optimization by providing a more powerful frontend (at the expense of the user's necessary knowledge) in order to converge better on feature selection & but slower on hyperparameter optimization of black boxes
• Load sparse data directly as dgCMatrix (sparse matrix)
• Plot massive amount of data in an easily readable picture
• Add unlimited colors to the Color Brewer palettes
• Add the ability to add linear equation coefficient to ggplot facets
• Add multiplot ggplot

Sparsity SVMLight converter benchmark:

• Benchmark to convert a dgCMatrix with 2,500,000 rows and 8,500 columns (1.1GB in memory) => 5 minutes
• I think it needs minimum hours if not days for the other existing converters for such size.
• Currently not merged on this repository: see https://github.com/Laurae2/sparsity !

Nice pictures:

• Partial Dependence for single observation analysis (5-variate example):

• Partial Dependence for multiple observation analysis (univariate example):

• LightGBM Feature Importance:

• xgboost Interactive Feature Importance:

• Automated Reporting with pretty tables:

• Interactive Symbolic Derivation:

• Interactive EDA using 3djs/Plotly/ggplot2:

• Interactive Feature Engineering Assistant:

Installing this package? (Unproper installation)

Proper version is at the end of this page.

If you already installed this package in the past, or you want to install this package super fast because you want the functions, run in R:

devtools::install_github("Laurae2/Laurae")

Running in a Virtual Machine and/or have no proxy redirection from R? Use the following alternative:

devtools::install_git("git://github.com/Laurae2/Laurae.git")

Need all R dependencies in one shot?:

devtools:::install_github("ramnathv/rCharts")
install.packages("https://cran.r-project.org/src/contrib/Archive/tabplot/tabplot_1.1.tar.gz", repos=NULL, type="source")
install.packages(c("data.table", "foreach", "doParallel", "rpart", "rpart.plot", "partykit", "tabplot", "partykit", "ggplot2", "ggthemes", "plotluck", "grid", "gridExtra", "RColorBrewer", "lattice", "car", "CEoptim", "DT", "formattable", "rmarkdown", "shiny", "shinydashboard", "miniUI", "Matrix", "matrixStats", "R.utils", "Rtsne", "recommenderlab", "Rcpp", "RcppArmadillo", "mgcv", "Deriv", "outliers", "MASS", "stringi"))
devtools:::install_github("Laurae2/sparsity")

Getting Failed with error: 'there is no package called 'sparsity'' ? Run install_github("Laurae2/sparsity") or install_git("git://github.com/Laurae2/sparsity.git") if you wish to hide this error or if you want to use the super fast column-compressed sparse matrix (dgCMatrix) -> SVMLight converter in R.

What you need?

If I am not missing stuff (please make a pull request if something is missing that must be added):

Manual installations:

• * Mandatory version of Microsoft/LightGBM (PR 33): https://github.com/Microsoft/LightGBM/tree/9895116d9e71a91b6722ca7ef1139c946fb608bf
• ** dmlc/xgboost PR 1855: https://github.com/dmlc/xgboost/commit/b21e658a0251d692d4e6ff3b614893c54e7d0c7c
• *** Laurae2/sparsity: https://github.com/Laurae2/sparsity

Installing dependencies?

• For LightGBM (use PR 33 please), please do NOT use: git clone --recursive https://github.com/Microsoft/LightGBM for the repository. Use my stable version which is aligned with Laurae package via git clone --recursive https://github.com/Laurae2/LightGBM. Then follow the installation steps (https://github.com/Microsoft/LightGBM/wiki/Installation-Guide).
• For xgboost, refer to my documentation for installing in MinGW: https://github.com/dmlc/xgboost/tree/master/R-package - If you encounter strange issues in Windows (like permission denied, etc.), please read: https://medium.com/@Laurae2/compiling-xgboost-in-windows-for-r-d0cb826786a5. Make sure you are using MinGW.
• sparsity: You must use Laurae's sparsity package (SVMLight I/O conversion) which can be found here: https://github.com/Laurae2/sparsity/blob/master/README.md - compilation simply requires writing devtools:::install_github("Laurae2/sparsity") (and having Rtools in Windows).
• tabplot: please use: install.packages("https://cran.r-project.org/src/contrib/Archive/tabplot/tabplot_1.1.tar.gz", repos=NULL, type="source"). The 1.3 version is "junk" since they added standard deviation which makes unreadable tableplots when it is too high, even if standard deviation is disabled.

Strange errors on first run

Sometimes you will get strange errors (like a corrupted documentation database) on the first load ever on the package. Restart R to get rid of this issue. It does not show up anymore afterwards.

Printed text is missing after interrupting LightGBM / xgboost

Write in your R console sink() until you get an error.

A lot of functions that worked are giving errors.

Write in your R console sink() until you get an error.

What is inside?

TO-DO:

• Add a super fast matrix to data.table converter
• Refactor LightGBM code
• Better handling of LightGBM arguments
• Better handling of LightGBM files
• Fuse Laurae2/sparsity 's SVMLight converter/reader and Laurae2/Laurae
• Add Differential Evolution algorithm for feature selection and hyperparameter simultaneous optimization (add another backend via another interface as it typically takes a lot of time for both) (cancelled)
• (Attempt to) Add automated non-linear feature creation using decision trees (cancelled)
• Provide more for LauraeML
• Provide dynamic shrinkage for Lextravagenza for maximally overfitting validation data

To add:

• xgboost grid search (LauraeML)
• xgboost unbalanced large dataset learning (cancelled)
• large sparse matrix loader for categorical data (cancelled)
• Categorical to Numeric converter: h2o's autoencoder, mxnet's autoencoder, t-SNE, Generalized Low Rank Models, largeVis, FeatureHashing - along with testing performance using xgboost
• Logloss brute force calibration
• Prediction Analyzer (analyze any type of model predictions, currently only binary)
• Automated Feature Creator (create automatically features using linear (^1), quadratic (^2), cubic (^3), quartic (^4), mean, and standard deviation of different random features as inputs) - all with a GUI to interrupt without losing data with full verbosity of search
• Automated Feature Analyzer (analyze created features and test against randomness of improval) - all with verbosity
• Leave-one-out encoding (encodes any categorical using a continuous variable such as the label or a feature)
• AND MANY MORE...

Extra contributors:

• @fakyras for the base R code for LightGBM.

Installing this package? (Proper installation)

If you need the modeling packages, you are going to need LightGBM and xgboost compiled. Also, xgboost requires to be installed afterwards as a R package. Using drat or CRAN version is not guaranteed to work with my package.

Linux users can skip xgboost (https://github.com/dmlc/xgboost/tree/master/R-package) and LightGBM (https://github.com/Microsoft/LightGBM/wiki/Installation-Guide) installation steps, as they are straightforward (compile source).

Windows users need MinGW (architecture x86_64) and Visual Studio 2015 Community (or any working version, starting from 2013). Prepare at least 10 GB.

xgboost (~1 GB in Windows)

This applies to Windows only. Linux users can just compile "out of the box" xgboost with the gcc tool chain and install easily the package in R.

Check first if you have RTools. If not, download a proper version here: https://cran.r-project.org/bin/windows/Rtools/

Check also whether you installed Git Bash or not. If not, install Git Bash (https://git-for-windows.github.io/).

Make sure you installed MinGW (mandatory) for x86_64 architecture.

Run in R: system('gcc -v')

• If you don't see MinGW, then edit the PATH variable appropriately so MinGW is FIRST.
• If you see MinGW, open Git Bash and run:

mkdir C:/xgboost
cd C:/xgboost
git clone --recursive https://github.com/dmlc/xgboost
cd xgboost
git submodule init
git submodule update
alias make='mingw32-make'
cd dmlc-core
make
cd ../rabit
make lib/librabit_empty.a
cd ..
cp make/mingw64_min.mk config.mk
make

This should compile xgboost perfectly out of the box on Windows. If you get an error at the last "make", it means you are not using MinGW or you messed up something in the steps.

Now, fire up an R session and run this:

setwd('C:/xgboost/xgboost/R-package')
library(devtools)
install()

If you get a "permission denied" error, go to C:\xgboost\xgboost\R-package, right-click on the “src” folder, select “Properties”:

• Under the “Security” tab, click “Edit”
• Click “Full control” to all group or user names (click on each group, click Full control for each)
• Click OK twice
• Right-click on the “src” folder, select “Properties”
• Under the “Security” tab, click “Advanced”
• Check “Replace all child object permission entries with inheritable permission entries from this object” (it is the last box at the bottom left of the opened tab).
• Click OK twice
• Run again install() in the R console

And you should have now xgboost compiled in Windows.

Check quickly that xgboost works:

library(xgboost)
set.seed(11111)
n=100
ncov=4
z=matrix(replicate(n,rnorm(ncov)),nrow=n)
alpha=c(-1,0.5,-0.25,-0.1)
za=z%*%alpha
p=exp(za)/(1+exp(za))
t=rbinom(n,1,p)
xgb.train(list(objective="binary:logitraw"), xgb.DMatrix(data=z,label=t), nrounds=10)

LightGBM installation (~10 GB in Windows)

• Deprecated soon: I recommend to use the official LightGBM R package I contribute to, it is a one-liner install in R and you do not even need Visual Studio (but only Rtools). LightGBM in Laurae's package will be deprecated soon. *

This applies to Windows only. Linux users can just compile "out of the box" LightGBM with the gcc tool chain

LightGBM use Visual Studio (2013 or higher) to build in Windows. If you do not have Visual Studio, follow this: download Visual Studio 2015 Community. It is free. When installing Visual Studio Community, use the default installation method. Otherwise, you might have random errors on the UI if you try a minimal installation. Prepare at least 8GB of free drive space. Install it with the Visual C++ additions (custom install, select the first box which has 3 subboxes - it should say you will install the Windows SDK blablabla - ignore the update failure error at the end).

Once you are done installing Visual Studio 2015 Community, reboot your computer.

Now, or if you skipped the installation step, clone the latest (CLEARLY UNRECOMMENDED) LightGBM repository by doing in Git Bash:

cd C:/xgboost
git clone --recursive https://github.com/Microsoft/LightGBM

If you want the stable (RECOMMENDED) version aligned to Laurae package, use git clone --recursive https://github.com/Laurae2/LightGBM instead. You have 99%+ guarantee to have a non-working version if you use the fully bleeding edge devel version of LightGBM with this package (well, most of the things work but it is refusing to train on data most of the times, even via direct command line).

Now the steps:

• Under C:/xgboost/LightGBM/windows, double click LightGBM.sln to open it in Visual Studio.
• Accept any warning pop up about project versioning issues (Upgrade VC++ Compiler and Libraries --> OK).
• Wait one minute for the loading.
• On the Solution Explorer, click "Solution 'LightGBM' (1 project)"
• On the bottom right tab (Properties), change the "Active config" to "Release|x64" (default is "Debug_mpi|x64")
• Compile the solution by pressing Ctrl+Shift+B (or click Build > Build Solution).
• Should everything be correct, you now have LightGBM compiled under C:\xgboost\LightGBM\windows\x64\Release

If you get an error while building (Windows SDK version blabla), then you will need the correct SDK for your OS. Start Visual Studio from scratch, click "New Project", select "Visual C++" and click "Install Visual C++ 2015 Tools for Windows Desktop". Then, attempt to build LightGBM.

If Visual Studio fails to load the "project", delete LightGBM folder and clone LightGBM repository again in Git Bash. If it still does not compile in Visual Studio, try adjusting the PATH to include the appropriate Windows SDK path. Restart Visual Studio and try compiling again. Another way: uninstall Visual Studio (using the installer), reboot, and reinstall using Custom install (and select all Visual C++ things, it must be the first box with 3 subboxes to check - which will tell you it will install the SDK etc.). Then, you should be able to compile it perfectly.

Once you compiled it (and after you installed everything else you need, like the Laurae package), create a folder named "test" in "C:/" (or any appropriate folder you have), and try to run the following in R (you will get two prompts: the first for the "temporary" directory you created, and the second for the LightGBM executable to select):

# Make sure you have data.table in case
setwd(choosedir(caption = "Select the temporary folder"))
library(Laurae)
library(stringi)

DT <- data.table(Split1 = c(rep(0, 50), rep(1, 50)), Split2 = rep(c(rep(0, 25), rep(0.5, 25)), 2))
DT$Split5 <- rep(c(rep(0, 5), rep(0.05, 5), rep(0, 10), rep(0.05, 5)), 4)
label <- as.numeric((DT$Split2 == 0) & (DT$Split1 == 0) & (DT$Split3 == 0) & (DT$Split4 == 0) | ((DT$Split2 == 0.5) & (DT$Split1 == 1) & (DT$Split3 == 0.25) & (DT$Split4 == 0.1) & (DT$Split5 == 0)) | ((DT$Split1 == 0) & (DT$Split2 == 0.5)))

trained <- lgbm.train(y_train = label,
                      x_train = DT,
                      bias_train = NA,
                      application = "binary",
                      num_iterations = 1,
                      early_stopping_rounds = 1,
                      learning_rate = 1,
                      num_leaves = 16,
                      min_data_in_leaf = 1,
                      min_sum_hessian_in_leaf = 1,
                      tree_learner = "serial",
                      num_threads = 1,
                      lgbm_path = lgbm.find(),
                      workingdir = getwd(),
                      validation = FALSE,
                      files_exist = FALSE,
                      verbose = TRUE,
                      is_training_metric = TRUE,
                      save_binary = TRUE,
                      metric = "binary_logloss")

tabplot

To have "more readable" tableplots for visualizations, you will need to install an old version of the tabplot package. You can do this by running in your R console:

install.packages("https://cran.r-project.org/src/contrib/Archive/tabplot/tabplot_1.1.tar.gz", repos=NULL, type="source")

Other packages

You can install the other packages by running in your R console:

install.packages(c("data.table", "foreach", "doParallel", "rpart", "rpart.plot", "partykit", "tabplot", "partykit", "ggplot2", "ggthemes", "plotluck", "grid", "gridExtra", "RColorBrewer", "lattice", "car", "CEoptim", "DT", "formattable", "rmarkdown", "shiny", "shinydashboard", "miniUI", "Matrix", "matrixStats", "R.utils", "Rtsne", "recommenderlab", "Rcpp", "RcppArmadillo", "mgcv", "Deriv", "outliers", "MASS", "stringi"))
devtools:::install_github("ramnathv/rCharts")
devtools:::install_github("Laurae2/sparsity")

Laurae

You can now install the Laurae package and use the fully fledged version of it.

devtools::install_github("Laurae2/Laurae")

Running in a Virtual Machine and/or have no proxy redirection from R? Use the following alternative:

devtools::install_git("git://github.com/Laurae2/Laurae.git")

Getting a package error while running install_github/install_git which is not "could not connect to server"? Make sure you have the package outlined in the error, which is required by devtools.