automl/SMAC3 automl/SMAC3

  • Stars
    star
    1,067
  • Rank 43,306 (Top 0.9 %)
  • Language
    Python
  • License
    Other
  • Created about 8 years ago
  • Updated about 2 months ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
automl/SMAC3
github

automl

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

Tests Documentation codecov Status

SMAC offers a robust and flexible framework for Bayesian Optimization to support users in determining well-performing hyperparameter configurations for their (Machine Learning) algorithms, datasets and applications at hand. The main core consists of Bayesian Optimization in combination with an aggressive racing mechanism to efficiently decide which of two configurations performs better.

SMAC3 is written in Python3 and continuously tested with Python 3.8, 3.9, and 3.10. Its Random Forest is written in C++. In further texts, SMAC is representatively mentioned for SMAC3.

Documentation

Roadmap

Important: Changes in v2.0

With the next big major release of SMAC, we drastically boosted the user experience by improving the APIs and how the pipelining is done (see changelog). All facades/intensifiers support multi-objective, multi-fidelity, and multi-threading natively now! That includes having an ask-and-tell interface and continuing a run wherever you left off. pSMAC is removed because when specifying the number of workers, SMAC automatically uses multi-threading for evaluating trials. When cleaning the code base, however, we removed the command-line interface (calling a target function from a script is still supported), and runtime optimization. Also, python 3.7 is not supported anymore. If you depend on those functionalities, please keep using v1.4.

We are excited to introduce the new major release and look forward to developing new features on the new code base. We hope you enjoy this new user experience as much as we do. πŸš€

Installation

This instruction is for the installation on a Linux system, for Windows and Mac and further information see the documentation.

Create a new environment with python 3.10 and make sure swig is installed either on your system or inside the environment. We demonstrate the installation via anaconda in the following:

Create and activate environment:

conda create -n SMAC python=3.10
conda activate SMAC

Install swig:

conda install gxx_linux-64 gcc_linux-64 swig

Install SMAC via PyPI:

pip install smac

If you want to contribute to SMAC, use the following steps instead:

git clone https://github.com/automl/SMAC3.git && cd SMAC3
make install-dev

Minimal Example

from ConfigSpace import Configuration, ConfigurationSpace

import numpy as np
from smac import HyperparameterOptimizationFacade, Scenario
from sklearn import datasets
from sklearn.svm import SVC
from sklearn.model_selection import cross_val_score

iris = datasets.load_iris()


def train(config: Configuration, seed: int = 0) -> float:
    classifier = SVC(C=config["C"], random_state=seed)
    scores = cross_val_score(classifier, iris.data, iris.target, cv=5)
    return 1 - np.mean(scores)


configspace = ConfigurationSpace({"C": (0.100, 1000.0)})

# Scenario object specifying the optimization environment
scenario = Scenario(configspace, deterministic=True, n_trials=200)

# Use SMAC to find the best configuration/hyperparameters
smac = HyperparameterOptimizationFacade(scenario, train)
incumbent = smac.optimize()

More examples can be found in the documentation.

Visualization via DeepCAVE

With DeepCAVE (Repo, Paper) you can visualize your SMAC runs. It is a visualization and analysis tool for AutoML (especially for the sub-problem hyperparameter optimization) runs.

License

This program is free software: you can redistribute it and/or modify it under the terms of the 3-clause BSD license (please see the LICENSE file).

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

You should have received a copy of the 3-clause BSD license along with this program (see LICENSE file). If not, see here.

Contacting us

If you have trouble using SMAC, a concrete question or found a bug, please create an issue. This is the easiest way to communicate about these things with us.

For all other inquiries, please write an email to smac[at]ai[dot]uni[dash]hannover[dot]de.

Miscellaneous

SMAC3 is developed by the AutoML Groups of the Universities of Hannover and Freiburg.

If you have found a bug, please report to issues. Moreover, we are appreciating any kind of help. Find our guidelines for contributing to this package here.

If you use SMAC in one of your research projects, please cite our JMLR paper:

@article{JMLR:v23:21-0888,
  author  = {Marius Lindauer and Katharina Eggensperger and Matthias Feurer and AndrΓ© Biedenkapp and Difan Deng and Carolin Benjamins and Tim Ruhkopf and RenΓ© Sass and Frank Hutter},
  title   = {SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {54},
  pages   = {1--9},
  url     = {http://jmlr.org/papers/v23/21-0888.html}
}

Copyright (C) 2016-2022 AutoML Group.

More Repositories

1

auto-sklearn

Automated Machine Learning with scikit-learn
Python
7,574
star
2

Auto-PyTorch

Automatic architecture search and hyperparameter optimization for PyTorch
Python
2,358
star
3

TabPFN

Official implementation of the TabPFN paper (https://arxiv.org/abs/2207.01848) and the tabpfn package.
Python
1,187
star
4

HpBandSter

a distributed Hyperband implementation on Steroids
Python
609
star
5

NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Python
521
star
6

RoBO

RoBO: a Robust Bayesian Optimization framework
Python
480
star
7

autoweka

Auto-WEKA
Java
330
star
8

awesome-transformer-search

A curated list of awesome resources combining Transformers with Neural Architecture Search
254
star
9

ConfigSpace

Domain specific language for configuration spaces in Python. Useful for hyperparameter optimization and algorithm configuration.
Python
202
star
10

TransformersCanDoBayesianInference

Official Implementation of "Transformers Can Do Bayesian Inference", the PFN paper
Python
183
star
11

HPOlib

HPOlib is a hyperparameter optimization library. It provides a common interface to three state of the art hyperparameter optimization packages: SMAC, spearmint and hyperopt. This package is discontinued, please read the longer note in the info box below.
Python
167
star
12

RobustDARTS

Understanding and Robustifying DARTS
Python
153
star
13

trivialaugment

This is the official implementation of TrivialAugment and a mini-library for the application of multiple image augmentation strategies including RandAugment and TrivialAugment.
Python
147
star
14

HPOBench

Collection of hyperparameter optimization benchmark problems
Python
137
star
15

pybnn

Bayesian neural network package
Jupyter Notebook
135
star
16

CARL

Benchmarking RL generalization in an interpretable way.
Python
129
star
17

CAAFE

Semi-automatic feature engineering process using Language Models and your dataset descriptions. Based on the paper "LLMs for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering" by Hollmann, MΓΌller, and Hutter (2023).
Python
124
star
18

nas_benchmarks

Python
91
star
19

ParameterImportance

Parameter Importance Analysis Tool
Python
75
star
20

nasbench301

Python
74
star
21

DEHB

Python
71
star
22

DeepCAVE

An interactive framework to visualize and analyze your AutoML process in real-time.
Python
70
star
23

HPOlib1.5

Python
69
star
24

nasbench-1shot1

Python
67
star
25

BOAH

BOAH: Bayesian Optimization & Analysis of Hyperparameters
Python
67
star
26

amltk

A build-it-yourself AutoML Framework
Python
62
star
27

labwatch

An extension to Sacred for automated hyperparameter optimization.
Python
59
star
28

learna

End-to-end RNA Design using deep reinforcement learning
Python
57
star
29

neps

Neural Pipeline Search (NePS): Helps deep learning experts find the best neural pipeline.
Python
54
star
30

CAVE

[deprecated] Configuration Assessment, Visualization and Evaluation
Python
46
star
31

PFNs

Our maintained PFN repository. Come here to train SOTA PFNs.
Python
44
star
32

zero-shot-automl-with-pretrained-models

Official repository for the paper "Zero-Shot AutoML with Pretrained Models"
Python
41
star
33

random_forest_run

C++
35
star
34

AutoFolio

Automated Algorithm Selection with Hyperparameter Optimization
Python
35
star
35

SEARL

Sample-Efficient Automated Deep Reinforcement Learning
Python
34
star
36

nes

Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
Python
31
star
37

LCBench

A learning curve benchmark on OpenML data
Jupyter Notebook
29
star
38

DACBench

A benchmark library for Dynamic Algorithm Configuration.
PDDL
28
star
39

mdp-playground

A python package to design and debug RL agents.
Python
28
star
40

RNAformer

Scalable Deep Learning for RNA Secondary Structure Prediction
Python
27
star
41

auto-sklearn-talks

Presentations on Auto-sklearn
Jupyter Notebook
24
star
42

multi-obj-baselines

Python
22
star
43

PFNs4BO

The official implementation of PFNs4BO: In-Context Learning for Bayesian Optimization
Jupyter Notebook
22
star
44

CVPR24-MedSAM-on-Laptop

Data-aware Fine-Tuning (DAFT) Code related to the CPVR24 Competition for Image Segmentation on a Laptop.
Python
22
star
45

tabpfn-client

Python
21
star
46

learning_environments

Python
20
star
47

pynisher

Python
20
star
48

DE-NAS

Jupyter Notebook
19
star
49

DAC

Dynamic Algorithm Configuration
Jupyter Notebook
19
star
50

nas-bench-x11

Python
18
star
51

ProbTransformer

Probabilistic Transformer: Modelling Ambiguities and Distributions for RNA Folding and Molecule Design
Python
17
star
52

Mighty

The Mighty cRL library you've been looking for! πŸ’ͺ
Python
17
star
53

TempoRL

Python
15
star
54

HPO_for_RL

This is the code of reproducing the results of our paper: On the importance of Hyperparameter Optimization for Model-based Reinforcement Learning
Python
15
star
55

hierarchical_nas_construction

Official repository for "Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars" (NeurIPS 2023)
Python
14
star
56

transfer-hpo-framework

Code accompanying https://arxiv.org/abs/1802.02219
Python
14
star
57

jahs_bench_201

The first collection of surrogate benchmarks for Joint Architecture and Hyperparameter Search.
Python
14
star
58

Squirrel-Optimizer-BBO-NeurIPS20-automlorg

Python
13
star
59

SVGe

Smooth Variational Graph Embeddings for Efficient Neural Architecture Search
Python
13
star
60

is_mamba_capable_of_icl

Jupyter Notebook
12
star
61

ChaLearn_Automatic_Machine_Learning_Challenge_2015

Python
11
star
62

GenericWrapper4AC

C++
11
star
63

HW-GPT-Bench

HW-GPT-Bench: Hardware-Aware Architecture Benchmark for Language Models
Python
11
star
64

QTT

A framework for QuickTune
Jupyter Notebook
11
star
65

ifBO

In-context Bayesian Optimization
Python
11
star
66

arlbench

HPO and Architecture Benchmarking for RL: Dynamically, Reactive and Efficient
Python
10
star
67

lcpfn

Jupyter Notebook
10
star
68

ASKL2.0_experiments

Jupyter Notebook
9
star
69

mf-prior-bench

A collection of multi-fidelity benchmarks with first class support for user priors
Python
8
star
70

LTO-CMA

Code for the paper "Learning Step-Size Adaptation in CMA-ES"
Python
8
star
71

EfficientNAS

Python
8
star
72

HPOlibConfigSpace

Python
8
star
73

paramsklearn

Python
8
star
74

multibeep

A Multi Armed Bandit library written in C++ with Python bindings
C
8
star
75

MODNAS

Official Repo for "Multi-objective Differentiable Neural Architecture Search"
Python
8
star
76

TabularTempoRL

Code for the paper "Towards TempoRL: Learning When to Act"
Python
8
star
77

CARP-S

A Framework for Comparing N Hyperparameter Optimizers on M Benchmarks.
Jupyter Notebook
8
star
78

HPOBenchExperimentUtils

Experiment code to run large-scale experimente with HPOBench
Python
7
star
79

hypersweeper

Hydra sweeper integration of our favorite optimization packages, utilizing ask-and-tell interfaces.
Python
6
star
80

HPOlib-hpconvnet

A wrapper for James Bergstras hyperopt convnet
Python
5
star
81

SPaCE

Jupyter Notebook
5
star
82

dac4automlcomp

DAC4AutoML Competition
HTML
5
star
83

IMFAS

Implicit Multi-Fidelity Algorithm Selection
Python
5
star
84

automl_common

This repository holds shared utilities that AutoML frameworks may benefit from.
Python
5
star
85

ParameterConfigSpace

parameter configuration space parser for SMAC format
Python
5
star
86

masif

MASIF: Meta-learned Algorithm Selection using Implicit Fidelity Information
Python
5
star
87

DAC4SGD

Python
5
star
88

automl_template

A template that provides all the tools to ensure the same project setup across all AutoML packages.
Python
5
star
89

AutoRL-Landscape

Python
4
star
90

SAWEI

Jupyter Notebook
4
star
91

hydra-smac-sweeper

Sweeper plugin based on SMAC for Hydra.
Python
4
star
92

DAC4RL

DAC4RL track of DAC4AutoML competition at AutoML Conf
Python
4
star
93

pi_is_back

Repo for "PI is back! Switching Acquisition Functions in Bayesian Optimization" (NeurIPS: Gaussian Process Workshop '22)
Python
4
star
94

BO-AFS

For BO: Select Acquisition Function (Schedule) with Meta-Learned Model Per-Run
Jupyter Notebook
4
star
95

HPOlib-AutoWEKA

Python
3
star
96

AutoDLComp19

AutoDL Competition Scripts 2019
Python
3
star
97

mf-prior-exp

Python
3
star
98

ICGen

Image Classification Dataset Generator
Python
3
star
99

AutomlCup2023

Code for the AutoMLCup 2023
Python
3
star
100

FOB

Python
3
star