• Stars
    star
    299
  • Rank 138,832 (Top 3 %)
  • Language
    MATLAB
  • License
    MIT License
  • Created almost 8 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Scientific machine learning (SciML) benchmarks, AI for science, and (differential) equation solvers. Covers Julia, Python (PyTorch, Jax), MATLAB, R

SciMLBenchmarks.jl: Benchmarks for Scientific Machine Learning (SciML) and Equation Solvers

Join the chat at https://julialang.zulipchat.com #sciml-bridged Global Docs

Build status

ColPrac: Contributor's Guide on Collaborative Practices for Community Packages SciML Code Style

SciMLBenchmarks.jl holds webpages, pdfs, and notebooks showing the benchmarks for the SciML Scientific Machine Learning Software ecosystem, including:

  • Benchmarks of equation solver implementations
  • Speed and robustness comparisons of methods for parameter estimation / inverse problems
  • Training universal differential equations (and subsets like neural ODEs)
  • Training of physics-informed neural networks (PINNs)
  • Surrogate comparisons, including radial basis functions, neural operators (DeepONets, Fourier Neural Operators), and more

The SciML Bench suite is made to be a comprehensive open source benchmark from the ground up, covering the methods of computational science and scientific computing all the way to AI for science.

Rules: Optimal, Fair, and Reproducible

These benchmarks are meant to represent good optimized coding style. Benchmarks are preferred to be run on the provided open benchmarking hardware for full reproducibility (though in some cases, such as with language barriers, this can be difficult). Each benchmark is documented with the compute devices used along with package versions for necessary reproduction. These benchmarks attempt to measure in terms of work-precision efficiency, either timing with an approximately matching the error or building work-precision diagrams for direct comparison of speed at given error tolerances.

If any of the code from any of the languages can be improved, please open a pull request.

Results

To view the results of the SciML Benchmarks, go to benchmarks.sciml.ai. By default, this will lead to the latest tagged version of the benchmarks. To see the in-development version of the benchmarks, go to https://benchmarks.sciml.ai/dev/.

Static outputs in pdf, markdown, and html reside in SciMLBenchmarksOutput.

Citing

To cite the SciML Benchmarks, please cite the following:

@article{rackauckas2019confederated,
  title={Confederated modular differential equation APIs for accelerated algorithm development and benchmarking},
  author={Rackauckas, Christopher and Nie, Qing},
  journal={Advances in Engineering Software},
  volume={132},
  pages={1--6},
  year={2019},
  publisher={Elsevier}
}

@article{DifferentialEquations.jl-2017,
 author = {Rackauckas, Christopher and Nie, Qing},
 doi = {10.5334/jors.151},
 journal = {The Journal of Open Research Software},
 keywords = {Applied Mathematics},
 note = {Exported from https://app.dimensions.ai on 2019/05/05},
 number = {1},
 pages = {},
 title = {DifferentialEquations.jl โ€“ A Performant and Feature-Rich Ecosystem for Solving Differential Equations in Julia},
 url = {https://app.dimensions.ai/details/publication/pub.1085583166 and http://openresearchsoftware.metajnl.com/articles/10.5334/jors.151/galley/245/download/},
 volume = {5},
 year = {2017}
}

Current Summary

The following is a quick summary of the benchmarks. These paint broad strokes over the set of tested equations and some specific examples may differ.

Non-Stiff ODEs

  • OrdinaryDiffEq.jl's methods are the most efficient by a good amount
  • The Vern methods tend to do the best in every benchmark of this category
  • At lower tolerances, Tsit5 does well consistently.
  • ARKODE and Hairer's dopri5/dop853 perform very similarly, but are both far less efficient than the Vern methods.
  • The multistep methods, CVODE_Adams and lsoda, tend to not do very well.
  • The ODEInterface multistep method ddeabm does not do as well as the other multistep methods.
  • ODE.jl's methods are not able to consistently solve the problems.
  • Fixed time step methods are less efficient than the adaptive methods.

Stiff ODEs

  • In this category, the best methods are much more problem dependent.
  • For smaller problems:
    • Rosenbrock23, lsoda, and TRBDF2 tend to be the most efficient at high tolerances.
    • Rodas4 and Rodas5 tend to be the most efficient at low tolerances.
  • For larger problems (Filament PDE):
    • QNDF and FBDF does the best at all normal tolerances.
    • The ESDIRK methods like TRBDF2 and KenCarp4 can come close.
  • radau is always the most efficient when tolerances go to the low extreme (1e-13)
  • Fixed time step methods tend to diverge on every tested problem because the high stiffness results in divergence of the Newton solvers.
  • ARKODE is very inconsistent and requires a lot of tweaking in order to not diverge on many of the tested problems. When it doesn't diverge, the similar algorithms in OrdinaryDiffEq.jl (KenCarp4) are much more efficient in most cases.
  • ODE.jl and GeometricIntegrators.jl fail to converge on any of the tested problems.

Dynamical ODEs

  • Higher order (generally order >=6) symplectic integrators are much more efficient than the lower order counterparts.
  • For high accuracy, using a symplectic integrator is not preferred. Their extra cost is not necessary since the other integrators are able to not drift simply due to having low enough error.
  • In this class, the DPRKN methods are by far the most efficient. The Vern methods do well for not being specific to the domain.

Non-Stiff SDEs

  • For simple 1-dimensional SDEs at low accuracy, the EM and RKMil methods can do well. Beyond that, they are simply outclassed.
  • The SRA and SRI methods both are very similar within-class on the simple SDEs.
  • SRA3 is the most efficient when applicable and the tolerances are low.
  • Generally, only low accuracy is necessary to get to sampling error of the mean.
  • The adaptive method is very conservative with error estimates.

Stiff SDEs

  • The high order adaptive methods (SRIW1) generally do well on stiff problems.
  • The "standard" low-order implicit methods, ImplicitEM and ImplicitRK, do not do well on all stiff problems. Some exceptions apply to well-behaved problems like the Stochastic Heat Equation.

Non-Stiff DDEs

  • The efficiency ranking tends to match the ODE Tests, but the cutoff from low to high tolerance is lower.
  • Tsit5 does well in a large class of problems here.
  • The Vern methods do well in low tolerance cases.

Stiff DDEs

  • The Rosenbrock methods, specifically Rodas5, perform well.

Parameter Estimation

  • Broadly two different approaches have been used, Bayesian Inference and Optimisation algorithms.
  • In general it seems that the optimisation algorithms perform more accurately but that can be attributed to the larger number of data points being used in the optimisation cases, Bayesian approach tends to be slower of the two and hence lesser data points are used, accuracy can increase if proper data is used.
  • Within the different available optimisation algorithms, BBO from the BlackBoxOptim package and GN_CRS2_LM for the global case while LD_SLSQP,LN_BOBYQA and LN_NELDERMEAD for the local case from the NLopt package perform the best.
  • Another algorithm being used is the QuadDIRECT algorithm, it gives very good results in the shorter problem case but doesn't do very well in the case of the longer problems.
  • The choice of global versus local optimization make a huge difference in the timings. BBO tends to find the correct solution for a global optimization setup. For local optimization, most methods in NLopt, like :LN_BOBYQA, solve the problem very fast but require a good initial condition.
  • The different backends options available for Bayesian method offer some tradeoffs beteween time, accuracy and control. It is observed that sufficiently high accuracy can be observed with any of the backends with the fine tuning of stepsize, constraints on the parameters, tightness of the priors and number of iterations being passed.

Interactive Notebooks

To generate the interactive notebooks, first install the SciMLBenchmarks, instantiate the environment, and then run SciMLBenchmarks.open_notebooks(). This looks as follows:

]add SciMLBenchmarks#master
]activate SciMLBenchmarks
]instantiate
using SciMLBenchmarks
SciMLBenchmarks.open_notebooks()

The benchmarks will be generated at your pwd() in a folder called generated_notebooks.

Note that when running the benchmarks, the packages are not automatically added. Thus you will need to add the packages manually or use the internal Project/Manifest tomls to instantiate the correct packages. This can be done by activating the folder of the benchmarks. For example,

using Pkg
Pkg.activate(joinpath(pkgdir(SciMLBenchmarks),"benchmarks","NonStiffODE"))
Pkg.instantiate()

will add all of the packages required to run any benchmark in the NonStiffODE folder.

Contributing

All of the files are generated from the Weave.jl files in the benchmarks folder. The generation process runs automatically, and thus one does not necessarily need to test the Weave process locally. Instead, simply open a PR that adds/updates a file in the "benchmarks" folder and the PR will generate the benchmark on demand. Its artifacts can then be inspected in the Buildkite as described below before merging. Note that it will use the Project.toml and Manifest.toml of the subfolder, so any changes to dependencies requires that those are updated.

Reporting Bugs and Issues

Report any bugs or issues at the SciMLBenchmarks repository.

Inspecting Benchmark Results

To see benchmark results before merging, click into the BuildKite, click onto Artifacts, and then investigate the trained results.

Manually Generating Files

All of the files are generated from the Weave.jl files in the benchmarks folder. To run the generation process, do for example:

]activate SciMLBenchmarks # Get all of the packages
using SciMLBenchmarks
SciMLBenchmarks.weave_file(joinpath(pkgdir(SciMLBenchmarks),"benchmarks","NonStiffODE"),"linear_wpd.jmd")

To generate all of the files in a folder, for example, run:

SciMLBenchmarks.weave_folder(joinpath(pkgdir(SciMLBenchmarks),"benchmarks","NonStiffODE"))

To generate all of the notebooks, do:

SciMLBenchmarks.weave_all()

Each of the benchmarks displays the computer characteristics at the bottom of the benchmark. Since performance-necessary computations are normally performed on compute clusters, the official benchmarks use a workstation with an AMD EPYC 7502 32-Core Processor @ 2.50GHz to match the performance characteristics of a standard node in a high performance computing (HPC) cluster or cloud computing setup.

More Repositories

1

DifferentialEquations.jl

Multi-language suite for high-performance solvers of differential equations and scientific machine learning (SciML) components. Ordinary differential equations (ODEs), stochastic differential equations (SDEs), delay differential equations (DDEs), differential-algebraic equations (DAEs), and more in Julia.
Julia
2,599
star
2

SciMLBook

Parallel Computing and Scientific Machine Learning (SciML): Methods and Applications (MIT 18.337J/6.338J)
HTML
1,841
star
3

ModelingToolkit.jl

An acausal modeling framework for automatically parallelized scientific machine learning (SciML) in Julia. A computer algebra system for integrated symbolics for physics-informed machine learning and automated transformations of differential equations
Julia
1,410
star
4

NeuralPDE.jl

Physics-Informed Neural Networks (PINN) Solvers of (Partial) Differential Equations for Scientific Machine Learning (SciML) accelerated simulation
Julia
961
star
5

DiffEqFlux.jl

Pre-built implicit layer architectures with O(1) backprop, GPUs, and stiff+non-stiff DE solvers, demonstrating scientific machine learning (SciML) and physics-informed machine learning methods
Julia
858
star
6

SciMLTutorials.jl

Tutorials for doing scientific machine learning (SciML) and high-performance differential equation solving with open source software.
CSS
710
star
7

Optimization.jl

Mathematical Optimization in Julia. Local, global, gradient-based and derivative-free. Linear, Quadratic, Convex, Mixed-Integer, and Nonlinear Optimization in one simple, fast, and differentiable interface.
Julia
708
star
8

OrdinaryDiffEq.jl

High performance ordinary differential equation (ODE) and differential-algebraic equation (DAE) solvers, including neural ordinary differential equations (neural ODEs) and scientific machine learning (SciML)
Julia
528
star
9

diffeqpy

Solving differential equations in Python using DifferentialEquations.jl and the SciML Scientific Machine Learning organization
Python
521
star
10

Catalyst.jl

Chemical reaction network and systems biology interface for scientific machine learning (SciML). High performance, GPU-parallelized, and O(1) solvers in open source software.
Julia
454
star
11

DataDrivenDiffEq.jl

Data driven modeling and automated discovery of dynamical systems for the SciML Scientific Machine Learning organization
Julia
405
star
12

SciMLSensitivity.jl

A component of the DiffEq ecosystem for enabling sensitivity analysis for scientific machine learning (SciML). Optimize-then-discretize, discretize-then-optimize, adjoint methods, and more for ODEs, SDEs, DDEs, DAEs, etc.
Julia
330
star
13

Surrogates.jl

Surrogate modeling and optimization for scientific machine learning (SciML)
Julia
328
star
14

DiffEqOperators.jl

Linear operators for discretizations of differential equations and scientific machine learning (SciML)
Julia
285
star
15

DiffEqGPU.jl

GPU-acceleration routines for DifferentialEquations.jl and the broader SciML scientific machine learning ecosystem
Julia
283
star
16

DiffEqDocs.jl

Documentation for the DiffEq differential equations and scientific machine learning (SciML) ecosystem
Julia
262
star
17

NeuralOperators.jl

DeepONets, (Fourier) Neural Operators, Physics-Informed Neural Operators, and more in Julia
Julia
262
star
18

DiffEqBase.jl

The lightweight Base library for shared types and functionality for defining differential equation and scientific machine learning (SciML) problems
Julia
254
star
19

LinearSolve.jl

LinearSolve.jl: High-Performance Unified Interface for Linear Solvers in Julia. Easily switch between factorization and Krylov methods, add preconditioners, and all in one interface.
Julia
245
star
20

Integrals.jl

A common interface for quadrature and numerical integration for the SciML scientific machine learning organization
Julia
226
star
21

NonlinearSolve.jl

High-performance and differentiation-enabled nonlinear solvers (Newton methods), bracketed rootfinding (bisection, Falsi), with sparsity and Newton-Krylov support.
Julia
226
star
22

DataInterpolations.jl

A library of data interpolation and smoothing functions
Julia
212
star
23

SciMLStyle

A style guide for stylish Julia developers
Julia
211
star
24

StochasticDiffEq.jl

Solvers for stochastic differential equations which connect with the scientific machine learning (SciML) ecosystem
Julia
209
star
25

ReservoirComputing.jl

Reservoir computing utilities for scientific machine learning (SciML)
Julia
206
star
26

Sundials.jl

Julia interface to Sundials, including a nonlinear solver (KINSOL), ODE's (CVODE and ARKODE), and DAE's (IDA) in a SciML scientific machine learning enabled manner
Julia
195
star
27

RecursiveArrayTools.jl

Tools for easily handling objects like arrays of arrays and deeper nestings in scientific machine learning (SciML) and other applications
Julia
167
star
28

MethodOfLines.jl

Automatic Finite Difference PDE solving with Julia SciML
Julia
160
star
29

diffeqr

Solving differential equations in R using DifferentialEquations.jl and the SciML Scientific Machine Learning ecosystem
R
140
star
30

JumpProcesses.jl

Build and simulate jump equations like Gillespie simulations and jump diffusions with constant and state-dependent rates and mix with differential equations and scientific machine learning (SciML)
Julia
138
star
31

SciMLBase.jl

The Base interface of the SciML ecosystem
Julia
129
star
32

NBodySimulator.jl

A differentiable simulator for scientific machine learning (SciML) with N-body problems, including astrophysical and molecular dynamics
Julia
128
star
33

ColPrac

Contributor's Guide on Collaborative Practices for Community Packages
Julia
123
star
34

DiffEqBayes.jl

Extension functionality which uses Stan.jl, DynamicHMC.jl, and Turing.jl to estimate the parameters to differential equations and perform Bayesian probabilistic scientific machine learning
Julia
121
star
35

LabelledArrays.jl

Arrays which also have a label for each element for easy scientific machine learning (SciML)
Julia
120
star
36

PolyChaos.jl

A Julia package to construct orthogonal polynomials, their quadrature rules, and use it with polynomial chaos expansions.
Julia
116
star
37

SymbolicNumericIntegration.jl

SymbolicNumericIntegration.jl: Symbolic-Numerics for Solving Integrals
Julia
116
star
38

ModelingToolkitStandardLibrary.jl

A standard library of components to model the world and beyond
Julia
112
star
39

PreallocationTools.jl

Tools for building non-allocating pre-cached functions in Julia, allowing for GC-free usage of automatic differentiation in complex codes
Julia
111
star
40

StructuralIdentifiability.jl

Fast and automatic structural identifiability software for ODE systems
Julia
110
star
41

ODE.jl

Assorted basic Ordinary Differential Equation solvers for scientific machine learning (SciML). Deprecated: Use DifferentialEquations.jl instead.
Julia
103
star
42

QuasiMonteCarlo.jl

Lightweight and easy generation of quasi-Monte Carlo sequences with a ton of different methods on one API for easy parameter exploration in scientific machine learning (SciML)
Julia
102
star
43

RuntimeGeneratedFunctions.jl

Functions generated at runtime without world-age issues or overhead
Julia
100
star
44

FEniCS.jl

A scientific machine learning (SciML) wrapper for the FEniCS Finite Element library in the Julia programming language
Julia
96
star
45

DiffEqCallbacks.jl

A library of useful callbacks for hybrid scientific machine learning (SciML) with augmented differential equation solvers
Julia
94
star
46

ExponentialUtilities.jl

Fast and differentiable implementations of matrix exponentials, Krylov exponential matrix-vector multiplications ("expmv"), KIOPS, ExpoKit functions, and more. All your exponential needs in SciML form.
Julia
93
star
47

EllipsisNotation.jl

Julia-based implementation of ellipsis array indexing notation `..`
Julia
80
star
48

EasyModelAnalysis.jl

High level functions for analyzing the output of simulations
Julia
79
star
49

AutoOptimize.jl

Automatic optimization and parallelization for Scientific Machine Learning (SciML)
Julia
78
star
50

ParameterizedFunctions.jl

A simple domain-specific language (DSL) for defining differential equations for use in scientific machine learning (SciML) and other applications
Julia
73
star
51

HighDimPDE.jl

A Julia package for Deep Backwards Stochastic Differential Equation (Deep BSDE) and Feynman-Kac methods to solve high-dimensional PDEs without the curse of dimensionality
Julia
71
star
52

SciMLExpectations.jl

Fast uncertainty quantification for scientific machine learning (SciML) and differential equations
Julia
64
star
53

MultiScaleArrays.jl

A framework for developing multi-scale arrays for use in scientific machine learning (SciML) simulations
Julia
64
star
54

SimpleNonlinearSolve.jl

Fast and simple nonlinear solvers for the SciML common interface. Newton, Broyden, Bisection, Falsi, and more rootfinders on a standard interface.
Julia
63
star
55

DiffEqNoiseProcess.jl

A library of noise processes for stochastic systems like stochastic differential equations (SDEs) and other systems that are present in scientific machine learning (SciML)
Julia
63
star
56

CellMLToolkit.jl

CellMLToolkit.jl is a Julia library that connects CellML models to the Scientific Julia ecosystem.
Julia
62
star
57

SparsityDetection.jl

Automatic detection of sparsity in pure Julia functions for sparsity-enabled scientific machine learning (SciML)
Julia
59
star
58

DelayDiffEq.jl

Delay differential equation (DDE) solvers in Julia for the SciML scientific machine learning ecosystem. Covers neutral and retarded delay differential equations, and differential-algebraic equations.
Julia
58
star
59

SciMLDocs

Global documentation for the Julia SciML Scientific Machine Learning Organization
Julia
57
star
60

DiffEqProblemLibrary.jl

A library of premade problems for examples and testing differential equation solvers and other SciML scientific machine learning tools
Julia
53
star
61

sciml.ai

The SciML Scientific Machine Learning Software Organization Website
CSS
53
star
62

DiffEqParamEstim.jl

Easy scientific machine learning (SciML) parameter estimation with pre-built loss functions
Julia
52
star
63

Static.jl

Static types useful for dispatch and generated functions.
Julia
52
star
64

GlobalSensitivity.jl

Robust, Fast, and Parallel Global Sensitivity Analysis (GSA) in Julia
Julia
51
star
65

DeepEquilibriumNetworks.jl

Implicit Layer Machine Learning via Deep Equilibrium Networks, O(1) backpropagation with accelerated convergence.
Julia
50
star
66

MinimallyDisruptiveCurves.jl

Finds relationships between the parameters of a mathematical model
Julia
49
star
67

DiffEqPhysics.jl

A library for building differential equations arising from physical problems for physics-informed and scientific machine learning (SciML)
Julia
48
star
68

OperatorLearning.jl

No need to train, he's a smooth operator
Julia
44
star
69

MuladdMacro.jl

This package contains a macro for converting expressions to use muladd calls and fused-multiply-add (FMA) operations for high-performance in the SciML scientific machine learning ecosystem
Julia
44
star
70

DiffEqDevTools.jl

Benchmarking, testing, and development tools for differential equations and scientific machine learning (SciML)
Julia
43
star
71

BoundaryValueDiffEq.jl

Boundary value problem (BVP) solvers for scientific machine learning (SciML)
Julia
42
star
72

SciMLOperators.jl

SciMLOperators.jl: Matrix-Free Operators for the SciML Scientific Machine Learning Common Interface in Julia
Julia
42
star
73

SBMLToolkit.jl

SBML differential equation and chemical reaction model (Gillespie simulations) for Julia's SciML ModelingToolkit
Julia
40
star
74

HelicopterSciML.jl

Helicopter Scientific Machine Learning (SciML) Challenge Problem
Julia
38
star
75

ADTypes.jl

Repository for automatic differentiation backend types
Julia
38
star
76

RootedTrees.jl

A collection of functionality around rooted trees to generate order conditions for Runge-Kutta methods in Julia for differential equations and scientific machine learning (SciML)
Julia
37
star
77

SciMLWorkshop.jl

Workshop materials for training in scientific computing and scientific machine learning
Julia
36
star
78

AutoOffload.jl

Automatic GPU, TPU, FPGA, Xeon Phi, Multithreaded, Distributed, etc. offloading for scientific machine learning (SciML) and differential equations
Julia
35
star
79

ModelOrderReduction.jl

High-level model-order reduction to automate the acceleration of large-scale simulations
Julia
33
star
80

ModelingToolkitCourse

A course on composable system modeling, differential-algebraic equations, acausal modeling, compilers for simulation, and building digital twins of real-world devices
Julia
33
star
81

DifferenceEquations.jl

Solving difference equations with DifferenceEquations.jl and the SciML ecosystem.
Julia
32
star
82

DASSL.jl

Solves stiff differential algebraic equations (DAE) using variable stepsize backwards finite difference formula (BDF) in the SciML scientific machine learning organization
Julia
31
star
83

FiniteVolumeMethod.jl

Solver for two-dimensional conservation equations using the finite volume method in Julia.
Julia
31
star
84

SteadyStateDiffEq.jl

Solvers for steady states in scientific machine learning (SciML)
Julia
30
star
85

TruncatedStacktraces.jl

Simpler stacktraces for the Julia Programming Language
Julia
28
star
86

PDESystemLibrary.jl

A library of systems of partial differential equations, as defined with ModelingToolkit.jl in Julia
Julia
28
star
87

DiffEqOnline

It's Angular2 business in the front, and a Julia party in the back! It's scientific machine learning (SciML) for the web
TypeScript
27
star
88

ReactionNetworkImporters.jl

Julia Catalyst.jl importers for various reaction network file formats like BioNetGen and stoichiometry matrices
Julia
26
star
89

StochasticDelayDiffEq.jl

Stochastic delay differential equations (SDDE) solvers for the SciML scientific machine learning ecosystem
Julia
25
star
90

DiffEqOnlineServer

Backend for DiffEqOnline, a webapp for scientific machine learning (SciML)
Julia
25
star
91

MathML.jl

Julia MathML parser
Julia
23
star
92

IRKGaussLegendre.jl

Implicit Runge-Kutta Gauss-Legendre 16th order (Julia)
Jupyter Notebook
23
star
93

SimpleDiffEq.jl

Simple differential equation solvers in native Julia for scientific machine learning (SciML)
Julia
22
star
94

DiffEqFinancial.jl

Differential equation problem specifications and scientific machine learning for common financial models
Julia
22
star
95

ModelingToolkitNeuralNets.jl

Symbolic-Numeric Universal Differential Equations for Automating Scientific Machine Learning (SciML)
Julia
22
star
96

SciPyDiffEq.jl

Wrappers for the SciPy differential equation solvers for the SciML Scientific Machine Learning organization
Julia
21
star
97

SciMLTutorialsOutput

Tutorials for doing scientific machine learning (SciML) and high-performance differential equation solving with open source software.
HTML
20
star
98

OptimalControl.jl

A component of the SciML scientific machine learning ecosystem for optimal control
Julia
20
star
99

MATLABDiffEq.jl

Common interface bindings for the MATLAB ODE solvers via MATLAB.jl for the SciML Scientific Machine Learning ecosystem
Julia
20
star
100

IfElse.jl

Under some conditions you may need this function
Julia
19
star