• Stars
    star
    455
  • Rank 96,175 (Top 2 %)
  • Language
    Python
  • License
    MIT License
  • Created about 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Myia prototyping

Myia

Myia is a new differentiable programming language. It aims to support large scale high performance computations (e.g. linear algebra) and their gradients. The main application Myia aims to support is research in artificial intelligence, in particular deep learning algorithms.

  • Define a model using a subset of Python, which is compiled to Myia (interfaces in other languages than Python may follow). This subset is general purpose and includes looping constructs and recursion. It excludes side effects and inplace operations.
  • Ask for the derivative of your model. Derivatives are fully supported for all control flow and all differentiable primitives.
  • Compile to efficient CPU and GPU code that optimizes use of your resources.

If you want to play with the current implementation, you can check out ALPHA.md

A short document explaining some of Myia's inner workings is available here

Status

Myia is currently under development and is not yet ready for use. We are optimistic about having an alpha version to play with around the start of 2020.

See Roadmap.

Motivation

Development in artificial intelligence has been undergoing a boom in the past decade, chiefly due to the success of deep neural networks. The training of a neural network is a sort of differentiable program: one writes a program to compute the output and a cost, and then one computes the derivative of that cost with respect to the model's parameters to determine how they should be updated.

Differentiation can be automated, but mainstream programming languages offer no support for this, hence the need for libraries or programming languages that can reliably support these applications.

The current leading solutions for deep learning fall in two camps:

Computation graph-based solutions such as TensorFlow, Theano and MXNet support automatic differentiation and are very well optimized, but they are not fully general, with only limited support for loops and none for general recursion. Thus models like recursive neural networks are tricky and awkward to write.

Operator overloading solutions such as PyTorch or Autograd use a dynamic approach to automatic differentiation which makes them much more general, but they are tightly coupled to the Python language and cannot reap the benefits of an optimizing compiler. They also involve a certain quantity of overhead per operation which discourages composing small cheap operations.

Myia's solution is to define a strongly-typed, general-purpose intermediate representation with an IR-level automatic differentiation transformation, which can then be compiled and optimized for various targets, thereby getting the best of both leading approaches.

Roadmap

Current

  • Parser: Supports def, if, for, while, operators, function calls, class and methods (limited support).
  • Intermediate representation: Implemented, with an array of utilities.
  • Debug VM: Faithfully runs the IR.
  • VM: Works on the simplified/optimized IR.
  • Primitives: Scalar primitives work, as well as map, reduce, broadcasting, 2D convolutions, concat/split, and many other operations.
  • Type system: Types are inferred without the need for annotations. Shapes can also be inferred. Myia supports recursive ADTs (e.g. tree data structures).
  • Optimization: Pattern-based optimizations, inlining, constant propagation, common subexpression elimination, closure conversion.
  • Automatic differentiation: Second order differentiation is not yet in working order.
  • GPU support: Using Relay or PyTorch.

In development

  • Compiler optimization: The compiler currently needs to be optimized to reduce compile times.
  • Auto-monadization: We are working to support print statements and random number generation through an auto-monadization system that can automatically keep track of the IO or RNG state.

Next steps

  • Error messages: We need to make sure that every likely mistake leads to an understandable and traceable error diagnosis.

Near future

  • Serialization: Serializing optimized graphs will allow for greater performance across runs and greater portability across systems.
  • Debugger: Intent is to have a step debugger for Myia. There used to be a working one for a previous version of the IR, so this should not pose a problem.
  • More Python syntax: break/continue.

After Beta

  • Even more Python syntax: Support for these features is not certain.
    • Augmented assignment (under restrictions)
    • yield and await
  • Support other languages: Which ones depend on demand. A new language is also a possibility.

Publications

Citation

If you use Myia for a scientific paper, please cite the above paper or mention Myia in the acknowledgements. It would be great if you could also let us know about it.

More Repositories

1

blocks

A Theano framework for building and training neural networks
Python
1,155
star
2

welcome_tutorials

Various tutorials given for welcoming new students at MILA.
Jupyter Notebook
985
star
3

fuel

A data pipeline framework for machine learning
Python
864
star
4

babyai

BabyAI platform. A testbed for training agents to understand and execute language commands.
Python
681
star
5

summerschool2015

Slides and exercises for the Deep Learning Summer School 2015 programming tutorials
Jupyter Notebook
391
star
6

atari-representation-learning

Code for "Unsupervised State Representation Learning in Atari"
Python
241
star
7

platoon

Multi-GPU mini-framework for Theano
Python
195
star
8

spr

Code for "Data-Efficient Reinforcement Learning with Self-Predictive Representations"
Python
155
star
9

blocks-examples

Examples and scripts using Blocks
Python
147
star
10

summerschool2016

Montréal Deep Learning Summer School 2016 material
Jupyter Notebook
100
star
11

paperoni

Search for scientific papers on the command line
Python
97
star
12

summerschool2017

Material for the Montréal Deep Learning Summer School 2017
Jupyter Notebook
78
star
13

gene-graph-conv

Towards Gene Expression Convolutions using Gene Interaction Graphs
Jupyter Notebook
74
star
14

milatools

Tools to connect to and interact with the Mila cluster
Python
60
star
15

Conscious-Planning

Implementation for paper "A Consciousness-Inspired Planning Agent for Model-Based Reinforcement Learning".
Python
58
star
16

SGI

Official code for "Pretraining Representations For Data-Efficient Reinforcement Learning" (NeurIPS 2021)
Python
51
star
17

ddxplus

Python
48
star
18

picklable-itertools

itertools. But picklable.
Python
38
star
19

climate-cooperation-competition

AI for Global Climate Cooperation: Modeling Global Climate Negotiations, Agreements, and Long-Term Cooperation in RICE-N. ai4climatecoop.org
Python
35
star
20

ivado-mila-dl-school-2019

IVADO/ Mila's Summer Deep Learning School
Jupyter Notebook
35
star
21

ivado-mila-dl-school-2021

Jupyter Notebook
33
star
22

blocks-extras

A collection of extensions to the Blocks framework
Python
27
star
23

DeepDrummer

Making the world a better place through AI-generated beats & grooves
Python
26
star
24

covid_p2p_risk_prediction

COVID19 P2P Risk Prediction Model & Dataset
Python
22
star
25

COVI-AgentSim

Covid-19 spread simulator with human mobility and intervention modeling.
Jupyter Notebook
20
star
26

Skipper

A PyTorch Implementation of Skipper
Python
20
star
27

cookiecutter-pyml

Python
19
star
28

milabench

Repository of machine learning benchmarks
Python
17
star
29

snektalk

Python
15
star
30

dlschool-ivadofr-a18

Ecole Mila/IVADO
Jupyter Notebook
12
star
31

COVI-ML

Risk model training code for Covid-19 tracing application.
Python
12
star
32

teamgrid

Multiagent gridworld for the TEAM project based on gym-minigrid
Python
12
star
33

ivado-mila-dl-school-2019-vancouver

Jupyter Notebook
11
star
34

mila-paper-webpage

Webpage template for MILA-affiliated papers
CSS
11
star
35

dlschool-ivadofr-h18

Ivado École d'hiver IVADO/MILA en apprentissage profond 2018
Jupyter Notebook
10
star
36

giving

Reactive logging
Python
9
star
37

training

Python
8
star
38

mila-docs

Mila technical documentation
8
star
39

Casande-RL

Casande-RL
Python
8
star
40

hardpicks

Deep learning dataset and benchmark for first-break detection from hardrock seismic reflection data
Python
7
star
41

ptera

Query and override internal variables in your programs
Python
5
star
42

ResearchTemplate

WIP: Research Template Repository
Python
5
star
43

mila_datamodules

Efficient Datamodules Customized for the Mila / CC clusters
Python
4
star
44

digit-detection

IFT6759 - Advanced projects in machine learning (Door Number Detection project)
Shell
4
star
45

Humanitarian_R-D

Jupyter Notebook
3
star
46

ansible-role-clockwork

Ansible role to install and configure clockwork
Jinja
3
star
47

SARC

FD#11499
Python
3
star
48

ansible-role-cobbler

Install and configure Cobbler service
Jinja
3
star
49

slurm-queue-time-pred

Slurm wait time prediction
Python
3
star
50

diffusion_for_multi_scale_molecular_dynamics

Python
3
star
51

cableinspect-ad-code

Code to prepare data and reproduce results from CableInspect-AD paper
Python
3
star
52

clockwork

Simple metrics to monitor slurm and produce reports.
Python
2
star
53

ansible-role-infiniband

Ansible role to configure InfiniBand interfaces
Jinja
2
star
54

tensorflow_dataloader

1
star
55

bcachefs

C implementation with Python 3.7 bindings of the BCacheFS
C
1
star
56

ansible-collection-proxmox

Ansible Collection to manage containers and virtual machines with Proxmox VE
1
star
57

mila-docs-chatbot

Python
1
star