Open MLOps - A Production-focused Open-Source Machine Learning Framework

Open MLOps is a set of open-source tools carefully chosen to ease user experience of conducting machine learning experiments and deploying machine learning models. Read our announcement blog post with more background here.

We also provide a step by step set-up guide and some other getting started tutorials.

In this repository, we provide these applications as Terraform modules with which the user will be able to install them into a Kubernetes cluster. The tools we provide are the following:

• Prefect for data flow automation
• Jupyter Hub for experimenting lab
• Dask for distributed computing
• Feast for Feature Store and Serving
• MLFlow for model registry and experiment tracking
• Seldon for model deployment

Other repositories

Modules

• Jupyter Hub
• Prefect
• Dask
• MLFlow
• Feast
• Seldon

Jupyter Hub

With the Jupyter Hub, we enabled a multi-user environment in which each of them can spawn a Jupyter server to do their experiments. Users can work on different environments being able to install any library necessary to meet their needs.

We provide a default Jupyter server image that comes with most of the data science packages installed. Users can use their own Jupyter server images as well.

Configuration

Below we provide a lists of the configurable parameters available and their default values.

Proxy configuration

The proxy receives the requests from the client’s browser and forwards all requests to the Hub. In the JupyterHub docs you can find a more in-depth explanation.

* Required parameters

Authentication configuration

JupyterHub’s OAuthenticator has support for enabling your users to authenticate via a third-party OAuth2 identity provider such as GitHub.

You can configure authentication using GitHub accounts and restrict what users are authorized based on membership in a GitHub organization.

See details on how to set up a GitHub Oauth here.

If you choose not to use GitHub to authenticate users, the DummyAuthenticator will be used as default. The Dummy Authenticator lets any user log in with the given password.

The dummy password is: a-shared-secret-password.

* Required parameters ** Required when oauth_github_enable is enabled

User configuration

Single user configuration refers to the default settings for each user logged in the JupyterHub.

A user can choose a Docker image to spawn a new Jupyter server. Each Docker image can have different libraries and environments installed. We use the singleuser_profile_list parameter to set up a list of default images available to the user. This parameter receives a list of maps that describes the image details such as the image location and description.

See an example:

[{
  display_name = "Prefect"
  description  = "Notebook with prefect installed"
  default      = true
  kubespawner_override = {
    image = "drtools/prefect:notebook-prefect"
  }
}]

You must pass the image pull secret if you provide an image located in a private container registry. The image pull secret parameter is defined as below:

default = [{
    name = ""
}]

Prefect

...

Agent

According to Prefect docs, Agents are lightweight processes for orchestrating flow runs. Agents run inside a user's architecture, and are responsible for starting and monitoring flow runs. During operation the agent process queries the Prefect API for any scheduled flow runs, and allocates resources for them on their respective deployment platforms.

Postgresql

Dask

...

Feast

...

MLFlow

...

Note: The variables docker_registry_server and docker_auth_key are optional and should only be used when pulling MLFlow's image from a private repository.

Seldon

Exposing Services

In order to access the services from outside the cluster, we need to expose them. Usually, this is done through Kubernetes Ingress resources. In this project, since we rely on Seldon to expose our prediction endpoints, we use Ambassador API Gateway as our ingress controller. Seldon Core works well with Ambassador, allowing a single ingress to be used to expose ambassador and running machine learning deployments can then be dynamically exposed through seldon-created ambassador configurations.

Ambassador

Ambassador is a Kubernetes-native API Gateway built on the Envoy Proxy. In addition to the classical routing capabilities of an ingress, it can perform sophisticated traffic management functions, such as load balancing, circuit breakers, rate limits, and automatic retries. Also, it has support for independent authentication systems, such as the ORY ecosystem.

Exposing a service in Ambassador

Ambassador is designed around a declarative, self-service management model. The core resource used to support application development teams who need to manage the edge with Ambassador is the Mapping resource. This resource allows us to define custom routing rules to our services. This routing configuration can achieved by applying a custom Kubernetes Resource like the following

# mapping.yaml
---
apiVersion: getambassador.io/v2
kind:  Mapping
metadata:
  name:  httpbin-mapping
spec:
  prefix: /httpbin/
  service: httpbin.httpbin_namespace

By applying this configuration with kubectl apply -f httpbin-mapping.yaml.

Terraform

Since this project uses Terraform to manage resources and, with the current version, it's still not possible to apply custom Kubernetes resource definitions, we need to add this YAML file inside the services annotation. One way to do this is by using Service's Metadata field

resource "kubernetes_service" "httpbin" {
  metadata {
    ...
    annotations = {
      "getambassador.io/config" = <<YAML
---
apiVersion: getambassador.io/v2
kind: Mapping
name: httpbin-mapping
service: httpbin.httpbin_namespace
prefix: /httpbin/
YAML
    }
  }
}

This will produce the same behaviour as applying the custom yaml file described above.

Authentication

Since we're exposing our services in the Internet, we need an Authentication and Authorization system to prevent unwanted users to accessing our services. Ambassador API Gateway can control the access by using an External Authentication Service resource (AuthService). An AuthService is an API that has a verification endpoint, which determines if the user can access this resource (returning 200 or not, 401). In this project, we rely on ORY ecosystem to enable authentication. ORY is an open-source ecosystem of services with clear boundaries that solve authentication and authorization.

Session Lifespan

The session lifespan of authenticated users can be managed through the /ory/kratos/values.yaml file. By default, the session lifespan is 24h, but it is currently set to 30 days.

kratos:
  config:
  ...
    session:
      cookie:
        domain: ${cookie_domain}
      lifespan: 720h

ORY Oathkeeper

ORY Oathkeeper is an Identity and Access Proxy. It functions as a centralized way to manage different Authentication and Authorization methods, and inform the gateway, whether the HTTP request is allowed or not. The Oathkeeper serves perfectly as an Ambassador's External AuthService.

Zero-Trust and Unauthorized Resources

Oathkeeper is rooted in the principle of "never trust, always verify,". This means that if no additional configuration is provided, the Oathkeeper will always block the incoming request. In practice, all endpoints exposed in Ambassador will be blocked by external requests, until further configuration is made.

Access Rules

To configure an access rule to ORY Oathkeeper, the file access-rule-oathkeeper.yaml is used. Example:

Allow all incoming requests

- id: oathkeeper-access-rule
  match:
    url: <{http,https}>://${hostname}/allowed-service/<**>
    methods:
      - GET
  authenticators:
    - handler: anonymous
  authorizer:
    handler: allow
  mutators:
    - handler: noop
  credentials_issuer:
    handler: noop

This configuration will register all the incoming requests as a guest user, thus, not performing any credentials validation.

Authorize on KRATOS

- id: httpbin-access-rule
  match:
    url: <{http,https}>://${hostname}/blocked-service/<**>
    methods:
      - GET
  authenticators:
    - handler: cookie_session
  authorizer:
    handler: allow
  mutators:
    - handler: id_token
  credentials_issuer:
    handler: noop
  errors:
    - handler: redirect
      config:
        to: http://${hostname}/auth/login

This configuration will force authenticating all incoming requests by checking a cookie_session, which configuration is specified in config-oathkeeper.yaml