Torpedo

Torpedo is a test suite to qualify storage providers for stateful containers running in a distributed environment. It tests various scenarios that applications encounter when running in Linux containers and deployed via schedulers such as Kubernetes, Marathon or Swarm.

CSI

CSI is a specification for Linux Container Storage Interfaces. It defines the control plane interaction between a cloud native scheduler such as Kubernetes, and a cloud native storage provider. The specification is available here.

The Torpedo test suite natively supports the CSI specification for external volume support into Kubernetes and Mesosphere. It operates as a CSI enabled orchestrator (scheduler) to communicate with external storage providers that support CSI.

Torpedo tests cover the various scheduler-storage integration points that are being addressed by the CSI specification (https://docs.google.com/document/d/1JMNVNP-ZHz8cGlnqckOnpJmHF-DNY7IYP-Di7iuVhQI/edit#) and how external volume providers like Portworx are able to support production level operational scenarios when it comes to storage, server, software or network failures.

Legacy support

Since CSI is currently still work in progress, most schedulers provide external volume support to Mesosphere or Kubernetes via DVDI or the Kubernetes native driver interface.

Docker volume driver interface (DVDI) provides the control path operations to create, mount, unmount and eventually delete an external volume and is documented here.

In order to support legacy storage drivers, Torpedo can also work with schedulers that still use the Docker volume driver interface.

Scenarios to Consider when Deploying Stateful Applications

Deploying ephemeral applications require less consideration when compared to stateful applications. When running stateful applications in production, administrators should take into account various runtime scenarios that may occur and chose an external storage provider that is capable of dealing with these situations. Examples of these scenarios are:

Runtime software hangs and crashes

• Container runtime engine (or scheduler) software failure, hang or crash: When a daemon, like Docker crashes, it can induce errors with an application's connectivity to the external storage. This problem is compounded when the storage provider itself, runs as a container. In general, you need to assume that user space code will either hang or crash, and the storage system needs to gracefully deal with this, without data loss, unavailability or corruption.
• External storage driver software failure, hang or crash: When the storage software itself crashes, the overall solution needs to make sure that there are no lost IOs (data loss), unavailability or corruption.

Network and host issues

• Network disconnect from the storage provider to the external environment: If a node on which the storage volume driver is running were to become disconnected from the network, the overall solution needs to make sure that the volume can be used on another node, and that there is no data loss or corruption.
• A node running a stateful application becomes permanently (or for a prolonged period of time) unreachable: In many cases, a node can become permanently unusable. In cases, such as AWS, when an EBS volume is attached to such a node, the overall solution needs to make sure that the volume or the data can somehow still be used on some other node in the cluster.
• A network partition in the cluster: When the scheduler cluster or the storage cluster gets partitioned in such a way that quorum is lost, the nodes that are still part of the quorum need to be able to use all of the data that was in the original cluster. Otherwise, this would lead to data unavailability.

Scheduler software issues

• Scheduler software attempts to deploy a stateful container on a node that is not part of the storage cluster: It is possible that the storage cluster and the scheduler cluster do not comprise of the same machines. The overall solution must prevent, or somehow make sure that when a stateful application is deployed on a non-storage node, that the application's storage requirements are fulfilled. Some approaches to handle this include the use of scheduler constraints and labels.
• Scheduler software attempts to bring up a new container/pod/task to use a storage volume prior to properly terminating the previous container/pod/task on a different host: Scheduler software, perhaps due to bugs or timing issues, may launch a new application stack on a new set of nodes that refer to a volume currently in use by an application stack being torn down. The overall solution must be capable of dealing with these transition scenarios, without application data loss or corruption.

Test Cases Run by Torpedo

Qualified External Storage Providers

To submit an external storage provider, please submit a PR with the output of the Torpedo test program and the specifics of the environment used.

Usage

Build

See How to build.

Run

See How to run.

Contributing

The specification and code is licensed under the Apache 2.0 license found in the LICENSE file of this repository.

See the Style Guide.

Sign your work

The sign-off is a simple line at the end of the explanation for the patch, which certifies that you wrote it or otherwise have the right to pass it on as an open-source patch. The rules are pretty simple: if you can certify the below (from developercertificate.org):

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
660 York Street, Suite 102,
San Francisco, CA 94110 USA

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

then you just add a line to every git commit message:

Signed-off-by: Joe Smith <[email protected]>

using your real name (sorry, no pseudonyms or anonymous contributions.)

You can add the sign off when creating the git commit via git commit -s.