• Stars
    star
    135
  • Rank 269,297 (Top 6 %)
  • Language
    Go
  • License
    MIT License
  • Created about 3 years ago
  • Updated 4 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

cheek: a pico-sized declarative job scheduler

cheek

cheek

GitHub tag (latest SemVer) dataroots codecov workflow Go Report Card Go Reference Awesome love

cheek is a pico-sized declarative job scheduler designed to excel in a single-node environment. cheek aims to be lightweight, stand-alone and simple. It does not compete for robustness.

Getting started

Fetch the latest version for your system below.

darwin-arm64 | darwin-amd64 | linux-386 | linux-arm64 | linux-amd64

You can (for example) fetch it like below, make it executable and run it. Optionally put the cheek on your PATH.

curl https://storage.googleapis.com/cheek-scheduler/darwin/amd64/cheek -o cheek
chmod +x cheek
./cheek

Everything about how you want the scheduler to function is defined in a schedule specification written in YAML. Start by creating this specification using the below example. Note, this structure should be more or less self-explanatory, if it is not, create an issue.

tz_location: Europe/Brussels # optionally set timezone to adhere to
jobs:
  foo:
    command: date
    cron: "* * * * *" # a cron string to specify when to run
    on_success:
      trigger_job: # trigger something on run
        - bar
  bar:
    command: # command to run, use a list if you want to pass args
      - echo
      - $foo
    env: # you can pass env variables
      foo: bar
  other_workingdir:
    command: pwd
    working_directory: ../testdata # specify the working directory of the job
  coffee:
    command: this fails
    cron: "* * * * *"
    retries: 3
    on_error:
      notify_webhook: # notify something on error
        - https://webhook.site/4b732eb4-ba10-4a84-8f6b-30167b2f2762
      notify_slack_webhook: # notify slack via a slack compatible webhook
        - https://webhook.site/048ff47f-9ef5-43fb-9375-a795a8c5cbf5

If your command requires arguments, please make sure to pass them as an array like in foo_job.

Note that you can set tz_location if the system time of where you run your service is not to your liking.

Scheduler

The core of cheek consists of a scheduler that uses the schedule specs defined in your yaml file to trigger jobs when they are due.

You can launch the scheduler via:

cheek run ./path/to/my-schedule.yaml

Check out cheek run --help for configuration options.

Web UI

cheek ships with a web UI that by default gets launched on port 8081. You can define the port on which it is accessible via the --port flag.

main-screen
main overview
detail
job detail

You can access the UI by navigating to http://localhost:8081. When cheek is deployed you are recommended to NOT make this port publicly accessible, instead navigate to the UI via an SSH tunnel.

The UI allows to get a quick overview on jobs that have run, that error'd and their logs. It basically does this by fetching the state of the scheduler and by reading the logs that (per job) get written to $HOME/.cheek/. Note that you can ignore these logs, output of jobs will always go to stdout as well.

Note, cheek prior to version 0.3.0 originally used to boast a TUI, which has since been removed.

Configuration

All configuration options are available by checking out cheek --help or the help of its subcommands (e.g. cheek run --help).

Configuration can be passed as flags to the cheek CLI directly. All configuration flags are also possible to set via environment variables. The following environment variables are available, they will override the default and/or set value of their similarly named CLI flags (without the prefix): CHEEK_PORT, CHEEK_SUPPRESSLOGS, CHEEK_LOGLEVEL, CHEEK_PRETTY, CHEEK_HOMEDIR.

Events & Notifications

There are two types of event you can hook into: on_success and on_error. Both events materialize after an (attempted) job run. Three types of actions can be taken as a response: notify_webhook, notify_slack_webhook and trigger_job. See the example below. Definition of these event actions can be done on job level or at schedule level, in the latter case it will apply to all jobs.

on_success:
  notify_webhook:
    - https://webhook.site/e33464a3-1a4f-4f1a-99d3-743364c6b10f
jobs:
  coffee:
    command: this fails # this will create on_error event
    cron: "* * * * *"
    on_error:
      notify_webhook:
        - https://webhook.site/e33464a3-1a4f-4f1a-99d3-743364c6b10f
  beans:
    command: echo grind # this will create on_success event
    cron: "* * * * *"

Webhooks are a generic way to push notifications to a plethora of tools. There is a generic way to do this via the notify_webhook option or a Slack-compatible one via notify_slack_webhook.

The notify_webhook sends a JSON payload to your webhook url with the following structure:

{
	"status": 0,
	"log": "I'm a teapot, not a coffee machine!",
	"name": "TeapotTask",
	"triggered_at": "2023-04-01T12:00:00Z",
	"triggered_by": "CoffeeRequestButton",
	"triggered": ["CoffeeMachine"] // this job triggered another one
}

The notify_slack_webhook sends a JSON payload to your Slack webhook url with the following structure (which is Slack app compatible):

{
	"text": "TeapotTask (exitcode 0):\nI'm a teapot, not a coffee machine!"
}

Docker

Check out the Dockerfile.example for an example on how to use cheek within the context of a Docker container. Note that this builds upon a published Ubuntu-based image build that you can find in the base Dockerfile.

Prebuilt images are available at ghcr.io/datarootsio/cheek:latest where latest can be replaced by a version tag. Check out the available images for an overview on available tags.

Available versions

If you want to pin your setup to a specific version of cheek you can use the following template to fetch your cheek binary:

Where:

  • os is one of linux, darwin
  • arch is one of amd64, arm64, 386
  • tag is one the available tags
  • shortsha is a 7-char SHA and most commits on main will be available

Acknowledgements

cheek is building on top of many great OSS assets. Noteable thanks goes to:

  • chota: for a pico sized css framework
  • gronx: for allowing me not to worry about CRON strings.

GitHub Contributors

More Repositories

1

ml-skeleton-py

A best-practices first project template that allows you to get started on a new machine learning project.
Python
142
star
2

databooks

A CLI tool to reduce the friction between data scientists by reducing git conflicts removing notebook metadata and gracefully resolving git conflicts.
Python
108
star
3

artyfarty

ggplot2 theme + palette presets
R
96
star
4

tf-profile

CLI tool to profile Terraform runs, written in Go
Go
94
star
5

tutorial-face-mask-detection

In this project, we develop a pipeline to detect unmasked faces in images. This can, for example, be used to alert people that do not wear a mask when entering a building.
Jupyter Notebook
86
star
6

terraform-aws-ecs-airflow

A terraform module that creates an airflow instance in AWS ECS.
HCL
53
star
7

fresh-coffee-listener

Using a raspberry pi, we listen to the coffee machine and count the number of coffee consumption
Python
51
star
8

prefect-dbt-flow

prefect integration for running dbt
Python
46
star
9

tutorial-great-expectations

A tutorial for the Great Expectations library.
Jupyter Notebook
39
star
10

terraform-module-azure-datalake

Terraform module for an Azure Data Lake
HCL
29
star
11

tutorial-streamlit-demo

Python
29
star
12

tutorial-mlops

MLOps exercise material.
Jupyter Notebook
24
star
13

bikefitting

Graduation project of the rootsacademy: A bikefitting application consisting of a website where users can upload a short video of themselves on a bike trainer and will receive a recommendation to either move the bike saddle up or down.
Python
20
star
14

terraform-module-kubeflow

Kubeflow deployment purely in Terraform
HCL
19
star
15

expiring-lru-cache

LRU caching with expiration period.
Python
17
star
16

terraform-aws-ecs-dagster

A terraform module that deploys Dagster to AWS, using ECS.
HCL
16
star
17

terraform-module-kubernetes-application

Terraform module for a kubernetes application
HCL
15
star
18

.github

πŸš€ Get started in our repos
Python
12
star
19

python-minimal-boilerplate

Some minimal Python boilerplate.
Python
11
star
20

github-stats-card

⭐️ a minimal but inclusive github stats badge ⭐️
TypeScript
11
star
21

dbt-fabric

Python
10
star
22

skeleton-pyspark

A best-practices first project template that allows you to get started on a new pyspark project
Python
9
star
23

terraform-provider-kubeflowpipelines

Terraform provider for Kubeflow pipelines API
Go
8
star
24

deploy-spark-streaming-ml

8
star
25

notion-dbs-data-quality

Using Great Expectations and Notion's API, this repo aims to provide data quality for our databases in Notion.
Python
8
star
26

anomalib-demo

Jupyter Notebook
6
star
27

OsConfig

Bash scrips for Data professionals OS configuration
Shell
4
star
28

knowledgebase_guardian

A minimal example to demonstrate how LLM's can assist in detecting contradictions in documents.
Python
3
star
29

phonehome

KISS telemetry for FOSS packages
Go
3
star
30

workshop-image-segmentation-style-transfer

Jupyter Notebook
2
star
31

terraform-module-azure-snowflake

HCL
2
star
32

mlflow-emissions-sdk

This package logs the carbon emissions of machine learning models 🌿
Python
2
star
33

rootsstyle

A dataroots inspired style for Matplotlib. Works with any visualization tools that builds upon Matplotlib (seaborn, pandas).
Python
2
star
34

snowflake-ml

Toy use case on how to use Snowflake as a full ML platform.
Python
2
star
35

rootscamp2019-session4

session 4 - building an API assets
Python
1
star
36

terraform-azurerm-aci-dagster

A terraform module that deploys Dagster to Azure.
HCL
1
star
37

dbt-fabric-demo

Demo project for dbt-fabric
1
star
38

tutorial-hyperparameter-optimization

Tutorial for Rootlabs@Lunch: Practical Hyperparameter Optimisation
Jupyter Notebook
1
star
39

transforming-tabular-data

Comparing Pandas v. Polars v. PyArrow v. DuckDB πŸΌπŸ»β€β„οΈπŸΉπŸ¦†
Jupyter Notebook
1
star
40

homebrew-tf-profile

This is a homebrew formulae for tf-profile.
Ruby
1
star