Discover moj-analytical-services/coffee-and-coding-public Open Source

Interactive notebooks containing demonstration code of the splink library

1,251

splink_demos

Now up to GDS frontend version v4.0.0

shinyGovstyle

CSS

airflow-pdf2embeddings

NLP tool for scraping text from a corpus of PDF files, embedding the sentences in the text and finding semantically similar sentences to a given search query.

xltabr: An R package for writing formatted cross tabulations (contingency tables) to Excel using openxlsx

xltabr

etl-pipeline-example

An example of an ETL pipeline that lays out generic DE processes. This is now out of date but still provides useful information

A python package to create a database on the platform using our moj data warehousing framework

etl_manager

IntroRTraining

Introductory R training

Fully unit tested utility functions for data engineering. Python 3 only.

dataengineeringutils3

DASD's coding principles for analytical projects

our-coding-standards

R package for formatting ggplot2 charts and applying MoJ corporate colours.

mojchart

user-guidance

User guidance for the MoJ Analytical Platform

How to write functions in R

writing_functions_in_r

Making and developing R packages

rpackage_training

pq-tool

Tool to analyse past parliamentary questions with visualisation in RShiny

splink_graph

pyspark-parallelised functions producing graph-theoretical metrics in connected component clusters for use in record-linkage (or other domains)

Python version of dbtools

pydbtools

Information for potential applicants to MoJ Data Engineering, including links to our work and information about our teams.

data-engineering-and-modelling-applicant-info

mojap-arrow-pd-parser

Conforms pandas to "correct" datatypes to ensure data in/out using CSV, JSONL and Parquet is read the same (using arrow).

Interact with files in s3 on the Analytical Platform

s3tools

mojrap

For generalised functions for RAP. If there are any functions in your RAP that will be useful to other people, please use this space to share them.

docker_spark_history_ui

A dockerised version of the spark history server which enables us to access metrics in the spark ui from a log generated by AWS glue

Dockerfile

graph-club

Tri-weekly hackathons and talks on Graph Theory and Network Analysis.

Generate synthetic datasets for linking

splink_synthetic_data

Short training session on RMarkdown, for JSAS

rmarkdown_training

mojspeakr

Formatting RMarkdown into govspeak for publishing on gov.uk

dataengineeringutils

A python package containing functions that help manage our data management processes on AWS

Docker image used to automatically validate data

data_linter

Fuzzy search for matching records and score search results according to how closely they match

fuzzyfinder

A repo to test the different open source AWS tools we use / maintain for Data Engineering

mojap-aws-tools-demo

Some thinking about Natural Language Processing

NLP-guidance

Basic wrapper functions to query data using boto3 and Athena

dbtools

splink_cluster_studio

Create interactive dashboards to visualise and analyse the outputs of data linking

Schema definitions and management of our metadata used by the Data Engineering Team at MoJ

mojap-metadata

Accessing Athena on the Analytical Platform

Rdbtools

splink_scalaudfs

Data linking functions in Scala, to be used in a Pyspark environment.

Scala

data_generator

Generates data using faker and our meta data schemas

An extension to the IntroRTraining course

rshiny-template

Template RShiny project

intro_r_training_extension

Little helper to write IAM policies

iam_builder

ggplotTraining

**DEPRECATED** See https://github.com/moj-analytical-services/user-guidance

mojSuppression

QA.that

platform_user_guidance

Infrastructure to allow data from the Analytical Platform to be accessed by other services

data-engineering-exports

Public repo with examples of goodtables

goodtables_test

splink_comparison_viewer

s3_data_packer

A package to generate random pairings for Coffee Roulette

Rs3tools

coffee_roulette_pairs

FuzzyMatchR

Reference page to link to R implementation of a probabilistic matching function

mojverse

The tidyverse equivalent for MoJ packages

intro_to_github_training

AWS-study-group-quizzes

I-RAP

data-engineering-template

Standard content, settings and hooks for data engineering

Shell

rmarkdown-vegawidget-template

A template for a deployed app that renders a markdown report

s3browser

A R Studio Addin that allows you to browse the files you have access to in S3

Generate datasets with known m and u probabilities to feed into the Fellegi Sunter model

splink_data_generation

A function that allows you to add superscripts and subscripts to cells in excel

RSuperscript

airflow_osrm_scrape

Scrapes the open streetmap routing machine for all combinations of LSOAs, and MSOAs

Data discovery tool that ingests metadata and makes it searchable. Uses metadata in the format required for https://github.com/moj-analytical-services/etl_manager

metadata_vis

CSS

OPG

airflow-de-intro-project

Guidance on learning SQL from square one (i.e. zero knowledge)

SQL_from_square_one

iceberg-evaluation

Judicial Workforce Modelling R Package

random-coffee-trials

Automating rct

jwmodel

airflow_get_index_of_multiple_deprivation

Airflow job to get dataset of index of multiple deprivation

Unit tested functions for cleaning data as part of ETL processes

datacleaningutils

Example of project with a Cronjob

rshiny-test

cronjob-template

shiny-headers-demo

gluejobutils

Python 2.7 utility functions to include with AWS glue jobs

A lookup table that maps local authorities to HMCTS regions.

actions-lint-python

lookup_hmcts_regions

Ministry of Justice Analytical Services GitHub workflow templates

.github

pq_scraper

Parliamentary Questions (PQ) scraper

a11ycharts

airflow-murad-ali-j-test

A package to lint data against our meta data schemas

vega-lite-away-day

data_linter_deprecated

Easy reading of published civil CSVs

predictr

kerins-shiny-app

civilreadr

template-airflow-python

Template repository for running airflow python tasks in Kubernetes/Docker

ap-tools-training

geoharmonise

splink_examples_synthetic_data

A sankey diagram for criminal history statistics

criminal_history_sankey

Testing kaniko to build Docker images

rshiny-xoen-kaniko-test

github-outside-collaborators

Manage outside collaborators on our Github repositories

Ruby

airflow-platform-user-data

Airflow job to gather platform user data from Auth0

A few wrappers and tools to use with Airflow on the Analytical Platform

mojap-airflow-tools