Discover moj-analytical-services/data_generator Open Source project

Apply GOV.UK styled components and formats in shiny

1,328

shinyGovstyle

CSS

splink_demos

Interactive notebooks containing demonstration code of the splink library

NLP tool for scraping text from a corpus of PDF files, embedding the sentences in the text and finding semantically similar sentences to a given search query.

airflow-pdf2embeddings

xltabr: An R package for writing formatted cross tabulations (contingency tables) to Excel using openxlsx

xltabr

etl-pipeline-example

An example of an ETL pipeline that lays out generic DE processes. This is now out of date but still provides useful information

MoJ coffee and coding sessions that can be made publicly available

coffee-and-coding-public

A python package to create a database on the platform using our moj data warehousing framework

etl_manager

IntroRTraining

Introductory R training

R package for formatting ggplot2 charts and applying MoJ corporate colours.

mojchart

dataengineeringutils3

Fully unit tested utility functions for data engineering. Python 3 only.

DASD's coding principles for analytical projects

our-coding-standards

User guidance for the MoJ Analytical Platform

user-guidance

How to write functions in R

writing_functions_in_r

Making and developing R packages

rpackage_training

data-engineering-and-modelling-applicant-info

Information for potential applicants to MoJ Data Engineering, including links to our work and information about our teams.

pq-tool

Tool to analyse past parliamentary questions with visualisation in RShiny

splink_graph

pyspark-parallelised functions producing graph-theoretical metrics in connected component clusters for use in record-linkage (or other domains)

Python version of dbtools

pydbtools

Conforms pandas to "correct" datatypes to ensure data in/out using CSV, JSONL and Parquet is read the same (using arrow).

mojap-arrow-pd-parser

Interact with files in s3 on the Analytical Platform

s3tools

mojrap

For generalised functions for RAP. If there are any functions in your RAP that will be useful to other people, please use this space to share them.

graph-club

Tri-weekly hackathons and talks on Graph Theory and Network Analysis.

Generate synthetic datasets for linking

splink_synthetic_data

A dockerised version of the spark history server which enables us to access metrics in the spark ui from a log generated by AWS glue

docker_spark_history_ui

Dockerfile

rmarkdown_training

Short training session on RMarkdown, for JSAS

mojspeakr

Formatting RMarkdown into govspeak for publishing on gov.uk

fuzzyfinder

Fuzzy search for matching records and score search results according to how closely they match

A python package containing functions that help manage our data management processes on AWS

dataengineeringutils

Docker image used to automatically validate data

data_linter

A repo to test the different open source AWS tools we use / maintain for Data Engineering

mojap-aws-tools-demo

Some thinking about Natural Language Processing

NLP-guidance

Basic wrapper functions to query data using boto3 and Athena

dbtools

splink_cluster_studio

Create interactive dashboards to visualise and analyse the outputs of data linking

Schema definitions and management of our metadata used by the Data Engineering Team at MoJ

mojap-metadata

Accessing Athena on the Analytical Platform

Rdbtools

splink_scalaudfs

Data linking functions in Scala, to be used in a Pyspark environment.

Scala

rshiny-template

Template RShiny project

intro_r_training_extension

An extension to the IntroRTraining course

Little helper to write IAM policies

iam_builder

ggplotTraining

mojSuppression

QA.that

splink_comparison_viewer

**DEPRECATED** See https://github.com/moj-analytical-services/user-guidance

platform_user_guidance

Infrastructure to allow data from the Analytical Platform to be accessed by other services

data-engineering-exports

Public repo with examples of goodtables

goodtables_test

s3_data_packer

A package to generate random pairings for Coffee Roulette

Rs3tools

coffee_roulette_pairs

FuzzyMatchR

Reference page to link to R implementation of a probabilistic matching function

mojverse

The tidyverse equivalent for MoJ packages

intro_to_github_training

AWS-study-group-quizzes

splink_data_generation

Generate datasets with known m and u probabilities to feed into the Fellegi Sunter model

Standard content, settings and hooks for data engineering

I-RAP

data-engineering-template

Shell

rmarkdown-vegawidget-template

A template for a deployed app that renders a markdown report

s3browser

A R Studio Addin that allows you to browse the files you have access to in S3

A function that allows you to add superscripts and subscripts to cells in excel

RSuperscript

airflow_osrm_scrape

Scrapes the open streetmap routing machine for all combinations of LSOAs, and MSOAs

Data discovery tool that ingests metadata and makes it searchable. Uses metadata in the format required for https://github.com/moj-analytical-services/etl_manager

metadata_vis

CSS

OPG

airflow-de-intro-project

Guidance on learning SQL from square one (i.e. zero knowledge)

SQL_from_square_one

iceberg-evaluation

Judicial Workforce Modelling R Package

random-coffee-trials

Automating rct

jwmodel

airflow_get_index_of_multiple_deprivation

Airflow job to get dataset of index of multiple deprivation

Unit tested functions for cleaning data as part of ETL processes

datacleaningutils

Example of project with a Cronjob

rshiny-test

cronjob-template

shiny-headers-demo

gluejobutils

Python 2.7 utility functions to include with AWS glue jobs

A lookup table that maps local authorities to HMCTS regions.

actions-lint-python

lookup_hmcts_regions

Ministry of Justice Analytical Services GitHub workflow templates

.github

pq_scraper

Parliamentary Questions (PQ) scraper

a11ycharts

airflow-murad-ali-j-test

A package to lint data against our meta data schemas

vega-lite-away-day

data_linter_deprecated

Easy reading of published civil CSVs

predictr

kerins-shiny-app

civilreadr

template-airflow-python

Template repository for running airflow python tasks in Kubernetes/Docker

A sankey diagram for criminal history statistics

ap-tools-training

criminal_history_sankey

geoharmonise

splink_examples_synthetic_data

Testing kaniko to build Docker images

rshiny-xoen-kaniko-test

github-outside-collaborators

Manage outside collaborators on our Github repositories

Ruby

airflow-platform-user-data

Airflow job to gather platform user data from Auth0

A few wrappers and tools to use with Airflow on the Analytical Platform

mojap-airflow-tools

Wrapper to simplify connection to Oracle databases

oracleConnectR

mojtext

Functions to automate text

mojtable

intro-to-python

Visualizing our technology choices

data-and-analytics-engineering-tech-radar