linkedin/datahub-gma

Stars
121
Rank 293,924 (Top 6 %)
Language
Java
License
Apache License 2.0
Created over 4 years ago
Updated 3 months ago

linkedin/datahub-gma

linkedin

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

General Metadata Architecture

General Metadata Architecture

Documentation | Roadmap | FAQ

📣 We've moved from Bintray to Artifactory!

As of version 0.2.45, we are only publishing versions to LinkedIn's Artifactory instance rather than Bintray, which is approaching end of life.

Introduction

General Metadata Architecture (GMA) is the backend for DataHub, LinkedIn's generalized metadata search & discovery tool. To learn more about DataHub, check out its GitHub page.

You should also visit GMA Architecture to get a better understanding of how GMA is implemented.

This repository contains the partial source code for GMA. It originally lived in the same GitHub repository as DataHub. We're still in the process of moving it all over.

Documentation

Releases

See Releases page for more details. We follow the SemVer Specification when versioning the releases and adopt the Keep a Changelog convention for the changelog format.

FAQs

Frequently Asked Questions about DataHub can be found here.

Features & Roadmap

Check out GMA's Features & Roadmap.

Contributing

We welcome contributions from the community. Please refer to our Contributing Guidelines for more details. We also have a contrib directory for incubating experimental features.

Community

Join our slack workspace for discussions and important announcements.

school-of-sre

At LinkedIn, we are using this curriculum for onboarding our entry-level talents into the SRE role.

css-blocks

High performance, maintainable stylesheets.

Burrow

Kafka Consumer Lag Checking

databus

Source-agnostic distributed change data capture system

Liger-Kernel

Efficient Triton Kernels for LLM Training

qark

Tool to look for several security related Android application vulnerabilities

dustjs

Asynchronous Javascript templating for the browser and server

cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.

rest.li

Rest.li is a REST+JSON framework for building robust, scalable service architectures using dynamic discovery and simple asynchronous APIs.

kafka-monitor

Xinfra Monitor monitors the availability of Kafka clusters by producing synthetic workloads using end-to-end pipelines to obtain derived vital statistics - E2E latency, service produce/consume availability, offsets commit availability & latency, message loss rate and more.

dexmaker

A utility for doing compile or runtime code generation targeting Android's Dalvik VM

greykite

A flexible, intuitive and fast forecasting library

ambry

Distributed object store

shiv

shiv is a command line utility for building fully self contained Python zipapps as outlined in PEP 441, but with all their dependencies included.

swift-style-guide

LinkedIn's Official Swift Style Guide

dr-elephant

Dr. Elephant is a job and flow-level performance monitoring and tuning tool for Apache Hadoop and Apache Spark

detext

DeText: A Deep Neural Text Understanding Framework for Ranking and Classification Tasks

luminol

Anomaly Detection and Correlation library

parseq

Asynchronous Java made easier

oncall

Oncall is a calendar tool designed for scheduling and managing on-call shifts. It can be used as source of dynamic ownership info for paging systems like http://iris.claims.

test-butler

Reliable Android Testing, at your service

goavro

PalDB

An embeddable write-once key-value store written in Java

brooklin

An extensible distributed system for reliable nearline data streaming at scale

iris

Iris is a highly configurable and flexible service for paging and messaging.

photon-ml

A scalable machine learning library on Apache Spark

URL-Detector

A Java library to detect and normalize URLs in text

coral

Coral is a translation, analysis, and query rewrite engine for SQL and other relational languages.

Hakawai

A powerful, extensible UITextView.

eyeglass

NPM Modules for Sass

opticss

A CSS Optimizer

LiTr

Lightweight hardware accelerated video/audio transcoder for Android.

kafka-tools

A collection of tools for working with Apache Kafka.

pygradle

Using Gradle to build Python projects

flashback

mock the internet

FeatureFu

Library and tools for advanced feature engineering

LayoutTest-iOS

Write unit tests which test the layout of a view in multiple configurations

FastTreeSHAP

Fast SHAP value computation for interpreting tree-based models

venice

Venice, Derived Data Platform for Planet-Scale Workloads.

Spyglass

A library for mentions on Android

dagli

Framework for defining machine learning models, including feature generation and transformations, as directed acyclic graphs (DAGs).

cruise-control-ui

Cruise Control Frontend (CCFE): Single Page Web Application to Manage Large Scale of Kafka Clusters

ml-ease

ADMM based large scale logistic regression

openhouse

Open Control Plane for Tables in Data Lakehouse

dph-framework

transport

A framework for writing performant user-defined functions (UDFs) that are portable across a variety of engines including Apache Spark, Apache Hive, and Presto.

spark-tfrecord

Read and write Tensorflow TFRecord data from Apache Spark.

isolation-forest

A Spark/Scala implementation of the isolation forest unsupervised outlier detection algorithm with support for exporting in ONNX format.

LiFT

The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows.

shaky-android

Shake to send feedback for Android.

pyexchange

Python wrapper for Microsoft Exchange

asciietch

A graphing library with the goal of making it simple to graphs using ascii characters.

python-avro-json-serializer

Serializes data into a JSON format using AVRO schema.

gdmix

A deep ranking personalization framework

li-apache-kafka-clients

li-apache-kafka-clients is a wrapper library for the Apache Kafka vanilla clients. It provides additional features such as large message support and auditing to the Java producer and consumer in the open source Apache Kafka.

dynamometer

A tool for scale and performance testing of HDFS with a specific focus on the NameNode.

Avro2TF

Avro2TF is designed to fill the gap of making users' training data ready to be consumed by deep learning training frameworks.

linkedin-gradle-plugin-for-apache-hadoop

dex-test-parser

Find all test methods in an Android instrumentation APK

cassette

An efficient, file-based FIFO Queue for iOS and macOS.

spaniel

LinkedIn's JavaScript viewport tracking library and IntersectionObserver polyfill

Hoptimator

Multi-hop declarative data pipelines

migz

Multithreaded, gzip-compatible compression and decompression, available as a platform-independent Java library and command-line utilities.

avro-util

Collection of utilities to allow writing java code that operates across a wide range of avro versions.

sysops-api

sysops-api is a framework designed to provide visability from tens of thousands of machines in seconds.

iceberg

A temporary home for LinkedIn's changes to Apache Iceberg (incubating)

DuaLip

DuaLip: Dual Decomposition based Linear Program Solver

kube2hadoop

Secure HDFS Access from Kubernetes

dynoyarn

DynoYARN is a framework to run simulated YARN clusters and workloads for YARN scale testing.

linkedin.github.com

Listing of all our public GitHub projects.

Tachyon

An Android library that provides a customizable calendar day view UI widget.

Cytodynamics

Classloader isolation library.

iris-relay

Stateless reverse proxy for thirdparty service integration with Iris API.

concurrentli

Classes for multithreading that expand on java.util.concurrent, adding convenience, efficiency and new tools to multithreaded Java programs

iris-mobile

A mobile interface for linkedin/iris, built for iOS and Android on the Ionic platform

lambda-learner

Lambda Learner is a library for iterative incremental training of a class of supervised machine learning models.

TE2Rules

Python library to explain Tree Ensemble models (TE) like XGBoost, using a rule list.

instantsearch-tutorial

Sample code for building an end-to-end instant search solution

PASS-GNN

self-focused

Helps make a single page application more friendly to screen readers.

tracked-queue

An autotracked implementation of a ring-buffer-backed double-ended queue

QueryAnalyzerAgent

Analyze MySQL queries with negligible overhead

performance-quality-models

Personalizing Performance model repository

Jupyter Notebook

data-integration-library

The Data Integration Library project provides a library of generic components based on a multi-stage architecture for data ingress and egress.

Iris-message-processor

Iris-message-processor is a fully distributed Go application meant to replace the sender functionality of Iris and provide reliable, scalable, and extensible incident and out of band message processing and sending.

smart-arg

Smart Arguments Suite (smart-arg) is a slim and handy python lib that helps one work safely and conveniently with command line arguments.

linkedin-calcite

LinkedIn's version of Apache Calcite

atscppapi

This library provides wrappers around the existing Apache Traffic Server API which will vastly simplify the process of writing Apache Traffic Server plugins.

forthic

high-school-trainee

LinkedIn Women in Tech High School Trainee Program

play-parseq

Play-ParSeq is a Play module which seamlessly integrates ParSeq with Play Framework

icon-magic

Automated icon build system for iOS, Android and Web

QuantEase

QuantEase, a layer-wise quantization framework, frames the problem as discrete-structured non-convex optimization. Our work leverages Coordinate Descent techniques, offering high-quality solutions without the need for matrix inversion or decomposition.

kafka-remote-storage-azure

play-restli

A library that simplifies building restli services on top of the play server.

spark-inequality-impact

Li-Airflow-Backfill-Plugin

Li-Airflow-Backfill-Plugin is a plugin to work with Apache Airflow to provide data backfill feature, ie. to rerun pipelines for a certain date range.

AlerTiger

Jupyter Notebook

diderot

A fast and flexible implementation of the xDS protocol

gobblin-elr

This is a read-only mirror of apache/gobblin