twitter/the-algorithm-ml

Stars
9,938
Rank 3,435 (Top 0.07 %)
Language
Python
License
GNU Affero Genera...
Created over 1 year ago
Updated 10 months ago

twitter/the-algorithm-ml

twitter

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Source code for Twitter's Recommendation Algorithm

This project open sources some of the ML models used at Twitter.

Currently these are:

The "For You" Heavy Ranker (projects/home/recap).
TwHIN embeddings (projects/twhin) https://arxiv.org/abs/2202.05387

This project can be run inside a python virtualenv. We have only tried this on Linux machines and because we use torchrec it works best with an Nvidia GPU. To setup run

./images/init_venv.sh (Linux only).

The READMEs of each project contain instructions about how to run each project.

the-algorithm

Source code for Twitter's Recommendation Algorithm

twemoji

Emoji for everyone. https://twemoji.twitter.com/

typeahead.js

typeahead.js is a fast and fully-featured autocomplete library

twemproxy

A fast, light-weight proxy for memcached and redis

finagle

A fault tolerant, protocol-agnostic RPC system

hogan.js

A compiler for the Mustache templating language

labella.js

Placing labels on a timeline without overlap.

scala_school

Lessons in the Fundamentals of Scala

AnomalyDetection

Anomaly Detection with R

scalding

A Scala API for Cascading

twitter-text

Twitter Text Libraries. This code is used at Twitter to tokenize and parse text to meet the expectations for what can be used on the platform.

TwitterTextEditor

A standalone, flexible API that provides a full-featured rich text editor for iOS applications.

opensource-website

Twitter's open source website, identifying projects we've released, organizations we support, and the work we do to support open source.

util

Wonderful reusable code from Twitter

algebird

Abstract Algebra for Scala

finatra

Fast, testable, Scala services built on TwitterServer and Finagle

effectivescala

Twitter's Effective Scala Guide

summingbird

Streaming MapReduce with Scalding and Storm

pelikan

Pelikan is Twitter's unified cache backend

ios-twitter-image-pipeline

Twitter Image Pipeline is a robust and performant image loading and caching framework for iOS clients

twurl

OAuth-enabled curl for the Twitter API

twitter-server

Twitter-Server defines a template from which services at Twitter are built

rezolus

Systems performance telemetry

communitynotes

Documentation and source code powering Twitter's Community Notes

activerecord-reputation-system

An Active Record Reputation System for Rails

compose-rules

Static checks to aid with a healthy adoption of Compose

fatcache

Memcache on SSD

rsc

Experimental Scala compiler focused on compilation speed

elephant-bird

Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.

cassovary

Cassovary is a simple big graph processing library for the JVM

Serial

Light-weight, fast framework for object serialization in Java, with Android support.

hbc

A Java HTTP client for consuming Twitter's realtime Streaming API

twemcache

Twemcache is the Twitter Memcached

vireo

Vireo is a lightweight and versatile video processing library written in C++11

innovators-patent-agreement

Innovators Patent Agreement (IPA)

twitter-korean-text

Korean tokenizer

scrooge

A Thrift parser/generator

BreakoutDetection

Breakout Detection via Robust E-Statistics

GraphJet

GraphJet is a real-time graph processing library.

twitter-cldr-rb

Ruby implementation of the ICU (International Components for Unicode) that uses the Common Locale Data Repository to format dates, plurals, and more.

bijection

Reversible conversions between types

chill

Scala extensions for the Kryo serialization library

ios-twitter-network-layer

Twitter Network Layer is a scalable and feature rich network layer built on top of NSURLSession for Apple platforms

hadoop-lzo

Refactored version of code.google.com/hadoop-gpl-compression for hadoop 0.20

storehaus

Storehaus is a library that makes it easy to work with asynchronous key value stores

rpc-perf

A tool for benchmarking RPC services

d3kit

D3Kit is a set tools to speed D3 related project development

scoot

Scoot is a distributed task runner, supporting both a proprietary API and Bazel's Remote Execution.

twitter-cldr-js

JavaScript implementation of the ICU (International Components for Unicode) that uses the Common Locale Data Repository to format dates, plurals, and more. Based on twitter-cldr-rb.

rustcommon

Common Twitter Rust lib

scala_school2

wordpress

The official Twitter plugin for WordPress. Embed Twitter content and grow your audience on Twitter.

ios-twitter-logging-service

Twitter Logging Service is a robust and performant logging framework for iOS clients

nodes

A library to implement asynchronous dependency graphs for services in Java

SentenTree

A novel text visualization technique

interactive

Twitter interactive visualization

joauth

A Java library for authenticating HTTP Requests using OAuth

thrift_client

A Thrift client wrapper that encapsulates some common failover behavior

hpack

Header Compression for HTTP/2

zktraffic

ZooKeeper protocol analyzer and stats gathering daemon

cache-trace

A collection of Twitter's anonymized production cache traces.

twemoji-parser

A simple library for identifying emoji entities within a string in order to render them as Twemoji.

sbf

tormenta

Scala extensions for Storm

whiskey

HTTP library for Android (beta)

hraven

hRaven collects run time data and statistics from MapReduce jobs in an easily queryable format

netty-http2

HTTP/2 for Netty

sqrl

A Safe, Stateful Rules Language for Event Streams

ccommon

focus

Focus aligns Git worktree content based on outlines of a repository's Bazel build graph. Focused repos are sparse, shallow, and thin and unlock markedly better performance in large repos.

metrics

dict_minimize

Access scipy optimizers from your favorite deep learning framework.

twitter.github.io

diffusion-rl

go-bindata

birdwatch

cloudhopper-commons

Cloudhopper Commons

.github

Twitter GitHub Organization-wide files

twitter-cldr-npm

TwitterCldr npm package

bazel-multiversion

Bazel rules to resolve, fetch and manage 3rdparty JVM dependencies with support for multiple parallel versions of the same dependency. Powered by Coursier.

libwatchman

A C interface to watchman

sslconfig

Twitter's OpenSSL Configuration

ios-twitter-apache-thrift

A thrift encoding and decoding library for Swift

gatekeeper-service

GateKeeper is a service built to automate the manual steps involved in onboarding, offboarding, and lost asset scenarios.

dodo

The Twitter OSS Project Builder

repo-scaffolding

Tools for creating repos based on open source standards and best practices

iago2

A load generator, built for engineers

caladrius

Performance modelling system for Distributed Stream Processing Systems (DSPS) such as Apache Heron and Apache Storm

ossdecks

Repository for Twitter Open Source Decks

curation-style-guide

Document Repository for Twitter's Curation Style Guide

analytics-infra-governance

Description of the process for how to commit, review, and release code to the Scalding OSS family (Scalding, Summingbird, Algebird, Bijection, Storehaus, etc)

gpl-commitment

Twitter's GPL Cooperation Commitment

second-control-probability-distributions

google-tag-manager-event-tag

google-tag-manager-base-tag