Discover lemire/IndexWikipedia Open Source project

This is a repository for the code posted on my blog

876

Code-used-on-Daniel-Lemire-s-blog

720

fast_double_parser

Fast function to parse strings into double (binary64) floating-point values, enforces the RFC 7159 (JSON standard) grammar: 4x faster than strtod

A compressed alternative to the Java BitSet class

574

javaewah

A simple integer compression library in Java

560

JavaFastPFOR

A simple C library for compressing lists of integers using binary packing

505

simdcomp

480

EWAHBoolArray

A compressed bitmap class in C++.

A C++ library to compress and intersect sorted lists of integers using SIMD instructions

430

SIMDCompressionAndIntersection

SIMD-accelerated base64 codecs

420

fastbase64

388

streamvbyte

Fast integer compression in C using the StreamVByte codec

372

FastPriorityQueue.js

a fast heap-based priority queue in JavaScript

header-only library to validate utf-8 strings at high speeds (using SIMD instructions)

328

fastvalidate-utf-8

286

fastrange

A fast alternative to the modulo reduction

272

fastmod

A C/C++ header file for fast 32-bit division remainders (and divisibility tests) on 64-bit hardware.

C library implementing the ridiculously fast CLHash hashing function

264

clhash

256

externalsortinginjava

External-Memory Sorting in Java

248

rollinghashcpp

Rolling Hash C++ Library

Speed-optimized BitSet implementation for modern browsers and JavaScript engines

185

FastBitSet.js

A simple script to help programmers who want to work within Docker

150

docker_programming_station

Shell

142

despacer

C library to remove white space from strings as fast as possible

133

StronglyUniversalStringHashing

Benchmark showing the we can randomly hash strings very quickly with good universality

Testing common random-number generators (RNG)

133

testingRNG

Fast decoder for VByte-compressed integers

131

MaskedVByte

115

cbitset

A simple bitset library in C

110

lbimproved

Dynamic Time Warping (DTW) library implementing lower bounds (LB_Keogh, LB_Improved...)

High-performance dictionary coding

105

dictionary

Fast random number generators: Vectorized (SIMD) version of xorshift128+

SIMDxorshift

csharpewah

Compressed bitmaps in C#

fastrand

Fast random number generation in an interval in Python: Up to 10x faster than random.randint.

bloofi

Bloofi: A java implementation of multidimensional Bloom filters

Rolling hash functions in Java

rollinghashjava

C library to pack and unpack short arrays of integers as fast as possible

LittleIntPacker

simdpcg

Vectorized version of the PCG random number generator

TypedFastBitSet.js

Speed-optimized BitSet implementation for modern browsers and JavaScript engines, uses typed arrays

TypeScript

FastIntegerCompression.js

Fast integer compression library in JavaScript

Pruning elements in SIMD vectors (i.e., packing left elements)

simdprune

testingmlp

Testing memory-level parallelism

FastDifferentialCoding

Fast differential coding functions (using SIMD instructions)

multiplatform_simd_recipes

SIMD recipes, for various platforms (collection of code snippets)

runningmaxmin

Fast maximum-minimum filters implemented in C++

Reproducible experimeents on UTF-8 validation using SIMD instructions

validateutf8-experiments

A Variable-Length String Hashing Library in C++

fasthashing

Simple C++ code to benchmark fast division algorithms

fast_division

C++ library to pack and unpack vectors of integers having a small range of values using a technique called Frame of Reference

FrameOfReference

SwiftBitset

A fast Bitset class in Swift

Swift

FastShuffleExperiments

How fast can we shuffle values?

A stable heap-based priority queue in JavaScript

programmingishard

Long-term book project

StablePriorityQueue.js

Fast implementations of the scancount algorithm: C++ header-only library

fastscancount

Vectorized (AVX) version of the game of life

SIMDgameoflife

CMemoryUsage

Measuring memory usage in C and C++

CRoaringUnityBuild

Dumps of CRoaring unity builds (for convenience)

SwiftWyhash

Fast random number generator in pure Swift

Swift

kodakimagecollection

Set of images made available royalty-free by Kodak

sparsebitmap

A simple sparse bitmap implementation in java

code from Daniel Lemire, A Better Alternative to Piecewise Linear Time Series Segmentation, SIAM Data Mining 2007, 2007.

PiecewiseLinearTimeSeriesApproximation

C++ implementation of Concise and WAH compressed bitsets

Concise

iOS app to test memory-level parallelism

MemoryLanes

How to "crack" xoroshiro128+: from two outputs, derive a possible seed

fastrandom

crackingxoroshiro128plus

Python

talks

Random material having to do with Daniel Lemire's talks

Some realistic tabular datasets for testing (CSV)

RealisticTabularDataSets

StarSchemaBenchmark

O'Neil et al.'s Star Schema Benchmark: curated code

vectorclass

Random number generator for large applications using vector instructions

Simple benchmark between compressed bitmap libraries in Java

simple_fastfloat_benchmark

simplebitmapbenchmark

Experiments on bit-slice indexing

zobristhashing

Zobrist hashing in C

BitSliceIndex

Vectorized intersections (research code)

SIMDIntersections

microbenchmarks

Private microbenchmarks

backward_multiplication

Multiplying... backward?

benchmarking positional population count

pospopcnt_avx512

Assembly

costofsafety

Quick experiment to see how expensive safety is in C, for research

constantdivisionbenchmarks

Benchmarks for constant-division problems (not a library! for research only!)

Code from article http://locklessinc.com/articles/fast_hash/

createfasthash

SwiftCallingCHeader

Calling a C header from Swift (example)

Swift

arraylayout

Pat Morin's arraylayout

TeX

WebAssemblyVSJavaScript

Project to compare the performance of WebAssembly with JavaScript

This is a set of row-reordering algorithms and data compression compression schemes implemented in C++.

rowreorderingcpplibrary

Captured as http://benchmarksgame.alioth.debian.org is closing

ComputerLanguageBenchmark

HTML

fastheap

Fast heap data structures in Go (golang)

iosbitmapdecoding

Experimenting with bitmap decoding on ios

notesdecours

Notes de cours

Removing Manually-Generated Boilerplate from Project Gutenberg e-Books

gutenberg-headers

A Go project to benchmark various bitmap implementations (this is not a library!)

gobitmapbenchmark

citationdata

Influential references dataset

simple_simdjson_python_wrapper

Simple use of simdjson from python (for research purposes)

Fast minimum-maximal filter in Python

pythonmaxmin

Python

jstypes

Doing C-like arithmetic and logical operations in JavaScript (full 64-bit support)

Hierarchical Bin Buffering C++ Library

hierarchicalbinbuffering

Supplemental tests for the fast_float library (credit: Nigel Tao)

fast_float_supplemental_tests

Do hash table use more memory than trees? A case study in Java.

HashVSTree

zone-file parsing benchmark

zone_benchmarks

DivisionBenchmark

A not-so-exciting benchmark to measure the performance of some division functions

A benchmark comparing roaring and concise

RoaringVSConciseBenchmark

Exercises pour s'approprier lucene, le moteur de recherche

exercices_lucene

Basic benchmark to compare different Roaring bitmap implementations in Rust

roaring_rust_benchmark

Rust

datasciencebook

Python

viewsizeestimation

Unassuming hashing-based view-size estimation techniques

A silly benchmark for Jackson (JSON parser)

100

jackson-json-bench