• Stars
    star
    158
  • Rank 237,095 (Top 5 %)
  • Language
    Haskell
  • License
    BSD 3-Clause "New...
  • Created about 11 years ago
  • Updated 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Composable, streaming, and efficient left folds

foldl

Use this foldl library when you want to compute multiple folds over a collection in one pass over the data without space leaks.

For example, suppose that you want to simultaneously compute the sum of the list and the length of the list. Many Haskell beginners might write something like this:

sumAndLength :: Num a => [a] -> (a, Int)
sumAndLength xs = (sum xs, length xs)

However, this solution will leak space because it goes over the list in two passes. If you demand the result of sum the Haskell runtime will materialize the entire list. However, the runtime cannot garbage collect the list because the list is still required for the call to length.

Usually people work around this by hand-writing a strict left fold that looks something like this:

{-# LANGUAGE BangPatterns #-}

import Data.List (foldl')

sumAndLength :: Num a => [a] -> (a, Int)
sumAndLength xs = foldl' step (0, 0) xs
  where
    step (x, y) n = (x + n, y + 1)

That now goes over the list in one pass, but will still leak space because the tuple is not strict in both fields! You have to define a strict Pair type to fix this:

{-# LANGUAGE BangPatterns #-}

import Data.List (foldl')

data Pair a b = Pair !a !b

sumAndLength :: Num a => [a] -> (a, Int)
sumAndLength xs = done (foldl' step (Pair 0 0) xs)
  where
    step (Pair x y) n = Pair (x + n) (y + 1)

    done (Pair x y) = (x, y)

However, this is not satisfactory because you have to reimplement the guts of every fold that you care about and also define a custom strict data type for your fold. Hand-writing the step function, accumulator, and strict data type for every fold that you want to use gets tedious fast. For example, implementing something like reservoir sampling over and over is very error prone.

What if you just stored the step function and accumulator for each individual fold and let some high-level library do the combining for you? That's exactly what this library does! Using this library you can instead write:

import qualified Control.Foldl as Fold

sumAndLength :: Num a => [a] -> (a, Int)
sumAndLength xs = Fold.fold ((,) <$> Fold.sum <*> Fold.length) xs

-- or, more concisely:
sumAndLength = Fold.fold ((,) <$> Fold.sum <*> Fold.length)

To see how this works, the Fold.sum value is just a datatype storing the step function and the starting state (and a final extraction function):

sum :: Num a => Fold a a
sum = Fold (+) 0 id

Same thing for the Fold.length value:

length :: Fold a Int
length = Fold (\n _ -> n + 1) 0 id

... and the Applicative operators combine them into a new datatype storing the composite step function and starting state:

(,) <$> Fold.sum <*> Fold.length = Fold step (Pair 0 0) done
  where
    step (Pair x y) n = Pair (x + n) (y + 1)

    done (Pair x y) = (x, y)

... and then fold just transforms that to a strict left fold:

fold (Fold step begin done) = done (foldl' step begin)

Since we preserve the step function and accumulator, we can use the Fold type to fold things other than pure collections. For example, we can fold a Producer from pipes using the same Fold:

Fold.purely Pipes.Prelude.fold ((,) <$> sum <*> length)
    :: (Monad m, Num a) => Producer a m () -> m (a, Int)

To learn more about this library, read the documentation in the main Control.Foldl module.

Quick start

Install the stack tool and then run:

$ stack setup
$ stack ghci foldl
Prelude> import qualified Control.Foldl as Fold
Prelude Fold> Fold.fold ((,) <$> Fold.sum <*> Fold.length) [1..1000000]
(500000500000,1000000)

How to contribute

Contribute a pull request if you have a Fold that you believe other people would find useful.

Development Status

Build Status

The foldl library is pretty stable at this point. I don't expect there to be breaking changes to the API from this point forward unless people discover new bugs.

License (BSD 3-clause)

Copyright (c) 2016 Gabriella Gonzalez All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

  • Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

  • Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

  • Neither the name of Gabriella Gonzalez nor the names of other contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

More Repositories

1

post-rfc

Blog post previews in need of peer review
2,189
star
2

haskell-nix

Nix and Haskell in production
Nix
1,135
star
3

turtle

Shell programming, Haskell style
Haskell
942
star
4

bench

Command-line benchmark tool
Haskell
867
star
5

simple-twitter

A bare-bones Twitter clone implemented in a single file
Nix
720
star
6

pipes

Compositional pipelines
Haskell
478
star
7

nixos-in-production

Source files for the book "NixOS in Production"
HCL
461
star
8

grace

A ready-to-fork interpreted functional language with type inference
JavaScript
385
star
9

Haskell-Morte-Library

A bare-bones calculus-of-constructions
Haskell
367
star
10

nix-diff

Explain why two Nix derivations differ
Haskell
328
star
11

slides

Slides from talks that I give
Haskell
294
star
12

optparse-generic

Auto-generate a command-line parser for your datatype
Haskell
207
star
13

Haskell-Typed-Spreadsheet-Library

Typed and composable spreadsheets
Haskell
187
star
14

terraform-nixos-ng

Terraform modules for NixOS and spiritual successor to the terraform-nixos project
HCL
122
star
15

Haskell-Annah-Library

Distributed programming language that desugars to Morte
Haskell
92
star
16

Haskell-Lens-Tutorial-Library

The missing tutorial module for the lens library
Haskell
81
star
17

macos-builder

Bootstrap a Linux build VM on macOS
Nix
78
star
18

haskell-in-the-large

A book about large-scale design in Haskell
75
star
19

Haskell-Errors-Library

Type-safe error handling
Haskell
64
star
20

Haskell-MVC-Library

Model-view-controller
Haskell
62
star
21

list-transformer

List monad transformer
Haskell
60
star
22

HasCal

Haskell embedding of PlusCal
Haskell
59
star
23

nixfmt

Automatic code formatter for the Nix programming language
Haskell
52
star
24

managed

A monad for managed values
Haskell
49
star
25

Haskell-MMorph-Library

Monad morphisms
Haskell
47
star
26

dhall-manual

The Dhall Configuration Language Manual
46
star
27

pipes-concurrency

Concurrency for the pipes ecosystem
Haskell
43
star
28

Purescript-to-Python

A compiler from Purescript to Python
Haskell
43
star
29

dhall-dot

Dhall support for the DOT language for graphviz
Dhall
38
star
30

suns-search

Fast all-atom protein structural search engine
Haskell
37
star
31

Haskell-Nix-Derivation-Library

Parse and render *.drv files
Haskell
35
star
32

servant-crud

Example CRUD web server+client using Servant
Haskell
34
star
33

Haskell-Total-Library

Exhaustive pattern matching using lenses, traversals, and prisms
Haskell
31
star
34

Haskell-Server-Generic-Library

Auto-generate a server for your datatype
Haskell
30
star
35

Haskell-Pipes-Parse-Library

Parsing for the pipes ecosystem
Haskell
28
star
36

Haskell-Pipes-Safe-Library

Safety for the pipes ecosystem
Haskell
26
star
37

Haskell-Bears-Library

Relational algebra
Haskell
25
star
38

Haskell-Succinct-Vector-Library

Succinct vectors
Haskell
20
star
39

graph

Dhall support for directed graphs with labeled vertices
Dhall
19
star
40

spire

Algorithmically solve Slay the Spire
Haskell
19
star
41

Haskell-Pipes-HTTP-Library

HTTP client with pipes interface
Haskell
18
star
42

Haskell-Optional-Args-Library

Optional function arguments
Haskell
16
star
43

defaultable-map

Applicative maps
Haskell
15
star
44

Haskell-Pipes-ByteString-Library

ByteString support for pipes
Haskell
14
star
45

dhall-semver

Dhall support for semantic version numbers
Dhall
14
star
46

dhall-kubernetes-charts

This repository is superseded by https://github.com/EarnestResearch/dhall-packages
13
star
47

Haskell-Index-Core-Library

Indexed Types
Haskell
13
star
48

Haskell-Pipes-Extras-Library

Miscellaneous utilities for pipes
Haskell
11
star
49

composable-path

Typed paths that are composable
Haskell
11
star
50

political-notes

Markdown
11
star
51

Haskell-MVC-Updates-Library

Concurrent and combinable updates
Haskell
10
star
52

Haskell-Event-Source-Library

Event sourcing library
Haskell
10
star
53

Haskell-RCPL-Library

Concurrent console input and output
Haskell
10
star
54

sig

Blazing fast signature detection
Haskell
10
star
55

Haskell-Break-Library

Break from a loop
Haskell
9
star
56

ghcjs-demo

Example GHCJS demo
Haskell
8
star
57

Haskell-Nordom-Library

A variation on Morte that is close to the machine (obsoleted in favor of Dhall)
Haskell
7
star
58

Haskell-Dhall-Edit-Library

Autogenerate a curses editor for a Dhall configuration file
Haskell
7
star
59

Haskell-MVC-Updates-Examples-Library

Examples for the mvc-updates library
Haskell
7
star
60

Haskell-DirStream-Library

Easily stream directory contents in constant memory
Haskell
7
star
61

Haskell-Free-Monads-Library

Free monads
Haskell
6
star
62

pipes-web

Deploy pipes over the web
Haskell
5
star
63

Haskell-Transformers-Free-Library

Free monad transformers
Haskell
5
star
64

suns-cmd

Command line client for the Suns search engine
Haskell
5
star
65

tiktoken

Haskell
5
star
66

pipes-platform

The full ecosystem of pipes libraries
Haskell
4
star
67

pipes-group

Group streams into substreams
Haskell
4
star
68

delta-lambda

test language for an automath style type system
Haskell
3
star
69

holepunch

NixOS configuration for tunneling an inbound SSH connection over an outbound HTTPS connection
Nix
3
star
70

Haskell-Pipes-Foldl-Library

Folding utilities for pipes
Haskell
3
star
71

Haskell-Crunch-Library

Fast serialization and deserialization
Haskell
3
star
72

Haskell-Aurora-Library

Haskell-Aurora-Library
Haskell
2
star
73

pipes-ecosystem

testing the pipes suite of libraries
Haskell
2
star
74

Haskell-Record-Library

Command line utility to record command environments and output
Haskell
2
star
75

pipes-process

Shell support for pipes
Haskell
2
star
76

Haskell-Pipes-Handle-Library

Model traditional handles using pipes
Haskell
1
star
77

twill

Twilio API interaction for Haskell
Haskell
1
star
78

min-standalone

Minimal reproducing case for standalone-haddock issue
Haskell
1
star
79

emitter

haskell pipes example
Haskell
1
star
80

code-kata-439

Haskell
1
star