• Stars
    star
    73
  • Rank 433,657 (Top 9 %)
  • Language
    Ruby
  • Created over 15 years ago
  • Updated over 10 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

brain-dead simple parallel processing for ruby
NAME

  forkoff

SYNOPSIS

  brain-dead simple parallel processing for ruby

URI

  http://rubyforge.org/projects/codeforpeople
  http://github.com/ahoward/forkoff

INSTALL

  gem install forkoff

DESCRIPTION

  forkoff works for any enumerable object, iterating a code block to run in a
  child process and collecting the results.  forkoff can limit the number of
  child processes which is, by default, 2.

SAMPLES

  
  <========< samples/a.rb >========>

  ~ > cat samples/a.rb

    # forkoff makes it trivial to do parallel processing with ruby, the following
    # prints out each word in a separate process
    #
    
      require 'forkoff'
    
      %w( hey you ).forkoff!{|word| puts "#{ word } from #{ Process.pid }"}

  ~ > ruby samples/a.rb

    hey from 7907
    you from 7908


  <========< samples/b.rb >========>

  ~ > cat samples/b.rb

    # for example, this takes only 4 seconds or so to complete (8 iterations
    # running in two processes = twice as fast)
    #
    
      require 'forkoff'
    
      a = Time.now.to_f
    
      results =
        (0..7).forkoff do |i|
          sleep 1
          i ** 2
        end
    
      b = Time.now.to_f
    
      elapsed = b - a
    
      puts "elapsed: #{ elapsed }"
      puts "results: #{ results.inspect }"

  ~ > ruby samples/b.rb

    elapsed: 4.19184589385986
    results: [0, 1, 4, 9, 16, 25, 36, 49]


  <========< samples/c.rb >========>

  ~ > cat samples/c.rb

    # forkoff does *NOT* spawn processes in batches, waiting for each batch to
    # complete.  rather, it keeps a certain number of processes busy until all
    # results have been gathered.  in otherwords the following will ensure that 3
    # processes are running at all times, until the list is complete. note that
    # the following will take about 3 seconds to run (3 sets of 3 @ 1 second).
    #
    
    require 'forkoff'
    
    pid = Process.pid
    
    a = Time.now.to_f
    
    pstrees =
      %w( a b c d e f g h i ).forkoff! :processes => 3 do |letter|
        sleep 1
        { letter => ` pstree -l 2 #{ pid } ` }
      end
    
    
    b = Time.now.to_f
    
    puts
    puts "pid: #{ pid }"
    puts "elapsed: #{ b - a }"
    puts
    
    require 'yaml'
    
    pstrees.each do |pstree|
      y pstree
    end

  ~ > ruby samples/c.rb

    
    pid: 7922
    elapsed: 3.37899208068848
    
    --- 
    a: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward (ruby)
       \-+- 07925 ahoward ruby -Ilib samples/c.rb
    
    --- 
    b: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward ruby -Ilib samples/c.rb
       \-+- 07925 ahoward ruby -Ilib samples/c.rb
    
    --- 
    c: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07923 ahoward ruby -Ilib samples/c.rb
       |-+- 07924 ahoward (ruby)
       \-+- 07925 ahoward ruby -Ilib samples/c.rb
    
    --- 
    d: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07932 ahoward ruby -Ilib samples/c.rb
       |--- 07933 ahoward ruby -Ilib samples/c.rb
       \--- 07934 ahoward ruby -Ilib samples/c.rb
    
    --- 
    e: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07932 ahoward (ruby)
       |-+- 07933 ahoward ruby -Ilib samples/c.rb
       \-+- 07934 ahoward (ruby)
    
    --- 
    f: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07932 ahoward (ruby)
       |-+- 07933 ahoward ruby -Ilib samples/c.rb
       \-+- 07934 ahoward ruby -Ilib samples/c.rb
    
    --- 
    g: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07941 ahoward ruby -Ilib samples/c.rb
       |--- 07942 ahoward ruby -Ilib samples/c.rb
       \--- 07943 ahoward ruby -Ilib samples/c.rb
    
    --- 
    h: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |-+- 07941 ahoward (ruby)
       |-+- 07942 ahoward ruby -Ilib samples/c.rb
       \--- 07943 ahoward ruby -Ilib samples/c.rb
    
    --- 
    i: |
      -+- 07922 ahoward ruby -Ilib samples/c.rb
       |--- 07942 ahoward (ruby)
       \-+- 07943 ahoward ruby -Ilib samples/c.rb
    


  <========< samples/d.rb >========>

  ~ > cat samples/d.rb

    # forkoff supports two strategies of reading the result from the child: via
    # pipe (the default) or via file.  you can select which to use using the
    # :strategy option.
    #
    
      require 'forkoff'
    
      %w( hey you guys ).forkoff :strategy => :file do |word|
        puts "#{ word } from #{ Process.pid }"
      end

  ~ > ruby samples/d.rb

    hey from 7953
    you from 7954
    guys from 7955



HISTORY
  1.1.0 
    - move to a model with one work queue and signals sent from consumers to
    producer to noitify ready state.  this let's smaller jobs race through a
    single process even while a larger job may have one sub-process bound up.
    incorporates a fix from http://github.com/fredrikj/forkoff which meant
    some processes would lag behind when jobs didn't have similar execution
    times.

  1.0.0
    - move to github

  0.0.4
    - code re-org
    - add :strategy option
    - default number of processes is 2, not 8

  0.0.1

    - updated to use producer threds pushing onto a SizedQueue for each consumer
      channel.  in this way the producers do not build up a massize parllel data
      structure but provide data to the consumers only as fast as they can fork
      and proccess it.  basically for a 4 process run you'll end up with 4
      channels of size 1 between 4 produces and 4 consumers, each consumer is a
      thread popping of jobs, forking, and yielding results.

    - removed use of Queue for capturing the output.  now it's simply an array
      of arrays which removed some sync overhead.

    - you can configure the number of processes globally with

        Forkoff.default['proccess'] = 4

    - you can now pass either an options hash

        forkoff( :processes => 2 ) ...

      or plain vanilla number

        forkoff( 2 ) ...

      to the forkoff call

    - default number of processes is 8, not 2
        

  0.0.0

    initial version

More Repositories

1

sekrets

sekrets is a command line tool and library used to securely manage encrypted files and settings in your rails' applications and git repositories.
Ruby
268
star
2

main

a class factory and dsl for generating command line programs real quick
Ruby
266
star
3

open4

open child process with handles on pid, stdin, stdout, and stderr: manage child processes and their io handles easily.
Ruby
194
star
4

map

the ruby container you've always wanted: an ordered string/symbol indifferent hash
Ruby
167
star
5

systemu

univeral capture of stdout and stderr and handling of child process pid for windows, *nix, etc.
Ruby
126
star
6

testy

a BDD testing framework for ruby that's mad at the world and plans to kick it's ass in 78 freakin lines of code
Ruby
98
star
7

configuration

pure ruby scoped configuration files.
Ruby
80
star
8

fattr

fattr.rb is a "fatter attr" for ruby and borrows heavily from the metakoans.rb ruby quiz
Ruby
73
star
9

session

session offers a persistent way to drive the shell (/bin/sh) via ruby
Ruby
63
star
10

dao

sa-weet data access object library for rails. top secret.
Ruby
61
star
11

lockfile

a ruby library for creating NFS safe lockfiles
Ruby
54
star
12

macaddr

cross platform mac address determination for ruby
Ruby
47
star
13

middleman-gibberish

password protect middleman pages - even on s3
JavaScript
47
star
14

threadify

threadify.rb makes it stupid easy to process a bunch of data using 'n' worker threads
Ruby
39
star
15

shared

shared.rb provides a super easy way to share code between classes or modules in a simple way.
Ruby
33
star
16

bj

Backgroundjob (Bj) is a brain dead simple, zero admin, background priority queue for Rails.
Ruby
31
star
17

tagz

tagz.rb generates html, xml, or any sgml variant like a small ninja running across the backs of a herd of giraffes swatting of heads like a mark-up weedwacker.
Ruby
31
star
18

demon

demon.rb - the ruby daemon library you've been waiting for
Ruby
29
star
19

slave

easy management of child process works over pipes and drb
Ruby
27
star
20

rails_build

A very small, very simple, very fast, and bullet proof static site generator built as a Rails 5 engine.
Ruby
25
star
21

objectpool

a simple, robust, generic thread-safe object pool for ruby
Ruby
20
star
22

rq

ruby queue is a zero-admin zero-configuration tool used to create instant unix clusters
Ruby
20
star
23

raptcha

low drain bamage, storage-less, session-less, plugin-less, zero admin, single-source-file secure captcha system for ruby and/or rails.
Ruby
19
star
24

arrayfields

allow keyword access to array instances.
Ruby
18
star
25

tumblr

a command line utility and library for the excellent tumblr blogging platform
Ruby
18
star
26

helene

helene is a plugin for writing rails applications on top of amazon's aws platform including sdb, s3, and sqs
Ruby
17
star
27

rego

run arbitrary commands easily when files change
Ruby
16
star
28

fbomb

fbomb is the dangerous flowdock bot
Ruby
16
star
29

fucking_favicons

fucking favicons fucking suck
Ruby
15
star
30

mongoid-haystack

a mongoid 3 zero-config, zero-integration, POLS pure mongo fulltext solution
Ruby
14
star
31

coxswain

encapsulate pre-forking master / worker pattern for ruby
Ruby
12
star
32

jquery.bires

bandwidth limited progressive image enhancement
JavaScript
12
star
33

hashish

awesome data access layer for rails/ruby projects
Ruby
12
star
34

rails_current

track current_STUFF mo betta
Ruby
9
star
35

assassin

no zombies ever, not even on `exit!` or `kill -9`
Ruby
9
star
36

default_url_options

all relative urls in rails all the time. even in mailers.
Ruby
8
star
37

forkhandle

a teeny library / design pattern for managing connections in a process and thread safe fashion
Ruby
7
star
38

rememberthemilk

simple (162 loc), json only, interface to the excellent RememberTheMilk API
Ruby
7
star
39

wrap

non-sucky :before and :after callbacks for any ruby class
Ruby
7
star
40

conducer

a model+view component for rails that combines the conductor and presenter pattern
Ruby
7
star
41

ro

ro is library for managing your site's content in git, as god intended.
Ruby
7
star
42

rails_default_url_options

you really can have default_url_options everywhere. even in mailers.
Ruby
7
star
43

options

options.rb handles keyword options in ruby in a simple and robust way
Ruby
6
star
44

irbcp

irbcp gives access to your system's clipboard (copy and paste) from irb
Ruby
6
star
45

rails_errors2html

simple and sane active_model error html rendering
Ruby
6
star
46

fukung

perhaps the most important ruby code EVAAARRR! gets random images from http://fukung.net.
Ruby
6
star
47

rails_nav

objectified navigation for rails
Ruby
5
star
48

cssjs

a zero learning curve zero contraints dsl for writing css stylesheets in javascript
JavaScript
5
star
49

pork

pork supports parallel programming in ruby using forked actors and durable sqlite message queues
Ruby
5
star
50

isolation

a small rails app to demonstrate what *you* don't understand about RDBMS transactions
Ruby
5
star
51

mongoid-fts

enable mongodb's new fulltext simply and quickly on your mongoid models, including pagination.
Ruby
4
star
52

lru_cache

a simple but efficient implementation of a size limited least recently used cache in ruby
Ruby
4
star
53

terminator

an external timeout mechanism based on processes and signals
Ruby
4
star
54

cdc

uber simple cross domain communication for javascript/iframes
JavaScript
4
star
55

coerce

a ruby library full of common cast/coercion operations
Ruby
4
star
56

rails_view

render views from anywhere. even without a controller context
Ruby
4
star
57

rails_helper

helper = Helper.new and helper.link_to(:foo)
Ruby
4
star
58

ggeocode

simple wrapper on google's new geocoding api
Ruby
4
star
59

suck

gem to show issues with gem/minigem load ordering
Shell
3
star
60

alpo

a library and design pattern for building sane web applications on top of the rails' stack
Ruby
3
star
61

backup.rake

rails' rake task for backup up and loading data+assets as yaml+files
Ruby
3
star
62

id3rename

id3rename is a program to do simple renaming of mp3 files
3
star
63

nfsutils

Ruby FileUtils for NFS
3
star
64

bookify

development moved to https://github.com/everlater/bookify
Ruby
3
star
65

senv

the 12-factor environment tool your mother told you to use
Ruby
3
star
66

codeforpeople

billions and billions of libs
3
star
67

ledis

a K.I.S.S auto-rotating redis logger for ruby/rails
Ruby
3
star
68

kgb

ultra lightweight javascript decision tree builder
JavaScript
3
star
69

image_cache

a small utility library to facility caching image uploads between form validation failures.
2
star
70

gnip-ruby

Ruby library for utilizing Gnip services.
Ruby
2
star
71

linked_list

a simple linked list implementation for ruby
Ruby
2
star
72

bucket

bucket is a command-line interface for amazon's s3 service
2
star
73

tmpdir_block

extends ruby's built-in Dir.tmpdir to accept a block.
Ruby
2
star
74

bestofyoutube

simple ruby library to grab some good video urls from http://bestofyoutube.com
Ruby
2
star
75

superhash

A general mechanism for defining attribute inheritance structures among objects of any type, not just classes
Ruby
2
star
76

testing.rb

adds the minimal features required in order to make test/unit not suck
Ruby
2
star
77

openobject

a simple property based container that's much more capable than a blankslate but far less polluted than ruby's built-in OpenStruct
Ruby
2
star
78

candy_store

hybrid session store that combines rails' built-in cookie based session store with its database backed one
Ruby
2
star
79

slug

a simple slug library. unicode prepared.
Ruby
2
star
80

markdown

my markdown script
Ruby
2
star
81

mongoid-bolt

mongoid-bolt is a concrete lock implementation and mixin.
Ruby
2
star
82

growltdf

growltdf is the greatest program evar. it let's you scrape http://cyclingnews.com for TDF updates and dump them into growl to you can keep up with the race while working.
2
star
83

nmap

narray + mmap = wicked fast persistent numerical arrays for ruby
Ruby
2
star
84

mob

background jobs for mongoid
Ruby
1
star
85

foobar

1
star
86

ydb

mo-betta the yaml/store.
Ruby
1
star
87

hrs

tracking teh hours from the cli
Ruby
1
star
88

mp3scrape

download shit-loads of mp3s from web pages
Ruby
1
star
89

wapp

golden.image
Ruby
1
star
90

upload_cache

a small utility library to facility caching http file uploads between form validation failures. designed for rails, but usable anywhere.
Ruby
1
star
91

forkify

forkify.rb makes it easy to process a bunch of data using 'n' worker processes
Ruby
1
star
92

ansible

magic unicorns for your deployz
Ruby
1
star
93

campfire

a command line script for using 37signal's campfire chat room
1
star
94

cast

a collection of casting methods for ruby
1
star
95

fifo

a very simple javascript fifo queue / cache
JavaScript
1
star
96

ey-cloud-recipes

A starter repo for custom chef recipes on EY's cloud platform
Ruby
1
star
97

test

1
star
98

one-click-hugo-cms

CSS
1
star
99

attributes

the implementation of attributes.rb borrows many of the best ideas from the metakoans.rb ruby quiz (ps. fattr >= attributes)
Ruby
1
star
100

gnip-expander

relay a gnip publisher stream to another, expanding shortened uris in the process
Ruby
1
star