• Stars
    star
    136
  • Rank 267,670 (Top 6 %)
  • Language
    JavaScript
  • Created about 14 years ago
  • Updated about 14 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Massively Distributed Browser-based Javascript Map Reduce Framework. node.js + socket.io (websockets) + webworkers + fun = global warming

Map Crowd Reduce

This project is a “SETI-at-home-like infrastructure for massively distributed cpu-intensive jobs based on HTML5 WebWorkers and node.js for distributing the tasks”. (quote from Pedro Teixeira’s blog).

This project made into the final SAPO Codebits 2010 programming contest, but unfortunately didn’t win :)

The code was built in an afternoon with the help of some friends. It was my first real-world JavaScript code, so there’s a lot to cleanup and improve :)

Quick Overview

The project consists of a node.js server that created a HTTP server on port 3000.

To start a new computation, you must open the main page, and supply 3 or 4 things:

A payload (optional)

You can supply a file payload that will be chuncked by the next function.

A split/segmenter JS function


  function s(input) {
    // optionally read the input and generate a list of segments to be mapped

    return [ ... ];
  }

This function will run on the server side, and will generate a list of jobs to map.

A map JS function


  function map(args) {
    // do heavy computation with args

    return result;
  }

This function will run on client side. Please generate as many as you want :)

A reduce JS function


  function reduce(arg) {
    // arg is [result, result, result, result]

    // reduce them :)
    return final_result;
  }

This function will run on the client side.

After you define the functions, the system gives you an URL that you can share with anyone wanting to contribute with CPU to the computation.

The client side functions are run on a WebWorker so it doesn’t block the main UI thread.

WebSockets are used for maintaining communication between server and clients.

Requirements / installation

On the client you need a modern browser (latest version of Firefox, Chrome and Webkit should work).

On the sever you will need latest node.js stable and npm, and the following node modules:

  • express
  • connect-form
  • uuid
  • haml
  • socket.io

Examples

With the code there are 2 prewritten examples:

  • find the number of primes between 0 and 60000 (brute force)

this uses a brute-force method of testing if each number is a prime, from 0 to 60000. it runs very fast and can be used to debug.

  • find a MD5 collision with the MD5 on all words with 5 or less alfa words

this lasts 15 minutes on a single machine. when I tested it on the main stage, 50 people connected their browsers at it lasted like 20 seconds (#win).

To run the examples you just open the root path of the web server and click on the button example. It will fill all the JS boxes with the prewritten code.

Limitations

The main limitation is how we handle and store the input and results. Right now, when you upload the payload, node.js loads it into memory to pass them to the s function.

Also, if your s function generates billion of jobs, the things could not work because all the data and arguments are kept in memory.

This also means that right now, if you close the server, you loose the current computation.

Security Considerations

There are two main security problems at the moment. First of all, the s function is run on the server side without any security check. I admit that I have no idea on if and how I could do that on node.js. Any hints (and patches) are appreciated :)

The second security problem is a feature :) At the end of the computation, the result of the reduce function is sent to all connected clients and injected in the DOM. This means that the reduce function can generate cool stuff like canvas with a beautiful fractal, but it also means that a malicious user can inject some nasty HTML on the client side.

Further work

As this was built as a quick hack, and by people with no experience on JS other that quick hacks on the browser, the code has a lot to improve. I’m specially looking into better modularization/encapsulation of the code.

As I’m writing this, some friends of mine are looking to build other cool examples, like building a distributed Mandelbrot fractal!

Also, I’m trying to get my feet wet with WebGL, because I think this could be improved by making some heavily number crunching on the client GPU :)

More Repositories

1

twitter-amqp-websocket-example

Simple example of Twitter + AMQP + RabbitMQ + EventMachine + HTML5 WebSocket with no polling
Ruby
172
star
2

fastimage

Finds the type and/or size of a remote image given its uri, by fetching as little as needed.
Go
90
star
3

rails-configd

Online rails configuration generator using etcd as source data
Go
34
star
4

motion-bt-low-energy-example

Simple example of using Bluetooth LE with RubyMotion to connect to a Heart Rate monitor
Ruby
29
star
5

gowalla-osx

[DEAD] Gowalla client(ish) for OS X with a MacRuby flavor
Objective-C
17
star
6

titanium-dropbox

Titanium Dropbox module for iOS
Objective-C
13
star
7

titanium-social

iOS6 Social module for Titanium
Objective-C
11
star
8

titanium-dropboxsync

DropboxSync module for Titanium Mobile iOS
Objective-C
8
star
9

titanium-webserver

Embed HTTP server module for Titanium iOS applications
Objective-C
8
star
10

tarpipe-ruby

Ruby bridge to tarpipe.com's REST API
Ruby
7
star
11

penetrate

penetrate for iOS maps network names into the default keys of known routers
Objective-C
7
star
12

titanium-keychain

Titanium mobile iOS Keychain module
Objective-C
6
star
13

ruby-http2xmpp

A HTTP to XMPP bridge written in Ruby
6
star
14

twitter-streamer-hub

node.js all-purpose, generic twitter UserStreams client hub
JavaScript
5
star
15

sinatra-capistrano-workshop

Code written during the Sinatra and Capistrano workshop at 06/03/2009, in Lisbon
Ruby
4
star
16

DudeNZB

Yet another Cocoa hellanzb frontend (with a MacRuby flavor)
Objective-C
4
star
17

whitme

Ruby library for the URL shortener service whit.me
Ruby
3
star
18

nzbmatrix_pogdesign_safari5

NzbMatrix PogDesign TV Calendar Safari 5 Extension
JavaScript
3
star
19

musicbox

workshop and fun experiment to access SAPO's MusicBox service as an API
Ruby
3
star
20

twitter_auth-with-couchdb

Twitter Auth, CouchDB flavor
Ruby
2
star
21

titanium-twitter

iOS5 Twitter iOS module for Titanium mobile
Objective-C
2
star
22

newzbin_xmpp

Ruby XMPP agent to search the newzbin.com website
Ruby
2
star
23

puny

Ruby library for the URL shortener service created by SAPO
Ruby
2
star
24

pogdesign_newzbin_chrome

Chrome extension to add newzbin download icons to pogdesign tv series calendar
JavaScript
2
star
25

validates_email_whitelist_of

Validates e-mail address format (RFC2822 and RFC3696) and whitelist / blacklist the domain part
Ruby
1
star