• Stars
    star
    100
  • Rank 338,748 (Top 7 %)
  • Language
    JavaScript
  • License
    Other
  • Created over 13 years ago
  • Updated about 8 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Social media crawling engine implemented totally browser side in JS

Hyve

http://github.com/Tawlk/hyve

About

Hyve (pronounced: "hive") is a Javascript library aiming to be a general solution for streaming the latest data from multiple social media networks, for any given topic, and abstracting all of them into one simple API.

Current Features

  • Plays nice with all major JS frameworks (jQuery, Mootools etc.)
  • Can run any Javascript function of your choice against each posting retrieved.
  • Outputs all data in a normalized format - USMF
  • Supports searching Facebook, Twitter, Identica, Foursquare, Flickr, Picasa, Wordpress, Youtube, Reddit, Digg, and Delicious
  • Indirectly supports Imgur and Vimeo by catching links shared via other services.
  • Expands urls from t.co and bitly
  • Streams from all services (default), or optionally only selected services
  • Streams personal feed of subscriptions using Oauth tokens (for supported networks)
  • Runs in the browser or on the server (Node.js)

Requirements

For running Hyve in Node.js, you'll need the request library.

You can grab it from npm:

$ npm install request

For running Hyve in the browser in production you'll need to compile it with the grunt library.

You can grab it from npm:

$ npm -g install grunt

Usage

Configuration Options

Data Queueing

When this is enabled all incoming data will be placed into a FIFO queue by so your application can dequeue items as appropriate.

Queued items will be stored in 'hyve.queue' as an array:

console.log(hyve.queue)

>>> {'text':[],'link':[],'video':[],'image':[],'checkin':[]}

Enable Queuing:

hyve.queue_enable = true

Dequeue Queued Item:

var item = hyve.queue.text[0]

hyve.dequeue(item)

Data Recall (Browser Only)

If enabled in conjunction with hyve.queue hyve will cache all queued items using HTML5 localStorage. Hyve will recall any cached items if the page containing hyve is closed and restarted.

Enable Recall:

hyve.recall_enable = true

API Key requirements

In order to use some functions within some services you must supply API Keys.

Foursquare will only return data if the client has HTML5 Geolocaton support. You must also define client_id and client_secret like so:

hyve.feeds.foursquare.client_id = 'your_application_client_id'
hyve.feeds.foursquare.client_secret = 'your_application_client_secret'

Flickr will work without authentication, however the 'views' statistic is not returned without auth. Without 'views' the 'weight' metric will be set to 0.

To get 'views' in Flickr you must specify an API key like so:

hyve.feeds.flickr.api_key = 'your_api_key',

To utilize bitly url un-shortening you must supply a username and API key:

hyve.feeds.bitly.login = 'your_username'
hyve.feeds.bitly.api_key = 'your_api_key'

OAuth Token definitions

In order to use functions of hyve that require OAuth authentication (like the friends/subcriptions stream) you must supply OAuth tokens for each involved service. Obtaining OAuth tokens is outside the scope of hyve.

// Facebook (Oauth2)
hyve.feeds.facebook.oauth_token = 'oauth_token'

// Flickr (Oauth2)
hyve.feeds.flickr.oauth_token = 'oauth_token'

// Youtube (Oauth2)
hyve.feeds.youtube.oauth_token = 'oauth_token'

// Twitter (Oauth1a)
hyve.feeds.twitter.oauth_token = 'oauth_token'
hyve.feeds.twitter.oauth_consumer_key = 'oauth_consumer_key'
hyve.feeds.twitter.oauth_signature = 'oauth_signature'
hyve.feeds.twitter.oauth_nonce = 'oauth_nonce'
hyve.feeds.twitter.oauth_timestamp = 'oauth_timestamp'

// Instagram (Oauth2)
hyve.feeds.instagram.access_token = 'access_token'

Commands

Public search real-time stream

hyve.search.stream(search_terms,callback_function,optional_service_list)

Public recently popular search

hyve.search.popular(search_term,callback_function,optional_service_list)

Personal friends/subscriptions stream

hyve.friends.stream(callback_function,optional_service_list)

Stop running streams

hyve.stop()

Deployment

Browser - Production

For production use in a browser you should use a single-file minified version of hyve.

Compile using grunt:

$ grunt concat min

You will then find a minified version of hyve at dist/hyve.min.js

Browser - Development

During development you can simply source the files you intend to work with:

<script type="text/javascript" src="hyve/src/hyve.core.js">
<script type="text/javascript" src="hyve/src/hyve.twitter.js">
<script type="text/javascript" src="hyve/src/hyve.facebook.js">

Node.JS

In Node.js simply require hyve.core.js and any modules you need.

var hyve = require('src/hyve.core.js')
require('src/hyve.twitter.js')
require('src/hyve.facebook.js')

Examples

Simple echo of Twitter, Facebook, and Identica with pure JS:

var myFunction = function(data){console.log(' + data.service +' : '+ data.text  +')}
var myQuery = 'android'
var myServices = ['Twitter','Facebook','Identica']

hyve.search.stream(myQuery,myFunction,myServices)

Simple Node.js example to output data from all services:

var hyve = require('src/hyve.core.js')
require('src/hyve.twitter.js')
require('src/hyve.facebook.js')

hyve.stream('android', function(data){
    console.log(data.service +' : '+ data.text);
})

Basic live search engine with jQuery:

<!DOCTYPE HTML>
<html>
    <head>
        <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
        <script src="dist/hyve.min.js" type="text/javascript"></script>
        <script type="text/javascript">
            $(document).ready(function() {
                $('#search').bind("click",function(){
                    query=$('#query').val();
                    hyve.search.stream(query,function(data){
                        $('#output').prepend($('<p>' + data.service +' : '+ data.text  +'</p>'))
                    });
                });
            });
        </script>
    </head>
    <body>
        <div id="input">
            <input id="query" type="text" />
            <button id="search">search</button>
        </div>
        <div id="output"></div>
    </body>
</html>

You can find more examples in the demos directory.

Notes

Questions/Comments? Please check us out on IRC via irc://freenode.net/#tawlk

More Repositories

1

youtube-dl

RIAA: Please go die in a fire.
Python
621
star
2

security-token-docs

Documentation of Security Tokens and their uses.
313
star
3

synt

Social sentiment flagger intended to judge given text as: positive, neutral or negative.
Python
129
star
4

raspi-hd44780

Python library for controlling the HD44780 LCD using 6 GPIO pins on the Raspberry Pi.
Python
51
star
5

kral

Provides a unified way to collect and distribute data live from potentially any social network
Python
38
star
6

dotvim

My ultimate vim setup (of doom)
Vim Script
24
star
7

visodea

An open source Kickstarter clone, aimed for more project privacy and control.
JavaScript
14
star
8

osae

Open Sentiment Analysis Engine - Allows humans to flag sentiment for random Twitter samples. The data is saved for use with machine learning sentiment classification projects like Synt
Python
12
star
9

gest

Experimental sentiment classification library in Javascript
JavaScript
12
star
10

dotfiles

All my dotfiles that I port around from system to system with me
Python
11
star
11

django-barcode-auth

Allow django auth system login with 2d barcodes/scanner
Python
11
star
12

rant

CLI driven, blog aware static site generator in Python
Python
10
star
13

django-orbited-chat

Simple "Comet" chat service with log persistence using Django, Orbited, and STOMP
Python
10
star
14

grfety

Code for http://grfety.me - A full screen real-time multi-user drawing application using NodeJS, SockJS, and HTML5 canvas
JavaScript
9
star
15

ewrl

A set of functions for unrolling shortened URLs to their origin and fetching Titles/Descriptions
Python
7
star
16

tawlk-old

The old django codebase for tawlk: A flexible framework for searching, collecting, archiving, interpreting, and responding to public social media, with a scalable web interface.
JavaScript
6
star
17

github-forged-sigs

Demo of GitHub forging git signatures
5
star
18

pixy

Raspberry Pi driven platform for scriptable robotic photo/video capture using digital cameras
Python
5
star
19

influencer

Visually compare the social influence of 2 keywords, using Kral
JavaScript
4
star
20

docker-kms-decrypt

Minimal container to decrypt AWS KMS secrets from env vars to an env file for sourcing to another container
Shell
4
star
21

baren

Baren.js is a 1d/2d barcode generator in javascript forked from bwip-js. This fork is intended to continue project development with public git version control as well as to rewrite the code library to have more modular API suitable for selective importing and easy integration into modern JS frameworks.
JavaScript
4
star
22

diblog

Drop-in PHP Blog with all the main features you would expect from a much heavier blogging solution. Drop it on a server and enjoy.
PHP
3
star
23

stuffdb

A tool to make it trivial to use a mobile phone or web browser to digitally attach special instructions, considerations, safety concerns, notes, or tips about any object in a lab/shop/office environment accessible by generated QR code labels.
Python
3
star
24

scripts

Various random scripts I have written for fun, clients, and to make my life easier.
Shell
3
star
25

python-abx

A simple GTK application for performing ABX audio compression testing using python and gstreamer.
Python
3
star
26

euler-solutions

My personal Project Euler solutions. Spoilers, obviously. This will probably take a very long time to complete...
Python
3
star
27

njkt

Proxy a given web site and inject scripts into it. Also remote reload and remotely change url for complete browser remote control.
JavaScript
3
star
28

craigmailer

Scrapes ads from Craigslist topics then sends them to you as emails you can reply to.
PHP
2
star
29

password-store

My fork of 'pass' ( https://passwordstore.org ) with support for multiple git repos
Shell
2
star
30

infra

Terraform configuration for all my personal infrastructure
HCL
2
star
31

weechat-helm

Helm chart for deploying weechat with access via ssh and relay
Smarty
2
star
32

digallery

Drop-in gallery creator with optional fancy stuff. Stick this in a folder full of images and enjoy.
PHP
2
star
33

pologen

Generates random convincing Poloroid(tm) collages from folders full of images.
PHP
2
star
34

createaband

Created to connect bands to musicians. Ran out of funding before it was done so... heres the code :-)
PHP
2
star
35

lrvick

The repository for my personal "rant" driven website
CSS
2
star
36

nodesb

WIP tool to compile nodejs projects into portable static binaries
Ruby
2
star
37

jquery-simpleslideshow

Most jquery slideshows are way over complicated to understand/use. I raged then made this.
JavaScript
2
star
38

difonts

Drop-in single-file solution for having dynamic text render in any TTF font you want in all major browsers.
PHP
2
star
39

eggweblog

Indexes and allows for searching of a directory of eggdrop-style IRC logs.
PHP
2
star
40

zorkin

Play zork online :D
1
star
41

DarkGmailComplete

Chrome Extension for completing the official Dark Gmail theme.
JavaScript
1
star
42

ircyte

Website for viewing and searching logs, users and stats for given IRC channels.
Python
1
star
43

synhax

Website code for synhax: Creative code snippets only a hacker would appreciate
Python
1
star
44

todoost

Todoost is a Django/HTML5 universal mobile/web instant Todo list app.
JavaScript
1
star
45

cloudprint

Google cloudprint proxy
Python
1
star
46

grapheneos-build

Build repo for GrapheneOS using the AOSP-build build system
Makefile
1
star
47

murfie

A Python CLI tool and library for interacting with the undocumented murfie.com API
Python
1
star
48

cli.life

A curated directory of tools for a CLI driven lifestyle.
1
star
49

jquery-githubindex

jQuery plugin to render an date-sorted html index with stats for any combination of repos for given Github users or individual repos. Useful for portfolios.
JavaScript
1
star
50

jquery-githubnav

jQuery driven stack to render a tree-driven explorer of all your Github code via the Github v3 API
1
star
51

essid-ad

Disposable battery powered ESP8266 ESSID advertisement beacons
Makefile
1
star
52

nfc-switch

Control the state of latching relays via NFC tags.
Arduino
1
star
53

sokral

Distant PHP ancestor of tawlk. Social search engine that that will scale to 3 or 4 users!
JavaScript
1
star
54

provisor

Server that provisions new accounts on a Linux system. Can be configured to allow new users to set up accounts for themselves. Ideal for hackerspaces and community shell servers.
Python
1
star
55

python-boilerplate

Python boilerplate project for a library with a class and testing/linting/coverage/compatibility via a makefile
Makefile
1
star
56

QAV250-Parts

1
star
57

weechat-docker

Docker container that exposes a weechat/tmux session over ssh
Shell
1
star
58

lrvick-old

The old django tree from my website (lrvick.net) before I switched everything over to rant
Python
1
star
59

gibrsh

Showcasing social media stupidity in real-time
1
star
60

jquery-picasagallery

Create a simple browsable image gallery from a given public Picasa account.
JavaScript
1
star
61

hdwallet

A CLI based hierarchical deterministic cryptocurrency wallet generator based on a 24 word mnemonic seed.
1
star