Discover TransparencyToolkit/UtilityScripts Open Source project

Stars
18
Rank 1,208,065 (Top 24 %)
Language
Ruby
License
GNU General Publi...
Created about 10 years ago
Updated about 6 years ago

TransparencyToolkit

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Scripts for managing scrapers

LookingGlass

Intuitive and configurable search interface for document archives.

Ruby

197

NSA-Data

NSA documents in machine readable form

Ruby

ICWATCH-Data

Resume data and scripts for managing it

Ruby

Harvester

Web crawling and document processing through a usable interface.

JavaScript

TransparencyToolkit

Main repository for Transparency Toolkit

LinkedInData

Scrapes all LinkedIn profiles including search terms.

Ruby

JSONToNetworkGraph

Generates network graphs from a JSON.

JavaScript

generalscraper

Scrapes all pages on any site you specify for keywords.

Ruby

DocManager

Universal backend for indexing, storing, and querying documents.

Ruby

IndeedScraper

Scraper for Indeed

ArchivePile

A read-only theme for publishing email archives using Mailpile

CSS

Transparency-Toolkit-Prototype

Analysis system for Transparency Toolkit.

Ruby

Twiddler

A user friendly tool for text processing, light NLP, and keyword extraction

JavaScript

LinkedinCrawler

Crawls public LinkedIn profiles

Ruby

dataspec-sii

Dataspec for SII

Ruby

LinkedinParser

A parser for LinkedIn profiles

Ruby

Catalyst

Text mining framework.

Ruby

Surveillance-Research-Data

Raw data and scripts for Surveillance Research Archive

Ruby

TwitterCrawler

A crawler for Twitter

Ruby

IndeedCrawler

Crawler for the resume website Indeed

Ruby

NameToEmail

Gets a list of potential emails from a JSON with names.

Ruby

JSONToMap

Converts a JSON with locations into a map with points.

Ruby

Thumbtack

An open narrative mapping tool to corroborate narratives across multiple sources and formats

Ruby

DesignAssets

A collection of branding, interfaces, and other visual resources!

HTML

JSONToChoropleth

Generates choropleth maps from JSONs.

Ruby

theme-snowden

A theme for LookingGlass for Snowden doc search

CSS

FacebookCrawler

A crawler for Facebook data from public web and Graph API

Ruby

EmailParser

A crawler for converting email files on disk to JSON

Ruby

ParseFile

OCRs document and extracts metadata

Ruby

ExtractPatterns

Extracts terms matching certain patterns. For finding new codewords and tracking mentions of known ones.

Ruby

TSJobCrawler

Collects listings for jobs that require security clearance.

Ruby

EntityExtractor

Extracts entities and terms matching certain patterns.

Ruby

IndeedParser

Parser for Indeed resumes

Ruby

transparencytoolkit.github.io

A styleguide site for Transparency Toolkit

CSS

CrawlerManager

API for calling crawlers

Ruby

dataspec-LinkedinCrawl

A LookingGlass dataspec file for data scraped form LinkedIn.com

PiplCollector

Request info from Pipl for all items in dataset

Ruby

DirCrawl

Runs block of code on every file in directory

Ruby

PiplRequest

Request profiles from Pipl

Ruby

JSONToChart

Converts JSONs to pretty charts

Ruby

dataspec-IndeedCrawl

A LookingGlass dataspec file for data scraped from Indeed.com

wlsearchscraper

Gets a list of results from the WikiLeaks search.

Ruby

Archiver

Archives URLs

Ruby

theme-pi

A theme for Privacy International collaborations

CSS

UploadConvert

Tools for converting documents uploaded to Transparency Toolkit to properly formatted JSONs.

Ruby

dataspec-GoogleCrawl

A dataspec for the Google crawler

dataspec-template

A starter template for LookingGlass json files

RequestManager

Manages scraper HTTP requests

Ruby

federalregisterscraper

Scraper for the Federal Register

Ruby

month-names

Names of months in multiple languages

dataspec-fbidhs

JSONCombiner

Combines JSONs.

Ruby

Test-Data

Test data for Transparency Toolkit development

HTML

ArchiveAdministrator

Archive administration system. Handles archive creation and user authentication.

CSS

DocUpload

Upload application for documents in archiving service.

CSS

dataspec-EmailCrawl

Dataspec for emails

IC-Company-Data

Intelligence contractors

Ruby

wordcloud

Changes word sizes in a document based on the number of times they occur.

Ruby

JSONCrossreference

Crossreferences JSONs and returns the matching data.

Ruby

OCRServer

OCR server for hosted archiving service

Ruby

ansible-role-lookingglass

Automates deployment of LookingGlass instances

Shell

NetworkGraph

Neo4j network graph generator prototype

CountryConvert

Converts 2-char ISO country codes to 3-char codes.

Ruby

classification-sensation

Parse classification-related information

Python

dataspec-LoadFiles

Dataspec for plain files loaded in via Harvester/DirCrawl.

dataspec-snowden

A dataspec for Snowden documents

TransparencyToolkit/UtilityScripts

TransparencyToolkit

Reviews

Repository Details

More Repositories