Discover palewire/sopr-activity Open Source project

A step-by-step guide to writing a web scraper with Python

392

first-web-scraper

Quickly import and export delimited data with Django support for PostgreSQL's COPY command

205

django-postgres-copy

A simple Python wrapper for the archive.is capturing service

180

archiveis

A simple Python wrapper and command-line interface for archive.org’s "Save Page Now" capturing service

161

savepagenow

An open-source archive that gathers, saves, shares and analyzes news homepages

149

news-homepages

JavaScript

132

cpi

Quickly adjust U.S. dollars for inflation using the Consumer Price Index (CPI)

The news homepage archive

118

pastpages.org

A simple Python wrapper for Google’s geocoder API

python-googlegeocoder

A step-by-step guide to publishing a simple news application.

first-news-app

U.S. government style guides acquired via the Freedom of Information Act

usa-style-guides

A Django app to download, extract and load campaign finance and lobbying activity data from the California Secretary of State's CAL-ACCESS database

django-calaccess-raw-data

A collection of the work of Edward Estlin Cummings, as it enters the public domain.

cummings.ee

SCSS

first-python-notebook

A step-by-step guide to analyzing data with Python and the Jupyter notebook.

Makefile

storytracker

Tools for tracking stories on news homepages

A simple Python wrapper for U.S. Census Geocoding Services API batch service

censusbatchgeocoder

Django template tools for printing filler, a technique from the days of hot type known as greeking

django-greeking

Inspect a URL and estimate if it contains a news story

storysniffer

A Django database field for storing YAML data

django-yamlfield

A step-by-step guide to publishing a standalone story from a dataset.

first-visual-story

CSS

amateur-satellite-database

The amateur satellites in space

An introduction to free, automated web scraping with GitHub’s powerful new Actions framework.

first-github-scraper

Save My News: A personal, permanent clipping service

savemy.news

A template for open-source Python repositories

python-open-source-template

Python notebooks analyzing campaign finance and lobbying activity data from California Secretary of State’s CAL-ACCESS database

python-calaccess-notebooks

Extracting the "dot plot" economic projections posted online by the Federal Open Market Committee

fed-dot-plot-scraper

A Django app to refine, review and republish campaign finance data drawn from the California Secretary of State’s CAL-ACCESS database

django-calaccess-campaign-browser

Materials from the Django bootcamp at NICAR 2010

nicar2010

The online home of the California Civic Data Coalition

californiacivicdata.org

A set of models and template tags for pulling in lists of content changes across applications.

django-correx

America's most comprehensive dictionary of campaign finance jargon. A free resource created by and for data journalists.

moneyinpolitics.wtf

SCSS

install-python-pipenv-pipfile

Easily install Python, pipenv and Pipfile packages in your GitHub Action

django-anss-archive

A Django application to archive real-time earthquake notifications from the USGS's Advanced National Seismic System

A Twitter bot that posts the latest public records requests filed and completed at muckrock.com

muckrockbot

palewi.re

The once and future blog

A step-by-step guide to creating a simple web application that empowers you to enlist reporters in data entry and refinement.

first-django-admin

A demonstration of how to deploy an Observable Framework dashboard via GitHub Pages.

observable-framework-cpi-example

A bot that posts photographs from the Los Angeles Public Library's digital collection

old-la-photos

Makefile

django-heroku-template

A template for Django projects hosted by Heroku

A plugin for Wordpress web sites to enable the Memento framework for time-based access

wordpress-memento-plugin

Perl

django-calaccess-processed-data

A Django app to transform and refine campaign-finance data from the California Secretary of State’s CAL-ACCESS database

Upload files to an archive.org collection in your GitHub Action

internet-archive-upload

Automated downloads of geographic information system data posted by the National Oceanic and Atmospheric Administration's National Hurricane Center and Central Pacific Hurricane Center

noaa-hurricane-gis-scraper

A bot that posts random images from the Library of Congress collection of Sanborn Fire Insurance Company maps

sanbornmaps

A custom Django model field that automatically archives a URL

django-urlarchivefield

800+ photographs and a digital elevation model from the USGSHawaiian Volcano Observatory's October 2022 Uncrewed Aircraft System (UAS) mission at Halema'uma'u crater, Kilauea Volcano, Hawaii

usgs-hawaii-volcano-drone-survey-october-2022

pandas-combine-workbooks-example

A Twitter bot that posts the latest METAR and EPA weather report for LAX airport

metar-weather-bot

An app for authoring background biographical matter on newsworthy people

django-a-matter

Learn how you can use Python and the Datawrapper API to create a limitless number of charts and maps

first-automated-chart

Automated monitoring of new and updated datasets posted to New York City's data portal

nyc-open-data-monitor

New York City pigeons generated by AI using a random adjective

random-pigeon-gpt

A task runner for the homepages.news open-source archive

news-homepages-runner

hackshackers-dec2010

A GeoDjango app that maps unemployment, meant to demonstrate concepts from my talk.

django-calaccess-lobbying-browser

A simple Django app browse California lobbying activity data from Cal-Access.

A set for helpers for Django web sites to enable the Memento framework for time-based access

django-memento-framework

A custom Django storage system for Internet Archive collections

django-internetarchive-storage

A simple Python wrapper for the MuckRock API

python-muckrock

Create an animated GIF from the PastPages news homepage archive

pastpages2gif

How many trees has Chicago planted? And where?

chicago-trees-analysis

Django helpers for automatically archiving URLs

django-autoarchive

A simple example of using storytracker and the PastPages API to conduct a link analysis

drudge-domain-analysis

Easy access to the official colors of every team in Major League Baseball

mlbcolors

Python notebooks exploring Major League Baseball data

baseball-notebooks

How active are news nerds on GitHub?

news-nerd-repo-analysis

An archive of the Rapture Index at raptureready.com

django-rapture

A script that posts raps by Del Tha Funkee Homosapien to @mistadobalina on Twitter

twitter-mistadobalina

my-first-github-scraper

Scripts for processing and analyzing federal lobbyist disclosure data reporting contributions to political campaigns

sopr-contribs

A feed of headlines with question marks in them.

questionheds

dorling-cartogram-example

Selections from the WFMT collection of Terkel’s radio interviews, delivered each Friday

nicar2011

An example Django app

studs-terkel-podcast

A bot that posts job openings at Reuters News

reuters-jobs

An open-source archive of campaign finance and lobbying disclosure data from the California Secretary of State’s CAL-ACCESS database

django-calaccess-downloads-website

Twitter bot that posts daily updates on a team’s chance to make the Major League Baseball postseason

mlb-postseason-bot

A regional map of Chicago based on the city's official designations

chicago-regions-map

A simple Python wrapper for the webcitation.org capturing service

webcitation

Analysis of the Opening Day starters for the Chicago Cubs baseball team

cubs-opening-day-analysis

Statistics about every U.S. Census tract mapped by @everytract

everytractcount

An official logo of NOAA's Hurricane Hunters released via FOIA

noaa-hurricane-hunters-logo

django-sitemaps-index-bug

A demonstration app that shows a problem I've encountered with django.contrib.sitemaps

Examples of how to remotely execute Jupyter Notebooks from other contexts.

jupyter-notebook-execution-examples

A Chef cookbook and Fabfile for deploying the California Civic Data Coalition's applications for analyzing the California Secretary of State’s CAL-ACCESS database on Amazon Web Services

django-calaccess-cookbook

Class materials for the Django bootcamp at the IRE 2010 conference in Las Vegas

ire2010

A quick and dirty data dump of the H1N1 flu vaccine locations that LA County public health currently buries in a PDF.

django-swineflu

A Django app to scrape campaign-finance data from the California Secretary of State’s CAL-ACCESS website

django-calaccess-scraped-data

A custom template for initializing a new Django project the Data Desk way.

django-project-template

Technical documentation for our pipeline of Django apps that download, extract, load and process the CAL-ACCESS database

django-calaccess-technical-documentation

Buildings marked as unsafe to occupy by the Cedar Rapids city government following the 2020 derecho storms

cedar-rapids-buildings-unsafe-after-derecho-2020

Public art created for light rail stations on the Los Angeles Metro's Regional Connector line.

my-first-visual-story

SCSS

regional-connector-art

nws-aurora

Download forecast data for Aurora Borealis and Aurora Australis from the National Weather Service

literary-notebooks

A Python hack to respond to a Twitter challenge to "select a random geographic point in the US, with the probability weighted by population."

random-tract

Export the e.e. cummings free poetry archive to a variety of ebook formats

ebook-exports

How the Yankee slugger's 2022 pace compares to the past

palewire

My GitHub profile

judge-home-run-analysis