mjhea0/Scrapy-Samples

Stars
196
Rank 198,553 (Top 4 %)
Language
Python
Created about 12 years ago
Updated over 8 years ago

mjhea0/Scrapy-Samples

mjhea0

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Scrapy examples crawling Craigslist

Scrapy Samples

This project contains sample scraping projects/tutorials as detailed in my blog. If you need help with downloading Scrapy and installing the dependencies, please watch this video - http://www.youtube.com/watch?v=eEK2kmmvIdw&feature=plcp

BaseSpider

This is a simple tutorial on how to write a crawler using Scrapy (BaseSpider) to scrape and parse Craigslist Nonprofit jobs in San Francisco and store the data to a CSV file.
Tutorial: http://mherman.org/blog/2012/11/05/scraping-web-pages-with-scrapy/
Video: http://www.youtube.com/watch?v=1EFnX1UkXVU

CrawlSpider

This tutorial adds to the BaseSpider code, in order to build a recursive crawler to scrape multiple pages. Make sure you read the first tutorial first.
Tutorial: http://mherman.org/blog/2012/11/08/recursively-scraping-web-pages-with-scrapy/
Video: http://www.youtube.com/watch?v=P-_TpZ54Vcw

Todo

Add more use cases

awesome-fastapi

A curated list of awesome things related to FastAPI

flaskr-tdd

Flaskr: Intro to Flask, Test-Driven Development (TDD), and JavaScript

awesome-flask

A curated list of awesome things related to Flask

cypress-visual-regression

Module for adding visual regression testing to Cypress

python-ruby

Should I Learn Python or Ruby?

passport-local-express4

check out the blog post

node-stripe-charge

node + stripe + express + bootstrap 4 (used for one time charges)

microservice-movies

sublime-setup-for-python

setting up sublime text for python development

flask-redis-queue

Example of how to handle background processes with Flask, Redis Queue, and Docker

thinkful-html

intro to html and css

thinkful-mentor

code samples, refactoring - for thinkful students

thinkful-angular

AngularJS by Example - Building a Bitcoin Investment Calculator

node-koa-api

flask-tracking

http://www.realpython.com

typescript-node-api

node-postgres-promises

flask-basic-registration

Basic user registration package for Flask.

passport-examples

social auth for node

node-postgres-todo

Flask-Landing

simple landing page to collect prelaunch emails

node-swagger-api

node-mocha-chai-tutorial

whiterspace

Another minimal theme for Octopress.

angular-gulp-browserify-seed

Angular starter featuring Gulp, Bower, and Browserify

node-docker-api

developing and testing microservices with docker

twitter-sentiment-analysis

Video tutorial and accompanying output for conducting text sentiment analysis in Twitter

mean-auth

User Auth with the MEAN Stack

testing-in-django

Example project for testing in Django.

efk-kubernetes

logging in kubernetes

node-postgres-sequelize

passport-local-knex

adding passport to a node app

angular-auth-ngrx

passport-social-auth

node-docker-workflow

http://mherman.org/blog/2015/03/06/node-with-docker-continuous-integration-and-delivery

passport-local

user authentication to Node.js with Passport.js.

node-express-ajax-craigslist

scraping craigslist

web2py

Intro to web2py

sinatra-blog

Sinatra + PostgreSQL + Heroku - on the center stage

flask-stripe

Flask + Stripe + User Registration

flask-spark-docker

Just a boilerplate for PySpark and Flask

python-decorators

into to python decorators

java-for-javascript

Java for JavaScript Developers

node-twitter-sentiment

sentiment analysis with node, express, and the twitter api

mocha-chai-knex

node-token-auth

angular-testing-tutorial

flask-intro

Introduction to Flask showing much of the basic functionality. Plus, I built a task manager application called FlaskTaskr.

node-stripe-example

react-intro

just a basic react intro

flask-rethink

a todo list with flask and rethinkdb

django-stripe

Django + Stripe + User Registration

express-testing-mocha-knex

testing an express app

programming-exercises

Michael Herman's stash of programming exercises

koa-passport-example

noLogo

minimalist octopress theme.

node-grpc-crud

python-devtest

new developer test

vue-intro

flow-node-api

node-getting-started

a little more than hello world!

pinky-promise

why sign an nda when you can just pinky swear instead?

node-workshop

learn javascript and node!

craigslist-housing-scraper

use sparingly, at your own risk

ancestry-scraper

twitter-sentiment-python

basic sentiment analysis in python

bootstrap3

getting started with bootstrap 3

flask-songs-api

octoplate

Octopress theme based on Twitter Bootstrap 3

testcafe-example

flask-ajax-form

Generate a dynamic form in Flask with AJAX

TDD-Django

Blog post and associated code repo describing TDD with Django and PyVows.

docker-workshop

django-1.5-template

django 1.5 + custom registration + bootstrap + heroku

testcafe-visual-regression

running-mate

simple MLOps tool for DS/ML teams

flask-geolocation

that's right: flask and geolocation

django-tutorial

Code from a brief tutorial showing the basics of installing Django and setting up a simple app.

django-tastypie-tutorial

Super Basic REST API with django-tastypie

node-express-swig-mongo

flask-scrapy-test

bottle-plotly-python

bottle.py and the plot.ly API

node-bootstrap3-template

node, express, bootstrap3, boilerplate, starter, template, mongo-ready

microservice-ping-pong

bottle-user-authentication

bottle.py user login/logout/authentication bootstrap3

django-tdd

django 1.6 test driven development!

express-crud-pg-promise

Just a basic Node + Express CRUD app with pg-promise

hibbert-django

node-oauth-openid

node-request-cheerio-scraper

scraping HN => CSV

angular4-crud

job-scraper

generator-herman-express

generator for a basic Node/Express boilerplate

docker-rails-ecs

deliver rails app to ecs

mean-token-auth

textbook

learning management system

mjhea0.github.com

code for my blog, powered by github pages and jekyll

ultimate-craigslist-scraper

get jobs. lots of 'em.

indeed-scraper

web2py-heroku

heroku + web2py