Discover ArchiveTeam/scrapy-thingy Open Source project

Wget-compatible web downloader and crawler.

1,330

wpull

HTML

551

ArchiveBot

ArchiveBot, an IRC bot for archiving websites

A Dockerfile for the ArchiveTeam Warrior

325

warrior-dockerfile

parler-grab

Ubuntu-Warrior

Scripts to build and boot warrior virtual machine containing Docker

Wget-AT is a modern Wget with Lua hooks, Zstandard (+dictionary) WARC compression and URL-agnostic deduplication.

114

wget-lua

IA.BAK

We back up a lot of stuff from around the web; now it's time to back up the Internet Archive, just in case.

Making a reusable toolkit for writing seesaw scripts

seesaw-kit

URLTeam's second generation of URL shortener archiving tools

terroroftinytown

Grabbing everything from reddit.

reddit-grab

NewsGrabber

Grabbing all news.

Saving all questions and answers from Yahoo! Answers.

yahooanswers-grab

Archiving all to-be-deleted NSFW tumblr blogs.

tumblr-grab

imgur-grab

Archiving imgur.

A configurable, reusable tracker with dashboard

universal-tracker

JavaScript

googleplus-grab

Archiving Google+.

The Seesaw pipeline grab script for the URLTeam (terroroftinytown) project

terroroftinytown-client-grab

wpull fork with fixes and faster parsing using html5-parser; used by grab-site; should go away when wpull is similarly improved

ludios_wpull

HTML

warrior-code2

Boot scripts for the ArchiveTeam Warrior 2

Archiving government FTPs.

ftp-gov-grab

warrior-code

Decentralized web archiving

WebArchiver

soundcloud-grab

Archiving https://500px.com/creativecommons

500px-grab

tinyback

A tiny web scraper

Gamemaker Sandbox Tracker items

gamemaker-sandbox-items

youtube-grab

Archiving all metadata from YouTube (everything except videos themselves due to size)

Archiving general youtube video metadata through innertube for dislikes removal.

youtube-dislikes-grab

Managing items for youtube-dislikes-grab.

youtube-dislikes-items

VideoBot

Specialised bot for periodical grabs and video/audio/etc. webpage scrapes.

Urlteam website, code, ... also, PONIES

urlteam-stuff

urls-grab

Archiving URLs (outlinks) from a variety of sources.

NewsGrabber-Warrior

Archiving Google Sites Classic.

google-sites-grab

flickr-grab

Grabbing Flickr images.

pastebin-grab

Archiving pastebin

Managing items for youtube-grab

youtube-items

wget-lua-forum-scripts

Downloading forums posts with Wget+Lua

http://www.archiveteam.org/index.php?title=Google_Reader

greader-grab

ftp-grab

Save all FTP sites!

Managing items for mediafire-grab.

mediafire-items

Roff

citeseerxpdf-grab

Grabbing all sources of CiteSeerX.

Grabbing twitch.tv videos

twitchtv-grab

mobileme-grab

Downloading MobileMe

Constructing a new warrior VM

warrior-preseed

Thinger to download FTP sites

ftp-nab

Saving courses from Coursera.

coursera-grab

Software behind tracker.tinyarchive.org - Warning: Very hacky code

tinyarchive

formspring-grab

Downloading Formspring

yahoomessages-grab

Archiving Yahoo Messages

splinder-grab

Archiving public telegram messages.

telegram-grab

ffnet-grab

Fanfictioning

Some scripts to process ArchiveTeam uploads

archiveteam-megawarc-factory

roblox-grab

Archiving roblox forums.

Grabbing sandbox.yoyogames.com

gamemaker-sandbox-grab

Grabbing as much of justin.tv's archives as possible

justintv-grab

sourceforge-grab

Archiving SourceForge.

Base Dockerfile for warrior project grab scripts

grab-base-df

Dockerfile

wikis-grab

Grabbing all wikis.

liveleak-grab

Archiving liveleak.com

Archiving Tencent Weibo (t.qq.com), 腾讯微博

tencent-weibo-grab

imdb-grab

Archiving IMDb.

Managing items for reddit-grab.

reddit-items

flashdomains-grab

Copy of domains-grab for Flash sites.

Create queue items for ftp-grab.

ftp-queue

NewLisp

tumblr-grab-test

Archiving Tumblr blogs (an ArchiveTeam Warrior testing project)

Heroku buildpack with the Archive Team Warrior

heroku-buildpack-archiveteam-warrior

An index of the MobileMe downloads

mobileme-index

Ruby

twitchtv-items

Managing twitch.tv items.

A better tracker with more features for ArchiveTeam

Universal-tracker-2

eroshare-grab

Grabbing everything from panoramio

panoramio-grab

Saving all images and content from Blingee.

blingee-grab

Archiving all videos from vid.me.

vidme-grab

Discovering items for livejournal-grab.

livejournal-discovery

github-grab

Archiving GitHub

Archiving mediafire.com URLs.

mediafire-grab

Grabbing all images and other stuff from Fur Affinity.

furaffinity-grab

Discovering twitch.tv content

yahoogroups-grab

Archiving Yahoo! Groups.

webs-grab

puush-grab

twitchtv-discovery-grab

vlive-grab

Archiving vlive.tv.

The services for NewsGrabber.

NewsGrabber-Services

Managing items for parler-grab.

parler-items

furaffinity-items

Readme instructions template for manually running pipeline grab scripts outside the warrior

standalone-readme-template

ArchiveBot-agents

Site-specific agents that work with ArchiveBot

Ruby

ua-grab

Archiving all of .ua.

Saving the full Google Code site!

googlecode-grab

pixiv-2-grab

Archiving pixiv2 images

miiverse-grab

Archiving miiverse

orkut-grab

Download all of Orkut

dpreview-grab

Archiving DPReview

A statistics monitor for the listerine download project @ Archive Team. Massive hack, no tests.

bottle

Ruby

halo-new-grab

Archiving Halo (round 2)

Managing items for googleplus-grab and googleplus2-grab.

googleplus-items

100

furaffinity-discovery