adbar/flux-toolchain

Stars
2
Language
Perl
Created about 12 years ago
Updated over 9 years ago

adbar/flux-toolchain

adbar

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Filtering and Language-identification for URL Crawling Seeds (FLUCS) a.k.a. FLUX-Toolchain

trafilatura

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

German-NLP

Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

simplemma

Simple multilingual lemmatizer for Python, especially useful for speed and efficiency

htmldate

Fast and robust date extraction from web pages, with Python or on the command-line

courlan

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

geokelone

integrates spatial and textual data processing tools into a modular software package which features preprocessing, geocoding, disambiguation and visualization

german-reddit

Extraction of a German Reddit Corpus

tweets-tools

Diverse tools used with Twitter data

jlcl-style

Experiments to modernize the LaTeX class of the JLCL

trafilatura_gui

toponyms

Old prototype for toponym extraction in historical texts written in German

url-compressor

A fast pattern-based URL compression for lists of links

zeitcrawler

Automatically exported from code.google.com/p/zeitcrawler

vardial-experiments

Experiments conducted on the occasion of the VarDial shared tasks

microblog-explorer

Perform crawls of social networks (identi.ca, reddit, friendfeed) to gather internal and external links and identify their language

coronakorpus

Material zum Aufbau eines deutschsprachigen COVID-19-Webkorpus / Building a corpus in German dedicated to coronavirus