< Java

Share Share Share

Awesome Java Web Crawling Libraries

• crawler-commons 228

  

  | updated 4 months ago | Apache License 2.0

  A set of reusable Java components that implement functionality common to any web crawler

• htmlunit 712

  

  | updated 11 months ago | Apache License 2.0

  HtmlUnit is a "GUI-Less browser for Java programs".

• nutch 2,653

  

  | updated 8 months ago | Apache License 2.0

  Apache Nutch is an extensible and scalable web crawler

• selenium 28,975

  

  | updated about 2 months ago | Apache License 2.0

  A browser automation framework and ecosystem.

• selenium_java 22

  

  | updated 26 days ago

  Collection of reusable loosely Selenium-related code - see https://github.com/sergueik/selenium_tests for strict Selenium test code examples

• tika 1,860

  

  | updated 8 months ago | Apache License 2.0

  The Apache Tika toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF).

• yauaa 677

  

  | updated 8 months ago | Apache License 2.0

  Yet Another UserAgent Analyzer