< Java

Share Share Share

Awesome Java Web Crawling Libraries

• crawler-commons 237

  

  | updated about 2 months ago | Apache License 2.0

  A set of reusable Java components that implement functionality common to any web crawler

• htmlunit 712

  

  | updated over 1 year ago | Apache License 2.0

  HtmlUnit is a "GUI-Less browser for Java programs".

• nutch 2,653

  

  | updated over 1 year ago | Apache License 2.0

  Apache Nutch is an extensible and scalable web crawler

• selenium 30,062

  

  | updated 4 months ago | Apache License 2.0

  A browser automation framework and ecosystem.

• selenium_java 23

  

  | updated 4 months ago

  Collection of reusable loosely Selenium-related code - see https://github.com/sergueik/selenium_tests for strict Selenium test code examples

• tika 1,860

  

  | updated over 1 year ago | Apache License 2.0

  The Apache Tika toolkit detects and extracts metadata and text from over a thousand different file types (such as PPT, XLS, and PDF).

• yauaa 677

  

  | updated over 1 year ago | Apache License 2.0

  Yet Another UserAgent Analyzer