VIPnytt/RobotsTxtParser VIPnytt/RobotsTxtParser

  • Stars
    star
    26
  • Rank 927,399 (Top 19 %)
  • Language
    PHP
  • License
    MIT License
  • Created over 8 years ago
  • Updated over 3 years ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
VIPnytt/RobotsTxtParser
github

VIPnytt

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

An extensible robots.txt parser and client library, with full support for every directive and specification.

More Repositories

1

SitemapParser

XML Sitemap parser class compliant with the Sitemaps.org protocol.
PHP
73
star
2

OPMLParser

OPML parser for PHP
PHP
2
star
3

CleanParam-URL-filter

Clean-Param URL filtering class - Find and remove any duplicate URL
PHP
2
star
4

UserAgentParser

User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets
PHP
1
star
5

RobotsTagParser

X-Robots-Tag HTTP header parser class
PHP
1
star