• Stars
    star
    305
  • Rank 136,879 (Top 3 %)
  • Language
    JavaScript
  • License
    MIT License
  • Created over 9 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Creates an XML-Sitemap by crawling a given site.

Sitemap Generator CLI

Travis David npm

Create xml sitemaps from the command line.

Generates a sitemap by crawling your site. Uses streams to efficiently write the sitemap to your drive. Is cappable of creating multiple sitemaps if threshold is reached. Respects robots.txt and meta tags.

Table of contents

Install

This module is available on npm.

npm install -g sitemap-generator-cli
# or execute it directly with npx (since npm v5.2)
npx sitemap-generator-cli https://example.com

Usage

The crawler will fetch all folder URL pages and file types parsed by Google. If present the robots.txt will be taken into account and possible rules are applied for each URL to consider if it should be added to the sitemap. Also the crawler will not fetch URL's from a page if the robots meta tag with the value nofollow is present and ignore them completely if noindex rule is present. The crawler is able to apply the base value to found links.

sitemap-generator [options] <url>

When the crawler finished the XML Sitemap will be built and saved to your specified filepath. If the count of fetched pages is greater than 50000 it will be splitted into several sitemap files and create a sitemapindex file. Google does not allow more than 50000 items in one sitemap.

Example:

sitemap-generator http://example.com

Options

sitemap-generator --help

  Usage: cli [options] <url>

  Options:

    -V, --version                           output the version number
    -f, --filepath <filepath>               path to file including filename (default: sitemap.xml)
    -m, --max-entries <maxEntries>          limits the maximum number of URLs per sitemap file (default: 50000)
    -d, --max-depth <maxDepth>              limits the maximum distance from the original request (default: 0)
    -q, --query                             consider query string
    -u, --user-agent <agent>                set custom User Agent
    -v, --verbose                           print details when crawling
    -c, --max-concurrency <maxConcurrency>  maximum number of requests the crawler will run simultaneously (default: 5)
    -r, --no-respect-robots-txt             controls whether the crawler should respect rules in robots.txt
    -l, --last-mod                          add Last-Modified header to xml
    -g, --change-freq <changeFreq>          adds a <changefreq> line to each URL in the sitemap.
    -p, --priority-map <priorityMap>        priority for each depth url, values between 1.0 and 0.0, example: "1.0,0.8 0.6,0.4"
    -x, --proxy <url>                       Use the passed proxy URL
    -h, --help                              output usage information

filepath

Path to file to write including the filename itself. Path can be absolute or relative. Default is sitemap.xml.

Examples:

  • sitemap.xml
  • mymap.xml
  • /var/www/sitemap.xml
  • ./sitemap.myext

maxConcurrency

Sets the maximum number of requests the crawler will run simultaneously (default: 5).

maxEntries

Define a limit of URLs per sitemap files, useful for site with lots of urls. Defaults to 50000.

maxDepth

Set a maximum distance from the original request to crawl URLs, useful for generating smaller sitemap.xml files. Defaults to 0, which means it will crawl all levels.

noRespectRobotsTxt

Controls whether the crawler should respect rules in robots.txt.

query

Consider URLs with query strings like http://www.example.com/?foo=bar as individual sites and add them to the sitemap.

user-agent

Set a custom User Agent used for crawling. Default is Node/SitemapGenerator.

verbose

Print debug messages during crawling process. Also prints out a summery when finished.

last-mod

add Last-Modified header to xml

change-freq

adds a line to each URL in the sitemap.

priority-map

add priority for each depth url, values between 1.0 and 0.0, example: "1.0,0.8 0.6,0.4"

License

MIT Β© Lars Graubner

More Repositories

1

sitemap-generator

Easily create XML sitemaps for your website.
JavaScript
406
star
2

jquery-offcanvas

An easy to use jQuery offcanvas plugin.
JavaScript
21
star
3

gatsby-plugin-fathom

Gatsby plugin to add Fathom tracking to your site.
JavaScript
16
star
4

larsgraubner.com

Personal website
Svelte
7
star
5

.dotfiles

πŸ› Lars' personal dotfiles and defaults for MacOS
Lua
5
star
6

node-w3c-validator-cli

Crawls a given site and checks for W3C validity.
JavaScript
5
star
7

true-micro-analytics

Simple analytics server using micro.
JavaScript
4
star
8

helpers

Loose collection of utility functions with expressiveness and minimal overhead in mind.
JavaScript
4
star
9

sass-boilerplate

Basic file structure and styles for new projects.
CSS
3
star
10

til

πŸ“ Today I learned
3
star
11

mobile-development-protips

Collection of useful mobile development tips.
3
star
12

docker-nginx-node

Sample Docker compose project with Nginx reverse proxy and Node.js
JavaScript
3
star
13

github-clone-all

Clone all of your Github repositories, including private repositories.
JavaScript
3
star
14

jquery-gmap

jQuery Wrapper for Google Maps API v3.
JavaScript
2
star
15

mqr

Functional matchMedia wrapper for media query handling.
JavaScript
2
star
16

graphql-typescript-starter

TypeScript
2
star
17

webpack-boilerplate-general

Simple Webpack Boilerplate
JavaScript
2
star
18

symfony-starter

PHP
2
star
19

react-browserify-sass-boilerplate

🚧 DEPRECATED - Base structure and files for frontend templates.
JavaScript
2
star
20

advent-of-code-2022

Go
1
star
21

secure-passphrase

Small PWA for generating secure passphrases
HTML
1
star
22

websitetalk.club

Clubhouse event website for discussing websites
TypeScript
1
star
23

go-ip

Simple service to return client IP
Go
1
star
24

url-shortener

URL shortener built with Laravel
PHP
1
star
25

lars

Code for my personal npm card: "npx lars"
JavaScript
1
star
26

better-switch

More intuitive version of a Javascript switch statement.
JavaScript
1
star
27

screenshotter

Simple service based on Symfony to create screenshots from websites.
PHP
1
star
28

password-chrome-extension

Easily create secure passwords with a single click.
JavaScript
1
star
29

algorithms

Algorithm practice with Go
Go
1
star
30

strong-pwgen-cli

πŸ”’Generate cryptographically strong passwords.
JavaScript
1
star