go-fasttld is a high performance effective top level domains (eTLD) extraction module that extracts subcomponents from URLs.
URLs can either contain hostnames, IPv4 addresses, or IPv6 addresses. eTLD extraction is based on the Mozilla Public Suffix List. Private domains listed in the Mozilla Public Suffix List like 'blogspot.co.uk' and 'sinaapp.com' are also supported.
Spot any bugs? Report them here
go get github.com/elliotwutingfeng/go-fasttld
First, build the CLI application.
# `git clone` and `cd` to the go-fasttld repository folder first
make build_cli
Afterwards, try extracting subcomponents from a URL.
# `git clone` and `cd` to the go-fasttld repository folder first
./dist/fasttld extract https://[email protected]%63.uk:5000/a/b\?id\=42
All of the following examples can be found at examples/demo.go
. To play the demo, run the following command:
# `git clone` and `cd` to the go-fasttld repository folder first
make demo
// Initialise fasttld extractor
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
// Extract URL subcomponents
url := "https://[email protected]%63.uk:5000/a/b?id=42"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
// Display results
fasttld.PrintRes(url, res) // Pretty-prints res.Scheme, res.UserInfo, res.SubDomain etc.
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | user | a.subdomain | example | a%63.uk | example.a%63.uk | 5000 | /a/b?id=42 | hostname |
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://127.0.0.1:5000"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | 127.0.0.1 | 127.0.0.1 | 5000 | ipv4 address |
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://[aBcD:ef01:2345:6789:aBcD:ef01:2345:6789]:5000"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | aBcD:ef01:2345:6789:aBcD:ef01:2345:6789 | aBcD:ef01:2345:6789:aBcD:ef01:2345:6789 | 5000 | ipv6 address |
go-fasttld supports the following internationalised label separators (IETF RFC 3490)
Full Stop | Ideographic Full Stop | Fullwidth Full Stop | Halfwidth Ideographic Full Stop |
---|---|---|---|
U+002E . |
U+3002 γ |
U+FF0E οΌ |
U+FF61 q |
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://brb\u002ei\u3002am\uff0egoing\uff61to\uff0ebe\u3002a\uff61fk"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | brb\u002ei\u3002am\uff0egoing\uff61to | be | a\uff61fk | be\u3002a\uff61fk | hostname |
You can use a custom public suffix list file by setting CacheFilePath
in fasttld.SuffixListParams{}
to its absolute path.
cacheFilePath := "/absolute/path/to/file.dat"
extractor, err := fasttld.New(fasttld.SuffixListParams{CacheFilePath: cacheFilePath})
Whenever fasttld.New
is called without specifying CacheFilePath
in fasttld.SuffixListParams{}
, the local cache of the default Public Suffix List is updated automatically if it is more than 3 days old. You can also manually update the cache by using Update()
.
// Automatic update performed if `CacheFilePath` is not specified
// and local cache is more than 3 days old
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
// Manually update local cache
if err := extractor.Update(); err != nil {
log.Println(err)
}
According to the Mozilla.org wiki, the Mozilla Public Suffix List contains private domains like blogspot.com
and sinaapp.com
.
By default, these private domains are excluded (i.e. IncludePrivateSuffix = false
)
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://google.blogspot.com"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | blogspot | com | blogspot.com | hostname |
You can include private domains by setting IncludePrivateSuffix = true
extractor, _ := fasttld.New(fasttld.SuffixListParams{IncludePrivateSuffix: true})
url := "https://google.blogspot.com"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | blogspot.com | google.blogspot.com | hostname |
You can ignore subdomains by setting IgnoreSubDomains = true
. By default, subdomains are extracted.
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://maps.google.com"
res, _ := extractor.Extract(fasttld.URLParams{URL: url, IgnoreSubDomains: true})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | com | google.com | hostname |
By default, internationalised URLs are not converted to punycode before extraction.
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://hello.δΈη.com"
res, _ := extractor.Extract(fasttld.URLParams{URL: url})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | hello | δΈη | com | δΈη.com | hostname |
You can convert internationalised URLs to punycode before extraction by setting ConvertURLToPunyCode = true
.
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://hello.δΈη.com"
res, _ := extractor.Extract(fasttld.URLParams{URL: url, ConvertURLToPunyCode: true})
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// | hello | xn--rhqv96g | com | xn--rhqv96g.com | hostname |
If the URL is invalid, the second value returned by Extract()
, error, will be non-nil. Partially extracted subcomponents can still be retrieved from the first value returned, ExtractResult.
extractor, _ := fasttld.New(fasttld.SuffixListParams{})
url := "https://example!.com" // invalid characters in hostname
color.New().Println("The following line should be an error message")
if res, err := extractor.Extract(fasttld.URLParams{URL: url}); err != nil {
color.New(color.FgHiRed, color.Bold).Print("Error: ")
color.New(color.FgHiWhite).Println(err)
}
fasttld.PrintRes(url, res) // Partially extracted subcomponents can still be retrieved
Scheme | UserInfo | SubDomain | Domain | Suffix | RegisteredDomain | Port | Path | HostType |
---|---|---|---|---|---|---|---|---|
https:// |
# `git clone` and `cd` to the go-fasttld repository folder first
make tests
# Alternatively, run tests without race detection
# Useful for systems that do not support the -race flag like windows/386
# See https://tip.golang.org/src/cmd/dist/test.go
make tests_without_race
# `git clone` and `cd` to the go-fasttld repository folder first
make bench
Benchmark Name | Source |
---|---|
GoFastTld | go-fasttld (this module) |
JPilloraGoTld | github.com/jpillora/go-tld |
JoeGuoTldExtract | github.com/joeguo/tldextract |
Mjd2021USATldExtract | github.com/mjd2021usa/tldextract |
Benchmarks performed on AMD Ryzen 7 5800X, Manjaro Linux.
go-fasttld performs especially well on longer URLs.
https://iupac.org/iupac-announces-the-2021-top-ten-emerging-technologies-in-chemistry/
Benchmark Name | Iterations | ns/op | B/op | allocs/op | Fastest |
---|---|---|---|---|---|
GoFastTld | 8037906 | 150.8 ns/op | 0 B/op | 0 allocs/op | βοΈ |
JPilloraGoTld | 1675113 | 716.1 ns/op | 224 B/op | 2 allocs/op | |
JoeGuoTldExtract | 2204854 | 515.1 ns/op | 272 B/op | 5 allocs/op | |
Mjd2021USATldExtract | 1676722 | 712.0 ns/op | 288 B/op | 6 allocs/op |
Benchmark Name | Iterations | ns/op | B/op | allocs/op | Fastest |
---|---|---|---|---|---|
GoFastTld | 6381516 | 181.9 ns/op | 0 B/op | 0 allocs/op | βοΈ |
JPilloraGoTld | 431671 | 2603 ns/op | 928 B/op | 4 allocs/op | |
JoeGuoTldExtract | 893347 | 1176 ns/op | 1120 B/op | 6 allocs/op | |
Mjd2021USATldExtract | 1030250 | 1165 ns/op | 1120 B/op | 6 allocs/op |
Benchmark Name | Iterations | ns/op | B/op | allocs/op | Fastest |
---|---|---|---|---|---|
GoFastTld | 833682 | 1424 ns/op | 0 B/op | 0 allocs/op | βοΈ |
JPilloraGoTld | 734790 | 1640 ns/op | 304 B/op | 3 allocs/op | |
JoeGuoTldExtract | 695475 | 1452 ns/op | 1040 B/op | 5 allocs/op | |
Mjd2021USATldExtract | 330717 | 3628 ns/op | 1904 B/op | 8 allocs/op |
Splitting on "." and taking the last element only works for simple eTLDs like com
, but not more complex ones like oseto.nagasaki.jp
.
go-fasttld stores eTLDs in compressed tries.
Valid eTLDs from the Mozilla Public Suffix List are appended to the compressed trie in reverse-order.
Given the following eTLDs
au
nsw.edu.au
com.ac
edu.ac
gov.ac
and the example URL host `example.nsw.edu.au`
The compressed trie will be structured as follows:
START
β β au π© β
β ββ edu β
β ββ nsw π© β
ββ ac
β β com π©
β β edu π©
ββ gov π©
=== Symbol meanings ===
π© : path to this node is a valid eTLD
β
: path to this node found in example URL host `example.nsw.edu.au`
The URL host subcomponents are parsed from right-to-left until no more matching nodes can be found. In this example, the path of matching nodes are au -> edu -> nsw
. Reversing the nodes gives the extracted eTLD nsw.edu.au
.
This module is a port of the Python fasttld module, with additional modifications to support extraction of subcomponents from full URLs, IPv4 addresses, and IPv6 addresses.