westandskif/rate-mirrors westandskif/rate-mirrors

  • Stars
    star
    325
  • Rank 129,350 (Top 3 %)
  • Language
    Rust
  • License
    Other
  • Created almost 4 years ago
  • Updated about 1 year ago
Twitter logo Share LinkedIn logo Share Facebook logo Share
westandskif/rate-mirrors
github

westandskif

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Everyday-use client-side map-aware mirror ranking tool (Arch Linux; Manjaro; custom ones)

Rate Mirrors

former Rate Arch Mirrors (changed in v0.4.0) - previous README

Tag Badge

This is a tool, which tests mirror speed for:

  • Arch Linux
    • including Chaotic-AUR
    • including Arch Linux CN
  • Arch Linux ARM
  • Manjaro
  • RebornOS
  • Artix Linux
  • CachyOS
  • EndeavourOS
  • any http/https mirrors via stdin.

It uses info about submarine cables and internet exchanges (kudos to TeleGeography for data) to jump between countries and find fast mirrors. And it's fast enough to run it before each system update (~30 seconds with default options).

Installation

  • ArchLinux AUR: yay -S rate-mirrors-bin - pre-built binary with statically linked musl
  • ArchLinux AUR: yay -S rate-mirrors - build binary from sources, linking glibc dynamically
  • Github releases: pre-built binary with statically linked musl

or build manually:

cargo build --release --locked

Usage

  • format is: rate-mirrors {base options} subcommand {subcommand options}
  • run rate-mirrors help to see base options, which go before subcommand
  • it doesn't need root, but if you wish just pass --allow-root option.

Here are supported subcommands:

Each subcommand has its own options, so run rate-mirrors arch --help to see arch specific options, which should go after arch sub-command.

  1. rate-mirrors arch โ€” fetches Arch Linux mirrors, skips outdated/syncing ones and tests them.

    To backup /etc/pacman.d/mirrorlist file and update it with the rated mirrors run the command below:

    export TMPFILE="$(mktemp)"; \
    sudo true; \
    rate-mirrors --save=$TMPFILE arch --max-delay=43200 \
      && sudo mv /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist-backup \
      && sudo mv $TMPFILE /etc/pacman.d/mirrorlist

    Or if you don't need a backup: rate-mirrors arch | sudo tee /etc/pacman.d/mirrorlist.

  2. rate-mirrors archarm โ€” fetches Arch Linux ARM mirrors and tests them

  3. rate-mirrors manjaro โ€” fetches Manjaro mirrors, skips outdated ones and tests them

  4. rate-mirrors rebornos โ€” fetches RebornOS mirrors and tests them

  5. rate-mirrors artix โ€” fetches Artix Linux mirrors and tests them

  6. rate-mirrors cachyos โ€” fetches CachyOS mirrors and tests them

  7. rate-mirrors endeavouros โ€” fetches/reads EndeavourOS mirrors, skips outdated ones and tests them

  8. rate-mirrors chaotic-aur - fetches Arch Linux Chaotic-AUR mirrors and tests them

  9. rate-mirrors archlinuxcn - fetches Arch Linux CN mirrors and tests them

  10. rate-mirrors stdin โ€” takes mirrors from stdin

Each string should comply with one of two supported formats:

- tab-separated url and country (either name or country code)
- tab-separated country and url โ€” just in case :)
- url

Urls should be what --path-to-test and --path-to-return are joined to.

e.g. we have a file with mirrors (countries are required for country-hopping):

https://mirror-a.mirrors.org/best-linux-distro/
US\thttps://mirror-b.mirrors.org/best-linux-distro/
https://mirror-c.mirrors.org/best-linux-distro/\tDE
https://mirror-d.mirrors.org/best-linux-distro/\tAustria

and we'd like to test it & format output for Arch:

cat mirrors_by_country.txt | \
    rate-mirrors --concurrency=40 stdin \
       --path-to-test="extra/os/x86_64/extra.files" \
       --path-to-return='$repo/os/$arch' \
       --comment-prefix="# " \
       --output-prefix="Server = "

Algorithm

The tool uses the following info:

  • submarine cable connections
  • number of internet exchanges per country and distances to weight country connections
  • continents to naively assume countries of the same continent are directly linked

e.g. steps for arch:

  1. fetch mirrors from Arch Linux - Mirror status as json

  2. skip ones, which havenโ€™t completed syncing (--completion=1 option)

  3. skip ones with delays-since-the-last-sync longer than 1 day (--max-delay option)

  4. sort mirrors by โ€œArch Linux - Mirror Statusโ€ score - the lower the better (--sort-mirrors-by=score_asc option)

  5. take the next country to explore (or --entry-country option, US by default -- no need to change)

  6. find neighbor countries --country-neighbors-per-country=3, using multiple strategies:

    • major internet hubs first ( first two jumps only )
    • closest by distance first ( every jump )

  7. take --country-test-mirrors-per-country=2 mirrors per country, selected at step 6, test speed and find 2 mirrors: 1 fastest and 1 with shortest connection time

  8. take countries of mirrors from step 7 and go to step 5

  9. after --max-jumps=7 jumps are done, take top M mirrors by speed (--top-mirrors-number-to-retest=5), test them with no concurrency, sort by speed and prepend to the resulting list

Example of everyday use on Arch Linux:

alias ua-drop-caches='sudo paccache -rk3; yay -Sc --aur --noconfirm'
alias ua-update-all='export TMPFILE="$(mktemp)"; \
    sudo true; \
    rate-mirrors --save=$TMPFILE arch --max-delay=21600 \
      && sudo mv /etc/pacman.d/mirrorlist /etc/pacman.d/mirrorlist-backup \
      && sudo mv $TMPFILE /etc/pacman.d/mirrorlist \
      && ua-drop-caches \
      && yay -Syyu --noconfirm'

Few notes:

  • the tool won't work with root permissions because it doesn't need them
  • ua- prefix means "user alias"
  • paccache from pacman-contrib package
  • yay is an AUR helper
  • sudo true forces password prompt in the very beginning

To persist aliases, add them to ~/.zshrc or ~/.bashrc (based on the shell you use)

Once done, just launch a new terminal and run:

ua-update-all

Output example:

Here is an example of running the arch mode from Belarus (output truncated):

# STARTED AT: 2021-06-23 21:44:49.694758291 +03:00
# ARGS: rate-mirrors arch
# FETCHED MIRRORS: 845
# MIRRORS LEFT AFTER FILTERING: 471
# JUMP #1
# EXPLORING US
# VISITED US
#     + NEIGHBOR ID (by HubsFirst)
#     + NEIGHBOR UK (by HubsFirst)
#     + NEIGHBOR CN (by HubsFirst)
#     + NEIGHBOR DE (by DistanceFirst)
#     + NEIGHBOR CA (by DistanceFirst)
#     + NEIGHBOR FR (by DistanceFirst)
# [US] SpeedTestResult { speed: 561.60 KB/s; elapsed: 1.087839405s; connection_time: 381.733267ms}
# [US] SpeedTestResult { speed: 1.32 MB/s; elapsed: 1.157819036s; connection_time: 276.577298ms}
# [UK] SpeedTestResult { speed: 2.65 MB/s; elapsed: 1.224126693s; connection_time: 261.353312ms}
# [UK] SpeedTestResult { speed: 5.48 MB/s; elapsed: 1.327602061s; connection_time: 94.138156ms}
# [ID] SpeedTestResult { speed: 179.68 KB/s; elapsed: 745.86963ms; connection_time: 670.100893ms}
# [DE] SpeedTestResult { speed: 1.98 MB/s; elapsed: 1.392069748s; connection_time: 106.878831ms}
# [DE] SpeedTestResult { speed: 1.76 MB/s; elapsed: 1.26812617s; connection_time: 223.358858ms}
# [CA] SpeedTestResult { speed: 1.04 MB/s; elapsed: 962.017229ms; connection_time: 517.499369ms}
# [CA] SpeedTestResult { speed: 1.67 MB/s; elapsed: 1.20327262s; connection_time: 296.884889ms}
# [FR] SpeedTestResult { speed: 2.50 MB/s; elapsed: 1.154907178s; connection_time: 341.534506ms}
# [FR] SpeedTestResult { speed: 2.25 MB/s; elapsed: 1.374341411s; connection_time: 118.696039ms}
#     TOP NEIGHBOR - CONNECTION TIME: UK - 94.138156ms
#     TOP NEIGHBOR - SPEED: UK - 5.48 MB/s
#
# JUMP #2
# EXPLORING UK
#     + NEIGHBOR NL (by HubsFirst)
#     + NEIGHBOR NO (by HubsFirst)
#     + NEIGHBOR AU (by HubsFirst)
#     + NEIGHBOR CZ (by DistanceFirst)
#     + NEIGHBOR CH (by DistanceFirst)
#     + NEIGHBOR SE (by DistanceFirst)
# ...
# JUMP #7
# ...

# RE-TESTING TOP MIRRORS
# [EE] SpeedTestResult { speed: 4.92 MB/s; elapsed: 1.320800025s; connection_time: 178.606272ms}
# [UK] SpeedTestResult { speed: 6.95 MB/s; elapsed: 791.741365ms; connection_time: 709.525808ms}
# [FI] SpeedTestResult { speed: 8.94 MB/s; elapsed: 1.376742398s; connection_time: 123.530685ms}
# [TR] SpeedTestResult { speed: 5.57 MB/s; elapsed: 1.243114863s; connection_time: 257.162963ms}
# [EE] SpeedTestResult { speed: 9.68 MB/s; elapsed: 1.41574022s; connection_time: 84.436077ms}
# ==== RESULTS (top re-tested) ====
#   1. [EE] SpeedTestResult { speed: 9.68 MB/s; elapsed: 1.41574022s; connection_time: 84.436077ms} -> http://mirror.cspacehostings.com/archlinux/
#   2. [FI] SpeedTestResult { speed: 8.94 MB/s; elapsed: 1.376742398s; connection_time: 123.530685ms} -> http://arch.mirror.far.fi/
#   3. [UK] SpeedTestResult { speed: 6.95 MB/s; elapsed: 791.741365ms; connection_time: 709.525808ms} -> http://archlinux.uk.mirror.allworldit.com/archlinux/
#   4. [TR] SpeedTestResult { speed: 5.57 MB/s; elapsed: 1.243114863s; connection_time: 257.162963ms} -> http://mirror.veriteknik.net.tr/archlinux/
#   5. [EE] SpeedTestResult { speed: 4.92 MB/s; elapsed: 1.320800025s; connection_time: 178.606272ms} -> https://mirror.cspacehostings.com/archlinux/
#   6. [BY] SpeedTestResult { speed: 3.66 MB/s; elapsed: 1.455269308s; connection_time: 44.256906ms} -> http://ftp.byfly.by/pub/archlinux/
#   7. [SE] SpeedTestResult { speed: 3.40 MB/s; elapsed: 1.270306507s; connection_time: 230.675741ms} -> https://mirror.osbeck.com/archlinux/
#   8. [BY] SpeedTestResult { speed: 3.40 MB/s; elapsed: 1.467606681s; connection_time: 32.547398ms} -> http://mirror.datacenter.by/pub/archlinux/
# ...
# FINISHED AT: 2021-06-23 21:45:15.555642390 +03:00
Server = http://mirror.cspacehostings.com/archlinux/$repo/os/$arch
Server = http://arch.mirror.far.fi/$repo/os/$arch
Server = http://archlinux.uk.mirror.allworldit.com/archlinux/$repo/os/$arch
Server = http://mirror.veriteknik.net.tr/archlinux/$repo/os/$arch
Server = https://mirror.cspacehostings.com/archlinux/$repo/os/$arch
Server = http://ftp.byfly.by/pub/archlinux/$repo/os/$arch
Server = https://mirror.osbeck.com/archlinux/$repo/os/$arch
Server = http://mirror.datacenter.by/pub/archlinux/$repo/os/$arch

Support

License

The tool is made available under the following Creative Commons License: Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0).

More Repositories

1

os-car-man

Open-source Car Manual
Python
8
star
2

graphima

An attempt to build full-featured WebAssembly-based monolith charts
Rust
5
star
3

user-config

Vim Script
1
star