• Stars
    star
    134
  • Rank 270,967 (Top 6 %)
  • Language
    Python
  • License
    GNU Lesser Genera...
  • Created over 3 years ago
  • Updated about 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

๐Ÿ“ Asynchronous pathlib for Python

๐Ÿ“ Async pathlib for Python

aiopath is a complete implementation of Python's pathlib that's compatible with asyncio, trio, and the async/await syntax.

All I/O performed by aiopath is asynchronous and awaitable.

Check out ๐Ÿ“‚ app_paths for an example of an app that uses aiopath, as well as the pyclean script here.

Use case

If you're writing asynchronous Python code and want to take advantage of pathlib's conveniences, but don't want to mix blocking and non-blocking I/O, then you can reach for aiopath.

For example, if you're writing an async web scraping script, you might want to make several concurrent requests to websites and save the responses to persistent storage:

from asyncio import run, gather

from aiohttp import ClientSession
from aiopath import AsyncPath


async def save_page(url: str, name: str):
  path = AsyncPath(name)

  if await path.exists():
    return

  async with ClientSession() as session:
    async with session.get(url) as response:
      content: bytes = await response.read()

  await path.write_bytes(content)


async def main():
  urls = [
    'https://example.com',
    'https://github.com/alexdelorenzo/aiopath',
    'https://alexdelorenzo.dev',
    'https://dupebot.firstbyte.dev'
  ]

  tasks = (
    save_page(url, f'{index}.html')
    for index, url in enumerate(urls)
  )

  await gather(*tasks)


run(main())

If you used pathlib instead of aiopath, tasks accessing the disk would block the event loop, and async tasks accessing the network would suspend until the event loop was unblocked.

By using aiopath, the script can access the network and disk concurrently.

Implementation

aiopath is a direct reimplementation of CPython's pathlib.py and shares some of its code. aiopath's class hierarchy directly matches the one from pathlib, where Path inherits from PurePath, AsyncPath inherits from AsyncPurePath, and so on.

With aiopath, methods that perform I/O are asynchronous and awaitable, and methods that perform I/O and return iterators in pathlib now return async generators. aiopath goes one step further, and wraps os.scandir() and DirEntry to make AsyncPath.glob() completely async.

aiopath is typed with Python type annotations, and if using the aiofile back end, it takes advantage of libaio for async I/O on Linux.

Usage

aiopath's API directly matches pathlib, so check out the standard library documentation for PurePath and Path.

Running examples

To run the following examples with top-level await expressions, launch an asynchronous Python REPL using python3 -m asyncio or an IPython shell.

You'll also need to install asynctempfile via PyPI, like so python3 -m pip install asynctempfile.

Replacing pathlib

All of pathlib.Path's methods that perform synchronous I/O are reimplemented as asynchronous methods. PurePath methods are not asynchronous because they don't perform I/O.

from pathlib import Path

from asynctempfile import NamedTemporaryFile
from aiopath import AsyncPath


async with NamedTemporaryFile() as temp:
  path = Path(temp.name)
  apath = AsyncPath(temp.name)

  # check existence
  ## sync
  assert path.exists()
  ## async
  assert await apath.exists()

  # check if file
  ## sync
  assert path.is_file()
  ## async
  assert await apath.is_file()

  # touch
  path.touch()
  await apath.touch()

  # PurePath methods are not async
  assert path.is_absolute() == apath.is_absolute()
  assert path.as_uri() == apath.as_uri()

  # read and write text
  text: str = 'example'
  await apath.write_text(text)
  assert await apath.read_text() == text

assert not path.exists()
assert not await apath.exists()

You can convert pathlib.Path objects to aiopath.AsyncPath objects, and vice versa:

from pathlib import Path
from aiopath import AsyncPath


home: Path = Path.home()
ahome: AsyncPath = AsyncPath(home)
path: Path = Path(ahome)

assert isinstance(home, Path)
assert isinstance(ahome, AsyncPath)
assert isinstance(path, Path)

# AsyncPath and Path objects can point to the same file
assert str(home) == str(ahome) == str(path)

# but AsyncPath and Path objects are not equivalent
assert not home == ahome

AsyncPath is a subclass of Path and PurePath, and a subclass of AsyncPurePath:

from pathlib import Path, PurePath
from aiopath import AsyncPath, AsyncPurePath


assert issubclass(AsyncPath, Path)
assert issubclass(AsyncPath, PurePath)
assert issubclass(AsyncPath, AsyncPurePath)
assert issubclass(AsyncPurePath, PurePath)

path: AsyncPath = await AsyncPath.home()

assert isinstance(path, Path)
assert isinstance(path, PurePath)
assert isinstance(path, AsyncPurePath) 

Check out the test files in the tests directory for more examples of how aiopath compares to pathlib.

Opening a file

You can get an asynchronous file-like object handle by using asynchronous context managers.

AsyncPath.open()'s async context manager yields an anyio.AsyncFile object.

from asynctempfile import NamedTemporaryFile
from aiopath import AsyncPath


text: str = 'example'

# you can access a file with async context managers
async with NamedTemporaryFile() as temp:
  path = AsyncPath(temp.name)

  async with path.open(mode='w') as file:
    await file.write(text)

  async with path.open(mode='r') as file:
    result: str = await file.read()

  assert result == text

# or you can use the read/write convenience methods
async with NamedTemporaryFile() as temp:
  path = AsyncPath(temp.name)

  await path.write_text(text)
  result: str = await path.read_text()
  assert result == text

  content: bytes = text.encode()

  await path.write_bytes(content)
  result: bytes = await path.read_bytes()
  assert result == content

Globbing

aiopath implements pathlib globbing using async I/O and async generators.

from aiopath import AsyncPath


home: AsyncPath = await AsyncPath.home()

async for path in home.glob('*'):
  assert isinstance(path, AsyncPath)
  print(path)

downloads: AsyncPath = home / 'Downloads'

if await downloads.exists():
  # this might take a while
  paths: list[AsyncPath] = \
    [path async for path in downloads.glob('**/*')]

Installation

Dependencies

  • A POSIX compliant OS, or Windows
  • Python 3.7+
  • requirements.txt

PyPI

$ python3 -m pip install aiopath

Python 3.9 and older

aiopath for Python 3.9 and older is available on PyPI under versions 0.5.x and lower.

Python 3.10 and newer

aiopath for Python 3.10 and newer is available on PyPI under versions 0.6.x and higher.

GitHub

Download a release archive for your Python version from the releases page.

Then to install, run:

$ python3 -m pip install -r requirements.txt
$ python3 setup.py install

Python 3.9 and older

aiopath for Python 3.9 and older is developed on the Python-3.9 branch.

Python 3.10 and newer

aiopath for Python 3.10 and newer is developed on the Python-3.10 branch.

Support

Want to support this project and other open-source projects like it?

Buy Me A Coffee

License

See LICENSE. If you'd like to use this project with a different license, please get in touch.

Credit

See CREDIT.md.

More Repositories

1

cast_control

๐Ÿ“บ Control Chromecasts from Linux and D-Bus
Python
468
star
2

onhold

๐Ÿ”Š Play sounds while and after shell jobs complete
Python
156
star
3

screenshot

๐Ÿ“ธ Automate capturing screenshots of apps and windows on macOS
Python
153
star
4

mpris_server

โ–ถ๏ธ Integrate MPRIS Media Player support into your app
Python
25
star
5

brightness

โ˜€๏ธ Dim your Mac's display from the command line via CoreDisplay
Python
25
star
6

limiter

โฒ๏ธ Easy rate limiting for Python using a token bucket algorithm, with async and thread-safe decorators and context managers
Python
20
star
7

parse_google_sms

๐Ÿ“ฑData forensics and recovery utility for Google Voice chats saved via Google Takeout
Python
18
star
8

save_skype

๐Ÿ’ฌ Data forensics and recovery utility for Skype chats and history
Python
13
star
9

cast_convert

๐Ÿ“ฝ๏ธ Convert videos to Chromecast compatible formats
Python
7
star
10

ding

Python
7
star
11

play_sounds

๐Ÿ”Š Play music and sounds in your Python scripts, synchronously and asynchronously.
Python
6
star
12

intel-gvtg

Scripts for managing Intel virtual gfx and KVM
Shell
6
star
13

strs

๐Ÿงต Easy string tools for the shell
Python
4
star
14

app_paths

๐Ÿ“‚ Like appdirs, but with pathlib, path creation and async support.
Python
4
star
15

grub2systemd

๐Ÿ’พ Convert grub.cfg to systemd-boot EFI loader entries
Python
4
star
16

unpackable

๐Ÿ“ฆ Destructure Python objects
Python
3
star
17

dreamhost-dyndns

Updates Dreamhost DNS Records via DynamicDNS (DynDNS) API
Python
3
star
18

delete_scrobbles

โŒ Automatically delete scrobbles on Last.fm
JavaScript
3
star
19

alexdelorenzo

2
star
20

disjoint_set

Union find / disjoint union set
Python
2
star
21

pinterest_tools

TamperMonkey tools for Pinterest
JavaScript
2
star
22

find-ubuntu-mirrors

๐Ÿ” Find Apt repository mirrors for Ubuntu
Shell
2
star
23

npm-user

๐Ÿ  Install npm packages as a non-root user
Shell
2
star
24

ubuntu-kernel

Download latest mainline kernel from Ubuntu's kernel PPA
Shell
2
star
25

rust-book

Code examples, tests, and responses from the Rust Book https://doc.rust-lang.org/book/
Assembly
2
star
26

transmission_user

Change the user of the transmission-daemon in Debian/Ubuntu
Shell
2
star
27

buf_stream

Rust
1
star
28

nth_py

Python
1
star
29

nth_rs

Return or exclude the nth item via stdin
Rust
1
star
30

minesweeper

minesweeper game + shoddily put together gui + a board solver
Python
1
star
31

cacheit

Caching decorator for Python callables
Python
1
star
32

buffers-rs

โŒ› A stream buffer backed by a temporary file.
Rust
1
star
33

userscripts

JavaScript
1
star
34

html_wrapper

HTML parser with lxml backend. Implements subset of BeautifulSoup API
Python
1
star
35

cheapbook

Send email/SMS notifications about Apple's refurb stock
Python
1
star
36

byte_lines

Iterate over lines of bytes from anything that implements std::io::Read
Rust
1
star
37

buffer

โŒ› A stream buffer backed by tempfile.SpooledTemporaryFile
Python
1
star
38

covid_schools

Script that puts together a COVID spreadsheet
Python
1
star
39

wordpress_menu

Drop down menu for Wordpress
JavaScript
1
star