๐ Async pathlib
for Python
aiopath
is a complete implementation of Python's pathlib
that's compatible with asyncio
, trio
, and the async/await
syntax.
All I/O performed by aiopath
is asynchronous and awaitable.
Check out ๐ app_paths
for an example of an app that uses aiopath
, as well as the pyclean
script here.
Use case
If you're writing asynchronous Python code and want to take advantage of pathlib
's conveniences, but don't want to mix blocking and non-blocking I/O, then you can reach for aiopath
.
For example, if you're writing an async web scraping script, you might want to make several concurrent requests to websites and save the responses to persistent storage:
from asyncio import run, gather
from aiohttp import ClientSession
from aiopath import AsyncPath
async def save_page(url: str, name: str):
path = AsyncPath(name)
if await path.exists():
return
async with ClientSession() as session:
async with session.get(url) as response:
content: bytes = await response.read()
await path.write_bytes(content)
async def main():
urls = [
'https://example.com',
'https://github.com/alexdelorenzo/aiopath',
'https://alexdelorenzo.dev',
'https://dupebot.firstbyte.dev'
]
tasks = (
save_page(url, f'{index}.html')
for index, url in enumerate(urls)
)
await gather(*tasks)
run(main())
If you used pathlib
instead of aiopath
, tasks accessing the disk would block the event loop, and async tasks accessing the network would suspend until the event loop was unblocked.
By using aiopath
, the script can access the network and disk concurrently.
Implementation
aiopath
is a direct reimplementation of CPython's pathlib.py
and shares some of its code. aiopath
's class hierarchy directly matches the one from pathlib
, where Path
inherits from PurePath
, AsyncPath
inherits from AsyncPurePath
, and so on.
With aiopath
, methods that perform I/O are asynchronous and awaitable, and methods that perform I/O and return iterators in pathlib
now return async generators. aiopath
goes one step further, and wraps os.scandir()
and DirEntry
to make AsyncPath.glob()
completely async.
aiopath
is typed with Python type annotations, and if using the aiofile
back end, it takes advantage of libaio
for async I/O on Linux.
Usage
aiopath
's API directly matches pathlib
, so check out the standard library documentation for PurePath
and Path
.
Running examples
To run the following examples with top-level await
expressions, launch an asynchronous Python REPL using python3 -m asyncio
or an IPython shell.
You'll also need to install asynctempfile
via PyPI, like so python3 -m pip install asynctempfile
.
pathlib
Replacing All of pathlib.Path
's methods that perform synchronous I/O are reimplemented as asynchronous methods. PurePath
methods are not asynchronous because they don't perform I/O.
from pathlib import Path
from asynctempfile import NamedTemporaryFile
from aiopath import AsyncPath
async with NamedTemporaryFile() as temp:
path = Path(temp.name)
apath = AsyncPath(temp.name)
# check existence
## sync
assert path.exists()
## async
assert await apath.exists()
# check if file
## sync
assert path.is_file()
## async
assert await apath.is_file()
# touch
path.touch()
await apath.touch()
# PurePath methods are not async
assert path.is_absolute() == apath.is_absolute()
assert path.as_uri() == apath.as_uri()
# read and write text
text: str = 'example'
await apath.write_text(text)
assert await apath.read_text() == text
assert not path.exists()
assert not await apath.exists()
You can convert pathlib.Path
objects to aiopath.AsyncPath
objects, and vice versa:
from pathlib import Path
from aiopath import AsyncPath
home: Path = Path.home()
ahome: AsyncPath = AsyncPath(home)
path: Path = Path(ahome)
assert isinstance(home, Path)
assert isinstance(ahome, AsyncPath)
assert isinstance(path, Path)
# AsyncPath and Path objects can point to the same file
assert str(home) == str(ahome) == str(path)
# but AsyncPath and Path objects are not equivalent
assert not home == ahome
AsyncPath
is a subclass of Path
and PurePath
, and a subclass of AsyncPurePath
:
from pathlib import Path, PurePath
from aiopath import AsyncPath, AsyncPurePath
assert issubclass(AsyncPath, Path)
assert issubclass(AsyncPath, PurePath)
assert issubclass(AsyncPath, AsyncPurePath)
assert issubclass(AsyncPurePath, PurePath)
path: AsyncPath = await AsyncPath.home()
assert isinstance(path, Path)
assert isinstance(path, PurePath)
assert isinstance(path, AsyncPurePath)
Check out the test files in the tests
directory for more examples of how aiopath
compares to pathlib
.
Opening a file
You can get an asynchronous file-like object handle by using asynchronous context managers.
AsyncPath.open()
's async context manager yields an anyio.AsyncFile
object.
from asynctempfile import NamedTemporaryFile
from aiopath import AsyncPath
text: str = 'example'
# you can access a file with async context managers
async with NamedTemporaryFile() as temp:
path = AsyncPath(temp.name)
async with path.open(mode='w') as file:
await file.write(text)
async with path.open(mode='r') as file:
result: str = await file.read()
assert result == text
# or you can use the read/write convenience methods
async with NamedTemporaryFile() as temp:
path = AsyncPath(temp.name)
await path.write_text(text)
result: str = await path.read_text()
assert result == text
content: bytes = text.encode()
await path.write_bytes(content)
result: bytes = await path.read_bytes()
assert result == content
Globbing
aiopath
implements pathlib
globbing using async I/O and async generators.
from aiopath import AsyncPath
home: AsyncPath = await AsyncPath.home()
async for path in home.glob('*'):
assert isinstance(path, AsyncPath)
print(path)
downloads: AsyncPath = home / 'Downloads'
if await downloads.exists():
# this might take a while
paths: list[AsyncPath] = \
[path async for path in downloads.glob('**/*')]
Installation
Dependencies
- A POSIX compliant OS, or Windows
- Python 3.7+
requirements.txt
PyPI
$ python3 -m pip install aiopath
Python 3.9 and older
aiopath
for Python 3.9 and older is available on PyPI under versions 0.5.x
and lower.
Python 3.10 and newer
aiopath
for Python 3.10 and newer is available on PyPI under versions 0.6.x
and higher.
GitHub
Download a release archive for your Python version from the releases page.
Then to install, run:
$ python3 -m pip install -r requirements.txt
$ python3 setup.py install
Python 3.9 and older
aiopath
for Python 3.9 and older is developed on the Python-3.9
branch.
Python 3.10 and newer
aiopath
for Python 3.10 and newer is developed on the Python-3.10
branch.
Support
Want to support this project and other open-source projects like it?
License
See LICENSE
. If you'd like to use this project with a different license, please get in touch.
Credit
See CREDIT.md
.