• Stars
    star
    1,163
  • Rank 39,871 (Top 0.8 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created over 9 years ago
  • Updated 15 days ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

asyncio support for botocore library using aiohttp

aiobotocore

https://travis-ci.com/aio-libs/aiobotocore.svg?branch=master Documentation Status Chat on Gitter

Async client for amazon services using botocore and aiohttp/asyncio.

This library is a mostly full featured asynchronous version of botocore.

Install

$ pip install aiobotocore

Basic Example

import asyncio
from aiobotocore.session import get_session

AWS_ACCESS_KEY_ID = "xxx"
AWS_SECRET_ACCESS_KEY = "xxx"


async def go():
    bucket = 'dataintake'
    filename = 'dummy.bin'
    folder = 'aiobotocore'
    key = '{}/{}'.format(folder, filename)

    session = get_session()
    async with session.create_client('s3', region_name='us-west-2',
                                   aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
                                   aws_access_key_id=AWS_ACCESS_KEY_ID) as client:
        # upload object to amazon s3
        data = b'\x01'*1024
        resp = await client.put_object(Bucket=bucket,
                                            Key=key,
                                            Body=data)
        print(resp)

        # getting s3 object properties of file we just uploaded
        resp = await client.get_object_acl(Bucket=bucket, Key=key)
        print(resp)

        # get object from s3
        response = await client.get_object(Bucket=bucket, Key=key)
        # this will ensure the connection is correctly re-used/closed
        async with response['Body'] as stream:
            assert await stream.read() == data

        # list s3 objects using paginator
        paginator = client.get_paginator('list_objects')
        async for result in paginator.paginate(Bucket=bucket, Prefix=folder):
            for c in result.get('Contents', []):
                print(c)

        # delete object from s3
        resp = await client.delete_object(Bucket=bucket, Key=key)
        print(resp)

loop = asyncio.get_event_loop()
loop.run_until_complete(go())

Context Manager Examples

from contextlib import AsyncExitStack

from aiobotocore.session import AioSession


# How to use in existing context manager
class Manager:
    def __init__(self):
        self._exit_stack = AsyncExitStack()
        self._s3_client = None

    async def __aenter__(self):
        session = AioSession()
        self._s3_client = await self._exit_stack.enter_async_context(session.create_client('s3'))

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        await self._exit_stack.__aexit__(exc_type, exc_val, exc_tb)

# How to use with an external exit_stack
async def create_s3_client(session: AioSession, exit_stack: AsyncExitStack):
    # Create client and add cleanup
    client = await exit_stack.enter_async_context(session.create_client('s3'))
    return client


async def non_manager_example():
    session = AioSession()

    async with AsyncExitStack() as exit_stack:
        s3_client = await create_s3_client(session, exit_stack)

        # do work with s3_client

Supported AWS Services

This is a non-exuastive list of what tests aiobotocore runs against AWS services. Not all methods are tested but we aim to test the majority of commonly used methods.

Service Status
S3 Working
DynamoDB Basic methods tested
SNS Basic methods tested
SQS Basic methods tested
CloudFormation Stack creation tested
Kinesis Basic methods tested

Due to the way boto3 is implemented, its highly likely that even if services are not listed above that you can take any boto3.client('service') and stick await infront of methods to make them async, e.g. await client.list_named_queries() would asynchronous list all of the named Athena queries.

If a service is not listed here and you could do with some tests or examples feel free to raise an issue.

Run Tests

Make sure you have development requirements installed and your amazon key and secret accessible via environment variables:

$ cd aiobotocore
$ export AWS_ACCESS_KEY_ID=xxx
$ export AWS_SECRET_ACCESS_KEY=xxx
$ pipenv sync --dev

Execute tests suite:

$ py.test -v tests

Enable type checking and code completion

Install types-aiobotocore that contains type annotations for aiobotocore and all supported botocore services.

# install aiobotocore type annotations
# for ec2, s3, rds, lambda, sqs, dynamo and cloudformation
python -m pip install 'types-aiobotocore[essential]'

# or install annotations for services you use
python -m pip install 'types-aiobotocore[acm,apigateway]'

# Lite version does not provide session.create_client overloads
# it is more RAM-friendly, but requires explicit type annotations
python -m pip install 'types-aiobotocore-lite[essential]'

Now you should be able to run Pylance, pyright, or mypy for type checking as well as code completion in your IDE.

For types-aiobotocore-lite package use explicit type annotations:

from aiobotocore.session import get_session
from types_aiobotocore_s3.client import S3Client

session = get_session()
async with session.create_client("s3") as client:
    client: S3Client
    # type checking and code completion is now enabled for client

Full documentation for types-aiobotocore can be found here: https://youtype.github.io/types_aiobotocore_docs/

Mailing List

https://groups.google.com/forum/#!forum/aio-libs

Requirements

awscli & boto3

awscli and boto3 depend on a single version, or a narrow range of versions, of botocore. However, aiobotocore only supports a specific range of botocore versions. To ensure you install the latest version of awscli and boto3 that your specific combination or aiobotocore and botocore can support use:

pip install -U 'aiobotocore[awscli,boto3]'

If you only need awscli and not boto3 (or vice versa) you can just install one extra or the other.

More Repositories

1

aiohttp

Asynchronous HTTP client/server framework for asyncio and Python
Python
14,457
star
2

aiomysql

aiomysql is a library for accessing a MySQL database from the asyncio
Python
1,721
star
3

aiopg

aiopg is a library for accessing a PostgreSQL database from the asyncio
Python
1,369
star
4

yarl

Yet another URL library
Python
1,200
star
5

aiokafka

asyncio client for kafka
Python
1,093
star
6

aiocache

Asyncio cache manager for redis, memcached and memory
Python
1,032
star
7

aiojobs

Jobs scheduler for managing background task (asyncio)
Python
820
star
8

janus

Thread-safe asyncio-aware queue for Python
Python
774
star
9

aiohttp-demos

Demos for aiohttp project
Makefile
705
star
10

async-lru

Simple LRU cache for asyncio
Python
695
star
11

aiomonitor

aiomonitor is module that adds monitor and python REPL capabilities for asyncio application
Python
644
star
12

async-timeout

asyncio-compatible timeout class
Python
536
star
13

aiozmq

Asyncio (pep 3156) integration with ZeroMQ
Python
420
star
14

aiodocker

Python Docker API client based on asyncio and aiohttp
Python
417
star
15

multidict

The multidict implementation
Python
386
star
16

aiosmtpd

A reimplementation of the Python stdlib smtpd.py based on asyncio.
Python
318
star
17

create-aio-app

The boilerplate for aiohttp. Quick setup for your asynchronous web service.
Python
306
star
18

aioodbc

aioodbc - is a library for accessing a ODBC databases from the asyncio
Python
304
star
19

aiohttp-devtools

dev tools for aiohttp
Python
248
star
20

aiohttp-session

Web sessions for aiohttp.web
Python
238
star
21

aiohttp-security

auth and permissions for aiohttp
Python
229
star
22

aiohttp-jinja2

jinja2 template renderer for aiohttp.web
Python
228
star
23

aiohttp-admin

admin interface for aiohttp application http://aiohttp-admin.readthedocs.io
Python
216
star
24

aiohttp-cors

CORS support for aiohttp
Python
199
star
25

aiohttp-sse

Server-sent events support for aiohttp
Python
197
star
26

aiohttp-debugtoolbar

aiohttp_debugtoolbar is library for debugtoolbar support for aiohttp
JavaScript
193
star
27

aiozipkin

Distributed tracing instrumentation for asyncio with zipkin
Python
184
star
28

aioftp

ftp client/server for asyncio (http://aioftp.readthedocs.org)
Python
183
star
29

aioelasticsearch

aioelasticsearch-py wrapper for asyncio
Python
139
star
30

aiomcache

Minimal asyncio memcached client
Python
139
star
31

aiosignal

aiosignal: a list of registered asynchronous callbacks
Python
130
star
32

pytest-aiohttp

pytest plugin for aiohttp support
Python
129
star
33

aiorwlock

Read/Write Lock - synchronization primitive for asyncio
Python
129
star
34

sockjs

SockJS Server
Python
118
star
35

frozenlist

`FrozenList` is a `list`-like structure that implements `collections.abc.MutableSequence` and can be made immutable.
Python
95
star
36

aiocassandra

Simple threaded cassandra wrapper for asyncio
Python
88
star
37

aiohttp-remotes

A set of useful tools for aiohttp.web server
Python
80
star
38

aiologstash

asyncio logging handler for logstash
Python
58
star
39

aiocouchdb

CouchDB client built on top of aiohttp (asyncio)
Python
53
star
40

aioamqp_consumer

consumer/producer/rpc library built over aioamqp
Python
36
star
41

aiohttp-mako

mako template renderer for aiohttp.web
Python
31
star
42

aioneo4j

asyncio client for neo4j
Python
30
star
43

aiosparql

An asynchronous SPARQL client library using aiohttp
Python
25
star
44

sort-all

Sort __all__ lists alphabetically
Python
25
star
45

aioga

Google Analytics client for asyncio
Python
22
star
46

aioppspp

IETF PPSP RFC7574 in Python/asyncio
Python
21
star
47

sphinxcontrib-asyncio

Sphinx extension to add asyncio-specific markups
Python
20
star
48

aiohttp-benchmarks

Python
17
star
49

aioloop-proxy

A proxy for asyncio.AbstractEventLoop for testing purposes
Python
14
star
50

aiohttp-flashbag

The library provides flashbag for aiohttp
Python
10
star
51

idna-ssl

Patch ssl.match_hostname for Unicode(idna) domains support
Python
9
star
52

aiohttp-bot

A bot for automating boring tasks
Python
8
star
53

aio-libs.github.io

aio-libs static site
HTML
8
star
54

dynoname

Dynamic name resolution for asyncio libraries
Python
8
star
55

.github

Organization wide community settings
Python
7
star
56

triagers

A repo for request for aio-libs triage
5
star
57

get-releasenote

GitHub action for getting release note from towncrier rendered file
Python
5
star
58

aiohappyeyeballs

Happy Eyeballs for pre-resolved hosts
Python
5
star
59

create-release

GitHub action for release creation
4
star
60

aiohttp-site

A site for aiohttp project
HTML
3
star
61

prepare-coverage

Temporarily store coverage in Artifact storage
Makefile
3
star
62

aiohttp-theme

Sphinx theme for aiohttp
Python
2
star
63

azure-pipelines

Common azure pipelines templates used by aio-libs
2
star
64

night-watch

Ops purposes repo: validates Travis CI configs of all the repos in @aio-libs on daily basis
2
star
65

upload-coverage

Upload coverage chunks previously stored by prepare-coverage action
Makefile
1
star