AWS CRT Python

Python 3 bindings for the AWS Common Runtime.

API documentation
Development guide for contributors to aws-crt-python's source code.

License

This library is licensed under the Apache 2.0 License.

Minimum Requirements:

Python 3.8+

Installation

To install from pip:

python3 -m pip install awscrt

To install from Github:

git clone https://github.com/awslabs/aws-crt-python.git
cd aws-crt-python
git submodule update --init
python3 -m pip install .

See Advanced Build Options for more info about building from source.

Fork and Multiprocessing

aws-crt-python uses background threads. This makes os.fork() unsafe. In a forked child process, all background threads vanish. The child will hang or crash when it tries to communicate with any of these (vanished) threads.

Unfortunately, Python's multiprocessing module defaults to using fork when it creates child processes (on POSIX systems except macOS, in Python versions 3.13 and earlier). multiprocessing is used under the hood by many tools that do work in parallel, including concurrent.futures.ProcessPoolExecutor, and pytorch.multiprocessing.

If you need to use multiprocessing with aws-crt-python, set it to use "spawn" or "forkserver" instead of "fork" (see docs). The Python community agrees, and multiprocessing will changes its default from "fork" to "spawn" in 3.14. It already uses "spawn" by default on macOS (because system libraries may start threads) and on Windows (because fork does not exist).

If you must use fork with aws-crt-python, you may be able to avoid hangs and crashes if you manage your threads very carefully:

Release all CRT resources with background threads (e.g. clean up any io.EventLoopGroup instances).
Join all CRT threads before forking (use common.join_all_native_threads() ).

For an example, see test.test_s3.py.S3RequestTest.test_fork_workaround .

macOS TLS Configuration

By default on macOS, aws-crt-python uses Apple Secure Transport for TLS. This provides FIPS-compliant cryptography and integration with the macOS Keychain (e.g. PKCS#12 credentials), but is limited to TLS 1.2.

To enable TLS 1.3 on macOS, set the environment variable:

export AWS_CRT_USE_NON_FIPS_TLS_13=1

This switches the TLS backend from Apple Secure Transport to s2n-tls with aws-lc as the underlying libcrypto. The tradeoffs are:

	Secure Transport (default)	s2n-tls (`AWS_CRT_USE_NON_FIPS_TLS_13=1`)
TLS versions	Up to TLS 1.2	Up to TLS 1.3
FIPS compliance	Yes	No
macOS Keychain integration	Yes (PKCS#12, system certs)	No

This variable is checked at runtime and only affects macOS. It has no effect on Linux (which always uses s2n-tls) or Windows (which always uses Schannel). Both TLS backends are compiled into the binary when building on macOS; the environment variable selects which one is used.

Keychain Behavior

Please note that on Mac, once a private key is used with a certificate, that certificate-key pair is imported into the Mac Keychain. All subsequent uses of that certificate will use the stored private key and ignore anything passed in programmatically. Beginning in v0.6.2, when a stored private key from the Keychain is used, the following will be logged at the "info" log level:

static: certificate has an existing certificate-key pair that was previously imported into the Keychain. Using key from Keychain instead of the one provided.

Logging

aws-crt-python provides two ways to configure logging from the CRT's native C libraries.

Option 1: Python logging integration (recommended)

Route CRT log messages through Python's standard logging module. Each CRT subsystem gets a hierarchical logger under awscrt (e.g. awscrt.io.event-loop, awscrt.http.http-connection, awscrt.s3.S3Client), so you can filter by package or subsystem.

import logging
from awscrt.logging import init_logging, CRT_LOG_FORMAT

handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(CRT_LOG_FORMAT))
logging.getLogger('awscrt').addHandler(handler)
init_logging(logging.DEBUG)

To filter to a specific CRT package:

# Only show IO logs at DEBUG, silence everything else
logging.getLogger('awscrt').setLevel(logging.WARNING)
logging.getLogger('awscrt.io').setLevel(logging.DEBUG)

logging.DEBUG maps to the CRT's Debug level. For the CRT's more verbose Trace logs (emitted at logging.DEBUG), call awscrt.logging.enable_trace_logs(); disable_trace_logs() restores the previous level.

Option 2: Direct file/stdout logging

Write CRT log output directly to a file, stdout, or stderr. This bypasses Python's logging module and is useful for quick debugging.

from awscrt.io import LogLevel, init_logging

init_logging(LogLevel.Debug, 'stdout')   # or 'stderr', or a file path

Which should I use?

Use awscrt.logging.init_logging (Option 1) if you want CRT logs integrated with your application's existing Python logging setup, with per-subsystem filtering.

Use awscrt.io.init_logging (Option 2) if you just want to dump all CRT logs to a file or console quickly.

These are mutually exclusive — use one or the other, not both.

Crash Handler

You can enable the crash handler by setting the environment variable AWS_CRT_CRASH_HANDLER=1 . This will print the callstack to stderr in the event of a fatal error.

Advanced Build Options

OpenSSL and LibCrypto

aws-crt-python does not use OpenSSL for TLS. On Windows, the OS's default TLS library (Schannel) is used. On Apple (macOS), both Secure Transport and s2n-tls are compiled in; the backend is selected at runtime (see macOS TLS Configuration below). On other Unix devices, s2n-tls is used. But s2n-tls uses libcrypto, the cryptography math library bundled with OpenSSL.

To simplify installation, aws-crt-python has its own copy of libcrypto. This lets you install a wheel from PyPI without having OpenSSL installed. Unix wheels on PyPI come with libcrypto statically compiled in. Code to build libcrypto comes from AWS-LC. AWS-LC's code is included in the PyPI source package, and the git repository includes it as a submodule.

If you need aws-crt-python to use the libcrypto included on your system, set environment variable AWS_CRT_BUILD_USE_SYSTEM_LIBCRYPTO=1 while building from source:

AWS_CRT_BUILD_USE_SYSTEM_LIBCRYPTO=1 python3 -m pip install --no-binary :all: --verbose awscrt

( --no-binary :all: ensures you do not use the precompiled wheel from PyPI)

aws-crt-python also exposes a number of cryptographic primitives. On Unix, those depend on libcrypto as described above. On Apple and Windows OS level crypto libraries are used whenever possible. One exception to above statement is that for ED25519 keygen on Windows and Apple, libcrypto is used as no viable OS level alternative exists. In that case Unix level notes about libcrypto apply to Apple and Windows as well. Libcrypto usage for ED25519 support is enabled on Windows and Apple by default and can be disabled by setting environment variable AWS_CRT_BUILD_DISABLE_LIBCRYPTO_USE_FOR_ED25519_EVERYWHERE as follows: (Note: ED25519 keygen functions will start returning not supported error in this case)

AWS_CRT_BUILD_DISABLE_LIBCRYPTO_USE_FOR_ED25519_EVERYWHERE=1 python3 -m pip install --no-binary :all: --verbose awscrt

( --no-binary :all: ensures you do not use the precompiled wheel from PyPI)

AWS_CRT_BUILD_USE_SYSTEM_LIBS

aws-crt-python depends on several C libraries that make up the AWS Common Runtime (libaws-c-common, libaws-c-s3, etc). By default, these libraries are built along with aws-crt-python and statically compiled in (their source code is under crt/).

To skip building these dependencies, because they're already available on your system, set environment variable AWS_CRT_BUILD_USE_SYSTEM_LIBS=1 while building from source:

AWS_CRT_BUILD_USE_SYSTEM_LIBS=1 python3 -m pip install .

If these dependencies are available as both static and shared libs, you can force the static ones to be used by setting: AWS_CRT_BUILD_FORCE_STATIC_LIBS=1

AWS_EXTRA_LIB_DIR

If you need to add additional library directories for the linker to search, set environment variable AWS_EXTRA_LIB_DIR while building from source. This sets the library_dirs parameter for the setuptools Extension, which tells the linker where to find libraries during the build process. These directories supplement (not replace) the linker's default search paths. This is useful when you have custom library installations in non-standard locations.

For more details about library_dirs , see the setuptools Extension documentation.

For a single directory:

AWS_EXTRA_LIB_DIR=/path/to/libs python3 -m pip install .

For multiple directories, separate them with the OS path separator ( : on Unix/macOS, ; on Windows):

# Unix/macOS
AWS_EXTRA_LIB_DIR=/path/to/libs:/another/path python3 -m pip install .

# Windows
AWS_EXTRA_LIB_DIR=C:\path\to\libs;D:\another\path python3 -m pip install .

Windows SDK Version

aws-crt-python builds against windows sdk version 10.0.17763.0 . This is the minimal version required for TLS 1.3 support on Windows. If you need a different Windows SDK version, you can set environment variable AWS_CRT_WINDOWS_SDK_VERSION=<version> while building from source:

Attribution

This library exposes native XXHash implementation (https://github.com/Cyan4973/xxHash).

Name		Name	Last commit message	Last commit date
.builder/actions		.builder/actions
.github		.github
awscrt		awscrt

Latest commit

History

Folders and files