Data in and out of any S3 store, in Python.
Nine verbs that read and write DataFrames, dicts, text and files by extension, on a Rust core that signs, retries and verifies every transfer. pip install s3pe, two environment variables, done.
Built to be trusted with the data
A client that moves the files a pipeline depends on has one job: either the bytes arrived, or you hear about it. Everything here serves that.
Formats by extension
.json → dict, .csv/.csv.gz/.tsv/.parquet/.xlsx → DataFrame, text files → str, anything else → bytes. format= overrides; extra arguments go to pandas.
A Rust core
Signing, retries and checks happen in the s3pe crate; the GIL is released during transfers. Wheels for Linux (x86_64, aarch64), macOS and Windows, Python 3.9 or later.
Signed the way every store accepts
AWS Signature V4 over the real SHA-256 of the body. No unsigned payloads, no aws-chunked trailers (which some stores reject), path-style URLs.
Retries that know when to stop
Connection failures and HTTP 429/500/502/503/504: three tries, 1 s then 2 s. Never on other 4xx.
Every byte accounted for
ETag against the local MD5 after a PUT; Content-Length and MD5 after a GET. Downloads land in a .s3pe-part file and are renamed when complete.
Typed exceptions
NotFoundError, AccessDeniedError, HttpError (with .status, .code), TransportError, ConfigError, IntegrityError, all under S3Error.
The nine verbs, and four more
Module-level verbs over a default client (s3pe.configure(bucket="my-bucket") sets it), and the same verbs as methods of s3pe.Client. Files above 64 MiB go up as a multipart upload (parts in parallel, aborted cleanly on failure); metadata and headers ride along on writes. No ACLs and no versioning, by design.
| Verb | Does |
|---|---|
read_s3 | GET, decoded by extension or format= |
write_s3 | PUT a DataFrame, dict, list, str, bytes, file-like or Path |
download_s3 | GET to a file, streaming; refuses to overwrite unless asked |
upload_s3 | PUT from a file, streaming |
list_s3 | every S3Object under a prefix, all pages |
exists_s3 | bool |
head_s3 | S3Object or None: size, type, date, ETag, metadata |
delete_s3 | one key or a list (batches of 1000) |
buckets_s3 | list of Bucket |
copy_s3 | server-side copy |
list_s3(delimiter="/") | one level: objects and prefixes (is_prefix) |
presign_s3 | a GET or PUT URL without credentials, up to 7 days |
sync_up_s3 / sync_down_s3 | a directory and a prefix mirrored by ETag, a manifest last |
A key is s3://bucket/key, or a bare key when the bucket is configured (S3PE_BUCKET or the client). Endpoint: S3PE_ENDPOINT (https://host[:port], required); credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
What happens in every call
The same six steps in the four editions.
Resolve and hash
s3://bucket/key or bucket plus key; SHA-256 for the signature, MD5 for the check.
Sign and send
SigV4 over host, date and payload hash; path-style, Content-Length, no chunked trailers.
Retry
Connection failures and 429/5xx, three tries, 1 s then 2 s. Other 4xx come straight back.
Verify and report
ETag and Content-Length checked; one line on stderr; typed errors otherwise.
Install & run
Python 3.9 or later; pandas and pyarrow only for tables. Changes between versions in the changelog.
$ pip install s3pe # bytes, str, dict/list and files
$ pip install "s3pe[pandas]" # also CSV, Parquet and Excel as DataFrames# an explicit client
s3 = s3pe.Client(endpoint="http://127.0.0.1:9000",
bucket="my-bucket",
access_key="AKIA…", secret_key="…")
s3.read("data/manifest.json")
s3.write(df, "reports/sales.csv", index=False)
s3.download("data/regions.geojson", "pe.geojson")
s3.upload("figure1.png", "reports/")
s3.head("etl/sales.parquet") # S3Object or None
s3pe.quiet() # or S3PE_QUIET=1# errors and formats
try:
s3pe.read_s3("s3://my-bucket/missing.json")
except s3pe.NotFoundError:
...
except s3pe.HttpError as e:
print(e.status, e.code)
raw = s3pe.read_s3("x.csv", bucket="b", format="bytes")
df = s3pe.read_s3("x.csv", bucket="b", sep=";", dtype=str)
s3pe.write_s3({"ok": True}, "s3://b/manifest.json")
s3pe.write_s3(df, "s3://b/series.csv.gz") # gzipped on the wayArchitecture
The Python package is a thin layer: formats and logging in Python, everything on the wire in the Rust crate through PyO3. R and Go reimplement the same specification; the four signers reproduce the AWS documentation vector. One repository, one version, one DESIGN.md.
