Data in and out of any S3 store, in one line.
The same nine verbs in four languages for any S3-compatible object store: AWS S3, Garage, MinIO, Ceph. Endpoint and credentials from the environment, format from the extension, every transfer signed, retried and checked.
Four editions, one behaviour
Pick the language of the project. Everything else is the same: defaults, signing, retries, integrity checks, log lines and error kinds. One repository, one version, one changelog.
R
Pure R on httr2 and cli, tidyverse style. Tibbles from CSV and Parquet, lists from JSON, sf from GeoJSON.
install.packages("s3pe")Python
A binding on the Rust core, wheels for every platform. DataFrames with pandas, dicts from JSON, bytes otherwise.
pip install "s3pe[pandas]"Rust
The core: SigV4, retries and checks over ureq and rustls. Also the s3pe command line.
cargo add s3pe
cargo install s3peGo
Standard library only, context on every call, typed errors for errors.Is.
go get github.com/StrategicProjects/s3peThe nine verbs
A key is s3://bucket/key, or a bare key when the bucket is configured. Nothing else to learn.
| Verb | Does | R and Python return |
|---|---|---|
read_s3 | GET an object into memory | decoded by extension: tibble / DataFrame, list / dict, text, raw bytes |
write_s3 | PUT an object from memory | encoded by extension: CSV, CSV.GZ, TSV, JSON, GeoJSON, Parquet, XLSX, RDS, text, bytes |
download_s3 | GET to a file, streaming | written as .s3pe-part, renamed when complete |
upload_s3 | PUT from a file, streaming | content type guessed from the key |
list_s3 | every object under a prefix, all pages | tibble / list of records: key, size, last_modified, etag |
exists_s3 | HEAD, 404 as false | logical / bool |
head_s3 | HEAD: size, type, date, ETag | one record, or nothing |
delete_s3 | DELETE, idempotent | nothing |
buckets_s3 | the buckets the credentials see | names and creation dates |
Rust and Go expose the same verbs (s3pe::read_s3, s3pe.ReadS3) and a Client for explicit settings; R has s3_config().
What happens in every call
The parts that make the difference between "it usually works" and "it worked, and we know it did".
Sign
AWS Signature V4 over the real SHA-256 of the payload. No unsigned payloads, no aws-chunked (some stores reject it). The four signers reproduce the AWS documentation vector.
Retry
Connection failures and HTTP 429, 500, 502, 503, 504: three tries, 1 s then 2 s. Never on other 4xx.
Verify
After an upload the server's ETag against the local MD5; after a download the byte count and the MD5 of what arrived.
Report
One line per transfer, silenced with S3PE_QUIET=1. Errors are typed: not found, access denied, HTTP, transport, configuration, integrity.
Tested three ways
Each edition runs against a local fake S3 that recomputes and checks every signature, against the AWS documentation vector, and against a real store when S3PE_ENDPOINT and S3PE_TEST_BUCKET are set.
Decisions on record
Why path-style, why our own signer, why no default endpoint: DESIGN.md keeps the reasons.
Configuration
Argument, then environment, then default. Three variables are enough: the endpoint and the two halves of the key.
~/.Renviron, .env or the shell
S3PE_ENDPOINT=https://s3.example.org
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
S3PE_BUCKET=my-bucket # optional: bare keysNever put the key in a script or a repository. In R, use_endpoint() and use_bucket() set the two for the session; in Python, s3pe.configure().
| Setting | Environment | Default |
|---|---|---|
| endpoint | S3PE_ENDPOINT, AWS_ENDPOINT_URL | required |
| region | AWS_REGION, AWS_DEFAULT_REGION | us-east-1 |
| credentials | AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY | required |
| bucket | S3PE_BUCKET | none: use s3://bucket/key |
| quiet | S3PE_QUIET=1 | one line per transfer |