Objects in and out of any S3 store, from Rust and the shell.
The core of the s3pe family: nine verbs, a builder, typed errors, and the s3pe command for scripts and servers. ureq and rustls, no system OpenSSL.
Built to be trusted with the data
A client that moves the files a pipeline depends on has one job: either the bytes arrived, or you hear about it. Everything here serves that.
Signed the way every store accepts
AWS Signature V4 over the real SHA-256 of the body. No unsigned payloads, no aws-chunked trailers (which some stores reject), path-style URLs.
Retries that know when to stop
Connection failures and HTTP 429/500/502/503/504: three tries, 1 s then 2 s, signed again with a fresh date. Never on other 4xx.
Every byte accounted for
ETag against the local MD5 after a PUT; Content-Length and MD5 after a GET. Downloads land in a .s3pe-part file and are renamed when complete.
Typed errors
Error::NotFound, AccessDenied, Http { status, code }, Transport, Config, Integrity. Match on them, do not parse messages.
Small and portable
ureq with rustls and bundled roots: no OpenSSL to find, builds on Linux, macOS and Windows, Rust 1.85 or later. #![forbid(unsafe_code)].
A command line too
cargo install s3pe gives s3pe ls | cat | get | put | rm | head | buckets, with exit codes for scripts and credentials read only from the environment.
The nine verbs, and four more
Free functions over a client built from the environment (endpoint and credentials), and the same verbs as methods of Client. Bytes in, bytes out: the crate does not pick formats, the Python and R editions do. Files above 64 MiB go up as a multipart upload (parts in parallel, aborted cleanly on failure); metadata and headers ride along on writes. No ACLs and no versioning, by design.
| Verb | Does |
|---|---|
read_s3 | GET into a Vec<u8> |
write_s3 | PUT bytes, content type guessed from the key |
download_s3 | GET to a file, streaming |
upload_s3 | PUT from a file, streaming, two passes |
list_s3 | every Object under a prefix, all pages |
exists_s3 | bool |
head_s3 | Option<Object>: size, type, date, ETag, metadata |
delete_s3 | idempotent; delete_many_s3 for batches of 1000 |
buckets_s3 | Vec<Bucket> |
copy_s3 | server-side copy |
list_dir_s3 | one level: objects and prefixes |
presign_s3 | a GET or PUT URL without credentials, up to 7 days |
sync_up_s3 / sync_down_s3 | a directory and a prefix mirrored by ETag, a manifest last |
A key is s3://bucket/key, or a bare key when the bucket is configured (S3PE_BUCKET or the client). Endpoint: S3PE_ENDPOINT (https://host[:port], required); credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
What happens in every call
The same six steps in the four editions.
Resolve and hash
s3://bucket/key or bucket plus key; SHA-256 for the signature, MD5 for the check.
Sign and send
SigV4 over host, date and payload hash; path-style, Content-Length, no chunked trailers.
Retry
Connection failures and 429/5xx, three tries, 1 s then 2 s. Other 4xx come straight back.
Verify and report
ETag and Content-Length checked; one line on stderr; typed errors otherwise.
Install & run
Rust 1.85 or later. Changes between versions in the changelog.
$ cargo add s3pe # library (+ CLI target)
$ cargo add s3pe --no-default-features # library only, no clap
$ cargo install s3pe # the s3pe command// an explicit client
let s3 = s3pe::Client::builder()
.endpoint("http://127.0.0.1:9000")
.credentials("AKIA…", "…")
.bucket("my-bucket")
.build()?;
let bytes = s3.read("data/manifest.json")?;
s3.download("data/regions.geojson", "pe.geojson")?;
if let Some(m) = s3.head("etl/sales.parquet")? {
println!("{} bytes", m.size);
}
match s3.read("missing") {
Err(s3pe::Error::NotFound { .. }) => {}
other => { other?; }
}# the command line
$ s3pe buckets
$ s3pe ls -l s3://my-bucket/data/
$ s3pe cat s3://my-bucket/data/manifest.json | jq .ultimo_mes
$ s3pe get s3://my-bucket/data/pe_municipios.geojson
$ s3pe put sales.parquet s3://my-bucket/etl/
↑ sales.parquet → s3://my-bucket/etl/sales.parquet (12.3 MB, 0.8 s)
$ s3pe rm s3://my-bucket/etl/old.parquet
$ s3pe --endpoint http://127.0.0.1:9000 -b my-bucket ls data/
# exit codes: 0 ok · 2 not found · 3 access denied · 4 configArchitecture
The crate is the core of the family: the Python package is a PyO3 binding on it, while R and Go reimplement the same specification in their own idiom. One repository, one version, one DESIGN.md.
