Objects in and out of any S3 store, in Go.
Nine verbs over net/http and crypto/hmac, context on every call, errors you can errors.Is. No cgo, no require in go.mod.
Built to be trusted with the data
A client that moves the files a pipeline depends on has one job: either the bytes arrived, or you hear about it. Everything here serves that.
Signed the way every store accepts
AWS Signature V4 over the real SHA-256 of the body. No unsigned payloads, no aws-chunked trailers (which some stores reject), path-style URLs.
Retries that know when to stop
Connection failures and HTTP 429/500/502/503/504: three tries, 1 s then 2 s, signed again with a fresh date. Never on other 4xx.
Every byte accounted for
ETag against the local MD5 after a PUT; Content-Length and MD5 after a GET. Downloads land in a .s3pe-part file and are renamed when complete.
Errors for errors.Is
ErrNotFound, ErrAccessDenied, ErrHTTP, ErrTransport, ErrConfig, ErrIntegrity, with the status, code and key in a *s3pe.Error for errors.As.
Standard library only
net/http, crypto/hmac, crypto/sha256, encoding/xml. No cgo, nothing to vet but the package itself. Go 1.22 or later.
Context everywhere
Every call takes a context.Context; cancel it and the transfer stops. A per-call timeout applies when the context has no deadline.
The nine verbs, and four more
Package-level verbs over a client built from the environment on first use, and the same verbs as methods of *Client. Bytes in, bytes out: the module does not pick formats, the Python and R editions do. Files above 64 MiB go up as a multipart upload (parts in parallel, aborted cleanly on failure); metadata and headers ride along on writes. No ACLs and no versioning, by design.
| Verb | Does |
|---|---|
ReadS3 | GET into a []byte |
WriteS3 | PUT bytes, content type guessed from the key |
DownloadS3 | GET to a file, streaming |
UploadS3 | PUT from a file, streaming, two passes |
ListS3 | every Object under a prefix, all pages |
ExistsS3 | bool |
HeadS3 | *Object or nil: size, type, date, ETag, metadata |
DeleteS3 | idempotent; DeleteManyS3 for batches of 1000 |
BucketsS3 | []Bucket |
CopyS3 | server-side copy |
ListDirS3 | one level: objects and prefixes |
PresignS3 | a GET or PUT URL without credentials, up to 7 days |
SyncUpS3 / SyncDownS3 | a directory and a prefix mirrored by ETag, a manifest last |
A key is s3://bucket/key, or a bare key when the bucket is configured (S3PE_BUCKET or the client). Endpoint: S3PE_ENDPOINT (https://host[:port], required); credentials: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
What happens in every call
The same six steps in the four editions.
Resolve and hash
s3://bucket/key or bucket plus key; SHA-256 for the signature, MD5 for the check.
Sign and send
SigV4 over host, date and payload hash; path-style, Content-Length, no chunked trailers.
Retry
Connection failures and 429/5xx, three tries, 1 s then 2 s. Other 4xx come straight back.
Verify and report
ETag and Content-Length checked; one line on stderr; typed errors otherwise.
Install & run
Go 1.22 or later. pkg.go.dev indexes each tag; changes between versions in the changelog.
$ go get github.com/StrategicProjects/s3pe// an explicit client
s3, err := s3pe.New(s3pe.Config{
Endpoint: "http://127.0.0.1:9000",
AccessKey: key, SecretKey: secret,
Bucket: "my-bucket",
})
data, err := s3.Read(ctx, "data/manifest.json")
_, err = s3.Download(ctx, "data/regions.geojson", "pe.geojson")
meta, err := s3.Head(ctx, "etl/sales.parquet")
if meta != nil {
fmt.Println(meta.Size, meta.LastModified)
}// errors you can test
_, err = s3.Read(ctx, "missing")
if errors.Is(err, s3pe.ErrNotFound) {
// 404: nothing there
}
var e *s3pe.Error
if errors.As(err, &e) {
log.Println(e.Status, e.Code, e.Key)
}
// quiet, or your own logger
s3, _ = s3pe.New(s3pe.Config{Quiet: true})
s3, _ = s3pe.New(s3pe.Config{Logger: log.Default()})Architecture
The Go module lives at the root of the family's repository, so its import path is the repository URL. It reimplements the shared specification on the standard library; the Rust crate is the reference, and the four signers reproduce the AWS documentation vector. One repository, one version, one DESIGN.md.
