An S3-compatible object store (AWS S3, MinIO, Garage, Ceph, …) is a
good place to publish data for analysts: a bucket per project, a key
pair that reads it and sometimes writes it. s3pe is the
client: nine verbs that move R objects and files in and out, with every
request signed, retried on transient failures and verified against the
server’s checksum.
This vignette is not evaluated: a store needs an endpoint and credentials.
Configuration
Put the endpoint and the key pair in ~/.Renviron and
restart R. S3PE_BUCKET is the bucket used when you do not
say which one.
S3PE_ENDPOINT=https://example.org
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
S3PE_BUCKET=my-bucketFor one session, use_endpoint() and
use_bucket() do the same. s3_config() shows
what the package resolved (the secret is never printed). Every verb
takes a config argument, so you can also build one
explicitly, for another store or another key:
library(s3pe)
s3_config()
cfg <- s3_config(endpoint = "https://example.org", bucket = "archive",
access_key = Sys.getenv("ARCHIVE_KEY"),
secret_key = Sys.getenv("ARCHIVE_SECRET"))Keys are either relative to the bucket or full
s3://bucket/key URLs; the URL wins.
Reading
read_s3() chooses the reader by the key’s extension:
# JSON -> list
meta <- read_s3("data/manifest.json")
meta$generated_at
# gzipped CSV -> tibble (readr); `...` reaches the reader
sales <- read_s3("series/sales.csv.gz", col_types = "Dncl")
# GeoJSON -> sf, when sf is installed
regions <- read_s3("data/regions.geojson")
# Parquet -> data frame (nanoparquet)
panel <- read_s3("etl/panel.parquet")
# anything else -> raw bytes; or ask for them
png <- read_s3("report/figure1.png")
txt <- read_s3("data/manifest.json", format = "text")A good publisher uploads a manifest last. If
manifest.json says the batch is 2026-08, every file of that
batch is already there: read it first and use it as the version of what
you cache.
Writing
write_s3() is the reverse: the extension decides the
format and the content type.
write_s3(panel, "report/panel.parquet")
write_s3(panel, "report/panel.csv.gz")
write_s3(list(generated_at = Sys.time(), rows = nrow(panel)),
"report/manifest.json", pretty = TRUE)
write_s3(model, "report/model.rds")
write_s3(c("# Report", "", "Text..."), "report/README.md")Upload the data first, the manifest last: readers that follow the convention above will then never see a half-written batch.
Files
upload_s3() and download_s3() move files
without reading them into R. A key ending in / gets the
file’s name appended; downloads are written to a .s3pe-part
file and renamed when complete and verified.
upload_s3("output/figure1.png", "report/")
download_s3("data/regions.geojson", "data/regions.geojson", overwrite = TRUE)The rest of the bucket
list_s3("data/") # tibble: key, size, last_modified, etag
head_s3("data/manifest.json") # one row, or NULL
exists_s3("data/manifest.json") # TRUE/FALSE
delete_s3("report/draft.csv")
buckets_s3()Big files, sharing and sync
Bodies above 64 MiB are uploaded in parts, with every part’s ETag
checked and the whole upload aborted on failure;
s3_config(multipart_threshold =, part_size =) tune it.
metadata = stores user metadata with the object
(head_s3() returns it) and headers = sends any
extra header.
upload_s3("exports/full.parquet", "archive/", bucket = "my-bucket",
metadata = c(run = "2026-10-23"))A pre-signed URL lets someone without credentials read (or write) one object for a while:
presign_s3("reports/report.pdf", bucket = "my-bucket", expires = 3600)sync_up_s3() and sync_down_s3() mirror a
directory and a prefix, moving only what differs (size and ETag).
last uploads a manifest after everything else, so readers
who check the manifest first never see a half-published batch;
delete = TRUE removes extras on the destination.
sync_up_s3("output/", "reports/2026/", bucket = "my-bucket", last = "manifest.json")
sync_down_s3("reports/2026/", "reports-2026/", bucket = "my-bucket")
copy_s3("reports/2026/manifest.json", "reports/latest.json", bucket = "my-bucket")
delete_s3(list_s3("tmp/", bucket = "my-bucket")$key, bucket = "my-bucket")When things go wrong
Every failure is an error with a class, so a pipeline stops by default and you can catch exactly what you expect:
-
s3pe_error_config: no endpoint, no credentials, no bucket. -
s3pe_error_not_found(404),s3pe_error_access(403),s3pe_error_http(other statuses, with$statusand$code). -
s3pe_error_transport: the server could not be reached, after three attempts with a 1 s and a 2 s pause. -
s3pe_error_integrity: the bytes that arrived are not the bytes that were sent.
Transfers print one line each
(↑ s3://my-bucket/report/panel.parquet (12.3 MB, 0.8 s));
silence them with options(s3pe.quiet = TRUE) or
S3PE_QUIET=1.
A local store
Inside the machine that hosts the store, or against a MinIO or Garage started for development, point the session at it:
use_endpoint("http://127.0.0.1:9000")