s3pe: A Small, Dependable Client for S3 Object Stores

Read and write data on any S3-compatible object store ('AWS S3', 'MinIO', 'Garage', 'Ceph', ...). read_s3() and write_s3() move data frames, lists and files, with the format chosen by the key's extension ('CSV', 'JSON', 'Parquet', 'RDS', 'GeoJSON'); list, inspect and delete objects with the same small vocabulary. Requests are signed with AWS Signature Version 4 over the real hash of the payload, transient failures are retried with exponential backoff, and every transfer is checked against the server's ETag and content length. Endpoint and credentials come from the environment and are never written to disk.

See also

Useful links:

• https://github.com/StrategicProjects/s3pe

• https://strategicprojects.github.io/s3pe/r/

• Report bugs at https://github.com/StrategicProjects/s3pe/issues

Author

Maintainer: André Leite leite@castlab.org (ORCID)

Authors:

• André Leite leite@castlab.org (ORCID)

• Marcos Wasiliew marcos.wasiliew@gmail.com (ORCID)

• Hugo Vasconcelos hugo.vasconcelos@ufpe.br (ORCID)

• Carlos Amorim carlos.agaf@ufpe.br (ORCID)

• Diogo Bezerra diogo.bezerra@ufpe.br (ORCID)

• Júlia Nascimento Barreto juliabarreto@gd.seplag.pe.gov.br (ORCID)