◆ Rust · Brazilian public health data

DATASUS .dbc files, read once, used anywhere.

One reader for the compressed dBase files that DATASUS publishes, and three ways to use it: a Rust crate with no dependencies, a command line that writes Parquet and CSV, and a Python package that returns Arrow tables.

CI crates.io PyPI DOI 10.5281/zenodo.24329435 License MIT

A small reader that does one thing

DATASUS distributes hospital admissions, outpatient production, births, deaths and facility records as .dbc: a dBase III file whose records are compressed with the PKWare Data Compression Library. This project reads them, and nothing else.

🦀

No dependencies in the core

datasus-core uses only the standard library: decompression, container detection and the dBase parser. Small to build, little to audit, easy to embed.

🧩

One core, three front ends

The command line, the Python package and your own Rust code call the same parser, so a file reads the same way everywhere.

🏹

Arrow out of the box

Typed record batches go straight to Parquet, pyarrow, polars or DuckDB. Python gets a stream through the Arrow PyCapsule interface, without copies.

🎚️

Read only what you need

Select columns, cap the number of rows, set column types, parse dates. Unselected fields are skipped, not read and thrown away.

🛡️

Safe on bad input

Malformed headers are rejected. Decompression stops at the size the header declares and never goes past 4 GiB, so a crafted file cannot expand without limit.

🔁

Same answer on every platform

Whitespace is the six ASCII characters and Latin-1 is strict ISO-8859-1, whatever the locale or operating system.

Install & use

Pick the front end that fits. All of them take the same options: columns to keep, maximum rows, trimming, encoding, type guessing, explicit column types and date parsing.

Command line

Convert to Parquet or CSV, inspect, decompress.

$ brew install strategicprojects/tap/datasus
# or
$ cargo install datasus-cli
# many files, one Parquet each (zstd)
$ datasus convert *.dbc --out-dir parquet/

# two columns, first 1000 rows, as CSV
$ datasus convert RDPE2401.dbc -o rd.csv \
    --select uf_zi,val_tot --n-max 1000

# back to a plain .dbf
$ datasus decompress RDPE2401.dbc

Outputs are written to a .part file and renamed on success; an existing output needs --force.

Python

datasuspy: wheels for Linux, macOS and Windows, Python 3.10 or later.

$ pip install datasuspy
import datasuspy

# a pyarrow.Table
table = datasuspy.read_dbc(
    "RDPE2401.dbc",
    select=["uf_zi", "val_tot"],
)
df = table.to_pandas()

import polars as pl
df = pl.from_arrow(
    datasuspy.read_dbc("RDPE2401.dbc"))

# in batches, for any Arrow consumer
reader = datasuspy.open_dbc(
    "RDPE2401.dbc", batch_size=100_000)

Options and behaviour on PyPI.

Rust

datasus-core for the reader, datasus-arrow for record batches. Rust 1.80 or later.

$ cargo add datasus-core
$ cargo add datasus-arrow
use datasus_core::{open, ReadOptions};

let opts = ReadOptions {
    parse_dates: true,
    ..ReadOptions::default()
};
let table = open("RDPE2401.dbc", &opts)?;
println!("{} rows x {} columns",
    table.n_rows(), table.n_cols());

// Vec<ColumnData>, one per column
let columns = table.collect();

Reference: datasus-core · datasus-arrow, also on docs.rs.

Using R? The datasusr package reads the same files into tibbles and also has what this project leaves out on purpose: the catalog of DATASUS sources, FTP listing, downloads and a cache.

How it works

The core never decides where the values live. It parses a record and hands each value to a Sink; the front end owns the columns.

The DBC file is a dBase header followed by a compressed record area. The core detects the container, decompresses the records up to the size the header declares and parses them, writing each value into a sink supplied by the front end: Arrow record batches for the command line and Python, or buffers of your own.
01

Detect

DBC, whole-file DBC, plain DBF or a truncated DBF: the layout is recognised from the bytes.

02

Decompress

A Rust translation of Mark Adler's blast.c, with a bound on the output.

03

Type the columns

Numeric fields without decimals are integers unless a value does not fit 32 bits.

04

Fill the sink

In batches of any size, so a large file never has to sit in memory twice.

openfrom_bytesReadOptionsTableCursorSinkColumnDatadecode_with_limitblastdatasus_arrow::BatchReaderdatasus_arrow::schema

Records flagged as deleted are dropped and n_max counts live records. Character columns are never null: a blank field is an empty string. Date fields are text unless parse_dates is set, and an invalid date is null. A header that declares more records than the file holds is a warning, not an error.

Speed

Time to read a whole file into memory, every column, on one macOS arm64 laptop. Lower is better.

Seconds to read a file

Best of three runs, except pyreaddbc + dbfread (one run, second file only).

datasuspy → pyarrow datasusr 0.1.1 (C reader) → R pyreaddbc + dbfread → pandas

The comparison with R is not like for like: R pays for creating its own strings, Arrow does not. With the same destination, an experimental build of datasusr on this core read 14 real files 1.5 to 1.9 times faster than its C reader. The command line, which also compresses and writes the Parquet file, took 1.68 s and 0.52 s on the two files above (one run each).

Checked against the C reader

The core is a port of the C reader of the datasusr R package (version 0.1.1, on CRAN). A script in the repository reads the same file with both and compares every value: 140 of 140 comparisons are identical, on synthetic files and on 11 real ones (hospital, outpatient, facilities, territorial tables), across column selection, row limits, column types, dates, trimming and encoding.

Where the two differ, by designC reader in Rdatasus-rs
Whitespace trimmed from a fieldisspace(): depends on the locale; on macOS under UTF-8 it also removes 0xA0The six ASCII whitespace characters, everywhere
Latin-1 textR on macOS and Windows converts it as Windows-1252Strict ISO-8859-1
Invalid sequences with encoding = "utf-8"Kept as they areReplaced by U+FFFD
Numeric field holding 0x1A or nan(...) (not seen in DATASUS files)Read as a number, because strtod() accepts themNull: only decimal numbers are numbers
✅

CI on three systems

clippy with -D warnings and the tests on Linux, macOS and Windows; the Python package is built and tested on every push.

🧪

The R tests, ported

The reader tests of datasusr run here too, on synthetic dBase files built in the test, plus cases for truncated input, padding and output limits.

⚖️

MIT

The decompressor is a translation of Mark Adler's blast.c, distributed under the zlib license. Cite the project by its DOI.