DATASUS .dbc files, read once, used anywhere.
One reader for the compressed dBase files that DATASUS publishes, and three ways to use it: a Rust crate with no dependencies, a command line that writes Parquet and CSV, and a Python package that returns Arrow tables.
A small reader that does one thing
DATASUS distributes hospital admissions, outpatient production, births, deaths and facility records as .dbc: a dBase III file whose records are compressed with the PKWare Data Compression Library. This project reads them, and nothing else.
No dependencies in the core
datasus-core uses only the standard library: decompression, container detection and the dBase parser. Small to build, little to audit, easy to embed.
One core, three front ends
The command line, the Python package and your own Rust code call the same parser, so a file reads the same way everywhere.
Arrow out of the box
Typed record batches go straight to Parquet, pyarrow, polars or DuckDB. Python gets a stream through the Arrow PyCapsule interface, without copies.
Read only what you need
Select columns, cap the number of rows, set column types, parse dates. Unselected fields are skipped, not read and thrown away.
Safe on bad input
Malformed headers are rejected. Decompression stops at the size the header declares and never goes past 4 GiB, so a crafted file cannot expand without limit.
Same answer on every platform
Whitespace is the six ASCII characters and Latin-1 is strict ISO-8859-1, whatever the locale or operating system.
Install & use
Pick the front end that fits. All of them take the same options: columns to keep, maximum rows, trimming, encoding, type guessing, explicit column types and date parsing.
Command line
Convert to Parquet or CSV, inspect, decompress.
$ brew install strategicprojects/tap/datasus
# or
$ cargo install datasus-cli
# many files, one Parquet each (zstd)
$ datasus convert *.dbc --out-dir parquet/
# two columns, first 1000 rows, as CSV
$ datasus convert RDPE2401.dbc -o rd.csv \
--select uf_zi,val_tot --n-max 1000
# back to a plain .dbf
$ datasus decompress RDPE2401.dbc
Outputs are written to a .part file and renamed on success; an existing output needs --force.
Python
datasuspy: wheels for Linux, macOS and Windows, Python 3.10 or later.
$ pip install datasuspy
import datasuspy
# a pyarrow.Table
table = datasuspy.read_dbc(
"RDPE2401.dbc",
select=["uf_zi", "val_tot"],
)
df = table.to_pandas()
import polars as pl
df = pl.from_arrow(
datasuspy.read_dbc("RDPE2401.dbc"))
# in batches, for any Arrow consumer
reader = datasuspy.open_dbc(
"RDPE2401.dbc", batch_size=100_000)
Options and behaviour on PyPI.
Rust
datasus-core for the reader, datasus-arrow for record batches. Rust 1.80 or later.
$ cargo add datasus-core
$ cargo add datasus-arrow
use datasus_core::{open, ReadOptions};
let opts = ReadOptions {
parse_dates: true,
..ReadOptions::default()
};
let table = open("RDPE2401.dbc", &opts)?;
println!("{} rows x {} columns",
table.n_rows(), table.n_cols());
// Vec<ColumnData>, one per column
let columns = table.collect();
Reference: datasus-core · datasus-arrow, also on docs.rs.
Using R? The datasusr package reads the same files into tibbles and also has what this project leaves out on purpose: the catalog of DATASUS sources, FTP listing, downloads and a cache.
How it works
The core never decides where the values live. It parses a record and hands each value to a Sink; the front end owns the columns.
Detect
DBC, whole-file DBC, plain DBF or a truncated DBF: the layout is recognised from the bytes.
Decompress
A Rust translation of Mark Adler's blast.c, with a bound on the output.
Type the columns
Numeric fields without decimals are integers unless a value does not fit 32 bits.
Fill the sink
In batches of any size, so a large file never has to sit in memory twice.
Records flagged as deleted are dropped and n_max counts live records. Character columns are never null: a blank field is an empty string. Date fields are text unless parse_dates is set, and an invalid date is null. A header that declares more records than the file holds is a warning, not an error.
Speed
Time to read a whole file into memory, every column, on one macOS arm64 laptop. Lower is better.
Seconds to read a file
Best of three runs, except pyreaddbc + dbfread (one run, second file only).
The comparison with R is not like for like: R pays for creating its own strings, Arrow does not. With the same destination, an experimental build of datasusr on this core read 14 real files 1.5 to 1.9 times faster than its C reader. The command line, which also compresses and writes the Parquet file, took 1.68 s and 0.52 s on the two files above (one run each).
Checked against the C reader
The core is a port of the C reader of the datasusr R package (version 0.1.1, on CRAN). A script in the repository reads the same file with both and compares every value: 140 of 140 comparisons are identical, on synthetic files and on 11 real ones (hospital, outpatient, facilities, territorial tables), across column selection, row limits, column types, dates, trimming and encoding.
| Where the two differ, by design | C reader in R | datasus-rs |
|---|---|---|
| Whitespace trimmed from a field | isspace(): depends on the locale; on macOS under UTF-8 it also removes 0xA0 | The six ASCII whitespace characters, everywhere |
| Latin-1 text | R on macOS and Windows converts it as Windows-1252 | Strict ISO-8859-1 |
Invalid sequences with encoding = "utf-8" | Kept as they are | Replaced by U+FFFD |
Numeric field holding 0x1A or nan(...) (not seen in DATASUS files) | Read as a number, because strtod() accepts them | Null: only decimal numbers are numbers |
CI on three systems
clippy with -D warnings and the tests on Linux, macOS and Windows; the Python package is built and tested on every push.
The R tests, ported
The reader tests of datasusr run here too, on synthetic dBase files built in the test, plus cases for truncated input, padding and output limits.
MIT
The decompressor is a translation of Mark Adler's blast.c, distributed under the zlib license. Cite the project by its DOI.