Write chunks (and embeddings) to Parquet or Feather

Persist a chunk tibble from docling_chunk() — optionally carrying the embedding list-column added by docling_embed() — to a columnar file via the arrow package. List-columns (headings, pages, doc_items, embedding) are stored natively, so the file round-trips back to the same shape with arrow::read_parquet() / arrow::read_feather() and is directly queryable from 'Python', 'DuckDB', 'Spark' and other 'Arrow'-aware tools — a convenient hand-off format for RAG indexing pipelines.

Usage

docling_write_chunks(chunks, path, ...)

Arguments

chunks

A data frame, typically from docling_chunk() or docling_embed().

path

Output file path. The format is chosen by extension: .parquet writes 'Parquet'; .feather or .arrow write 'Feather' v2 (the 'Arrow' IPC format).

...

Additional arguments passed on to arrow::write_parquet() or arrow::write_feather() (for example compression).

Value

path, invisibly.

See also

docling_chunk(), docling_embed()

Examples

if (FALSE) { # \dontrun{
doc    <- docling_convert("paper.pdf")
chunks <- docling_chunk(doc, max_tokens = 512)
docling_write_chunks(chunks, "paper-chunks.parquet")

# Read back, or query from any Arrow-aware tool
arrow::read_parquet("paper-chunks.parquet")
} # }