Skip to contents

tantivyr brings fast, local full-text search to R. It wraps the Tantivy search engine — a Rust library inspired by Apache Lucene — to let you index a data frame or a collection of documents and search it with BM25 ranking, structured filters, snippet highlighting and incremental updates, all on your own machine.

The search engine is the work of Paul Masurel and the Tantivy contributors (MIT license); tantivyr is an independent R binding and is not affiliated with, sponsored by or endorsed by the Tantivy project or by Quickwit, Inc. See License and credits.

It is built for text in Portuguese and English (stemming and stop words included), making it a good fit for public documents, news clippings, extracted PDF text, transcripts and legal acts.

Installation

Install the released version from CRAN:

install.packages("tantivyr")

Or the development version from GitHub:

# install.packages("pak")
pak::pak("StrategicProjects/tantivyr")

Building from source needs a Rust toolchain (cargo and rustc 1.88 or later). CRAN binaries for Windows and macOS do not.

Quick start

The fastest way in is tnt_index_df(): point it at a data frame, say which columns are text and which are filters, and search.

library(tantivyr)

news <- data.frame(
  id    = 1:4,
  title = c(
    "Orçamento público aprovado pelo congresso",
    "Reforma tributária avança no senado",
    "Nova lei de licitações entra em vigor",
    "Congresso debate orçamentos municipais"
  ),
  year  = c(2022L, 2023L, 2024L, 2024L)
)

idx <- tnt_index_df(
  news,
  text     = title,
  filters  = year,
  stemmer  = "portuguese",
  stopwords = TRUE
)

# BM25 search — note the Portuguese stemmer matches "orçamentos" too
tnt_search(idx, "orçamento")
#> # A tibble: 2 × 4
#>   score    id title                                      year
#>   <dbl> <dbl> <chr>                                     <dbl>
#> 1 0.710     1 Orçamento público aprovado pelo congresso  2022
#> 2 0.710     4 Congresso debate orçamentos municipais     2024

Filters, ordering and highlighting

Filters can be written as plain comparisons (year >= 2024) or as Tantivy query strings. Snippets come back as <field>_snippet columns.

# structured filter
tnt_search(idx, "", filter = year >= 2024)
#> # A tibble: 2 × 4
#>   score    id title                                   year
#>   <dbl> <dbl> <chr>                                  <dbl>
#> 1     1     4 Congresso debate orçamentos municipais  2024
#> 2     1     3 Nova lei de licitações entra em vigor   2024

# highlighted snippets
tnt_search(idx, "congresso", highlight = title)$title_snippet
#> [1] "Orçamento público aprovado pelo <b>congresso</b>"
#> [2] "<b>Congresso</b> debate orçamentos municipais"

# order by a fast field instead of relevance
tnt_search(idx, "", order_by = year, desc = TRUE)[, c("title", "year")]
#> # A tibble: 4 × 2
#>   title                                      year
#>   <chr>                                     <dbl>
#> 1 Congresso debate orçamentos municipais     2024
#> 2 Nova lei de licitações entra em vigor      2024
#> 3 Reforma tributária avança no senado        2023
#> 4 Orçamento público aprovado pelo congresso  2022

Explicit schema and incremental updates

For full control, declare a schema and manage the index yourself. Indexes can live on disk (and be reopened later) or in memory.

sch <- tnt_schema(
  id    = tnt_i64(),
  title = tnt_text(stemmer = "portuguese", stored = TRUE),
  body  = tnt_text(stemmer = "portuguese")
)

idx <- tnt_index(path = tempfile(), schema = sch)

idx |>
  tnt_add(data.frame(id = 1L, title = "Edital de licitação", body = "...")) |>
  tnt_commit()

# update (replace by key) and delete, then commit
idx |>
  tnt_update(data.frame(id = 1L, title = "Edital retificado", body = "..."), by = id) |>
  tnt_commit()

tnt_count(idx, "edital")
#> [1] 1

How it compares

tantivyr is for retrieval: ranked, filtered, highlighted search over text you control, with an index that updates incrementally and persists to disk. It is not a database and not an embeddings/semantic-search tool — it is the local, dependency-light BM25 engine that R has been missing.

License and credits

The R package is MIT licensed, © tantivyr authors (see LICENSE).

tantivyr bundles and statically links third-party code, which remains the copyright of its authors:

  • Tantivy, the search engine, is copyright (c) 2018 by the Tantivy project authors (Paul Masurel and the others listed in Tantivy’s AUTHORS file) and MIT licensed.
  • Tantivy’s dependencies and the extendr bridge are vendored as Rust crates in src/rust/vendor.tar.xz. Most are used under the MIT license; a few are available only under Apache-2.0, BSD-3-Clause, Zlib or Unicode-3.0 (among them the Snowball stemmers and the Zstandard library).

inst/AUTHORS lists every bundled crate with its version, license and copyright holders, and inst/COPYRIGHTS reproduces Tantivy’s notice and the notices and license texts of the crates that are not under MIT. In an installed package they are at system.file("AUTHORS", package = "tantivyr") and system.file("COPYRIGHTS", package = "tantivyr").

tantivyr is not affiliated with, sponsored by or endorsed by the Tantivy project or Quickwit, Inc.