A RAG pipeline

library(crawlee)

Beyond crawling, crawlee provides three helpers to turn collected text into a retrieval-ready corpus for retrieval-augmented generation (RAG): cr_chunk(), cr_embed() and cr_export(). They operate on plain tibbles, so they slot in right after cr_collect().

1. Crawl and collect text

pages <- crawler("https://books.toscrape.com/") |>
  cr_options(max_requests = 100) |>
  cr_on_html(function(ctx) {
    ctx$push_data(list(
      url   = ctx$request$url,
      title = ctx$page |> rvest::html_element("title") |> rvest::html_text2(),
      text  = ctx$page |> rvest::html_element("body") |> rvest::html_text2()
    ))
    ctx$enqueue_links(glob = "*/catalogue/*")
  }) |>
  cr_run() |>
  cr_collect()

2. Chunk

cr_chunk() splits text into overlapping windows. On a data frame, name the text column; every other column is carried along as per-chunk metadata (so each chunk keeps its url and title).

chunks <- cr_chunk(pages, text = text, size = 1000, overlap = 200, by = "char")
chunks
#> columns: doc_id, chunk_id, chunk, text, n_chars, url, title

Use by = "word" to size chunks in words instead of characters.

3. Embed

cr_embed() is provider-agnostic: crawlee never calls an embedding service itself. You pass embed_fn, a function that maps a character vector to a numeric matrix (one row per input) or a list of numeric vectors. It is applied in batches and adds an embedding list-column.

# A real embedder typically calls an HTTP API (any provider) with httr2:
embed_fn <- function(texts) {
  # return a length(texts) x d numeric matrix
  resp <- httr2::request("https://api.example.com/v1/embeddings") |>
    httr2::req_auth_bearer_token(Sys.getenv("EMBEDDINGS_API_KEY")) |>
    httr2::req_body_json(list(input = texts)) |>
    httr2::req_perform()
  do.call(rbind, lapply(httr2::resp_body_json(resp)$data, \(x) unlist(x$embedding)))
}

embedded <- cr_embed(chunks, embed_fn, batch_size = 32)

For a quick local experiment you can pass any function — even a trivial one:

fake_embed <- function(x) matrix(nchar(x), nrow = length(x), ncol = 1)
embedded <- cr_embed(chunks, fake_embed)

4. Export for retrieval

cr_export() writes the chunk table (with embeddings) to a retrieval-friendly format. parquet and jsonl preserve the embedding vectors natively; csv and duckdb serialise them to a [...] string.

cr_export(embedded, "corpus.parquet", format = "parquet")
cr_export(embedded, "corpus.jsonl", format = "jsonl")
cr_export(embedded, "corpus.duckdb", format = "duckdb", table = "chunks")

End to end

crawler("https://books.toscrape.com/") |>
  cr_options(max_requests = 100) |>
  cr_on_html(function(ctx) {
    ctx$push_data(list(
      url  = ctx$request$url,
      text = ctx$page |> rvest::html_element("body") |> rvest::html_text2()
    ))
    ctx$enqueue_links(glob = "*/catalogue/*")
  }) |>
  cr_run() |>
  cr_collect() |>
  cr_chunk(text = text, size = 1000, overlap = 200) |>
  cr_embed(embed_fn) |>
  cr_export("corpus.parquet", format = "parquet")

From here, load corpus.parquet into your vector store or do nearest-neighbour search in R to retrieve chunks for a prompt.