Fetch all data from an API in chunks — af_fetch

Iteratively calls af_fetch() with an advancing offset, stopping when a chunk comes back empty or total_limit is reached, then row-binds the chunks into one tibble. Columns listed in the API profile's drop_cols are removed.

Usage

af_fetch_all(
  api,
  name,
  total_limit = Inf,
  chunk_size = 50000L,
  query = list(),
  verbosity = 0L
)

Arguments

api: An apifetch_api object (see af_api()).
name: The token name to authenticate with (looked up via the API's service).
total_limit: Maximum number of records to retrieve in total. Default Inf (all available).
chunk_size: Records to request per chunk. Default 50000.
query: A named list of additional query-string filters. Default empty.
verbosity: 0 (silent, default), 1 (progress messages), or 2 (progress plus full HTTP request/response details).

Value

A tibble with all retrieved records.

Examples

if (FALSE) { # \dontrun{
api <- af_api("https://www.bigdata.pe.gov.br/api/buscar",
              service = "BigDataPE", auth = af_auth_raw(),
              pagination = af_paginate_offset("header"),
              drop_cols = "Mensagem")
af_fetch_all(api, "dengue", total_limit = 500, chunk_size = 100)
} # }