Introduction to sidrar

Renato Prado Siqueira

Overview

sidrar is an R interface to SIDRA (Sistema IBGE de Recuperação Automática), the system through which the Brazilian Institute of Geography and Statistics (IBGE) publishes aggregate statistics.

The usual workflow is:

  1. find a table with search_sidra();
  2. inspect its available dimensions with info_sidra(); and
  3. optionally build and inspect a request with sidra_query() and sidra_plan();
  4. retrieve a selection with get_sidra(); or
  5. split and collect a large explicit request with sidra_split() and sidra_collect().

Network-dependent examples are not evaluated while the vignette is built.

Installation

Install the released version from CRAN:

install.packages("sidrar")

Install the development version from GitHub with pak:

# install.packages("pak")
pak::pak("rpradosiqueira/sidrar")

Find and inspect a table

search_sidra() searches titles in IBGE’s official aggregate catalog. Its result is a character vector whose names are the SIDRA table codes:

library(sidrar)

search_sidra("IPCA")
search_sidra(c("contas", "nacionais"))

The search is case- and accent-insensitive. When several terms are supplied, all terms must occur in the title, but they need not be adjacent.

Once you have a code, inspect the periods, variables, classifications, categories, and territorial levels accepted by the table:

metadata <- info_sidra(7060)
names(metadata)
metadata$variable
metadata$classific_category
metadata$geo

Set wb = TRUE to open the official table descriptor in the default browser. The function no longer prompts for confirmation:

info_sidra(7060, wb = TRUE)

For programmatic work, use the normalized discovery layer. It preserves codes as character strings and returns stable base R data frames:

catalog <- sidra_catalog()
metadata <- sidra_metadata(7060)
periods <- sidra_periods(7060)
locations <- sidra_locations(7060, "N1")

names(metadata)
metadata$variables
metadata$classifications
metadata$categories

The legacy search_sidra() and info_sidra() contracts remain unchanged. If SIDRA’s descriptor returns a recognized browser challenge, info_sidra() uses official aggregate metadata and periods. It discloses unavailable descriptor-specific names, geographic counts, and variable availability exceptions in attr(metadata, "sidrar_metadata") when metadata is its return value; it does not invent equivalent fields. wb = TRUE still opens the original descriptor. options(sidrar.fallback = FALSE) disables this route.

Build a structured request

This request retrieves the monthly IPCA for the general index in Campo Grande, Mato Grosso do Sul, over the 12 most recent periods:

ipca <- get_sidra(
  x = 7060,
  variable = 63,
  period = c(last = 12),
  geo = "City",
  geo.filter = list(City = 5002704),
  classific = "c315",
  category = list(7169)
)

geo.filter may also select every unit inside a higher territorial level. For example, the following pattern requests cities inside Mato Grosso do Sul:

get_sidra(
  x = 7060,
  variable = 63,
  period = "last",
  geo = "City",
  geo.filter = list(State = 50),
  classific = "c315",
  category = list(7169)
)

The existing defaults remain unchanged: descriptive headers are enabled, format = 4 requests codes and names, digits = "default" uses the table’s standard precision, and variable = "allxp" excludes automatically generated percentage variables.

Build the same request without downloading values and inspect the selections whose cardinality can be determined offline:

query <- sidra_query(
  x = 7060,
  variable = 63,
  period = sprintf("2024%02d", 1:12),
  geo = "City",
  geo.filter = list(City = 5002704),
  classific = "c315",
  category = list(7169)
)

query$url
sidra_plan(query)

No service limit is assumed by the planner. If a limit is known for the current request, pass it explicitly with sidra_plan(query, limit = ...).

Split and collect batches

When one dimension contains many explicit members, split it into disjoint batches and collect them sequentially:

batches <- sidra_split(query, by = "period", size = 6)
data <- sidra_collect(batches, provenance = TRUE)
sidra_provenance(data)

sidra_collect() requires identical names and column types across batches. It does not sort or deduplicate rows. Category members containing a space are SIDRA sums and are kept indivisible in structured queries. URLs can now be split directly; period selectors all, first, last, and ranges use the official inventory without inventing calendar periods. Other special selectors require explicit codes. A geographic filter can be split only for a query with one non-Brazil territorial level; multiple levels would repeat the unchanged levels in every batch.

Version 0.6.0 adds opt-in period batching and resumable local checkpoints:

url <- "/t/6468/n1/all/n2/all/n3/all/v/4099/p/all/h/n"
batches <- sidra_split(url, "period", size = 8)
batches$resolution$selection

data <- sidra_collect(
  url, batch_size = 8, value_type = "both",
  checkpoint = "sidrar-pnad", provenance = TRUE
)
# Repeat the same call after an interruption to reuse completed batches.
sidra_provenance(data)$batch_accessed_at
sidra_provenance(data)$resumed

batch_size limits periods, not cells: one period can still exceed the API limit and require splitting another explicit dimension. get_sidra() remains unchanged and does not split or store values. Checkpoints freeze the period inventory, including relative selectors when batch_size is omitted, and validate settings, package version, checksums, and schemas before reuse. They are trusted local files, separate from the metadata cache. An existing checkpoint is never silently cleared; use a new directory for a fresh extract. Stored and new batches may span source revisions, so inspect access times. The result is still combined in memory, not queried from disk. A stale .sidrar-lock after a hard crash must only be removed after confirming there is no running collector. URL geographic splitting requires direct codes; use a structured query for containment filters.

For comparisons between services or collections, align by territorial-level, location, period, variable, and classification codes, not row positions. Names can change and two territorial levels can reuse a location code.

Territorial views (G) and extinct territorial units (/u/y) are available through additive arguments:

sidra_query(1612, geo_view = 44, classific = character())
sidra_query(
  1612,
  geo = "State",
  geo.filter = list(c(20, 34)),
  include_extinct = TRUE,
  classific = character()
)

Use an API path or full URL

If a query was assembled elsewhere, pass either its path:

get_sidra(
  api = "/t/7060/n1/all/v/63/p/last/c315/7169"
)

or the complete official HTTPS URL:

get_sidra(
  api = paste0(
    "https://apisidra.ibge.gov.br/values/",
    "t/7060/n1/all/v/63/p/last/c315/7169"
  )
)

Full URLs are restricted to the official https://apisidra.ibge.gov.br/values endpoint. Percent-encoded segments such as %20 are preserved. If the path contains /h/n, the first observation is kept as data instead of being interpreted as a header.

Preserve SIDRA’s special values

SIDRA uses symbols with specific meanings, including "-" for an absolute zero, "X" for an inhibited value, ".." when a value does not apply, and "..." when it is unavailable. Earlier versions returned a numeric Valor column, so special symbols became NA. That remains the default for compatibility.

Use value_type = "character" to keep the symbols directly:

raw <- get_sidra(
  api = "/t/1849/n3/all/v/811/p/2018/c12762/all",
  value_type = "character"
)

Use value_type = "both" to keep numeric Valor and append Valor_raw:

both <- get_sidra(
  api = "/t/1849/n3/all/v/811/p/2018/c12762/all",
  value_type = "both"
)

Network behavior

Requests use HTTPS, UTF-8 decoding, an identifying user agent, a timeout, and limited retries for transient failures. Customize the timeout and retry count with:

options(
  sidrar.timeout = 120,
  sidrar.retries = 4
)

Catalog and metadata caching is explicit and disabled by default. Value responses are not cached automatically; explicit collection checkpoints are separate from the metadata cache:

metadata <- sidra_metadata(7060, cache = TRUE)
sidra_cache_info()
sidra_cache_clear()

Use refresh = TRUE to bypass and replace a cached discovery entry.

Cloudflare browser challenges are identified as sidrar_challenge_error. For compatible values queries, get_sidra() and sidra_collect() can use IBGE’s official aggregate API v3 as an alternative endpoint. This fallback supports multiple geographic levels, explicit periods and ranges, complete or first/latest period selections, standard variable/category selections, and the default descriptor format. Dimension columns follow the original URL; observation order remains that returned by the alternative service. It preserves the requested headers and special-value handling, and announces when the alternative is used. Set options(sidrar.fallback = FALSE) to disable it. Unsupported selections retain the original challenge error with an explanation in fallback_reason.

Explicit decimal precision is accepted only when numeric values already have the requested decimal places. Otherwise sidrar_fallback_precision_error (also a sidrar_parse_error) prevents silent re-rounding or invented digits. Default precision preserves the received values; maximum precision is not supported by this alternative. Automatic classification discovery can use official aggregate metadata when the SIDRA descriptor returns a challenge.

Every fallback response is checked for complete dimension fields, textual identifiers, duplicate observation keys, and codes outside explicit filters. Missing explicitly requested members generate sidrar_incomplete_warning, not fabricated rows or zeros: legitimate sparse tables need not form a full Cartesian product. Full coverage of all and contextual geographic membership still require comparison with current metadata.

For HTTP 429 or 503, the client honors a valid Retry-After header, including HTTP dates. The default maximum accepted server delay is 60 seconds; set options(sidrar.retry_after_max = 120) to allow up to two minutes. The value must be one finite positive number of seconds; invalid settings use 60, and options(sidrar.retry_after_max = NULL) restores that default. If another attempt would exceed this limit, sidrar_retry_after_error carries the delay in retry_after and the configured limit in retry_after_max; it does not trigger an early retry or change the per-attempt timeout.

pnad <- get_sidra(
  api = "/t/6468/n1/all/n2/all/n3/all/v/4099/p/all/d/v4099%201",
  value_type = "both"
)

Collection provenance records actual source endpoints in urls and the original queries in requested_urls when fallback occurs. If both official endpoints fail, report the URL, time, and the error’s cf_ray to IBGE; increasing retries cannot solve an interactive browser challenge.

Invalid parameters and API limits are reported with the response returned by SIDRA. See the official API help for the complete query syntax and current service limits.