Package {fedstatAPIr}

fedstatAPIr: Unofficial API for Fedstat (Rosstat EMISS System) for Automatic and Efficient Data Queries

Description

An API for automatic data queries to the fedstat https://www.fedstat.ru, using a small set of functions with a common interface.

An API for automatic data queries to the fedstat https://www.fedstat.ru, using a small set of functions with a common interface.

Author(s)

Maintainer: Denis Krylov deniskrylovvit@gmail.com

Authors:

• Denis Krylov deniskrylovvit@gmail.com

• Dmitry Kibalnikov d.kibalnikov@gmail.com

See Also

Useful links:

• https://github.com/DenchPokepon/fedstatAPIr

• Report bugs at https://github.com/DenchPokepon/fedstatAPIr/issues

Free the disk space used by a finished download

Description

Deletes the saved parts of a complete fedstat_download(). Save the table from fedstat_collect() first: after cleanup the download cannot be collected or continued again.

Usage

fedstat_cleanup(x)

Arguments

Details

Incomplete, changed or actively used downloads are not cleaned. Cleanup can be repeated if it was interrupted.

Value

Invisibly, the download folder. Only the part files listed in the manifest are removed; the manifest, diagnostic log and any other files stay.

See Also

fedstat_download(), fedstat_collect()

Examples

## Not run: 
data <- fedstat_collect("cpi-download")
saveRDS(data, "cpi.rds")
fedstat_cleanup("cpi-download")

## End(Not run)

Get the table from a finished download

Description

Combines the saved parts of a complete fedstat_download() into one table. It works offline and can be repeated until you remove the parts with fedstat_cleanup().

Columns: the indicator value in ObsValue; one column per filter with its name, and a matching ⁠_code⁠ column with Fedstat's classification code; time columns such as Time. If Fedstat renamed a category during the download, every row uses the newest name. If Fedstat stopped offering some selected values, a message lists them.

Usage

fedstat_collect(x, remove_parts = FALSE, try_to_parse_ObsValue = TRUE)

Arguments

Value

A data.frame. The whole table is loaded into memory; for very large downloads read the individual part files listed in manifest.rds instead. Saved parts keep the original text values.

See Also

fedstat_download(), fedstat_cleanup()

Examples

## Not run: 
data <- fedstat_collect("cpi-download")
saveRDS(data, "cpi.rds")

## End(Not run)

Filters data_ids based on filters that are given in JSON format

Description

For advanced use. Most downloads should use fedstat_download(), which splits large selections, pauses between requests and saves progress.

Filters indicator data_ids with given filters taking into account possible filters specification errors and default filters.

filters should use filter_field_title in names and filter_value_title in values as they are presented on fedstat.ru. If for some reason the specified filters do not return the expected result, it is worth inspecting possible filter values in data_ids to see if the strings are defined correctly (e.g. encoding issues, mixing latin and cyrillic symbols)

filter_value_title currently supports the following special values:

1. asterix (*), it's alias for "select all possible filter values for this filter field"

All filter fields with more than one possible value must be explicitly specified in filters. Use "*" to select all values for a filter field. Filter fields with only one possible value are auto-selected.

Internally normalized filter_field_title and filter_value_title are used (all lowercase, removed extra whitespaces) to compare the equality of data_ids and filters

Usage

fedstat_data_ids_filter(data_ids, filters = list(), disable_warnings = FALSE)

Arguments

{
 "filter_field_title1": ["filter_value_title1", "filter_value_title2"],
 "filter_field_title2": ["filter_value_title1", "filter_value_title2"],
 ...
}

Value

data.frame, filtered data_ids

See Also

fedstat_get_data_ids, fedstat_post_data_ids_filtered

Examples

## Not run: 
# Get data filters identificators for CPI
# filter the data_ids to get data for january of 2023
# for all goods and services for Russian Federation
data_ids_filtered <- fedstat_get_data_ids("31074") %>%
    fedstat_data_ids_filter(
        filters = list(
            "Territory" = "Russian Federation",
            "Year" = "2023",
            "Period" = "January",
            "Types of goods and services" = "*"
        )
    )

# Not actual filter field titles and filter values titles because of ASCII requirement for CRAN

## End(Not run)

(Deprecated) Download indicator data from fedstat.ru in one request

Description

Deprecated: use fedstat_download() and fedstat_collect(). This function sends the whole selection to Fedstat as one request, with no pauses or saved progress. Large selections are slow, often fail and put heavy load on Fedstat. It is kept so that existing scripts keep working. Replace data <- fedstat_data_load_with_filters("31074", filters = filters) with job <- fedstat_download("31074", filters = filters) and data <- fedstat_collect(job).

Size is the number of filter combinations: the product of the number of selected values of every filter. Up to 50,000 combinations, a deprecation message is shown once per session. Above 50,000, every call warns. Above 100,000, the function stops unless allow_large_requests = TRUE, and above 250,000 it always stops: such requests load Fedstat for all its users and rarely succeed.

This function is a wrapper for the other functions of the package to provide a simple one function API for fedstat.ru

There are two basic terms in this API: filter_field and filter_value

The ⁠filter field⁠ reflects the individual property of the data point. For example, Year, Region, Unit of measurement, etc. Each filter field has its own title (filter_field_title), it is simply a human-readable word or phrase (e.g. "Year", "Region") that reflects the essence of the property by which filtering takes place

The ⁠filter value⁠ reflects the individual property specific value of the data point. (e.g. 2021 for the Year, "Russian Federation" for the region, etc.) It also has a title (filter_value_title) with the same purpose as filter_field_title

filters should use filter_field_title in names and filter_value_title in values as they are presented on fedstat.ru. If for some reason the specified filters do not return the expected result, it is worth using fedstat_get_data_ids separately and inspecting possible filter values in data_ids to see if the strings are defined correctly (e.g. encoding issues, mixing latin and cyrillic symbols)

filter_value_title currently supports the following special values:

1. asterix (*), alias for "select all possible filter values for this filter field"

All filter fields with more than one possible value must be explicitly specified in filters. Use "*" to select all values. Unspecified multi-value filters will produce an error.

Internally normalized filter_field_title and filter_value_title are used (all lowercase, removed extra whitespaces) to compare the equality of data_ids and filters

Usage

fedstat_data_load_with_filters(
  indicator_id,
  ...,
  filters = list(),
  timeout_seconds = 180,
  retry_max_times = 3,
  disable_warnings = FALSE,
  httr_verbose = NULL,
  loading_steps_verbose = TRUE,
  return_type = c("data", "dictionary"),
  try_to_parse_ObsValue = TRUE,
  allow_large_requests = FALSE
)

Arguments

{
 "filter_field_title1": ["filter_value_title1", "filter_value_title2"],
 "filter_field_title2": ["filter_value_title1", "filter_value_title2"],
 ...
}

Value

data.frame with filtered indicator data from fedstat.ru

See Also

fedstat_download, fedstat_get_data_ids, fedstat_data_ids_filter, fedstat_post_data_ids_filtered, fedstat_parse_sdmx_to_table

Examples

## Not run: 
# Download CPI data
# for all goods and services for Russian Federation
data <- fedstat_data_load_with_filters(
    indicator_id = "31074",
    filters = list(
        "Territory" = "Russian Federation",
        "Year" = "2023",
        "Period" = "January",
        "Types of goods and services" = "*"
    )
)
# Not actual filter field titles and filter values titles because of ASCII requirement for CRAN

## End(Not run)

Download the code lists (dictionary) of an indicator

Description

Returns the classifications Fedstat uses in the data of an indicator: for every coded column, all its codes and their names. The data from fedstat_collect() has a column with names (such as s_grtov) and a column with codes (s_grtov_code) for each classification; this table lists every code, including ones absent from your download.

Fedstat sends these lists only together with data, so the function downloads a small piece of data and keeps only its code lists. Any piece will do: every export carries the complete lists. It first tries the selection the website shows when the indicator page opens, and then small random selections if that one is empty.

Usage

fedstat_dictionary(
  indicator_id,
  filters = NULL,
  ...,
  attempts = 5L,
  timeout_seconds = 180,
  retry_max_times = 3L,
  request_pause = c(10, 20),
  verbose = TRUE
)

Arguments

Details

The code lists come from the SDMX export, not from the indicator page. Names usually match the filter values on the page, but codes are a different system from the IDs in fedstat_get_data_ids(). Year, period and unit of measurement are not coded and are not listed.

Each try is one small request (at most 5,000 filter combinations), with a pause between tries.

Value

A data.frame with one row per code: field_id (the column name in the data), field_title (the classification name), value_id (the code) and value_title (its name).

See Also

fedstat_show_filters() for the filters of the indicator page.

Examples

## Not run: 
dictionary <- fedstat_dictionary("31074")
# All codes of one classification
dictionary[dictionary$field_id == "s_grtov", c("value_id", "value_title")]

## End(Not run)

Download Fedstat data in small parts that survive interruptions

Description

Downloads one indicator for the filter values you choose and saves it piece by piece in cache_dir. Fedstat builds every combination of the selected values (years x months x territories x products ...), so a large selection is split into parts of at most max_nominal_cells_request combinations. Parts are requested one at a time, with pauses in between. This keeps the load on Fedstat light and makes long downloads finish.

If a run stops (time is up, Fedstat is unavailable, R was closed), run the same call again: saved parts are skipped. When the download is complete, fedstat_collect() returns the table.

Usage

fedstat_download(
  indicator_id,
  filters = list(),
  cache_dir = NULL,
  ...,
  max_nominal_cells_request = 50000,
  allow_large_requests = FALSE,
  preview_first = FALSE,
  total_time_limit_seconds = 3600,
  timeout_seconds = 180,
  retry_max_times = 2L,
  request_pause = c(30, 60),
  verbose = TRUE,
  log_file = file.path(cache_dir, "download.log"),
  token_policy = c("reuse", "refresh"),
  max_depth = 20L,
  max_failures = 6L,
  recovery_max_times = 3L,
  recovery_pause = 60,
  access_pause = 300
)

Arguments

Value

A fedstat_download object: the folder (path), whether the download is complete, a table of parts, and the values Fedstat stopped offering during the download (unavailable). The data stay on disk; use fedstat_collect() to load them. If the run stops, an error is raised and saved parts are kept for the next run.

Resuming

Run the same call with the same filters and cache_dir. Filters may be reordered or differ in case and spacing. A different selection needs a different folder. "*" means the values offered when the download started: values Fedstat adds later are not included. Each run has its own time allowance; saving a part that has already arrived may run slightly over it. Only one R session can use a folder at a time. If R crashed, the folder can stay marked as in use; the error message says how to clear it.

Request size

Size is counted in filter combinations, the product of the number of selected values of every filter, not in rows: many combinations can be empty. A part over the limit is split along its largest filter into the fewest groups that fit. Fedstat checks every combination, even empty ones, so large requests load it for all its users and often fail. Every function that requests data uses the same limits: a warning above 50,000 combinations, allow_large_requests = TRUE needed above 100,000, and no request above 250,000.

Reliability

Requests are sequential and paced. A failed request is retried after a pause. A part is halved only when its size is the likely cause: a request timeout, a gateway timeout (HTTP 504), an export marked as truncated, a refusal that persists after renewing the session, or a part whose export Fedstat drops or answers with a server error twice in a run. Fedstat often answers a request that is too heavy this way, and its indicator page may fail for a while afterwards. When Fedstat can't be reached, its page fails, or it limits the request rate, the download pauses to let it recover. After the second split caused by failures in a run, a warning suggests a smaller max_nominal_cells_request. Splits made after failures are undone on the next run if none of their parts was saved. Fedstat answers some empty selections with a redirect; a part counts as empty only when Fedstat's table preview confirms zero rows.

When Fedstat changes its filters

The selected filters are compared with the indicator page at the start of every run, when the session is renewed and before completion. Filter and value IDs identify what is requested. If Fedstat adds or removes a whole filter, the download stops: start again in a new folder. A selected value that Fedstat no longer offers is left out of unfinished parts, with a warning, and listed in unavailable. Renamed values continue with a warning, and fedstat_collect() uses the newest name for each code. A changed row/column layout on the website is ignored. If an export contains values that were not requested, the download stops. A change in meaning that keeps the same IDs and names cannot be detected.

Completion

A download is complete when every part has been received and checked, and the saved parts together contain no duplicate observations. This check needs some free disk space in cache_dir but never loads the whole table. A completed download opens again without network access. A long download is not a snapshot of one moment: Fedstat may publish revisions while it runs. Saved parts keep values exactly as Fedstat sent them.

Diagnostic log

download.log is a JSON Lines file with request stages, timings, counts and recovery decisions. It never contains HTTP headers, cookies, tokens or response bodies.

See Also

fedstat_collect(), fedstat_cleanup()

Examples

## Not run: 
job <- fedstat_download("31074", filters = "*", cache_dir = "cpi-download")
data <- fedstat_collect(job)

# Filter names as on the indicator page (English placeholders: CRAN needs ASCII)
filters <- list("Year" = "2024", "Period" = "January", "Territory" = "*")
# Without cache_dir, a folder is chosen for you and its path is shown
job <- fedstat_download("31074", filters)

## End(Not run)

Get data filters ids from fedstat.ru indicator web page

Description

For advanced use. Most downloads should use fedstat_download(), which splits large selections, pauses between requests and saves progress.

To query data from fedstat we need to POST some filters in form of filter numeric identificators. Most filters don't have some rule from which their ids can be generated based on filters titles and values. It seems like these ids are just indexes in the fedstat inner database. So in order to get the data, we first need to get the ids of the filter values by parsing specific part of java script source code on indicator web page.

Usage

fedstat_get_data_ids(
  indicator_id,
  ...,
  timeout_seconds = 180,
  retry_max_times = 3,
  httr_verbose = NULL
)

Arguments

Details

It is known that the fedstat lags quite often. Sometimes site never responds at all. This is especially true for the most popular indicators web pages. In this regard, by default, a GET request is sent 3 times with a timeout of 180 seconds and with initially small, but growing exponentially, pauses between requests.

As a rule, requests to the indicator web page take much longer than requests to get the data itself. A POST request for data is sent to https://www.fedstat.ru/indicator/downloadData.do?format={format} for all indicators and is often quite fast.

Token lifetime is controlled by Fedstat. Reuse may succeed, but tokens can be rejected. The wrapper fedstat_data_load_with_filters refreshes refused tokens while retaining the resolved filters.

Correct filter_field_object_ids are needed to get data. For the sdmx format, these ids do not change anything, except for the standard data sorting, but their incorrect specification will lead either to incomplete data loading or to no data at all. For the excel format, these ids determine the form of data presentation, as in the data preview on the fedstat site. For now only default filter_field_object_ids are used, which are parsed from java script source code on indicator web page. Users can specify filter_field_object_ids for each filter_field in resulting data_ids table.

The returned data.frame also carries session and CSRF token metadata as attributes (fedstat_handle, fedstat_base_url, fedstat_csrf_token, fedstat_csrf_token_name, fedstat_indicator_id). The fedstat_handle attribute holds an httr::handle object that preserves session cookies for the downstream POST request. These are used internally by fedstat_post_data_ids_filtered and must not be removed. A shared in-memory session attribute keeps refreshed tokens consistent across filtered copies. Metadata can be reused; session handles are not a disk cache.

The fedstat_default_selection attribute lists the values the website selects when the indicator page opens, as "<filter_field_id>_<filter_value_id>". fedstat_show_filters() prints the filters in a readable form.

Value

data.frame with all character type columns:

1. filter_field_id - id for filter field;

2. filter_field_title - filter field title string representation;

3. filter_value_id - id for filter field value;

4. filter_value_title - filter field value title string representation;

5. filter_field_object_ids - special strings that define the location of the filters fields. It can take the following values: lineObjectIds (filters in lines), columnObjectIds (filters in columns), filterObjectIds (hidden filters for all data);

See Also

fedstat_show_filters, fedstat_data_ids_filter, fedstat_post_data_ids_filtered

Examples

## Not run: 
# Get data filters identificators for CPI
data_ids <- fedstat_get_data_ids("31074")

## End(Not run)

Download indicator information

Description

Download indicator information from https://www.fedstat.ru/organizations/
Result table contains fedstat indicator id which is needed to request fedstat data
Indicator with condition hidden == TRUE shows disabled records in fedstat hence ones might not be requested

Usage

fedstat_indicator_info()

Value

data.frame

See Also

fedstat_get_data_ids

Examples

## Not run: 
# Get all indicator info
indicators <- fedstat_indicator_info()

## End(Not run)

Database of all indicator names presented on the fedstat.ru with hierarchical grouping

Description

Allows researchers to search for interesting indicators more easily. The database can be refreshed by the user via fedstat_indicator_info().

Usage

fedstat_indicators_names_database

Format

A data frame with the following variables:

department

the name of the department from which the data comes

group

thematic grouping of the indicator

id

indicator id (used in URL and API calls)

title

indicator name/title

hidden

boolean, TRUE if indicator is hidden on the website

Details

The version shipped with the package was last refreshed on 2026-09-04.

Source

https://fedstat.ru/organizations/

Parse sdmx raw bytes to data.frame

Description

For advanced use. Most downloads should use fedstat_download(), which splits large selections, pauses between requests and saves progress.

Parses sdmx raw bytes received in response to POST request. This function is a wrapper around readsdmx::read_sdmx, in addition to reading data, automatically adds columns with values from lookup tables. Can also return full data codes dictionary for the indicator

Usage

fedstat_parse_sdmx_to_table(
  data_raw,
  return_type = c("data", "dictionary"),
  try_to_parse_ObsValue = TRUE
)

Arguments

Value

data.frame

See Also

fedstat_parse_sdmx_to_table

Examples

## Not run: 
# Get data filters identificators for CPI
# filter the data_ids to get data for january of 2023
# for all goods and services for Russian Federation
# POST filters and download data in sdmx format
# Parse raw sdmx to data.frame
data <- fedstat_get_data_ids("31074") %>%
    fedstat_data_ids_filter(
        filters = list(
            "Territory" = "Russian Federation",
            "Year" = "2023",
            "Period" = "January",
            "Types of goods and services" = "*"
        )
    ) %>%
    fedstat_post_data_ids_filtered() %>%
    fedstat_parse_sdmx_to_table()

# Not actual filter field titles and filter values titles because of ASCII requirement for CRAN

## End(Not run)

Post data filters ids to fedstat.ru and download specified subset of data

Description

For advanced use. Most downloads should use fedstat_download(), which splits large selections, pauses between requests and saves progress.

Creates a request body from data_ids and sends it to https://www.fedstat.ru/indicator/downloadData.do?format={format}. Gets an sdmx or excel with data in binary format.

The request includes a CSRF token that is automatically extracted by fedstat_get_data_ids and passed through via data.frame attributes. Token reuse may succeed, but Fedstat can reject a token at any time. This primitive sends one POST; the convenience wrapper and batch downloader own retry and token-refresh decisions.

If no CSRF token is found in data_ids attributes (e.g. when using cached data_ids from a previous session), the function will attempt to fetch a fresh token automatically, using the same HTTP configuration.

sdmx raw bytes can be passed to fedstat_parse_sdmx_to_table to create a data.frame or to rawToChar and writeLines to create an xml file

excel raw bytes can be passed to writeBin to create an xls file

Request size is limited to protect Fedstat. Size is the number of filter combinations: the product of the number of selected values of every filter. Above 50,000 combinations this function warns, above 100,000 it stops unless allow_large_requests = TRUE, and above 250,000 it always stops. fedstat_download() splits larger selections into parts.

Usage

fedstat_post_data_ids_filtered(
  data_ids,
  ...,
  data_format = c("sdmx", "excel"),
  timeout_seconds = 180,
  retry_max_times = 3,
  httr_verbose = NULL,
  allow_large_requests = FALSE
)

Arguments

Value

raw bytes (sdmx or excel)

See Also

fedstat_parse_sdmx_to_table

Examples

## Not run: 
# Get data filters identificators for CPI
# filter the data_ids to get data for january of 2023
# for all goods and services for Russian Federation
# POST filters and download data in sdmx format
data <- fedstat_get_data_ids("31074") %>%
    fedstat_data_ids_filter(
        filters = list(
            "Territory" = "Russian Federation",
            "Year" = "2023",
            "Period" = "January",
            "Types of goods and services" = "*"
        )
    ) %>%
    fedstat_post_data_ids_filtered()

# Not actual filter field titles and filter values titles because of ASCII requirement for CRAN

## End(Not run)

Show the filters of an indicator, ready to copy into your code

Description

Prints every filter of an indicator with the number of its values and a few examples, followed by a filters <- list(...) you can copy into fedstat_download(). That list repeats the selection the website shows when the indicator page opens; edit it to choose what you need.

Usage

fedstat_show_filters(
  indicator_id,
  ...,
  timeout_seconds = 180,
  retry_max_times = 3L
)

Arguments

Value

Invisibly, the printed filters list. Filters with only one value are left out: they are selected automatically.

See Also

fedstat_dictionary() for the codes used in the data.

Examples

## Not run: 
filters <- fedstat_show_filters("31074")
job <- fedstat_download("31074", filters)

# All values of one filter
ids <- fedstat_get_data_ids("31074")
unique(ids[c("filter_field_title", "filter_value_title")])

## End(Not run)

Get data ids from java script source

Description

Get data ids from java script source

Usage

parse_js1(script)

Arguments

Value

json in form of list with data ids

Get default data ids object ids from java script source

Description

Get default data ids object ids from java script source

Usage

parse_js2(script)

Arguments

Value

json in form of list with 3 character vectors for lineObjectIds, columnObjectIds, filterObjectIds, which consist of filters_id

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

magrittr

%>%