---
title: "PX-WEB API Interface for R"
author: "Mans Magnusson, Leo Lahti et al."
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    toc: true
vignette: >
  %\VignetteIndexEntry{PX-WEB API Interface for R}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}  
---


```{r basecode, message=FALSE, eval=FALSE, echo=FALSE}
# Below are code to run to setup the data

# Get PXWEB levels
px_levels <- pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/")
px_levels
save(px_levels, file = "vignettes/px_levels_example.rda")

# Get PXWEB metadata about a table
scb_table_url <- paste0(
  "https", "://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy"
)
px_meta <- pxweb_get(scb_table_url)
px_meta
save(px_meta, file = "vignettes/px_meta_example.rda")


# Example Download
pxweb_query_list <-
  list(
    "Civilstand" = c("*"), # Use "*" to select all
    "Kon" = c("1", "2"),
    "ContentsCode" = c("BE0101N1"),
    "Tid" = c("2015", "2016", "2017")
  )
pxq <- pxweb_query(pxweb_query_list)

pxd <- pxweb_get(
  scb_table_url,
  pxq
)
save(pxd, file = "vignettes/pxd_example.rda")

pxq$response$format <- "json-stat"
pxjstat <- pxweb_get(
  scb_table_url,
  pxq
)
save(pxjstat, file = "vignettes/pxjstat_example.rda")

pxq$response$format <- "px"
pxfp <- pxweb_get(
  scb_table_url,
  pxq
)
save(pxfp, file = "vignettes/pxfp_example.rda")

pxweb_cite_example <- capture.output(pxweb_cite(pxd))

```


This R package provides tools to access [PX-WEB
API](https://www.scb.se/en/services/open-data-api/api-for-the-statistical-database/). Your
[contributions](https://ropengov.org/community/) and [bug reports and other feedback](https://github.com/ropengov/pxweb) are
welcome!


We can find more information on the PX-Web/PC-Axis API [here](https://www.scb.se/en/services/statistical-programs-for-px-files/px-web/).

## Introduction

PXWEB is an API structure developed by Statistics Sweden and other national statistical institutions (NSI) to disseminate public statistics in a structured way. This API enables downloading and using data from statistical agencies without using a web browser direct over HTTP/HTTPS.

The `pxweb` R package connects any PXWEB API to R and facilitates the access, use and referencing of data from PXWEB APIs.

### Available data sources and tools

[A number of organizations](https://www.scb.se/en/services/statistical-programs-for-px-files/px-web/pxweb-examples/) use PXWEB to distribute hierarchical data. You can browse the available data sets at:

 * [Statistics Sweden](http://www.statistikdatabasen.scb.se/pxweb/en/ssd/) with [API Description](https://www.scb.se/en/services/open-data-api/api-for-the-statistical-database/)
 * [Statistics Finland](https://stat.fi/til/aihealuejako.html) [StatFi API Description](https://statfin.stat.fi/api1.html)
 * [Other organizations using PX-WEB](https://www.scb.se/en/services/statistical-programs-for-px-files/px-web/pxweb-examples/)

### About PXWEB APIs

The data in PXWEB APIs consists of metadata and data
parts. Metadata is structured in a hierarchical node tree, where each node contains information about subnodes. The leaf nodes have information on which the dimensions are available for
the data at that leaf node.

## Installation

To install the latest stable release version from CRAN, just use:

```{r install1, eval=FALSE}
install.packages("pxweb")
```


To install the latest stable release version from GitHub, just use:

```{r install2, eval=FALSE}
library("remotes")
remotes::install_github("ropengov/pxweb")
```

Test the installation by loading the library:

```{r test, message=FALSE, warning=FALSE, eval=TRUE}
library(pxweb)
```

A tutorial is included with the package with:
```r
vignette(topic="pxweb")
```


### Installation issues

We also recommend setting the UTF-8 encoding since each API may have local specific letters:

```{r locale, eval=FALSE}
Sys.setlocale(locale = "UTF-8")
```


## Accessing PXWEB from R

There are two ways of using the `pxweb` R package to access data, either interactively or using the core functions. To access data, two parts are needed, an URL to the data table in the API and a query specifying what data is of interest. 

## Interactive use

The simplest way of using `pxweb` is to use it interactively, navigate the API to the data of interest, and then set up the query of interest.
When selecting values for a table variable, use `*` to select all available values, separate multiple choices with `,`, and use `:` to select a range of choices.
If a variable can be eliminated from the query, use `e` to eliminate it.

```{r standardquery, message=FALSE, eval=FALSE}
# Navigate through all pxweb api:s in the R package API catalogue
d <- pxweb_interactive()

# Get data from SCB (Statistics Sweden)
d <- pxweb_interactive("api.scb.se")

# Fetching data from statfi (Statistics Finland)
d <- pxweb_interactive("statfin.stat.fi")

# Fetching data from StatBank (Statistics Norway)
d <- pxweb_interactive("data.ssb.no")

# To see all available PXWEB APIs use
pxweb_apis <- pxweb_api_catalogue()
```

In the example above, we use the interactive functionality from the PXWEB API root, but we could use any path to the API.

```{r inapi, message=FALSE, eval=FALSE}
# Start with a specific path.
d <- pxweb_interactive("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A")
```

This functionality also means that we can navigate any PXWEB API, irrespectively of if they are a part of the R package API catalogue or not. Just supply an URL to somewhere in the API and then navigate the API from there.

Due to new CRAN policies, it is not possible to use an R function to edit the API catalogue of the R package, but editing them can be done quickly from R using `file.edit()`.

```{r, message=FALSE, eval=FALSE}
file.edit(pxweb_api_catalogue_path())
```

Although, if the `pxweb` is installed again, it will overwrite the old API catalogue. So the easiest way is to add a PXWEB API to the global catalogue. To do this, do a pull request at the pxweb GitHub page [here](https://github.com/rOpenGov/pxweb).

## Direct use

Under the hood, the pxweb package uses the `pxweb_get()` function to access data from the PXWEB API. It also keeps track of the API's time limits and splits big queries into optimal downloadable chunks. If we use `pxweb_get()` without a query, the function either returns a PXWEB LEVELS object or a PXWEB METADATA object. What is returned depends on if the URL points to a table in the API or not. Here is an example of a PXWEB LEVELS object.

```{r levels, message=FALSE, eval=FALSE}
# Get PXWEB levels
px_levels <- pxweb_get("https://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/")
px_levels
```
```{r , message=FALSE, eval=TRUE, echo=FALSE}
load("px_levels_example.rda")
px_levels
```

And if we use `pxweb_get()` for a table, a PXWEB METADATA object is returned.

```{r meta, message=FALSE, eval=FALSE}
# Get PXWEB metadata about a table
scb_table_url <- paste0(
  "https", "://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy"
)
px_meta <- pxweb_get(scb_table_url)
px_meta
```

```{r , message=FALSE, eval=TRUE, echo=FALSE}
load("px_meta_example.rda")
px_meta
```


### Creating data queries

To download data, we need both the URL to the table and a query specifying what parts of the table are of interest. An URL to a table is an URL that will return a metadata object if not a query is supplied. Creating a query can be done in three main ways. The first and most straightforward approach is to use `pxweb_interactive()` to explore the table URL and create a query interactively.

```{r, message=FALSE, eval=FALSE}
d <- pxweb_interactive(scb_table_url)
```

The interactive function returns an invisible list.
The list always contains the table URL and the query, even if the data is not downloaded.
If you choose to download data before exiting the interactive session, the list also contains a `data` element with the downloaded data.
This is different from `pxweb_get()`, which returns the API response itself.

```{r , message=FALSE, eval=TRUE, echo=FALSE}
# save(d, file = "d_example.rda")
load("d_example.rda")
```

```{r, message=FALSE, eval=TRUE}
names(d)
d$url
d$query
```

We can also turn the query into a JSON query that we can use outside R.

```{r interactive_query, message=FALSE, eval=TRUE}
pxweb_query_as_json(d$query, pretty = TRUE)
```


The second approach is to specify the query either as an R list or a JSON object. Some Statistical Agencies, such as Statistics Sweden, supply queries directly as a JSON object on their web pages. We can use these queries directly. Below is another example of a JSON query for the table above. For details on setting up a JSON query, see the PXWEB API documentation. 

```
{
  "query": [
    {
      "code": "Civilstand",
      "selection": {
        "filter": "item",
        "values": ["OG", "G", "ÄNKL", "SK"]
      }
    },
    {
      "code": "Kon",
      "selection": {
        "filter": "item",
        "values": ["1", "2"]
      }
    },
    {
      "code": "ContentsCode",
      "selection": {
        "filter": "item",
        "values": ["BE0101N1"]
      }
    },
    {
      "code": "Tid",
      "selection": {
        "filter": "item",
        "values": ["2015", "2016", "2017"]
      }
    }
  ],
  "response": {
    "format": "json"
  }
} 
```

To use this JSON query, we store the JSON query as a file and supply the path to the file to the "`pxweb_query()` "function.

```{r pxq, message=FALSE, eval=FALSE}
pxq <- pxweb_query("path/to/the/json/query.json")
```

Finally, we can create a PXWEB query from an R list where each list element is a variable and selected observation.

```{r rquery, message=FALSE, eval=TRUE}
pxweb_query_list <-
  list(
    "Civilstand" = c("*"), # Use "*" to select all
    "Kon" = c("1", "2"),
    "ContentsCode" = c("BE0101N1"),
    "Tid" = c("2015", "2016", "2017")
  )
pxq <- pxweb_query(pxweb_query_list)
pxq
```

Some PXWEB API v1 tables support additional selection filters, such as
aggregation filters. These can be expressed in an R query with
`pxweb_selection()`. For example, Statistics Finland tables may expose regional
aggregations where the API filter identifies the aggregation file and the value
selects the aggregated region.

```{r v1_aggregation_query, message=FALSE, eval=FALSE, purl=FALSE}
statfin_url <- paste0(
  "https://statfin.stat.fi",
  "/PxWeb/api/v1/en/StatFin/raku/15er.px"
)

aggregation_query <- list(
  alue_23_20260101 = pxweb_selection(
    filter = "agg:_Regions 2026.agg",
    values = "MK01"
  ),
  timeperiod_y = "2025",
  contentscode = "rakennus_lkm"
)

pxd <- pxweb_get_data(
  statfin_url,
  query = aggregation_query,
  column.name.type = "code",
  variable.value.type = "code"
)
```

We can validate the query against the metadata object to asses that we can use the query. This validation is done automatically when the data is fetched with `pxweb_get()` but can also be done manually.

```{r validate_query, message=FALSE, eval=TRUE}
pxweb_validate_query_with_metadata(pxq, px_meta)
```

### Downloading data

When we have the URL to a data table and a query, we can download the data with "`pxweb_get()` ". The function returns a `pxweb_data` object that contains the downloaded data.

```{r, message=FALSE, eval=FALSE}
pxd <- pxweb_get(
  scb_table_url,
  pxq
)
pxd
```

```{r , message=FALSE, eval=TRUE, echo=FALSE}
load("pxd_example.rda")
pxd
```

If we instead want a JSON-stat object, we change the response format to JSON-stat, and we will get a JSON-stat object returned. 

```{r, message=FALSE, eval=FALSE}
pxq$response$format <- "json-stat"
pxjstat <- pxweb_get(
  scb_table_url,
  pxq
)
pxjstat
```

```{r , message=FALSE, eval=TRUE, echo=FALSE}
load("pxjstat_example.rda")
pxjstat
```

Some return formats return files. Then, these responses are stored in the R `tempdir()` folded, and the file paths are returned by `pxweb_get()`. Currently, `px` and `sdmx` formats can be downloaded as files, but file an issue if you need other response formats.

```{r, message=FALSE, eval=FALSE}
pxq$response$format <- "px"
pxfp <- pxweb_get(
  scb_table_url,
  pxq
)
pxfp
```

```{r , message=FALSE, eval=TRUE, echo=FALSE}
load("pxfp_example.rda")
pxfp
```

If the queries are large (contain more values than the PXWEB API maximum allowed values), the query is chunked into optimal chunks and is then downloaded sequentially. PXWEB data objects are then combined into one large PXWEB data object, while JSON-stat objects are returned as a list of JSON-stat objects, and other files are stored in `tempdir()` as separate files.

For more advanced connections to the API, the `pxweb_advanced_get()` gives the flexibility to access the underlying HTTP calls using `httr` and log the HTTP calls for debugging.

### PXWEB API v2 tables

PXWEB API v2 tables use table identifiers and separate metadata and data endpoints. The `pxweb_get()` function handles this internally: when a query is supplied for a v2 table, the request is sent to the table's `/data` endpoint and JSON-stat2 is requested by default.

Use `pxweb_search()` to find v2 tables from an API root.

```{r, message=FALSE, eval=FALSE}
search_results <- pxweb_search(
  "population",
  api_url = paste0("https:", "//", "statistikdatabasen.scb.se", "/api/v2"),
  lang = "en",
  page_size = 5
)

search_results[, c("id", "label", "metadata_url")]
```

```{r, message=FALSE, eval=FALSE}
v2_url <- paste0(
  "https:", "//", "statistikdatabasen.scb.se",
  "/api/v2/tables/TAB638/metadata?lang=sv"
)

v2_query <- list(
  Region = "00",
  Civilstand = "OG",
  Alder = "0",
  Kon = "1",
  ContentsCode = "BE0101N1",
  Tid = "2024"
)

v2_data <- pxweb_get(v2_url, query = v2_query)
as.data.frame(v2_data, column.name.type = "code", variable.value.type = "code")
```

#### Query helpers for PXWEB API v2

PXWEB API v2 tables may expose value sets and aggregation codelists, for example age groups, regional groupings or other ready-made classifications. These are represented in the API with extra URL parameters such as `codelist[Variable]` and `outputValues[Variable]`. The helper functions below let you express these choices directly in the R query instead of writing those API details by hand.

The helpers can be mixed with ordinary character values:

```{r, message=FALSE, eval=FALSE}
v2_query <- list(
  Region = pxweb_all(),
  Alder = pxweb_aggregation("agg_Ålder5år_1"),
  Kon = c("1", "2"),
  ContentsCode = "BE0101N1",
  Tid = pxweb_latest()
)

v2_df <- pxweb_get_data(
  v2_url,
  query = v2_query,
  column.name.type = "code",
  variable.value.type = "code"
)
```

The most common helpers are:

```{r, message=FALSE, eval=FALSE}
pxweb_all()                         # all values for a variable
pxweb_latest()                      # latest time period, resolved from metadata
pxweb_top(10)                       # top 10 values
pxweb_bottom(10)                    # bottom 10 values
pxweb_aggregation("agg_Ålder5år_1") # use an aggregation codelist
pxweb_valueset("vs_Ålder1årG")      # use a value set codelist
```

If only some values from a codelist should be requested, supply them with `value_codes`.

```{r, message=FALSE, eval=FALSE}
v2_query <- list(
  Alder = pxweb_aggregation(
    "agg_Ålder5år_1",
    value_codes = c("0-4", "5-9", "10-14")
  ),
  Kon = "1",
  ContentsCode = "BE0101N1",
  Tid = pxweb_latest()
)
```

The exact aggregation and value set identifiers are found in the v2 metadata for a table. Use `pxweb_codelists()` to list them without inspecting the raw API response by hand.

```{r, message=FALSE, eval=FALSE}
v2_meta <- pxweb_get(v2_url)
pxweb_codelists(v2_meta, variable = "Alder")
```

`pxweb_latest()` is resolved against the table metadata before the data request is sent. By default it uses the placeholder `"9999"` in the query object and replaces it with the last available value for that variable from the metadata. It is mainly intended for time variables.

If you want a data frame directly, use `pxweb_get_data()`.

```{r, message=FALSE, eval=FALSE}
v2_df <- pxweb_get_data(
  v2_url,
  query = v2_query,
  column.name.type = "code",
  variable.value.type = "code"
)
```

In v2 JSON-stat2 output, the content variable is retained as a dimension, such as `ContentsCode`, and observations are returned in a generic `value` column.

We can then convert the downloaded PXWEB data objects to a `data. frame` or to a character matrix. The character matrix contains the "raw" data while `data. frame` returns an R `data.frame` in a tidy format. This conversion means missing values (such as ".." are converted to `NA`) in a `data. frame`. Using the arguments `variable.value.type` and `column.name.type`, we can choose if we want the code or the text column names and value types.

```{r, message=FALSE, eval=TRUE}
pxdf <- as.data.frame(pxd, column.name.type = "text", variable.value.type = "text")
head(pxdf)
```


```{r, message=FALSE, eval=TRUE}
pxdf <- as.data.frame(pxd, column.name.type = "code", variable.value.type = "code")
head(pxdf)
```

Similarly, we can access the raw data as a character matrix with `as.matrix`.

```{r, message=FALSE, eval=TRUE}
pxmat <- as.matrix(pxd, column.name.type = "code", variable.value.type = "code")
head(pxmat)
```

### Access data footnotes/comments

In addition to the data, the PXWEB DATA object may also contain comments for the data. This can be accessed using `pxweb_data_comments()` function.

```{r, message=FALSE, eval=TRUE}
pxdc <- pxweb_data_comments(pxd)
pxdc
```

In this case, we did not have any comments. If we have comments, we can turn the comments into a `data. frame` with one comment per row.

```{r, message=FALSE, eval=FALSE}
as.data.frame(pxdc)
```

## Citation

Finally, if we use the data, we can easily create a citation for a `pxweb_data` object using the `pxweb_cite()` function. For full reproducibility, please also cite the package.

```{r, message=FALSE, eval=FALSE}
pxweb_cite(pxd)
```

```{r, message=FALSE, eval=TRUE, echo=FALSE}
load("pxweb_cite_example.rda")
cat(pxweb_cite_example, sep = "\n")
```


## Known issues and troubleshooting

See [TROUBLESHOOTING.md](https://github.com/rOpenGov/pxweb/blob/master/TROUBLESHOOTING.md) for a list of current known issues.


## Licensing

This work can be freely used, modified and distributed under the open license specified in the [DESCRIPTION file](https://github.com/rOpenGov/pxweb/blob/master/DESCRIPTION).


## Session info

We created this vignette with

```{r sessioninfo, message=FALSE, warning=FALSE}
sessionInfo()
```