Automatic downloading of MRIO databases

Pymrio includes functions to automatically download some of the publicly available global EE MRIO databases. This is currently implemented for EXIOBASE 3, OECD and WIOD.

The functions described here download the raw data files. Thus, they can also be used for post processing by other tools.

EXIOBASE 3 download

EXIOBASE 3 is licensed under the Creative Commons Attribution 4.0 International-license. Thus you can remix, tweak, and build upon EXIOBASE 3, even commercially, as long as you give credit to the EXIOBASE compilers. The suggested citation for EXIOBASE 3 is Stadler et al 2018. You can find more information, links to documentation as well as concordance matrices on the EXIOBASE 3 Zenodo repository. The download function of pymrio also downloads the files from this repository.

To download, start with:

[1]:
import pymrio

and define a folder for storing the data:

[2]:
exio3_folder = "/tmp/mrios/autodownload/EXIO3"

With that we can start the download with (this might take a moment):

[3]:
exio_meta = pymrio.download_exiobase3(
    storage_folder=exio3_folder, system="pxp", years=[2011, 2012]
)

The command above will download the latest EXIOBASE 3 tables in the product by product classification (system=’pxp’) for the years 2011 and 2012. Both parameters (system and years) are optional and when omitted the function will download all available files.

The function returns the meta data for the release (which is stored in metadata.json in the download folder). You can inspect the meta data by:

[4]:
print(exio_meta)
Description: EXIOBASE3 metadata file for pymrio
MRIO Name: EXIO3
System: pxp
Version: 10.5281/zenodo.3583070
File: /tmp/mrios/autodownload/EXIO3/metadata.json
History:
20210223 15:40:31 - FILEIO -  Downloaded https://zenodo.org/record/4277368/files/IOT_2012_pxp.zip to IOT_2012_pxp.zip
20210223 15:38:16 - FILEIO -  Downloaded https://zenodo.org/record/4277368/files/IOT_2011_pxp.zip to IOT_2011_pxp.zip

By default, the download_exiobase3 fetches the latest version of EXIOBASE3 available at the EXIOBASE 3 Zenodo repository. To download one of the previous versions specify the DOI with the doi parameter:

[5]:
prev_version_storage = "/tmp/mrios/autodownload/EXIO3_7"
exio_meta_37 = pymrio.download_exiobase3(
    storage_folder=prev_version_storage,
    system="ixi",
    years=2004,
    doi="10.5281/zenodo.3583071",
)
[6]:
print(exio_meta_37)
Description: EXIOBASE3 metadata file for pymrio
MRIO Name: EXIO3
System: ixi
Version: 10.5281/zenodo.3583071
File: /tmp/mrios/autodownload/EXIO3_7/metadata.json
History:
20210223 15:43:42 - FILEIO -  Downloaded https://zenodo.org/record/3583071/files/IOT_2004_ixi.zip to IOT_2004_ixi.zip

Currently (Feb 2021), the following versions are available. Please double-check at the EXIOBASE 3 Zenodo repository (a box at the left sidebar titled ‘Versions’)

  • Version 3.7: 10.5281/zenodo.3583071 (only ixi files from 1995 to 2011 are available)
  • Version 3.8: 10.5281/zenodo.4277368

WIOD download

WIOD is licensed under the Creative Commons Attribution 4.0 International-license. Thus you can remix, tweak, and build upon WIOD, even commercially, as long as you give credit to WIOD. The WIOD web-page suggest to cite Timmer et al. 2015 when you use the database. You can find more information on the WIOD webpage.

The download function for WIOD currently processes the 2013 release version of WIOD.

To download, start with:

[7]:
import pymrio

Define a folder for storing the data

[8]:
wiod_folder = "/tmp/mrios/autodownload/WIOD2013"

And start the download with (this will take a couple of minutes):

[9]:
wiod_meta = pymrio.download_wiod2013(storage_folder=wiod_folder)

The function returns the meta data for the release (which is stored in metadata.json in the download folder). You can inspect the meta data by:

[10]:
print(wiod_meta)
Description: WIOD metadata file for pymrio
MRIO Name: WIOD
System: IxI
Version: data13
File: /tmp/mrios/autodownload/WIOD2013/metadata.json
History:
20210223 15:46:26 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/water/wat_may12.zip to wat_may12.zip
20210223 15:46:25 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/materials/mat_may12.zip to mat_may12.zip
20210223 15:46:25 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/land/lan_may12.zip to lan_may12.zip
20210223 15:46:24 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/AIR/AIR_may12.zip to AIR_may12.zip
20210223 15:46:24 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip
20210223 15:46:23 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EM/EM_may12.zip to EM_may12.zip
20210223 15:46:22 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip
20210223 15:46:21 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/SEA/WIOD_SEA_July14.xlsx to WIOD_SEA_July14.xlsx
20210223 15:46:20 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/update_sep12/wiot/wiot09_row_sep12.xlsx to wiot09_row_sep12.xlsx
20210223 15:46:15 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx
 ... (more lines in history)

The WIOD database provide data for several years and satellite accounts. In the default case, all of them are downloaded. You can, however, specify years and satellite account.

You can specify the years as either int or string (2 or 4 digits):

[11]:
res_years = [97, 2004, "2005"]

The available satellite accounts for WIOD are listed in the WIOD_CONFIG. To get them import this dict by:

[12]:
from pymrio.tools.iodownloader import WIOD_CONFIG
[13]:
WIOD_CONFIG
[13]:
{'url_db_view': 'http://www.wiod.org/database/wiots13',
 'url_db_content': 'http://www.wiod.org/',
 'mrio_regex': 'protected.*?wiot\\d\\d.*?xlsx',
 'satellite_urls': ['http://www.wiod.org/protected3/data13/SEA/WIOD_SEA_July14.xlsx',
  'http://www.wiod.org/protected3/data13/EU/EU_may12.zip',
  'http://www.wiod.org/protected3/data13/EM/EM_may12.zip',
  'http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip',
  'http://www.wiod.org/protected3/data13/AIR/AIR_may12.zip',
  'http://www.wiod.org/protected3/data13/land/lan_may12.zip',
  'http://www.wiod.org/protected3/data13/materials/mat_may12.zip',
  'http://www.wiod.org/protected3/data13/water/wat_may12.zip']}

To restrict this list, you can either copy paste the urls or automatically select the accounts:

[14]:
sat_accounts = ["EU", "CO2"]
res_satellite = [
    sat
    for sat in WIOD_CONFIG["satellite_urls"]
    if any(acc in sat for acc in sat_accounts)
]
[15]:
res_satellite
[15]:
['http://www.wiod.org/protected3/data13/EU/EU_may12.zip',
 'http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip']
[16]:
wiod_meta_res = pymrio.download_wiod2013(
    storage_folder="/tmp/foo_folder/WIOD2013_res",
    years=res_years,
    satellite_urls=res_satellite,
)
[17]:
print(wiod_meta_res)
Description: WIOD metadata file for pymrio
MRIO Name: WIOD
System: IxI
Version: data13
File: /tmp/foo_folder/WIOD2013_res/metadata.json
History:
20210218 15:29:34 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot01_row_apr12.xlsx to wiot01_row_apr12.xlsx
20210218 15:29:33 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot00_row_apr12.xlsx to wiot00_row_apr12.xlsx
20210218 15:29:32 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip
20210218 15:29:31 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip
20210218 15:29:30 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx
20210218 15:29:27 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot97_row_apr12.xlsx to wiot97_row_apr12.xlsx
20210218 15:29:26 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot05_row_apr12.xlsx to wiot05_row_apr12.xlsx

Subsequent download will only catch files currently not present in the folder, e.g.:

[18]:
additional_years = [2000, 2001]
wiod_meta_res = pymrio.download_wiod2013(
    storage_folder="/tmp/foo_folder/WIOD2013_res",
    years=res_years + additional_years,
    satellite_urls=res_satellite,
)

only downloads the years given in additional_years, appending these downloads to the meta data file.

[19]:
print(wiod_meta_res)
Description: WIOD metadata file for pymrio
MRIO Name: WIOD
System: IxI
Version: data13
File: /tmp/foo_folder/WIOD2013_res/metadata.json
History:
20210218 15:29:34 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot01_row_apr12.xlsx to wiot01_row_apr12.xlsx
20210218 15:29:33 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot00_row_apr12.xlsx to wiot00_row_apr12.xlsx
20210218 15:29:32 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip
20210218 15:29:31 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip
20210218 15:29:30 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx
20210218 15:29:27 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot97_row_apr12.xlsx to wiot97_row_apr12.xlsx
20210218 15:29:26 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot05_row_apr12.xlsx to wiot05_row_apr12.xlsx

To catch all files, irrespective if present in the storage_folder or not pass overwrite_existing=True

OECD download

The OECD Inter-Country Input-Output tables (ICIO) are available on the OECD webpage. There is no specific licence given for the these tables, but the webpage state that “Data can be downloaded for free” (per July 2019).

The download function works for both, the 2016 and 2018 release.

To download the data, we first define the folder for storing the data (these will be created if they do not exist yet):

[20]:
oecd_folder_v2018 = "/tmp/mrios/autodownload/OECD_2018"
oecd_folder_v2016 = "/tmp/mrios/autodownload/OECD_2016"

Than we can start the download with

[21]:
meta_2018 = pymrio.download_oecd(storage_folder=oecd_folder_v2018)

Be default, the 2018 release of the OECD - ICIO tables are downloaded. To retrieve the 2016 version, pass “version=’v2016”.

As for WIOD, specific years can be specified by passing a list of years:

[22]:
meta_2016 = pymrio.download_oecd(
    storage_folder=oecd_folder_v2016, version="v2016", years=[2003, 2008]
)

Both functions return the meta data describing the download progress and MRIO info. Thus:

[23]:
print(meta_2018)
Description: OECD-ICIO download
MRIO Name: OECD-ICIO
System: IxI
Version: v2018
File: /tmp/mrios/autodownload/OECD_2018/metadata.json
History:
20210223 16:00:46 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=9f579ef3-4685-45e4-a0ba-d1acbd9755a6 to ICIO2018_2015.zip
20210223 15:59:29 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=0190bd9d-31d0-4171-bd1c-82d96b88e469 to ICIO2018_2014.zip
20210223 15:58:34 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=8c8ac674-1b6c-4c8e-94d1-158f06285659 to ICIO2018_2013.zip
20210223 15:57:23 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=cfd03495-8a90-4449-8097-a30f06853cab to ICIO2018_2012.zip
20210223 15:56:04 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=dc48c8c0-f200-487a-aecb-0c2c17fe3ddf to ICIO2018_2011.zip
20210223 15:54:27 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=16d04830-3c27-47a5-bc03-e429d27f585e to ICIO2018_2010.zip
20210223 15:52:54 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=4cc79090-d1ee-48b6-a252-e75312d32a1c to ICIO2018_2009.zip
20210223 15:51:25 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=1fd2fc03-c140-46f4-818e-9a66b671ff70 to ICIO2018_2008.zip
20210223 15:50:08 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=c4d4c21d-00db-48d8-9f9a-f722fcdca494 to ICIO2018_2007.zip
20210223 15:48:55 - FILEIO -  Downloaded http://stats.oecd.org/wbos/fileview2.aspx?IDFile=da62c835-f4fa-4450-bf19-1dd60f88a385 to ICIO2018_2006.zip
 ... (more lines in history)

Eora26 download

Eora26 requires registration prior to download and therefore an automatic download has not been implemented. For further information check the download instruction at the Eora26 example notebook.

EXIOBASE download (previous version 1 and 2)

Previous EXIOBASE version requires registration prior to download and therefore an automatic download has not been implemented. For further information check the download instruction at the EXIOBASE example notebook.