Automatic downloading of MRIO databases

Pymrio includes functions to automatically download some of the publicly available global EE MRIO databases. This is currently implemented for WIOD and Eora26.

The functions described here download the raw data files. Thus, they can also be used for post processing by other tools.

WIOD download

WIOD is licensed under the Creative Commons Attribution 4.0 International-license. Thus you can remix, tweak, and build upon WIOD, even commercially, as long as you give credit to WIOD. The WIOD web-page suggest to cite Timmer et al. 2015 when you use the database. You can find more information on the WIOD webpage.

The download function for WIOD currently processes the 2013 release version of WIOD.

To download, start with:

In [1]:

import pymrio

Define a folder for storing the data

In [2]:

wiod_folder = '/tmp/mrios/WIOD2013'

And start the download with (this will take a couple of minutes):

In [3]:

wiod_meta = pymrio.download_wiod2013(storage_folder=wiod_folder)

The function returns the meta data for the release (which is stored in metadata.json in the download folder). You can inspect the meta data by:

In [4]:

print(wiod_meta)

Description: WIOD metadata file for pymrio
MRIO Name: WIOD
System: ixi
Version: data13
File: /tmp/mrios/WIOD2013/metadata.json
History:
20180111 10:11:06 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/water/wat_may12.zip to wat_may12.zip
20180111 10:11:05 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/materials/mat_may12.zip to mat_may12.zip
20180111 10:11:05 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/land/lan_may12.zip to lan_may12.zip
20180111 10:11:04 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/AIR/AIR_may12.zip to AIR_may12.zip
20180111 10:11:03 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip
20180111 10:11:02 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EM/EM_may12.zip to EM_may12.zip
20180111 10:11:02 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip
20180111 10:11:01 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/SEA/WIOD_SEA_July14.xlsx to WIOD_SEA_July14.xlsx
20180111 10:11:00 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/update_sep12/wiot/wiot09_row_sep12.xlsx to wiot09_row_sep12.xlsx
20180111 10:10:58 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx
 ... (more lines in history)

The WIOD database provide data for several years and satellite accounts. In the default case, all of them are downloaded. You can, however, specify years and satellite account.

You can specify the years as either int or string (2 or 4 digits):

In [5]:

res_years = [97,2004,'2005']

The available satellite accounts for WIOD are listed in the WIOD_CONFIG. To get them import this dict by:

In [6]:

from pymrio.tools.iodownloader import WIOD_CONFIG

In [7]:

WIOD_CONFIG

Out[7]:

{'satellite_urls': ['http://www.wiod.org/protected3/data13/SEA/WIOD_SEA_July14.xlsx',
  'http://www.wiod.org/protected3/data13/EU/EU_may12.zip',
  'http://www.wiod.org/protected3/data13/EM/EM_may12.zip',
  'http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip',
  'http://www.wiod.org/protected3/data13/AIR/AIR_may12.zip',
  'http://www.wiod.org/protected3/data13/land/lan_may12.zip',
  'http://www.wiod.org/protected3/data13/materials/mat_may12.zip',
  'http://www.wiod.org/protected3/data13/water/wat_may12.zip'],
 'url_db_content': 'http://www.wiod.org/',
 'url_db_view': 'http://www.wiod.org/database/wiots13'}

To restrict this list, you can either copy paste the urls or automatically select the accounts:

In [8]:

sat_accounts = ['EU', 'CO2']
res_satellite = [sat for sat in WIOD_CONFIG['satellite_urls']
                 if any(acc in sat for acc in sat_accounts)]

In [9]:

res_satellite

Out[9]:

['http://www.wiod.org/protected3/data13/EU/EU_may12.zip',
 'http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip']

In [10]:

wiod_meta_res = pymrio.download_wiod2013(storage_folder='/tmp/foo_folder/WIOD2013_res',
                                         years=res_years,
                                         satellite_urls=res_satellite)

In [11]:

print(wiod_meta_res)

Description: WIOD metadata file for pymrio
MRIO Name: WIOD
System: ixi
Version: data13
File: /tmp/foo_folder/WIOD2013_res/metadata.json
History:
20180111 10:22:41 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip
20180111 10:22:41 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip
20180111 10:22:40 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx
20180111 10:22:38 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot97_row_apr12.xlsx to wiot97_row_apr12.xlsx
20180111 10:22:37 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot05_row_apr12.xlsx to wiot05_row_apr12.xlsx

Subsequent download will only catch files currently not present in the folder, e.g.:

In [12]:

additional_years = [2000, 2001]
wiod_meta_res = pymrio.download_wiod2013(storage_folder='/tmp/foo_folder/WIOD2013_res',
                                         years=res_years + additional_years,
                                         satellite_urls=res_satellite)

only downloads the years given in additional_years, appending these downloads to the meta data file.

In [13]:

print(wiod_meta_res)

Description: WIOD metadata file for pymrio
MRIO Name: WIOD
System: ixi
Version: data13
File: /tmp/foo_folder/WIOD2013_res/metadata.json
History:
20180111 10:22:46 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot01_row_apr12.xlsx to wiot01_row_apr12.xlsx
20180111 10:22:45 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot00_row_apr12.xlsx to wiot00_row_apr12.xlsx
20180111 10:22:41 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip
20180111 10:22:41 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip
20180111 10:22:40 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx
20180111 10:22:38 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot97_row_apr12.xlsx to wiot97_row_apr12.xlsx
20180111 10:22:37 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot05_row_apr12.xlsx to wiot05_row_apr12.xlsx

To catch all files, irrespective if present in the storage_folder or not pass overwrite_existing=True

Eora26 download

Eora26 provides a simplified, symmetric version of the full Eora database.

The Eora website (worldmrio) states that Eora is free for use at degree-granting academic institutions. All other users must license the data. Further information can be obtained by contacting info@worldmrio.com .

Prior to the access to the Eora database, you have to agree to this license agreement and commit to cite Lenzen et al. 2012 and Lenzen et al. 2013.

The same applies when using the automatic downloader and you have to explicitly confirm to these requirements during the download.

Setup the download with

In [14]:

import pymrio
eora_folder = '/tmp/mrios/eora26'

Start the download with (this can take some minutes). Before the download starts, the function will ask you to agree with the conditions stated on the Eora website (worldmrio).

In [15]:

eora_meta = pymrio.download_eora26(storage_folder=eora_folder, prices=['bp'])

The Eora MRIO is free for academic (university or grant-funded) work at degree-granting institutions. All other uses require a data license before the results are shared.

 When using Eora, the Eora authors ask you cite these publications:

 Lenzen, M., Kanemoto, K., Moran, D., Geschke, A. Mapping the Structure of the World Economy (2012). Env. Sci. Tech. 46(15) pp 8374-8381. DOI:10.1021/es300171x

 Lenzen, M., Moran, D., Kanemoto, K., Geschke, A. (2013) Building Eora: A Global Multi-regional Input-Output Database at High Country and Sector Resolution, Economic Systems Research,  25:1, 20-49, DOI:10.1080/09535314.2013.769 938


Do you agree with these conditions [y/n]: y

In [16]:

print(eora_meta)

Description: Eora metadata file for pymrio
MRIO Name: Eora
System: ixi
Version: v199.82
File: /tmp/mrios/eora26/metadata.json
History:
20180111 10:26:35 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1995_bp.zip to Eora26_1995_bp.zip
20180111 10:26:26 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1996_bp.zip to Eora26_1996_bp.zip
20180111 10:26:17 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1997_bp.zip to Eora26_1997_bp.zip
20180111 10:25:57 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1998_bp.zip to Eora26_1998_bp.zip
20180111 10:25:47 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1999_bp.zip to Eora26_1999_bp.zip
20180111 10:25:37 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2000_bp.zip to Eora26_2000_bp.zip
20180111 10:25:21 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2001_bp.zip to Eora26_2001_bp.zip
20180111 10:25:08 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2002_bp.zip to Eora26_2002_bp.zip
20180111 10:24:58 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2003_bp.zip to Eora26_2003_bp.zip
20180111 10:24:46 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2004_bp.zip to Eora26_2004_bp.zip
 ... (more lines in history)

As in the case of the WIOD downloader, you can restrict the

  1. years to download by passing years=[list of int or str - 4 digits]
  2. force the overwriting of existing files by passing overwrite_existing=True

Satellite accounts, however, can not be restricted since they are included in one file.

In the default case, the tables in basic prices are downloaded. To catch the purchaser price tables pass prices='pp' or prices=['bp', 'pp'] to catch both price systems.

EXIOBASE download

EXIOBASE requires registration prior to download and therefore an automatic download has not been implemented. For further information check the download instruction at the EXIOBASE example notebook.