{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automatic downloading of MRIO databases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pymrio includes functions to automatically download some of the publicly available global EE MRIO databases.\n",
    "This is currently implemented for [EXIOBASE 3](https://doi.org/10.5281/zenodo.3583070), [OECD](https://www.oecd.org/sti/ind/inter-country-input-output-tables.htm) and [WIOD](http://www.wiod.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The functions described here download the raw data files. Thus, they can also be used for post processing by other tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXIOBASE 3 download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EXIOBASE 3 is licensed under the [Creative Commons Attribution ShareAlike 4.0 International-license](https://creativecommons.org/licenses/by-sa/4.0/legalcode). Thus you can use EXIOBASE 3 for analyses as well as remix, tweak, and build uponit, even commercially, as long as you give credit to the EXIOBASE compilers and share your results/new databases under the same licence. The suggested citation for EXIOBASE 3 is [Stadler et al 2018](https://doi.org/10.1111/jiec.12715). You can find more information, links to documentation as well as concordance matrices on the [EXIOBASE 3 Zenodo repository](https://doi.org/10.5281/zenodo.3583070). The download function of pymrio also downloads the files from this repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download, start with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pymrio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and define a folder for storing the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "exio3_folder = \"/tmp/mrios/autodownload/EXIO3\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With that we can start the download with (this might take a moment):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "exio_downloadlog = pymrio.download_exiobase3(\n",
    "    storage_folder=exio3_folder, system=\"pxp\", years=[2011, 2012]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command above will download the latest EXIOBASE 3 tables in the product\n",
    "by product classification (system='pxp') for the years 2011 and 2012. Both\n",
    "parameters (system and years) are optional and when omitted the function will\n",
    "download all available files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function returns a log of the download data (which is stored in ```download_log.json``` in the download folder).\n",
    "You can inspect the meta data by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: Download log of EXIOBASE3\n",
      "MRIO Name: EXIO3\n",
      "System: pxp\n",
      "Version: 10.5281/zenodo.3583070\n",
      "File: /tmp/mrios/autodownload/EXIO3/download_log.json\n",
      "History:\n",
      "20230323 09:32:39 - FILEIO -  Downloaded https://zenodo.org/record/5589597/files/IOT_2012_pxp.zip to IOT_2012_pxp.zip\n",
      "20230323 09:26:03 - FILEIO -  Downloaded https://zenodo.org/record/5589597/files/IOT_2011_pxp.zip to IOT_2011_pxp.zip\n",
      "20230323 09:19:45 - NOTE -  Download log created\n",
      "20230323 09:19:45 - NOTE -  python_version: 3.9.16\n",
      "20230323 09:19:45 - NOTE -  pymrio_version: 0.5.0dev\n",
      "20230323 09:19:45 - NOTE -  os: Linux\n",
      "20230323 09:19:45 - NOTE -  hostname: NTNU09417\n",
      "20230323 09:19:45 - NOTE -  username: konstans\n"
     ]
    }
   ],
   "source": [
    "print(exio_downloadlog)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the download_exiobase3 fetches the latest version of EXIOBASE3\n",
    "available at the [EXIOBASE 3 Zenodo repository](https://doi.org/10.5281/zenodo.3583070).\n",
    "To download one of the previous versions specify the DOI with the doi\n",
    "parameter:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prev_version_storage = \"/tmp/mrios/autodownload/EXIO3_7\"\n",
    "exio_downlog_37 = pymrio.download_exiobase3(\n",
    "    storage_folder=prev_version_storage,\n",
    "    system=\"ixi\",\n",
    "    years=2004,\n",
    "    doi=\"10.5281/zenodo.3583071\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: Download log of EXIOBASE3\n",
      "MRIO Name: EXIO3\n",
      "System: ixi\n",
      "Version: 10.5281/zenodo.3583071\n",
      "File: /tmp/mrios/autodownload/EXIO3_7/download_log.json\n",
      "History:\n",
      "20230323 09:44:54 - FILEIO -  Downloaded https://zenodo.org/record/3583071/files/IOT_2004_ixi.zip to IOT_2004_ixi.zip\n",
      "20230323 09:43:49 - NOTE -  Download log created\n",
      "20230323 09:43:49 - NOTE -  python_version: 3.9.16\n",
      "20230323 09:43:49 - NOTE -  pymrio_version: 0.5.0dev\n",
      "20230323 09:43:49 - NOTE -  os: Linux\n",
      "20230323 09:43:49 - NOTE -  hostname: NTNU09417\n",
      "20230323 09:43:49 - NOTE -  username: konstans\n"
     ]
    }
   ],
   "source": [
    "print(exio_downlog_37)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "We also recommend to specifiy a specific DOI version even when using the latest version of EXIOBASE. In that way the used version is documented in the code and can be reproduced in case a newer EXIOBASE version becomes available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "## WIOD download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**DUE TO A RESTRUCTERING OF THE WIOD WEBPAGE THIS IS CURRENTLY BROKEN.**\n",
    "\n",
    "\n",
    "WIOD is licensed under the [Creative Commons Attribution 4.0 International-license](http://creativecommons.org/licenses/by/4.0/). Thus you can remix, tweak, and build upon WIOD, even commercially, as long as you give credit to WIOD. The WIOD web-page suggest to cite [Timmer et al. 2015](http://doi.wiley.com/10.1111/roie.12178) when you use the database. You can find more information on the [WIOD webpage](http://www.wiod.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The download function for WIOD currently processes the [2013 release version of WIOD](http://www.wiod.org/database/wiots13)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download, start with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymrio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a folder for storing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "wiod_folder = \"/tmp/mrios/autodownload/WIOD2013\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And start the download with (this will take a couple of minutes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "wiod_meta = pymrio.download_wiod2013(storage_folder=wiod_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function returns the meta data for the release (which is stored in ```metadata.json``` in the download folder).\n",
    "You can inspect the meta data by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: WIOD metadata file for pymrio\n",
      "MRIO Name: WIOD\n",
      "System: IxI\n",
      "Version: data13\n",
      "File: /tmp/mrios/autodownload/WIOD2013/metadata.json\n",
      "History:\n",
      "20210223 15:46:26 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/water/wat_may12.zip to wat_may12.zip\n",
      "20210223 15:46:25 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/materials/mat_may12.zip to mat_may12.zip\n",
      "20210223 15:46:25 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/land/lan_may12.zip to lan_may12.zip\n",
      "20210223 15:46:24 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/AIR/AIR_may12.zip to AIR_may12.zip\n",
      "20210223 15:46:24 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip\n",
      "20210223 15:46:23 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EM/EM_may12.zip to EM_may12.zip\n",
      "20210223 15:46:22 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip\n",
      "20210223 15:46:21 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/SEA/WIOD_SEA_July14.xlsx to WIOD_SEA_July14.xlsx\n",
      "20210223 15:46:20 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/update_sep12/wiot/wiot09_row_sep12.xlsx to wiot09_row_sep12.xlsx\n",
      "20210223 15:46:15 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx\n",
      " ... (more lines in history)\n"
     ]
    }
   ],
   "source": [
    "print(wiod_meta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The WIOD database provide data for several years and satellite accounts.\n",
    "In the default case, all of them are downloaded. You can, however, specify\n",
    "years and satellite account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can specify the years as either int or string (2 or 4 digits):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "res_years = [97, 2004, \"2005\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The available satellite accounts for WIOD are listed in the ```WIOD_CONFIG```.\n",
    "To get them import this dict by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymrio.tools.iodownloader import WIOD_CONFIG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'url_db_view': 'http://www.wiod.org/database/wiots13',\n",
       " 'url_db_content': 'http://www.wiod.org/',\n",
       " 'mrio_regex': 'protected.*?wiot\\\\d\\\\d.*?xlsx',\n",
       " 'satellite_urls': ['http://www.wiod.org/protected3/data13/SEA/WIOD_SEA_July14.xlsx',\n",
       "  'http://www.wiod.org/protected3/data13/EU/EU_may12.zip',\n",
       "  'http://www.wiod.org/protected3/data13/EM/EM_may12.zip',\n",
       "  'http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip',\n",
       "  'http://www.wiod.org/protected3/data13/AIR/AIR_may12.zip',\n",
       "  'http://www.wiod.org/protected3/data13/land/lan_may12.zip',\n",
       "  'http://www.wiod.org/protected3/data13/materials/mat_may12.zip',\n",
       "  'http://www.wiod.org/protected3/data13/water/wat_may12.zip']}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "WIOD_CONFIG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To restrict this list, you can either copy paste the urls or automatically select the accounts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sat_accounts = [\"EU\", \"CO2\"]\n",
    "res_satellite = [\n",
    "    sat\n",
    "    for sat in WIOD_CONFIG[\"satellite_urls\"]\n",
    "    if any(acc in sat for acc in sat_accounts)\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://www.wiod.org/protected3/data13/EU/EU_may12.zip',\n",
       " 'http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_satellite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "wiod_meta_res = pymrio.download_wiod2013(\n",
    "    storage_folder=\"/tmp/foo_folder/WIOD2013_res\",\n",
    "    years=res_years,\n",
    "    satellite_urls=res_satellite,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: WIOD metadata file for pymrio\n",
      "MRIO Name: WIOD\n",
      "System: IxI\n",
      "Version: data13\n",
      "File: /tmp/foo_folder/WIOD2013_res/metadata.json\n",
      "History:\n",
      "20210218 15:29:34 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot01_row_apr12.xlsx to wiot01_row_apr12.xlsx\n",
      "20210218 15:29:33 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot00_row_apr12.xlsx to wiot00_row_apr12.xlsx\n",
      "20210218 15:29:32 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip\n",
      "20210218 15:29:31 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip\n",
      "20210218 15:29:30 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx\n",
      "20210218 15:29:27 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot97_row_apr12.xlsx to wiot97_row_apr12.xlsx\n",
      "20210218 15:29:26 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot05_row_apr12.xlsx to wiot05_row_apr12.xlsx\n"
     ]
    }
   ],
   "source": [
    "print(wiod_meta_res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Subsequent download will only catch files currently not present in the folder, e.g.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "additional_years = [2000, 2001]\n",
    "wiod_meta_res = pymrio.download_wiod2013(\n",
    "    storage_folder=\"/tmp/foo_folder/WIOD2013_res\",\n",
    "    years=res_years + additional_years,\n",
    "    satellite_urls=res_satellite,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "only downloads the years given in ```additional_years```, appending these downloads to the meta data file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: WIOD metadata file for pymrio\n",
      "MRIO Name: WIOD\n",
      "System: IxI\n",
      "Version: data13\n",
      "File: /tmp/foo_folder/WIOD2013_res/metadata.json\n",
      "History:\n",
      "20210218 15:29:34 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot01_row_apr12.xlsx to wiot01_row_apr12.xlsx\n",
      "20210218 15:29:33 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot00_row_apr12.xlsx to wiot00_row_apr12.xlsx\n",
      "20210218 15:29:32 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/CO2/CO2_may12.zip to CO2_may12.zip\n",
      "20210218 15:29:31 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/EU/EU_may12.zip to EU_may12.zip\n",
      "20210218 15:29:30 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot04_row_apr12.xlsx to wiot04_row_apr12.xlsx\n",
      "20210218 15:29:27 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot97_row_apr12.xlsx to wiot97_row_apr12.xlsx\n",
      "20210218 15:29:26 - FILEIO -  Downloaded http://www.wiod.org/protected3/data13/wiot_analytic/wiot05_row_apr12.xlsx to wiot05_row_apr12.xlsx\n"
     ]
    }
   ],
   "source": [
    "print(wiod_meta_res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To catch all files, irrespective if present in the storage_folder or not pass ```overwrite_existing=True```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## OECD download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The OECD Inter-Country Input-Output tables (ICIO) are available on the [OECD webpage.](https://www.oecd.org/sti/ind/inter-country-input-output-tables.htm) There is no specific license given for the these tables, but the webpage state that \"Data can be downloaded for free\" (per June 2023) and to cite this database \"OECD (2021), OECD Inter-Country Input-Output Database, http://oe.cd/icio\".\n",
    "\n",
    "Currently OECD provides three versions 2016, 2018, and 2021, which are available through pymrio\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download the data, we first define the folder for storing the data (these will be created if they do not exist yet):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "oecd_folder_v2021 = \"/tmp/mrios/autodownload/OECD_2021\"\n",
    "oecd_folder_v2018 = \"/tmp/mrios/autodownload/OECD_2018\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Than we can start the download with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_2021 = pymrio.download_oecd(storage_folder=oecd_folder_v2021)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the 2021 release of the OECD - ICIO tables are downloaded.\n",
    "To retrieve other versions, simply pass \"version='v<version date>'\", for example to get the 2018 release, pass \"version='v2018'\" .\n",
    "\n",
    "As for WIOD, specific years can be specified by passing a list of years:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_2018 = pymrio.download_oecd(\n",
    "    storage_folder=oecd_folder_v2018, version=\"v2016\", years=[2003, 2008]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, the 2021 release of OECD is only available in 5-year bundles starting from 1995 to 2018\n",
    "1995-1999, 2000-2004, 2005-2009, 2010-2014, 2015-2018\n",
    "\n",
    "Therefore whenever 2021 release is used, it is recommended to pass years as a list of the wanted bundles in string forms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_2021 = pymrio.download_oecd(\n",
    "    storage_folder=oecd_folder_v2021, years=[\"1995-1999\", \"2015-2018\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otherwise the corresponding bundles for the entered years would be downloaded, for example if year 2003 is requested, the bundle 2000-2004 would be downloaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_2021 = pymrio.download_oecd(storage_folder=oecd_folder_v2021, years=[2003, 2012])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The bundles 2000-2004 and 2010-2014 would be downloaded\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>\n",
    "<br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function returns a log for the download progress and MRIO info:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: OECD-ICIO download\n",
      "MRIO Name: OECD-ICIO\n",
      "System: IxI\n",
      "Version: v2021\n",
      "File: \\tmp\\mrios\\autodownload\\OECD_2021\\download_log.json\n",
      "History:\n",
      "20230706 15:35:10 - FILEIO -  Downloaded https://stats.oecd.org/wbos/fileview2.aspx?IDFile=2c2f499f-5703-4034-9457-2f7518e8f2fc to ICIO2021_2010-2014.zip\n",
      "20230706 15:31:47 - FILEIO -  Downloaded https://stats.oecd.org/wbos/fileview2.aspx?IDFile=8adf89dd-18b4-40fe-bc7f-c822052eb961 to ICIO2021_2000-2004.zip\n",
      "20230706 15:29:43 - NOTE -  Download log created\n",
      "20230706 15:29:43 - NOTE -  python_version: 3.11.2\n",
      "20230706 15:29:43 - NOTE -  pymrio_version: 0.5.1\n",
      "20230706 15:29:43 - NOTE -  os: Windows\n",
      "20230706 15:29:43 - NOTE -  hostname: NTNU09702\n",
      "20230706 15:29:43 - NOTE -  username: hazimh\n"
     ]
    }
   ],
   "source": [
    "print(log_2021)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Eora26 download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eora26 provides a simplified, symmetric version of the full Eora database. \n",
    "\n",
    "Downloading the Eora data requires registration through the [Eora website (worldmrio)](http://www.worldmrio.com) . \n",
    "Currently (August 2023), open Eora26 data are only offered for the years 1990 - 2016.\n",
    "\n",
    "Therefore, you have to have the Eora account email and password in advance to use the downloader."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the download with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import pymrio\n",
    "\n",
    "eora_folder = \"/tmp/mrios/eora26\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Start the download with (passing the email and password of the eora account)\n",
    "If you pass invalid email or password, you get a prompt about that and asked to enter them again, otherwise the function will download the data (this can take some minutes)\n",
    "\n",
    "This will download all available years 1990 - 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Eora account with this email was not found\n",
      "\n",
      "            Please try again or register a new user at the following website:\n",
      "\n",
      "            https://worldmrio.com/login.jsp\n",
      "The password for this email account is incorrect\n",
      "\n",
      "            Please try again\n"
     ]
    }
   ],
   "source": [
    "eora_log = pymrio.download_eora26(\n",
    "    storage_folder=eora_folder,\n",
    "    email=\"<Your Eora account email>\",\n",
    "    password=\"<Your Eora account password>\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: Eora metadata file for pymrio\n",
      "MRIO Name: Eora\n",
      "System: ixi\n",
      "Version: v199.82\n",
      "File: /tmp/mrios/eora26/metadata.json\n",
      "History:\n",
      "20180111 10:26:35 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1995_bp.zip to Eora26_1995_bp.zip\n",
      "20180111 10:26:26 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1996_bp.zip to Eora26_1996_bp.zip\n",
      "20180111 10:26:17 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1997_bp.zip to Eora26_1997_bp.zip\n",
      "20180111 10:25:57 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1998_bp.zip to Eora26_1998_bp.zip\n",
      "20180111 10:25:47 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_1999_bp.zip to Eora26_1999_bp.zip\n",
      "20180111 10:25:37 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2000_bp.zip to Eora26_2000_bp.zip\n",
      "20180111 10:25:21 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2001_bp.zip to Eora26_2001_bp.zip\n",
      "20180111 10:25:08 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2002_bp.zip to Eora26_2002_bp.zip\n",
      "20180111 10:24:58 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2003_bp.zip to Eora26_2003_bp.zip\n",
      "20180111 10:24:46 - FILEIO -  Downloaded http://worldmrio.com/ComputationsM/Phase199/Loop082/simplified/Eora26_2004_bp.zip to Eora26_2004_bp.zip\n",
      " ... (more lines in history)\n"
     ]
    }
   ],
   "source": [
    "print(eora_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the case of the WIOD downloader, you can restrict the \n",
    "\n",
    "1) years to download by passing ```years=[list of int or str - 4 digits]```\n",
    "\n",
    "2) force the overwriting of existing files by passing ```overwrite_existing=True```\n",
    "\n",
    "Satellite accounts, however, can not be restricted since they are included in one file.\n",
    "\n",
    "The tables are in basic prices as it is the only price system available to download for Eora26."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXIOBASE download (previous version 1 and 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previous EXIOBASE version requires registration prior to download and therefore an automatic download has not been implemented.\n",
    "For further information check the download instruction at the [EXIOBASE example notebook.](working_with_exiobase.ipynb#Getting-EXIOBASE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GLORIA download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Global Resource Input Output Assessment (GLORIA) database are available to download through a dropbox folder [GLORIA database.](https://www.dropbox.com/sh/o4fxq94n7grvdbk/AABhKvEVx0UuMvz4dQ4NlWC8a?d) There is no specific licence given for the this database.\n",
    "\n",
    "Currently (as per May 2024), there are five available versions 53, 54, 55, 57 and 59 (Release 056 was only\n",
    "distributed to a limited number of users for feedback).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The download function can (as per April 2023) download all four versions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To download, start with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pymrio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a folder for storing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "gloria_folder = \"/tmp/mrios/autodownload/GLORIA2014\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And start the download with (this will take a couple of minutes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gloria_log_2014 = pymrio.download_gloria(storage_folder=gloria_folder)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function returns the download log data for the release (which is stored in ```download_log.json``` in the download folder).\n",
    "You can inspect the log data by:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Description: Download log of Gloria\n",
      "MRIO Name: GLORIA\n",
      "System: IxI\n",
      "Version: 57\n",
      "File: /tmp/mrios/autodownload/GLORIA2014/download_log.json\n",
      "History:\n",
      "20230418 07:51:15 - NOTE -  Download log created\n",
      "20230418 07:51:15 - NOTE -  python_version: 3.11.3\n",
      "20230418 07:51:15 - NOTE -  pymrio_version: 0.5.0dev\n",
      "20230418 07:51:15 - NOTE -  os: Linux\n",
      "20230418 07:51:15 - NOTE -  hostname: instance-exio-svr\n",
      "20230418 07:51:15 - NOTE -  username: hazimh\n",
      "20230418 07:50:38 - FILEIO -  Downloaded https://dl.dropboxusercontent.com/sh/o4fxq94n7grvdbk/AACposoILlF8OETb20yh4st7a/latest_release/057/GLORIA_MRIO_Loop057_part_I_MRIOdatabase/GLORIA_MRIOs_57_2014.zip?dl=0 to GLORIA_MRIOs_57_2014.zip\n",
      "20230418 07:46:30 - NOTE -  Download log created\n",
      "20230418 07:46:30 - NOTE -  python_version: 3.11.3\n",
      "20230418 07:46:30 - NOTE -  pymrio_version: 0.5.0dev\n",
      " ... (more lines in history)\n"
     ]
    }
   ],
   "source": [
    "print(gloria_log_2014)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default the function downloads all years for the final release (59 as per May 2024), but the year and version can be specified by passing them to the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "gloria_log_v053_2012 = pymrio.download_gloria(gloria_folder, year=2012, version=53)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "jupytext": {
   "formats": "ipynb,py:light"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}