Loading, saving and exporting data¶
Pymrio includes several functions for data reading and storing. This section presents the methods to use for saving and loading data already in a pymrio compatible format. For parsing raw MRIO data see the different tutorials for working with available MRIO databases.
Here, we use the included small test MRIO system to highlight the different function. The same functions are available for any MRIO loaded into pymrio. Expect, however, significantly decreased performance due to the size of real MRIO system.
In [1]:
import pymrio
io = pymrio.load_test().calc_all()
Basic save and read¶
To save the full system, use:
In [2]:
save_folder_full = '/tmp/testmrio/full'
io.save_all(path=save_folder_full)
Out[2]:
<pymrio.core.mriosystem.IOSystem at 0x7fad28466390>
To read again from that folder do:
In [3]:
io_read = pymrio.load_all(path=save_folder_full)
The fileio activities are stored in the included meta data history field:
In [4]:
io_read.meta
Out[4]:
Description: test mrio for pymrio
MRIO Name: testmrio
System: pxp
Version: v1
File: /tmp/testmrio/full/metadata.json
History:
20180110 15:50:46 - FILEIO - Added satellite account from /tmp/testmrio/full/factor_inputs
20180110 15:50:46 - FILEIO - Added satellite account from /tmp/testmrio/full/emissions
20180110 15:50:46 - FILEIO - Loaded IO system from /tmp/testmrio/full
20180110 15:50:46 - FILEIO - Saved testmrio to /tmp/testmrio/full
20180110 15:50:46 - MODIFICATION - Calculating accounts for extension emissions
20180110 15:50:45 - MODIFICATION - Calculating accounts for extension factor_inputs
20180110 15:50:45 - MODIFICATION - Calculating aggregated final demand
20180110 15:50:45 - MODIFICATION - Leontief matrix L calculated
20180110 15:50:45 - MODIFICATION - Coefficient matrix A calculated
20180110 15:50:45 - MODIFICATION - Industry output x calculated
... (more lines in history)
Storage format¶
Internally, pymrio stores data in csv format, with the ‘economic core’ data in the root and each satellite account in a subfolder. Metadata as file as a file describing the data format (‘file_parameters.json’) are included in each folder.
In [5]:
import os
os.listdir(save_folder_full)
Out[5]:
['emissions',
'factor_inputs',
'metadata.json',
'file_parameters.json',
'population.txt',
'unit.txt',
'L.txt',
'A.txt',
'x.txt',
'Y.txt',
'Z.txt']
The file format for storing the MRIO data can be switched to a binary pickle format with:
In [6]:
save_folder_bin = '/tmp/testmrio/binary'
io.save_all(path=save_folder_bin, table_format='pkl')
os.listdir(save_folder_bin)
Out[6]:
['emissions',
'factor_inputs',
'metadata.json',
'file_parameters.json',
'population.pkl',
'unit.pkl',
'L.pkl',
'A.pkl',
'x.pkl',
'Y.pkl',
'Z.pkl']
This can be used to reduce the storage space required on the disk for large MRIO databases.
Storing or exporting a specific table or extension¶
Each extension of the MRIO system can be stored separetly with:
In [7]:
save_folder_em= '/tmp/testmrio/emissions'
In [8]:
io.emissions.save(path=save_folder_em)
Out[8]:
<pymrio.core.mriosystem.Extension at 0x7fad28485208>
This can than be loaded again as separate satellite account:
In [9]:
emissions = pymrio.load(save_folder_em)
In [10]:
emissions
Out[10]:
<pymrio.core.mriosystem.Extension at 0x7fad18c9ecf8>
In [11]:
emissions.D_cba
Out[11]:
| region | reg1 | reg2 | ... | reg5 | reg6 | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sector | food | mining | manufactoring | electricity | construction | trade | transport | other | food | mining | ... | transport | other | food | mining | manufactoring | electricity | construction | trade | transport | other | |
| stressor | compartment | |||||||||||||||||||||
| emission_type1 | air | 2.056183e+06 | 179423.535893 | 9.749300e+07 | 1.188759e+07 | 3.342906e+06 | 3.885884e+06 | 1.075027e+07 | 1.582152e+07 | 1.793338e+06 | 19145.604911 | ... | 4.209505e+07 | 1.138661e+07 | 1.517235e+07 | 1.345318e+06 | 7.145075e+07 | 3.683167e+07 | 1.836696e+06 | 4.241568e+07 | 4.805409e+07 | 3.602298e+07 |
| emission_type2 | water | 2.423103e+05 | 25278.192086 | 1.671240e+07 | 1.371303e+05 | 3.468292e+05 | 7.766205e+05 | 4.999628e+05 | 8.480505e+06 | 2.136528e+05 | 3733.601474 | ... | 4.243738e+06 | 7.307208e+06 | 4.420574e+06 | 5.372216e+05 | 1.068144e+07 | 5.728136e+05 | 9.069515e+05 | 5.449044e+07 | 8.836484e+06 | 4.634899e+07 |
2 rows × 48 columns
As all data in pymrio is stored as pandas DataFrame, the full pandas stack for exporting tables is available. For example, to export a table as excel sheet use:
In [12]:
io.emissions.D_cba.to_excel('/tmp/testmrio/emission_footprints.xlsx')
For further information see the pandas documentation on import/export.