{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Adding a new technology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MUSE is made up of a number of different [input files](../inputs/index.rst). These, however, can be broadly split into two:\n",
    "\n",
    "- [Simulation settings](../inputs/toml.rst)\n",
    "\n",
    "- [Simulation data](../inputs/inputs_csv.rst)\n",
    "\n",
    "Simulation settings specify how a simulation should be run. For example, which sectors to run, for how many years, the benchmark years and what to output. In this context, benchmark years are the years in which the model is solved. In the examples following, we solve for every 5 years, ie. 2020, 2025, 2030...\n",
    "\n",
    "Whereas, simulation data parametrises the technologies involved in the simulation, or the number and kinds of agents.\n",
    "\n",
    "To create a customised case study it is necessary to edit both of these file types.\n",
    "\n",
    "Simulation settings are specified in a TOML file. [TOML](https://github.com/toml-lang/toml) is a simple, extensible and intuitive file format well suited  for specifying small sets of complex data.\n",
    "\n",
    "Simulation data is specified in [CSV](https://en.wikipedia.org/wiki/Comma-separated_values). This is a common format used for larger datasets, and is made up of columns and rows, with a comma used to differentiate between entries. \n",
    "\n",
    "MUSE requires at least the following files to successfully run:\n",
    "\n",
    "* a single [simulation settings TOML file](../inputs/toml.rst) for the simulation as a whole\n",
    "* a file indicating initial market price [projections](../inputs/projections.rst)\n",
    "* a file describing the [commodities in the simulation](../inputs/commodities.rst)\n",
    "* for generalized sectors:\n",
    "   * a file descring the [agents](../inputs/agents.rst)\n",
    "   * a file descring the [technologies](../inputs/technodata.rst)\n",
    "   * a file descring the [input commodities](../inputs/commodities_io.rst) for each technology\n",
    "   * a file descring the [output commodities](../inputs/commodities_io.rst) for each technology\n",
    "   * a file descring the [existing capacity](../inputs/existing_capacity.rst) of a given\n",
    "     sector\n",
    "* for each preset sector:\n",
    "   * a csv file describing consumption for the duration of the simulation\n",
    "   \n",
    "For a full description of these files see the [input files section](../inputs/index.rst). To see how to customise an example, continue on this page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Addition of solar PV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will add solar photovoltaics to the default model seen in the [example page](../running-muse-example.ipynb). To achieve this, we must modify some of the input files shown in the above section. To get the default model and be able to edit the files use:\n",
    "\n",
    "```bash\n",
    "python -m muse --model default --copy PATH/TO/COPY/THE/MODEL/TO\n",
    "```\n",
    "\n",
    " You can modify the files in your favourite spreadsheet editor or text editor such as VSCODE, Excel, Numbers, Notepad or TextEdit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Technodata Input "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within the default folder there is the ```settings.toml``` file, input folder and technodata folder. To add a technology within the power sector, we must open the ```technodata``` folder followed by the ```power``` folder.\n",
    "\n",
    "At this point, we must note that we require consistency in input and output units. For example, if capacity is in PJ, the same basis would be needed for the output files `CommIn.csv` and `CommOut.csv`. In addition, across sectors a commodity needs to maintain the same unit. In these examples, we use the unit petajoule (PJ).\n",
    "\n",
    "Next, we will edit the ```CommIn.csv``` file, which specifies the commodities consumed by solar photovoltaics.\n",
    "\n",
    "The table below shows the original ```CommIn.csv``` version in normal text, and the added column and row in **bold**.\n",
    "\n",
    "\n",
    "|ProcessName|RegionName|Time|Level|electricity|gas|heat|CO2f|wind|**solar**|\n",
    "|-----------|----------|----|-----|-----------|---|----|----|----|----|\n",
    "|Unit|-|Year|-|PJ/PJ|PJ/PJ|PJ/PJ|kt/PJ|PJ/PJ|**PJ/PJ**|\n",
    "|gasCCGT|R1|2020|fixed|0|1.67|0|0|0|**0**|\n",
    "|windturbine|R1|2020|fixed|0|0|0|0|1|**0**|\n",
    "|**solarPV**|**R1**|**2020**|**fixed**|**0**|**0**|**0**|**0**|**0**|**1**|\n",
    "\n",
    "    \n",
    "We must first add a new row at the bottom of the file, to indicate the new solar photovoltaic technology:\n",
    "\n",
    "- we call this technology ```solarPV``` \n",
    "- place it in region ```R1``` \n",
    "- the data in this row is associated to the year 2020\n",
    "- the input type is fixed\n",
    "- solarPV consumes solar\n",
    "\n",
    "As the solar commodity has not been previously defined, we must define it by adding a column, which we will call solar. We fill out the entries in the solar column, ie. that neither `gasCCGT` nor `windturbine` consume solar.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat this process for the file: ```CommOut.csv```. This file specifies the output of the technology. In our case, solar photovoltaics only output ```electricity```. This is unlike ```gasCCGT``` which also outputs ```CO2f```, or carbon dioxide.\n",
    "\n",
    "|ProcessName|RegionName|Time|Level|electricity|gas|heat|CO2f|wind|**solar**|\n",
    "|-|-|-|-|-|-|-|-|-|----|\n",
    "|Unit|-|Year|-|PJ/PJ|PJ/PJ|PJ/PJ|kt/PJ|PJ/PJ|**PJ/PJ**|\n",
    "|gasCCGT|R1|2020|fixed|1|0|0|91.67|0|**0**|\n",
    "|windturbine|R1|2020|fixed|1|0|0|0|0|**0**|\n",
    "|**solarPV**|**R1**|**2020**|**fixed**|**1**|**0**|**0**|**0**|**0**|**0**|\n",
    "\n",
    "Similar to the the ```CommIn.csv```, we create a new row, and add in the solar commodity. We must ensure that we call our new commodity and technologies the same as the previous file for MUSE to successfully run. ie `solar` and `solarPV`.\n",
    "\n",
    "Please note that we use flat forward extension of the values when only one value is defined. For example, in the `CommOut.csv` we only provide data for the year 2020. Therefore for the benchmark years, 2025, 2030, 2035... we assume the data remains unchanged from 2020."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next file to modify is the ```ExistingCapacity.csv``` file. This file details the existing capacity of each technology, per benchmark year. For this example, we will set the existing capacity to be 0. Please note, that the model interpolates between years linearly. \n",
    "\n",
    "|ProcessName|RegionName|Unit|2020|2025|2030|2035|2040|2045|2050|\n",
    "|-|-|-|-|-|-|-|-|-|-|\n",
    "|gasCCGT|R1|PJ/y|1|1|0|0|0|0|0|\n",
    "|windturbine|R1|PJ/y|0|0|0|0|0|0|0|\n",
    "|**solarPV**|**R1**|**PJ/y**|**0**|**0**|**0**|**0**|**0**|**0**|**0**|\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the ```technodata.csv``` contains parametrisation data for the technology, such as the cost, growth constraints, lifetime of the power plant and fuel used. The technodata file is too long for it all to be displayed here, so we will truncate the full version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we will only define the parameters: ```processName```, ```RegionName```, ```Time```, ```Level```,```cap_par```, ```Fuel```,```EndUse```,```Agent2``` and ```Agent1```\n",
    "\n",
    "We shall copy the existing parameters from the ```windturbine``` technology for the remaining parameters that can be seen in the ```technodata.csv``` file for brevity. You can see the full file [here](https://github.com/SGIModel/MUSE_OS/blob/main/docs/tutorial-code/1-add-new-technology/1-introduction/technodata/power/Technodata.csv).\n",
    "\n",
    "Again, flat forward extension is used here. Therefore, as in this example we only provide data for the benchmark year 2020, 2025 and the following benchmark years will keep the same characteristics, e.g. costs, for each benchmark year of the simulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|ProcessName|RegionName|Time|Level|cap_par|cap_exp|…|Fuel|EndUse|Agent2|\n",
    "|-|-|-|-|-|-|-|-|-|-|\n",
    "|Unit|-|Year|-|MUS$2010/PJ_a|-|…|-|-|Retrofit|\n",
    "|gasCCGT|R1|2020|fixed|23.78234399|1|…|gas|electricity|1|\n",
    "|windturbine|R1|2020|fixed|36.30771182|1|…|wind|electricity|1|\n",
    "|**solarPV**|**R1**|**2020**|**fixed**|**30**|**1**|...|**solar**|**electricity**|**1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Global inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, navigate to the ```input``` folder, found at \n",
    "\n",
    "```bash\n",
    "{muse_installation_location}src/muse/data/example/default/input\n",
    "```\n",
    "\n",
    "We must now edit each of the files found here to add the new solar commodity. Due to space constraints we will not display all of the entries contained in the input files. The edited files can be viewed [here](https://github.com/SGIModel/MUSE_OS/tree/main/docs/tutorial-code/1-add-new-technology) however."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```BaseYearExport.csv``` file defines the exports in the base year. For our example we add a column to indicate that there is no export for solar. However, it is important that a column exists for our new commodity.\n",
    "\n",
    "It is noted, however, that the `BaseYearImport.csv` as well as the `BaseYearExport.csv` files are optional files to define exogenous imports and exports; all values are set to zero if they are not used.\n",
    "\n",
    "|RegionName|Attribute|Time|electricity|gas|heat|CO2f|wind|**solar**|\n",
    "|-|-|-|-|-|-|-|-|-|\n",
    "|Unit|-|Year|PJ|PJ|PJ|kt|PJ|**PJ**|\n",
    "|R1|Exports|2010|0|0|0|0|0|**0**|\n",
    "|R1|Exports|2015|0|0|0|0|0|**0**|\n",
    "|...|...|...|...|...|...|...|...|**...**|\n",
    "|R1|Exports|2100|0|0|0|0|0|**0**|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```BaseYearImport.csv``` file defines the imports in the base year. Similarly to ```BaseYearExport.csv```, we add a column for solar in the ```BaseYearImport.csv``` file. Again, we indicate that solar has no imports.\n",
    "\n",
    "|RegionName|Attribute|Time|electricity|gas|heat|CO2f|wind|**solar**|\n",
    "|-|-|-|-|-|-|-|-|-|\n",
    "|Unit|-|Year|PJ|PJ|PJ|kt|PJ|**PJ**|\n",
    "|R1|Imports|2010|0|0|0|0|0|**0**|\n",
    "|R1|Imports|2015|0|0|0|0|0|**0**|\n",
    "|...|...|...|...|...|...|...|...|**...**|\n",
    "|R1|Imports|2100|0|0|0|0|0|**0**|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```GlobalCommodities.csv``` file is the file which defines the commodities. Here we give the commodities a commodity type, CO2 emissions factor and heat rate. For this file, we will add the solar commodity, with zero CO2 emissions factor and a heat rate of 1.\n",
    "\n",
    "|Commodity|CommodityType|CommodityName|CommodityEmissionFactor_CO2|HeatRate|Unit|\n",
    "|-|-|-|-|-|-|\n",
    "|Electricity|Energy|electricity|0|1|PJ|\n",
    "|Gas|Energy|gas|56.1|1|PJ|\n",
    "|Heat|Energy|heat|0|1|PJ|\n",
    "|Wind|Energy|wind|0|1|PJ|\n",
    "|CO2fuelcomsbustion|Environmental|CO2f|0|1|kt|\n",
    "|**Solar**|**Energy**|**solar**|**0**|**1**|**PJ**|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```projections.csv``` file details the initial market prices for the commodities. The market clearing algorithm will update these throughout the simulation, however, an initial estimate is required to start the simulation. As solar energy is free, we will indicate this by adding a final column.\n",
    "\n",
    "Please note that the unit row is not read by MUSE, but used as a reference for the user. The units should be consistent across all input files for MUSE; MUSE does not carry out any unit conversion.\n",
    "\n",
    "|RegionName|Attribute|Time|electricity|gas|heat|CO2f|wind|**solar**|\n",
    "|-|-|-|-|-|-|-|-|-|\n",
    "|Unit|-|Year|MUS\\$2010/PJ|MUS\\$2010/PJ|MUS\\$2010/PJ|MUS\\$2010/kt|MUS\\$2010/PJ|**MUS\\$2010/PJ**|\n",
    "|R1|CommodityPrice|2010|14.81481472|6.6759|100|0|0|**0**|\n",
    "|R1|CommodityPrice|2015|17.89814806|6.914325|100|0.052913851|0|**0**|\n",
    "|...|...|...|...|...|...|...|...|**...**|\n",
    "|R1|CommodityPrice|2100|21.39814806|7.373485819|100|1.871299697|0|**0**|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running our customised simulation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are able to run our simulation, with the new solar power technology.\n",
    "\n",
    "To do this we run the same run command as previously in the anaconda command prompt:\n",
    "\n",
    "```bash\n",
    "python -m muse settings.toml\n",
    "```\n",
    "\n",
    "The output should be similar to the output [here](../example-output.ipynb). However, expect the simulation to take slightly longer to run. This is due to the additional calculations made."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the simulation has run successfully, you should now have a folder in the same location as your `settings.toml` file called `Results`. The next step is to visualise the results using the data analysis library ```pandas```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will import the ```MCACapacity.csv``` file into pandas and print the first 5 lines using the ```head()``` command.\n",
    "\n",
    "Make sure to change the file path of `\"../tutorial-code/1-add-new-technology/introduction/Results/MCACapacity.csv\"` to where the `MCACapacity.csv` is on your computer, otherwise you will receive an error when you import the csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mca_capacity = pd.read_csv(\n",
    "    \"../tutorial-code/1-add-new-technology/1-introduction/Results/MCACapacity.csv\"\n",
    ")\n",
    "mca_capacity.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will only visualise the power sector in this example, as this was the only sector we changed. Therefore, we filter for this sector, sum the capacity for each technology, and then plot a stacked bar chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "power_capacity = (\n",
    "    mca_capacity[mca_capacity.sector == \"power\"]\n",
    "    .groupby([\"technology\", \"year\"])\n",
    "    .sum()\n",
    "    .reset_index()\n",
    ")\n",
    "\n",
    "ax = power_capacity.pivot(index=\"year\", columns=\"technology\", values=\"capacity\").plot(\n",
    "    kind=\"bar\", stacked=True\n",
    ")\n",
    "ax.set_ylabel(\"Capacity (PJ)\")\n",
    "ax.tick_params(axis=\"x\", labelrotation=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now see that there is solarPV in addition to windturbine and gasCCGT, when compared to the example [here](../running-muse-example.ipynb)! That's great and means it worked! \n",
    "\n",
    "The difference in uptake of `solarPV` compared to `windturbine` is due to the fact that ```solarPV``` has a lower ```cap_par``` cost of 30, compared to the ```windturbine```. Meaning that ```solarPV``` outcompetes both ```windturbine``` and ```gasCCGT``` in the electricity market."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Solar Price"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will observe what happens if we increase the price of solar to be more expensive than wind in the year 2020, but then reduce the price of solar in 2040. By doing this, we should observe an increase in wind in the first few benchmark years of the simulation, followed by a transition to solar as we approach the year 2040.\n",
    "\n",
    "To achieve this we have to modify the `Technodata.csv`, `CommIn.csv` and `CommOut.csv` files.\n",
    "\n",
    "First, we will amend the `Technodata.csv` file as follows:\n",
    "\n",
    "|ProcessName|RegionName|Time|Level|cap_par|cap_exp|…|Fuel|EndUse|Agent2|\n",
    "|-|-|-|-|-|-|-|-|-|-|\n",
    "|Unit|-|Year|-|MUS$2010/PJ_a|-|…|-|-|Retrofit|\n",
    "|gasCCGT|R1|2020|fixed|23.78234399|1|…|gas|electricity|1|\n",
    "|**gasCCGT**|**R1**|**2040**|**fixed**|**23.78234399**|**1**|**…**|**gas**|**electricity**|**1**|\n",
    "|windturbine|R1|2020|fixed|36.30771182|1|…|wind|electricity|1|\n",
    "|**windturbine**|**R1**|**2040**|**fixed**|**36.30771182**|**1**|**…**|**wind**|**electricity**|**1**|\n",
    "|solarPV|R1|2020|fixed|**40**|1|...|solar|electricity|1|\n",
    "|**solarPV**|**R1**|**2040**|**fixed**|**30**|**1**|**...**|**solar**|**electricity**|**1**|\n",
    "\n",
    "Notice that we must provide entries for 2040 for the other technologies, `gasCCGT` and `windturbine`. For this example, we will keep these the same as before, copying and pasting the rows. \n",
    "\n",
    "Here, we increase the ```cap_par``` variable by 10 for `solarPV` in the year 2020, to be a total of 40, and then reduce `cap_par` by 10 in 2040, again for `solarPV`. \n",
    "\n",
    "MUSE uses interpolation for the years which are unknown. So in this example, for the benchmark years between 2020 and 2040 (2025, 2030, 2035), MUSE uses interpolated `cap_par` values. The interpolation mode can be set in the `settings.toml` file, and defaults to linear interpolation. This example uses the default setting for interpolation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will modify the `CommIn.csv` file.\n",
    "\n",
    "For this step, we have to provide the input commodities for each technology, in each of the years defined in the `Technodata.csv` file. So, for this example we are required to provide entries for the years 2020 and 2040 for each of the technologies. For now, we won't change the 2040 values compared to the 2020. Therefore, we just need to copy and paste each of the entries for each of the technologies, as shown below:\n",
    "\n",
    "|ProcessName|RegionName|Time|Level|electricity|gas|heat|CO2f|wind|solar|\n",
    "|-|-|-|-|-|-|-|-|-|-|\n",
    "|Unit|-|Year|-|PJ/PJ|PJ/PJ|PJ/PJ|kt/PJ|PJ/PJ|PJ/PJ|\n",
    "|gasCCGT|R1|2020|fixed|0|1.67|0|0|0|0|\n",
    "|**gasCCGT**|**R1**|**2040**|**fixed**|**0**|**1.67**|**0**|**0**|**0**|**0**|\n",
    "|windturbine|R1|2020|fixed|0|0|0|0|1|0|\n",
    "|**windturbine**|**R1**|**2040**|**fixed**|**0**|**0**|**0**|**0**|**1**|**0**|\n",
    "|solarPV|R1|2020|fixed|0|0|0|0|0|1|\n",
    "|**solarPV**|**R1**|**2040**|**fixed**|**0**|**0**|**0**|**0**|**0**|**1**|\n",
    "\n",
    "We must do the same for the `CommOut.csv` file. For the sake of brevity we won't show you this, but the link to the file can be found [here](https://github.com/SGIModel/MUSE_OS/blob/main/docs/tutorial-code/1-add-new-technology/2-scenario/technodata/power/CommOut.csv).\n",
    "\n",
    "We will now rerun the simulation, using the same command as previously and visualise the new results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We must import the new ```MCACapacity.csv``` file again, and then visualise the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mca_capacity = pd.read_csv(\n",
    "    \"../tutorial-code/1-add-new-technology/2-scenario/Results/MCACapacity.csv\"\n",
    ")\n",
    "power_capacity = (\n",
    "    mca_capacity[mca_capacity.sector == \"power\"]\n",
    "    .groupby([\"technology\", \"year\"])\n",
    "    .sum()\n",
    "    .reset_index()\n",
    ")\n",
    "ax = power_capacity.pivot(index=\"year\", columns=\"technology\", values=\"capacity\").plot(\n",
    "    kind=\"bar\", stacked=True\n",
    ")\n",
    "ax.set_ylabel(\"Capacity (PJ)\")\n",
    "ax.tick_params(axis=\"x\", labelrotation=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the results, we can see that `windturbine` increases rapidly between 2025 and 2040. However, between the years 2030 and 2050 `solarPV` increases. This is because of the changing cost of `solarPV` during these years. \n",
    "\n",
    "For the full example with the completed input files see [here](https://github.com/SGIModel/MUSE_OS/tree/main/docs/tutorial-code/1-add-new-technology/2-scenario)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next section we will add a new agent to the simulation. Note, that we will keep the additional `solarPV` technology, as well as the changing costs in 2040."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}