{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Extending MUSE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One key feature of the generalized sector's implementation is that it should be easy to\n", "extend. As such, MUSE can be made to run custom python functions,\n", "as long as these inputs and output of the function follow a standard specific to each\n", "step. We will look at a few here. \n", "\n", "Below is a list of possible hooks, referenced by their\n", "implementation in the MUSE model:\n", "\n", "- `register_interaction_net` in `muse.interactions`: a list of lists of agents\n", " that interact together.\n", "- `register_agent_interaction` in `muse.interactions`: Given a list of\n", " interacting agents, perform the interaction.\n", "- `register_production` in `muse.production`: A method to compute the production\n", " from a sector, given the demand and the capacity.\n", "- `register_initial_asset_transform` in `muse.hooks`: Allows any kind of transformation to be applied to the assets of an agent, prior to investing.\n", "- `register_final_asset_transform` in `muse.hooks`: After computing the investment, this sets the assets that will be owned by the agents.\n", "- `register_demand_share` in `muse.demand_share`: During agent investment, this is the share\n", " of the demand that an agent will try and satisfy.\n", "- `register_filter` in `muse.filters`: A filter to remove technologies from consideration, during agent investment. \n", "- `register_objective` in `muse.objectives`: A quantity which allows an agent to compare technologies during investment.\n", "- `register_decision` in `muse.decisions`: A transformation applied to aggregate multiple objectives into a single objective during agent investment, e.g. via a weighted sum.\n", "- `register_investment` in `muse.investment`: During agent investment, matches\n", " the demand for future investment using the decision metric above.\n", "- `register_output_quantity` in `muse.output.sector`: A sectorial quantity to output for\n", " postmortem analysis.\n", "- `register_output_sink` in `muse.outputs`: A _place_ to store an output\n", " quantity, e.g. a file with a given format, a database on premise or on the cloud,\n", " etc...\n", "- `register_cached_quantity` in `muse.outputs.cache`: A global quantity to output for\n", " postmortem analysis.\n", "- `register_carbon_budget_fitter` in `muse.carbon_budget`\n", "- `register_carbon_budget_method` in `muse.carbon_budget`\n", "- `register_sector`: Registers a function that can create a sector from a muse\n", " configuration object." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extending outputs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "MUSE can be used to save custom quantities as well as data for analysis. There are two steps to this process:\n", " \n", "- Computing the quantity of interest\n", " \n", "- Store the quantity of interest in a sink\n", "\n", "In practice, this means that we can compute any quantity, such as capacity or\n", "consumption of an energy source and save it to a csv file, or a netcdf file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Output extension\n", "\n", "To demonstrate this, we will compute a new edited quantity of consumption, then save it as a text file.\n", "\n", "The current implementation of the quantity of consumption found in `muse.outputs.sector` filters out values of 0. In this example, we would like to maintain the values of 0, but do not want to edit the source code of MUSE.\n", "\n", "This is rather simple to do using MUSE's hooks.\n", "\n", "First we create a new function called `consumption_zero` as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from typing import List, Optional, Text\n", "\n", "from muse.outputs.sector import market_quantity, register_output_quantity\n", "from xarray import DataArray, Dataset\n", "\n", "\n", "@register_output_quantity\n", "def consumption_zero(\n", " market: Dataset,\n", " capacity: DataArray,\n", " technologies: Dataset,\n", "):\n", " \"\"\"Current consumption.\"\"\"\n", " result = (\n", " market_quantity(market.consumption, sum_over=\"timeslice\", drop=None)\n", " .rename(\"consumption\")\n", " .to_dataframe()\n", " .round(4)\n", " )\n", " return result" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The function we created takes three arguments. These arguments (`market`, `capacity` and `technology`) are mandatory for the `@register_output_quantity` hook. Other hooks require different arguments. \n", "\n", "Whilst this function is very similar to the `consumption` function in `muse.outputs.sector`, we have modified it slightly by allowing for values of `0`.\n", "\n", "The important part of this function is the `@register_output_quantity` decorator. This decorator ensures that this new quantity is addressable in the TOML file. Notice that we did not need to edit the source code to create our new function.\n", "\n", "Next, we can create a sink to save the output quantity previously registered. For this example, this sink will simply dump the quantity it is given to a file, with the \"Hello world!\" message:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from typing import Any\n", "\n", "from muse.outputs.sinks import register_output_sink, sink_to_file\n", "\n", "\n", "@register_output_sink(name=\"txt\")\n", "@sink_to_file(\".txt\")\n", "def text_dump(data: Any, filename: Text) -> None:\n", " from pathlib import Path\n", "\n", " Path(filename).write_text(f\"Hello world!\\n\\n{data}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The code above makes use of two dectorators: `@register_output_sink` and `@sink_to_file`. \n", "\n", "`@register_output_sink` registers the function with MUSE, so that the sink is addressable from a TOML file. The second one, `@sink_to_file`, is optional. This adds some nice-to-have features to sinks that are files. For example, a way to specify filenames and check that files cannot be overwritten, unless explicitly allowed to.\n", "\n", "Next, we want to modify the TOML file to actually use this output type. To do this, we add a section to the output table:\n", "\n", "```toml\n", "[[sectors.residential.outputs]]\n", "quantity = \"consumption_zero\"\n", "sink = \"txt\"\n", "filename = \"{cwd}/{default_output_dir}/{Sector}{Quantity}{year}{suffix}\"\n", "```\n", "\n", "The last line above allows us to specify the name of the file. We could also use `sector` above or `quantity`.\n", "\n", "There can be as many sections of this kind as we like in the TOML file, which allow for multiple outputs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we first copy the default model provided with muse to a local subfolder called \"model\". Then we read the `settings.toml` file and modify it using python. You may prefer to modify the `settings.toml` file using your favorite text editor. However, modifying the file programmatically allows us to\n", "routinely run this notebook as part of MUSE's test suite and check that the tutorial it is still up\n", "to date." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "from muse import examples\n", "from toml import dump, load\n", "\n", "model_path = examples.copy_model(overwrite=True)\n", "settings = load(model_path / \"settings.toml\")\n", "new_output = {\n", " \"quantity\": \"consumption_zero\",\n", " \"sink\": \"txt\",\n", " \"overwrite\": True,\n", " \"filename\": \"{cwd}/{default_output_dir}/{Sector}{Quantity}{year}{suffix}\",\n", "}\n", "settings[\"sectors\"][\"residential\"][\"outputs\"].append(new_output)\n", "dump(settings, (model_path / \"modified_settings.toml\").open(\"w\"))\n", "settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now run the simulation. There are two ways to do this. From the command-line, where we can do:\n", "\n", " python3 -m muse data/commercial/modified_settings.toml \n", "\n", "(note that slashes may be the other way on Windows). Or directly from the notebook:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import logging\n", "\n", "from muse.mca import MCA\n", "\n", "logging.getLogger(\"muse\").setLevel(0)\n", "mca = MCA.factory(model_path / \"modified_settings.toml\")\n", "mca.run();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now check that the simulation has created the files that we expect. We also check that our \"Hello, world!\" message has printed:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "all_txt_files = sorted((Path() / \"Results\").glob(\"Residential*.txt\"))\n", "assert \"Hello world!\" in all_txt_files[0].read_text()\n", "all_txt_files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Our model output the files we were expecting and passed the `assert` statement, meaning that it could find the \"Hello world!\" messages in the outputs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Cached quantities\n", "\n", "The result of intermediate calculations are often useful for post-mortem analysis or\n", "simply to have a more detailed picture of the evolution of the calculation over time.\n", "The process of adding a new quantity to cache and output has three steps:\n", "\n", "1. Register the function with `register_cached_quantity` that will deal with the \n", " consolidation of the cached quantity prior to outputting in such a way it can be\n", " accepted by one of the sinks. It can also be used to modify what is saved, filtering\n", " by technologies or agents, for example.\n", "2. Cache the quantity in each iteration of the market using\n", " `muse.outputs.cache.cache_quantity` in the relevant part of your code.\n", "3. Indicate in the TOML file that you want to save that quantity, and where.\n", "\n", "The last point is identical to requesting a sector quantity to be saved, already\n", "described in the previous section, but with information placed in the global section of\n", "the TOML file rather than within a sector.\n", "\n", "All functions registered with `register_investment` or `register_objective` are\n", "automatically cached, i.e. `cache_quantity` is called within the hook taking as input\n", "the output of the investment or objective function. In particular, investment functions\n", "calculate both the `capacity` and the `production` after investment has been made. There\n", "is already registered a function that will deal with the cached `capacity` but here we\n", "we are going to register an alternative that will cache only the `capacity` related to\n", "assets present in `retro` agents:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from typing import MutableMapping, Text\n", "\n", "import pandas as pd\n", "import xarray as xr\n", "from muse.outputs.cache import consolidate_quantity, register_cached_quantity\n", "\n", "\n", "@register_cached_quantity(overwrite=True)\n", "def capacity(\n", " cached: List[xr.DataArray],\n", " agents: MutableMapping[Text, MutableMapping[Text, Text]],\n", ") -> pd.DataFrame:\n", " \"\"\"Consolidates the cached capacity into a single DataFrame to save.\n", "\n", " Args:\n", " cached (List[xr.DataArray]): The list of cached arrays during the calculation of\n", " the time period with the capacity.\n", " agents (MutableMapping[Text, MutableMapping[Text, Text]]): Agents' metadata.\n", "\n", " Returns:\n", " pd.DataFrame: DataFrame with the consolidated data for retro agents.\n", " \"\"\"\n", " consolidated = consolidate_quantity(\"capacity\", cached, agents)\n", " return consolidated.query(\"category == 'retrofit'\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above function is nearly identical to `muse.outputs.cache.capacity` but filtering\n", "the output such that only information related to retorfit agents is included in the\n", "output. As a function with the same name intended to cache the `capacity` already\n", "exists, we have to set `overwrite = True` in the decorator, so that it replaces the\n", "built in version.\n", "\n", "The `consolidate_quantity` function is a convenient tool to extract the last records\n", "from the list of cached DataArrays and put it together with the agent's metadata in a\n", "DataFrame, but you can code your own solution to put together an output that the chosen\n", "sink can digest." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Next, we need to indicate in the TOML file that we want to cache that quantity. To do\n", "that, we write the following in the global section:\n", "\n", "```toml\n", "[[outputs_cache]]\n", "quantity = \"capacity\"\n", "sink = \"aggregate\"\n", "filename = \"{cwd}/{default_output_dir}/Cache{Quantity}.csv\"\n", "index = false\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `aggregate` sink already exist, so we do not need to create it. If you want to\n", "customise further how to save the data, create your own as described above.\n", "\n", "The next steps are similar to those already described: create a modified settings file,\n", "run the simulation and check that the output we have created indeed is what we wanted." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "from muse import examples\n", "from toml import dump, load\n", "\n", "model_path = examples.copy_model(overwrite=True)\n", "settings = load(model_path / \"settings.toml\")\n", "new_output = {\n", " \"quantity\": \"capacity\",\n", " \"sink\": \"aggregate\",\n", " \"index\": False,\n", " \"filename\": \"{cwd}/{default_output_dir}/Cache{Quantity}.csv\",\n", "}\n", "settings[\"outputs_cache\"] = []\n", "settings[\"outputs_cache\"].append(new_output)\n", "dump(settings, (model_path / \"modified_settings.toml\").open(\"w\"))\n", "settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now run the simulation. There are two ways to do this. From the command-line, where we can do:\n", "\n", " python3 -m muse data/commercial/modified_settings.toml \n", "\n", "(note that slashes may be the other way on Windows). Or directly from the notebook:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import logging\n", "\n", "from muse.mca import MCA\n", "\n", "logging.getLogger(\"muse\").setLevel(0)\n", "mca = MCA.factory(model_path / \"modified_settings.toml\")\n", "mca.run();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now check that the simulation has created the file that we expect: the cached\n", "capacity only for retrofit agents:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cache_files = sorted((Path() / \"Results\").glob(\"Cache*\"))\n", "assert len(cache_files) == 1\n", "cached = pd.read_csv(cache_files[0])\n", "assert tuple(cached.category.unique()) == (\"retrofit\",)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding TOML parameters to the outputs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It would be useful if we could pass parameters from the TOML file to our new functions `consumption_zero` and `text_dump`. For example, in our previous iteration the consumption output was aggregating the data by `\"timeslice\"`, by hardcoding the variable. We can pass a parameter which could do this by setting the `sum_over` parameter to be `True`. In addition, we could change the message output by a new `text_dump` function.\n", "\n", "Not all hooks are this flexible (for historical reasons, rather than any intrinsic difficulty). However, for outputs, we can do this as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "@register_output_quantity(overwrite=True)\n", "def consumption_zero( # noqa: F811\n", " market: Dataset,\n", " capacity: DataArray,\n", " technologies: Dataset,\n", " sum_over: Optional[List[Text]] = None,\n", " drop: Optional[List[Text]] = None,\n", " rounding: int = 4,\n", "):\n", " \"\"\"Current consumption.\"\"\"\n", " result = (\n", " market_quantity(market.consumption, sum_over=sum_over, drop=drop)\n", " .rename(\"consumption\")\n", " .to_dataframe()\n", " .round(rounding)\n", " )\n", " return result\n", "\n", "\n", "@register_output_sink(name=\"txt\", overwrite=True)\n", "@sink_to_file(\".txt\")\n", "def text_dump(data: Any, filename: Text, msg: Optional[Text] = \"Hello, world!\") -> None: # noqa: F811\n", " from pathlib import Path\n", "\n", " Path(filename).write_text(f\"{msg}\\n\\n{data}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We simply added parameters as arguments to both of our functions: `consumption_zero` and `text_dump`.\n", "\n", "Note: The overwrite argument allows us to overwrite previously defined registered functions. This is useful in a notebook such as this. But it should not be used in general. If overwrite were false, then the code would issue a warning and it would leave the TOML to refer to the original functions at the beginning of the notebook. This is useful when using custom modules.\n", "\n", "Now we can modify the output section to take additional arguments:\n", "\n", " [[sectors.commercial.outputs]]\n", " quantity.name = \"consumption_zero\"\n", " quantity.sum_over = \"timeslice\"\n", " sink.name = \"txt\"\n", " sink.filename = \"{cwd}/{default_output_dir}/{Sector}{Quantity}{year}{suffix}\"\n", " sink.msg = \"Hello, you!\"\n", " sink.overwrite = True\n", " \n", "Here, we still want to use the `consumption_zero` function and the `txt` sink. But we would like to change the message from \"Hello world!\" to \"Hello you!\" within the `TOML` file.\n", " \n", "Now, both sink and quantity are dictionaries which can take any number of arguments. Previously, we were using a shorthand for convenience. Again, we create a new settings file, and run this with our new parameters, which interface with our new functions." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "from muse import examples\n", "from toml import dump, load\n", "\n", "model_path = examples.copy_model(overwrite=True)\n", "settings = load(model_path / \"settings.toml\")\n", "settings[\"sectors\"][\"residential\"][\"outputs\"] = [\n", " {\n", " \"quantity\": {\"name\": \"consumption_zero\", \"sum_over\": \"timeslice\"},\n", " \"sink\": {\n", " \"name\": \"txt\",\n", " \"filename\": \"{cwd}/{default_output_dir}/{Sector}{Quantity}{year}{suffix}\",\n", " \"msg\": \"Hello, you!\",\n", " \"overwrite\": True,\n", " },\n", " }\n", "]\n", "\n", "dump(settings, (model_path / \"modified_settings_2.toml\").open(\"w\"))\n", "settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We then run the simulation again:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "mca = MCA.factory(model_path / \"modified_settings_2.toml\")\n", "mca.run();" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And we can check the parameters were used accordingly:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "all_txt_files = sorted((Path() / \"Results\").glob(\"Residential*.txt\"))\n", "assert len(all_txt_files) == 7\n", "assert \"Hello, you!\" in all_txt_files[0].read_text()\n", "all_txt_files" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, we can see that the number of output files generated were as we expected and that our new message \"Hello, you!\" was found within these files. This means that our output and sink functions worked as expected." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Where to store new functionality" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As previously demonstrated, we can easily add new functionality to MUSE. However,\n", "running a jupyter notebook is not always the best approach. It is also possible to store\n", "functions in an arbitrary Python file, such as the following:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile mynewfunctions.py\n", "from typing import Any, Text\n", "\n", "from muse.outputs.sinks import register_output_sink, sink_to_file\n", "\n", "\n", "@register_output_sink(name=\"txt\")\n", "@sink_to_file(\".txt\")\n", "def text_dump(data: Any, filename: Text) -> None:\n", " from pathlib import Path\n", " Path(filename).write_text(f\"Hello world!\\n\\n{data}\")\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can then tell the TOML file where to find it:\n", "\n", "```python\n", "plugins = \"{cwd}/mynewfunctions.py\"\n", "\n", "[[sectors.commercial.outputs]]\n", "quantity = \"capacity\"\n", "sink = \"dummy\"\n", "overwrite = true\n", "```\n", "\n", "Alternatively, `plugin` can also be given a list of paths rather than just a single one, as done below." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "settings = load(model_path / \"settings.toml\")\n", "settings[\"plugins\"] = [\"{cwd}/mynewfunctions.py\"]\n", "settings[\"sectors\"][\"residential\"][\"outputs\"] = [\n", " {\"quantity\": \"capacity\", \"sink\": \"dummy\", \"overwrite\": \"true\"}\n", "]\n", "dump(settings, (model_path / \"modified_settings.toml\").open(\"w\"))\n", "settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Next steps" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the next section we will output a technology filter, to stop agents from investing in a certain technology, and a new metric to combine multiple objectives." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.8.2 ('.venv': venv)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.2" }, "vscode": { "interpreter": { "hash": "bef583514e26a7e735428d308331ee3372c412f577c9d62d7c3154d972034ba3" } } }, "nbformat": 4, "nbformat_minor": 4 }