8.2. Simulation settings TOML file

This section details the TOML input for MUSE. The format for TOML files is described in this previous section. Here, however, we focus on sections and attributes that are specific to MUSE.

The TOML file can be read using read_settings(). The resulting data is used to construct the market clearing algorithm directly in the MCA's factory function.

8.2.1. Main section

This is the topmost section. It contains settings relevant to the simulation as a whole.

time_framework = [2020, 2025, 2030, 2035, 2040, 2045, 2050]
regions = ["USA"]
interpolation_mode = 'linear'
log_level = 'info'
currency = 'USD'

equilibrium_variable = 'demand'
maximum_iterations = 100
tolerance = 0.1
tolerance_unmet_demand = -0.1

time_framework

Required. List of years for which the simulation will run.

regions

List of regions in the model.

interpolation_mode (optional, default = linear)

interpolation when reading the initial market. Options are:

linear interpolates using a linear function.
nearest uses nearest-neighbour interpolation. That is, the closest known data point is used as the prediction for the data point to interpolate.
zero assumes that the data point to interpolate is zero.
slinear uses a spline of order 1. This is a similar method to linear interpolation.
quadratic uses a quadratic equation for interpolation. This should be used if you expect your data to take a quadratic form.
cubic interpolation is similar to quadratic interpolation, but uses a cubic function for interpolation.

log_level (optional, default = info)

verbosity of the output. Valid options, from the highest to the lowest level of verbosity, are: debug, info, warning, error, critical.

currency (optional)

The currency used for prices (e.g. “USD”, “EUR”, “MUS$2010”). All prices in all input files should be expressed in this currency. For example, if the currency is “USD”, then capital costs should be expressed as “USD” per capacity unit, and price projections for a commodity with units “PJ” should be expressed as “USD/PJ”. No currency conversions are performed in the model, and the choice of currency has no impact on the simulation - it is purely for documentation purposes.

equilibirum_variable (optional, default = demand)

whether equilibrium of demand or prices should be sought.

maximum_iterations (optional, default = 100)

Maximum number of iterations when searching for equilibrium.

tolerance (optional, default = 0.1)

Tolerance criteria when checking for equilibrium. e.g. 0.1 signifies that 10% of a deviation is allowed among the iterative value of either demand or price over a year per region.

tolerance_unmet_demand (optional, default = -0.1)

Criteria checking whether the demand has been met.

excluded_commodities (optional)

List of commodities excluded from the equilibrium considerations.

plugins (optional)

Path or list of paths to extra python plugins, i.e. files with registered functions such as register_output_quantity().

8.2.2. Global input files

Defines the paths specific simulation data files. The paths can be formatted as explained in the TOML primer.

[global_input_files]
projections = '{path}/Projections.csv'
global_commodities = '{path}/GlobalCommodities.csv'

projections: Path to a csv file giving initial market projection. See Commodity Price Projections.
global_commodities: Path to a csv file describing the comodities in the simulation. See Global Commodities.

8.2.3. Timeslices

Time-slices represent a sub-year disaggregation of commodity demand. Generally, timeslices are expected to introduce several levels, e.g. season, day, or hour. For example:

[timeslices]
level_names = ["month", "day", "hour"]
winter.weekday.night = 396
winter.weekday.morning = 396
winter.weekday.afternoon = 264
winter.weekday.early-peak = 66
winter.weekday.late-peak = 66
winter.weekday.evening = 396
winter.weekend.night = 156
winter.weekend.morning = 156
winter.weekend.afternoon = 156
winter.weekend.evening = 156
spring-autumn.weekday.night = 792
spring-autumn.weekday.morning = 792
spring-autumn.weekday.afternoon = 528
spring-autumn.weekday.early-peak = 132
spring-autumn.weekday.late-peak = 132
spring-autumn.weekday.evening = 792
spring-autumn.weekend.night = 300
spring-autumn.weekend.morning = 300
spring-autumn.weekend.afternoon = 300
spring-autumn.weekend.evening = 300
summer.weekday.night = 396
summer.weekday.morning  = 396
summer.weekday.afternoon = 264
summer.weekday.early-peak = 66
summer.weekday.late-peak = 66
summer.weekday.evening = 396
summer.weekend.night = 150
summer.weekend.morning = 150
summer.weekend.afternoon = 150
summer.weekend.evening = 150

This input introduces three levels, via level_names: month, day, hour. Other simulations may want fewer or more levels. The month level is split into three points of data, winter, spring-autumn, summer. Then day splits out weekdays from weekends, and so on. Each line indicates the number of hours for the relevant slice. It should be noted that the slices are not a cartesian products of each levels. For instance, there no peak periods during weekends. All that matters is that the relative weights (i.e. the number of hours) are consistent and sum up to a year.

8.2.4. Standard sectors

A MUSE model requires at least one sector. Sectors are declared in the TOML file by adding a subsection to the sectors section:

[sectors.residential]
type = 'default'
[sectors.power]
type = 'default'

Above, we’ve added two sectors, residential and power. The name of the subsection is only used for identification. In other words, it should be chosen to be meaningful to the user, since it will not affect the model itself.

A sector accepts these attributes:

type

Defines the kind of sector this is. There are two options:

default: defines a standard sector
presets: defines a preset sector (see below)

priority

An integer denoting which sectors runs when. Lower values imply the sector will run earlier. Later sectors can depend on earlier sectors for the their input. If two sectors share the same priority, then their order is not defined. Indeed, it should indicate that they can run in parallel. For simplicity, the keyword also accepts standard values:

preset: 0
demand: 10
conversion: 20
supply: 30
last: 100

Defaults to last.

interpolation (optional, default = linear)

Interpolation method used to fill missing years in the technodata. Available interpolation methods depend on the underlying scipy method’s kind attribute. Years outside the data range will always be back/forward filled with the closest available data.

dispatch_production (optional, default = share)

The method used to calculate supply of commodities after investments have been made.

MUSE provides three methods in muse.dispatch:

share: assets each supply a proportion of demand based on their share of total capacity. This method is most appropriate for sectors where demand is inherently distributed across assets and cannot be shifted between them (e.g. residential heating systems, where assets in each household meets their own share of the demand). See muse.dispatch.share_based_production().
merit-order: supply in each timeslice is obtained by dispatching the cheapest assets first until the demand is met. This method is best suited to sectors where supply is pooled across assets and short-run marginal cost is the dominant driver of dispatch decisions (e.g. electricity production). See muse.dispatch.merit_order_production().
maximum: the production is the maximum production for the existing capacity and the technology’s utilization factor. This is largely intended for testing purposes. See muse.dispatch.maximum_production().

Additional methods can be registered with muse.dispatch.register_production()

technodata

Path to a csv file containing the characterization of the technologies involved in the sector, e.g. lifetime, capital costs, etc.. See Technodata.

technodata_timeslices (optional)

Path to a csv file describing the utilization factor and minimum service factor of each technology in each timeslice. See Technodata Timeslices.

commodities_in

Path to a csv file describing the inputs of each technology involved in the sector. See Commodity Inputs/Outputs.

commodities_out

Path to a csv file describing the outputs of each technology involved in the sector. See Commodity Inputs/Outputs.

timeslice_level (optional)

This represents the level of timeslice granularity over which commodity flows out of the sector are balanced (e.g. if “day”, the sector will aim to meet commodity demands on a daily basis, rather than an hourly basis). If not given, defaults to the finest level defined in the global timeslices section. Note: If technodata_timeslices is used, the data in this file must match the timeslice level of the sector (e.g. with global timeslice levels “month”, “day” and “hour”, if a sector has “day” as the timeslice level, then technodata_timeslices must have columns “month” and “day”, but not “hour”)

Sectors contain a number of subsections:

subsectors

Subsectors group together agents into separate groups servicing the demand for different commodities. There must be at least one subsector, and there can be as many as required. For instance, a one-subsector setup would look like:

[sectors.gas.subsectors.all]
agents = '{path}/gas/Agents.csv'
existing_capacity = '{path}/gas/ExistingCapacity.csv'

A two-subsector could look like:

[sectors.gas.subsectors.methane_and_ethanol]
agents = '{path}/gas/me_agents.csv'
existing_capacity = '{path}/gas/me_existing.csv'
commodities = ["methane", "ethanol"]

[sectors.gas.subsectors.natural]
agents = '{path}/gas/nat_agents.csv'
existing_capacity = '{path}/gas/nat_existing.csv'
commodities = ["refined", "crude"]

In the case of multiple subsectors, it is important to specify disjoint sets of commodities so that each subsector can service a separate demand. The subsectors accept the following keywords:

agents

Path to a csv file describing the agents in the sector. See Agents.

existing_capacity

Path to a csv file describing the initial capacity of the sector. See Existing Capacity.

lpsolver (optional, default = scipy)

The solver for linear problems to use when figuring out investments. The solvers are registered via register_investment(). At time of writing, three are available:

scipy solver (default from v1.3): Formulates investment as a true LP problem and solves it using the scipy solver.
adhoc solver: Simple in-house solver that ranks the technologies according to cost and service the demand incrementally.

demand_share (optional, default = standard_demand)

A method used to split the MCA demand into separate parts to be serviced by specific agents. The appropriate choice depends on the type of agents being used in the simulation. There are currently two options:

standard_demand() (default): The input demand is split amongst new agents. New agents get a share of the increase in demand over the investment period, as well as the demand that occurs from decommissioned assets.
new_and_retro(): The input demand is split amongst both new and retrofit agents. New agents get a share of the increase in demand over the investment period, whereas retrofit agents are assigned a share of the demand that occurs from decommissioned assets.

constraints (optional, defaults to full list)

The list of constraints to apply to the LP problem solved by the sector. By default all of the following are included:

demand(): a lower-bound of the production decision variables specifying the target demand.
max_production(): an upper bound limiting how much can be produced for a given capacity.
max_capacity_expansion(): an upper bound limiting how much the capacity can grow during each investment event.
search_space(): a binary (on-off) constraint specifying which technologies are considered for investment.
minimum_service(): a lower constraint for production for those technologies that need to keep a minimum production.
demand_limiting_capacity(): limits the combined capacity to be installed to the demand of the peak timeslice.

output

Outputs are made up of several components. MUSE is designed to allow users to mix-and-match both how and what to save.

output is specified as a TOML array, e.g. with double brackets. Each sector can specify an arbitrary number of outputs, simply by adding an extra output row.

A single row looks like this:

[[sectors.commercial.outputs]]
filename = '{cwd}/Results/{Sector}/{Quantity}/{year}{suffix}'
quantity = "capacity"
sink = 'csv'
overwrite = true

The following attributes are available:

quantity

Name of the quantity to save. The options are capacity, consumption, supply and costs. Users can also customize and create further output quantities by registering with MUSE via muse.outputs.mca.register_output_quantity(). See muse.outputs.sector for more details.

sink

the sink is the place (disk, cloud, database, etc…) and format with which the computed quantity is saved. Currently only sinks that save to files are implemented. The following sinks are available: “csv”, “netcfd”, “excel” and “aggregate”. Additional sinks can be added by interested users, and registered with MUSE via muse.outputs.sinks.register_output_sink(). See muse.outputs.sinks for more details.

filename

defines the format of the file where to save the data. There are several standard values that are automatically substituted:

cwd: current working directory, where MUSE was started
path: directory where the TOML file resides
sector: name of the current sector (.e.g. “commercial” above)
Sector: capitalized name of the current sector
quantity: name of the quantity to save (as given by the quantity attribute)
Quantity: capitablized name of the quantity to save
year: current year
suffix: standard suffix/file extension of the sink

Defaults to {cwd}/{default_output_dir}/{Sector}/{Quantity}/{year}{suffix}.

overwrite (optional, default = False)

If False MUSE will issue an error and abort, instead of overwriting an existing file. This prevents important output files from being overwritten.

Additional sink parameters

You can pass additional parameters that will be forwarded to the underlying save function. For example, when using the “csv” sink, you could use float_format = “%.6f” to increase the precision of floating point numbers in the output file (default is 4 decimal places). For a complete list of available parameters, see the documentation for the respective save function (e.g., pandas.to_csv for CSV outputs).

For example, the following would save supply data for the commercial sector as a separate file for each year:

[[sectors.commercial.outputs]]
quantity = "supply"
sink = "csv"
filename = "{cwd}/{default_output_dir}/{Sector}/{Quantity}/{year}{suffix}"
overwrite = true

There is a special output sink for aggregating over years (i.e. a single output file for all years). It can be invoked as follows:

[[sectors.commercial.outputs]]
quantity = "supply"
sink = "aggregate"
filename = "{cwd}/{default_output_dir}/{Sector}/{Quantity}.csv"

Note that the aggregate sink always overwrites the final file, since it will overwrite itself.

interactions (optional)

Defines interactions between agents. These interactions take place right before new investments are computed. The interactions can be anything. They are expected to modify the agents and their assets. MUSE provides a default set of interactions that have new agents pass on their assets to the corresponding retro agent, and the retro agents pass on the make-up of their assets to the corresponding new agents.

interactions are specified as a TOML array, e.g. with double brackets. Each sector can specify an arbitrary number of interactaction, simply by adding an extra interaction row.

There are two orthogonal concepts to interactions:

a net defines the set of agents that interact. A set can contain any number of agents, whether zero, two, or all agents in a sector. See muse.interactions.register_interaction_net().
an interaction defines how the net actually interacts. See muse.interactions.register_agent_interaction().

In practice, we always consider sequences of nets (i.e. more than one net) that interact using the same interaction function.

Hence, the input looks something like the following:

[[sectors.commercial.interactions]]
net = 'new_to_retro'
interaction = 'transfer'

new_to_retro is a function that figures out all “new/retro” pairs of agents. Whereas transfer is a function that performs the transfer of assets and information between each pair.

Furthermore, it is possible to pass parameters to either the net of the interaction as follows:

[[sectors.commercial.interactions]]
net = { name = "some_net", param = "some value" }
interaction = { name = "some_interaction", param = "some other value" }

The parameters will depend on the net and interaction functions. Neither “new_to_retro” nor “transfer” take any arguments at this point. MUSE interaction facilities are defined in muse.interactions.

An error is raised if and empty network is found. This is the case, for example, if a “new_to_retro” type of network has been defined but no retro agents are included in the sector.

8.2.5. Preset sectors

The commodity production, commodity consumption and product prices of preset sectors are determined exogeneously. They are known from the start of the simulation and are not affected by the simulation.

A common example would be the following, where commodity consumption is defined exogeneously (see consumption_path):

[sectors.commercial_presets]
type = 'presets'
priority = 0
consumption_path = "{path}/commercial_presets/*Consumption.csv"

Alternatively, you may define consumption as a function of macro-economic data, i.e. population and GDP (see Correlation Demand Files):

[sectors.commercial_presets]
type = 'presets'
priority = 0
timeslice_shares_path = '{path}/commercial_presets/TimesliceShareCommercial.csv'
macrodrivers_path = '{path}/commercial_presets/Macrodrivers.csv'
regression_path = '{path}/commercial_presets/regressionparameters.csv'

The following attributes are accepted:

type (required): See the attribute in the standard mode, type. Preset sectors are those with type “presets”.
priority (required): See the attribute in the standard mode, priority.

consumption_path

CSV files, one per year. This attribute can include wild cards, i.e. ‘*’, which can match anything. For instance: consumption_path = “{cwd}/Consumption*.csv” will match any csv file starting with “Consumption” in the current working directory. The file names must include the year for which it defines the consumption, e.g. Consumption2015.csv.

The CSV format should follow the format described in the Preset commodity demands document.

supply_path

CSV file, one per year, indicating the amount of commodities produced. It follows the same format as consumption_path.

demand_path: Incompatible with consumption_path or macrodrivers_path. A CSV file containing the consumption in the same format as Commodity Price Projections.

macrodrivers_path: Incompatible with consumption_path or demand_path. Path to a CSV file giving the profile of the macrodrivers. Also requires regression_path.

regression_path: Incompatible with consumption_path. Path to a CSV file giving the regression parameters with respect to the macrodrivers. Also requires macrodrivers_path.
timeslice_shares_path: Incompatible with consumption_path or demand_path. Optional csv file giving shares per timeslice. The timeslice share definition needs to have a consistent number of timeslices as the sectoral level time slices. Requires macrodrivers_path.

8.2.5.1. Carbon market (optional)

This section contains the settings related to the modelling of the carbon market. If omitted, it defaults to not including the carbon market in the simulation.

For example

[carbon_budget_control]
budget = [1000, 800, 600, 400, 200, 0]
commodities = ["CO2"]

budget

Yearly budget. There should be one item for each year the simulation will run. In other words, if given and not empty, this is a list with the same length as time_framework from the main section. If not given or an empty list, then the carbon market feature is disabled. Defaults to an empty list.

commodities

Commodities that make up the carbon market.

control_undershoot (optional, default = False)

Whether to control carbon budget undershoots. This parameter allows for carbon tax credit from one year to be passed to the next in the case of less carbon being emitted than the budget.

control_overshoot (optional, default = False)

Whether to control carbon budget overshoots. If the amount of carbon emitted is above the carbon budget, this parameter specifies whether this deficit is carried over to the next year.

method (optional, default = bisection)

Method used to equilibrate the carbon market. Available options are fitting and bisection, however this can be expanded with the @register_carbon_budget_method hook in muse.carbon_budget.

These methods solve the market with a number of different carbon prices, aiming to find the carbon price at which emissions (pooled across all regions) are equal to the carbon budget. The obtained carbon price applies to all regions.

The fitting method samples a number of different carbon prices to build a regression model (linear or exponential) of emissions as a function of carbon price. This regression model is then used to estimate the carbon price at which the carbon budget is met.

The bisection method uses an iterative approach to settle on a carbon price. Starting with a lower and upper-bound carbon price, it iteratively halves this price interval until the carbon budget is met to within a user-defined tolerance, or until the maximum number of iterations is reached. Generally, this method is more robust for markets with a complex, nonlinear relationship between emissions and carbon price, but may be slower to converge than the fitting method.

method_options

Additional options for the specified carbon method.

Parameters for the bisection method:

max_iterations (optional, default = 5): maximum number of iterations.
tolerance (optional, default = 0.1): tolerance for convergence. E.g. 0.1 means that the algorithm will terminate when emissions are within 10% of the carbon budget.
early_termination_count (optional, default = 5): number of iterations with no change in the carbon price before the algorithm will terminate.
price_penalty (optional, default = 0.1): penalty factor applied to carbon price when selecting optimal solution when convergence isn’t reached. For example, if the carbon price is measured in units of MUSD/kt, a price penalty of 1000 means that a price increase of 1 MUSD/kt will only be accepted if it reduces emissions by at least 1000 kt.

Parameters for the fitting method:

fitter (optional, default = linear): the regression model used to approximate model emissions. Predefined options are linear (default) and exponential. Further options can be defined using the @register_carbon_budget_fitter hook in muse.carbon_budget.
sample_size (optional, default = 5): number of price samples used.

Shared parameters:

refine_price (optional, default = False): If True, applies an upper limit on the carbon price.
price_too_high_threshold (optional, default = 10): upper limit on the carbon price.
resolution (optional, default = 2): Number of decimal places to solve the carbon price to. When using the bisection method, increasing this value may increase the time taken to solve the carbon market.

8.2.6. Output cache (for advanced users)

outputs_cache

This option behaves exactly like outputs for sectors and accepts the same options but controls the output of cached quantities instead. This option is NOT available for sectors themselves (i.e using [[sector.commercial.outputs_cache]] will have no effect). See muse.outputs.cache for more details.

A single row looks like this:

[[outputs_cache]]
quantity = "production"
sink = "aggregate"
filename = "{cwd}/{default_output_dir}/Cache{Quantity}.csv"