12. API

MUSE model.

12.1. Market Clearing Algorithm

12.1.1. Main MCA

class FindEquilibriumResults(converged, market, sectors)

Result of find equilibrium.

converged

Alias for field number 0

market

Alias for field number 1

sectors

Alias for field number 2

class MCA(sectors, market, outputs=None, outputs_cache=None, time_framework=[2010, 2020, 2030, 2040, 2050, 2060, 2070, 2080, 2090], equilibrium=True, equilibrium_variable='demand', maximum_iterations=100, tolerance=0.1, tolerance_unmet_demand=-0.1, excluded_commodities=None, carbon_budget=None, carbon_commodities=None, debug=False, control_undershoot=False, control_overshoot=False, carbon_method='bisection', method_options=None)

Market Clearing Algorithm.

The market clearing algorithm is the main object implementing the MUSE model. It is responsible for orchestrating how the sectors are run, how they interface with one another, with the general market and the carbon market.

classmethod factory(settings)

Loads MCA from input settings and input files.

Parameters:

settings – namedtuple with the global MUSE input settings.

Returns:

The loaded MCA

find_equilibrium(market)

Specialised version of the find_equilibrium function.

Parameters:

market – Commodities market, with the prices, supply, consumption and demand.

Returns:

A tuple with the updated market (prices, supply, consumption and demand) and sector.

run()

Initiates the calculation, starting with the loop over years.

This method starts the main MUSE loop, going over the years of the simulation. Internally, it runs the carbon budget loop, which updates the carbon prices, if needed, and the equilibrium loop, which tries to reach an equilibrium between prices, demand and supply.

Returns:

None

update_carbon_budget(market, year_idx)

Specialised version of the update_carbon_budget function.

Parameters:

• market – Commodities market, with the prices, supply, consumption and demand.

• year_idx – Index of the year of interest.

Returns:

An updated market with prices, supply, consumption and demand.

update_carbon_price(market)

Calculates the updated carbon price.

Parameters:

market – Market with the prices, supply, consumption and demand.

Returns:

The new carbon price.

class SingleYearIterationResult(market, sectors)

Result of iterating over sectors for a year.

Convenience tuple naming naming the return values from of single_year_iteration().

market

Alias for field number 0

sectors

Alias for field number 1

check_demand_fulfillment(market, tol)

Checks if the supply will fulfill all the demand in the future.

If it does not, it logs a warning.

Parameters:

• market – Commodities market, with the prices, supply, consumption and demand.

• tol – Tolerance for the unmet demand.

Returns:

True if the supply fulfils the demand; False otherwise

check_equilibrium(market, int_market, tolerance, equilibrium_variable, year=None)

Checks if equilibrium has been reached.

This function checks if the difference in either the demand or the prices between iterations if smaller than certain tolerance. If is, then it is assumed that the process has converged.

Parameters:

• market – The market values in this iteration.

• int_market – The market values in the previous iteration.

• tolerance – Tolerance for reaching equilibrium.

• equilibrium_variable – Variable to use to calculate the equilibrium condition.

• year – year for which to check changes. Default to minimum year in market.

Returns:

True if converged, False otherwise.

find_equilibrium(market, sectors, maxiter=3, tol=0.1, equilibrium_variable='demand', tol_unmet_demand=-0.1, excluded_commodities=None, equilibrium=True)

Runs the equilibrium loop.

If convergence is reached, then the function returns the new market. If the maximum number of iterations are reached, then a warning issued in the log and the function returns with the current status.

Parameters:

• market – Commodities market, with the prices, supply, consumption and demand.

• sectors – A list of the sectors participating in the simulation.

• maxiter – Maximum number of iterations.

• tol – Tolerance for reaching equilibrium.

• equilibrium_variable – Variable to use to calculate the equilibrium condition.

• tol_unmet_demand – Tolerance for the unmet demand.

• excluded_commodities – Commodities to be excluded in check_demand_fulfillment

• equilibrium – if equilibrium should be reached. Useful to testing.

Returns:

A tuple with the updated market (prices, supply, consumption and demand), sectors, and convergence status.

single_year_iteration(market, sectors)

Runs one iteration of the sectors (runs each sector once).

Parameters:

• market – An initial market with prices, supply, consumption.

• sectors – A list of the sectors participating in the simulation.

Returns:

A tuple with the new market and sectors.

12.1.2. Carbon Budget

CARBON_BUDGET_FITTERS = {'Exponential': <function exponential>, 'Linear': <function linear>, 'exponential': <function exponential>, 'linear': <function linear>}

Dictionary of carbon budget fitters.

CARBON_BUDGET_FITTERS_SIGNATURE

carbon budget fitters signature.

alias of Callable[[ndarray, ndarray, int], float]

CARBON_BUDGET_METHODS = {'Bisection': <function bisection>, 'Fitting': <function fitting>, 'bisection': <function bisection>, 'fitting': <function fitting>}

Dictionary of carbon budget methods checks.

CARBON_BUDGET_METHODS_SIGNATURE

carbon budget fitters signature.

alias of Callable[[Dataset, Callable, DataArray, list], float]

class EmissionsCache(market, equilibrium, commodities)

Cache of emissions at different price points for bisection algorithm.

If a price is queried that is not in the cache, it calculates the emissions at that price using solve_market and stores the result in the cache.

adjust_bounds(lb_price, ub_price, emissions_cache, target, resolution=2)

Adjust the bounds of the carbon price for the bisection algorithm.

As emissions can be a discontinuous function of the carbon price, this method is used to improve the solution search when discontinuities are met, improving the bounds search.

Parameters:

• lb_price – Value of carbon price at lower bound

• ub_price – Value of carbon price at upper bound

• emissions_cache – Dictionary of emissions at different price points

• target – Carbon budget

• resolution – Number of decimal places to solve the carbon price to

Returns:

New lower and upper bounds for the carbon price.

bisect_bounds(lb_price, ub_price, emissions_cache, target, resolution=2)

Bisects the bounds of the carbon price.

bisect_bounds_inverted(lb_price, ub_price, emissions_cache, target, resolution=2)

Bisects the bounds of the carbon price, in the case of inverted bounds.

bisection(market, equilibrium, carbon_budget, commodities, refine_price=False, price_too_high_threshold=10, max_iterations=5, tolerance=0.1, early_termination_count=5, resolution=2, price_penalty=0.1)

Applies bisection algorithm to escalate carbon price and meet the budget.

A carbon market is meant as a pool of emissions for all the modelled regions; therefore, the carbon price applies to all modelled regions. Bisection applies an iterative estimations of the emissions varying the carbon price until convergence or stop criteria are met. Builds on ‘register_carbon_budget_method’.

Parameters:

• market – Market, with the prices, supply, consumption and demand

• equilibrium – Method for searching market equilibrium

• carbon_budget – DataArray with the carbon budget

• commodities – List of carbon-related commodities

• refine_price – Boolean to decide on whether carbon price should be capped, with the upper bound given by price_too_high_threshold

• price_too_high_threshold – Upper limit for carbon price

• max_iterations – Maximum number of iterations for bisection

• tolerance – Maximum permitted deviation of emissions from the budget

• early_termination_count – Will terminate the loop early if the last n solutions are the same

• resolution – Number of decimal places to solve the carbon price to

• price_penalty – Penalty factor applied to carbon price when selecting optimal solution when convergence isn’t reached. Higher values favor lower prices when emissions are similar.

Returns:

New value of global carbon price

create_sample(carbon_price, current_emissions, budget, size=4)

Calculates a sample of carbon prices to estimate the adjusted carbon price.

For each of these prices, the equilibrium loop will be run, obtaining a new value for the emissions. Out of those price-emissions pairs, the final carbon price will be estimated.

Parameters:

• carbon_price – Current carbon price,

• current_emissions – Current emissions,

• budget – Carbon budget,

• size – Number of points in the sample.

Returns:

An array with the sample prices.

decrease_bounds(lb_price, ub_price, emissions_cache, target, resolution=2)

Decreases the lb of the carbon price, and sets the ub to the previous lb.

exp_guess_and_weights(prices, emissions, budget)

Estimates initial values for the exponential fitting algorithm and the weights.

The points closest to the budget are used to estimate the initial guess. They also have the highest weight.

Parameters:

• prices – An array with the sample carbon prices,

• emissions – An array with the corresponding emissions,

• budget – The carbon budget for the time period.

Returns:

The initial guess and weights

exponential(prices, emissions, budget)

Fits the prices-emissions pairs to an exponential function.

Once that is done, an optimal carbon price is estimated

Parameters:

• prices – An array with the sample carbon prices,

• emissions – An array with the corresponding emissions,

• budget – The carbon budget for the time period.

Returns:

The optimal carbon price.

fitting(market, equilibrium, carbon_budget, commodities, refine_price=False, price_too_high_threshold=10, sample_size=5, fitter='linear', resolution=2)

Used to solve the carbon market.

Given the emission of a period, adjusts carbon price to meet the budget. A carbon market is meant as a pool of emissions for all the modelled regions; therefore, the carbon price applies to all modelled regions. The method solves an equation applying a fitting of the emission-carbon price relation.

Parameters:

• market – Market, with the prices, supply, and consumption

• equilibrium – Method for searching market equilibrium

• carbon_budget – limit on emissions

• commodities – list of commodities to limit (ie. emissions)

• refine_price – Boolean to decide on whether carbon price should be capped, with the upper bound given by price_too_high_threshold

• price_too_high_threshold – threshold on carbon price

• sample_size – sample size for fitting

• fitter – method to fit emissions with carbon price

• resolution – Number of decimal places to solve the carbon price to

Returns:

adjusted carbon price to meet budget

Return type:

new_price

increase_bounds(lb_price, ub_price, emissions_cache, target, resolution=2)

Increases the ub of the carbon price, and sets the lb to the previous ub.

linear(prices, emissions, budget)

Fits the prices-emissions pairs to a linear function.

Once that is done, an optimal carbon price is estimated

Parameters:

• prices – An array with the sample carbon prices,

• emissions – An array with the corresponding emissions,

• budget – The carbon budget for the time period.

Returns:

The optimal carbon price.

linear_guess_and_weights(prices, emissions, budget)

Estimates initial values for the linear fitting algorithm and the weights.

The points closest to the budget are used to estimate the initial guess. They also have the highest weight.

Parameters:

• prices – An array with the sample carbon prices,

• emissions – An array with the corresponding emissions,

• budget – The carbon budget for the time period.

Returns:

The initial guess and weights

register_carbon_budget_fitter(function=None)

Decorator to register a carbon budget function.

register_carbon_budget_method(function=None)

Decorator to register a carbon budget function.

solve_market(market, equilibrium, commodities, carbon_price)

Solves the market with a new carbon price and returns the emissions.

Parameters:

• market – Market, with the prices, supply, consumption and demand

• equilibrium – Method for searching market equilibrium

• commodities – List of carbon-related commodities

• carbon_price – New carbon price

Returns:

Emissions at the new carbon price.

update_carbon_budget(carbon_budget, emissions, year_idx, over=True, under=True)

Adjust the carbon budget in the far future if emissions too high or low.

This feature can allow to simulate overshoot shifting.

Parameters:

• carbon_budget – budget for future year,

• emissions – emission for future year,

• year_idx – index of year for estimation,

• over – if True, allows overshoot,

• under – if True, allows undershoot.

Returns:

An adjusted threshold for the future year

12.2. Sectors and associated functionality

Define a sector, e.g. aggregation of agents.

There are three main kinds of sectors classes, encompassing three use cases:

• Sector: The main workhorse sector of the model. It contains only on kind of data, namely the agents responsible for holding assets and investing in new assets.

• PresetSector: A sector that is meant to generate demand for the sectors above using a fixed formula or schedule.

All the sectors derive from AbstractSector. The AbstractSector defines two abstract functions which should be declared by derived sectors. Abstract here means a common programming practice where some concept in the code (e.g. a sector) is given an explicit interface, with the goal of making it easier for other programmers to use and implement the concept.

• AbstractSector.factory(): Creates a sector from input data

• AbstractSector.next(): A function which takes a market (demand, supply, prices) and returns a market. What happens within could be anything, though it will likely consists of dispatch and investment.

New sectors can be registered with the MUSE input files using muse.sectors.register.register_sector().

@register_sector(sector_class=None, name=None)

Registers a sector so it is available MUSE-wide.

Example

>>> from muse.sectors import AbstractSector, register_sector
>>> @register_sector(name="MyResidence")
... class ResidentialSector(AbstractSector):
...     pass

12.2.1. AbstractSector

class AbstractSector

Abstract base class for sectors.

Sectors are part of type hierarchy with AbstractSector at the apex: all sectors should derive from AbstractSector directly or indirectly.

MUSE only requires two things of a sector. Sector should be instanstiable via a factory() function. And they should be callable via next().

AbstractSector declares an interface with these two functions. Sectors which derive from it will be warned if either method is not implemented.

abstract classmethod factory(name, settings)

Creates class from settings named-tuple.

abstract next(mca_market)

Advance sector by one time period.

12.2.2. Sector

class Sector(name, technologies, supply_prod, subsectors=[], interactions=None, outputs=None, timeslice_level=None)

Base class for all sectors.

property agents

Iterator over all agents in the sector.

property capacity

Aggregates capacity across agents.

The capacities are aggregated leaving only two dimensions: asset (technology, installation date, region), year.

convert_to_global_timeslicing(market)

Converts market data to global timeslicing.

convert_to_sector_timeslicing(market)

Converts market data to sector timeslicing.

classmethod factory(name, settings)

Creates class from settings named-tuple.

interactions

A function for outputting data for post-mortem analysis.

market_variables(market, technologies)

Computes resulting market: production, consumption, and costs.

name

Timeslice level for the sector (e.g. “month”).

next(mca_market)

Advance sector by one time period.

Parameters:

mca_market – Market with demand, supply, and prices.

Returns:

A market containing the supply offered by the sector, it’s attendant consumption of fuels and materials and the associated costs.

outputs

Computes production as used to return the supply to the MCA.

It can be anything registered with @register_production.

save_outputs(year)

Calls the outputs function with the current output data.

subsectors

Parameters describing the sector’s technologies.

supply_prod

Full supply, consumption and costs data for the most recent year.

timeslice_level

Subsectors controlled by this object.

12.2.3. Subsector

class Subsector(agents, commodities, demand_share=None, constraints=None, investment=None, name='subsector', expand_market_prices=False, timeslice_level=None)

Agent group servicing a subset of the sectorial commodities.

12.2.4. PresetSector

class PresetSector(presets, interpolation_mode='linear', name='preset')

Sector with outcomes fixed from the start.

classmethod factory(name, settings)

Constructs a PresetSectors from input data.

interpolation_mode

Interpolation method

name

Name by which to identify a sector

next(mca_market)

Advance sector by one time period.

presets

Market across time and space.

12.2.5. Production

Various ways and means to compute production.

Production is the amount of commodities produced by an asset. However, depending on the context, it could be computed several ways. For instance, it can be obtained straight from the capacity of the asset. Or it can be obtained by matching for the same commodities with a set of assets.

Production methods can be registered via the @register_production production decorator. Registering a function makes the function accessible from MUSE’s input file. Production methods are not expected to modify their arguments. Furthermore they should conform the following signatures:

@register_production
def production(
    market: xr.Dataset, capacity: xr.DataArray, technologies: xr.Dataset, **kwargs
) -> xr.DataArray:
    pass

param market:

Market, including demand and prices.

param capacity:

The capacity of each asset within a market.

param technologies:

A dataset characterising the technologies of the same assets.

param **kwargs:

Any number of keyword arguments

returns:

A xr.DataArray with the amount produced for each good from each asset.

PRODUCTION_SIGNATURE

Dictionary of production methods.

alias of Callable[[Dataset, DataArray, Dataset, str], DataArray]

maximum_production(market, capacity, technologies, timeslice_level=None)

Production when running at full capacity.

Full capacity is limited by the utilization factor. For more details, see muse.quantities.maximum_production().

register_production(function)

Decorator to register a function as a production method.

See also

muse.production

supply(market, capacity, technologies, timeslice_level=None)

Service current demand equally from all assets.

“Equally” means that equivalent technologies are used to the same percentage of their respective capacity.

12.2.6. Agent Interactions

Modes of interactions between agents.

Interactions between agents are modelled via two orthogonal concepts:

• a net is a set of agents which interact in some way

• an interaction proper is a function that takes a net and actually performs the interaction.

Hence, there are two registrators in this this module, register_interaction_net(), and register_agent_interaction(). The first registers functions that take the full set of agents as input and returns a sequence of nets. It is expected each net of the sequence will be applied the same interaction. The second registrator registers the interaction proper: it takes agents as arguments and returns nothing. It is expected to modify the agents in-place.

factory(inputs=None)

Creates an interaction functor.

new_to_retro_net(agents, first_category='newcapa')

Interactions between new and retrofit agents.

register_agent_interaction(function)

Decorator to register an agent to agent(s) interaction function.

An agent interaction function takes at least two agents and makes them interact in some way.

An agent interaction function also takes as argument a sector object. This object should not be modified in any way. But it can be queried for parameters, if the specific agent interaction function requires it. This is most likely the same configuration object passed on to the interaction net function.

register_interaction_net(function)

Decorator to register a function computing interaction nets.

An interaction net function takes as input the list of all agents and returns the list of all interactions, where an interaction is a list of at least two interacting agents.

An interactiont-net function also takes as argument a sector object. This object should not be modified in any way. But it can be queried for parameters, if the specific interaction-net function requires it.

transfer_assets(from_, to_)

Transfer assets from first agent to second agent.

12.3. Agents and associated functionalities

Holds all building agents.

agents_factory(params_or_path, capacity, technologies, regions=None, year=None, **kwargs)

Creates a list of agents for the chosen sector.

create_newcapa_agent(capacity, year, region, share, search_rules='all', merge_transform='new', quantity=0.3, housekeeping='clean', retrofit_present=True, **kwargs)

Creates newcapa agent from muse primitives.

If there are no retrofit agents present in the sector, then the newcapa agent need to be initialised with the initial capacity of the sector.

create_retrofit_agent(technologies, capacity, share, year, region, decision='mean', **kwargs)

Creates retrofit agent from muse primitives.

class AbstractAgent(name='Agent', region='', assets=None, category=None, quantity=1, timeslice_level=None)

Base class for all agents.

assets

Current stock of technologies.

category

Attribute to classify different sets of agents.

filter_input(dataset, **kwargs)

Filter inputs for usage in agent.

For instance, filters down to agent’s region, etc.

name

Name associated with the agent.

abstract next(technologies, market, demand)

Increments agent to the next time point (e.g. performing investments).

Performs investments to meet demands, and increments agent.year to the investment year.

Parameters:

• technologies – dataset of technology parameters for the investment year

• market – market dataset covering the current year and investment year

• demand – data array of demand for the investment year

quantity

Attribute to classify different agents’ share of the population.

region

Region the agent operates in.

timeslice_level

Timeslice level for the agent.

tolerance = 1e-12

tolerance criteria for floating point comparisons.

uuid

A unique identifier for the agent.

class Agent(name='Agent', region='USA', assets=None, search_rules=None, objectives=None, decision=None, year=2010, maturity_threshold=0, housekeeping=None, merge_transform=None, demand_threshold=None, category=None, asset_threshold=0.0001, quantity=1, spend_limit=0, timeslice_level=None, **kwargs)

Standard agent that does not perform investments.

_housekeeping

Transforms applied on the old and new assets.

It could mean using only the new assets, or merging old and new, etc… It can be any function registered with register_final_asset_transform().

asset_housekeeping()

Reduces memory footprint of assets.

Performs tasks such as:

• remove empty assets

• remove years prior to current

decision

Transforms applied on the assets at the start of each iteration.

It could mean keeping the assets as are, or removing assets with no capacity in the current year and beyond, etc… It can be any function registered with register_initial_asset_transform().

demand_threshold

Threshold below which assets are not added.

merge_transform

Threshold below which the demand share is zero.

This criteria avoids fulfilling demand for very small values. If None, then the criteria is not applied.

next(technologies, market, demand)

Increments agent to the next time point (e.g. performing investments).

Performs investments to meet demands, and increments agent.year to the investment year.

Parameters:

• technologies – dataset of technology parameters for the investment year

• market – market dataset covering the current year and investment year

• demand – data array of demand for the investment year

objectives

Creates single decision objective from one or more objectives.

search_rules

Market share threshold.

Threshold when and if filtering replacement technologies with respect to market share.

spend_limit

One or more objectives by which to decide next investments.

year

Search rule(s) determining potential replacement technologies.

This is a string referring to a filter, or a sequence of strings referring to multiple filters, applied one after the other. Any function registered via muse.filters.register_filter can be used to filter the search space.

class InvestingAgent(*args, constraints=None, investment=None, **kwargs)

Agent that performs investment for itself.

add_investments(technologies, investments, investment_year)

Add new assets to the agent.

constraints

Creates a set of constraints limiting investment.

invest

Method to use when fulfilling demand from rated set of techs.

next(technologies, market, demand)

Iterates agent one turn.

The goal is to figure out from market variables which technologies to invest in and by how much.

This function will modify self.assets and increment self.year. Other attributes are left unchanged. Arguments to the function are never modified.

12.3.1. Objectives

Valuation functions for replacement technologies.

Objectives are used to compare replacement technologies. They should correspond to a single well defined economic concept. Multiple objectives can later be combined via decision functions.

Objectives should be registered via the @register_objective decorator. This makes it possible to refer to them by name in agent input files, and nominally to set extra input parameters.

The factory() function creates a function that calls all objectives defined in its input argument and returns a dataset with each objective as a separate data array.

Objectives are not expected to modify their arguments. Furthermore they should conform the following signatures:

@register_objective
def comfort(
    technologies: xr.Dataset,
    demand: xr.DataArray,
    prices: xr.DataArray,
    **kwargs
) -> xr.DataArray:
    pass

param technologies:

A data set characterising the technologies from which the agent can draw assets. This has been pre-filtered according to the agent’s search space.

param demand:

Demand to fulfill.

param prices:

Commodity prices.

param kwargs:

Extra input parameters. These parameters are expected to be set from the input file.

Warning

The standard agent csv file does not allow to set these parameters.

returns:

A DataArray with at least two dimension corresponding to replacement and asset. A timeslice dimension may also be present.

capacity_to_service_demand(technologies, demand, *args, **kwargs)

Minimum capacity required to fulfill the demand.

capital_costs(technologies, demand, *args, **kwargs)

Capital costs for input technologies.

comfort(technologies, demand, *args, **kwargs)

Comfort value provided by technologies.

efficiency(technologies, demand, *args, **kwargs)

Efficiency of the technologies.

emission_cost(technologies, demand, prices, timeslice_level=None, *args, **kwargs)

Emission cost for each technology when fulfilling whole demand.

Given the demand share \(D\), the emissions per amount produced \(E\), and the prices per emittant \(P\), then emissions costs \(C\) are computed as:

\[C = \sum_s \left(\sum_cD\right)\left(\sum_cEP\right),\]

with \(s\) the timeslices and \(c\) the commodity.

equivalent_annual_cost(technologies, demand, prices, timeslice_level=None, *args, **kwargs)

Equivalent annual costs (or annualized cost) of a technology.

See muse.costs.equivalent_annual_cost() for more details.

factory(settings='LCOE')

Creates a function computing multiple objectives.

The input can be a single objective defined by its name alone. Or it can be a single objective defined by a dictionary which must include at least a “name” item, as well as any extra parameters to pass to the objective. Or it can be a sequence of objectives defined by name or by dictionary.

fixed_costs(technologies, demand, *args, **kwargs)

Fixed costs associated with a technology.

Given a factor \(\alpha\) and an exponent \(\beta\), the fixed costs \(F\) are computed from the capacity fulfilling the current demand \(C\) as:

\[F = \alpha * C^\beta\]

\(\alpha\) and \(\beta\) are “fix_par” and “fix_exp” in Techno-data, respectively.

fuel_consumption_cost(technologies, demand, prices, timeslice_level=None, *args, **kwargs)

Cost of fuels when fulfilling whole demand.

lifetime_levelized_cost_of_energy(technologies, demand, prices, timeslice_level=None, *args, **kwargs)

Levelized cost of energy (LCOE) of technologies over their lifetime.

See muse.costs.lifetime_levelized_cost_of_energy() for more details.

The LCOE is set to zero for those timeslices where the production is zero, normally due to a zero utilisation factor.

net_present_value(technologies, demand, prices, timeslice_level=None, *args, **kwargs)

Net present value (NPV) of the relevant technologies.

See muse.costs.net_present_value() for more details.

register_objective(function)

Decorator to register a function as a objective.

Registers a function as a objective so that it can be applied easily when sorting technologies one against the other.

The input name is expected to be in lower_snake_case, since it ought to be a python function. CamelCase, lowerCamelCase, and kebab-case names are also registered.

12.3.2. Search Rules

Various search-space filters.

Search-space filters return a modified matrix of booleans, with dimension (asset, replacement), where asset refer to technologies currently managed by the agent, and replacement to all technologies the agent could consider, prior to filtering.

Filters should be registered using the decorator register_filter(). The registration makes it possible to call then from the agent by specifying the search_rule attribute. The search_rule attribute is string or list of strings specifying the filters to apply one after the other when considering the search space.

Filters are not expected to modify any of their arguments. They should all follow the same signature:

@register_filter
def search_space_filter(
    agent: Agent,
    search_space: xr.DataArray,
    *,
    additional_argument1,
    additional_argument2,
    **kwargs,
) -> xr.DataArray:
    pass

param agent:

the agent relevant to the search space. The filters may need to query the agent for parameters, e.g. the current year, the tolerance, etc.

param search_space:

the current search space.

param any additional arguments must follow the * argument:

arguments.

param and must be passed as keyword:

arguments.

returns:

A new search space with the same data-type as the input search-space, but with potentially different values.

In practice, an initial search space is created by calling a function with the signature given below, and registered with register_initializer(). The initializer function returns a search space which is passed on to a chain of filters, as done in the factory() function.

Functions creating initial search spaces should have the following signature:

@register_initializer
def search_space_initializer(
    agent: Agent,
    demand: xr.DataArray,
    technologies: xr.Dataset,
) -> xr.DataArray:
    pass

param agent:

the agent relevant to the search space. The filters may need to query the agent for parameters, e.g. the current year, the tolerance, etc.

param demand:

share of the demand per existing reference technology (e.g. assets).

param technologies:

A data set characterising the technologies from which the agent can draw assets.

returns:

An initial search space

compress(agent, search_space, **kwargs)

Compress search space to include only potential technologies.

This operation reduces the size of the search space along the replacement dimension, such that are left only technologies that will be considered as replacement for at least by one asset. Unlike most filters, it does not change the data, but rather changes how the data is represented. In other words, this is mostly an optimization for later steps, to avoid unnecessary computations.

currently_existing_tech(agent, search_space, *, market, **kwargs)

Only consider technologies that currently exist in the market.

This filter only allows technologies that exists in the market and have non- zero capacity in the current year. See currently_referenced_tech for a similar filter that does not check the capacity.

currently_referenced_tech(agent, search_space, *, market, **kwargs)

Only consider technologies that are currently referenced in the market.

This filter will allow any technology that exists in the market, even if it currently sits at zero capacity (unlike currently_existing_tech which requires non-zero capacity in the current year).

factory(settings=None, separator='->')

Creates filters from input TOML data.

The input data is standardized to a list of dictionaries where each dictionary contains at least one member, “name”.

The first dictionary specifies the initial function which creates the search space from the demand share, the market, and the dataset describing technologies in the sectors.

The next entries are applied in turn and transform the search space in some way. In other words the process is more or less:

search_space = initial_filter(
    agent, demand, technologies=technologies, market=market
)
for afilter in filters:
    search_space = afilter(
        agent, search_space, technologies=technologies, market=market
    )
return search_space

initial_filter is simply first filter given on input, if that filter is registered with register_initializer(). Otherwise, initialize_from_technologies() is automatically inserted.

identity(agent, search_space, **kwargs)

Returns search space as given.

initialize_from_technologies(agent, demand, *, technologies, **kwargs)

Initialize a search space from existing technologies.

maturity(agent, search_space, *, market, **kwargs)

Only allows technologies that have achieve a given market share.

Specifically, the market share refers to the capacity for each end- use.

reduce_asset(agent, search_space, **kwargs)

Reduce over assets.

register_filter(function)

Decorator to register a function as a filter.

Registers a function as a filter so that it can be applied easily when constraining the technology search-space.

The name that the function is registered with defaults to the function name. However, it can also be specified explicitly as a keyword argument. In any case, it must be unique amongst all search-space filters.

register_initializer(function)

Decorator to register a function as a search-space initializer.

same_enduse(agent, search_space, *, technologies, **kwargs)

Only allow for technologies with at least the same end-use.

same_fuels(agent, search_space, *, technologies, **kwargs)

Filters technologies with the same fuel type.

Determines fuel type based on the energy commodities used as inputs by the technology.

similar_technology(agent, search_space, *, technologies, **kwargs)

Filters technologies with the same type.

with_asset_technology(agent, search_space, **kwargs)

Search space also contains its asset technology for each asset.

12.3.3. Decision Methods

Decision methods combining several objectives into ones.

Decisions methods create a single scalar from multiple objectives. To be available from the input, functions implementing decision methods should follow a specific signature:

@register_decision
def weighted_sum(objectives: Dataset, parameters: Any, **kwargs) -> DataArray:
    pass

param objectives:

An dataset where each array is a separate objective

param parameters:

parameters, such as weights, whether to minimize or maximize, the names of objectives to consider, etc.

param kwargs:

Extra input parameters. These parameters are expected to be set from the input file.

Warning

The standard agent csv file does not allow to set these parameters.

returns:

A data array with ranked replacement technologies.

epsilon_constraints(objectives, parameters, mask=None)

Minimizes first objective subject to constraints on other objectives.

The parameters are a sequence of tuples (name, minimize, epsilon), where name is the name of the objective, minimize is True if minimizing and false if maximizing that objective, and epsilon is the constraint. The first objective is the one that will be minimized according to:

Given objectives \(O^{(i)}_t\), with \(i \in [|1, N|]\) and \(t\) the replacement technologies, this function computes the ranking with respect to \(t\):

\[\mathrm{ranking}_{O^{(i)}_t < \epsilon_i} O^{(0)}_t\]

The first tuple can be restricted to (name, minimize), since epsilon is ignored.

The result is the matrix \(O^{(0)}\) modified such minimizing over the replacement dimension value would take into account the constraints and the optimization direction (minimize or maximize). In other words, calling result.rank(‘replacement’) will yield the expected result.

factory(settings='mean')

Creates a decision method based on the input settings.

lexical_comparison(objectives, parameters)

Lexical comparison over the objectives.

Lexical comparison operates by binning the objectives into bins of width w_i = min_j(p_i o_i^j). Once binned, dimensions other than asset and technology are reduced by taking the max, e.g. the largest constraint. Finally, the objectives are ranked lexographically, in the order given by the parameters.

The result is an array of tuples which can subsequently be compared lexicographically.

mean(objectives, *args, **kwargs)

Mean over objectives.

register_decision(function, name)

Decorator to register a function as a decision.

Registers a function as a decision so that it can be applied easily when aggregating different objectives together.

retro_epsilon_constraints(objectives, parameters)

Epsilon constraints where the current tech is included.

Modifies the parameters to the function such that the existing technologies are always competitive.

retro_lexical_comparison(objectives, parameters)

Lexical comparison over the objectives.

Lexical comparison operates by binning the objectives into bins of width w_i = p_i o_i, where i are the current assets. Once binned, dimensions other than asset and replacement are reduced by taking the max, e.g. the largest constraint. Finally, the objectives are ranked lexographically, in the order given by the parameters.

The result is an array of tuples which can subsequently be compared lexicographically.

single_objective(objectives, parameters)

Single objective decision method.

It only decides on minimization vs maximization and multiplies by a given factor. The input parameters can take the following forms:

• Standard sequence [(objective, direction, factor)], in which case it must have only one element.

• A single string: defaults to standard sequence [(string, 1, 1)]

• A tuple (string, bool): defaults to standard sequence [(string, direction, 1)]

• A tuple (string, bool, factor): defaults to standard sequence [(string, direction, factor)]

weighted_sum(objectives, parameters)

Weighted sum over normalized objectives.

The objectives are each normalized to [-1, 1] over the replacement dimension by dividing by the maximum absolute value. Furthermore, the dimensions other than asset and replacement are reduced by taking the mean.

More specifically, the objective function is:

\[\sum_m c_m \frac{A_m - \min(A_m)}{\max(A_m) - \min(A_m)}\]

where sum runs over the different objectives, c_m is a scalar coefficient, A_m is a matrix with dimensions (existing tech, replacement tech). max(A) and min(A) return the largest and smallest component of the input matrix. If c_m is positive, then that particular objective is minimized, whereas if it is negative, that particular objective is maximized.

12.3.4. Investment Methods

Investment decision.

An investment determines which technologies to invest given a metric to determine preferred technologies, a corresponding search space of technologies, and the demand to fulfill.

Investments should be registered via the decorator register_investment. The registration makes it possible to call investments dynamically through compute_investment, by specifying the name of the investment. It is part of MUSE’s plugin platform.

Investments are not expected to modify any of their arguments. They should all have the following signature:

@register_investment
def investment(
    costs: xr.DataArray,
    search_space: xr.DataArray,
    technologies: xr.Dataset,
    constraints: List[Constraint],
    **kwargs
) -> xr.DataArray:
    pass

param costs:

specifies for each asset which replacement technology should be invested in preferentially. This should be an integer or floating point array with dimensions asset and replacement.

param search_space:

an asset by replacement matrix defining allowed and disallowed replacement technologies for each asset

param technologies:

a dataset containing all constant data characterizing the technologies.

param constraints:

a list of constraints as defined in constraints.

returns:

A data array with dimensions asset and technology specifying the amount of newly invested capacity.

INVESTMENT_SIGNATURE

Investment signature.

alias of Callable[[DataArray, DataArray, Dataset, list[Dataset], Any], DataArray | Dataset]

cliff_retirement_profile(technical_life, investment_year, **kwargs)

Cliff-like retirement profile from current year.

Computes the retirement profile of all technologies in technical_life. Assets with a technical life smaller than the input time-period should automatically be renewed.

We could just return an array where each year is represented. Instead, to save memory, we return a compact view of the same where years where no change happens are removed.

Parameters:

• technical_life – lifetimes for each technology

• investment_year – The year in which the investment is made

• **kwargs – arguments by which to filter technical_life, if any.

Returns:

A boolean DataArray where each each element along the year dimension is true if the technology is still not retired for the given year.

register_investment(function)

Decorator to register a function as an investment.

The output of the function can be a DataArray, with the invested capacity, or a Dataset. In this case, it must contain a DataArray named “capacity” and, optionally, a DataArray named “production”. Only the invested capacity DataArray is returned to the calling function.

12.3.5. Demand Share

Demand share computations.

The demand share splits a demand amongst agents. It is used within a sector to assign part of the input MCA demand to each agent.

Demand shares functions should be registered via the decorator register_demand_share.

Demand share functions are not expected to modify any of their arguments. They should all have the following signature:

@register_demand_share
def demand_share(
    agents: Sequence[AbstractAgent],
    demand: xr.Dataarray,
    technologies: xr.Dataset,
    **kwargs
) -> xr.DataArray:
    pass

param agents:

a sequence of agent relevant to the demand share procedure. The agent can be queried for parameters specific to the demand share procedure. For instance, :py:func`new_and_retro` will query the agents for the assets they own, the region they are contained with, their category (new or retrofit), etc…

param demand:

DataArray of commodity demands for the current year and investment year.

param technologies:

a dataset containing all constant data characterizing the technologies.

param kwargs:

Any number of keyword arguments that can parametrize how the demand is shared. These keyword arguments can be modified from the TOML file.

returns:

The unmet consumption. Unless indicated, all agents will compete for a the full demand. However, if there exists a coordinate “agent” of dimension “asset” giving the uuid of the agent, then agents will only service that part of the demand.

DEMAND_SHARE_SIGNATURE

Demand share signature.

alias of Callable[[Sequence[AbstractAgent], DataArray, Dataset, Any], DataArray]

new_and_retro(agents, demand, technologies, timeslice_level=None)

Splits demand across new and retro agents.

The input demand is split amongst both new and retrofit agents. New agents get a share of the increase in demand for the investment year, whereas retrofit agents are assigned a share of the demand that occurs from decommissioned assets.

Parameters:

• agents – a list of all agents. This list should mainly be used to determine the type of an agent and the assets it owns. The agents will not be modified in any way.

• demand – commodity demands for the current year and investment year.

• technologies – quantities describing the technologies.

• timeslice_level – the timeslice level of the sector (e.g. “hour”, “day”)

Pseudo-code:

1. the capacity is reduced over agents and expanded over timeslices (extensive quantity) and aggregated over agents. Generally:

   \[A_{a, s}^r = w_s\sum_i A_a^{r, i}\]

   with \(w_s\) a weight associated with each timeslice and determined via muse.timeslices.distribute_timeslice().

2. An intermediate quantity, the unmet demand \(U\) is defined from \(P[\mathcal{M}, \mathcal{A}]\), a function giving the production for a given market \(\mathcal{M}\), the associated consumption \(\mathcal{C}\), and aggregate assets \(\mathcal{A}\):

   \[U[\mathcal{M}, \mathcal{A}] = \max(\mathcal{C} - P[\mathcal{M}, \mathcal{A}], 0)\]

   where \(\max\) operates element-wise, and indices have been dropped for simplicity. The resulting expression has the same indices as the consumption \(\mathcal{C}_{c, s}^r\).

   \(P\) is the maximum production, given by <muse.quantities.maximum_production>`.

3. the new demand \(N\) is defined as:

   \[N = \min\left( \mathcal{C}_{c, s}^r(y + \Delta y) - \mathcal{C}_{c, s}^r(y), U[\mathcal{M}^r(y + \Delta y), \mathcal{A}_{a, s}^r(y)] \right)\]

4. the retrofit demand \(R\) is defined from the identity

   \[C_{c, s}^r(y + \Delta y) = P[\mathcal{M}^r(y+\Delta y), \mathcal{A}_{a, s}^r(y + \Delta y)] + N_{c, s}^r + R_{c, s}^r\]

   In other words, it is the share of the forecasted consumption that is serviced neither by the current assets still present in the investment year, nor by the new agent.

5. then each new agent gets a share of \(N\) proportional to it’s

   share of the production, \(P[\mathcal{A}_{a, s}^{r, i}(y)]\). Then the share of the demand for new agent \(i\) is:

   \[N_{c, s, t}^{i, r}(y) = N_{c, s}^r \frac{\sum_\iota P[\mathcal{A}_{s, t, \iota}^{r, i}(y)]} {\sum_{i, t, \iota}P[\mathcal{A}_{s, t, \iota}^{r, i}(y)]}\]

6. similarly, each retrofit agent gets a share of \(N\) proportional to it’s

   share of the decommissioning demand, \(D^{r, i}_{t, c}\). Then the share of the demand for retrofit agent \(i\) is:

   \[R_{c, s, t}^{i, r}(y) = R_{c, s}^r \frac{\sum_\iota\mathcal{D}_{t, c, \iota}^{i, r}(y)} {\sum_{i, t, \iota}\mathcal{D}_{t, c, \iota}^{i, r}(y)}\]

Note that in the last two steps, the assets owned by the agent are aggregated over the installation year. The effect is that the demand serviced by agents is disaggregated over each technology, rather than not over each model of each technology (asset).

See also

indices, quantities, Agent investments, decommissioning_demand(), maximum_production()

register_demand_share(function)

Decorator to register a function as a demand share calculation.

unmet_demand(demand, capacity, technologies, timeslice_level=None)

Share of the demand that cannot be serviced by the existing assets.

\[U[\mathcal{M}, \mathcal{A}] = \max(\mathcal{C} - P[\mathcal{M}, \mathcal{A}], 0)\]

\(\max\) operates element-wise, and indices have been dropped for simplicity. The resulting expression has the same indices as the consumption \(\mathcal{C}_{c, s}^r\).

\(P\) is the maximum production, given by <muse.quantities.maximum_production>.

unmet_forecasted_demand(agents, demand, technologies, timeslice_level=None)

Forecast demand that cannot be serviced by non-decommissioned current assets.

12.3.6. Constraints:

Investment constraints.

Constraints on investments ensure that investments match some given criteria. For instance, the constraints could ensure that only so much of a new asset can be built every year.

Functions to compute constraints should be registered via the decorator register_constraints(). This registration step makes it possible for constraints to be declared in the TOML file.

Generally, LP solvers accept linear constraints defined as:

\[\begin{split}A x \\leq b\end{split}\]

with \(A\) a matrix, \(x\) the decision variables, and \(b\) a vector. However, these quantities are dimensionless. They do no have timeslices, assets, or replacement technologies, or any other dimensions that users have set up in their model. The crux is to translate from MUSE’s data-structures to a consistent dimensionless format.

In MUSE, users can register constraints functions that return fully dimensional quantities. The matrix operator is split over the capacity decision variables and the production decision variables:

\[\begin{split}A_c .* x_c + A_p .* x_p \\leq b\end{split}\]

The operator \(.*\) means the standard elementwise multiplication of xarray, including automatic broadcasting (adding missing dimensions by repeating the smaller matrix along the missing dimension). Constraint functions return the three quantities \(A_c\), \(A_p\), and \(b\). These three quantities will often not have the same dimension. E.g. one might include timeslices where another might not. The transformation from \(A_c\), \(A_p\), \(b\) to \(A\) and \(b\) happens as described below.

• \(b\) remains the same. It defines the rows of \(A\).

• \(x_c\) and \(x_p\) are concatenated one on top of the other and define the columns of \(A\).

• \(A\) is split into a left submatrix for capacities and a right submatrix for production, following the concatenation of \(x_c\) and \(x_p\)

• Any dimension in \(A_c .* x_c\) (\(A_p .* x_p\)) that is also in \(b\) defines diagonal entries into the left (right) submatrix of \(A\).

• Any dimension in \(A_c .* x_c\) (\(A_p .* x_b\)) and missing from \(b\) is reduced by summation over a row in the left (right) submatrix of \(A\). In other words, those dimensions become part of a standard tensor reduction or matrix multiplication.

There are two additional rules. However, they are likely to be the result of an inefficient definition of \(A_c\), \(A_p\) and \(b\).

• Any dimension in \(A_c\) (\(A_b\)) that is neither in \(b\) nor in \(x_c\) (\(x_p\)) is reduced by summation before consideration for the elementwise multiplication. For instance, if \(d\) is such a dimension, present only in \(A_c\), then the problem becomes \((\\sum_d A_c) .* x_c + A_p .* x_p \\leq b\).

• Any dimension missing from \(A_c .* x_c\) (\(A_p .* x_p\)) and present in \(b\) is added by repeating the resulting row in \(A\).

Constraints are registered using the decorator register_constraints(). The decorated functions must follow the following signature:

@register_constraints
def constraints(
    demand: xr.DataArray,
    capacity: xr.DataArray,
    search_space: xr.DataArray,
    technologies: xr.Dataset,
    **kwargs,
) -> Constraint:
    pass

demand:

The demand for the sectors products in the investment year. In practice it is a demand share obtained in demand_share. It is a data-array with dimensions including asset, commodity, timeslice.

capacity:

A data-array with dimensions technology and year defining the existing capacity of each technology in the current year and investment year.

search_space:

A matrix asset vs replacement technology defining which replacement technologies will be considered for each existing asset.

technologies:

Technodata characterizing the competing technologies in the investment year.

**kwargs:

Any other parameter.

demand(demand, capacity, search_space, technologies, **kwargs)

Constraints production to meet demand.

factory(settings=None)

Creates a list of constraints from standard settings.

The standard settings can be a string naming the constraint, a dictionary including at least “name”, or a list of strings and dictionaries.

max_capacity_expansion(demand, capacity, search_space, technologies, **kwargs)

Max-capacity addition, max-capacity growth, and capacity limits constraints.

Limits by how much the capacity of each technology owned by an agent can grow in a given year. This is a constraint on the agent’s ability to invest in a technology.

Let \(L_t^r(y)\) be the total capacity limit for a given year, technology, and region. \(G_t^r(y)\) is the maximum growth. And \(W_t^r(y)\) is the maximum additional capacity. \(y=y_0\) is the current year and \(y=y_1\) is the year marking the end of the investment period.

Let \(\mathcal{A}^{i, r}_{t, \iota}(y)\) be the current assets, before investment, and let \(\Delta\mathcal{A}^{i,r}_t\) be the future investments. The the constraint on agent \(i\) are given as:

\[ \begin{align}\begin{aligned}L_t^r(y_0) - \sum_\iota \mathcal{A}^{i, r}_{t, \iota}(y_1) \geq \Delta\mathcal{A}^{i,r}_t\\(y_1 - y_0 + 1) G_t^r(y_0) \sum_\iota \mathcal{A}^{i, r}_{t, \iota}(y_0) - \sum_\iota \mathcal{A}^{i, r}_{t, \iota}(y_1) \geq \Delta\mathcal{A}^{i,r}_t\\(y_1 - y_0)W_t^r(y_0) \geq \Delta\mathcal{A}^{i,r}_t\end{aligned}\end{align} \]

The three constraints are combined into a single one which is returned as the maximum capacity expansion, \(\Gamma_t^{r, i}\). The maximum capacity expansion cannot impose negative investments: Maximum capacity addition:

\[\Gamma_t^{r, i} \geq 0\]

\(L_t^r(y)\), \(G_t^r(y)\) and \(W_t^r(y)\) default to np.inf if not provided (i.e. no constraint). If all three parameters are not provided, no constraint is applied (returns None).

max_production(demand, capacity, search_space, technologies, *, timeslice_level=None, **kwargs)

Constructs constraint between capacity and maximum production.

Constrains the production decision variable by the maximum production for a given capacity.

register_constraints(function)

Registers a constraint with MUSE.

See muse.constraints.

search_space(demand, capacity, search_space, technologies, **kwargs)

Removes disabled technologies.

12.3.7. Initial and Final Asset Transforms

Pre and post hooks on agents.

asset_merge_factory(settings='new')

Returns a function for merging new investments into assets.

Available merging functions should be registered with @register_final_asset_transform.

clean(agent, assets)

Removes empty assets.

housekeeping_factory(settings='clean')

Returns a function for performing initial housekeeping.

For instance, remove technologies with no capacity now or in the future. Available housekeeping functions should be registered with @register_initial_asset_transform.

merge_assets(old_assets, new_assets)

Adds new assets to old along asset dimension.

New assets are assumed to be nonequivalent to any old_assets. Indeed, it is expected that the asset dimension does not have coordinates (i.e. it is a combination of coordinates, such as technology and installation year).

After merging the new assets, quantities are back-filled along the year dimension. Further missing values (i.e. future years the old_assets did not take into account) are set to zero.

new_assets_only(old_assets, new_assets)

Returns newly invested assets and ignores old assets.

noop(agent, assets)

Return assets as they are.

old_assets_only(old_assets, new_assets)

Returns old assets and ignores newly invested assets.

register_final_asset_transform(function)

Decorator to register a function to merge new investments into current assets.

The transform is applied a the very end of the agent iteration. It can be any function which takes as input the current set of assets, the new assets, and any number of keyword arguments. The function must return a “merge” of the two assets.

For instance, the new assets could completely replace the old assets (new_assets_only()), or they could be summed to the old assets (merge_assets()).

register_initial_asset_transform(function)

Decorator to register a function for cleaning or transforming assets.

The transformation is applied at the start of each iteration. It any function which take an agent and assets as input and any number of keyword arguments, and returns the transformed assets. The agent should not be modified. It is only there to query the current year, the region, etc.

12.4. Reading the inputs

Ensemble of functions to read MUSE data.

read_settings(settings_file, path=None)

Loads the input settings for any MUSE simulation.

Loads a MUSE settings file. This must be a TOML formatted file. Missing settings are loaded from the DEFAULT_SETTINGS. Custom Python modules, if present, are loaded and checks are run to validate the settings and ensure that they are compatible with a MUSE simulation.

Parameters:

• settings_file – A string or a Path to the settings file

• path – A string or path to the settings folder

Returns:

A dictionary with the settings

Ensemble of functions to read MUSE data.

read_attribute_table(path)

Read a standard MUSE csv file for price projections.

read_csv_agent_parameters(filename)

Reads standard MUSE agent-declaration csv-files.

Returns a list of dictionaries, where each dictionary can be used to instantiate an agent in muse.agents.factories.factory().

read_global_commodities(path)

Reads commodities information from input.

read_initial_assets(filename)

Reads and formats data about initial capacity into a dataframe.

read_initial_market(projections, base_year_import=None, base_year_export=None)

Read projections, import and export csv files.

read_io_technodata(filename)

Reads process inputs or outputs.

There are four axes: (technology, region, year, commodity)

read_macro_drivers(path)

Reads a standard MUSE csv file for macro drivers.

read_presets(paths, columns='commodity', indices=('RegionName', 'Timeslice'), drop=('Unnamed: 0',))

Read consumption or supply files for preset sectors.

read_regression_parameters(path)

Reads the regression parameters from a standard MUSE csv file.

read_technodictionary(filename)

Reads and formats technodata into a dataset.

There are three axes: technologies, regions, and year.

read_technologies(technodata_path_or_sector=None, technodata_timeslices_path=None, comm_out_path=None, comm_in_path=None, commodities=None, sectors_directory=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/muse-os/checkouts/v1.4.3/docs/data'))

Reads data characterising technologies from files.

Parameters:

• technodata_path_or_sector – If comm_out_path and comm_in_path are not given, then this argument refers to the name of the sector. The three paths are then determined using standard locations and name. Specifically, technodata looks for a “technodataSECTORNAME.csv” file in the standard location for that sector. However, if comm_out_path and comm_in_path are given, then this should be the path to the the technodata file.

• technodata_timeslices_path – This argument refers to the TechnodataTimeslices file which specifies the utilization factor per timeslice for the specified technology.

• comm_out_path – If given, then refers to the path of the file specifying output commmodities. If not given, then defaults to “commOUTtechnodataSECTORNAME.csv” in the relevant sector directory.

• comm_in_path – If given, then refers to the path of the file specifying input commmodities. If not given, then defaults to “commINtechnodataSECTORNAME.csv” in the relevant sector directory.

• commodities – Optional. If commodities is given, it should point to a global commodities file, or a dataset akin to reading such a file with read_global_commodities. In either case, the information pertaining to commodities will be added to the technologies dataset.

• sectors_directory – Optional. If paths_or_sector is a string indicating the name of the sector, then this is a path to a directory where standard input files are contained.

Returns:

A dataset with all the characteristics of the technologies.

read_timeslice_shares(path=PosixPath('/home/docs/checkouts/readthedocs.org/user_builds/muse-os/checkouts/v1.4.3/docs/data'), sector=None)

Reads sliceshare information into a xr.Dataset.

Additionally, this function will try and recover the timeslice multi- index from a import file “Timeslices{sector}.csv” in the same directory as the timeslice shares. Pass None if this behaviour is not required.

SETTINGS_CHECKS = {'check_budget_parameters': <function check_budget_parameters>, 'check_global_data_files': <function check_global_data_files>, 'check_interpolation_mode': <function check_interpolation_mode>, 'check_iteration_control': <function check_iteration_control>, 'check_log_level': <function check_log_level>, 'check_sectors_files': <function check_sectors_files>}

Dictionary of settings checks.

SETTINGS_CHECKS_SIGNATURE

settings checks signature.

alias of Callable[[dict], None]

register_settings_check(function)

Decorator to register a function as a settings check.

Registers a function as a settings check so that it can be applied easily when validating the MUSE input settings.

There is no restriction on the function name, although is should be in lower_snake_case, as it is a python function.

12.5. Writing Outputs

12.5.1. Sinks

Sinks where output quantities can be stored.

Sinks take as argument a DataArray and store it somewhere. Additionally they take a dictionary as argument. This dictionary will always contains the items (‘quantity’, ‘sector’, ‘year’) referring to the name of the quantity, the name of the calling sector, the current year. They may contain additional parameters which depend on the actual sink, such as ‘filename’.

Optionally, a description of the storage (filename, etc) can be returned.

The signature of a sink is:

@register_output_sink(name="netcfd")
def to_netcfd(quantity: DataArray, config: Mapping) -> Optional[Text]:
    pass

exception FiniteResourceException

Raised when a finite resource is exceeded.

OUTPUT_SINKS = {'Aggregate': <class 'muse.outputs.sinks.YearlyAggregate'>, 'Csv': <function to_csv>, 'Excel': <function to_excel>, 'FiniteResourceLogger': <function finite_resource_logger>, 'Nc': <function to_netcdf>, 'Netcdf': <function to_netcdf>, 'ToCsv': <function to_csv>, 'ToExcel': <function to_excel>, 'ToNetcdf': <function to_netcdf>, 'Xlsx': <function to_excel>, 'YearlyAggregate': <class 'muse.outputs.sinks.YearlyAggregate'>, 'Yearlyaggregate': <class 'muse.outputs.sinks.YearlyAggregate'>, 'aggregate': <class 'muse.outputs.sinks.YearlyAggregate'>, 'csv': <function to_csv>, 'excel': <function to_excel>, 'finite-resource-logger': <function finite_resource_logger>, 'finiteResourceLogger': <function finite_resource_logger>, 'finite_resource_logger': <function finite_resource_logger>, 'finiteresourcelogger': <function finite_resource_logger>, 'nc': <function to_netcdf>, 'netcdf': <function to_netcdf>, 'to-csv': <function to_csv>, 'to-excel': <function to_excel>, 'to-netcdf': <function to_netcdf>, 'toCsv': <function to_csv>, 'toExcel': <function to_excel>, 'toNetcdf': <function to_netcdf>, 'to_csv': <function to_csv>, 'to_excel': <function to_excel>, 'to_netcdf': <function to_netcdf>, 'tocsv': <function to_csv>, 'toexcel': <function to_excel>, 'tonetcdf': <function to_netcdf>, 'xlsx': <function to_excel>}

Stores a quantity somewhere.

OUTPUT_SINK_SIGNATURE

Signature of functions used to save quantities.

alias of Callable[[DataArray | DataFrame, int, Any], str | None]

class YearlyAggregate(final_sink=None, sector='', axis='year', **kwargs)

Incrementally aggregates data from year to year.

register_output_sink(function=None)

Registers a function to save quantities.

sink_to_file(suffix)

Simplifies sinks to files.

The decorator takes care of figuring out the path to the file, as well as trims the configuration dictionary to include only parameters for the sink itself. The decorated function returns the path to the output file.

standardize_quantity(function)

Helps standardize how the quantities are specified.

This decorator adds three keyword arguments to an input function:

• set_index: A dictionary or any argument accepted by pandas.DataFrame.set_index(). Ignored if not specified. If specified, a call to pandas.DataFrame.reset_index() is made first.

• sort_index: A dictionary or any argument accepted by pandas.DataFrame.sort_index(). Ignored if not specified.

• keep_columns: a string or a list of strings with the names of the columns to keep. Ignored if not specified.

The three functions are applied in the order given, assuming an input is specified.

to_csv(quantity, filename, **params)

Saves data array to csv format, using pandas.to_csv.

Parameters:

• quantity – The data to be saved

• filename – File to which the data should be saved

• params – A configuration dictionary accepting any argument to pandas.to_csv

to_excel(quantity, filename, **params)

Saves data array to csv format, using pandas.to_excel.

Parameters:

• quantity – The data to be saved

• filename – File to which the data should be saved

• params – A configuration dictionary accepting any argument to pandas.to_excel

to_netcdf(quantity, filename, **params)

Saves data array to csv format, using xarray.to_netcdf.

Parameters:

• quantity – The data to be saved

• filename – File to which the data should be saved

• params – A configuration dictionary accepting any argument to xarray.to_netcdf

12.5.2. Sectorial Outputs

Output quantities.

Functions that compute sectorial quantities for post-simulation analysis should all follow the same signature:

@register_output_quantity
def quantity(
    capacity: xr.DataArray,
    market: xr.Dataset,
) -> Union[xr.DataArray, DataFrame]:
    pass

They take as input the current capacity profile, aggregated across a sectoar, a dataset containing market-related quantities, and a dataset characterizing the technologies in the market. It returns a single xr.DataArray object.

The function should never modify it’s arguments.

OUTPUTS_PARAMETERS

Acceptable Datastructures for outputs parameters

alias of str | Mapping

OUTPUT_QUANTITIES = {'Capacity': <function capacity>, 'Consumption': <function consumption>, 'Costs': <function costs>, 'Supply': <function supply>, 'capacity': <function capacity>, 'consumption': <function consumption>, 'costs': <function costs>, 'supply': <function supply>}

Quantity for post-simulation analysis.

OUTPUT_QUANTITY_SIGNATURE

Signature of functions computing quantities for later analysis.

alias of Callable[[Dataset, DataArray, Dataset, Any], DataFrame | DataArray]

capacity(market, capacity, rounding=4, **kwargs)

Current capacity.

consumption(market, capacity, sum_over=None, drop=None, rounding=4, **kwargs)

Current consumption.

costs(market, capacity, sum_over=None, drop=None, rounding=4, **kwargs)

Current costs.

factory(*parameters, sector_name='default')

Creates outputs functions for post-mortem analysis.

Each parameter is a dictionary containing the following:

• quantity (mandatory): name of the quantity to output. Mandatory.

• sink (optional): name of the storage procedure, e.g. the file format or database format. When it cannot be guessed from filename, it defaults to “csv”.

• filename (optional): path to a directory or a file where to store the quantity. In the latter case, if sink is not given, it will be determined from the file extension. The filename can incorporate markers. By default, it is “{default_output_dir}/{sector}{year}{quantity}{suffix}”.

• any other parameter relevant to the sink, e.g. pandas.to_csv keyword arguments.

For simplicity, it is also possible to given lone strings as input. They default to {‘quantity’: string} (and the sink will default to “csv”).

register_output_quantity(function=None)

Registers a function to compute an output quantity.

supply(market, capacity, sum_over=None, drop=None, rounding=4, **kwargs)

Current supply.

12.6. Quantities

Collection of functions to compute model quantities.

This module is meant to collect functions computing quantities of interest to the model, e.g. maximum production for a given capacity, etc, especially where these functions are used in different areas of the model.

Functions for calculating costs (e.g. LCOE, EAC) are in the costs module.

capacity_in_use(production, technologies, max_dim='commodity', timeslice_level=None, **filters)

Capacity-in-use for each asset, given production.

Conceptually, this operation is the inverse of production.

Parameters:

• production – Production from each technology of interest.

• technologies – xr.Dataset describing the features of the technologies of interests. It should contain fixed_outputs and utilization_factor.

• max_dim – reduces the given dimensions using max. Defaults to “commodity”. If None, then no reduction is performed.

• filters – keyword arguments are used to filter down the capacity and technologies. Filters not relevant to the quantities of interest, i.e. filters that are not a dimension of capacity or technologies, are silently ignored.

• timeslice_level – the desired timeslice level of the result (e.g. “hour”, “day”)

Returns:

Capacity-in-use for each technology, whittled down by the filters.

capacity_to_service_demand(demand, technologies, timeslice_level=None)

Minimum capacity required to fulfill the demand.

consumption(technologies, production, prices=None, timeslice_level=None)

Commodity consumption when fulfilling the whole production.

Firstly, the degree of technology activity is calculated (i.e. the amount of technology flow required to meet the production). Then, the consumption of fixed commodities is calculated in proportion to this activity.

In addition, if there are flexible inputs, then the single lowest-cost option is selected (minimising price * quantity). If prices are not given, then flexible consumption is not considered.

Parameters:

• technologies – Dataset of technology parameters. Must contain fixed_inputs, flexible_inputs, and fixed_outputs.

• production – DataArray of production data. Must have “timeslice” and “commodity” dimensions.

• prices – DataArray of prices for each commodity. Must have “timeslice” and “commodity” dimensions. If not given, then flexible inputs are not considered.

• timeslice_level – the desired timeslice level of the result (e.g. “hour”, “day”)

Returns:

A data array containing the consumption of each commodity. Will have the same dimensions as production.

emission(production, technologies, timeslice_level=None)

Computes emission from current products.

Parameters:

• production – Commodity-level production for a series of assets.

• technologies – technologies dataset containing fixed_output.

• timeslice_level – the desired timeslice level of the result (e.g. “hour”, “day”)

Returns:

A data array containing emissions (and only emissions).

maximum_production(technologies, capacity, timeslice_level=None, **filters)

Production for a given capacity.

Given a capacity \(\mathcal{A}_{t, \iota}^r\), the utilization factor \(\alpha^r_{t, \iota}\) and the the fixed outputs of each technology \(\beta^r_{t, \iota, c}\), then the result production is:

\[P_{t, \iota}^r = \alpha^r_{t, \iota}\beta^r_{t, \iota, c}\mathcal{A}_{t, \iota}^r\]

The dimensions above are only indicative. The function should work with many different input values, e.g. with capacities expanded over time-slices \(t\) or agents \(i\).

Parameters:

• capacity – Capacity of each technology of interest. In practice, the capacity can refer to asset capacity, the max capacity, or the capacity-in-use.

• technologies – xr.Dataset describing the features of the technologies of interests. It should contain fixed_outputs and utilization_factor.

• filters – keyword arguments are used to filter down the capacity and technologies. Filters not relevant to the quantities of interest, i.e. filters that are not a dimension of capacity or technologies, are silently ignored.

• timeslice_level – the desired timeslice level of the result (e.g. “hour”, “day”)

Returns:

capacity * fixed_outputs * utilization_factor, whittled down according to the filters and the set of technologies in capacity.

minimum_production(technologies, capacity, timeslice_level=None, **filters)

Minimum production for a given capacity.

Given a capacity \(\mathcal{A}_{t, \iota}^r\), the minimum service factor \(\alpha^r_{t, \iota}\) and the the fixed outputs of each technology \(\beta^r_{t, \iota, c}\), then the result production is:

\[P_{t, \iota}^r = \alpha^r_{t, \iota}\beta^r_{t, \iota, c}\mathcal{A}_{t, \iota}^r\]

The dimensions above are only indicative. The function should work with many different input values, e.g. with capacities expanded over time-slices \(t\) or agents \(i\).

Parameters:

• capacity – Capacity of each technology of interest. In practice, the capacity can refer to asset capacity, the max capacity, or the capacity-in-use.

• technologies – xr.Dataset describing the features of the technologies of interests. It should contain fixed_outputs and minimum_service_factor.

• timeslices – xr.DataArray of the timeslicing scheme. Production data will be returned in this format.

• filters – keyword arguments are used to filter down the capacity and technologies. Filters not relevant to the quantities of interest, i.e. filters that are not a dimension of capacity or technologies, are silently ignored.

• timeslice_level – the desired timeslice level of the result (e.g. “hour”, “day”)

Returns:

capacity * fixed_outputs * minimum_service_factor, whittled down according to the filters and the set of technologies in capacity.

production_amplitude(production, technologies)

Calculates the degree of technology activity based on production data.

We do this by dividing the production data by the output flow per unit of activity. Taking the max of this across all commodities, we get the minimum units of technology activity required to meet (at least) the specified production of all commodities.

For example: A technology has the following reaction: 1A -> 2B + 3C If production is 4B & 6C, this is equal to a production amplitude of 2

Parameters:

• production – DataArray with commodity-level production for a set of technologies. Must have timeslice and commodity dimensions. May also have other dimensions e.g. region, year, etc.

• technologies – Dataset of technology parameters

Returns:

DataArray with production amplitudes for each technology in each timeslice. Will have the same dimensions as production, minus the commodity dimension.

supply(capacity, demand, technologies, timeslice_level=None)

Production and emission for a given capacity servicing a given demand.

Supply includes two components, end-uses outputs and environmental pollutants. The former consists of the demand that the current capacity is capable of servicing. Where there is excess capacity, then service is assigned to each asset a share of the maximum production (e.g. utilization across similar assets is the same in percentage). Then, environmental pollutants are computing as a function of commodity outputs.

Parameters:

• capacity – number/quantity of assets that can service the demand

• demand – amount of each end-use required. The supply of each process will not exceed its share of the demand.

• technologies – factors bindings the capacity of an asset with its production of commodities and environmental pollutants.

• timeslice_level – the desired timeslice level of the result (e.g. “hour”, “day”)

Returns:

A data array where the commodity dimension only contains actual outputs (i.e. no input commodities).

12.7. Demand Matching Algorithm

Collection of demand-matching algorithms.

At it’s simplest, the demand matching algorithm solves the following problem,

• given a demand for a commodity \(D_d\), with \(d\in\mathcal{D}\)

• given processes to supply these commodities, with an associated cost per process, \(C_{d, i}\), with \(i\in\mathcal{I}\)

Match demand and supply while minimizing the associated cost.

\[ \begin{align}\begin{aligned}\min_{X} \sum_{d, i} C_{d,i} X_{d, i}\\X_{d, i} \geq 0\\\sum_o X_o \geq D_d\end{aligned}\end{align} \]

The basic algorithm proceeds as follows:

1. sort all costs \(C_{d, i}\) across both \(d\) and \(i\)

2. for each cost \(c_0\) in order:

   1. find the set of indices \(\mathcal{C}\subseteq\mathcal{D}\cup\mathcal{I}\) for which

      \[\forall (d, i) \in \mathcal{C}\quad C_{d, i} == c_0\]

   2. determine the partial result for the current cost

      \[\forall (d, i) \in \mathcal{C}\quad X_{d, i} = \frac{D_d}{|i\in\mathcal{C}|}\]

      Where \(|i\in\mathcal{C}|\) indicates the number of indices \(i\) in \(\mathcal{C}\).

However, in practice, the problem to solve often contains constraints, e.g. a constraint on production \(\sum_d X_{d, i} \leq M_i\). The algorithms in this module try and solve these constrained problems one way or another.

demand_matching(demand, cost, *constraints, protected_dims=None)

Demand matching over heterogeneous dimensions.

This algorithm enables demand matching while enforcing constraints on how much an asset can produce. Any set of dimensions can be matched. The algorithm is general with respect to the dimensions in demand and cost. It also enforces constraints over sets of indices.

\[ \begin{align}\begin{aligned}\min_{X} \sum_{d, i} C_{d, i} X_{d, i}\\X_{d, i} \geq 0\\\sum_i X_{d, i} \geq D_d\\M_{(d, i) \in \mathcal{R}^{(\alpha)}}^{(\alpha)} \geq \sum_{(d, i)\notin\mathcal{R}^{(\alpha)}} X_{d, i}\end{aligned}\end{align} \]

Where \(\alpha\) is an index running over constraints, \(\mathcal{R}^{(\alpha)}\subseteq\mathcal{D}\cup\mathcal{I}\) is a subset of indices.

The algorithm proceeds as described in muse.demand_matching. However, an extra step is added to ensure that the solutions falls within the convex-hull formed by the constraints. This projects the current solution onto the constraint. Hence, the solution will depend on the order in which the constraints are given.

1. sort all costs \(C_{d, m}\) across both \(d\) and \(m\)

2. for each cost \(c_0\) in order:

   1. find the set of indices \(\mathcal{C}\)

      \[ \begin{align}\begin{aligned}\mathcal{C}\subseteq\mathcal{D}\cup\mathcal{I}\\\forall (d, i) \in \mathcal{C}\quad C_{d, i} == c_0\end{aligned}\end{align} \]

   2. determine an interim partial result for the current cost

      \[\forall (d, i) \in \mathcal{C}\quad \delta X_{d, i} = \frac{1}{|i\in\mathcal{C}|}\left( D_d - \sum_{j \in \mathcal{I}} X_{d, j}\right)\]

      Where \(|i\in\mathcal{C}|\) indicates the number of \(i\) indices in \(\mathcal{C}\). The expression in the parenthesis is the currently unserviced demand.

   3. Loop over each constraint \(\alpha\). Below we drop the index \(\alpha\) over constraints for simplicity.

      1. Determine the excess over the constraint:

         \[E_{(d, i) \in \mathcal{R}} = \max\left\{ 0, \sum_{(d, i)\notin\mathcal{R}}\left( X_{d, i} + \delta X_{d, i} \right) - M_{(d, i) \in \mathcal{R}} \right\}\]

      2. Correct \(\delta X\) as follows:

         \[ \begin{align}\begin{aligned}\forall (d, i) \in \mathcal{C}\cap\mathcal{R}\quad \delta X\prime_{d, i} = E_{(d, i)} \frac{\delta X_{(d, i)}}{ \sum_{(e, j)\in \mathcal{C}\cap\mathcal{R}} \delta X_{(e,j)} }\\ \forall (d, i) \notin \mathcal{R}, (d, i)\in\mathcal{C} \quad \delta X\prime_{d, i} = 0\end{aligned}\end{align} \]

      3. Set \(\delta X = \max(0, \delta X - \delta X\prime)\)

A more complex problem would see independent dimensions for each quantity. In that, case we can reduce to the original problem as shown here

\[ \begin{align}\begin{aligned}C_{d, i, c} = \min_cC\prime_{d, i, c}\\D_d = \sum_{d\prime} D\prime_{d, d\prime}\\M_r = \sum_m M\prime_{r, m}\\X_{d, d\prime, i, m, c} = \left(C\prime_{d, i, c} == C_{d, i}\right) \frac{M\prime_{r, m}}{M_r} \frac{D\prime_{d, d\prime}}{D_d} X_{d, i}\end{aligned}\end{align} \]

A dimension could be shared by all quantities, in which case each point along that dimension is treated as independent.

Similarly, if a dimension is shared only by the demand and a constraint but not by the cost, then the problem can be reduced a set of problems independent along that direction.

Parameters:

• demand – Demand to match with production. It should have the same physical units as max_production.

• cost – Cost to minimize while fulfilling the demand.

• *constraints – each item is a separate constraint \(M_r\).

• protected_dims – Dimensions that will not be modified

Returns:

An array with the joint dimensionality of max_production, cost, and demand, containing the supply that fulfills the demand. The units of this supply are the same as demand and max_production.

12.8. Miscellaneous

12.8.1. Timeslices

Timeslice utility functions.

broadcast_timeslice(data, ts=None, level=None)

Convert a non-timesliced array to a timesliced array by broadcasting.

If data is already timesliced in the appropriate scheme, it will be returned unchanged.

Parameters:

• data – Array to broadcast.

• ts – Dataarray with timeslice weights. If None, defaults to the global timeslice.

• level – Level to broadcast to. If None, use the finest level of ts.

compress_timeslice(data, ts=None, level=None, operation='sum')

Convert a fully timesliced array to a coarser level.

The operation can be either ‘sum’, or ‘mean’: - sum: sum values at each compressed timeslice level - mean: take a weighted average of values at each compressed timeslice level,

according to the timeslice weights in ts

Parameters:

• data – Timesliced array to compress. Must have the same timeslicing as ts.

• ts – Dataarray with timeslice weights. If None, defaults to the global timeslice.

• level – Level to compress to. If None, don’t compress.

• operation – Operation to perform (“sum” or “mean”). Defaults to “sum”.

distribute_timeslice(data, ts=None, level=None)

Convert a non-timesliced array to a timesliced array by distribution.

Takes non-timesliced data and distributes it over the timeslice dimension according to the timeslice weights in ts. The sum of the output over all timeslices will be equal to the input. If data is already timesliced in the appropriate scheme, it will be returned unchanged.

Parameters:

• data – Array to distribute.

• ts – Dataarray with timeslice weights. If None, defaults to the global timeslice.

• level – Level to distribute to. If None, use the finest level of ts.

drop_timeslice(data)

Drop the timeslice variable from a DataArray.

If the array doesn’t contain the timeslice variable, return the input unchanged.

expand_timeslice(data, ts=None, operation='distribute')

Convert a timesliced array to a finer level.

The operation can be either ‘distribute’, or ‘broadcast’ - distribute: distribute values over the new timeslice level(s) according to

timeslice weights in ts, such that the sum of the output over all timeslices is equal to the sum of the input

• broadcast: broadcast values across over the new timeslice level(s)

Parameters:

• data – Timesliced array to expand.

• ts – Dataarray with timeslice weights. If None, defaults to the global timeslice.

• operation – Operation to perform (“distribute” or “broadcast”). Defaults to “distribute”.

get_level(data)

Get the timeslice level of a DataArray.

setup_module(settings)

Sets up module singletons.

sort_timeslices(data, ts=None)

Sorts the timeslices of a DataArray according to a reference timeslice.

This will only sort timeslices to match the reference if the data is at the same timeslice level as the reference. Otherwise, it will sort timeslices in alphabetical order.

Parameters:

• data – Timesliced DataArray to sort.

• ts – Dataarray with reference timeslices in the appropriate order

timeslice_max(data, ts=None)

Find the max value over the timeslice dimension, normalized for timeslice length.

This first annualizes the value in each timeslice by dividing by the fraction of the year that the timeslice occupies, then takes the maximum value

Parameters:

• data – Timesliced DataArray to find the max of.

• ts – Dataarray with relative timeslice lengths. If None, defaults to the global

• timeslice.

12.8.2. Commodities

Methods and types around commodities.

class CommodityUsage(value)

Flags to specify the different kinds of commodities.

For details on how enum’s work, see python’s documentation. In practice, CommodityUsage centralizes in one place the different kinds of commodities that are meaningful to the generalized sector, e.g. commodities that are consumed by the sector, and commodities that produced by the sectors, as well commodities that are, somehow, environmental.

With the exception of CommodityUsage.OTHER, flags can be combined in any fashion. CommodityUsage.PRODUCT | CommodityUsage.CONSUMABLE is a commodity that is both consumed and produced by a sector. CommodityUsage.ENVIRONMENTAL | CommodityUsage.ENERGY | CommodityUsage.CONSUMABLE is an environmental energy commodity consumed by the sector.

CommodityUsage.OTHER is an alias for no flag. It is meant for commodities that should be ignored by the sector.

CONSUMABLE = 1

Commodity which can be consumed by the sector.

ENERGY = 8

Commodity which is a fuel for this or another sector.

ENVIRONMENTAL = 4

Commodity which is a pollutant.

OTHER = 0

Not relevant for current sector.

PRODUCT = 2

Commodity which can be produced by the sector.

property name

Hack to get the name of the flag consistently across python versions.

check_usage(data, flag, match='all')

Match usage flags with input data array.

Parameters:

• data – sequence for which to match flags elementwise.

• flag – flag or combination of flags to match. The input can be a string, such as “product | environmental”, or a CommodityUsage instance. Defaults to “other”.

• match – one of: - “all”: should all flag match. Default. - “any”, should match at least one flags. - “exact”, should match each flag and nothing else.

Examples

>>> from muse.commodities import CommodityUsage, check_usage
>>> data = [
...     CommodityUsage.OTHER,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.ENVIRONMENTAL | CommodityUsage.PRODUCT,
...     CommodityUsage.ENVIRONMENTAL,
... ]

Matching “all”:

>>> check_usage(data, CommodityUsage.PRODUCT).tolist()
[False, True, True, False]

>>> check_usage(data, CommodityUsage.ENVIRONMENTAL).tolist()
[False, False, True, True]

>>> check_usage(
...     data, CommodityUsage.ENVIRONMENTAL | CommodityUsage.PRODUCT
... ).tolist()
[False, False, True, False]

Matching “any”:

>>> check_usage(data, CommodityUsage.PRODUCT, match="any").tolist()
[False, True, True, False]

>>> check_usage(data, CommodityUsage.ENVIRONMENTAL, match="any").tolist()
[False, False, True, True]

>>> check_usage(data, "environmental | product", match="any").tolist()
[False, True, True, True]

Matching “exact”:

>>> check_usage(data, "PRODUCT", match="exact").tolist()
[False, True, False, False]

>>> check_usage(data, CommodityUsage.ENVIRONMENTAL, match="exact").tolist()
[False, False, False, True]

>>> check_usage(data, "ENVIRONMENTAL | PRODUCT", match="exact").tolist()
[False, False, True, False]

Finally, checking no flags has been set can be done with:

>>> check_usage(data, CommodityUsage.OTHER, match="exact").tolist()
[True, False, False, False]
>>> check_usage(data, None, match="exact").tolist()
[True, False, False, False]

is_consumable(data)

Any consumable.

Examples

>>> from muse.commodities import CommodityUsage, is_consumable
>>> data = [
...     CommodityUsage.CONSUMABLE,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.ENVIRONMENTAL,
...     CommodityUsage.PRODUCT | CommodityUsage.CONSUMABLE,
...     CommodityUsage.ENVIRONMENTAL | CommodityUsage.PRODUCT,
... ]
>>> is_consumable(data).tolist()
[True, False, False, True, False]

is_enduse(data)

Non-environmental product.

Examples

>>> from muse.commodities import CommodityUsage, is_enduse
>>> data = [
...     CommodityUsage.CONSUMABLE,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.ENVIRONMENTAL,
...     CommodityUsage.PRODUCT | CommodityUsage.CONSUMABLE,
...     CommodityUsage.ENVIRONMENTAL | CommodityUsage.PRODUCT,
... ]
>>> is_enduse(data).tolist()
[False, True, False, True, False]

is_fuel(data)

Any consumable energy.

Examples

>>> from muse.commodities import CommodityUsage, is_fuel
>>> data = [
...     CommodityUsage.CONSUMABLE,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.ENERGY,
...     CommodityUsage.ENERGY | CommodityUsage.CONSUMABLE,
...     CommodityUsage.ENERGY | CommodityUsage.CONSUMABLE
...         | CommodityUsage.ENVIRONMENTAL,
...     CommodityUsage.ENERGY | CommodityUsage.CONSUMABLE
...         | CommodityUsage.PRODUCT,
...     CommodityUsage.ENERGY | CommodityUsage.PRODUCT,
... ]
>>> is_fuel(data).tolist()
[False, False, False, True, True, True, False]

is_material(data)

Any non-energy non-environmental consumable.

Examples

>>> from muse.commodities import CommodityUsage, is_material
>>> data = [
...     CommodityUsage.CONSUMABLE,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.ENERGY,
...     CommodityUsage.ENERGY | CommodityUsage.CONSUMABLE,
...     CommodityUsage.CONSUMABLE | CommodityUsage.ENVIRONMENTAL,
...     CommodityUsage.ENERGY | CommodityUsage.CONSUMABLE
...         | CommodityUsage.PRODUCT,
...     CommodityUsage.CONSUMABLE | CommodityUsage.PRODUCT,
... ]
>>> is_material(data).tolist()
[True, False, False, False, False, False, True]

is_other(data)

No flags are set.

Examples

>>> from muse.commodities import CommodityUsage, is_other
>>> data = [
...     CommodityUsage.OTHER,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.PRODUCT | CommodityUsage.OTHER,
... ]
>>> is_other(data).tolist()
[True, False, False]

is_pollutant(data)

Environmental product.

Examples

>>> from muse.commodities import CommodityUsage, is_pollutant
>>> data = [
...     CommodityUsage.CONSUMABLE,
...     CommodityUsage.PRODUCT,
...     CommodityUsage.ENVIRONMENTAL,
...     CommodityUsage.PRODUCT | CommodityUsage.CONSUMABLE,
...     CommodityUsage.ENVIRONMENTAL | CommodityUsage.PRODUCT,
... ]
>>> is_pollutant(data).tolist()
[False, False, False, False, True]

12.8.3. Regression functions

Functions and functors to compute macro-drivers.

class Exponential(interpolation='linear', base_year=2010, **kwargs)

Regression function: exponential

This functor is a regression function registered with MUSE as ‘exponential’.

class ExponentialAdj(interpolation='linear', base_year=2010, **kwargs)

Regression function: exponentialadj

This functor is a regression function registered with MUSE as ‘exponentialadj’.

class Linear(interpolation='linear', base_year=2010, **kwargs)

a * population + b * (gdp - gdp[2010]/population[2010] * population).

class Logistic(interpolation='linear', base_year=2010, **kwargs)

Regression function: logistic

This functor is a regression function registered with MUSE as ‘logistic’.

class LogisticSigmoid(interpolation='linear', base_year=2010, **kwargs)

Regression function: logisticsigmoid

This functor is a regression function registered with MUSE as ‘logisticsigmoid’.

class Loglog(interpolation='linear', base_year=2010, **kwargs)

Regression function: loglog

This functor is a regression function registered with MUSE as ‘loglog’.

endogenous_demand(regression_parameters, drivers, sector=None, **kwargs)

Endogenous demand based on macro drivers and regression parameters.

factory(regression_parameters, sector=None)

Creates regression functor from standard MUSE data for given sector.

register_regression(Functor=None, name=None)

Registers a functor with MUSE regressions.

Regression functors are registered with MUSE so that the functors can be called easily on created.

functor name that the functor is registered with defaults to the snake_case version of the functor name. However, it can also be specified explicitly as a keyword argument. In any case, it must be unique amongst all registered regression functor.

12.8.4. Functionality Registration

Registrators that allow pluggable data to logic transforms.

registrator(decorator=None, registry=None, logname=None, loglevel='Debug')

A decorator to create a decorator that registers functions with MUSE.

This is a decorator that takes another decorator as an argument. Hence it returns a decorator. It simplifies and standardizes creating decorators to register functions with muse.

The registrator expects as non-optional keyword argument a registry where the resulting decorator will register functions.

Furthermore, the final function (the one passed to the decorator passed to this function) will emit a standardized log-call.

Example

At it’s simplest, creating a registrator and registering happens by first declaring a registry.

>>> REGISTRY = {}

In general, it will be a variable owned directly by a module, hence the all-caps. Creating the registrator then follows:

>>> from muse.registration import registrator
>>> @registrator(registry=REGISTRY, logname='my stuff',
...              loglevel='Info')
... def register_mystuff(function):
...     return function

This registrator does nothing more than register the function. A more interesting example is given below. Then a function can be registered:

>>> @register_mystuff(name='yoyo')
... def my_registered_function(a, b):
...     return a + b

The argument ‘yoyo’ is optional. It adds aliases for the function in the registry. In any case, functions are registered with default aliases corresponding to standard name variations, e.g. CamelCase, camelCase, and kebab-case, as illustrated below:

>>> REGISTRY['my_registered_function'] is my_registered_function
True
>>> REGISTRY['my-registered-function'] is my_registered_function
True
>>> REGISTRY['yoyo'] is my_registered_function
True

A more interesting case would involve the registrator automatically adding functionality to the input function. For instance, the inputs could be manipulated and the result of the function could be automatically transformed to a string:

>>> from muse.registration import registrator
>>> @registrator(registry=REGISTRY)
... def register_mystuff(function):
...     from functools import wraps
...
...     @wraps(function)
...     def decorated(a, b) -> str:
...         result = function(2 * a, 3 * b)
...         return str(result)
...
...     return decorated

>>> @register_mystuff
... def other(a, b):
...     return a + b

>>> isinstance(REGISTRY['other'](-3, 2), str)
True
>>> REGISTRY['other'](-3, 2) == "0"
True

12.8.5. Utilities

Collection of functions and stand-alone algorithms.

agent_concatenation(data, dim='asset', name='agent', fill_value=0)

Concatenates input map along given dimension.

Example

Lets create sets of random assets to work with. We set the seed so that this test can be reproduced exactly.

>>> from muse.examples import random_agent_assets
>>> rng = np.random.default_rng(1234)
>>> assets = {i: random_agent_assets(rng) for i in range(5)}

The concatenation will create a new dataset (or datarray) combining all the inputs along the dimension “asset”. The origin of each datum is retained in a new coordinate “agent” with dimension “asset”.

>>> from muse.utilities import agent_concatenation
>>> aggregate = agent_concatenation(assets)
>>> aggregate 
<xarray.Dataset> Size: 4kB
Dimensions:     (asset: 19, year: 12)
Coordinates:
    agent       (asset) int32 76B 0 0 0 0 0 1 1 1 2 2 2 2 3 3 3 4 4 4 4
  * year        (year) int64 96B 2033 2035 2036 2037 ... 2046 2047 2048 2049
    installed   (asset) int64 152B 2030 2025 2030 2010 ... 2025 2030 2010 2025
    technology  (asset) <U9 684B 'oven' 'stove' 'oven' ... 'oven' 'thermomix'
    region      (asset) <U9 684B 'Brexitham' 'Brexitham' ... 'Brexitham'
Dimensions without coordinates: asset
Data variables:
    capacity    (asset, year) float64 2kB 26.0 26.0 26.0 56.0 ... 62.0 62.0 62.0

Note that the dtype of the capacity has changed from integers to floating points. This is due to how xarray performs the operation.

We can check that all the data from each agent is indeed present in the aggregate.

>>> for agent, inventory in assets.items():
...    assert (aggregate.sel(asset=aggregate.agent == agent) == inventory).all()

However, it should be noted that the data is not always strictly equivalent: dimensions outside of “assets” (most notably “year”) will include all points from all agents. Missing values for the “year” dimension are forward filled (and backfilled with zeros). Others are left with “NaN”.

avoid_repetitions(data, dim='year')

List of years such that there is no repetition in the data.

It removes the central year of any three consecutive years where all data is the same. This means the original data can be reobtained via a linear interpolation or a forward fill. See muse.utilities.interpolate_capacity().

The first and last year are always preserved.

broadcast_over_assets(data, template, installed_as_year=True)

Broadcasts an array to the shape of a template containing asset-level data.

The dimensions of many arrays (such as technology datasets) are fully explicit, in that each concept (e.g. ‘technology’, ‘region’, ‘year’) is a separate dimension. However, other datasets (e.g capacity), are presented on a per-asset basis, containing a single ‘asset’ dimension with with coordinates such as ‘region’, ‘installed’ (year of installation), and ‘technology’. This latter representation is sparse if not all combinations of ‘region’, ‘installed’ and ‘technology’ are present.

This function broadcasts the first representation to the shape and coordinates of the second, selecting the appropriate values for each asset (see example below).

Note: this is not necessarily limited to technology datasets. For example, it could also be used on a dataset of commodity prices to select prices relevant to each asset (e.g. if assets exist in multiple regions).

Parameters:

• data – The dataset/data-array to broadcast

• template – The dataset/data-array to use as a template

• installed_as_year –

  True means that the “year” dimension in ‘data` corresponds to the year that the asset was installed. This will commonly be the case for most technology parameters (e.g. var_par/fix_par are specified the year that an asset is installed, and fixed for the lifetime of the asset). In this case, data must have a year coordinate for every possible “installed” year in the template.

  Conversely, if the values in data apply to the year of activity, rather than the year of installation, installed_as_year should be False. An example would be commodity prices, which can change over the lifetime of an asset. In this case, if “year” is present as a dimension in data, it will be maintained as a separate dimension in the output.

Example

Define the data array: >>> import xarray as xr >>> technologies = xr.DataArray( … data=[[1, 2, 3], [4, 5, 6]], … dims=[‘technology’, ‘region’], … coords={‘technology’: [‘gasboiler’, ‘heatpump’], … ‘region’: [‘R1’, ‘R2’, ‘R3’]}, … )

This array contains a value for every combination of technology and region (e.g. this could refer to the efficiency of each technology in each region). For simplicity, we are not including a “year” dimension in this example.

Define the assets template: >>> assets = xr.DataArray( … data=[10, 50], … dims=[“asset”], … coords={ … “region”: ([“asset”], [“R1”, “R2”]), … “technology”: ([“asset”], [“gasboiler”, “heatpump”])}, … )

We have two assets: a gas boiler in region R1 and a heat pump in region R2. In this case the values don’t matter, but could correspond to the installed capacity of each asset, for example.

We want to select the values from the technology array that correspond to each asset in the template. To do this, we perform broadcast_over_assets on technologies using assets as a template: >>> broadcast_over_assets(technologies, assets, installed_as_year=False) <xarray.DataArray (asset: 2)> Size: 16B array([1, 5]) Coordinates:

technology (asset) <U9 72B ‘gasboiler’ ‘heatpump’ region (asset) <U2 16B ‘R1’ ‘R2’

Dimensions without coordinates: asset

The output array has an “asset” dimension which matches the template. Each value in the output is the value in the original technology array that matches the technology & region of each asset.

check_dimensions(data, required=(), optional=())

Ensure that an array has the required dimensions.

This will check that all required dimensions are present, and that no other dimensions are present, apart from those listed as optional.

Parameters:

• data – DataArray or Dataset to check dimensions of

• required – List of dimension names that must be present

• optional – List of dimension names that may be present

clean_assets(assets, year)

Cleans up and prepares asset for current iteration.

• removes data from before the specified year

• removes assets for which there is no capacity now or in the future

coords_to_multiindex(data, dimension='asset')

Creates a multi-index from flattened multiple coords.

future_propagation(data, future, threshold=1e-12, dim='year')

Propagates values into the future.

Example

Data should be an array with at least one dimension, “year”:

>>> coords = dict(year=list(range(2020, 2040, 5)), fuel=["gas", "coal"])
>>> data = xr.DataArray(
...     [list(range(4)), list(range(-5, -1))],
...     coords=coords,
...     dims=("fuel", "year")
... )

future is an array with exactly one year in its year coordinate, or that coordinate must correspond to a scalar. That one year should also be present in data.

>>> future = xr.DataArray(
...     [1.2, -3.95], coords=dict(fuel=coords['fuel'], year=2025), dims="fuel",
... )

This function propagates into data values from future, but only if those values differed for the current year beyond a given threshold:

>>> from muse.utilities import future_propagation
>>> future_propagation(data, future, threshold=0.1)  
<xarray.DataArray (fuel: 2, year: 4)> Size: 64B
array([[ 0. ,  1.2,  1.2,  1.2],
       [-5. , -4. , -3. , -2. ]])
Coordinates:
  * year     (year) int32 16B 2020 2025 2030 2035
  * fuel     (fuel) <U4 32B 'gas' 'coal'

Above, the data for coal is not sufficiently different given the threshold. hence, the future values for coal remain as they where.

The dimensions of future do not have to match exactly those of data. Standard broadcasting is used if they do not match:

>>> future_propagation(
...    data, future.sel(fuel="gas", drop=True), threshold=0.1
... )  
<xarray.DataArray (fuel: 2, year: 4)> Size: 64B
array([[ 0. ,  1.2,  1.2,  1.2],
       [-5. ,  1.2,  1.2,  1.2]])
Coordinates:
  * year     (year) int32 16B 2020 2025 2030 2035
  * fuel     (fuel) <U4 32B 'gas' 'coal'
>>> future_propagation(
...     data, future.sel(fuel="coal", drop=True), threshold=0.1
... )  
<xarray.DataArray (fuel: 2, year: 4)> Size: 64B
array([[ 0.  , -3.95, -3.95, -3.95],
       [-5.  , -4.  , -3.  , -2.  ]])
Coordinates:
  * year     (year) int32 16B 2020 2025 2030 2035
  * fuel     (fuel) <U4 32B 'gas' 'coal'

interpolate_capacity(data, year)

Interpolates capacity data to the given years.

Capacity between years is interpolated linearly. Capacity outside the range of the data is set to zero.

Parameters:

• data – DataArray containing the capacity data

• year – Year or years to interpolate to

lexical_comparison(objectives, binsize, order=None, bin_last=True)

Lexical comparison over the objectives.

Lexical comparison operates by binning the objectives into bins of width binsize. Once binned, dimensions other than asset and technology are reduced by taking the max, e.g. the largest constraint. Finally, the objectives are ranked lexographically, in the order given by the parameters.

Parameters:

• objectives – xr.Dataset containing the objectives to rank

• binsize – bin size, minimization direction (+ -> minimize, - -> maximize), and (optionally) order of lexicographical comparison. The order is the one given binsize.data_vars if the argument order is None.

• order – Optional array indicating the order in which to rank the tuples.

• bin_last – Whether the last metric should be binned, or whether it should be left as a the type it already is (e.g. no flooring and no turning to integer.)

Result:

An array of tuples which can subsequently be compared lexicographically.

merge_assets(capa_a, capa_b, dimension='asset')

Merge two capacity arrays.

multiindex_to_coords(data, dimension='asset')

Flattens multi-index dimension into multi-coord dimension.

nametuple_to_dict(nametup)

Transforms a nametuple of type GenericDict into an OrderDict.

reduce_assets(assets, coords=None, dim='asset', operation=None)

Combine assets along given asset dimension.

This method simplifies combining assets across multiple agents, or combining assets across a given dimension. By default, it will sum together assets from the same region which have the same technology and the same installation date. In other words, assets are identified by the technology, installation year and region. The reduction happens over other possible coordinates, e.g. the owning agent.

More specifically, assets are often indexed using what xarray calls a dimension without coordinates. In practice, there are still coordinates associated with assets, e.g. technology and installed (installation year or version), but the value associated with these coordinates are not unique. There may be more than one asset with the same technology or installation year.

For instance, with assets per agent defined as \(A^{i, r}_o\), with \(i\) an agent index, \(r\) a region, \(o\) is the coordinates identified in coords, and \(T\) the transformation identified by operation, then this function computes:

\[R_{r, o} = T[\{A^{i, r}_o; \forall i\}]\]

If \(T\) is the sum operation, then:

\[R_{r, o} = \sum_i A^{i, r}_o\]

Example

Lets construct assets that do have duplicates assets. First we construct the dimensions, using fake data:

>>> data = xr.Dataset()
>>> data['year'] = 'year', [2010, 2015, 2017]
>>> data['installed'] = 'asset', [1990, 1991, 1991, 1990]
>>> data['technology'] = 'asset', ['a', 'b', 'b', 'c']
>>> data['region'] = 'asset', ['x', 'x', 'x', 'y']
>>> data = data.set_coords(('installed', 'technology', 'region'))

We can check there are duplicate assets in this coordinate system:

>>> processes = set(
...     zip(data.installed.values, data.technology.values, data.region.values)
... )
>>> len(processes) < len(data.asset)
True

Now we can easily create a fake two dimensional quantity per process and per year:

>>> data['capacity'] = ('year', 'asset'), np.arange(3 * 4).reshape(3, 4)

The point of reduce_assets is to aggregate assets that refer to the same process:

>>> reduce_assets(data.capacity)  
<xarray.DataArray 'capacity' (year: 3, asset: 3)> Size: 36B
array([[ 0,  3,  3],
       [ 4,  7, 11],
       [ 8, 11, 19]])
Coordinates:
  * year        (year) int32 12B 2010 2015 2017
    installed   (asset) int32 12B 1990 1990 1991
    technology  (asset) <U1 12B 'a' 'c' 'b'
    region      (asset) <U1 12B 'x' 'y' 'x'
Dimensions without coordinates: asset

We can also specify explicitly which coordinates in the ‘asset’ dimension should be reduced, and how:

>>> reduce_assets(
...     data.capacity,
...     coords=('technology', 'installed'),
...     operation = lambda x: x.mean(dim='asset')
... )  
<xarray.DataArray 'capacity' (year: 3, asset: 3)> Size: 72B
array([[ 0. ,  1.5,  3. ],
       [ 4. ,  5.5,  7. ],
       [ 8. ,  9.5, 11. ]])
Coordinates:
  * year        (year) int32 12B 2010 2015 2017
    technology  (asset) <U1 12B 'a' 'b' 'c'
    installed   (asset) int32 12B 1990 1991 1990
Dimensions without coordinates: asset

tupled_dimension(array, axis)

Transforms one axis into a tuples.

12.8.6. Examples

Example models and datasets.

Helps create and run small standard models from the command-line or directly from python.

To run from the command-line:

python -m muse --model default

Other models may be available. Check the command-line help:

python -m muse --help

The same models can be instantiated in a python script as follows:

from muse import example
model = example.model("default")
model.run()

model(name='default', test=False)

Fully constructs a given example model.

File logging is added if test is False.

Parameters:

• name – Name of the model to load.

• test – If True, the logging to file is not added.

Returns:

The MCA model.

technodata(sector, model='default')

Technology for a sector of a given example model.