Streamline workflow

[FabianHofmann]

Hey all,

I am opening a discussion on potential routes for refactoring the workflow. I guess this is most interesting for @fneum @lkstrp @coroa and @nworbmot

This is only a draft and summarizes what I have in mind, see it as inspiration and kick-off with some random aspects rather than a fixed, clear picture.

Target

• Harmonize electricity only and sector coupled routines
• Modularize sectors, allow different combinations of sector additions and underlying constraints
• Streamline the workflow, reduce the number of workflow steps to a feasible minimum
• Configure target files through config rather than wildcards

Restructuring the workflow

To have a clear set of steps I would propose to follow four steps in order to create the solved network (with arbitray combination of sectors)

1. network creation (base)
2. network simplification and clustering (clustered)
3. network composition, ie. addition of all components (composed)
4. network optimization (solved)

The target file name of the network would then be base_clustered_composed_solved.

The target file and thus the process are not dependent on wildcards and is configured solely through the (scenario) configuration. Logically there is now big difference to the current elec-only network, only that the solved step is integrates the current prepare_network.

First Q: Are these steps sufficient? Or anything missing here, already?

Implementation strategy

If we decide for going into this direction, I would still try to

• keep the number of breaking changes to a minimum
• reuse most of the current scripts, reordering/refactoring them could essentially be seen as a independent (however synced) task

Thus the plan would include

1. Keep all scripts and module for now, only add new ones
2. Make all functions in add_electricity.py/prepare_network.py/prepare_sector_network.py/add_existing_base_year.py standalone functions (not dependent on wildcards or the snakemake object).
3. Introduce the option to set the {clusters} wildcard to "clustered" with the value taken from the configuration.
4. Allow add_electricity to optionally create carrier hubs for thermal generation as we know it from prepare_sector_network.py. This is not fully necessary, as far as I see it
5. Add a new script compose_network.py which contains the composition of the network. It is importing all the functions from the original scripts (add_electricity, prepare_sector_network, add_existing_baseyear) and essentially combining their main sections.
6. Add a new file compose.smk which defines the rule for composing the network and possibly others. The rule input is base_clustered and the output is base_clustered_composed.
7. Define a myopic optimization function in PyPSA that allows to user to update the network for each horizon from the previous one. It works in the same way as the perfect foresight optimization using a multi-indexed snapshot index and requires only one command. The loop over solve->add_brownfield->solve would then be happening in pypsa only.
8. Add a logic to run the compose step for multiple horizons in the compose.smk file. Which are then merged into one base_clustered_composed network. This contains all the components of the network for all horizons. This should likely be a separate rule in the compose.smk file.
9. ...

Open points

• There might be the alternative to run compose for each horizon with the previous horizon result as input.
• (probably many others)

[fneum]

Nice that we are moving away from almost all wildcards (except for {run} and {planning_horizon})!

I like the idea of changing the workflow structure first, unifying the existing "compose steps" in a single "compose rule" and then using that as our workspace to modularize based on the scenario settings.

Step 1. ok!

For 2., one also needs to consider time_aggregation which currently sits between prepare_network and prepare_sector_network. The challenge here is with myopic scenarios where the time aggregation should be consistent for different planning horizons.

Step 3. (removing {cluster}) is quickly done and pretty much independent of the rest of the restructuring.

Step 4. seems unnecessary. Instead, one could focus on re-writing how power plants are added and make that function dependent on the config (e.g., as links or generators) after step 5.

Step 6. ok!

For steps 7/8., I am unsure yet if this can all be done within PyPSA (although this is not a show-stopper for starting with 1-6). I think with this approach we may run into limits with regards to including new data / custom changes between investment periods. It could also have advantages to have the investment periods as different snakemake jobs for deployment of the cluster (i.e. saving intermediate states and not starting from scratch if optimisation fails in last investment period).

These are my thoughts for now.

[FabianHofmann]

Short follow up on an internal discussion:

To internalize the myopic and perfect-foresight option, the compose step would allow to take a composed network from a previous horizon as input. The output of the compose would then include the previous horizon and append the current horizon. Depending on myopic versus perfect foresight, the input network is already solved or not.

Example workflow for composing and solving a myopic network:

• Step compose 2030
  input: clustered
  output: composed 2030

• Step solve 2030
  input: composed 2030
  output: solved 2030

• Step compose 2035
  input: composed 2030
  output: composed 2035

• Step solve 2035
  input: composed 2035
  output: solved 2035

...

• Step solved 2050
  input: composed 2050
  output: solved

The output then contains a solved network with all horizons included.

Example workflow for composing and solving a perfect foresight network:

• Step compose 2030
  input: clustered
  output: composed 2030

• Step compose 2035
  input: composed 2030
  output: composed 2035

...

until 2050. The output then contains a composed network with all horizons included. The network is not solved yet. The final step then optimizes the composed network including all horizons.

• Step solve 2050
  input: composed 2050
  output: solved

In both cases, the final output is a solved network solved with all horizons included.

This setup would require to integrate add_existing_baseyear, add_brownfield and prepare_perfect_foresight into the compose logic.

[tgilon]

There is a small typo in the compose 2035 for the myopic workflow. The input would be solved 2030.

* Step compose 2035
  input: solved 2030
  output: composed 2035