Incorporate a model of demand flexibility

[lanesmith]

Purpose

The purpose of this new code is to incorporate a model of demand flexibility into the production cost model. This is a first iteration of the model and will be updated in the future to (1) become more computationally efficient and (2) incorporate the modeling efforts of our external collaborators working on sectoral electrification.

What is the code doing?

The new code includes a model of demand flexibility and supporting functionality to prepare the relevant flexibility data and process the new results. Here is a more in-depth breakdown of the additions by file:

• src/REISE.jl: brief updates to the overall framework to allow demand flexibility to be considered.
• src/loop.jl: very brief update to include the Flexibility object. While not needed in this iteration of the code, information from the Flexibility object will almost certainly be needed once we start considering the look-ahead period.
• src/model.jl: this contains the bulk of the code additions. The JuMP model is updated to include new decision variables and constraints that are needed for the demand flexibility model. Some preexisting constraints needed to be updated following the inclusion of the demand flexibility model. Functionality is included to map the demand flexibility profile data (which is originally by load zone) to each bus that contains demand flexibility (which is currently every bus with demand).
• src/types.jl: create a new Flexibility object to store relevant demand flexibility information. Update preexisting types as needed.
• src/read.jl: provide functionality to read in a single demand flexibility profile and other associated demand flexibility parameters (e.g., shifting duration). The original plan was to allow for multiple profiles to be read in, thereby allowing the different demand sectors to remain disaggregated. However, after internal discussions, it seems like it is probably best to aggregate these profiles; this functionality would probably be best suited for PreREISE (similar to what is done in PreREISE/prereise/gather/demanddata/nrel_efs/aggregate_demand.py). Right now, I'm just reading in the demand flexibility parameters (currently just duration, though curtailment cost could be included at a later time) from a very basic .csv file. I'm open to suggestions on how to best store and read in this data (though this design may remain pretty fluid depending whether the duration parameter is static or more dynamic depending on weather).
• src/query.jl: accesses the load_shift_dn and load_shift_up variables from the VariablesOfInterest object.
• src/save.jl: saves the load_shift_dn and load_shift_up variables.
• pyreisejl/utility/extract_data.py: allows the load_shift_dn and load_shift_up variables to be accessed and saved as .pkl files.
• README.md: update the documentation to include new the demand flexibility model.

Where to look?

Most of the new code is located within the src folder to update the model (src/REISE.jl, src/loop.jl, and src/model.jl) and the supporting functions to prepare the data (src/read.jl and src/types.jl) and process the results (src/query.jl and src/save.jl). I also update pyreisejl/utility/extract_data.py to allow the Python wrapper functions to work with the new model. Lastly, I also updated README.md to appropriately document the new model.

Testing

The best that I can tell, none of the preexisting tests need to be modified following the inclusion of this code. Additionally, none of the code that I added appears to warrant its own tests (based on current testing convention within this repository). I did run some simple tests on my local machine to serve as a proof of concept. Essentially, the demand flexibility model behaved in a way that was to be expected, with demand trying to shift from times of higher system prices (which can be visualized via the LMP representation) to times of lower system prices. Predictably, the inclusion of the demand flexibility model (in its current state) really slows down the solution time, even when using Gurobi. While I don't expect the difference to be as stark when executed on the server, these results indicate the need for some form of feature reduction, which will be incorporated in the future. For a little more information on these results, please refer to my recent presentation, which is located in the Deliverables folder of my Explorations folder.

Visuals

While I don't include any visuals here, there are a couple limited visuals in the presentation that I mentioned in the previous section. That presentation is located in the Deliverables folder of my Explorations folder.

Time estimate

I'm not sure. I tried to submit this PR a little earlier, so there should be fewer lines of code to review than in the past. However, it does incorporate a new model and works across quite a few different files in REISE.jl.

[danielolsen]

It seems like with this change we can shed more load than we have available at a bus?

[lanesmith]

I ended up constraining the load_shed variable separate from the variable creation. When demand flexibility is enabled, load_shift_up and load_shift_dn will have an effect on the upper bound of load_shed. I wasn't sure how flexible setting constraints within the variable creation was, so that's why I moved it to its own constraint.

[danielolsen]

Here is how we modify variable bounds as we step through intervals: https://github.com/Breakthrough-Energy/REISE.jl/blob/develop/src/loop.jl#L86

[lanesmith]

Thanks for pointing this out! I was just going to reply to your next comment suggesting that this is one of changes that I forgot to account for. Is it preferred to keep the load_shed variable the same as it was originally (include the constraints in the variable creation) and instead account for load_shift_up and load_shift_dn in loop.jl if demand flexibility is allowed?

[danielolsen]

We need to make sure that load shedding is appropriately constrained in the first iteration of the loop (when we run _build_model), as well as appropriately updated in each iteration. We will have to do the same for flexibility (if present).

[BainanXia]

@danielolsen I'm about to point the same thing out.

[lanesmith]

I have updated loop.jl, so load_shed and the demand flexibility variables (load_shift_up and load_shift_dn) should be properly constrained in each iteration.

[danielolsen]

What do the values in the new flexibility.csv input represent? Are they fractions of the demand that are flexible over the course of the year?

[lanesmith]

What do the values in the new flexibility.csv input represent? Are they fractions of the demand that are flexible over the course of the year?

They are an amount of demand (in MW) that is considered to be flexible at that time. So, if you divide the amount of flexible demand provided by flexibility.csv by the total demand provided by demand.csv, you would get the percentage of demand at that bus that is flexible.

[BainanXia]

Looking through the conversations here, I think all the comments have been addressed so far, right? @lanesmith

[lanesmith]

Looking through the conversations here, I think all the comments have been addressed so far, right? @lanesmith

Yes, I believe so.

[BainanXia]

Very nice. Thanks @lanesmith Let's get @danielolsen 's green light before merging.

[lanesmith]

Very nice. Thanks @lanesmith Let's get @danielolsen 's green light before merging.

Sounds good! There is definitely room for improvements (some ideas for some of these improvements have been outlined in some issues), but I think this functionality is a good first version. It should serve as a good foundation for some of the collaborations that are ready to look at demand flexibility.

[danielolsen]

The return is not necessary here, Julia automatically returns the value of the last-evaluated expression.
See https://docs.julialang.org/en/v1/manual/functions/#The-return-Keyword

[BainanXia]

Agree. But I will vote for keeping these return statements since they are more explicit and readable even for people not familiar with Julia.

[danielolsen]

In that case, there are a few places where we should add them for consistency. See #127

[lanesmith]

Okay, so we'll keep the return statements? I can add return statements to some of the functions that don't have them, like _make_branch_map and _make_sets.

[danielolsen]

I think that would be a good idea, thanks for taking care of that.

[lanesmith]

I added return statements to all functions where it made sense. I believe this should close #127

[danielolsen]

Can we build up the right-hand-side of this constraint (based on the ordering of variables/parameters here, not based on the 'traditional' RHS definition which is parameters only) using a JuMP expression? I.e. start with the bus demand, add load shifting as necessary, and then constrain based on this expression?

Or equivalently, start with an expression containing load shedding, add load shifting as necessary, and constrain this expression by the bus demand?

[lanesmith]

Yeah, that's a good idea. That would definitely be cleaner.

[lanesmith]

I believe I have implemented something similar to what you were describing.

[lanesmith]

@danielolsen, I refactored read_demand_flexibility close to what we discussed on Friday. However, I wasn't able to get rid of the nested try/catch statements that are used for reading the demand flexibility parameters. It appears that variables created within a try statement aren't stored in memory and cannot be used outside of the try statement (unless, I'm guessing, you make it a global variable, which seems unnecessary for this application, or pre-allocate the memory outside of the try statement, which again doesn't seem necessary for this application). The nested try/catch statements appear to work how we would want them to, with errors raised in the inner try statement only being caught by the inner catch statement and not tripping the outer catch statement.

[danielolsen]

@danielolsen, I refactored read_demand_flexibility close to what we discussed on Friday. However, I wasn't able to get rid of the nested try/catch statements that are used for reading the demand flexibility parameters. It appears that variables created within a try statement aren't stored in memory and cannot be used outside of the try statement (unless, I'm guessing, you make it a global variable, which seems unnecessary for this application, or pre-allocate the memory outside of the try statement, which again doesn't seem necessary for this application). The nested try/catch statements appear to work how we would want them to, with errors raised in the inner try statement only being caught by the inner catch statement and not tripping the outer catch statement.

Yeah, Julia's scoping rules can be pretty unintuitive, especially coming from Python. I think the modifications to read_demand_flexibility make it much clearer.

[danielolsen]

Thanks. There's one place where a function gets extra parameters that it doesn't use, once this is removed I think this is ready to merge.

[lanesmith]

Thanks. There's one place where a function gets extra parameters that it doesn't use, once this is removed I think this is ready to merge.

Great, thank @BainanXia and @danielolsen for the feedback!