STOCKHOLM SWEDEN 2016,
Cross-market optimization for a
hydro pumped storage using
dynamic programming
JULIA CAROLINE SUC
KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING
using dynamic programming
Master thesis in cooperation with Statkraft
JULIA SUC
Stockholm 2015
Electric Power Systems
School of Electrical Engineering
Kungliga Tekniska Högskolan
IR-EE-SB 2000:099
Following the liberalization of the electricity sector and the increase of renewables, the design of the electricity markets has evolved. Optimization techniques and models for power system generation scheduling including multiple markets are nowadays crucial.
In this thesis, a solution has been developed in order to optimize the schedule of a hydro pumped storage while considering electricity markets with different time resolutions. The model is based on the German market design and uses deterministic dynamic programming techniques. Based on past data, the final model allows backtesting analyses to assess previous strategies. The usage of forecasted prices and forecasted executions in the reserve markets gives the possibility to provide planning and indicators to the traders. Unique price scenarios and executions scenarios are considered due to the deterministic characteristic of the model.
The optimization can compare simultaneously different spot markets and provides the corresponding optimal schedule. Adjustments can be made to consider the tertiary reserve market. In the tertiary reserve market, each offer corresponds to an upward or downward regulation for a given period i.e. an increase or decrease in the injected energy in the power system. Only exclusive options are considered for a fixed volume chosen at the beginning of the simulation: either a possible upward regulation is offered, or a downward for a given period and the fixed volume. In addition, a successful offer for the primary or secondary reserve market can be assumed as a fixed parameter at the beginning of the simulation.
A demonstration of the optimization is provided for a hydro pumped storage defined in the thesis and validates the usage of dynamic programming. The simulations highlight the interest in considering multiple markets with high time resolution to increase the potential profit. Moreover, the financial interest of the reserve markets is shown for a hydro pumped storage. The model allows an assessment of the different trading possibilities in the reserve markets.
Suggested improvements to the model and potential for future work can be found in the final chapter of this thesis.
i
Efter liberaliseringen av elsektorn och ökande förnyelsebar andelen, har designen av el- marknaden utvecklats. Optimerings metoder för elproduktions planering med flera olika marknader är avgörande.
I denna avhandling, har en lösning utvecklats för att optimera produktions planerin- gen av ett pumpkraftverk som tar hänsyn till flera elmarknader med olika tidsupplösningar.
Modellen baseras på den Tyska marknads designen och deterministisk dynamisk program- mering. Baserad på historisk data, kan den slutgiltiga modellen användas för backtesting av tidigare strategier. Användningen av prognostiserade priser och utförande i reserv marknader ger möjligheten at förse planering och indikationer till handlaren.
Optimeringen kan jämföra olika spot marknader samt den tertiär reservmarknaden sam- tidigt och förser den motsvarande produktionsplanen. I tertiär marknaden motsvarar varje bud en upp eller ner reglering i den givna tidsperioden. I början av simuleringen avges reglerings håll för en fast volym: antingen är budet en upp eller ner reglering för en fast volym för en given tidsperiod. Ett lyckat bud i primär och sekundär marknaden kan antas som en konstant parameter i början av simuleringen.
En demonstration av optimeringen förses för ett pumpkraftverk definierat i avhandlingen och bekräftar användningen av dynamisk programmering. Simuleringarna framhäver in- tresset för att ta hänsyn till flera marknader med höga tidsupplösningar för potentiella vinst. Det finansiella intresset för en reserv marknad visas för ett pumpkraftverk. Modellen tillåter antagande för olika handelsmöjligheter i reserv marknader.
I sista kapitlet av avhandlingen finns förslag till förbättringar och framtida arbete.
ii
First of all, I thank deeply my supervisor at Statkraft, Konstantin Wiegandt for his guidance during my thesis work and also to allow me to perform this really interesting master thesis.
I express my gratitude to Professor Mikael Amelin at KTH Royal Institute of Technology for approving this project as master’s thesis work. I also thank Dina Khastieva for being my supervisor at KTH and for providing useful comments and remarks during the writing of my master thesis.
Finally, I especially thank Jon, Tobias and Paul for their support during my master thesis project.
iii
1 Introduction 1
1.1 Background . . . 1
1.2 Problem Definition . . . 2
1.3 Scope of the Thesis . . . 3
1.4 Overview of the Report . . . 3
2 Motivation and Literature Review 4 2.1 Schedule Optimization Including Reserve Markets . . . 4
2.2 Different Time Resolution Optimization Problem . . . 4
2.3 Application of Dynamic Programming . . . 4
3 The Electricity Market 6 3.1 Electricity Market Structure . . . 6
3.2 Spot Market . . . 7
3.2.1 Day-Ahead Market . . . 8
3.2.2 Intraday Market . . . 9
3.3 Reserve Market Design in Germany . . . 11
3.3.1 Technical Specifications . . . 12
3.3.2 Procurement System . . . 14
4 The Hydro Pumped Storage 16 4.1 Technical Description . . . 16
4.2 Overall Efficiency . . . 17
4.3 Turbine and Pump Efficiency . . . 18
4.4 Flexibility . . . 18
4.5 Start up Costs . . . 19
4.6 General Advantages and Disadvantages . . . 20
4.7 Grid Fees . . . 21
4.8 Water Value . . . 21
5 Dynamic Programming 22 5.1 Dynamic Programming Methodology . . . 22
5.2 Mathematical Description for a Deterministic Problem . . . 23
5.3 Methodology . . . 24
5.4 Running Time . . . 25
5.5 Advantages and Drawbacks . . . 25
6 Model and Methodology 26 6.1 Inputs and Outputs of the Model . . . 26
6.2 Hydro Pumped Storage Model . . . 27
6.3 Electricity Market Model . . . 29
6.3.1 Spot Markets . . . 29
iv
6.3.2 Reserve Markets . . . 29
6.3.2.1 Primary Reserve Market . . . 29
6.3.2.2 Secondary Reserve Market . . . 30
6.3.2.3 Tertiary Reserve Market . . . 31
6.4 Application of Dynamic Programming . . . 32
6.4.1 General Presentation . . . 32
6.4.2 Including 1h-resolution markets . . . 36
6.4.3 Including Primary Reserve . . . 39
6.4.4 Including Secondary Reserve . . . 39
6.4.5 Including Tertiary Reserve . . . 40
6.4.6 Running Time . . . 41
6.4.7 Water Value . . . 42
7 Results and Discussion 43 7.1 General Input Data . . . 43
7.2 Choice of the Initial State . . . 43
7.3 Choice of the Operating Situation . . . 45
7.4 Spot Markets Optimization . . . 47
7.5 Demonstration of the Optimization Process . . . 49
7.5.1 Inputs . . . 49
7.5.2 Results . . . 49
7.6 Comparison of Reserve Markets . . . 54
7.6.1 Inputs . . . 55
7.6.2 Profit Comparison . . . 56
7.7 Strategies for Successive Optimization . . . 58
8 Conclusions and Outlook 60
Bibliography 62
3.1 Time frame of the EPEX and EXAA trading periods . . . 7
3.2 Principle of EPEX and EXAA spot markets [1] . . . 8
3.3 Schematic explanation of the use of 15 min resolution (right) instead of one hour (left) . . . 10
3.4 Frequency change after primary control action [2] . . . 13
4.1 Efficiencies depending on the discharge [3] . . . 19
4.2 Head-flow ranges of small hydro turbines [3] . . . 19
6.1 Profile of the power generation to deliver positive SRL . . . 30
6.2 Profile of the power generation to deliver negative SRL . . . 31
6.3 Schematic explanation of the optimization at stage t including the 1h markets 37 7.1 Profit from the 04-2015 to the 09-2015 with respect to the initial state of the hydro pumped storage . . . 44
7.2 Prices on EPEX 15 min for the beginning of 04-2015 . . . 44
7.3 Profit repartition over 6 months in the different operating situations . . . . 46
7.4 Monthly profit repartition in the different operating situations . . . 46
7.5 Profit repartition over 6 months in the different situations . . . 47
7.6 Monthly profit repartition in the different situations . . . 48
7.7 Percentage of the volume traded for each spot market from the 30.03.2015 to the 04.10.2015 . . . 48
7.8 EPEX prices from the 07.09.2015 to the 16.09.2015 . . . 50
7.9 Prices used for the MRL blocks from the 07.09.2015 to the 16.09.2015 . . . 50
7.10 Trading schedule from the 07.09.2015 to the 16.09.2015 . . . 50
7.11 Variation of the energy stored from the 07.09.2015 to the 16.09.2015 . . . . 51
7.12 Water value pumping and consumption from the 07.09.2015 to the 16.09.2015 51 7.13 Water value generating and generation from the 07.09.2015 to the 16.09.2015 52 7.14 Schedule on the 15 min and 1 h EPEX market from the 07.09.2015 to the 16.09.2015 . . . 52
7.15 Production and consumption schedules from the 07.09.2015 to the 16.09.2015 52 7.16 Schedule in the MRL market, negative MRL (left), positive MRL (right) . . 53
7.17 Required capacity price for the second week of Sept, negative MRL (left), positive MRL (right) . . . 53
7.18 MRL execution for Sept (links), for the 2nd week of Sept (right) . . . 54
7.19 Detailed profits . . . 54
7.20 Profits for the 1st (left) and the 2nd (right) week of September . . . 56
7.21 Profits for the 3rd (left) and the 4th (right) week of September . . . 56
7.22 Profits for the 1st (left) and the 2nd (right) week of April . . . 57
7.23 Profits for the 3rd (left) and the 4th (right) week of April . . . 57
7.24 Description of the trading strategy 1 . . . 58
vi
3.1 Day-Ahead auction characteristics . . . 9
3.2 Intraday call auction characteristics . . . 9
3.3 EPEX intraday characteristics . . . 10
4.1 Detailed efficiencies for a hydro pumped storage [4] . . . 18
4.2 Switching times of reversible pump turbines [5] . . . 20
7.1 Hydro Pumped Storage Characteristics . . . 43
7.2 Benchmarking of the profits in the different situations S with respect to the best situation (%). . . 46
7.3 Running time comparison in each situation S. . . 46
7.4 Benchmarking of the profits in the different market situations (%) . . . 48
7.5 Prices offered in the reserve markets . . . 56
7.6 Volume offered in the reserve markets . . . 56
vii
1.1 Background
In recent years the design of the German electricity market has changed significantly. The deregulation of the electricity sector has been followed by a promotion of renewable energy through the Energiewende policy. The target is to reach a share of renewables in Germany’s gross energy consumption of 30% by 2030, 45% by 2040 and 60% by 2050 [6]. In order to fulfill these goals, new regulatory measures, which address topics such as climate change, security of energy supply and affordable energy prices have been required and are still discussed.
The electricity is a special commodity, which cannot be stored. Its quality and cor- rect delivery are crucial for the society and a perfect balance between production and consumption must be ensured at any time. The introduction of renewables, which are hardly predictable, has created new challenges in this field. New regulations have been introduced to deal with the increased share of renewables. The different markets considered in the scope of this thesis: day ahead and intraday energy markets as well as the reserve markets, have been adapted. Reserve power, traded in the reserve markets, is required to match consumption with production in case of imbalance. Reserves ensure a balanced and safe power grid. In these markets, new designs have been created by synchronization and interconnection of the four previous distinct regional control areas in Germany. A common web-based tendering platform for the reserve markets has been launched in 2006.
Subsequently, from 2008 to 2010 the four transmission operators were obliged to gradually coordinate their operations [7]. In 2010, after the requirement to phase-out nuclear, regula- tory changes have been introduced to encourage the development of renewables [7]. These decisions have resulted in multiple new options to trade electricity especially in the reserve markets.
Optimizing the trading and operating schedule of power plants is essential for energy providers. However, the complexity increases in the case of a power plant that disposes fast controllable generation units, which can be used to trade in the reserve markets. The problem of deciding whether to bid in the intraday or day-ahead energy markets or to commit the units for balancing purposes is non trivial. The problem becomes more complex as these options have different time resolutions and restrictions that partially exclude or complement each other.
In this perspective, power plants such as hydro pumped storage become especially in- teresting as they can be used to trade on several types of electricity markets. In Germany, they consist of the intraday and day-ahead energy markets and primary, secondary and tertiary reserve markets. A large range of possible options is covered.
1
1.2 Problem Definition
The purpose of the project is to establish an optimal schedule for a hydro pumped storage based on different trading possibilities. The model should be defined in order to highlight the financial interest of the different markets and motivate further studies. Prices and executed volume in the reserve market are supposed to be known and given as inputs. The model is developed based on a specific hydro pumped storage but should remain general.
Thus, the characteristics of the power plant should be parameters of the problem and are described as the following:
• a storage capacity defined as an energy storage in MWh.
• a number of identical reversible machines which can be operated as pump and turbine.
• characteristics of the pumping mode: (consumption) power and efficiency.
• characteristics of the turbine mode: 3 generation modes each defined by a power and an efficiency.
• start-up costs for generating (identical for all the machines).
• start-up costs (identical) for pumping (identical for all the machines).
The possibility to buy and sell electric power at different types of markets with different time resolution products is considered. The products refer to the different kind of trades that can be made in a market. The different markets are:
• intraday market: 1 market of 15 min resolution and 1 market of 1 h resolution.
• day ahead market: 2 markets with 15 min resolution (EXAA 15 min and EPEX 15 min) and 1 market of 1 h resolution (EPEX 1 h).
• primary reserve used mainly to stabilize the gird frequency: 1 market of 7 days resolution which consists of 1 product.
• secondary reserve used mainly to maintain the stabilized frequency at its nominal value: 1 market of 7 days resolution which consists of 4 products.
• tertiary reserve used in case of serious imbalance: 1 market of 1 day resolution which consists of 12 products.
The present study develops a trading tool which is able to deliver the optimal schedule for a hydro pumped storage according to different independent cases:
• the optimal schedule is provided by simultaneous optimization on markets from the day ahead and/or intraday markets. All the bids are assumed to be successfull. The schedule can also include the trades in the tertiary reserve markets. Upward or downward regulation corresponding to an increase or decrease in the energy fed in the power system can be offered. Only exclusive offers are considered: upward or downward for a fixed volume defined at the beginning of the simulation are traded for a given period.
• the primary and secondary reserve markets are consider to assess their financial interest. In addition to the pervious case, a bid is assumed to be successful in these reserve markets. The offered power and the offered price are chosen at the beginning of the simulation. The schedule is then provided by taking into account the given offer and by simultaneous optimization on different time resolution spot markets (15 min and 1 h) and tertiary reserve.
1.3 Scope of the Thesis
The model has been developed by using dynamic programming techniques and has been limited to a deterministic approach. The uncertainties regarding the prices in the spot markets (intraday and day ahead markets) or the volumes executed in the reserve markets have not been taken into account. Statkraft has provided the data from the electricity markets.
The present study offers backtesting to analyze the past obtained results as well as the possibility to study different trading options. This work sheds light on the interest of further possible research that could be carried out in this field.
1.4 Overview of the Report
Chapter 2 presents a review on papers published in the topic of planning schedule for a hydro pumped storage considering reserve markets.
Chapter 3 contains a presentation of the electricity markets and products considered for the hydro pumped storage. It provides an overview of the specified characteristics of the Germany electricity markets used in the optimization.
Chapter 4 describes briefly how a hydro pumped storage works and its main charac- teristics for the modeling.
Chapter 5 contains the theory behind dynamic programming and the reasons to use it in the developed model. This method is restricted to deterministic cases.
Chapter 6 details how the key elements, the power plant and the markets, have been modeled and how dynamic programming techniques have been implemented. Necessary assumptions are also explained and discussed.
Chapter 7 contains a demonstration of the program’s usage as well as a discussion and comparison of different trading options.
Chapter 8 draws conclusions about the master thesis work and displays some thoughts and leads to enhance the model in further works.
Hydro pumped storages were traditionally integrated in the electric power system to pro- duce during the peak load and pump the water and store energy in the upper reservoir when the demand is low. Unit commitment problems have been developed in which the pumped storage unit is used in combination with other plants [8].
However, the new legislations in the German electricity markets create new opportu- nities for the energy producers to trade electricity and optimize their profit. The interest of hydro-pumped storages relies on their flexibility and their ability to deliver energy on different electricity markets [9]. The optimal scheduling problem can be considered as a self-scheduling problem where hydro pumped storage are considered individually. The optimal bidding strategy consists of finding the suitable time slot, generation/consumption and price, which maximize the profit.
Research has been carried out on different aspects of the problem: optimization of power plant schedule by including the reserve markets, optimization considering markets with 1 h and 15 min resolutions, application of dynamic programming for hydro problems. A review of interesting papers in these categories is given as example of what has already been done.
2.1 Schedule Optimization Including Reserve Markets
Research has been carried out to consider the reserve markets in order to maximize the profit of a power plant. However, the models consider most often mixed portfolios including combined heat and power plants [10], [11]. In general, only few researchers have addressed the case of hydro pumped storage separately and emphasized the share of profits that could be obtained in the reserve markets [12]. According to [12], the consideration of the spot market alone severely underestimates the possible income of pumped-storage units.
2.2 Different Time Resolution Optimization Problem
Paper [13] provides a review and comparison of different unit commitment methods for a set of generators while including uncertainty from wind production. Commitments with resolutions of 1 h and 15 min are considered. The results and running time of the simulations considering each case are analyzed. The results show that the 15 min schedule lead to more efficient commitments and dispatch decisions, which increase the profit.
2.3 Application of Dynamic Programming
Paper [11] presents the advantage of using dynamic programming for scheduling large pumped storage plants. In [14] possible methods including dynamic programming are presented to develop an optimization-based model of the Nordic power market including cas- cade reservoirs. In order to circumvent the issue of dimensionality and make the algorithm
4
faster, further techniques have been tried such as Differential Dynamic Programming [15], Approximate Dynamic Programming [16], Dual Dynamic Programming [17]. However, the literature focuses mainly on multiple reservoir systems and hybrid systems integrating renewables and not on scheduling problems for one independent hydro pumped storage unit.
This thesis aims at delivering a useful tool for energy traders to maximize the profit from a hydro pumped storage by considering the different available trading opportunities in the German electricity markets. The mentioned points are combined in order to deliver a clear overview of the possible trading options to the traders. The choice of using dynamic programming has been legitimated by its previous applications in this field and its possible usage to easily deliver valuable complementary information to the traders.
This section aims at delivering an overview of the electricity market with a focus on the German markets design. The optimization of a hydro pumped storage done in chapter 7 is based on the markets described in this section.
In the power system, a supply side with one or more generators delivers power to one or more loads in the demand side through an electric grid [1]. The electricity can be produced through different forms of generation and can be categorized into nuclear, thermal and renewable sources. These sources of energy have a marginal cost that has to be covered if the power plant is dispatched to meet the demand. The consumers correspond to the final users of the electricity produced. Their needs build the demand curve, which shows daily and monthly patterns and can vary significantly concerning peak hours and annual energy consumption. The production and consumption happen on different places. The grid owners operate and maintain the grid to ensure an adequate quality of power supply and make the physical link between supply and demand. The system operator is then responsible for the technical operation of the power system, which among other things means to be responsible for frequency control and to ensure that the players are balanced.
Actually, in order to deliver safe electrical power, production and consumption should be equal. Imbalance creates instability, which can lead to blackout and should be avoided.
However, nowadays large-scale storage systems are not economically feasible and not widely used. As the electricity market is liberalized, electricity markets have been developed to balance the system and ensure reserves for unplanned events.
Due to historical reasons the German electricity market was divided in territorial mo- nopolies. Following the EU-directive of 1996, the National Energy Act of 1998 was voted in Germany to liberalize the electricity sector. However, large network energy supply companies still had a large influence and the legal unbundling was enacted with the Na- tional Energy Act of 2005. In addition, the Bundesnetzagentur a regulatory agency has been created [18]. The production and retail area are liberalized. In Germany the market organization is defined as a bilateral contracts model or decentralized market [19]. The players trade through a power pool defined as a transparent exchange market where supply and demand is matched resulting in a unique price. In parallel, they may sell and purchase freely with other players. The price of the pool serves as a guideline. The transactions are then reported to the system operator that controls the fulfillment of balance responsibility requirements. It should be noticed that Germany and Austria are fully interconnected.
They could be considered as one market, however, for historical reasons, two spot markets exist.
3.1 Electricity Market Structure
Electricity markets consists of multiple markets with different delivery time. They aim at creating an agreed schedule to deliver electricity and cover the demand in a safe way:
6
ensuring stable frequency and avoiding blackout. The different players agree on these markets to consume or generate a certain amount of power during a given period. The markets are:
• Day-ahead Market
• Intraday Market
• Forward Market
The study of the forward market is out of the scope of this thesis as it deals with agreements on deliveries in the future, up to six years in advance [2]. The thesis focuses on markets, which lead to a physical delivery at the latest during the following week.
The electricity deliveries for the next day are traded in the day-ahead market. The producers send bids to the system operators stating the time of delivery, the price and the power that they are willing to deliver or consume. Their goal is to maximize their profit. After a clearing time, the accepted bids are announced and each producer has a schedule to fulfill for the next day. In the intraday market the schedule of the generators is adjusted up to 30 minutes before delivery [20]. The goal is to deal with unplanned events in order to fulfill the day-ahead commitments i.e. to be balanced over the balancing period.
In Germany the balancing periods corresponds to the quarter-hours. If the system is not balanced at the time of delivery, the Transmission System Operators, TSOs, increase or decrease the production fed in the grid by the use of the balancing reserves. The players who were imbalanced during the required balancing period are charged using the imbalance settlement system. However, this process is not described in this thesis.
In addition to power exchange places, contracts can also be traded over the counter i.e. direct transaction between buyer and seller. However, these contracts are not consider in this thesis.
3.2 Spot Market
In the spot market, the players are responsible for delivering a schedule in which they buy to cover their forecasted demand and sell their expected production. The aim is to get the most advantageous price. In Germany it is possible to trade on two different exchanges:
EPEX and EXAA. Both cover the German and Austrian markets. Figure 3.1 shows the different timeframes for the markets, which are described in the next sections. The dotted lines correspond to the intraday markets were the trades happen continuously up to 30 minutes before delivery. Unborken lines refer to auction where the trades happen the day before delivery.
Figure 3.1: Time frame of the EPEX and EXAA trading periods
How does the spot market operate ?
Each participant submits bids before a clearing time. The bid indicates a price for which the player is willing to buy/sell a given quantity for a given period in the future. All the
selling bids are then sorted according to their price and form the offer curve also known as merit order. The buying bids form the demand curve. The intersection between the two curves fixes the price for the given period. The generators offering a lower price than the settled price at this time are then dispatched and paid at the settled price. This can be visualized as a group of plants ordered in such a way that the ones with the lowest marginal cost (cost to operate) are dispatched first followed by the second cheaper and so on until the demand is fulfilled. The last plant in the merit-order is the one with the highest marginal cost and is generating at a level close or below its full capacity while the cheapest plants deliver at maximal capacity. Figure 3.2 represents this system. The power plants with a price lower than the clearing price are dispatched to cover the demand.
Figure 3.2: Principle of EPEX and EXAA spot markets [1]
3.2.1 Day-Ahead Market
The day-ahead energy market lets market participants commit to buy or sell wholesale electricity one day before the day of delivery.
The prices for EPEX and EXAA are revealed at different points in time, even though the same hourly trading periods are considered. Paper [21] studies how the European spot markets are influenced by the EXAA prices. Table 3.1 shows the general characteristics of the two markets [20], [22]. Block products corresponding to an offer over periods longer as 1 hour are not considered in this study and therefore are not presented in this section. It should be noticed that EXAA integrates quarter-hour resolution into the day ahead auction to avoid arbitrage between the quarter hours, hours and longer blocks. The average of four-quarter hour prices equals the price of the respective full hour [22]. Therefore, an offer over 1 h corresponds to the combination of 4 successive 15 min offers.
In the rest of this thesis, EPEX 1 h refers to the prices with 1 h resolution in the EPEX day-ahead market while EXAA 15 min refers to the prices with 15 min resolution in the EXAA day-ahead market.
Intraday call auction
In order to increase the efficiency and accuracy of the market, EPEX also offers an intraday call auction. It consists in a quarterly hour resolution market, which can be traded before
Table 3.1: Day-Ahead auction characteristics
Characteristics EPEX EXAA
Trading procedure Daily Auction Daily Auction
Trading days Everyday Working day
Tradable contracts Hour Hour and Quarter hour
Order book opening At 00:00 am At 8:00 am
starting 45 days before delivery day one day before delivery day Order book closes Daily at 12:00 pm Daily at 10:12 am
Publication time At 12:42 pm Around 10:20 am
Delivery procedure According to the merit order According to the merit order Min/max prices -500 / +3000e/MWh -150 / +3000e/MWh
Minimum volume size 0.1 MW 0.1 MW
Volume intervals 0.1 MW 0.1 MW
Price characteristics e/MWh one decimal digit e/MWh one decimal digit Quantity characteristics MW one decimal digit MW one decimal digit
3 pm the day before delivery. Its creation was especially motivated to favor the increase of renewables [23]. A shorter trading period allows producers to model in a better way the solar ramp. Their offer can follow more accurately their production and they can more easily be balanced over the required 15-minutes balancing period. This is represented in figure 3.3. The red line represents the ramp in produced power when the sun rises. The continuous blue line represents the power sold. With 15 min resolution, it is theoretically possible to have as much positive and negative imbalance. The market characteristics are presented in table 3.2 [20]. In the rest of this thesis, the prices in the EPEX 15 min market refers to the prices for the EPEX intraday call auction.
Table 3.2: Intraday call auction characteristics
Characteristics EPEX
Trading procedure Daily Auction
Trading days Everyday
Tradable contracts Quarter hour
Order book opening 24 hours per day starting 45 days before delivery day Order book closes Daily at 3:00 pm
Publication time Around 3:10 pm
Delivery procedure According to the merit order Min/max prices -3000 / +3000e/MWh Minimum volume size 0.1 MW
Volume intervals 0.1 MW
Price characteristics In e/MWh with one decimal digit Quantity characteristics In MW with one decimal digit
3.2.2 Intraday Market
The Intraday Market lets market participants buy and sell wholesale electricity during the course of the operating day. In this market the players try to balance the differences between day-ahead commitments and the actual real-time demand and production of
Figure 3.3: Schematic explanation of the use of 15 min resolution (right) instead of one hour (left)
electricity. Table 3.3 summarizes the characteristics of this market [22]. In the rest of the thesis, Intraday 15 min (respectively Intraday 1 h) refers to the prices with 1 h (respectively 15 min) resolution in the EPEX intraday market.
Table 3.3: EPEX intraday characteristics
Characteristics EPEX
Trading procedure Continuous trade
Trading days 24h/7
Tradable contracts Quarter hour or hour
Order book opening 3 pm for hourly trades the day before 4 pm for quarterly trades the day before
Deal closes As soon as two orders (demand/offer) are compatible Min/max prices -9 999 / +9 999e/MWh
Minimum volume size 0.1 MW Volume intervals 0.1 MW
Price characteristics Increment of 0.01 e/MWh Quantity characteristics In MW with one decimal digit
3.3 Reserve Market Design in Germany
The liberalization of the electricity market in the late 20th century has brought new necessities for adequate coordination to secure the power grid. Power is crucial in our societies: the safety and the quality of the delivery have to be ensured at every instant.
The electronic devices are designed to work at a nominal frequency: 50 Hz in Europe.
Therefore, it is crucial to avoid fluctuation in the frequency, which means to ensure that production and consumption are equal. This can be understood from the following example.
A situation of equilibrium between consumption and production can be considered. A sudden increase in consumption creates a loss of equilibrium. In order to balance the load increase, the rotational energy stored in all synchronous machines connected to the system is used. This causes the rotor speed and therefore the frequency to decrease due to their strong connection in synchronous machines. A load higher than the generation would create the opposite reactions resulting in an increase in the frequency. The role of the electric power system is to transport the electricity over distances to the consumers and to balance all fluctuations that occur. However the technical characteristics of the elec- tricity system such as its small storage capacity make this requirement more difficult to fulfill.
Deviations from the schedule fixed in the day ahead market may appear due to the following effects:
• Unplanned power plant outages
• Unplanned outages of transmission lines
• Deviations in the load forecast
• Deviations in the meteorological prognosis which create wind or solar forecast devia- tions
The electricity generation from renewables has been growing rapidly in the last few years.
Due to their stochastic nature, the balance requirement becomes more complex. Reserve markets help matching production with consumption in case of unplanned events.
Before the liberalization, the integrated utility was in charge of balancing the grid. Nowa- days in a liberalized sector, the market participants (i.e. supply companies, large consumers) are required to be balanced within a specified timeframe, quarter hours in Germany. The energy schedules are submitted daily to the TSOs, which are then responsible of the techni- cal balance of the system in real time via the use of flexible reserves available in the reserve market. According to the European Network of Transmission System Operators, ENTSO-E, each TSO has to ensure the physical equilibrium between generation and consumption at every time through the use of three reserves: primary, secondary and tertiary reserves.
The system operator buys capacities on these markets in advance. These capacities represent additional power or reduction of power that can be executed to balance the system according to technical specifications. The TSOs activate bids to feed-in additional electricity into the network or to withdraw electricity from the network. Theses bids correspond to different kind of products available in the reserve market. For historical reasons, the German grid is divided in four areas with a different TSO for each: Tennet, 50Herz, TransnetBW and Amprion. Since the 1st December 2007, the four TSOs tender control power on their common platform www.regelleistung.net. There are three types of reserve markets and participants have to meet specific requirements before bidding. These prequalification are specific to each reserve.
The three types of reserve markets:
• Primärregelung: primary reserve (PRL)
• Sekundärregelung: secondary reserve (SRL)
• Minutenreserve: tertiary reserve (MRL)
Each reserve is used differently to balance the system and is divided in different products.
The auctions for the reserve markets happen independently of all other energy transactions.
For the producers, which are profit-maximizer, this creates more alternatives. The opti- mization of the profit has to take into account the expected profit from generating and trading electricity in the spot market and the expected profit from allocating capacity to the reserves and executing their bid when required by the TSOs.
3.3.1 Technical Specifications
Upward regulation is defined as an increase in the generation to cover the load and uses positive reserve while downward regulation consists in a decrease of the generation and uses negative reserve. The three reserves are intended to deliver both regulations through their products.
Disturbances occur when the generation doesn’t match the consumption. This causes the frequency to deviate from the nominal value on which are currently based the elec- trical devices. Activating the primary control stabilizes the system at a new frequency.
The activation is done automatically. Actually, synchronous machines have the same frequency as the grid. As explained, a lack of power causes the machines to slow down as their rotational energy is used to cover the load, which results in a frequency decrease.
In case of a surplus of power, the mechanism is reversed. To prevent this effect, some power plants have governors that adjust the generation according to the change in frequency.
Once the primary control has stabilized the frequency, the second control is called through the secondary reserve to restore the frequency to the nominal value and releases the use of primary control reserves. In contrast to the primary control, the secondary control is local.
The control area affected by the power imbalance is required to stabilize the system. In addition, it ensures that the time deviation of the system remains low. The time deviation corresponds to the integral of the error in the frequency with respect to time [1]
ti = Z t
0
f (τ ) − fnominal
fnominal dτ (3.1)
where t is the continuous time, f the frequency and ti the time deviation
The accumulated errors in frequency produce a permanent bias in clocks which yields to permanent errors in the energy exchange and should be corrected.
In contrast to the primary control process, secondary control reserve is automatically activated in the control areas where the system imbalances occur. It is implemented as proportional integral controller. Thus, no steady-state deviation remains: the control area where the imbalance happens is supplied with a surplus of power until the frequency reaches its nominal value. After this point the primary reserve will be activated again for the next failures.
The tertiary control uses tertiary reserve and is usually activated manually by the TSOs.
It is used to free up or complement the previous reserves.
Primary reserve [24]
Figure 3.4 describes the change in frequency due to a small perturbation being corrected by primary control and defines the notations used.
• Execution of primary control: automatically for frequency deviations exceeding ± 20 mHz [25]
• Maximum quasi-steady-state frequency: the deviation should not exceed ± 180 mHz
• Allowed range of the instantaneous frequency: [49.2 : 50.8] Hz
• Time response: reserve fully activated within 15s
• Availability: for at least 15 minutes
Figure 3.4: Frequency change after primary control action [2]
Secondary reserve
The following specifications should be ensured:
• Execution of secondary control: automatically
• Time response: reserve fully activated within 30s Tertiary reserve
The following specification should be ensured:
• Execution of tertiary control: manually
• Time response: reserve fully activated within 15min
The technical requirements are the most demanding for the primary reserve as it requires high flexible generating units. In decreasing order of technical complexity come then the secondary and tertiary reserves. As a consequence all the power plants cannot enter these markets.
Dimensioning of the reserves
The European Network of Transmission System Operators, ENTSO-E, fixes the amount of primary reserve that should be allocated in the continental European synchronously interconnected system. It represents 3000MW and according to the calculation rule Germany has to ensure 568 MW. However, 628 MW are tendered on the German Internet platform as it includes a share of 25 MW from the demand of Switzerland and 35 MW of the demand of the Netherlands.
The dimensioning of the second and tertiary reserve is done locally and differs between the different European TSOs. In Germany, the TSOs determine together every three months (in March, June, September and December) for the next quarter the necessary provision of secondary and tertiary reserve. The capacity for the second reserve is approximately ± 2000 MW and for the tertiary reserve ± 2500 MW.
3.3.2 Procurement System
The three types of reserve are offered as pay-as-bid. If the bid is successful, the producers earn their offered price times their offered quantity. All the bids are ranked in a merit order until the total offered volume is enough to cover the demand in the reserve markets.
These bids are then successfully selected. In the three markets, the providers specify a capacity price to compensate for the cost and loss induced by allocating a defined amount of power to the reserves instead of using it to trade in the day-ahead or intraday markets.
For the secondary and tertiary markets they also specify an energy price to reward the actual energy required by the TSO. Actually this amount is not known in advance and creates cost for the suppliers.
The cheapest bids for the capacity price are selected in order to reach the required capacity for the different reserves. For the secondary and tertiary reserves, the suppliers specify in addition an energy price for the executed energy. If two bids have the same capacity price, their place in the merit order is defined by their energy price. When the reserve is required to balance the system, the TSO executes the bids according to the merit order for the energy price [26]. In the first primary reserve, the actual use (i.e. how much energy is actually required) is not rewarded separately. The net execution after increase and decrease in production to offer primary reserve is considered as zero.
Primary Reserve
The characteristics of the German primary reserve market are as follows.
• Auction time: the week before delivery on Tuesday before 3 pm
• Auction schedule: weekly from Monday to Sunday
• Products: symmetric positive and negative primary control has to be offered, the producers commit their units to deliver upward and downward regulation at any time
• Minimum bid: 1 MW
• Minimum increment of the bid: 1 MW
• Availability: the offered capacity of primary reserve has to be 100% available
• Allocation: merit order for the capacity price and pay-as-bid
• Reward: for the allocated capacity
Secondary Reserve
The characteristics of the German secondary reserve market are as follows.
• Auction time: the week before delivery on Wednesday before 3 pm
• Auction schedule: weekly from Monday to Sunday
• Products: Peak (HT) from Monday to Friday between 8 am and 8 pm without public holiday, positive control or negative control for the whole peak period. Off-peak (NT) the rest of the week, positive or negative control for the whole off-peak period
• Minimum bid: 5 MW
• Minimum increment of the bid: 1 MW
• Availability: the offered capacity of secondary reserve has to be 100% available
• Allocation: merit order for the energy price and pay-as-bid
• Reward: for the allocated capacity and executed power Tertiary Reserve
The characteristics of the German tertiary reserve market are as follows.
• Auction time: from Monday to Friday before 10 am for the following day and the following days-off (week-ends, national holidays)
• Auction schedule: for one working day, week-ends, national holidays
• Products: each day is divided in 6 blocks of 4 hours with the possibility to offer positive or negative reserve for the whole considered block
• Minimum bid: 5 MW
• Minimum increment of the bid: 1 MW
• Availability: the offered capacity of secondary reserve has to be 100% available
• Allocation: merit order for the energy price and pay-as-bid
• Reward: for the allocated capacity and executed power
Comparison
Primary and secondary reserves are procured in weekly tenders while tertiary reserve is delivered in daily tenders. They are then divided in different delivery periods. For primary reserves, power plants are committed for the whole week rather than for secondary reserve they are committed for their offered time-slices called peak and off-peak. Tertiary reserve has the shorter tendering periods with a daily auction consisting in six time-slices of 4 hours. One of the interests of decreasing and slicing the tendering periods is to lead to a more efficient dispatch and to allow smaller operators to enter these markets [27]. This aims at making the market more competitive and efficient and avoids market manipulation [27].
This chapter provides key concepts to understand the operation of hydro pumped storages.
These elements are further used to model and optimize the schedule of a hydro pumped storage.
The main idea of a hydro pumped storage is to store energy as hydraulic potential energy by pumping water from a low-level into a high-level reservoir. This energy is then used to differ the electricity production to a later point in time. When pumping, a motor transforms electrical energy in mechanical energy by making a pump rotating, which sends the water to the upper reservoir. When generating, the water is discharged which drives the turbines connected to the generators. This produces electricity. A hydro pumped storage can consume and generate electricity. Due to imperfect efficiencies, pumping requires more energy than the actual energy produced by the pumped water. Therefore, a hydro pumped storage is a net consumer. Its interest relies on its flexibility and its ability to consume at low market price and generate at high market price. It is an economical and flexible way of storing large amounts of energy. Its storage capacity is only limited by the size of the available upper reservoir. Many forms of energy storage have been installed but hydro pumped storage are the most widely used with a capacity of more than 127000 MW [28] in operation worldwide. It represents 99% of the world storage capacity [28].
4.1 Technical Description
A hydro-pumped storage plant has two operation modes: generation and consumption.
While generating, water flows from an upper reservoir through a dam and turns a turbine connected to a generator and creates electricity. When consuming, power is taken from the power grid to run the electric motor, which drives the pump turbine. Water from the lower reservoir is then pumped into the upper reservoir.
A conventional hydro pumped storage is composed of an upper reservoir and a dam, which holds water back. The stored water in the upper reservoir is considered as stored potential energy, which depends on the head i.e. the height between the water surface and the turbine. Water conductors drive the water when the dam opens and lead to the turbine.
They are sized to match the flow required by the power plant capacity. The water exiting the turbine flows into a lower reservoir instead of flowing downstream and is used during the pumping process. The power plant house contains the elements enabling the plant to consume and produce electricity. It hosts the turbines and pumps. Ternary systems with a separate pump and turbine and reversible machines with a reversible pump-turbine exist. A motor transform electrical energy into mechanical energy and a generator does the opposite.
Transformers convert the generator’s voltage up to high voltage on the transmission grid or down to the voltage used by the pumped storage. The plant is then linked to the power
16
grid through power lines.
The energy stored in the upper reservoir is given by the potential energy of the water:
Epot= ρgV H (4.1)
Where ρ is the density of the water in kg/m3, g is the acceleration of gravity 9.86 m/s2 V is the volume in m3
H is the height in m
The head of the power plant is a key component. A higher head allows to store more energy within a given storage volume and it produces a higher electrical power with a given cross sectional area of penstocks.
4.2 Overall Efficiency
The process of pumping water and releasing it back is not 100% efficient. The amount of energy required to pump a given amount of water is higher than the energy that can be produced by releasing this amount of water. The overall efficiency of the plant is defined as the ratio between the energy used to pump a nominal volume of water and the energy obtained after releasing it.
The main losses are:
• the viscous friction in tunnels and penstock
• the hydraulic and mechanical losses within the pump and the turbine
• the electrical and mechanical losses within the motor and the generator
The losses due to the friction can be represented through the constant Hvf,puin the pumping mode and Hvf,tu in the generating mode. These values are homogeneous to a distance.
The efficiency from the pump, the turbine, the motor and the generator are respectively noted ηpu, ηtu, ηmot, ηgen. By using the same notation as in (4.1) the obtained electrical energy while discharging a volume V can be written as:
Eel,pump= ρgV (H − Hvf,tu)ηtuηgen (4.2)
While the energy required to store this volume is given by:
Eel,turbine = ρgV (H + Hvf,pu) 1
ηpuηmot (4.3)
We can thus define the overall efficiency as ηoverall = Eel,turbine
Eel,pump = H − Hvf,tu
H + Hvf,puηpuηtuηmotηgen (4.4) In reality other losses due to seepage and evaporation should be taken into account [4].
Table 4.1 shows a more comprehensive description of the different efficiencies that have to be taken into account in the process of pumping and delivering water. The overall efficiency is between 75 and 80%.
Table 4.1: Detailed efficiencies for a hydro pumped storage [4]
Low % High % Generating Components
Water conductors 97.40 98.50
Pump turbine 91.50 92.00
Generator motor 98.50 99.00
Transformer 99.50 99.70
Subtotal 87.35 89.44
Losses and leakage 98.00 99.50 Pumping Components
Water conductors 97.60 98.50
Pump turbine 91.60 92.50
Generator motor 98.70 99.00
Transformer 99.50 99.80
Subtotal 97.80 90.02
Total 75.15 80.12
4.3 Turbine and Pump Efficiency
The power generated by the power plant P in watts (W) can be defined with the previous notations by [4]:
P = QHρgη (4.5)
where Q is the fluid flow in cubic meters per second (m3/s) and η is the efficiency.
From this equation, it is clear that the produced power depends on the head, the flow and the efficiency of the power plant. However, these variables are strongly linked. By fixing one parameter, the other one can be optimized. For example the head and the flow are dependent on each other: if the head is high, the flow can be designed to be low.
Reciprocally, if the flow can be maximized the head can be reduced [29]. Different turbines are relevant for each situation as seen in 4.2. In the same way, the flow and the efficiency present dependencies. In this study the head is fixed. The relation of interest is then the link between efficiency and flow. The turbines are designed to generate at a nominal flow.
Figure 4.1 represents the efficiencies at the designed and reduced flows for different kind of turbines. The efficiency can then be written as a function of the flow and the power output can be described by:
P = f (Q, η(Q)) (4.6)
4.4 Flexibility
Hydro pumped storages are flexible: a daily use and a quick start are possible. Its fast power delivery capability is essential for the qualification in the reserve market. Table 4.2 shows the switching time for the different operations depending on the state of a reversible machine. The change of the generation mode may depend on the hydro pumped storage.
However, based on the asset owned by Statkraft the time change is supposed to fulfill the requirements for the three types of reserve.
Figure 4.1: Efficiencies depending on the discharge [3]
Figure 4.2: Head-flow ranges of small hydro turbines [3]
4.5 Start up Costs
Any start and stop of the machine in the pumping and generating mode induce abrasion of valves, mechanical equipment and generator, especially windings and insulation. An increasing number of starts increases the need of monitoring and maintenance as well as more frequent refurbishment. For a hydro pumped storage, the damages are caused by the so-called hydraulic transient. It describes the disturbance in a fluid between two steady state levels and results in:
Table 4.2: Switching times of reversible pump turbines [5]
Physical operation Time range [s] Average time [s]
Standstill to generating 60-100 80
Pumping to generating 90-120 105
Standstill to pumping 160-240 200
Generating to pumping - 420
Change of generation level <15 <15
• High pressure
• Vacuum conditions
• Cavitation
• Hydraulic vibrations
They cause damages and stresses in the pipelines, tunnels, valves or other components in the long time. More detailed explanation can be found in [30]. Therefore, the cycle process of starting and shutting down the plant creates costs due to the increase in maintenance and the loss of water at the beginning and end of a cycle. Moreover, the life span of the plant is reduced. These costs have to be taken into account in the scheduling process [31].
A cycle of start and stop of the pumping or generating mode is associated to a so called start up cost, added at each start. It is an estimation of the total cost created by the repetitive starts and stops of the machine’s mode during the whole lifetime of the plant in relation to the estimated number of cycles during this period.
4.6 General Advantages and Disadvantages Advantages
Due to recent development and use of renewable energy has experienced rapid growth over the past few years. The aim is now to ensure a stable production independent from the fluctuations in the renewable production and from possible outages as well as to utilize the most efficiently the current power capacity. In this context, the development of storage systems is crucial as described in [32]. The technical characteristics of the hydro pumped storage explain its high share among the storage technologies [33]. Its advantages are summarized in the following points.
Energy transfer
The immediate production can be stored and utilized at an appropriate time i.e. low-load energy into peak-load energy. The peak hour prices can be decreased. Due to the fast response of hydro pumped storages, temporary losses due to unplanned events or changes in the forecasts for renewables can be compensated. For the generation companies, hydro pumped storages can be used as a backup to fulfill the commercial obligation of pre-sold energy supply and avoid the potential penalties for being imbalanced. Paper [34] shows the positive advantage of combining wind power plants and pumped storages in order to avoid imbalance penalties. The imbalance is charged at a more volatile price than the price in the market, which can result in high costs.
Network stability
The flexibility and fast availability of the stored energy can help provide instant response to change in the demand. In this context being able to run intermittently the plant to follow
the demand secures the power grid and helps power-quality control. Hydro pumped storages can offer available power as reserve used by the TSOs to balance varying electricity demand and production. Due to the remuneration of the reserve, the possible range of trading transactions is increased. Besides, hydro pumped storages are suitable for black starts i.e.
its operation can be restored without relying on the external transmission network [35].
Disadvantages
The main drawbacks of hydro pumped storages are their high capital costs. The facility is also dependent on the location. Finding a suitable place with head and water reduce the number of possible products. The environmental impacts of hydro pumped storages have also to be taken into account in every project [36].
4.7 Grid Fees
Since January 2008, German hydro-pumped storages are charged for their use of the transmission lines for consumption purpose. The net consumption of the plant is charged at a price called grid fee established by the TSOs. This creates an additional cost that has to be taken into account in the optimization process. Moreover, in order to reduce the consumption during peak hours, the TSOs define timeframes, depending on the seasons, during which consumption from hydro pumped storage is charged at a really high price [37].
During these timeframes the net output of the plant should then be equivalent to a production.
4.8 Water Value
Water value indicates the value of adding an amount of water in the reservoir. It can be seen as the expected marginal value of the energy stored in the reservoir. It is used by the traders to maximize the profit by choosing to release water for generation versus storing it for future use. If the price on the market is higher than the water value it is more profitable to generate as to store water in the reservoir for the next stage. An equivalent water value for pumping can be defined.
In order to deliver a cross-market optimization for a hydro pumped storage, dynamic programming method is used. Its principle and main characateristics are explained and discussed in this chapter. The introduced notations are then used in chapter 6 to be adapted to the given problem.
Using dynamic programming, large optimization problems are solved by solving a se- quence of optimization sub problems. The problem is recursive and the solutions to the sub problems are then memorized and reused to solve the general problem. This method is based on Bellman’s Principle of Optimality:
An optimal policy has the property that whatever the previous decisions were the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decisions. (Bellman, 1957) [14]
This means that, whatever the previous decisions leading to the current system state were, the remaining decisions have to be optimal for the subproblem.
5.1 Dynamic Programming Methodology
To apply dynamic programming, the problem should present the following characteristics:
• The optimal solution to a problem is composed of optimal solutions to sub problems
• If the different solutions lead to the same optimum, the choice between the different solutions does not matter
The problem can be described according to the following structure. This analysis is based on the methods given in [38] and [14] by defining:
The stages of the problem. The stages are the sequential order of the optimization problem. To use dynamic programming the subproblem at a given stage requires knowing the optima from the previous sub problems.
The states of the problem. The state contains the information required to solve the subproblem at a given stage.
The possible decision in the given state and stage. The decision affects and updates the state of the problem for the next stage.
The exogenous information. It corresponds to external data known at the beginning of each stage.
22
The transition function. This function determines the evolution of the system in the following stage depending on the current state and decision.
The contribution function. It represents the objective function that has to be op- timized over the whole period reduced to the given stage.
The recursive relationship between the optimal decision at a stage and at the previous one. Due to the Bellman Optimality principle, the optimal decision at a given stage and state depends on the decision at this stage and the optima in the updated state in the following stage.
5.2 Mathematical Description for a Deterministic Problem
In a deterministic problem it is assumed that all the information required to solve the given problem are available, therefore the effect of any variable can be computed with certainty.
The considered optimization problem consists of solving [39]
M axd∈∆U (s, d, v)
subject to G(s, d, v) ≥ 0 (5.1)
where s is the state of the system, d the decision to optimize among ∆, v is a control variable that can be chosen in every period by the decision-maker, U is the objective function and G represents the constraints connecting the state and control variables.
According to the Bellman optimality concept, the problem can be sequential. There- fore the solution is a set of optimal decisions and states and the control variables can be divided in stages: d1,d2,...,dT, s1,s2,...,sT and v1,v2,...,vT where T in our case correspond to the period of simulation. The ensembles of possible states and decisions at each time is supposed time independent and can be written respectively ∆ and Σ. U and G are assumed to be time-separable. For t in [0..T]:
U (s0, ..., sT −1; d0...dT −1; v0...vT −1) =
U0(s0, d0, v0) + U1(s1, d1, v1) + ... + UT −1(sT −1, dT −1, vT −1) + S(sT) (5.2) where S(sT) is a chosen value for the last stage.
G function corresponds to the transition equations i.e. rule to change of state, and has the following structure, for s in Σ:
s1= G0(s0, d0, v0) s2= G1(s1, d1, v1)
...
sT = GT −1(sT −1, dT −1, vT −1)
(5.3)
The problem of optimizing the objective function over the entire time horizon can be written again as:
M axd0..dT −1∈∆T
T −1
X
t=0
Ut(st, dt, vt) + S(sT) (5.4)
subject to st+1= Gt(st, dt, vt)∀t ∈ [0..T-1], ∀s ∈ Σ, ∀d ∈ ∆ s0 = ¯s0 given
This problem can be divided in sub problems:
M axdt0..dT −1∈∆T
T −1
X
t=t0
Ut(st, dt, vt) + S(sT) (5.5)
subject to st+1 = Gt(st, dt, vt) ∀t ∈ [0..T-1], ∀s ∈ Σ, ∀d ∈ ∆ st0 = ¯st0 given
According to Bellman the optimal solution of the sub problems will lead to the optimal solution for the whole. Therefore the sub problems for the final stage are solved, followed by the previous one and so on. The general problem will then be solved at the last step using all the memorized solutions to the sub problems. If Vt(st) is the value of the solved sub problems between stages t and T, then for every t in [0..T], si in Σ and dj in ∆:
Vt(st) =maxdt∈∆[Ut(st, dt, vt) + Vt+1(st+1)] ∀st∈ Σ (5.6)
subject to st+1= Gt(st, dt, vt) ∀st∈ Σ, ∀dt∈ ∆ st= ¯st given ∀st∈ Σ
VT = ¯VT given
This problem is defined as the Bellman backward relationship and is used in this thesis.
With this formulation U is the contribution function, G the transition function and V the recursive function also called in the literature Profit-to-go function (PTG).
5.3 Methodology
Backward recursion
The optimization problem is solved by solving the recursion formula of the PTG. Starting from the last stage of the optimization horizon it is assumed that:
V¯T = 0 (5.7)
The problem is then solved recursively by storing the results for each possible starting state at each stage so for every s0 in Σ at each stage t and reusing them for solving the previous stage.
Forward step
Once all the optimal decisions for each state at each stage are known, it is possible to find the set of optimal decisions for the period starting from the initial one. Thanks to the backward recursion, the optimal decision at the first stage with the given initial decision is known as d∗0, thus the state at stage 1 is given by:
st+1= Gt(st, d∗t, vt) with t = 0 (5.8) The reconstruction of the optimal decision path over the stages continue in the same way by using the optimal decisions d∗t stored for each state and stage.
