Competition in Markets for Ancillary Services? The Implications of Rising Distributed Generation

Ancillary services are electricity products which include balancing energy, frequency regulation, voltage support, constraint management and reserves. Traditionally they have been procured by system operators from large conventional power plants, as by-products of the production of energy. This paper discusses the use of markets to procure ancillary services in the face of potentially higher demand for them, caused by rising amounts of intermittent renewable generation. We discuss: the nature of markets for ancillary services; what we really mean by ancillary services; how they are impacted by the rise of distributed generation; how they are currently procured; how they relate to the rest of the electricity system; the current state of evidence on ancillary services markets; whether these markets ever be as competitive as conventional wholesale energy markets, and offer some conclusions.


Introduction
Ancillary services are electricity products relating to wholesale electricity other than those traded through traditional wholesale electrical energy markets 3 .Roughly, they can be characterised as covering balancing energy, frequency regulation, voltage support, constraint management and reserves.They are related to power quality.These products 4 have traditionally been supplied to system operators at fixed or negotiated prices usually related to the opportunity cost of them being provided by traditional fossil fuel and conventional hydro generators.Over time some of these products have been available from more formal markets in some jurisdictions (e.g. for balancing energy and frequency regulation, for congestion and for reserves in markets such as PJM or GB).
Rising amounts of intermittent distributed generation (wind and solar) on the electricity network creates the potential need for more ancillary services relative to total electrical energy supplied: greater fluctuation in system frequency, more potential for high volts export constrained areas, more local network congestion (because generation is more distributed) and higher reserve requirements.At the same time, distributed energy resources (DERs) can provide more ancillary services, both by distributed generation itself being incentivised to mitigate its impacts on the system or via new technologies (such as batteries or active demand management).
The focus of this paper however is on whether markets for ancillary services, usually run by system operators5 , can ever be as competitive as wholesale energy markets?Energy markets -where these have been allowed to develop -in general are characterised by increasingly deep market arrangements which make it difficult for them to be excessively monopolised in the absence of significant market power relating to incumbent generators.Good examples of these wide area markets are across PJM in the US, the increasingly integrated markets in Europe or the National Electricity Market (NEM) in Australia.In what follows we discuss: the nature of markets for ancillary services; what we really mean by ancillary services; how they are impacted by the rise of distributed generation; how they are currently procured; how they relate to the rest of the electricity system; the current state of evidence on ancillary services markets; whether these markets ever be as competitive as conventional wholesale energy markets, and offer some conclusions.

Markets for ancillary services?
Many discussions of ancillary services start with the physics of electrical energy (see for example, Kirchen andStrbac, 2019, p.141ff andCreti andFontini, 2019, p.185ff) and how this gives rise to the unique products that make up the sorts of ancillary services that we see system operators procuring.We will come back to this, but it is instructive, for the purpose of this paper to start somewhere else.
The focus of this paper is on markets and where they are appropriate, so let us start with focussing on these.The most relevant economic paper in this context is Coase (1937) on The Theory of the Firm.This paper focuses the discussion of production efficiency on the question of the appropriate use of in-house production vs external production mediated via a market.The decision of a firm to go to the market to procure inputs to its production is a choice.Coase makes clear that intra-firm production is an alternative.What matters is which has the lower transactions costs, where these reflect the costs of assuring quality of supply.Williamson (1975) formally suggested that the decision to produce in-house production is a decision which trades off the production cost advantages of outsourcing in terms of increased scale with the transaction costs disadvantages of having to assure the external quality of boughtin inputs.Richardson (1972) pointed out that outsourcing itself can be closer to in-house production if it takes the form of an exclusive long-term contract or closer a pure decentralised market if inputs are acquired via spot market trading.
Ancillary services are something that transmission system operators could produce in-house and without use of the market, or could contract for long term with exclusive contracts or indeed procure via 'spot' markets.When we discuss ancillary services we need to bear in mind that 'spot' markets are not always desirable or efficient ways of procuring inputs, even though over time there would seem to be a tendency to make more use of them in other areas of economic procurement (there has been a general trend to market based outsourcing).
A key insight of Coase is that the capitalist firm makes the decision on how much outsourcing to do and how much keep within its own internal planning system.As such one of the key efficiency drivers of the capitalist system is that it should arrive at a more optimal privately determined mix of planning and markets.An important implication of this for ancillary services markets is the extent to which the choice of particular types of markets (or the use of an external market at all) is actually globally efficient or one mandated by the regulator/central government.
In general, market outsourcing works well when the product being outsourced is well defined and separable from in-house activities, there is sufficient demand to justify the fixed costs of market participation and the product is capable of being provided by many costefficient/innovative suppliers who can compete directly with each other.This is clearly true of wholesale electrical energy markets, it is likely to be less true of many ancillary services.Internal production works well in conditions of uncertainty about how much to procure and/or how the costs of different quality features trade off with each other.In-house production can also be a good way to manage external suppliers who would otherwise exercise market power.Indeed, in house intra-firm production is a form of private regulation (as well as planning).
Ancillary services, as we shall see, are often poorly defined as products, associated with significant uncertainties about how much to procure and may be subject to significant market power within the limited local area that they are needed (especially for voltage support and constraint management).These are traditional reasons why they have been supplied in-house or via longer-term contracts rather than via spot markets.None of the recent textbooks we looked at on electricity markets cite Coase's (1937) paper (though it is mentioned in Joskow and Schmalensee, 1983).

What do we mean by ancillary services and are they unique to electricity?
Different markets give rise to different lists of ancillary services.Stoft (2002) discusses ancillary services in the US under 6 headings: real power balancing, voltage stability, transmission security, economic dispatch, financial transaction enforcement and black start.He then lists four real power balancing services: regulation of frequency, energy imbalance, spinning operating reserve and supplemental operating reserve.Stoft's lists are all the activities of the system operator.
Economic dispatch may be an objective of a US independent system operator/regional transmission organisation (ISO/RTO) like PJM but it is a natural characteristic of any system where generators are clearly incentivised to profit maximise as they will make their least cost plant available.Most of ISOs/RTO have adopted centralised wholesale markets based on a security constrained bid-based economic dispatch model (Joskow, 2019).In Great Britain, under self-dispatch where power plants inform the system operator if they want to run, the system operator does not need to ensure economic dispatch of all power plants, the profit motive does that.Financial transaction enforcement (what is called 'settlement' in Great Britain) is important but it need not be done by the system operator -it could be done by a third party.PJM currently defines ancillary services as being only about frequency regulation, reserves and black start on its website.A longer list of ancillary services (including reactive power among others) is provided in the State of Market Report for PJM prepared by Monitoring Analytics.Only regulation and reserves (synchronised and supplemental) are provided through market mechanisms, the others are provided on a cost basis.
National Grid (2017a) defined ancillary services in Great Britain under four basic products categories that the system operator needed to procure: frequency regulation, voltage control, system security (including constraint management and black start) and reserves.These are in addition to its use of the balancing market for real time energy and the operation of the longer term (1 year and 4 year ahead capacity markets).In 2016 there were around 30 products that the system operator might be procuring.This was in addition to a balancing energy market.This has since been rationalised to only 22 (and falling).For frequency, alone there were three products: primary, secondary and fast (PFR, SFR, FFR), corresponding to the speed at which providers could respond and the length of time for which they could hold their response quantity.
A key point is that the number of defined ancillary service products being procured by the system operator is a matter of history rather than purely a matter of the nature of the product.The product in many cases is not well defined.Some authors (e.g.Greve et al., 2018, who suggest trying to have one frequency response auction with bids evaluated against each other on the basis of willingness to pay for speed and quantity of response) have suggested rationalising the number of products down and trying to reduce the capacity for arbitrary product boundary categories.The GB system operator, National Grid ESO, has reported that it is working on the rationalisation, simplification and improvement on the current balancing services (National Grid, 2017b, 2018).
As we alluded to earlier it is fashionable -among electrical engineers -to suggest the need for ancillary services is an unusual feature of electricity markets and arises from the peculiar physics of electricity.That physics involves: the need to balance supply and demand at all nodes in very close to real time; the presence of loop-flows which mean that power in meshed AC networks can flow down multiple pathways between the generator and the load; and that associated thermal, voltage and dynamic stability limits need to be observed in power transmission and distribution (see Leautier, 2019;Biggar and Hesamzadh, 2014, p.66).
It is important to state upfront that the need for firms producing a product to manage the unique chemistry or physics of the production and delivery process is not at all unusual.Nor is it particularly unusual that stability limits need to be observed and rates of change are important.Indeed, it is quite common that producer-distributors need to take actions which are unique to that industry (one thinks of the management of data packets, transportation of living tissue and aluminium production as being good examples of interesting physics and chemistry requiring adjustments to the production process), that does not make the economics of assuring quality and continuity of supply unusual.Furthermore, the actual physics of power networks is not that unusual: the public internet and transport networks share many similar characteristics.One of the great insights that drove power market reform was a scepticism (on the part of economists, such as Weiss (1975) and Joskow and Schmalensee (1983)) that the peculiar physics of electricity should prevent it being traded like other commodities.
Nor are the characteristics of service quality unique to electricity.Other network industries are not just concerned about delivering quantities of product, but care greatly about service quality which may be particular to them (and be very precisely defined).Thus financial transaction transmission networks (such as credit card systems) are interested in accuracy and speed which is every bit as significant as in electricity networks (Visa verifies a transaction in 3 secs globally).The public internet has to manage both data privacy issues and guard against pack loss.Transport networks of various types (e.g.air transport) work to passenger safety standards which are similar to the standards for delivery of electricity (i.e. more than 99.995%6 ).
Similarly, it is also true that regulatory limits on the ability to vary real time prices is not unusual to electricity.Indeed, this is a common feature of all regulated industries, that governments and their regulatory agencies limit the use of price discrimination, especially by location, for final customers.More interestingly, it is a common feature of unregulated industries that private companies often engage in self-limitation of price variation because their customers prefer this (because it engenders trust), it simplifies marketing messages and reduces the likelihood of interference from regulators.Indeed, many platform markets go further and make their products free to whole classes of customer (across multiple geographies).Thus, exposing final customers to differential prices which reflect real time network conditions or local cost variance is in general not a feature of most markets -it is mostly left for network owners to manage those risks internally.
It is also common to say that the electricity industry has been restructured in a wayseparating generation, networks and retail (and increasingly also system operation) -that gives rise to problems of coordination and requirements for market solutions which are unique to electricity.This is also not universally true: several other network industries have had forced separation of different stages of production (e.g.gas, telecoms) and many others have voluntarily separated stages of production (e.g.water, transport, financial services).
Managing a transaction boundary is common to many industries and this does not require a unique set of market arrangements for the assurance of final product quality.It is however true that certain aspects of outsourcing are forced in some electricity markets, when they could be more efficiently carried out in-house.Most industries simply carry excess capacity or curtail demand rather than use formal markets to match supply and demand and maintain quality in real time (though many large firms do use small third parties to help them manage peak demand, e.g.logistics firms).Carrying excess capacity is a way to reduce the market power of third parties who may be only required occasionally and hence take opportunities for price gouging when they can.It is also true that the real-time impacts of failure to match electricity supply and demand in real time can be more pervasive than in other sectors (though even this is not entirely clear as failures of payment and transport systems do have significant widespread effects in the short run).

Why is competition in the market for ancillary services related to the rise of distributed generation?
The rise of distributed generation is very real in some electricity markets under deep decarbonisation targets and/or where renewables are nearing grid-parity in cost.Distributed power production is expected to continue increasing significantly, with a forecast of around 110 GW by 2023 (with circa 55 GW in 2014) in Western/Eastern Europe and North America (VGB, 2018)7 .Most of these renewables are expected to be added to distribution networks.Much of these renewables will be in the form of solar and wind, which are intermittent and give rise to physical challenges which need to be managed.For instance, according to the Australian Renewable Energy Agency (ARENA)8 , more than 2m households in Australia (out of 8.3m)9 have already installed rooftop solar systems.
Rising intermittent renewables, at least in theory, may increase the demand for ancillary services, as flexibility services needed to support the absorption of renewables.Indeed, one study for Great Britain predicts that by 2030 the value of non-balancing market ancillary services will be 25% of the total value of the wholesale energy market, from 2% in 2015 (see for example Poyry and Imperial College, 2017, p.21, who suggest costs will rise from £0.6b to c.£3.9b).Evidence from the South Australian region of the NEM also suggests the importance of Frequency Control Ancillary Services (FCAS) may rise; as the table below notes FCAS averaged $3.4m per annum 2012-2014, then increased to $35m+ following the exit of the last coal plant in 2015. 10We consider balancing, frequency regulation, voltage control, constraint management (which we take as the main system security product) and reserves in turn.
Table 1: Frequency Control Ancillary Services costs in the NEM (South Australia region) Wind and solar power are subject to difficult to predict intermittency and have daily loading patterns which can be extreme (with high ramp rates).This may increase the requirement for balancing energy, which is based on the difference between predicted supply and demand.It is important not to over-emphasise this because wind and solar forecasting are improving and intermittency is not the same unpredictability, especially in aggregate for whole electricity markets.The trend to more power being traded through balancing markets will be offset if intra-day trading through power exchanges becomes more prevalent.A spot market which operates near to real time and over shorter time period windows (say 5 mins as in the NEM) will offset underlying rises in the demand for balancing energy.However, there is still much local variation which may reflect cloud cover or local wind patterns which effect particular generation facilities and affect the supply at each node.
The presence of extremely variable wind and solar generators which are not synchronised to the system frequency does mean the frequency varies more than in the past.For instance, NEM Frequency (excluding the Tasmanian region) following several coal plant closures (5000MW+) and rising levels of variable renewable energy did show a significant percentage decline in the percentage of the time it was operating inside the normal frequency range (from February 2018 to February 2019).This triggered an increase in AEMO demand for FCAS Regulation Services, from 130MW to 220MW+ in order to bring Frequency back within the 'Normal Operating Frequency Band' of 50Hz +/-0.015Hz for > 99% of time.This change was implemented in March 2019.
Fossil fuel generators connected to the system naturally adjust to maintain system frequency when loaded between their minimum and maximum stability limits.More accuracy in wind and solar forecasting can partly adjust for this and there is an issue of whether there will inevitably be more MWh required for the maintenance of system frequency within required limits -more small variations around the target frequency can be partly offset by less large frequency variations arising from the loss of single large fossil-fuel generating units.Voltage must be maintained within minimum and maximum limits at all nodes.Voltage varies when too much power/too little power is injected relative to demand.This gives rise to the need for reactive power (MVars) to be either injected or withdrawn.Voltage can vary at every node and MVars do not travel far from where they are injected or withdrawn.So the adjustment of MVars is a local service (as opposed to the maintenance of system frequency which is system wide).Distributed generation gives rise to two new sources of MVar requirements.First, when distribution systems are in low demand but high generation, they export power and voltage rises.MVars need to be withdrawn.Second, when distribution wires are built for generation they can absorb MVars when not in use, hence MVars need to be injected (or the line can be isolated).Indeed, rising demand for MVars (to be withdrawn or injected) can be mitigated by better management of the existing distribution networks.

Year
Traditionally large scale fossil fuel generators have the capacity to supply/withdraw MVars easily in the quantities required and do this at the national transmission level, whereas distributed generators have smaller and more limited ranges within which they can do this and this gives rise to more local variation in MVar requirements.Another option is optimising the operation of the distribution network assets (Strbac et al., 2018).
The intermittent nature of wind and solar power added to the distribution system exacerbates constraint management issues.This is because distribution systems are initially sized for loads and it is not likely to be optimal to re-size them for peak export of renewables, especially given that the peaks may be very peaky and coincident peaks from multiple generators interacting with local loads are rare.The willingness to pay for resizing for export is different than the willingness to pay for peak import.Load customers on distribution networks are willing to pay the value of lost load to avoid interruption.Generator customers on distribution networks are only likely to be willing to pay the market value of the electricity curtailed.This suggests rising requirement for constraint management at the nodal level and hence a role for locational marginal prices to signal where best to locate generation and, increasingly, new loads (or storage).
Intermittent renewables suggest a need for increased (capacity) reserves, in the event of a deviation from the average level of output or due to expected daily or seasonal variation in wind and solar output.Reserve markets take many forms, the most important being markets for short term operating reserve (in Great Britain) or capacity markets.Reserve markets involve paying capacity to be available at short notice rather than simply relying on short run market prices.A key question is whether intermittency, which is fairly predictable, can be best handled by short run energy prices, rather than via reserve markets?In the past reserves have consisted of paying simply the marginal cost of making existing fossil fuel plants (built for the energy market) available, rather than also compensating for the upfront capital cost of the reserve capacity (for capacity built for the reserve market).However in markets with declining demand and lots of existing fossil fuel generation capacity it is not immediately clear that higher quantities of reserves will give rise to high per MW prices for reserve capacity.Indeed, capacity market prices have been low in PJM and GB recently.
Overall, we would expect rising shares of intermittent renewables to increase the quantity of ancillary services demanded relative to the underlying total quantity of energy demand.
However, it is important to point out that this is mitigated to some extent by better renewable generation forecasting and by increasing load flexibility (e.g. from the presence of electric vehicle loads).Furthermore, the ISO demand/requirement for frequency response (and other ancillary services) is small relative to total system (energy) demand.Both of them act to reduce the demand for ancillary services from traditional providers.
It is quite another question as to whether rising demand for ancillary services means rising prices for ancillary services.This depends on the supply curve of ancillary services.Distributed generators can provide ancillary services as well as creating the demand for them.Thus, the net increase in demand (for new market players) is limited by this.It is also true that new providers of ancillary services, such as flexible loads and local storage can more easily provide the local ancillary services required for voltage and constraint management, than in the past.These technologies can provide ancillary services very competitively (with each other) and their costs are coming down significantly.At the same time, existing fossil fuel generators, faced with declining energy demand, can still compete strongly to provide them.This suggests that the estimate of £3.9bn in 2030 for Great Britain is already likely to be an overestimate (Poyry and Imperial College, 2017)

How are ancillary services procured today and how might their markets develop?
Ancillary services are typically procured in three ways.First, via a mandatory response which is required as a condition of being connected to the network.This may or may not be compensated at a fixed price or at opportunity cost.Second, via a long-term bilateral contract (between the transmission system operator (TSO)/ distribution system operator (DSO) and the service provider) for a specific service at a specific location.Third, via a market based procurement mechanism on the basis of invited bids.These can involve regular auctions typically for one or six months ahead, or real-time co-optimised dispatch.
Taking each of the five main ancillary service products in turn, we can discuss the role of markets in their provision, drawing on the experience of GB, PJM and NEM.
In real energy balancing, this is very close to existing wholesale markets for electricity.Intraday and formal balancing markets work in similar ways to day-ahead and longer term energy markets on the basis of price bids.Exposure to balancing markets is strongly disincentivised because if the market is long on generation prices in the balancing markets will be low relative to longer duration markets for generators, while if the market is short retailers buying in the balancing markets will pay more than they would have done through longer duration markets.Hence both generators and retailers have incentives to minimise trading in balancing markets.
The price series for the balancing market in GB shows that a generator receiving payment from the balancing market would on average have received significantly less than if they had sold day-ahead in the power exchange, similarly a retailer would have paid significantly more.
In addition, the size of balancing markets is small at c.3% of total electrical energy consumed by final consumers in GB.
Markets for frequency response are common in advanced markets.They typically pay both an availability per MW and a utilisation payment per MWh.GB, PJM and NEM all have frequency response markets.In the case of GB there have been up to 4 frequency response (FR) products procured through market processes.Typically, these operate on a monthly basis for one month ahead.However, in many electricity systems frequency response is compensated per MWh of actual energy injected and generators are simply mandated to provide frequency response services.A comprehensive list of European countries with information about the type of payments (availability and/or utilisation) applicable to key ancillary services can be found at SEDC (2017).
Markets are used for reserves in the leading markets, though many electricity systems around the world still make use of fixed payments for capacity.Reserve markets can be for short term operating reserve, through to longer term capacity markets.What matters for real time operation of the system are the short run reserve markets.PJM defines different kinds of reserve products, listed in Table 1.The operating reserves design involves two types: Tier 1 and Tier 2 (for synchronised reserve).However, there are some inconsistencies in the current methodology in pricing and operational requirements (Hogan and Pope, 2019).PJM recently filled at FERC a proposal to consolidate Tier 1 and Tier 2 into one product (synchronised reserve) and the adoption of a more robust Operating Reserve Demand Curve (ORDC).The list of reserve products procured in GB and Australia, along with the frequency response ones, is shown in different kinds of payments depending on the operation mode (AEMO, 2017).More recently some jurisdictions have begun to experiment with local markets for reactive power.National Grid (the GB transmission system operator) and UKPN (the distribution system operator for the London area) are currently trialling the use of auctions to supply RP to four transmissiondistribution boundary nodes in the south east of England near London.This trial aims to test the ability of DERs to supply RP via a weekly auction 12 .
PJM famously uses nodal pricing for constraint management 13 within its transmission system area 14 .These prices are calculated on the basis of the day-ahead bids/offers for every 5 minutes.They reflect the actual condition of the grid and network congestion.It is important to say that this is not a real-time auction, because the bids/offers are not adjusted in real time.By contrast in GB, constraints are managed by a mixture of contracts for constraint management and constraint payments whereby generators and loads are paid to adjust their positions to manage constraints.This is procured via tenders or bilateral contracts and the payments are determined in the balancing market (which are taken out of calculation of the balancing prices).These adjustments occur in a system where zonal annual charging for transmission gives a long run signal on which zones generators and loads should connect in.
In Australia, congestion due to constraints on transmission networks is solved via the dispatch process in a zonal (regional) pricing model (AEMO, 2018) 15 .There are five interconnected electrical regions, each one with a designated regional reference node 16 , where the regional reference price is set.An improvement of the methodology is currently under evaluation.A dynamic regional pricing approach is being evaluated by the Australian Energy Market Commission.In the case of congestion due to transmission constraints, pricing regions would be dynamically created (see AEMC, 2018b).In comparison with regional/zonal pricing, nodal pricing allows a more efficient dispatch that reflects better transmission system constraints (due to lower spatial granularity) however its implementation can be challenging (IRENA, 2019).Shifting from regional to nodal prices may also adversely affect liquidity in forward markets -at least in the short run.
Where fixed prices are still used to compensate ancillary services these prices were originally calculated to reflect the opportunity cost to a fossil generation of providing ancillary services rather than energy (if already running) or start-up costs if they were not already running.The extent to which these prices (e.g. for reactive power) still reflect opportunity cost is questionable.
In terms of charging for ancillary services these were historically procured by the transmission system operator and charged out to final electricity customers as part of their transmission charges (this was the case under the CEGB in GB).However, over time ancillary services markets have developed and charging methodologies have become more sophisticated.For instance in GB, balancing services use of system (BSUoS) charges recover the cost of balancing the transmission system and are paid by generators and retailers17 .Ancillary services may sometimes be procured by distribution companies and charged to their customers, within systems where distributed generation has become very significant.
The rise of distributed generation has put a new focus on the use of market mechanisms to procure ancillary services.This is partly because the case for market mechanisms becomes stronger in the face of rising uncertainty about where and when ancillary services will be required, but there are many potential providers.This is a good reason to make more use of markets, even if quantity and location of ancillary services, if known with certainty, could be procured more cheaply in-house or via a long-term contract.
The rise of distributed generation has also pushed distribution companies to consider market arrangements for procuring local ancillary services related to voltage service and constraint management.These could be in the form of using local markets for reactive power or extending locational marginal pricing to distribution system nodes.This is because the same trends that have given rise to distributed generation exist for storage and demand side management.Lower IT and metering costs mean that the average size of a provider of ancillary services can be reduced.However, it remains to be seen as to the extent to which distribution companies will make use of sophisticated mechanisms of procurement rather than resorting to fixed price contracts.A good example of divergence in this space is in GB where one distribution company in GB, WPD, decided to set a fixed price of £300 / MWh for constraint management (WPD, 2019), while another, UKPN, decided to only procure services via competitive tenders.

How do ancillary services markets relate to the rest of the electricity supply industry?
Biggar and Hesamzadl (2014) helpfully characterise the electricity system as needing to undertake short run operational tasks, long run investment tasks and appropriate risk management activities to supply electricity.They also point out that the system should make efficient use of generation, available loads and the existing network, in the very short run, short run and in the long run via efficient investment in generation, electricity consuming devices and the network.
Put like this the 'optimisation' of the electricity system is extremely complicated and depends on overlapping timescales and the interaction of multiple decision makers.It involves combinations of planning and markets.Overlaid with all of this is: regulation of prices and pricing methodologies (which limit both average prices and price discrimination) and government industrial policy towards both electricity investments and electricity consumers.This may favour specific generation technologies (such as wind and solar), particular locations of loads and types of consumer energy equipment (such as smart meters).Two important economic principles need to be borne in mind: Coase on theory of the firm and the difficulty of specifying the role of the market prices vs internal planning; and the theory of second best (following Lipsey and Lancaster, 1956) where introducing more price differentiation in one part of the system is not guaranteed to improve overall system welfare, it may indeed worsen it.
Markets for ancillary services sit within this complex picture and it is by no means clear that sharper short run price signals for one type of service within the system is an improvement over a more planned system, at the margin, which more carefully trades off overall system costs.For example, in the South Australian NEM region, falling levels of inertia were dealt with via a regulated solution, 3 Sync Condensers at a capital cost of $180m (perhaps $10m annual cost) -in a market which turns over $1bn per annum it is hard to imagine spot markets being worth establishing for this narrow system requirement.Indeed, the practice of cooptimisation per se which trades off more than one different market costs more carefully is not guaranteed in an n-th best world to be an improvement.Independence of different costs -which might allow market prices in one sub-area of the electricity system to not affect optimality of the system as a whole -helps but it is not pervasive in the electricity system.
A key challenge in markets for ancillary services is the trade-off between network solutions and market solutions.Building a bigger network reduces the demand for local ancillary services such as voltage support or constraint management.However, network solutions occur on different time frame to ancillary services markets.They involve assets which might last 40 or more years and are funded under a different risk-management regime (at regulated rates of return) 18 .Network solutions can be about thicker wires or about network assets (such as reactive compensators or grid connected batteries).
Another challenge is time frame.Markets work well when there is a seamless connection between short run market prices and long run market prices, when long run contract prices exist which reflect the time series of short run prices.This can create financial instruments which allow investors in assets which play in short run markets to hedge their risk.This would seem to be important for ancillary services which are by their nature subject to highly volatile demand/supply and pricing in the short run.Some system operators have had to create specific markets which hedge this risk: such as those for financial transmission rights (to hedge against the use of nodal prices to manage short run constraints in the transmission system) and capacity markets to create long run markets (4 years+) for reserve capacity.One observation is that these long-run hedging markets are very patchy across ancillary services.They generally do not exist for frequency response), and not at all for voltage support.
Even in markets for electrical energy the existence of long run markets has been an issue.Such markets do not exist for power for more than 3 years, or if they do the product is rarely traded.Instead most generators have chosen to hedge longer-term by integrating with retailers directly -another Coasian solution to how to organise production efficiently in the presence of a lot of uncertainty about the future supply/demand balance and outturn prices.Thus, something like the classic Schweppian (Schweppe et al., 1988) insight that nodal prices can help manage short run network congestion (popularised to great effect by Hogan, 1992) must be put in context: it is only signalling one among many network characteristics and may not give rise to an overall efficiency improvement in the absence of the pricing of all other network characteristics and the fact that only parts of the network are priced and only some network connected loads and generators are exposed to these prices.A system of constraint management that was more centralised, could give rise to better long-run investment in constraint reduction could easily be more globally efficient.
Rising distributed generation may exacerbate the global optimisation problem.It does this by giving rise to much greater fluctuations in the use of the network than in the past and hence the need to trade off different investments in load flexibility, generation and networks more carefully.With large transmission level generators and steady load growth the required investments are less subject to trade-offs, especially between longer run investments and short term ancillary service markets.Good examples would be the requirement to manage intermittency, which could either involve increasing market interconnection (over a wider area) or local solutions which could be on the demand or supply side (or even markets for 'fast ramping' to deal with the Duck Curve issue arising from the rapid decline of solar generation in the evening).This suggests many more trade-offs across multiple overlapping scales and timeframes.The idea that 'just' introducing a short-term local market is the solution to this problem is to limit and direct the solution in a particular and not-necessarily optimal way.
7. What is the evidence on the impact of rising distributed generation on markets for ancillary services?

Cost, price and quantity impacts
In Europe the share of renewable energy in total electricity supply has risen from 15% to 30% in the last 10 years.It set to rise again to perhaps 55% in the next 10 years (see Chyong et al., 2019).In some countries the rise has been even larger.For instance in Denmark, Germany, Ireland, the UK and Australia (inter-alia) the share of renewables in electricity generation 19 .has risen sharply (with around 60.3%, 34.4%, 30.1%, 28.1% and 16% respectively by 2017).In PJM the share of renewables is still low.For instance by 2016, the share of PJM's installed capacity consisted of 33% coal, 33% of natural gas, 18% nuclear and only 6% renewables (including hydro)20 .Much of the new renewable capacity has been in distributed generation (especially in the UK and Germany) and a significant amount has been in non-dispatchable renewable generation (though biomass remains a significant share of the rise in renewables).
However the actual impact so far on the demand for ancillary services has been modest.
In GB the growth of the cost of ancillary services has been modest at perhaps a rise of 25% over 5 years nominal terms (much less in real terms).This is not quite the full picture of the impact of renewable electricity.We have seen the introduction of a capacity market (whose value is now £8.40/KW) 21and there has been significant network reinforcement to support renewables, in particular, a large investment in interconnectors with other countries.It is observed that balancing costs (exc.transmission constraints) have increased by around 12% (annual figures) between 2013/14 and 2017/18, in real terms, see Figure 1.In PJM, we observe that the annual average price of ancillary services has increased around 1% (annual compound average, real) for the period 1999 and 2018, see Figure 2. Reactive power ancillary service is getting more relevant than frequency regulation, especially in the last 5 years.In contrast with GB, PJM co-optimises regulation with reserves and energy in order to minimise the cost of the three products.AEMO also uses co-optimisation techniques.In contrast with PJM and GB, in NEM ancillary service costs have increased materially especially in the last two years.Looking at the figures it is noted that this increase in mainly driven by the upward trend in frequency control costs (FCAS contingency followed by FCAS regulation).Figure 3   Looking behind the figures in more detail we find that the quantity of ancillary services being procured has been rising, but that prices have been moderating (i.e. in line with competitive responses, including from battery storage).
A clear picture that emerges, so far, is that the implications of rising amounts of renewable generation for ancillary services is modest.This is not to say that better use could not be made of ancillary services markets: maybe there has been too much network investment and not enough competitive procurement.

Analysis of markets for ancillary services
There has been relatively little literature on the analysis of markets for ancillary services.Most of the current literature is focused on technical aspects related to the integration of intermittent resources and their capability to provide ancillary services (including those from distributed generation such as solar PV, wind) especially linked to network operation, optimisation methods and algorithms, quality issues and planning.Anaya and Pollitt (2018) provide further details about specific technical studies related to ancillary services.Only a few of them consider competition in ancillary services.For instance, Rebours et al. (2007) evaluate the economic features of eight system operators in their procurement of frequency and voltage control ancillary services.They identify the different methods for procuring those services and identify common features among the different jurisdictions, observing a lack of competition for some of them such as voltage control.Zhou et al. (2016) perform a survey of ancillary services (with a focus on frequency regulation and reserves only) in the ISOs/RTOs from the US.They find common trends among system operators and identify that frequency regulation are the ones with the highest price following by spinning and non-spinning reserves.They identify also PJM as the one with the largest market for frequency regulation (by revenue and capacity) and for spinning reserves (by capacity).discusses the ancillary service markets in the US and BRIC countries (Brazil, Russia, India and China).They explore the main issues that each of these markets is facing in order to improve competition (i.e.weak legal systems, poor ancillary service market design, and other nonenergy related factors).They find that India and China are at the developing and exploratory stages, respectively.

Innovation
There would appear to be significant innovations taking place around ancillary services and a lot of promise arising from new technologies that can help with better energy forecast (consumption and generation) such as artificial intelligence and associated machine learning and deep learning.In fact, different electric utilities (e.g.Duke Energy, Southern California Edison, First Energy) are already using these in order to solve business challenges (including customer engagement applications and forecasting) 22 .The addition of more distributed energy resources (i.e.distributed generation units) within the electricity system and the opportunity to trade flexibility services (i.e.balancing services) from them, would add more complexity to the electricity network.This could be reflected by the trading of new products, new market participants (i.e.aggregated DERs including storage), need for better forecasting, the availability of more data to manage and process and further coordination between parties23 etc. Blockchain is among the technologies that can help to improve short run operation in the presence of smaller intermittent generation by shortening the small scale transaction validation time.However, some limitations need to be taken into consideration too especially for real time transactions (DENA, 2019).All these developments are being driven by rising distributed generation.
There are currently several initiatives (in the form of projects, start-ups etc.) that are trading with DERs in the provision of different ancillary services and the number is rising.Many of them use one or a combination of the technologies described previously.Kufeoglu et al.
(2019) identify a list of 40 energy-related start-ups in order to explore their business model.They consider key business model dimensions (value proposition, value creation, value revenue).They find that around 35% of them report having utility partners and many of them use blockchain technology.Having these kinds of partnerships may increase the chance of better financial support (in fact, many of them have received customer based funds) and reduces regulatory obstacles (the use of a regulatory sandbox is also observed here, especially in the UK).
Based on this, it is expected that the procurement of DERs, using less conventional approaches (such as the ones procured via DSOs, local energy communities, among others) could increase over time.The Open Networks project of the Energy Networks Association in the UK suggests that 270 MW of flexibility services were procured by distribution companies in 2017 rising to 800 MW+ in 2018 (ENA, 2019).

Who will run future markets for ancillary services?
A major question going forward is who will be responsible for future ancillary services markets and the division of responsibility between the transmission level system operator (TSO) and the distribution system operator (DSO).In Europe there is a further question of the division of responsibilities between national system operators and pan-European system operators.
Energy balancing, frequency regulation and the provision of reserves are best conducted at the transmission system level.In Europe, there is scope for having pan-European or regional markets for these within the framework of the European single market for electricity (see Pollitt, 2019) which is largely complete for day-ahead wholesale energy.
For voltage support and constraint management this will increasingly need to take place within distribution systems driven by the rise of distributed generation.At the very least markets will need to be organised at this level.It is another matter whether they should be run by the existing transmission system operator as an extension of their existing markets or whether they should be a new service provided by the distribution system operator or whether an independent entity would be the most suitable for this role.
The fundamental issue here is what is the value of extending the role of the ISO/TSO versus extending the role of the DSO?This is a Coasian question and depends on the relative efficiency of different types of firms in different jurisdictions.In many systems DSOs are small and lacking in market management experience (see Kufeoglu et al., 2018), while TSOs are the source of expertise in running formal ancillary services markets.The voltage boundary between TSOs and DSOs is a historic accident and does not reflect what might be optimal for system operation: indeed, that is one of the reasons for the move towards ISOs, which can reflect better the mathematical optimisation of the system without worrying about the underlying ownership of the network assets.On the other hand, DSOs have relationships with their customers directly connected to the distribution system and can work with them to physically facilitate their participation in ancillary services markets.
A reasonably sized DSO can be sophisticated enough to run off the shelf ancillary services procurement software.There is no doubt that some unusually large distribution companies such as Enedis -the national distributor business of EdF -in France or ENEL Distribution in Italy could.However, one might expect that medium sized private distribution companies such as ConEd in New York, UKPN in London and Ausgrid in Sydney could run local markets for ancillary services at the same time as their transmission level counterparts NYISO, National Grid Electricity System Operator or AEMO who run market wide their balancing, frequency regulation and reserves markets.Indeed, there would be some regulatory benefit to setting these DSOs up in competition to their transmission level counterparts as procurers of ancillary services.
However, the eventual division of responsibilities is an open question given the nature of economies of scale and scope in running markets and the linkages between markets (the cooptimisation question).Having markets coordinated centrally has some cost advantages in terms of driving IT costs down, though it potentially reduces innovation and increases IT risks.It also might be preferred by market participants who may themselves want to participate in several markets simultaneously.Indeed, one solution for a storage facility is to sign one long term contract with the system operator and let the system operator decide how to operate the facility and for what service in real time (something which was possible under the DRAM procurement scheme in California).The alternative of leaving individual DERs to optimise their own bids across multiple markets being conducted at multiple scales is challenging for the DERs.
One new possibility afforded by local procurement of ancillary services by DSOs is the ability to co-optimise between regulated network investments and DER ancillary service solutions.
Assuming that the regulatory regime rewards this (as it intends to under RIIO in Great Britain and REV in New York), this should allow the DSO to offer a contact to a DER to provide constraint management or voltage support in lieu of upgrading its network.This might wholly or partially finance the DER.

Will ancillary services markets ever be as competitive as wholesale energy markets?
In considering the prospects for ancillary services markets, it is interesting to ask will they ever be as competitive as wholesale electrical energy markets?
The simple answer would seem to be no.This is important because we should be careful to suggest that we need to rely on them more in the future.
There are a number of elements to this.
First, ancillary services markets are largely constructs of the system operator.This means they are subject to a certain degree of arbitrariness (in defining maximum allowed speed of response or minimum size participation criteria24 ) and a lack of transparency around the price formation process.It is difficult to construct a price and quantity series for national ancillary services markets and some prices are subject to confidentiality (e.g.constraint payments to generators under negotiated contracts in GB).While one can find information on total costs of ancillary services for GB it is more difficult to see how they were arrived at -the division of payments through the balancing mechanism for balancing and non-balancing services is decided internally.National Grid have begun to rationalise their markets and this has led to some products being removed suggesting that unlike energy markets there is a risk that DERs relying on certain types of payments in the future are subject to the fundamental risk that their offered service might simply not be procured in the future.
Second, unlike markets for energy, ancillary services markets are not usually two-sided (i.e. a market where both supply and demand are competitively determined).Thus prices do not arise as the interaction of modelable supply and demand.Those who create the demand for ancillary services are not those who pay for it when it comes to frequency regulation, voltage support, constraint management (even in PJM most loads are not exposed to nodal prices) and reserves.This introduces the further arbitrariness that these services can often be over-procured by the system operator or subject to bargaining back by network companies whose own incentives to build network or invest in other regulated assets (such as reactive assets) may impact on the quantity of ancillary services demanded.The capacity market is a classic example of this where there have both been cases of over-procurement of capacity (in the early years of the GB market) and bargaining back by US state regulators approving new plants to reduce capacity payments or the GB system operator uprating interconnectors (in recent years) to reduce capacity market prices.
Third, markets for ancillary services will be risky for investors, raising the cost of capital and encouraging a particular sort of DER response (i.e.ones with low capital cost).These investments will compete poorly with network owned assets (the in-house solution) and network solutions in terms of value for money for customers.
Over time, each of these limitations could be partially addressed.Rising amounts of distributed generation both raise demand for ancillary services and provide competitive sources of ancillary service provision, however some of the core issues which limit the competitiveness of short-term non-energy ancillary service markets25 will remain.

Some conclusions
Ancillary service markets are currently small relative to markets for day-ahead and longer term wholesale energy.The rise of intermittent renewable energy would seem to increase the quantities of ancillary services required, however this is not quite the same as meaning that the value of ancillary services will become significantly larger in the future than it is now.Better wind and solar forecasting, combined with sharper real time energy price signals will offset the need for more frequency regulation, voltage support, constraint management and reserves.The presence of electric vehicles also offers significantly more potential for managing the electricity system relative to now and could provide a cheap source of ancillary services in the same way that fossil fuel power plants have done traditionally.
The overall optimisation of the electricity system involves the interaction of ancillary services, wholesale energy and networks and is not dependent on the nature of the market for ancillary services.Ancillary services trade off with wholesale energy and network operation and investment in ways that may mean that co-optimisation or internalisation is preferable to a stand-alone spot ancillary services market.The Coasian question of the optimal allocation of activity between in house and market transactions requires relatively more attention than Schweppian question of how to increase optimal decentralised price signals.
Ancillary services markets are limited by the overarching role of the system operator, whose monopoly purchaser position acts to create investment risk relative to wholesale energy markets.
Regulators need to carefully evaluate changes to ancillary services procurement, given the split incentives which exist for them to be acquired in a system optimal manner.The question of whether they are best acquired in the future by the transmission level system operator or the distribution system operator is an open one for voltage support and constraint management.
There is, however, considerable scope for innovation in the provision of ancillary services and the targeting payment for them.At this stage, there remains much room for experimentation globally as shares of distributed generation rise on the electricity system.

Figure 1 :
Figure 1: Trend of balancing costs in GB (exc.Transmission constraints)

Figure 2 :
Figure 2: Ancillary services average cost per component in PJM (USD)

Figure 3 :
Figure 3: Ancillary services costs per component in the NEM

Table 2 :
Anaya and Pollitt, 2018)optimisation with wholesale energy markets is observed in the PJM and NEM markets(PJM, 2018: AEMO, 2015).Comparison of different frequency response and reserve products procured in NEM, GB and PJM Voltage control markets are underdeveloped (seeAnaya and Pollitt, 2018). Tyically, reactive power must be provided locally and has been managed through mandatory requirements to provide it at fixed prices and a reliance on network owned reactive assets (such as shunt reactors).GB entirely procures reactive power (RP) at fixed prices (OPRS), with a parallel auction process (EPRS) having withered away to no useable offers.PJM compensates generators for RP capability based on American Electric Power (AEP) methodology and pays opportunity cost.NEM makes some use of a national RP market and participants may receive In PJM two types of automated signals are generated for regulation: Reg.D (fast response), Reg.A (slow response).Source: AEMO (2015), NGESO website (Balancing Services), PJM website (Ancillary Services) depicts this trend.