Background

Let’s recall a few basic facts and then apply them to DER telemetry. There is math behind all this but as usual we will not need to solve any equations. The shy genius Claude Shannon1 provided the foundation of information theory and telecommunications with (among other things) his definition of information entropy:

Basically, this equation quantifies the “disorder” of a data set (its entropy - essentially the uncertainty in the representation of the information about a random variable). It quantifies the average number of bits (when using base 2 logarithms) required to encode the state of the underlying random variable.

We are going to adapt this idea as the basis for some very practical grid architectural structure. If a data set has more bits than the number needed to represent the actual information, then we have an opportunity to reduce the data volume while preserving the information. The data reduction can be done using data analytics, which are defined as data transformations that reduce Shannon entropy. In practice, this means aggregating2 raw (incoming) data and then extracting the relevant information, which results in a data volume reduction (see Figure 1). Note that we are not talking about data compression since there is no intention to expand back to the original data set.

This is not really very mysterious in practice. Consider, for example, converting a set of AC voltage samples into an RMS voltage value. We may use, say, 256 16-bit samples per cycle for two cycles of an AC voltage waveform to create one 32-bit value of RMS voltage per cycle pair. The original data set contains the information about RMS voltage, but spread over 8,192 bits, whereas the extracted RMS value needs only 32 bits to represent it. By aggregating samples, the uncertainty in the voltage RMS value is reduced, resulting in decreased entropy, allowing for a reduced data volume representation of the extracted information (calculation of the RMS value in this case). We can think of an analytic as having two parts: aggregation and extraction, although they may both be accomplished in a single step.

Starting from this basis, we can treat DER telemetry data flows in an organized fashion for purposes of scalability and also define key functionality of grid telemetry interfaces.

In fact, applying Laminar structure combined with the entropy reduction principle yields a number of useful architectural and systemic insights, as Figure 2 above illustrates. Not only are DER telemetry data volumes affected as functions of infrastructural scale, but so are telemetry statistics3 and system dynamics. Notice that this logical dataflow structure mirrors the Laminar coordination structural command disaggregation model, albeit with flow in the opposite direction.

Application to High-DER Power Grids

So how does this help us deal with DER data flow scaling? If we apply the entropy-reducing data analytic concept combined with Laminar coordination, we can devise a multiscalar structure for DER telemetry management that delivers information to each level as needed without the scaling problem of centralizing all the raw data. See Figure 3 below.

This structure determines the locations and essential functions of ten interface types, as well as six analytics stack types that support the interface functions and thereby support the distributed DER telemetry functionality.

From the bottom up in Figure 3:

Level 1- Domain Concentrators

The Domain Concentrators perform entropy/volume reduction on the telemetry from the devices within their domains (premises, microgrids, etc.) and present the resultant information to the appropriate Feeder Coordinator. Each Domain Concentrator has one northbound interface, one standardized analytics stack (per domain type), and any southbound interfaces necessary to connect to the various devices and systems that may exist within each domain.

Level 2/3 - Feeder/Substation Coordinator Nodes

The Feeder Coordinator performs entropy/volume reduction on the telemetry from the domain concentrators in its service area and presents the resultant information to the Substation Coordinator. The Feeder Coordinator has one northbound interface, one standardized analytics stack, and sufficient instances of its southbound interface to receive the information from as many Domain Concentrators as necessary.

The Substation Coordinator performs entropy/volume reduction on the telemetry from the feeder coordinators in its service area and presents the resultant information to the DSO. The Substation Coordinator has one northbound interface, one standardized analytics stack, and sufficient instances of its southbound interface to receive the information from as many feeders as the substation has.

Level 4 - DSO/SO

The Distribution System Operator, in conjunction with the host Distribution Network Operator, performs entropy/volume reduction on the telemetry from the substation coordinators in its service area and presents the resultant information to the System Operator. The DSO has one northbound interface, one standardized analytics stack, and sufficient instances of its southbound interface to receive the information from as many substations as there are in its distribution system.

Of course, the System Operator has sufficient instances of its southbound interface to receive the DSO information from as many DSOs as there are in the system. Also, at each level, coordinators may exchange aggregated information with its peers.

Summary

The model of Figure 3 defines a multiscalar analytics structure, based on Laminar coordination framing and entropy reduction. The set of six analytics stack types defines a telemetry analytics superstack. Each of the interface types and analytics stacks can be standardized and used across the entire power system. This structure forms a multi-layer platform, which provides DER telemetry scalability, system decoupling and resilience (see also the Layering and Platforms discussion in Section 5.2 of the Grid Architecture Cookbook). The approach described above is used now to some extent, but it can be formalized into an actual architectural approach that determines structure, components (concentrators, coordinators, and analytics stacks), interface locations and functions, and also informs standards. This layered, distributed structure is inherently more resilient and scalable than the massive centralized database/data-mining approach.

Caveats

This model is simplified in that it does not show the effects of aggregators/VPPs and cloud services of the equipment vendors4 but these can be accommodated. More about that another time.

The model describes idealized logical telemetry flows, not physical network data packet flows. Physical communications networks and data flows do not necessarily comport with these diagrams and that is both bad news and good news. The bad news is that communications connectivity varies considerably from location to location and can be quite unreliable for a number of reasons. Actual packet flows over real communications networks can take time-changing variable-latency paths that can be much less direct that the idealized figure above shows. Electric utilities may own and operate some distribution communication networks but may rely upon commercial internet for connections to residential and commercial/industrial premises. Consequently, the communications and computation platforms for feeder coordinators may5 or may not exist on distribution grids.

The good news is that virtualization can be used to provide the feeder coordinator functions either within the substation in conjunction with the substation coordinator, or at some other location, provided the communications networks can support them. More about that later, too.

Given the above and looking forward, it is useful to be clear on a number of distributed system and distributed control concepts. Integrating existing concepts like hidden coupling, Laminar coordination, and ugly loop avoidance with material presented here yields some strong new architectural principles.

Definitions

Capability - the ability to perform certain actions or achieve specific outcomes.

Functionality - the related group of tasks, operations, or services (functions) that a grid can supply or carry out. Functions combine to implement capabilities.

Component - uniquely identifiable, non-trivial, nearly-independent device, individual, organization, organism, element, building block, part, or sub-assembly. Components combine through structure to form systems that have functionalities and therefore capabilities.

System - a set of interacting, interrelated, or interdependent components working together to form a unified whole with a specific purpose or goal. Note that this is a potentially recursive concept.

Entity - in the Grid Architecture work, a person or organization of persons (such as a business or regulator). This term has many definitions in other fields but those are not helpful in this context.1

Centralized system - a system where data flows to a central hub where all logic and control are determined; commands flow back to individual elements from the hub.

Decentralized system - a system where multiple separate entities operate independently with at most some small amount of supervision.

Distributed system - a decentralized system where the parts cooperate to solve a common problem - this implies some form of communication among the parts.

Coordination - the means by which a set of decentralized elements cooperate to solve a common problem, thus becoming a distributed system.

Transmission/Distribution Coordination (TDC) - in power grids, the systematic operational alignment of utility and non-utility assets to provide electricity delivery.2

Complexity - the state of having many interrelated, interconnected, intricate, or interwoven elements and interfaces.

Scale - size, extent, scope, or range of any quantifiable aspect or property of a concept, entity, behavior, effect, object, component, or system.

Multiscalar structure - a structure that has substantially the same characteristics at any scale.

Background: Centralized, Decentralized, and Distributed Systems and the System Triangle

There is a view that centralized and distributed systems are polar opposites. Not correct - the system triangle below illustrates the actual relationships of the system definitions listed above.

Real systems do not necessarily reside at any one of the three corners. They tend to be more complicated than that, and so actual systems may be located somewhere inside the triangle. Also, complex systems such as power grids have multiple component sub-systems and these may have different locations in the triangle and may even be nested. A system may be distributed but include an apparently centralized part (like a control center) but this does not make it a centralized system. The control center can be one of the nodes in a distributed system, as can a cloud resource. The same is true for real time electricity markets: the part that is usually thought of as the market, namely the part that collects bids, facilitates trades, and performs market clearings (market place, as opposed to market system), is normally centralized but that does not make the whole market system centralized. It is necessary to examine functions, dataflows, coordination, and control structures to determine system type.

Principle: Use the Triangle Interior

We can use the triangle concept in the architecture and design of complex systems. By choosing where various subsystems are located inside the system triangle, we can specify a system that accounts for various real constraints, whereas a single point (triangle corner) model would not be able to do so. In the DER integration case, this is not just possible, it is necessary (more about that in future postings). Think of the entire perimeter plus interior of the triangle3 as bounded continuous degrees of freedom for structuring grid control systems. Place systems anywhere on or inside the triangle as appropriate.

Background: Coordination in Control Engineering

The diagram below is a rough control/coordination taxonomy applicable to electric power grids.

In classical control theory, coordination is a subset topic of distributed control. Individual control elements are coordinated to cooperate on solving a shared control problem. Such systems typically use one of three distributed control coordination methods: state sharing, goal decomposition, or model decomposition. For Grid Architecture, we switch this around a bit and make coordination supervisory.

Principle: Inversion of Coordination and Control

In the Grid Architecture discipline, we invert the distributed control/coordination principle and treat control as a subset of coordination. This inversion gives grid architects a better way to deal with grid operational coordination elements that reside outside of the grid control systems. We focus on the layered decomposition approach to coordination because it yields Laminar structural models we can apply to the system architecture as a whole. We could try to apply layered decomposition to the total system all the way from the system operator to individual devices, but this is not practical in the near term. The best immediate use for Laminar structure is to coordinate the entities (including DSOs and aggregators/VPPs) involved in DER integration. In other words, TDC.

While Laminar coordination is helpful for TDC in the short term, this still leaves us with a partial system control problem, and the gap becomes crucial right in the range where scale concerns crop up, namely, as we approach the distribution edge. Laminar structure will help us there too, but more about that later.

Background: Grid Scales

Consider three types of scale for power grid control systems4 (the figures below apply to the US power systems):

Temporal scale reaches from microseconds to days (12 orders of magnitude) or, if we include planning, to years (15 orders of magnitude). The chart below was developed by A. von Meier.

Geographic scale reaches from meters (connection point service drops) to service areas for bulk power systems operators and reliability coordination (about 7 orders of magnitude). The chart below was derived by me from one created by A. von Meier.

Infrastructural scale refers to numbers of grid elements and reaches from as low as one or a few regional interconnections to hundreds of millions of edge connection points (about 8 orders of magnitude).

How do we deal with such wide-ranging scales?

Principle: Scalable Structures

Use multiscalar structures to deal with grid scales. Multiscalar structures look more or less similar (but not identical) at any grid scale when viewed graphically and more importantly, the inherent properties of the structures work the same way at any grid scale. Multiscalar structures provide conceptual integrity (similar problems are solved in similar ways at any scale), provide re-usable structure templates, and have intrinsic scalability.5 In addition to previously discussed coordination structures, scalable data flow structures and distributed computation structures can be created methodically as well.

Significance for Grid Control Architecture

Here are three principles to add to your grid architecture toolkit:

1. Control is a subset of coordination. Use coordination structure to organize TDC and distributed control.

2. Coordination/control structure should be multiscalar on one or more of three grid scale dimensions: temporal, geographic, and infrastructural.

3. The system triangle is a continuum. Let go of the one-corner view; use the degrees of freedom implied in the system triangle interior to specify system types.

Electric power systems organizations and product vendors must provide new functions to meet new requirements but cannot change everything at once. It is imperative to make incremental no-regrets near-term changes that will become integral to the future state grid. The three principles listed above are part of the Grid Architecture tool set that helps you get you there.

More to come.

The Good

Control systems routinely use negative feedback loops for regulation and stabilization. While the negative feedback principle has been around for a long time in various self-regulating arrangements (flyball governors for steam engines and chimney turbines for air flow and temperature regulation, for example), the use of negative feedback in electronic systems really commenced with the invention of the negative feedback electronic amplifier by Harold Black in 1928. Using feedback for stabilization and control is a foundational element of control engineering.

Feedback loops are used by design in the control of bulk power systems (see Fig. 1). In fact, in a conventional grid control system, there are many nested feedback control loops. It is a load-following power balance regulator.

Primary dispatchable generator control consists of three (non-nested) feedback loops (see Fig. 2).

Secondary generator control uses the PI portion of the venerable PID control for the balance function called frequency regulation (see Fig. 3). This loop updates on a 4-second or 6-second cycle (depending on the balancing authority).

System imbalance (tertiary control) and scheduling are actually implemented as model predictive receding horizon controls in feedback loop form. This is true even when real time markets are used as part of the dispatch mechanism. Tertiary control typically operates on 5-minute and 15-minute cycles. Scheduling typically operates on day-ahead, intra-day, and hourly cycles, using 1-hour time steps (see Fig. 4).

Bulk power system stabilization with power electronics uses feedback as well.

At the distribution level, various feedback loops are used for Volt/VAR regulation, stabilization (D-STATCOM), and synchronization (see grid-following inverters).

The point is, these are all designed. These grid loop structures are what we mean by the Good. Next, we head into darker territory.

The Bad

On the other hand, there may be control loops in a power system that are not designed. These arise from hidden coupling at the distribution level when distributed energy resources (DERs)1 are being managed for grid operations. This doesn’t just result in feedback loops, it can cause complex, interconnected loops. Figure 5 illustrates several ways that hidden coupling can occur. Part of the coupling is always electrical but the rest is in how control processes are connected to grid edge devices.

The presence of DER aggregators can be problematic if the aggregators can have interpenetrated service areas. In such cases, the aggregators (each of which is a controller for some DERs) end up interacting via hidden coupling.2 The DER coupled controls also interact with the other existing control loops, such as Volt/VAR regulation and the control loops of the bulk power system. Figure 6 shows several control dependencies, any of which can be created through hidden coupling.

Control dependency can result in controllers interacting with, competing with, or even outright interfering with each other. Not good for distribution reliability.

This is what we mean by the Bad. Finally, we head into the Land of Naïve Compromises.

The Ugly

On the other other hand (ok, running out of hands here) feedback loops can arise in a rather insidious manner. This occurs when processes involving multiple grid entities are created without regard for structural concerns. We see this most often in the creation of hybrid or ad hoc DSO arrangements if edge resource aggregators are involved with wholesale market operations.3 Due to the large number of endpoints being controlled and the number of entities involved, loop structures can arise through the combination of entity interconnections and process flows.

This can be very hard to detect, due to organizational complexity.

These loops create the possibility of inadvertent feedback; also, passing coordination through processes in multiple entities (see Fig. 7) introduces latency stacking. The combination can easily lead to instability, as we have seen.

And of course, all of this is inside the outer loops shown in Figure 1.

If the update rates are slow (such as with day-ahead demand response), then loop instability problem may not manifest. But if the update rates are fast (15-minute cycle or faster) then feedback loop dynamics can cause unintended behavior. This has shown up as price oscillation in transactive system field trials, and oscillation of limits on PV power injection in distribution grid flexible connection arrangements.

Now that’s Ugly.

What’s a Grid Architect to do (or not do)?

First, what not to do:

1. Don’t try to smother the problem by tightening down injection limits. The extreme curtailment involved will leave a lot of DER value untapped.

2. Don’t try to dampen oscillations by introducing ad hoc low pass filters. Either use a rigorous control engineering approach (impractical, to say the least), or better, avoid non-essential loops in the first place (see below).

Instead, do this:

1. Use architecture that enables the things the grid is supposed to do, and precludes the things it is not supposed to do. Do this by applying structure devoid of tier bypassing, coordination gapping, and hidden coupling, as well as inadvertent loop formation (see Fig. 8). An appropriate structure comes from the Laminar framework developed from layered decomposition (see Section 8.3 in the Grid Architecture Cookbook). Note that using the proper structure has significant implications for entity roles and for interfaces. Other structures that comport with Laminar form are possible.

   

2. Employ coordination methods/algorithms that ensure that no coordination gaps or tier bypassing exist, and that account for aggregator interactions. In other words, actually coordinate all of the assets, entities, processes, and controls, just as you would coordinate protection.

Final comment

Now you see one reason why control theory is included in the GA stack pack.

This starts out looking hard, but stay with me. It turns out to be easy and practical.

Layered Decomposition

This is actually a solution technique for optimization problems involving many coupled constraints. It involves decomposing the problem into subproblems, and then using a recursive iteration process to obtain the solutions to all of the subproblems. When rolled up, this approach converges to the solution of the original large optimization problem. Figure 1 illustrates the decomposition concept.

Decompositions may be done multiple times, and do not have to be done on all subproblems. Decompositions have two forms:

1. Primal decompositions produce coordination signals that behave like resource allocations.

2. Dual decompositions produce coordination signals that behave like prices.

Decompositions may be mixed from layer to layer. Each layer iterates with the layer below it, and so on recursively. While it might seem logical to try to formulate a grand central DER coordination problem this way (and many have tried), there are two overwhelming drawbacks to doing so:

• Convergence - convergence rate depends on both the nature of the decompositions and the structure of the resultant decomposition structure including both horizontal fan out and depth. Convergence can take a very large number of iterations, although there are some ways to speed that up.1

• Input data requirements - formulations for coordination on a real distribution system require vast amounts of real time data from the distribution system. As the Boston School found out,2 this is a massive roadblock to using this approach to calculate coordination controls, as most distribution systems in the US do not provide such data. In fact, many electric distribution system do not even have distribution SCADA. Others have some but have significant observability gaps, which is why I worked with a team to create an observability evaluation tool.3

Given the above, I do not recommend trying to use this for creating distribution control algorithms.

There is a better, more practical way to use the layered decomposition idea.

Laminar Structure

If we examine the representation of the process in Figure 1, we can identify an underlying structure. Looking at the problem/subproblem decomposition model, we can identify a basic element that occurs over and over at every level. Decompositions are composed of structured arrangements of this basic building block (see Fig. 2).

Using the primary coordination domain structure as a building block, and using the problem-subproblem decomposition form as the system structure generator, we can build Laminar Coordination Frameworks that can map to any distribution system. The structure is recursive and multiscalar.

We say the mathematics induces a structure. This is a core axiom of Grid Architecture.

Such a framework has a number of valuable properties:

• Extensibility – the composable nature of laminar coordination domains means that a framework can be made to fit an existing grid structure, can be built out incrementally, and can be extended incrementally when grid structure changes.

• Boundary deference – the decomposition method and composability of coordination domains enables the creation of an interface wherever one is needed to accommodate a system or organizational boundary.

• Local objective support (selfish optimization inside global coordination) – by introducing additional objective terms at any particular coordinator node, local objectives can be integrated into the overall solution. This is a form of goal decomposition (from distributed control theory).

• Constraint fusion – by adding in constraints as needed at any coordinator node, local constraints can be accommodated in a distributed fashion.

• Scalability – since coordination signals do not need to aggregate up or down the coordination chain, no communication scalability issues arises due to depth of the coordination chain. Layered decomposition can be used to create new layers as needed if the southbound fan-out for any particular node becomes too large, thus providing structural scalability.

• Securability – the inherent form of the coordination framework and consequent coordination signal flows provides a degree of regularity that supports signature and traffic analytic security measures much more so than arbitrary networking for unstructured coordination schemes.

• Flexibility – since primal and dual decomposition can both be used, both resource allocation and price allocation approaches can be accommodated as appropriate.

So, how might we use this?

Figure 3 shows an example of a Laminar coordination framework mapped to a distribution system in simplified form.

In the figure, the coordination domains are indicated inside red boxes. The (logical) inter-domain communication buses provide inter-layer and inter-domain data exchange, while intra-domain communication buses provide data flows within coordination domains (see Fig. 4).

The node at the very top of the framework does not need a northbound interface and terminal nodes at the bottom do not need southbound interfaces. Complete Laminar coordination frameworks could be assembled in this way. Could be, but won’t be.

“But wait, there’s more!” as the Ronco guy used to say.

There is a much more practical way to use all this.

Practical Application to Electric Distribution Grids

The foregoing looks like it is starting to involve a lot of technology, right? Stuff that isn’t standard utility systems or equipment. After all, aren’t DERMS and some interface standards enough? Well, yes and no.

Being neglectful of structural issues when integrating distribution grid edge-connected resources can lead to a serious problem: hidden coupling. You can read about it here and here. It can also lead to looping and excess latency accumulation.

The solution to these issues is to prevent them from occurring, using…

you guessed it…

Laminar coordination structure!

Now, we don’t want to go into a whole new layer of technology for this, but it turns out we don’t have to. We can use the concepts from above to derive some extremely practical rules that are easily applied to real distribution operational coordination problems.

So here is the basic Laminar structure template:

The lines show how elements are connected. Note that data and control flow along these lines only. Sticking to this structure framework rigorously will automatically prevent hidden coupling and loops from being created. Using this form as an architectural constraint eliminates accidental creation of hidden coupling or looping.

We could do this down to every single device on a distribution system, including infrastructure devices as well as edge-connected resources, even down into individual devices inside premises. But that would be impractical, so don’t try to go there.

The best use for Laminar coordination is in coordinating the entities (organizations and consumers) involved in DER/CER/GER4 integration. In other words, Transmission/Distribution Coordination.

Arranging the connections, interactions, and roles of System Operators, DSOs, distribution utilities, aggregators/VPPs, meter readers, third party data services organizations, vendors (where their cloud systems are in play), and the like is where you should apply Laminar coordination.

So there you have it- some fancy math provides extremely practical and easy-to-apply architectural guidelines that avoid hidden coupling and looping.

Diving Deeper

Ok, if you want the deep math, here is some of the hard stuff - two papers on layered decomposition.

A Tutorial on Decomposition Methods for Network Utility Maximization

Decomposition Num

563KB ∙ PDF file

Download

Download

Layering as Optimization Decomposition: A Mathematical Theory of Network Architectures

Layering

3.27MB ∙ PDF file

Download

Download

Figure 1 below shows a simple negative feedback loop control without perturbation (top) and with perturbation (middle). The FCEs provide the control effort to drive the system. In a motion control system, these are the motors; in an electric grid these are the power generators. “System” is the thing being controlled (robot, power grid, chemical process, etc.).

Negative feedback reduces the impact of the perturbation of the plant output in a way well-understood in control theory and feedback amplifier design. The ability of the control loop to do so depends on the loop gain and availability of sufficient reserve control effort. The second point is important in understanding how this plays out with a conventional power grid control system and Variable Energy Resources (VERs), also known as Variable Renewable Energy (VRE) resources.1

A specialized version of feedback control is the regulator, which is designed to drive a system to an equilibrium state. In this case the control looks like the bottom diagram in Figure 1. It may operate from an error signal, in which case it is a tracking control (such as for radar tracking of moving targets).

Application to Power Grids

Let’s look at a simplified diagram of a 20th Century bulk power system from a control point of view (Fig. 2).

This is a little bit more complicated than the diagrams in Figure 1, but we can see the basic feedback form. Since the input to the control loop is balance error (imbalance), we have a demand-tracking balance regulator. There are nested loops but the function of this control is to cause generation to track demand by driving imbalance to zero, which is sometimes called load-following control. The actual control system is more complex than this diagram and is quite sophisticated.

Now let’s add in the VER (see Fig. 3).

Comparing this with the bottom diagram in Figure 1, we can easily see that the VER constitutes a perturbation to the control system, where the system “output” is imbalance and imbalance is derived from the difference between resources and demand. Because the VER is stochastic and not dispatchable, it creates random fluctuations in system balance, the very thing the control system regulates. (Yes, there can be curtailment, but that is not the issue here).

The whole system still has to stay in balance (imbalance = 0), else it will collapse, resulting in wide area blackout. So the rest of the system has to compensate for the perturbations VER causes. Now, remember I said the ability of the control loop to compensate depends on availability of sufficient reserve control effort? Grids are designed with a certain amount of reserve capacity to deal with fluctuations in load and other contingencies. Introducing VER at scale increases the amount of fast dispatchable reserves necessary to maintain balance.2

So now we see a key challenge for conventional grid controls in the presence of large amount of VER: having sufficient fast reserves to provide the necessary control effort to maintain balance in the presence of VER fluctuation. In the US Pacific Northwest, BPA used to employ hydro generation to balance wind, but they eventually ran out of hydro capacity and hydro capacity is susceptible to droughts anyway. California is trying to manage solar fluctuations with giant bulk energy storage units but they are doing it wrong and have had such severe problems with ramping deficiency around sundown (see duck curves) that electric vehicle owners were told to refrain from charging their EVs between 5 pm and 9 pm. The state of New York’s energy plan calls for something they refer to as Dispatchable Emission-Free Resources (DEFRs) to solve the imbalance problem but in the next sentence they admit that no such thing exists. Brilliant example of magic box thinking. DEFRs are deferred indefinitely.

Comments

The jurisdictions that want to make extensive use of VER can pursue two options: 1) figure out how to use large amounts of storage as shock absorbers to manage VER power flow volatility propagation; 2) employ a new form of control.3 These are not mutually exclusive.

The first option has major obstacles: current storage technology (Li ion) is expensive, hazardous to operate, has limited cycling capacity, and makes use of toxic minerals. The approach of trying to use storage as a generator with positive or negative output, or as a grid services device, or an outage ride-through mechanism are increasingly problematic as VER increases, absent a significant breakthrough in storage cost, cycling, and safety.

The second option also has a significant challenge: replacing the deeply embedded grid controls with something new. This is especially true if the VER power is to be injected at the distribution level, such as with extensive rooftop solar PV. The resource bifurcation caused by extensive use of DER/CER/GER4 is a fundamental structural change to power grids that has to be met with a homologous control architecture. To be sure, the old bulk power systems have distributed infrastructure, but have centralized controls. Trying to apply that existing centralized control to a highly distributed system is problematic and that is why we are seeing efforts to create DSOs.5

Here is a handy rule:

Highly distributed systems need distributed coordination and control.

The nature of the actual control algorithms that will be implemented in the DSO framework is still to be determined. We do know some things that don’t work, like Transactive Energy, DLMP, and Grand Central Optimization.

1. Service to consumers, not control of them

2. Abundance thinking instead of constraint thinking

3. Ubiquitous use of buffering for volatility decoupling

4. Fast (sub-minute to sub-second) grid dynamics at all levels

5. Focus on reliability and resilience

While I was not trying to predict the future, I want to point out that I said in the paper that an S2 grid could employ micro nuclear reactors and I said that when used for facilities such as military bases, the reactors would be rated for 5 MWe. I got to this using Grid Architecture principles.

The US Department of Energy and Department of War just flew a micro nuclear reactor from California to Utah on a transport plane, for use at a military base. Its power rating: 5 MWe.

Here is the S2 paper.

Envisioning An S2 Grid

3.87MB ∙ PDF file

Download

Download

“But, Curmudgeon Architect,” you say, “why can’t we just use enterprise IT/TOGAF?”1 Well, TOGAF has neither the scope nor the scalability for grid problems, which are inherently Ultra-Large-Scale (ULS). TOGAF only contemplates enterprise data processing/management, but grid architecture must deal with electrical infrastructure, industry structure, regulatory structure, market-control structure, digital superstructure, coordination framework, and sector coupling infrastructures. These structures are all complex in themselves, and all are interconnected in highly complex ways. GA also deals with a much broader range of stakeholders and public policy issues than IT does. The implications of ULS complexity are pervasive and require special architectural methods. Back during the Smart Grid Wars, I saw many times how enterprise architects were overwhelmed by the torrent of use cases generated by the IT approach. To deal this basic scale issue, ULS architecture development is driven from systemic issues rather than use cases (see Fig. 1).

Electric grids have ULS complexity and therefore are ULS systems. ULS architecture development requires a whole systems view, a focus on structure, and use of articulated fundamental principles (see Fig. 2). That is why Grid Architecture (GA) methodology exists.

TOGAF doesn’t give you any guidance whatsoever on how to meet the system requirements. It tells you how to write down what you did, but does not tell you what to do. It is an elaborate documentation scheme, not an architecture development methodology. This is why, as has been noted in the literature on system architecture, different IT architects produce widely varying “solutions” to the same architectural requirement set.

Grid Architecture methodology solves that problem. The use of principles in the development of grid architecture (as opposed to ad hoc approaches used in enterprise IT architecture) derives from the fundamental axiom of Grid Architecture:

Mathematics can induce structure.

So where do we get the math to do this? GA draws upon a very large set of disciplines to identify and apply principles from which we extract practical structures and guidelines for architecture specification. GA uses four knowledge stacks: tools, domain knowledge, disciplines, and methods (see Fig. 3). These stacks provide practical answers for both structures and derivative solutions. I have used all of these stack elements at one time or another.

“But Curmudgeon Architect,” you say, ”do we have to be able to do all this math to do Grid Architecture?” Well, mostly no, since we have boiled much of this down to practical rules, methods, and results you can use. But even with these resources, you need the GA methodology to create grid architectures. Having the tools is not enough - you need to know how to use them.

So let’s see how the methodology is organized. Figure 4 shows the process in five parts: initialization, requirements definition, architecture synthesis, stakeholder engagement, and finalization. On the surface, this looks like many architecture and design processes, but three key blocks (requirements definition, synthesis, and engagement) are aligned with ULS concepts as applied to electric power systems.

Requirements definition is focused on developing a validated set of systemic issues (not use cases) and consequent selection of architectural bases and principles. The synthesis block uses the principles selected for the project and the stack pack to translate the objectives and systemic issues into an architecture. Note that stakeholder engagement runs in parallel with requirements definition and architecture synthesis throughout both of those processes.

The flow diagram in Figure 5 shows the transformation from objectives to architecture via intermediate stages carried out in the Architecture Synthesis block. This process is shown as being time-linear but in practice can involve some amount of iteration (see discussion around Figure 7 below).

GA derives capabilities from a compact organized set of systemic issues, and then decomposes those capabilities into functions to create an architecture with specified characteristics. Use case methods try to improvise a set of functions to cover a massive unorganized set of low-level actions (the use cases) and then see what behavior might emerge. In other words, the GA method works top-down; use-case driven methods work bottom-up.

Specification of structures comes from applying the math from the stack pack (see Fig. 3). “But Curmudgeon Architect,” you say, “how can one person have all of that knowledge?” This is why we use architecture teams that employ Subject Matter Experts (SMEs). The team is organized with a small core and concentric rings of SMEs and stakeholders (see Fig. 6).

There is a great deal to understand about how architecture teams are structured and operated, but for now, know that the small core is necessary to ensure that the architecture has conceptual integrity (a topic in itself) and the outer rings provide both subject matter knowledge and stakeholder engagement.

There are additional aspects of the process that have their own rigor. For example, the Synthesis block contains the process step “Perform role assignment.” Role/responsibility assignment is a significant aspect of the architecture; in a ULS architecture this is not something that can be done ad hoc. Hence, there is a formal and rigorous process for carrying out this assignment. Similarly, stakeholder engagement is formalized with stages and process steps, including Active Reviews of Intermediate Architectures (ARIA), a method adopted from the Carnegie Mellon Software Engineering Institute.

We also divide system characteristics into two groups: those that are seen from outside the system by the users (arbitrarily labeled “qualities”) and those seen from the inside by the system builders and operators (labeled “properties”). We found this distinction to be necessary when dealing with the large heterogeneous stakeholders groups common to ULS systems to avoid writing ambiguous or conflicting requirements. In addition, the relationships between capabilities and qualities and between properties and functions play a role in the creation and iteration of the proposed architecture (see Fig. 7). The concept illustrated in Figure 7 is inherent in the Synthesis block of Figure 4.

Finally, a word about definitions. Every methodology compiles lists of definitions but Grid Architecture puts definitions into definition structures. Doing so converts a simple domain-specific technical dictionary into a practical and powerful language for reasoning about the ULS system. This is typically created in the initialization phase.

Summary Comments

The GA methodology for developing grid architectures provides architects with rigorous ways to develop ULS architectures.

1. Structure is crucial to grid architecture. It determines what a system is capable of and what it will not do.

2. Mathematics can induce structure.

3. Employing math-induced structures result in systems with predictable properties and behavior.

4. Top down derivation of architecture is stronger than bottom-up conglomeration.

Grid Architecture methodology enables the architect to create architectures, not just document them. You need this methodology.

Bon Appétit!

Grid Architecture Cookbook

3.2MB ∙ PDF file

Download

Download

Power grids are enormously complex, and correspondingly they have extremely complex control systems. Let’s take a look at the many kinds of control that are used in power grids:

1. Internal conventional generator control, consisting of three parts: 1 - fuel burn (fossil) or flow gate (hydro) or damping rod (nuclear) control; 2 - turbine speed control; and 3 - exciter control; see Fig. 1.

2. Load sharing (via droop control, an amazing distributed control mechanism that operates without any communication systems).

3. Dispatch and curtailment (resource setpoint control for conventional bulk power generation and Variable Renewable Energy).

4. Receding horizon model predictive control (for determination of resource dispatch schedules and transmission grid settings for bulk power system control; see Fig. 2).

5. Power flow control (VFT1, phase shifting, reactance injection, power electronics).

6. Frequency regulation (secondary generator control/ACE; see Fig. 3).

7. Balance/interchange control (tertiary generator control).

8. Voltage regulation (tap changers, serial line drop compensators, reactive support, power electronics).

9. Reactive power compensation (capacitors, power electronics, synchronous condensers).

10. Stabilization (e.g., damping inter-area oscillations and other phenomena using system inertia, power electronics, reserve dispatch, and ancillary services dispatch).

11. Synchronization (adjusting AC phase to synchronize an AC generator or inverter with the grid).

12. Bulk energy storage control (power flows and state of charge management).

13. Injection control (inverter grid forming/following/supporting; see Fig. 4).

14. Flow limiting (DOEs2, power meter limiters, flexible connections).

15. Circuit connectivity control (switching).

16. Direct and indirect load control (demand response).

17. Aggregator/VPP coordination (systematic operational alignment of utility and non-utility assets to provide electricity delivery).

Note that all of this is happening simultaneously.

It is fair to say that that generator unit commitment and various planning processes (especially the short term operational planning) should be included in the scope of grid control and also, that the role that markets play in generator dispatch and transmission configuration in areas where wholesale real time markets exist shows that such markets are integral parts of grid control as well. See Fig. 5.

Given the use of power electronics in aspects of grid control, time scales for grid controls run from days (even months if we include the full scope of operational planning) to sub-microsecond (for power electronics internal control). See Fig. 6.

In addition to standard control structures, the emergence of GER/DER/CER3 has led to considerations about decentralized and distributed control. Unfortunately, too many people think in terms of dichotomies (“distributed is the polar opposite of centralized”) when the issue is more complex than that. See Fig. 7.

Systems can fall anywhere in or on this triangle. Lack of understanding of this and related realities is problematic for schemes based on distribution-edge connected resources.

Electric power delivery systems are integral to our health, safety, and prosperity. They are not like smart phone apps or video games; electricity delivery can be a matter of life and death. So, to the would-be electricity industry disruptors out there: before telling us about your magic box innovations, please learn something about electricity delivery systems and the industry first.

The Commission Chair actually asked me to explain to the whole Commission what a platform was and why it mattered. (I did.)

Now, platform is a very useful architectural concept and structure. It can take more than one apparent form, but ultimately it depends on the concept of layering. By this I do not mean layered decomposition for coordination frameworks - that is a topic for another day. I mean system components structured in layers. Such structures generally show up as either onions or lasagna.

A platform is a stable collection of components arranged in a layered structure that provides fundamental or commonly-needed capabilities and services to a variable set of uses or applications through well-defined interoperable interfaces. Platforms may have a single layer but most have three or more. A common example is the arrangement found in PCs and laptop computers.

The platform in this case is the three layers: hardware, BIOS1, and Operating System. Hardware may have many variations, but a (hardware-specific) BIOS “evens them out” for the operating system, which then provides a uniform way for any application to make use of the hardware without needing to know any of the details of that hardware. The BIOS insulates the operating system from hardware variations, and the operating system provides open standard interfaces for the applications. This points to a powerful aspect of platforms: future-proofing. The underlying hardware can change without requiring any changes to the applications. Alternately, the hardware does not need to change to accommodate new applications. All good.

Now consider the basic structure of the internet. It is often referred to core-edge and may be depicted something like an onion. But we can “unwrap” it to see the layers.

In fact, when we talk about something being like an onion, how do we describe looking into it to understand it - we say “peel back the layers!”

So, why do we care about the layered platform structure? It confers many advantages:

1. Separates foundation functions from end uses (“applications”) via layering.

2. Provides a set of services and capabilities that are useful to many applications.

3. The platform is stable over time, while the applications may change frequently.

4. Provides decoupling of changes between applications and underlying infrastructure.

5. May scale (adjust resources) to support variable demands from applications.

6. Third parties can freely create applications that use the platform (must have open standard interfaces to do this).

7. The value of the platform is spread across many uses.

Layering in platforms can also improve resilience. Use of Resilience Algebra shows how a three-layer platform can have significantly greater resilience than a two layer platform. Given the discussion of computer platforms above, this is not surprising; the math confirms it.

Just how far could we carry the platform concept? How about this far?

The figure illustrates a “macro-platform” consisting of three layers: the electric distribution infrastructure, distribution sensing, and the distribution communication system (including services). This is a pretty radical departure from traditional distribution system design, but in a dynamic distributed-control environment, it would provide considerable flexibility, resilience, and future-proofing. To my knowledge, nobody does this today. Alas.

So, when it comes to your platform, onion or lasagna?

I have also read any number of architectures from others. I have even done comparative analyses of groups of architectures. Some were good; some were not.

Below are seven maxims distilled from the above. These subtle but powerful truisms take experience to appreciate, so well-meaning but naïve newbies often stumble over them. Following the list, you can read about the rookie mistake of hyper-orthogonalization, a consequence of not understanding maxims 3 and 4. I have seen this - it does not end well.

By the way, people generally find maxim 6 to be the most surprising, but I have seen this too.

Architectural Maxims

1. Architectures are frameworks, not designs. Know when to stop detailing; design is the engineer’s or developer’s job. Architectures supply the minimum constraints necessary to ensure proper system “shape” and free up coordinated downstream design creativity.

2. System architecture development is more than tabulating use cases and then drawing a block diagram, a cartoon, or a cloud illustration. It requires an understanding of systemic issues that are generally not directly discernible just from consideration of a set of use cases.

3. The architecture problem decomposition that seems easiest for humans to understand is almost never the best for efficient implementation. The logical decomposition will look quite regular, whereas the effective decomposition will often seem oblique.

4. An architecture that looks neat and elegant is probably not real. Real architectures tend to be messy and inelegant to some degree, especially when there are significant legacy elements. Strong architectures can diminish, but not eliminate, messiness.

5. Recognize that the system architecture discipline involves both methodology (rigor) and art (inventiveness). Don’t get carried away with the artistry part. One person’s work of art is another person’s kitchen sink stain. Strive for principles-based rigor.

6. Absent a rigorous foundation and framework, the structure of a product or system will tend to mirror the structure of the organization that produced it.

7. System architecture development is not a committee function but hearing the stakeholders is crucial, so proper processes and team structure that ensure all stakeholders participate effectively is key.

The Fallacy of Hyper-Orthogonalization

The architect’s role is to manage complexity, but not to obscure or ignore it. Architectures that are very clean-looking are generally not realistic, especially in an environment with many legacy elements. Maxims 3 and 4 above hint at a problem that is common among inexperienced system architects: the desire to create elegant architectures by defining an extensive and rigid set of highly orthogonal unitary elements that seem easily comprehensible; these are often touted as the basis for building capabilities by composing the elements like building blocks. This hyper-orthogonalization yields diagrams that seem aesthetically appealing. However, such architectures do not persist through design and implementation because they fail to consider real constraints, are too expensive to implement, and result in poor performance.

A real architecture will look more like a set of distorted oblique slices through a hyper-orthogonal architecture. Decompositions and structures will be made to fit real cost/performance criteria and to deal with real legacy or other constraints. Some of that is not pretty.

________________________________________

So there you have it: distilled experience.

1. An architecture is a complete model of a complex system that enables reasoning about the system’s behavior and performance. You should be able to use the architecture to answer questions about the system - capabilities, characteristics, behavior, etc.

2. A good architecture is one that meets the needs of the stakeholders (especially the users) to their satisfaction, does not violate established principles of system architecture, and takes into account the relevant (external) qualities and (internal) properties as per stakeholder requirements.

3. A proper architecture is comprised of three kinds of elements: structure, black box components, and externally visible characteristics.

4. Good architectures have conceptual integrity (devoid of unnecessary complexities or ‘exceptions,’ similar problems are solved in similar ways, exhibits principles-based cohesion, etc.), can direct a builder to cost-effective structures that can be completed within a reasonable time frame, are conceptually acceptable to the stakeholders3, and provide utility to the users.

5. Essential functionality drives complexity, not architectural “elegance.” An architecture that looks perfectly neat and elegant is probably not real. Real architectures tend to be messy and inelegant to some degree, especially when there are significant legacy elements.

6. Architectural structures should have formal bases to minimize ad hoc configurations with unknown properties. Architectural features should trace back to architectural principles; traceability should be documented in the architecture specification.

7. Architectures should feature a small number of interaction patterns and structures (interfaces should function in consistent ways throughout).

8. Architectures must be agnostic to commercial products, tools, and business models.

9. Architecture specifications must produce clearly identified enforceable key constraints on the system implementation.

10. Architecture must be consumable (i.e., understandable) by the users. Architecture should be documented with representations that stakeholders can understand. A set of key term definitions with definition structure4 must be included in the specification.

11. Each component of the architecture should be responsible for only a specific feature or functionality, or aggregation of cohesive functionality. Components should be coupled only through explicit structure, avoiding hidden coupling where possible.5,6,7

12. Architectures define interfaces and platforms.

There is much more to Grid Architecture than this, but these key principles are foundational.

Hayek found in particular that the idea of concentrating all necessary knowledge into a central planning authority was not workable for several reasons:

• To solve an economic optimization problem as typically posed requires that data for the whole society be, as he put it, “given to a single mind” i.e., concentrated in one place or system or organization

• It requires knowledge that is dispersed among many people, with no possibility that any individual or group of experts could be capable of acquiring it all

• The necessary data exists in small portions that are not integrated, are incomplete, and may be contradictory

• Some data can only be found locally and require special understanding to even recognize

• The data can be transitory and may change on an unpredictable time line

• The statistical aggregates beloved of economists are not useful in dealing with constant small changes that make up actual economic dynamics

Hayek suggested that markets have value precisely because they enable participants to benefit from widely dispersed knowledge that is never gathered into one place and where no one has more than the slightest fragment of that knowledge, including market designers and operators, and government planners. He also indicated that some of the knowledge is fleeting in nature and can only be recognized by people with specialized understanding of local conditions and opportunities (“unorganized knowledge of the circumstances of time and place”), knowledge disdained by the economists of Hayek’s time. He portrayed prices as both distillers of information and as motivators. Price communicates essential information in a very compressed and, as he described it, “symbolic” form - information that is beyond the horizon of any individual actor to acquire. Price, he argued, is a coordination mechanism and it also enables division of labor and can use knowledge that is local in time and space as well as knowledge that is “unorganized” and never available to central planners and markets. In effect, it is crowdsourcing - a term that was unknown in 1945.

The applicability of this thinking to present-day electric power systems is striking. Electricity market economists and power system control academics both still engage in the fallacies of centralization and the perceived need to have complete information in a central location for operational management of the extended grid. Through Hayek, we can see that both centralized electricity markets and grand central grid optimization schemes of the type advocated by what I call the “Boston school”1 are problematic in the emerging 21^st Century grid environment, something that seems to be borne out by market failures in Texas, Australia, and elsewhere. This is not to say that grid controls are better than electricity markets or vice versa. In fact, they are two sides of the same power system coin.2

Not only do Hayek’s principles apply to markets and controls, they also apply to grand central grid data repositories, such as those proposed for distribution-connected devices and resources. However, federated databases with distributed responsibilities for data management are keys to resolving the data repository issue for power grids.3

Which brings us to the question of AI and the grid. At least one pricey AI startup and (separately) a lavishly funded non-profit foundation have proclaimed that AI will revolutionize the grid, operating it in some unspecified imaginary optimal fashion by pulling vast amounts of grid data into its arcane inner workings. Talk about science fiction magic box thinking!

Unfortunately, none of these champagne wishes and caviar dreams are backed by anything resembling an actual reason why AI could do this, or any demonstration of such an ability at any scale. The proponents have shown no understanding of electric power grids and what problems need to be solved, and cannot even explain how the magic AI grid ops would work, but yet “know” that AI will be superior in “running the grid.” Since AI is centralized (those giant power-hogging, water-sucking data centers), it will run into the same issue as the Boston school schemes: the limits of knowledge. The knowledge and data that the AI proponents think they will be able to obtain to feed their neural network beasts are not available and cannot be acquired by absorbing the Library of Congress and scraping the New York Times website. And then there is the matter of that little hallucination problem.4 So, it’s really more like root beer wishes and cheese cracker dreams with these people.

Instead, a distributed method of coordination that avoids the issues described above is needed. A distributed5 coordination framework derived from layered decomposition6 can satisfy these requirements. It provides coordination signals that encode information from the entire system without the need to concentrate that knowledge in anyone place, system, or device. Some of the characteristics of the layered decomposition approach are:

• It is not centralized

• It can provide Hayek’s “symbols” in the forms of resource allocation signals (controls) and/or price signals (markets), which can be mixed in the same coordination framework

• It does not have to accumulate data across tiers or layers and hence is scalable, an issue Hayek implied but did not address directly in his essay

• It can enable use of local constraints and objectives (“local selfish optimization inside global coordination”)

• It does not have to depend on theoretical optimality7

• It can be capable of reacting quickly to sources of power flow volatility, avoiding the latency problems that are growing in conventional grid systems as device dynamics move to faster and faster time scales

• It provides an orderly but flexible structure that avoids hidden coupling, tier bypassing, and coordination gapping – common flaws in centralized optimization approaches

If we are to transition to the power grid we need for 21^st Century, the sophistry of grand centralism and complete knowledge must be replaced by a rigorous distributed coordination framework approach to affordable delivery of abundant electricity.

Rather than present an entire U.S. present state reference model here, let’s look at a number of key items/issues related to DER integration that come from consideration of such a model.

Distribution systems are not structured like transmission grids

• Electrical physics applies across the board but the structure and operation of distribution systems is not the same as for transmission systems. Three-phase distribution feeders are generally unbalanced. Laterals may be one, two, or three phases. Most feeders are simple radials, although in urban and suburban settings variations exist, including dense meshes, partial meshes, open loops, and backfeed inter-ties.

• Many distribution systems have little or no distribution SCADA, and laterals are often fused (although new technology is gradually replacing fuses with more sophisticated devices and newer distribution control devices have built-in line sensing). Traditional distribution voltage regulation, protection, and power flow controls are not made for reverse power flows and so must be upgraded.

• A distribution feeder is not an infinite bus. This point must be taken to heart by those that think it does not matter what is connected to the distribution feeders. Many field trials are misleading in this regard due to low scale.

• A service transformer secondary bus is a dead short across the set of loads it serves. This matters when using inverters to inject power into the grid via a secondary. The same applies to two inverters connected to the same primary feeder phase(s).

• Many distribution feeders are structured as” large-wire/small-wire”, meaning that wire sizes decrease as you move away from the substation. Consequently, putting new large loads at the ends of feeders or attempting to inject large amounts of current near feeder ends is not feasible on these circuits.

• Many distribution feeders are already operating near maximum capacity and cannot take much additional load (like EV chargers) or significant injection without upgrade. Reconductoring is the answer; using residential DER and load turndown or usage time-shifting as non-wires alternatives is not.

• The closer you get to the distribution edge, the less you can benefit from the inherent smoothing of DER aggregation, so power flow volatility increases, in the limit it approaches the volatility of a single device.

Load behavior is complex

Distribution systems, substations, feeders, and feeder sections all have individual load curves that peak at different times. They do not align with each other and more importantly, do not align with the bulk power system demand peak. Whiteboard Flatlanders, take note of this peak demand divergence: trying to control loads or usage in a naïve cluster-coupled manner to accomplish precise system peak time shifting or shaving leaves the grid kinda screwed.

Rooftop solar PV, behind the meter storage, and remotely managed responsive loads induce power flow volatility that propagates into bulk power systems,1 and also cause apparent load to be less than actual load. Lack of visibility of this effect at the system operator level creates issues in terms of planning ready/spinning reserves and maintaining balance. In addition, forecasting of rooftop solar is, shall we say, iffy. Before this can be considered anywhere near reliable at scale, somebody has to become orders of magnitude better at weather prediction. Nobody does it well enough now.

DER dependability/availability concerns and aggregator woes abound

Private DER and third party DER aggregators do not have the same obligations as bulk power generator owner/operators. Relying upon DER for stable system operation is problematic due to dependability issues. Homeowners can and do opt out of letting their home systems be commanded by aggregators or grid operators and do so without notice. DER operators and aggregators can and have exited their businesses abruptly.

In fact, most DER operators actually have not wanted to provide energy into the bulk system in the U.S. because they did not want to be held to the rigorous contractual, regulatory, and technical requirements of generators. They only want to provide apparent load reduction. As far as supplying grid services goes, the economics are not favorable to the owners of the DER. Grid services is a small pie, presently divided among incumbent suppliers. For DER aggregators to play, they must take market share away from the incumbents and divide it among their subscribers. Given that some DER like solar are not dispatchable, this leaves the DER aggregators at a competitive disadvantage to the incumbents and as the number of DER subscribers increases, their shares of the services pie decrease. Vast numbers of DERs? Well then, in the limit, as the number of DERs involved increases, compensation for each individual DER goes to zero. It is not worthwhile for the DER owners to let their systems and devices be used that way.

But DER aggregators face other hurdles in the U.S. Most aggregators that were tying to use residential DER have abandoned those plans for economic reasons. They have either exited the business altogether or have offered their platforms for sale to distribution network operators and told them that the distribution operators should run DER for themselves. The reason for this is that the aggregators discovered that after they paid to recruit DER subscribers2 (essentially marketing costs), paid the installer and maintenance organizations, paid for access to the devices via the manufacturers, and could no longer get government subsidies, there was not enough margin left over for the aggregators to survive.

Consequently, U.S. FERC Order 2222 has largely become irrelevant.3

Had the foregoing not been the case, another problem of an architectural nature would have arisen. It has been a fantasy of the grid chattering class that there would be large numbers of competitive DER aggregators. If they were not to be given exclusive territories, then the interleaving of aggregator subscribers would result in entanglement coupling (a form of hidden coupling, which could change arbitrarily over time). This would have created significant coordination issues with implications for voltage regulation and feeder reliability when power is being injected into feeders from the DERs. This is an example of where the “feeder is not an infinite bus” issue matters. Dispatching active DER in aggregation clusters instead of taking DER electrical location and grid state into account (oh, wait, see Sophistries discussion later in this essay) would, how you say, have a high degree of fail.4 And, scalewise, it can get much worse than the diagram below illustrates.

If aggregators were to be given exclusive territories or if they managed to absorb other aggregators (shakeouts would inevitably occur), then they could develop market power. Also not awesome.

DER communication and device management problems persist

Dynamic management of DER on a large scale for support of grid operations is problematic. Communication connectivity (generally internet) to edge devices varies greatly in availability, capability, and reliability. Interfaces vary widely and when API changes cause devices to drop offline, device owners are often not able to deal with the necessary upgrades. In the U.S., control access for solar and storage inverters and home smart thermostats generally must go through the manufacturers’ clouds.5 Since the two main manufacturers do not make data available from the devices, this cuts off valuable sources of information that could have been used to support granular DER management, so as to deal with some of the technical issues listed above.

Sophistries of transactive energy, DLMP, and prices-to-devices schemes revealed

There are some technical challenges associated with transactive and other price-based schemes for DER management. Communications and interfaces limitations have already been addressed above. These technical challenges are real but are probably solvable in time. A bigger issue is that to make these schemes workable requires knowledge of complete grid connectivity and grid state everywhere on the distribution system but for many distribution systems, this is not available. This limitation is in the hands of the distribution network operators; the DER aggregators cannot change the deficiencies that exist in distribution grid observability. Distribution locational marginal prices (DLMP) schemes fail for this and fairness reasons. Single price-to-devices schemes (price broadcast as a signal) suffer from cluster coupling, which cannot take local conditions into account.

But the real problem for transactive and price-based DER management is that the underlying premise upon which these schemes depend is a fallacy. No rational consumer is willing to bid on electricity for home consumption every five minutes, so these schemes depend on consumers being able to know and express their marginal utility curves - in the worst case, for every electricity-using device they own. Automated approaches expect the devices to be able to have these curves to bid for electricity, and in some cases, even provide the curves to a centralized transactive or other market clearing system on demand.6

News flash: consumers do not know what electricity marginal utility curves are or how to figure out what their own are or should be. And never mind that these curves are time-varying.

Imagine consumers being asked to specify forward-looking marginal utility curves for their various appliances and passive loads, and detailing how the curves change with time and circumstances.

One transactive energy field trial attempted to solve this problem by providing home controllers that had a simple three-position manual selector knob, with positions for “Economical”, “Normal”, and “Comfort” operational options. Each option had a pre-programmed marginal utility function that had been created by the project designers. In other words, the researchers imposed marginal utility functions on the consumers that the researchers had made up. The consumers had no idea what these curves did or how they affected what the consumers paid for electricity. On every cycle, each home controller had to upload its currently-selected curve to a central clearing process which then solved an optimization problem across all the curves to determine a clearing price that was supposed to induce a particular DER response for the grid.

None of that was based on basic principles; the researchers created something that “worked” even though the hack that made it operate was just the project developers imposing their own preferred functions on the consumer systems in place of the unknowable consumer marginal utility curves. In one field trial, the transactive process proved to exhibit oscillating prices and had to be modified in an ad hoc manner to dampen the oscillations. No wonder nobody uses it.

Scaling issues emerge

Oh, did “optimization” come up in the conversation? Optimization invariably raises its ugly head in the context of managing massive numbers of DERs. If DER penetration is to become as large as the dreamers envision, scaling is a significant issue. Not just from the standpoint of transporting all the data (Claude Shannon can help with this), but also in terms of computation. Depending on the actual optimization methods used, computation times can suddenly break drastically upward as a function of the number of elements being optimized. In the world of DER imagineers, the numbers of elements is projected to be orders of magnitude greater than what is optimized for the grid today.

Plus, optimization can be brittle.

Final comment

The U.S. power grid is a shared public good, not just a big Tinkertoy for researchers who like to solve nail puzzles or would-be entrepreneurs who think that they will use the latest techno-fad to “disrupt” the electric power industry and so make themselves rich. Deep understanding of how things work and why they work is necessary to make grid changes, but this understanding has all to often been neglected in the rush to push poorly thought-out processes and gimmicks onto the U.S. electricity-consuming public.

Grid Related Control Taxonomy

Control engineering is a large discipline, both very broad and very deep, and the literature on control theory and practice is enormous and quite mathematical. It is not practical to characterize all of it here, so in Figure 1, I have assembled an abbreviated taxonomy of control approaches in electric power systems. It focuses more on control structure, less on control laws or algorithms.

Note that wholesale electricity markets are included in this view of grid control, particularly in Receding Horizon Model Predictive (RHMP) control. We generally use the term market-control to indicate the close relationship between the two, but here we will just refer to control. Markets are in there.

Grid Control Problems

In the Grid Architecture work, we look at whole system problems, so Figure 2 provides a sample list of power grid control problems, including control of Behind-The-Meter (BTM) resources:

I revised this list somewhat from the one in the Better Balance posting to highlight external (grid) control of inverter-based resources, variable structure and parametric flow controls, volatility management (reliability reserve control, generation ramping for solar and wind compensation, etc.), and storage state of charge control (storage power flow is included in general power flow control),2 since these are important emerging trends.

Previous to the emerging large scale penetration of distribution edge connected resources, the problems shown above mapped to one or more of the elements of the control taxonomy. Some of the control solutions use combinations of more than one control approach. The Exchanged Shared State and Layered Decomposition elements are not yet widely used but are there because we will be looking at them in the context of DER/CER integration later.

A few mapping examples:

1. Tertiary generator control - model predictive receding horizon tracking control

2. Generator load sharing - distributed global state control

3. Rooftop solar injection curtailment via DOEs3 - scalar state flow control endpoint bounds

4. Secondary generator control (ACE) - close loop servo-regulator control

5. Thermostat setbacks for DER/CER programs - supervisory dynamic setpoints

6. Static VAr compensation - stabilization

7. Interface control for inverter-based resources - synchronization

It is common for controls to be nested, composed, or arranged in hierarchies or other metastructures.4 For example, generator secondary control is a servo regulator loop nested inside multiple tertiary and supervisory RHMP tracking controls. Consequently, an overall control architecture can become quite complex and inappropriate metastructure can lead to hidden coupling issues.

A Few Words About Systems

Another aspect of control architecture applies to systems in general and to controls specifically. It also brings up something I find Barney-the-Dinosaur-level annoying. I can’t tell you how many times I have seen or heard the following: “Centralized and distributed systems are polar opposites.” Sometimes it is phrased as “Centralized and decentralized systems are at the polar opposite ends of a spectrum.”

People, people, no. Just no.

Actually, it takes a tri-polar diagram to show the relationships among centralized, decentralized, and distributed systems, keeping the following definitions in mind:

1. Centralized System – data and signals flow to a central hub where all logic and control are determined; data and commands flow back to individual elements.

2. Decentralized System – multiple separate entities operate independently with, at most, some small amount of supervision.

3. Distributed System – decentralized system where the parts cooperate to solve a common problem. This implies some form of peer-to-peer interaction.

Read a deeper discussion about this in my posting here. Figure 3 shows the key diagram that indicates how to think about these concepts. The spectral axes are autonomy, coordination, and dispersal.

Systems can fit anywhere inside or on the triangle. Most systems do not end up being “pure” and so are not quite at the corners. Some systems have a portion that is essentially distributed and a portion that is centralized, so it can be necessary to decompose a system to fully understand its relationship to this chart (organized wholesale electricity markets are mostly distributed but have a centralized clearing mechanism, for example). To fully understand control system structure, it is necessary to understand the control metastructure.

What Does This Mean for DER/CER Integration?

We must specify a metastructure for DER/CER control and then specify the control structures within that metastructure to set the control architecture in the context of the whole system architecture. At that point, the designers can specify the control algorithms and implementations. But before we can do that, we must have an understanding of structural, electrical, behavioral, and even financial issues related to distribution-connected resources and loads. We will examine these issues next.

Now, I am sure the electricity market economists are frantically waving their hands to say “But, but, wait, markets solve all the problems, we don’t need controls.” First let me point out that in the Grid Architecture discipline, we use the hybrid terminology market-control because these two are much closer to each other in terms of methods and complementarity than their proponents might have you believe. Second, the two trends I mentioned in the first paragraph are causing shifts in how market-controls must work because electricity markets are not able to handle low-to-zero marginal cost resources. In addition, increased speed of grid dynamics, combined with the sheer scale of planned distribution-connected resources challenge the ability of markets to respond in a timely fashion. You can see more about how the US 20th Century Bulk Power System was controlled here, and how the US BPS electricity markets are structured and integrated with controls here.

With all that as preamble, let’s use control engineering thinking to look at the currently hot topic of Distribution System Operators. The DSO issue arose because of questions about how electricity market-controls could handle the introduction of vast amounts of Distributed Energy Resources. The IT approach has failed to recognize the inherent large scale grid and control structure issues and so has resulted in messy ad hoc arrangements, some of which have unfortunately been codified into regulation in the US.

In the diagram below, the power system is depicted with a good bit of abstraction. We must always remember that the abstraction deliberately hides some details (like the details you can find at the links above), so be wary of making the reification mistake. With that caution in mind, have a look at the power system-with-DSO model in the figure below. It shows only one distribution system but of course, in practice there would be many.

We are going to dissect this diagram to illuminate the control structure. Let’s start by setting aside details of the electrical infrastructure for now and consider the major operational elements:

In this diagram there are three feedback control systems, arranged in a specific structural form (more about that later). The first control loop (SO-BPS) involves the System Operator (SO) and the bulk power system:

The Bulk Power System has more than a dozen control functions (see the links above) but we are going to focus on what most people think about in connection with DSOs: resource dispatch. Notice that I show the system operator control function for this as Receding Horizon/Model Predictive Control, regardless of whether there is a real time market or not. Grid operators and especially electricity market economists do not think of this function as being that, but it is.1 We can clearly see the feedback control structure for this portion of the grid from this depiction. Here the demand from distribution appears as reference input to the control loop, which makes sense when you realize that the grid was originally developed as a load following system.

The second control loop (SO-DSO) involves the system operator and a DSO:

In this loop, the DSO provides consolidated views of its distribution system to the SO, and acts to disaggregate bulk dispatches from the SO to the DSO. In this manner, the SO sees the distribution system as a consolidated load/resource at the TDI and does not need detailed information from inside the distribution grid.2 There is one of these control structures for each distribution system TDI interconnection point. Note that power flows across the TDI may occur in either direction (see the first figure).

The third control loop (DSO-D grid) involves the DSO and the distribution system:

In the layered decomposition (“Laminar”) approach, the DSO acts as controller/coordinator for the resources of the distribution system with input from the SO (note that DSO is a set of functions or role, assigned to the distribution system operator). The DSO manages local and bulk resources to ensure electricity delivery in the distribution service area. It may also manage resources provided to bulk power system from the extended distribution system.

These models are simplified to show macro structure and so do not illustrate various functions that are necessary to grid operation but don’t change the essential forms shown here. By understanding these major structures in terms of control and coordination, rather than via IT, we can reason about the integration and use of DSOs not just statically but also in terms of dynamic behavior.

Remarks

• TDI is a boundary, not a layer or zone or platform.

• This model is essentially a “Total DSO” model.

• This is a distributed control system model. Multiple independent controls cooperate to solve a common problem.

• This model uses Laminar (layered decomposition) structure, although that may not be immediately obvious from the way it is drawn.3

• This structure provides for well-defined interfaces and clear role/function assignments, without any need for intermediate entities or platforms.

• This structure and model enables data/information scalability.

In the 1950s, Bellman stated his now famous Principle of Optimality:

An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision.

A few years ago, we applied Bellman’s Principle to solve the problem of simultaneously maximizing short-term and long term resilience improvements in an electric distribution system. You can find the description of this solution in the Resilience Compendium. Using a combination of Bellman’s Principle and merit order selection, we solved the problem forward in time. [3], [4]

Hold onto that thought.

Bellman also created dynamic programming, which is both an analytical solution method and a computing algorithm. It is used in control engineering to solve complex trajectory problems but is also used in finance, bioinformatics, communication network routing, and other fields. It involves the use of backward induction, meaning problems are solved backward in time.

Hold onto that thought, too.

Now let’s look at a solution method from modern control theory that combines forward time and backward time. The stochastic servo regulator control problem1 has two main parts: the measurement (state estimator) part and the regulator (state feedback control) part. As the controller operates, the control setting at any time affects the both the future system trajectory and the future state uncertainty. The controller has the dual role of controlling the system trajectory and the controlling the quality of state measurements, while dealing with cost and other constraints.

The solution includes determining the observer functions (called the Kalman gains), by solving forward in time, and the feedback control functions (the control gains), by solving backward in time. The system can then be put into motion from its initial (present) state to move through a trajectory determined by the controller to a future end state, despite perturbations and uncertainty the system may encounter along the way.

The concept of solving dynamic problems both forward and backward in time has a sound mathematical basis, thanks to Richard Bellman, Rudolf Kálmán, and many others.

What does all this have to do with transitioning to a future S2 power grid?

Well, let’s see - we must manage the process of moving a very complex system (the grid in the largest sense of all the elements: infrastructure, systems, organizations, processes, markets, regulations, finance, etc.) from its present state through much uncertainty and many perturbing forces in a complex social environment to a desired future end state (the S2 grid). We want the transition to be as orderly as possible and to proceed in an effective a manner. Seems like a complex constrained dynamic control problem to me!

Now, we’re not going to be able to solve the entire S2 grid transition problem using a set of matrix differential equations (bummer, because I would love to do that), but, we can apply the concepts just discussed in a notional way. Of course, we would tackle this in a robust manner.

Turns out somebody actually thought about this in the context of power grid transition. There is a paradigm for using forward/backward time in the grid transition problem called Present Forward/Future Back.2 As we can see, although the idea did not come directly from the math described above, the approach is supportable by principles from modern control theory.3

Understanding Bellman’s work can help us take our power grids back to the future.

So, stand on the shoulders of a giant and use these ideas in your grid transition planning.

References

1. PEI Fellows, A Gambit for Grid 2035, April 2021, Pacific Energy Institute.

2. Jeffrey Taft, Envisioning an S2 Grid, July 2023, Pacific Energy Institute.

3. Paul De Martini and Jeff Taft, Distribution Resilience and Reliability Planning, PNNL-32574, January 2022.

4. Paul De Martini, et. al, Integrated Resilient Distribution Planning, PNNL-32833, May 2022.

First, a few maxims about optimization:

• “Optimum” does not mean ideal, it means “the best available compromise, given the constraints.”

• There are conditions under which an optimum may not exist.

• An optimum might exist (from a math standpoint), but not be feasible to implement.

• Optima can be broad and shallow, in which case being off optimum a bit makes very little difference in outcomes. Or, optima can be sharp, in which case a slight shift in operating point can make a large difference in outcomes.

• Optima may often be found right at the border between the feasible solution set and non-feasible solutions.

• A slight external condition shift can invalidate an “optimal” solution.

Consider the linear programming problem to help visualize these last two points: a slight shift in conditions can leave a formerly optimal solution outside the bounds of feasibility, as the figure below illustrates.

The sharper the optimum is, as compared to other solutions (an economist might say the more value it extracts), the more sensitive to condition perturbations it tends to be. So, for a sharp optimum, the chances are greater that a shift in underlying conditions (or an incorrect representation of the constraint) will cause the presumably optimal solution to fall off the optimization peak or outside the set of feasible solutions. The error can have severe consequences in an operational environment.

In other words, optima that make the biggest differences can also be the most brittle.

So then, what is better than optimal?

Rather than seeking the holy grail of an optimal solution, it is often better to seek a robust solution.2 A robust solution is one that is relatively insensitive to variations in exogenous factors and model parameters. Such a solution will be suboptimal from the standpoint of traditional grid techno-economic objectives but will be superior to the traditional “optimal” solution under conditions of volatility, subsystem or component failure, and external system stress. In essence, we are introducing a new criterion into the problem, namely the need for the solution to be resilient.3 Once that criterion is taken into account, the new robust solution becomes stronger than the traditional techno-economic optimal solution, hence, “Better Than Optimal.”4

So where is this better solution to be found? Right in the neighborhood of the traditional optimal solution. Generally, optimization problems have what is known as a feasible set of solutions - ones that may satisfy the problem constraints and can be realized (implemented). Among the feasible solutions, there may be one particular solution that extremizes an objective function (setting aside the issue of local vs. global optima for the moment). Identifying the feasible solution set may in itself be difficult, and the process of locating the optimum within that set can be computationally demanding to the point of scaling unsustainably as the number of elements (such as DER devices) increases.

Rather than seeking and using the “optimal” solution, we can choose another point from the feasible solution set that is better from the standpoint of robustness. In fact, since we are choosing for robustness rather than pure extremization, there are likely many candidates to be found in the vicinity of the pure optimum - in other words, on a broader but slightly less than optimum curve, or slightly more interior to the feasible solution set than the traditional optimum. Such solutions can be easier to find than the theoretical optimum and the incremental loss of value due to being slightly off-optimal can be kept small. The idea here is to trade off the small incremental advantage of the theoretically optimal but brittle solution for the improved resilience of the less-optimal solution.

Don’t let slavish devotion to the dainty concept of optimality prevent you from using the durable solution that consistently gets the job done.

Remember: robust solution > pure optimal solution. This math is on the final.

Insufficient architectural rigor is one cause, for sure; that is a sin of omission.

But there is a pernicious sin of commission when it comes to bad power grid ideas, and it starts with the useful concept of abstraction. Abstraction is the process of hiding details or removing characteristics from a model of something real in order to create a focused representation. We use abstraction in science, engineering, system architecture, computing, and a host of other disciplines and endeavors. It can be viewed as a kind of data compression process. It is useful so long as the users understand what details are being hidden by the abstraction.

But in the hands of the Whiteboard Flatlanders, Abstraction’s baneful twin often turns up - namely, Reification.

Reification occurs when someone views abstractions as being concrete realities. This is, in a sense, a fallacious inversion of abstraction - treating the abstraction as if it is actually the thing it represents. The trouble develops when people build a Jenga Tower of specious schemes upon those reified “realities.” Data modelers can really exacerbate the situation and then you end up with “architectures” containing absurd magic boxes.

This is how you get stuff like Transactive Energy, dispatchable emission-free resources, treating SGAM as an actual architectural tool, and a host of other Jenga Tower ideas. These faux realities and often bizarre data models engender misguided notions such as:

• demand response (load turndown) is energy storage;

• electrons from renewable energy sources can be physically routed through the grid independently of other electrons (because “green” electrons are tagged differently from all other electrons);

• ancillary services providers sell frequency to the grid;

• distribution service transformers have a set of independent output ports, one for each load.

I have personally encountered each of these misguided statements and much more from various sources, most of whom should know better.1

The negative consequences of reification in the grid transformation domain can be severe:

• Misinforming public policy and regulation;

• Misleading consumer expectations;

• Wasting resources on inappropriate field trials;

• Misdirecting both product and grid investment strategies;

• Fostering poor system designs;

• Detracting from focus on practical grid solutions.

If only there were some methods and tools to help people think critically about the grid. Oh, right.

And knock off the reification!

Watch this short video for a more detailed explanation of reification.

Let’s take a serious look at the architectural concept of modularity and its two key characteristics: cohesion and coupling. It’ll be totally tubular.

The concepts of modularity, module cohesion, and module coupling we use came from software engineering (where module cohesion used to be called module strength) but they are widely applicable to general complex system architectures and are used in Grid Architecture in particular.

First, some definitions:

• A module is a distinct unitary functional element that can easily be added, removed, or replaced in a system.

• Module cohesion is the degree to which elements of a module, component, or function belong together and are jointly focused on a single well-defined goal.

• Module (information) coupling refers to the types of information and manner of sharing between any two modules, components, or functions. In general, it is the degree to which a pair of modules interact or depend on each other (unilaterally or bilaterally). Note that the concept can be extended to electrical coupling, coordination coupling, etc.

• Modular composability is the ability to combine modules into various subsystems and whole systems. Composability informs module interfaces and follows from the combination of high cohesion and low coupling.

Cohesion and coupling are externally observable characteristics that do not depend on knowledge of how any module is implemented. Cohesion applies to individual modules, whereas coupling applies to module pairs. So how do we measure them?

Software engineers have definitions for gradations of both cohesion and coupling. We have developed numerical values and formulas for the gradations so we could apply these concepts quantitively to Grid Architecture grid resilience calculations, but they can be adapted for any type of complex system modularity.

Table 1 shows a classification and quantification schema for module cohesion. A higher value is better than a lower one. The d values (0.0 < d < 1.0) in the table are resilience determinants (called d-block values) used in Resilience Algebra formulations. If you do not have a basis for assigning d-block values, use a default value (d = 0.632) with the formulas in the score column of Table 1.

Table 2 shows a classification and quantification schema for calculating the amount of module information coupling. A lower value for coupling is generally better than a higher one. If the data numbers are not available, you can just use the type (Γ_AB) values.

Note that this is different from calculating equivalent d-block values for information-coupled modules. To do that, we use the method and values given in Table 3.

The notions of modularity and its two characteristics, cohesion and coupling, are multi-scale concepts. They work for code modules, they work for regional scale grids, and everything in between.

So what do we do with this stuff? Several things:

1. Evaluate module specifications (we want high cohesion).

2. Evaluate specifications of subsystems or whole systems composed of multiple modules (we want high cohesion for each module and low coupling between each module pair).

3. Cluster groups of function modules for roles/responsibilities assignment (use Table 2).

4. Use the evaluations with Resilience Algebra to choose from among system structural alternatives (use Table 3).

As an example of the last item, consider Figure 1 (requires some knowledge of Resilience Algebra - this example is included in one of the papers of the Resilience Compendium). For each configuration, equivalent d-block values are computed, taking into account module coupling. The upper case is a typical siloed structure, with two systems, each of which has its own set of sensors and sensor communication networks. The two systems are connected via back-end integration to share information. Depending on the type of data sharing, the equivalent d-block value is either 0.172 (message coupling) or 0.125 (common coupling).

The lower case shows a different structure for the two systems, where they are each connected to a two-layer macro platform (communications over sensors). For this structure, the d-block equivalent is 0.456. The structural change in how module coupling is done makes a significant difference in the essential resilience of the subsystem.

Understanding modularity and its key properties is an important aspect of Grid Architecture, or in fact architecture for any complex system.

So when you are headed out to the architecture point break to shoot the system curl, remember to have plenty of modularity in your utility belt.

Cowabungaaa!, er, Banzaiii!, er, carry on.

By the way, when I was just starting on the US DOE GMLC Grid Architecture project, I was lectured by a “seasoned” guy about what he said was the most important step I needed to take in that multi-year effort. He said the first thing I should do for the grid was label things because he claimed Dwight ”Steez” Eisenhower did this for the Interstate Highway System project by creating a numbering system for the roads before doing anything else! I guess the guy who told me this was afraid the Label-Industrial Complex would end up dominating the grid.