Substation protection and control architectures went through a number of changes over the last 100 years. They evolved from electro-mechanical and solid state relays with primitive communications infrastructure, based on hard-wired copper wiring and use of analog telephone lines, to integrated protection and control systems utilizing Ethernet networks and based on advanced protocols such as IEC 61850. The current integrated substation automation systems based on IEC 61850 architecture are using two level communication networks – process bus, that connects merging units (MU) and intelligent integrated devices (IEDs) and station bus, that connects IEDs and control SCADA system. The process bus carries digitized primary equipment data in the form of sampled values (SV) and GOOSE messages whereas station bus carries mostly control data. In future Centralized Substation Protection and Control (CPC) systems, IEDs as standalone devices will disappear and their functions will be integrated into the centralized computing platforms (and to a lesser extent into the MUs). This will lead to converged network architecture with process and control buses physically connected but logically separated. The computing platform will provide protection, control, management and monitoring functions. MUs will have a minimum intelligence and may integrate only primary equipment protection functions. This centralized protection and control architecture will require a reliable and secure communications infrastructure. In principle, this communications infrastructure connects MUs with computing platforms and as a result must support all types of protection and control functions. These networks must handle processing of real time mission-critical events in a few milliseconds without data loss while supporting SCADA/control traffic at the same time. Robust network redundancy protocols are the key in building resilient networks. The latest IEC 62439-3 HSR/PRP [4] protocols are considered one of the best standardized options available for substation CPC communications architecture as they provide true zero-frame loss communication. There are also emerging technologies like Time Sensitive Networks (TSN) and Software Defined Networks (SDN) etc. that will further improve Ethernet networks performance by making them deterministic and easy to manage and configure.

KEYWORDS

Centralized protection and control (CPC) – resilience – redundancy – communications – infrastructure – converged networks – protection and control (P&C)

1 Introduction

The Smart Grid communications infrastructure is converging to Ethernet/IP based networks; the same applies to future substation CPC systems. Despite this clear trend, there is still a mix of many legacy type communication technologies used in today’s substations: different types of serial interfaces and time-division multiplexing (such as DDS, G.703, IEEE C37.94 etc.) used in majority of protective relays with Sonnet/SDH in the backbone. The mission critical nature of P&C systems imposes very specific requirements on network performance. A number of protection schemes require lossless communication with minimum latency and packet delay variations (packets jitter). Lossless communication is also extremely important in minimizing overall latency and preventing retransmits; for example, digitized sensor data carried in the form of Sampled Values (SV) is not retransmitted – the loss of a few samples can prevent signal restoration and impact P&C functions. The existing properties of Ethernet, as a best effort network, require enhancements to address the mission critical nature of P&C systems – until then, the circuit-switched networks will still co-exist in the substation environment. Advancements in developing Deterministic Ethernet (DE), adoption of lossless redundant schemes like HSR/PRP and other improvements will solve these performance issues thereby making Ethernet the network for future CPC systems. The CPC system is defined as a “system comprised of a high-performance computing platform capable of providing protection, control, monitoring, communication and asset management functions by collecting the data those functions require using high-speed, time synchronized measurements within a substation” [1].

At its core, the future CPC architecture will consist of three major blocks – Intelligent Merging units (IMUs) that carry time-synchronized data from primary equipment, resilient communications infrastructure and CPC Computing (CCPC) devices [3]. The conceptual diagram can be presented as follows:

Figure 1 Conception view of the future Ethernet based CPC system

The key elements of this system are:

1. Data acquisition
a. IMUs
2. Centralized Protection and Control Computing (CPCC)
3. Communications network
a. Ethernet based, IEC 61850 etc.
b. Resilient, lossless, low latency, low PDV, etc.
c. Timing – GPS, 1588 with power profile, TSN, etc.
4. Protection based on data acquired a. Differential protection, Reclosers, Breakers etc.
5. HMI, local SCADA,
a. Secure gateway a. Firewall, IDS, NERC CIP compliant etc.

RTU and SCADA are replaced with centralized computing devices that enable new estimation and control algorithms. In the next two sections we will review existing and future substation communications technologies.
2. Existing Ethernet Technologies

The reasons that Ethernet /IP networks will eventually replace all other technologies in substation automation systems (SAS) are the same that made this technology dominant in the enterprise and service provider world – economies of scale, simplicity, scalability and legacy technologies components obsolescence. For example, components for Sonnet/SDH networks will become obsolete as the technology nears its end in the enterprise and service provider markets.

CPC requires resilient and time sensitive communications infrastructure, however packet based networks may present a challenge to achieve this goal. There is a number of existing Ethernet technologies that mitigate the non-deterministic nature of Ethernet and have been successfully implemented in the substation environment. These technologies are related to redundancy (as a key building block of any resilient network) and timing performance.

The standard redundancy protocols like STP, RSTP support mesh and/or ring architectures with recovery times ranging from over 20s for the older protocols like STP to millisecond levels for RSTP (IEEE 802.1D-2004). Most of the ring topologies are proprietary and almost every vendor has implemented one. MRP (IEC 62439-2) is one of the popular standard ring technologies that provide decent performance and allows for interoperability between different vendor’s equipment.

There is also a mix of ring and chain topologies – networks that can achieve single digit millisecond reconfiguration time. For example, we have tested the network, as per diagram below (Figure 2), and received 1-6ms switchover time depending on the location of the disconnect. This topology is scalable and can be used as a building block for fast reconfigurable networks.

Figure 2 iRing/Chain network topology

The latest IEC 62439-3 protocol – High-availability Seamless Redundancy (HSR) and Parallel Redundancy Protocol (PRP) [4] – is the only lossless technology that guarantees 0s switchover time. This emerging technology is slowly becoming the dominant redundant protocol in substation networks and will be integral to future CPC communications infrastructure.

3 Future centralized substation networks

Convergence of substation networks started with the introduction of IEC 61850 standard that defines Ethernet based scalable, resilient network infrastructure. At the moment, legacy protocols are converted to Ethernet by using device servers for serial and circuit emulation for TDM technologies.

The rate of CPC technology adoption depends on the financial impact of replacing existing mature substation systems, labour force expertise, component obsolesce etc. In Greenfield installations this change is already happening; for example, the CPC system, as shown in Figure 1, has already been implemented in a pilot project in Russia (see [2], sec V).

The circuit-switched networks will probably be the last non-packet based technology standing until Ethernet enhancements will provide the same low levels of latency and packet delay variation (PDV) – the critical parameters for differential and other protection schemes.

There are a number of success stories in using existing technologies that initially were developed for the carrier grade Ethernet like Multiprotocol Label Switching (MPLS) with its Operations, Administration, and Maintenance (OAM). However, complexity and difficulties in provisioning make this technology challenging in the P&C environment.

The newer approaches in making Ethernet truly deterministic like IEEE 802.1x TSN and SDN are promising. For example, TSN is adding circuit-switched properties (such as fixed timeslots) to Ethernet which will further reduce latencies and PDVs.

Future CPC communications infrastructure will be based on Ethernet/IP and have the following key elements:
– Redundant networks/protocols: Predominantly HSR/PRP with the use of RSTP where applicable.
– Timing: IEEE 1588 based with power profile, TSN, SDN, etc.
– Secure communications: NERC CIP complaint and secure (firewall, IDS, authentications, certificates management, role-based access control (RBAC), lightweight directory access protocol (LDAP), etc.
– Ease of configuration: Software defined networking (SDN) with OpenView interface and/or similar advancements in the way systems are managed, configured and provisioned.
The key considerations for building resilient CPC communications infrastructure based on HSR/PRP networks are listed below:
– Redundancy: Need to provide end to end redundancy connecting I/O sensors, IMUs with computing platforms
– Data segregation and bandwidth planning: Protection and control data needs to be segregated in a way that prevents high bandwidth process bus multicast SV and Goose messaging from interfering with the lower bandwidth unicast control data (such as MMS). The network protocols like IGMP snooping, multicast filtering and VLANs can be used to achieve this result.
– Latency: Need to use hardware implementation of switching protocols to keep latencies low. In the case of HSR networks, latencies can also be reduced by splitting HSR rings to minimize the number of devices per ring and using QuadBoxes to interconnect them. If latency requirements are relaxed, then savings can be achieved by simplifying network topologies and using lower bit rates and a higher device count in one HSR ring.
– Greenfield or brownfield installation: To preserve investment in the case of brownfield installation, PRP can be used to utilize existing LAN infrastructure; in greenfield installations, to save on Capital Expenses (CapEx), HSR can be used.

These considerations can be demonstrated in a couple of possible CPC implementations as per the diagrams below.

Figure 3 Centralized P&C, mixed HSR/PRP networks

Figure 4 Centralized P&C, PRP networks

In Fig. 2, the system is built using a mix of HSR and PRP networks. Fig. 3 shows a PRP only solution. These examples represent converged process and station buses; buses are physically connected in a single network but have logically separated data and control planes.

The high bandwidth digitalized data from MUs in the form of SVs and GOOSE messages will be mostly contained in the process bus allowing CPCC to make P&C decisions. The low bandwidth control data will be mostly contained in the station bus. At the same time, both data and control information will be accessible by every node in the system, according to the IEC 61850 communications model.

The data/control planes segregation can be achieved by using a number of Ethernet protocol features like VLAN, QoS and multicast filtering. For example, RedBoxes connecting HSR and PRP networks in Figure 3 are configured as Ethernet bridges with multicast filtering which segregate multicast process data with unicast control traffic. This type of network design reduces process bus latency and PDV, which is closing the gap between packet and circuit-switched network performance.

Conclusion

Economies of scale, Smart grid requirements and the need for standardized efficient energy systems will further drive convergence of multiple substation automation systems to unified centralized substation P&C architecture. This CPC architecture will utilize the ever increasing computing power and resilient, deterministic Ethernet communications infrastructure.

4 Bibliography

[1] Meliopoulos et al. (2010), Substation of the Future: A Feasibility Study. PSERC Publication 10-17.
[2] Looking into the Future Protection, Automation and Control Systems. WG K15 on Centralized Substation Protection and Control, IEEE Power System Relaying Committee, IEEE transactions, 2015.
[3] Centralized substation protection and control, IEEE PES Power system relaying committee, Report WG K15, 2015.
[4] High-availability Seamless Redundancy (HSR) and Parallel Redundancy Protocol (PRP), IEC 62439-3 Ed 2.0, 2012.
[5] Communication networks and systems for power utility automation – Part 90-4: Network engineering guidelines, IEC 61850-90-4, Edition 1.0 2013.
[6] A. Gopalakrishnan, Applications of Software Defined Networks in Industrial Automation,
http://www.academia.edu/2472112/Application_of_Software_Defined_Networks_in_Industrial_Automation
[7] IEEE Standard Profile for Use of IEEE 1588™ Precision Time Protocol in Power System Application. IEEE Power & Energy Society, IEEE Std C37.238™, 2011.

About the author: Yuri Luskind is the CTO at iS5 Communications. He has over 25 years of experience in designing communication products for enterprise and industrial applications working for companies like Fore Systems, Marconi Communications and RuggedCom. At iS5, Yuri is leading development of the next generation industrial communications equipment. Yuri received his MSEE from Moscow University (MIIT) and MBA from Queen’s University, Canada. He holds two US patents and has contributed to a number of publications as a member of IEEE and CIGRE.