A pipeline company in the Midwest provides pipeline transportation services to deliver gas and diesel fuel. Several years ago, a telephone fiber optic cable was cut near the host computer facility. This took out the SCADA network that monitored the pipeline system. At the time, the SCADA network used multi-point analog phone lines and 1200 bps 202T modems.

The pipeline company recognized that it could not be without control of the pipeline pumping stations for any length of time. The company has to monitor its stations at all times. There were more than 30 stations throughout the pipeline system that required constant monitoring, regardless of the condition of the SCADA system telephone lines.

The fiber optic cable cut left the SCADA system out of service for approximately three days. During that time, company personnel manned the stations, a dozen at a time, and moved from station to station, with numerous phone calls to verify various aspects of operations. The company recognized that manual operations meant increased risk, and that managing and minimizing risk is a high priority in the pipeline business.

Improving the network

That extended outage prompted the company to investigate and hopefully implement a better way to back up the SCADA communications, to have redundancy and fast cut over to a backup system. They wanted a system that was automatic, or nearly so, and would result in restored communications, not manual operation of the pumping stations.

The pipeline company began conversations with frame relay service vendors and frame relay equipment vendors. They asked for quotations from several frame relay service providers and evaluated several frame relay access devices (FRADs). The company ended up selecting Global Crossing for frame relay service. Global Crossing was already providing other telecommunications services to the company, which gave Global Crossing an economic edge and a positive service history. The Data Comm for Business Broadcast Polling FRAD (BPF) was selected for the frame relay equipment. The BPF transport the asynchronous serial SCADA polling data over the frame relay network.

The primary goal of the pipeline company was to increase the reliability of the communications network. Losing the network for days was not acceptable. After implementing frame relay, which has now been in service for several years, the company can look at their experience and measure the success of the move to frame relay. And the news is good. The frame relay service is a winner in several categories, more than originally anticipated.

The new network

The frame relay network now has two host sites. The primary host site is in one state, the backup site is located in an adjacent state. The host computers at both host locations and every SCADA device at the pumping stations attach to a BPF. The host BPF and the pumping station BPFs encapsulate the async polls and responses in frame relay packets for transport over the frame relay network. Poll responses from the pumping stations are sent back to both the primary and the backup host site.

The primary host site is the one which is normally polling the pumping stations. The backup site becomes the active host site if there is a failure at the primary host location. At the inactive host site, the poll responses always arrive but are ignored by the SCADA host computer. An upstream oil company that furnishes the refined products controls whether the primary site or the backup site is polling the pumping stations.

Within the frame relay network, a PVC (permanent virtual circuit) is created between each of the SCADA devices and each of the hosts. A PVC is created by assigning DLCIs (data link control identifiers) to the end points. Each SCADA PLC at a pumping station is assigned two DLCIs, one that defines a circuit to the primary host site; the other DLCI defines a circuit to the backup site. At both the primary and the backup sites, there is one DLCI defined for each remote pumping station.

Reliability

Reliability was a primary goal for the system. Frame relay is a winner in this category. First of all, poll/response error rates are virtually nil with frame relay. The old analog network had a poll/response success rate of 96%. With frame relay, the poll/response success rate is 99.8%.

Reliable switching to the backup host site in the event of failure is a simple accomplishment with frame relay. Global Crossing feeds the primary and the backup sites with separate frame relay links into the network. If there is a communications failure with the link to the backup site, the primary site will keep running. If there is a problem with the link to the primary site, it will take minutes or less to make the switch from one site to the other as the active host. Since all the SCADA PLCs are already sending poll responses to both sites, there are no changes that need to be made in the field.

The frame relay technology makes it easier to troubleshoot communications problems. This is due to better diagnostics. Diagnostics in the BPFs include the ability to see all the DLCIs that are mapped to create virtual circuits. If there is network congestion, this information is available from the BPF. Response time to the pumping stations can be measured. Error statistics are available from the BPF. The pipiline company or Global Crossing can easily and quickly get circuit statistics that show inbound and outbound traffic to each location.

Flexibility

The frame relay provides flexibility that was impossible to achieve on the analog network. The host computer serial port and the SCADA PLC serial port speeds are no longer tied directly to the speed of the communications link. The frame relay lines run at 56 Kbps. The host computer ports now run at 19.2 Kbps and the PLCs are 9.6 Kbps. The pumping station BPFs all have four serial ports. Some locations now have two PLCs, each operating at a different serial data rate. Ethernet has been added to the BPFs at locations that need Ethernet connections.

The most valuable feature of the frame relay network is to introduce peer to peer communication. Connecting one pumping station to another for peer-to-peer links is a simple matter of having Global Crossing add DLCIs to the mapping. Two DCLIs logically connected to each other create a new PVC, or permanent virtual circuit. Peer-to-peer allows communications from PLC to PLC, not just host to PLC. Peer-to-peer links provide closed loop control between stations. Closed loop control between pumping stations is a safety issue. If there is a path loss in the pipeline, the source (the pump) can be shut off. Pumping stations can share information regarding pressure, temperature, specific gravity, etc.

Ethernet

The pipeline company has recently added Ethernet to the BPFs to provide access to local pumping station HMI (Human Machine Interface) information from remote locations. The HMI is typically graphic information that is available only via Ethernet.

Cell modem backup

Another addition to the frame relay network is to utilize BPFs with backup capability at several of the key pumping stations. These pumping stations are especially critical and must always be monitored. The BPFs with backup capability are connected to Airlink wireless packet data modems. If communications over the frame relay network is lost, the remote BPF initiates a wireless modem call to the SCADA control center. The BPF knows to use the wireless modem because the BPF will sense the loss of the frame relay connection within a minute of the link loss. The backup link via the wireless modem takes about 5 seconds to connect. When the frame relay link is restored, the remote BPF automatically drops the modem call and resumes communications through the frame relay network.

Savings

The pipeline company did not make cost savings the primary objective, but the savings are substantial by moving from analog to frame relay. The frame relay network costs only 60% of what the analog network cost. The 60% savings amounts to nearly a quarter of a million dollars in annual savings. While seeing substantial savings, the frame relay system provides higher speeds, more reliability, greater pipeline safety, simple and reliable backup, and a more flexible network.

Speed

The SCADA system now exhibits has faster poll to response times. The polling cycle time has been increased from every 7 or 8 seconds to every 3 or 4 seconds. This doubles the surveillance time. With the (previous) 1200 bps analog network, the company knew only if the valves were open or closed. The faster polling now allows them to see valve travel open or closed, not just the state of being open or closed. Pressure information is updated twice as often. This means failures are determined sooner, which translates to increased pipeline safety.

Network facts

The pipeline company still uses very little of the 56 kbps bandwidth. The daily circuit statistics show 8 Kbps or less used at each network location. The frame relay circuits are 56 kbps and the CIR (Committed Information Rate) is 16 Kbps. The network traffic is less than half of the CIR, less than 15% of the line capacity. The response time of the frame relay network is very low. The BPFs have a “ping” function for measuring response time, and the response time varies between 35 and 40 milliseconds.