What if a technician could remotely monitor any SCADA channel from his office, or even his home, and determine if polls were being issued and responses were being received? What a tremendous tool and time-saver that would be! That is precisely what El Paso’s technicians and engineers are doing every day.

El Paso Corporation is the nation’s largest natural gas pipeline company, operating over 43,000 miles of pipe to bring clean, efficient, natural gas to millions of homes and businesses across the United States.

Its Telecom & Network Services group has built and operates a communications network that includes channels for SCADA to poll remote terminal units (RTUs) all over the United States. It includes microwave systems, radio systems, satellite systems, telephone systems, and every combination of the above.

To troubleshoot such a complex SCADA network, the telecom group also developed both hardware and software technician tools to allow the company’s operations personnel to monitor and diagnose problems for quick resolution. The goal here is to discuss two of those tools, the SCADA protocol analyzer called ElPasoPA, and the data tap, which appears in two forms.

El Paso SCADA System

While SCADA is as old as industry, there are several details and three key components in the current El Paso SCADA system which will make an understanding of the tech tools much more intuitive – the ElPasoPA, the PA Service, and the data tap. epSCADA is the name given to the SCADA master software used at El Paso. It was developed by El Paso’s Operations and Engineering Computer Systems (OECS) group with a goal of serving all of the pipelines that make up the El Paso system. The major protocols identified were Modbus (both ASCII and RTU), BSAP, and Fisher ROC. El Paso’s expert real-time programmers plied their trade for several years, and the result is a stable and flexible platform for all SCADA applications on the El Paso system.

El Paso’s OECS group wisely included a protocol analyzer, ElPasoPA, similar to a previous version developed by the telecom group. An outside contractor, Imperious Technology, Inc., was hired to implement the new analyzer to El Paso specifications. This would provide all the normal functions of monitoring and decoding the SCADA traffic in multiple protocols, serve as an MTU for polling RTU’s, and act as an RTU to accept polls and respond with fixed messages.

The contractor provided a module, PA Service, that would run on the same server as epSCADA, and would allow multiple technicians to connect through it to a single port on a terminal server. It also monitored the IP activity in and out of the master, and made that available to the technician over IP as well. The design goal was for 1,000 technicians to be able to connect with zero impact to epSCADA.

This software tool, ElPasoPA with the PA Service, represents a quantum step of innovation for SCADA monitoring software, and when combined with a hardware data tap developed by El Paso, creates a package that is unbeatable. It allows technicians to monitor parts of the network from anywhere that they can connect to the company network – even from home in the middle of the night – when things seem to break.

A brief SCADA review

A SCADA system consists of just three parts, as illustrated in Figure 1 – the master, the slave, and a communications channel. There may be thousands of RTU’s, and many communications channels and types, but the basic block diagram remains the same.

With the advent of the internet and the explosion in TCP/IP data communications, it was perfectly natural to adopt some of the new hardware and techniques which became available. In the old days, if ten channels of SCADA were required, then ten wires came out of the computer serving as the SCADA master. One of the first new devices to appear was the terminal server. It allowed many “terminals” to be connected to a computer from another building or across the country over the internet.

The modern SCADA system uses the terminal server to extend its reach out to the field office, where it converts TCP/IP data packets to RS-232 data packets. While it is true that it may also be for printers or personal computers, in this case it serves to connect to the field remotes (or RTU’s) through any number of “last mile” communications systems.

The terminal sever

The terminal server is currently used at every entry to a field SCADA channel at El Paso Corporation. It serves as a starting point for troubleshooting, and as the first occurrence of RS-232, it must be configured to generate the serial parameters required by the RTU’s on a given channel (Figure 2).

With the terminal server and its many multiplexed channels came the addition of a test tool called a Data tap (Figure 3). This was a device that monitored the outgoing RS-232 polls and the incoming RS-232 responses on the SCADA system, and combined them into the receive data line on a secondary terminal server port. This is what allows a technician to connect to the “monitor” port of the terminal server and see the out-going polls, and the in-coming responses, and to do it from any place where he can connect to the network. This remote monitoring ability multiplied the company’s technicians with the ability to see how the system was behaving, and to form a preliminary diagnosis before ever getting on the road.

It was not long before the data tap was routinely installed at every terminal server that was supporting SCADA communications. The data tap is mounted above every terminal server, and is shown in Figure 3. It monitors 4-Ports and has LED indicators to show activity on the TX and RX RS-232 data lines (Figure 4).

SCADA communications protocol

The ElPasoPA consists of software that decodes SCADA polls and responses, and displays both the raw data from the line and the decoded values. As seen in Figure 5, the display shows the raw data and the decoded data values. While the software takes care of all the protocol details with the bits, bytes, CheckSum, etc., it leaves the technician free to examine what is going on with the SCADA network. The decoded data is not identified as to engineering units and field assignment, but it is easy for a technician to see that it is valid data. That makes ElPasoPA the perfect troubleshooting tool. A technician can log into the monitor port on the terminal server to see live data.

One can see that there are many options to assist the technician or engineer in decoding any level of detail needed, and when combined with the ability to do this in several protocols, the analyzer becomes a very powerful tool. In addition, the ElPasoPA reports odd messages and fragments with error messages to assist the technician in troubleshooting. SCADA problems are rarely related to Check Sums or CRC’s, but are usually the result of timing on radio systems or incorrectly set polling or serial parameters.

The analyzer supports the following protocols:

  • Modbus ASCII
  • Modbus RTU
  • Gould Modicon ASCII
  • Gould Modicon RTU
  • BSAP – Bristol Synchronous/Asynchronous Protocol
  • Fisher ROC – Remote Operations Controller.

Additionally, the ElPasoPA supports the following modes of operation:

  • IP Sniffer
  • Monitor – Serial and IP
  • MTU – Serial and IP
  • RTU – Serial and IP
  • Load file – loads log files recorded during operation, and parses log as new data.
  • Custom screen – allows for engineering units and field tag definitions.

The sketch shown in Figure 6 highlights the many capabilities that are available with the ElPasoPA in conjunction with the PA Service operating on the epSCADA master. This represents a true quantum jump in monitoring capability for SCADA systems.

The chief innovation in this fourth-generation El Paso Corporation analyzer is represented in Figure 6 as the dotted lines indicating TCP/IP protocol and the use of the PA Service inside the same computer (server) as epSCADA – the SCADA master. The IP Monitor and IP Sniffer are important enough to discuss separately below.

The IP Monitor

Direct connection to a terminal server monitor port is simple and fast. It allows all the functionality of the analyzer, and was available in the third-generation product. The only problem is that only one device can connect to any given port on a terminal server. It is often helpful to allow multiple people to access the same port, either to collaborate on a problem, or for completely independent reasons.

Connection through the PA Service solves the problem. The PA server connects to the terminal server locking out others, but a large number of individuals can connect to the PA Service and see the same data. The service acts like a data-sharing tool, which is extremely helpful. Without this ability, the first person would see the data, and presumably the problem, go by and then he would have to get off the port to allow the second person to log on and hopefully see the same problem.

The IP Sniffer

Here, the ElPasoPA is connected through the PA Service and actually sniffs the IP port of the SCADA master, decoding the polls and responses going out and coming into the actual computer performing the polling. This gives the field technician the upper hand for a change when the reported problem appears to be no response. If the response is actually there, it becomes the problem of the SCADA software team.

These types of problems usually end up being a configuration problem. This mode is also very handy if there is no data tap installed, because it gives the technician a view that is independent of the problem as seen by the operations controllers. From a philosophical standpoint, it eliminates the “IP Cloud” which may not be well specified, and may actually involve routes and hardware that change, depending on traffic.

For instance, an incorrect setting on a terminal server could result in SCADA responses from field RTU’s not being passed through the IP to the SCADA master. However, the data tap would show good data being received. The IP Sniffer would show that the system controller was not losing his mind, and that the data was not making it to the SCADA machine.

Hardware tools

There are various pieces of hardware that every technician must lug around in his tool kit. This discussion will focus on the most prominent and particularly the El Paso–specific SCADA tools, those that were developed by telecom and network services within El Paso Corporation for its technicians.

Most of the parts shown in Figure 7 are available from any electronics store and most large “everything” stores. The DB9 to DB25 adapters allow the use of all the DB9 Tech-Tools, with the original DB25 connectors found on most radios and modems. The unusual tools are the two shown in the bottom right – the pocket monitor and the data tap. They are discussed individually below.

The pocket monitor is strictly an RS232 break-out box, without the break-out pins. Its chief benefit is the layout of the bi-directional LED’s. By putting the DTE lamps on the top row, and the DCE lamps on the bottom row, a technician can spot problems in most RS232 setups and easily fix them. Only the seven most important signals are represented.

El Paso buys these from a vendor with a modification that the negative RS-232 voltage produces green, which then flashes to red when the voltage goes high to indicate zeroes or active handshake signals. By simply walking up to a given RS-232 situation, breaking the link and inserting the pocket monitor, the technician can usually deduce any hardware problem. Of course, the software problems are tougher, but if they involve timing or a lack of transmission, the pocket monitor spots that very quickly. Every technician that comes near RS-232 in the course of his job should have one of these.

The Hand-held data tap is actually the precursor to the terminal server data tap. It has no LED’s and only monitors a single port, but if a technician needs to break into an RS-232 SCADA channel and monitor both sides of the exchange, this is the only way to do it. The monitor circuit is simply a resistor diode or gate with some biasing. The tap port on top connects directly to the technician’s laptop via a straight cable. It is built for El Paso by the same vendor that does the pocket monitor, but the circuit is strictly an El Paso telecom design. Actually, it now appears in the vendor’s catalog as a standard part.

This device requires that the data being monitored be half-duplex, as are most SCADA system poll-response sequences. It also requires that the software on the laptop be able to display the information being gathered. For instance, a simple terminal emulation program (like Microsoft HyperTerminal) will do a good job of displaying the raw polls and responses from a Modbus RTU protocol exchange, but it cannot decode the data. However, the same software would not do as well showing the Modbus RTU protocol. The ElPasoPA SCADA Analyzer, of course, is designed to handle many protocols.

Serial modes

The serial modes include the Monitor, the MTU, and the RTU functions. While these all have their IP counterpart, the serial functions are particularly useful when in the field visiting a site. Using the hardware tools and the ElPasoPA, most SCADA communications problems can be easily diagnosed and corrected.

When a technician arrives on-site to diagnose a SCADA problem, he may by able to see a problem from the LED’s on the communications or RTU equipment onsite. If not, he may break the RS-232 link at the communications equipment and install his hand-held data tap. With the data tap in place and connected to the serial port of a laptop computer running ElPasoPA in the serial monitor mode, the same poll-response sequences should be visible as what he saw earlier at the terminal server data tap. If something is wrong, he should be able to quickly decide if it is the communications equipment or the RTU. As always, the name of the game is: “Are there valid polls directed to this RTU address, and is the RTU sending back valid responses?” This is one of the first functions to be built in any analyzer and is available in many “demo” analyzers. However, the full functions of the ElPasoPA are available for display, custom polling list, data logging and retrieval, and the custom screens. The ElPasoPA RTU mode has a data list where data may be defined for responses.

USB to serial converters

All of the serial troubleshooting is done using the hardware tools and the serial port on most laptop computers. In the past few years, computer manufacturers have discontinued the once ubiquitous DB9 serial port. To compensate, many vendors of USB to serial converters have jumped into the mix. Many of these perform well electrically, but do a very poor job if one requires the RS-232 handshaking lines to perform to specifications.

In SCADA operations, the industry often uses RTS to key on a radio or modem prior to transmitting to avoid losing the beginning of a message, while the radio or modem keys up and begins transmitting a carrier. El Paso Telecom & Network Services has run extensive tests measuring input impedance, output impedance, and open circuit output voltage on over a dozen of these devices. Further, the timing between RTS and TD was measured. In all but two, the RTS signal dropped prematurely, chopping off the data being transmitted. The manufacturers were all notified and the few that responded said they did not have access to the source code and could not fix it. Only those passing the timing test could be recommended for SCADA troubleshooting.

More often, SCADA radios are using “key-on-data” to avoid that type of timing problem. The radio transmitter is controlled internally and stays on until the data stops arriving at the serial port. The radio still returns CTS to remain RS-232 compliant.

Conclusion

The ElPasoPA SCADA protocol analyzer is a fabulous tool used by more than 1,000 technicians and engineers at El Paso Corporation to perform SCADA trouble-shooting and diagnostics involving other serial communications in the plants. Its use with the PA Service and the data taps represents a quantum step improvement in SCADA monitoring tools. The hardware tools are packaged and given to every technician that attends the basic SCADA course, or maintains SCADA-related equipment.

The result is a team that works much smarter and faster at responding to SCADA outages across all of the company’s pipelines. The ability to remotely monitor the system, prior to hitting the road, saves technicians countless hours worth of unnecessary trips. This helps to keep El Paso Corporation’s pipeline operations safe, efficient, and competitive.

Acknowledgment

Based on a paper presented at the ENTELEC 2008 Conference and Expo in Houston, Texas, on April 9, 2008.