Cameras, a useful tool

At our shop in Baytown I find that our little digital camera is one of the most useful in our arsenal. I admit its not thought of as useful as our multimeters and o'scopes but it can save a lot of work and surely aids in communication.

While working on an old 650 Watt Sorensen power supply it became apparent that a small 4 winding toroid transformer had to be changed. I needed to rewind it with some fresh new magnetic wire. Now I could try to make notes on where each of the leads go and try to keep track of each of their windings and polarities. The simple thing to do of course is take a picture, print it out on the laser/ink jet jot a few notes and I have some pretty foolproof documentation. Using Photoshop or just about any other photo editing software one could put text tags mark up a photo that will be emailed or saved to disk. A picture can also simplify a written procedure by showing just what a specific setup is supposed to look like.

Often part numbers can be confusing a particular widget might have a zillion options and a dozen build options a picture just might be the way to communicate exactly what widget is under discussion.

Some companies have pretty tight security policies on cameras in the workplace. We as support people have to live within whatever guidelines that are given us. However in most cases justification can be made in the time and money that a camera can save in documentation and communications. I am seeing more shops and support people with cameras available when they need them.

Labels: Electronics, Industry, Troubleshooting

posted by Ray L. @ 12:31 PM 1 Comments Links to this post.

Nuclear Plant DataStorm

In August of last year Unit 3 of the Browns Ferry nuclear power plant in Alabama experienced a manual emergency shutdown (SCRAM). The Reactor got shut down there was no Radiation released and all the shutdown systems worked as they were supposed to.
so, what happened?

Apparently a ethernet connected PLC device generated what the investigative committee termed a "data storm":

The U.S. House of Representative's Committee on Homeland Security called this week for the Nuclear Regulatory Commission (NRC) to further investigate the cause of excessive network traffic that shut down an Alabama nuclear plant.

During the incident, which happened last August at Unit 3 of the Browns Ferry nuclear power plant, operators manually shut down the reactor after two water recirculation pumps failed. The recirculation pumps control the flow of water through the reactor, and thus the power output of boiling-water reactors (BWRs) like Browns Ferry Unit 3. An investigation into the failure found that the controllers for the pumps locked up following a spike in data traffic -- referred to as a "data storm" in the NRC notice -- on the power plant's internal control system network. The deluge of data was apparently caused by a separate malfunctioning control device, known as a programmable logic controller (PLC).

In other words PLC controller that had nothing directly to do with the valve controllers that caused the shutdown. The errant controller, was babbling or (in Government speak) causing a datastorm which in effect disabled the motorized valve controllers. What we don't know is the real cause of the babbeling PLC. Could it have been the PLC itself or caused by an external Denial of Service attack?

"Conversations between the Homeland Security Committee staff and the NRC representatives suggest that it is possible that this incident could have come from outside the plant," Committee Chairman Bennie G. Thompson (D-Miss.) and Subcommittee Chairman James R. Langevin (D-RI) stated in the letter. "Unless and until the cause of the excessive network load can be explained, there is no way for either the licensee (power company) or the NRC to know that this was not an external distributed denial-of-service attack."

The article continues to describe a couple of instances where Virrii and worms have contributed to major power shortages.

There are lessons to be learned from this one incident.

While IT administration and security issues often times can be perceived by us in the front lines a nuisance issues. Network security is critical when control issues are involved.

I find it interesting that no one has been able to nail down whether the PLC controller that brought the network down has a real harware issue or not. One time events are tough!!! It does sound like the problem though was most likely the PLC itself or some internal communications within the plant.

No digital contagion has been fingered in the latest incident, said Terry Johnson, spokesman for the Tennessee Valley Authority, the public power company that runs the Browns Ferry power plant.

"The integrated control system (ICS) network is not connected to the network outside the plant, but it is connected to a very large number of controllers and devices in the plant," Johnson said. "You can end up with a lot of information, and it appears to be more than it could handle."

The device responsible for flooding the network with data appears to be a programmable logic controller (PLC) connected to the plant's Ethernet network, according to an NRC information notice on the incident (PDF). The PLC controlled Unit 3's condensate demineralizer -- essentially a water softener for nuclear plants. The flood of data spewed out by the malfunctioning controller caused the variable frequency drive (VFD) controllers for the recirculation pumps to hang.

Such failures are common among PLC and supervisory control and data acquisition (SCADA) systems, because the manufacturers do not test the devices' handling of bad data, said Dale Peterson, CEO of industrial system security firm DigitalBond.

"What is happening in this marketplace is that vendors will build their own (network) stacks to make it cheaper," Peterson said. "And it works, but when (the device) gets anything that it didn't expect, it will gag."

In many cases, a simple vulnerability scan will even cause the devices to crash, Peterson said. During tests in an electrical substation, Nessus running in safe scan mode crashed devices, he said. In some cases, sending out broadcast data on the network will crash several of connected devices, he added.

"If you were to test any control systems that have any more than three or four (different) network-connected devices, they could be knocked over very easily," Peterson said.

Labels: Networking, Troubleshooting

posted by Ray L. @ 8:32 AM 0 Comments Links to this post.

Finding the Hot Spots

It can be helpful to find what is heating up, when attempting to repair a piece of equipment. I had an old Army instructor who drilled into our heads, "Whats the first thing you look for after you turn on the equipment?" The answer is "look for smoke and fire." The first step in testing an unknown piece of equipment is to look for smoke and fire. Check for strange smells or a visible wisp of smoke. At the most primitive level we just look for burned components or perhaps some smoke and fire. While this can be destructive, most of the damage is already done by the time we get a chance to open it up and look around.

There are tools to help monitor temperature:

• Infrared temperature monitors are real handy. They can find hot spots fast and is an inexpensive solution. We use a couple of inexpensive EXTECH in our shops. A major advantage to these devices is a person does not need to actually make contact with the equipment under test. This is a big advantage when checking out CRT monitors and radio equipment.



• Infrared thermal imaging is another method. Although this method is getting more inexpensive every year, it is still pricey. These devices are great to see all the components that are getting hot and isolating the cause. As they say, a picture is worth a thousand words, or in this case several thousand dollars.
• Another method is thermocouple digital thermometers. These are often available as calibration equipment and are generally inexpensive. We have several in our shops from a high-priced Fluke 724 Temperature Calibrator to an inexpensive Omega 871 thermometer. Generally speaking, a thermocouple can be attached using either kapton or masking tape. A real advantage to thermocouple thermometers is they can be monitored over a period of time. The Omega we use has a milivolt output so that temperature can be monitored and logged over a period of a few days.

Although temperature isn't the first thing a technician thinks of for test equipment, it often is a valuable tool when chasing down intermittent and stability problems.

Labels: Electronics, Troubleshooting

posted by Ray L. @ 4:50 AM 0 Comments Links to this post.