Water pumps need firewalls too. That's what operators of the Tennessee Valley Authority's (TVA's) Browns Ferry Nuclear Plant discovered last August when they were forced to manually shut down one of their plant's two reactors after networking problems caused two water pumps to fail and threatened the stability of the plant itself.

On Aug. 19 2006, operators found themselves in a potentially dangerous "high power, low flow condition," when one of the plant's two operating reactors was not recirculating enough water to properly cool itself, the U.S. Nuclear Regulatory Commission (NRC) said in a note sent to operators last month. Operators were forced to perform a "manual scram," or shutdown of the plant.

Built in 1974 in northern Alabama, Browns Ferry was once the world's largest nuclear reactor.

Although the Browns Ferry incident wasn't anywhere close to a nuclear meltdown, it was a serious situation, said Eric Byres, CEO of Byres Security, in Nanaimo, British Columbia. "They realized that their recirculation system wasn't working and they were in danger of something undesirable happening," said Byres, an expert in industrial systems security who was consulted on the matter.

The cause of the pump's failures? "Excessive traffic" on the closed Ethernet network used by the plant's control systems, the NRC said.

The NRC report said the origin of this excessive traffic was unclear, but Byres suspects that the problem was due to faulty networking code the controllers used by the plant's recirculation pumps. They may have suffered from the same well-documented networking flaw that has taken down similar systems in food processing, steel, and pulp plants in the past, Byres said. "I'm personally aware of at least a dozen incidents at this point that relate to this particular fault," he said.

Although he declined to name the manufacturer of this product, Byres said that it has a known bug that can cause a crash by generating too much networking traffic. "It's like the loud guy at the bar standing at the table," he said. "It kind of cuts down on the ability of everyone else to have a decent conversation."

After the incident, Browns Ferry's operators began developing firewalls for the different controllers on their network as well as a network firewall device to limit the traffic between devices within the plant's internal network, the NRC said.

Two members of the U.S. House of Representatives believe that more should be done, however. Last week Rep. Bennie Thompson, (D-Miss.), and James Langevin, (D-Rhode Island), wrote the chairman of the NRC expressing their concern that the Browns Ferry failure may have been due to an outside attack. "Without a thorough, independent review of the logs and associated data, the assumption that this incident is not an outside attack is unjustifiable," they wrote.

Byres agreed that a more thorough investigation would be useful. "It's unfortunate that they didn't dig in further," he said. "These types of glitches are well known."

Virtually no control systems use firewalls at present, although that is starting to change, Byres said. "The idea is still in its infancy, but -- particularly in the oil and gas industry -- there's starting to be a lot of interest in doing this sort of defense-in-depth," he said.

Water-pump firewalls are coming none too soon, Byres believes.