Software at issue in city water SNAFU
Apparently, the problem last week with the Baldwin City water supply can be traced to faulty software.
Baldwin City Administrator Jeff Dingman put the water issue -- where a line broke near Lawrence Jan. 1, was fixed, but a bushing in a valve broke when turned back on, left the city with no supply, water towers went empty and a boil advisory had to be put in effect -- on Monday's city council agenda. He put the bottom line up front.
"The water is fine," said Dingman. "Always has been."
But, because the levels in the water towers got so low last Tuesday before the problem was discovered, the Kansas Department of Health and Environment issued a boil advisory, informing residents that it was best to boil drinking and cooking water until tests could determine if bacteria had entered the system. All but two restaurants in Baldwin closed for two days as a result. The boil advisory was lifted Thursday afternoon after six water samples taken from various locations in Baldwin were found to be fine.
"What failed?" asked Mayor Gary Walbridge.
City Utility Director Bill Winegar said the biggest problem was the alarm system that should have alerted crews that the city's two 750,000-gallon water towers were low on water didn't go off.
"Basically, the alarm didn't go off," said Winegar. "How or why it happened, I can't tell you. It won't happen again."
He also explained problems with the telephone and walkie talkie systems, before fielding another question from Council Member Ted Brecheisen Jr.
"Are we going to have someone look at the gauges manually on days off?" said Brecheisen. "I know you've had problems with that system in the past."
"We've talked about that," said Winegar.
But, other than questions from Council President Amy Cleavinger regarding notification of Baldwin residents and businesses, that was the end of the water discussion.
The Signal had other questions and Dingman provided those answers Tuesday morning.
"First, the alarm system did not work," said Dingman. "This system does not use phone lines, but uses telemetry. There are transponders at the various points -- meters, water tanks -- that relay information to the system controller in the public works office. This controller collects the data and a computer connected to the controller analyzes the data. For whatever reason, the water tank alarms were not listed as being present -- they definitely were there at one time."
The city called in the system vendor, Micro-Comm, Inc., of Lenexa, to trouble shoot and find out why it failed.
"In discussion with the vendor, there were some components in the controller that were changed out last fall and the though process is that this altered the program that contained the alarm settings," he said. "That's the best explanation we've come up with on why the alarm settings were no longer there. Nothing in the field -- at the meters, pump station or water towers -- failed. Nothing in the system controller at public works failed. The computer that analyzes the controller's data -- which is a machine dedicated to that use -- didn't actually fail, but it was a software setting -- more precisely, lack of software setting -- within the control software on that computer that did not process an alarm."
In analyzing what went wrong, another problem was found and corrected. Even if the alarm software was set properly, notification still wouldn't have happened.
"If all of what I've explained had worked properly, there would have been problems when the system tried to dial out to page our on-call staff person that there was a problem," said Dingman. "This is the compatibility issue between the controller/alarm system and the phone system we had installed last year. It was a matter of the controller trying to pick up a line that was on the phone system instead of the line set up specifically for and dedicated to the dialer. This has been fixed and the system is now again operating with its own dedicated phone line that is not run through our phone system at all and doesn't need to be."
And, to make sure everything was how it should be, tests were run last week on the entire system.
"When the vendor was here last week to troubleshoot all of these problems, the alarms were all re-programmed and re-set, and the issue with the phone line was addressed and solved with the dedicated line," he said. "They put the system through several troubleshooting scenarios by creating false alarms and following it through the paces to notification of the on-call technician. When the on-call tech gets a page, he can call into the system and it will tell him what kind of alarm is being sent, which was also tested."
Dingman, in answering Cleavinger's questions regarding notification, said city staff had done the best it could and did considered going door to door with notes, but time did not allow that.
On Thursday, KDHE completed its test and determined the city's water supply was safe. Although restaurants owners who were closed for two days declined to provide figures on what it cost them, one who wished to be anonymous summed up the feelings on whether they were happy with two days off.
"No. Lost revenue," she said.