National Weather Service: Building a Better Forecast Model

It's a step up from today's system, with which forecasts are compiled by regions or metropolitan areas, not neighborhoods. Forecasters have to correlate and decipher textual data manually from different weather detection systems. This text-based weather data leaves a lot to interpretation, so it isn't always accurate, says Randy Chambers, operational manager of the NWS' Network Control Facility, or NCF. The NDFD database makes it easier to glean information from the reams of embedded weather data. "So there's more uniformity and accuracy in the forecasts," Chambers says.

But the weather service's new database was nearly grounded before the NWS issued the first neighborhood forecast. When NWS technicians test-ran NDFD on the agency's customized network simulation tool last year, they determined that the flashy new graphical weather data would saturate its clustered Linux server architecture. The simulation showed that for several minutes each hour, the NDFD database application would consume nearly 90 percent of the existing Dell 2650 Linux cluster's computing resources. As NDFD expanded and evolved with more graphics, the servers would get slower. And the slower the servers, Chambers says, the staler the forecast data.

So this fall, the weather service will buy new Linux superservers, with multigigabytes of memory and a terabyte or more of storage capacity. The existing servers are sufficient for running the early phases of the database system, which is less graphical. Meantime, the weather service will reconfigure the servers to get more mileage out of them. One option is to separate the server cluster and divide up the RAID 5 drives, Chambers says. "Then we can use all the processing power of both 2650s and spread the drives among the servers," he says.

For the new database superservers, NWS is considering an all-in-one Linux superserver architecture that would support the NDFD Informix database, collaboration server, Web server and the NDFD processing system. There are trade-offs with this combo superserver, though. "We're wrestling with whether to throw everything into one bucket. It makes sense from an engineering perspective to have one huge server, with a backup," Chambers says. "But if that server goes down, you've lost all these pieces."

The NWS's Advanced Communications Network Model (ACNM) simulation tool, meanwhile, is like a forecasting system for the weather service's network. The weather service uses it to test-run technology or products before it buys them (see The Hard Sell, below). ACNM is based on OPNET Technologies' Modeler simulation software and runs on a 1.9-GHz Microsoft Windows 2000 server with 2 GB of RAM. It gets real-time data on the weather service's IP/frame relay WAN from the agency's Agilent NetMetrix network monitor.The weather service has already seen a return on its $250,000 investment in ACNM, Chambers says. Aside from the database server savings, the simulation tool also helped the NWS determine the bandwidth it needed for a new radar system. ACNM found the radar system requires DS-3 circuits rather than additional T1 lines, which saved the agency millions of dollars in unnecessary T1 costs.

And the weather service may be consulting ACNM again soon, this time to help solve backup problems with the IFPS Red Hat Linux servers at its 122 forecasting sites. When one of these servers crashes, weather data is automatically mirrored to the central NCF server, which then acts as a backup. Chambers says the NWS needs each server to be able to transmit its configuration file back to the central server within 30 minutes, and then to have its files available within 10 minutes so forecasts aren't missed. That won't be easy. "How are we going to get a 20-MB file over this network from each site within a half an hour?" Chambers wonders. The weather service will simulate compressing the configuration files over ACNM. That way, NWS can determine how fast the files would travel using a new compression technique developed for meteorological data.

Changes Like the Weather

The IFPS application uses Esri ArcGIS' geographical information system package for customizing forecasts down to neighborhood streets, as well as its PNG (portable network graphics) and text-based displays of the weather data stored in the database. It generates the PNG files on the weather service's intranet server, and later this year, those graphics will be available on the weather service's public Web site. IFPS and NDFD ride atop the NWS' IP-frame relay WAN, the Advanced Weather Interactive Processing System network.

Although the NWS typically develops much of its own code, the agency's recent $10 million Red Hat 7.3 Linux makeover is a sign of the times: NWS' move from Unix to Linux is nearly complete, and now it's looking at other open-source code like MySQL for its database archive at the NCF. That doesn't mean it will throw away any of its custom-built software for the database, Chambers says.The open-source push is all about cutting development time and costs for the agency. "If we use open source, we can come up with some new, neat applications and reduce our development costs," Chambers says.

Post a comment or question on this story.

Tell us about you Network and we may profile it in a future issue. Send e-mail to [email protected] or call (516) 562-5914.

Weather Service finds proof is in the concept

The Advanced Communications Network Model simulation tool wasn't the National Weather Service's first attempt at modeling its network. The federal agency had tried it before, but the modeling techniques and tools it had tested were too new at the time, so the initiatives failed. "There was also a shortage of in-house expertise then," says Randy Chambers, operational manager of the NWS' Network Control Facility, or NCF.Timing was everything. The latest simulation tool, OPNET Technologies' Modeler, was a mature product and the NWS had a clear business case for network simulation: NWS was in the process of developing its new National Digital Forecasting Database (NDFD), and the NWS IT group needed a way to measure the impact the new graphics-heavy database would have on its infrastructure--the WAN, servers and applications. "There were a lot of unknowns and a lot of expense associated with that," Chambers says. "So rather than just throwing money at these things, we were trying to find a way to manage our fiscal resources while doing some capacity planning and systems engineering. We had less money, so we had to spend it more intelligently."

Although network simulation seemed the obvious choice, Chambers and his supervisor had to do some prodding and cajoling to sell the concept, especially with the skeptics who had experience with previous modeling tools.

"The folks who had bad experiences with other modeling efforts were not an easy sell," Chambers says. In all, it took about 18 months to get everyone on board with ACNM. What eventually sold the most jaded cynics was the prototype Chambers' team built. "ACNM answered all questions with a high degree of accuracy," he notes. Its test data on server capacity and other readings rang true. "The tool's output was validated with real-world data," Chambers says.

In today's cloudier IT budget climate, however, a project like ACNM probably wouldn't fly. Like other government agencies, NWS is facing millions of dollars in budget cuts. A simulation tool would be considered nice to have but not essential, Chambers says. It's mainly the workstations and server upgrades that make the budget these days. "You're not going to get money for this model--it's just not there now," he says.

Randy Chambers -- Operational Manager, National weather service's; Network control facility, silver spring, MD.Randy Chambers, 49, runs the NCF, the main site for the NWS' massive Advanced Weather Information Processing System network and systems. Chambers heads up NWS' operations, analysis, engineering and product support. He also oversees the primary contractor that runs the NCF and the new IFPS and NDFD systems, Northrop Grumman Information Technology. Chambers has been with the NWS, which is part of the federal government's National Oceanographic and Atmospheric Administration, for 21 years; he has worked in the IT field for 30 years. He holds a B.S. in computer science from the University of Arizona.

Why weather forecasters get the forecast wrong: I don't think NWS forecasters get the forecast wrong. I believe that media newscasters do.

Seriously, though, NWS is continuing to improve the accuracy of its forecasts. We've shortened the lead time for issuing severe storm watches and warnings, and the meteorological models contain more information, which improves precision.

What you can't predict about the weather: Budgets for the systems that collect weather data.

Biggest mistake made in technology: Falling in love with bleeding-edge technology and forgetting the process for implementing it.When it's not just a job: I enjoy thinking of ways to stretch the IT envelope to improve customer support and services. The product isn't any good if no one wants it. Each year, the NWS issues a customer service/support survey to our 135 field sites. The results from the latest survey two months ago indicate that 92 percent of the field sites rated the NCF's level of service to be good or excellent.

Best advice: Three things I live by: I work for the taxpayer as a public servant. Take the job seriously, but don't take yourself seriously. Understand to be understood.

For fun: Lifting weights, working out, music, reading.

Wheels: 1999 Honda Passport. I drive it on my 160-mile round trip to work each day from Fredericksburg, Va., to Silver Spring, Md.