Can Linux Be Made To Do 'Hard' Time?

Chicago — For Kevin Morgan, hard real-time Linux is a matter of determination. Never mind that much of the embedded industry is leery. Never mind that the history of Unix is littered with the tales of others who have tried to write hard real-time kernels but have been eaten up by the effort. Morgan sees it as an issue of plain old hard work.

"This isn't rocket science," said Morgan, the vice president of engineering for MontaVista Software Inc. (Sunnyvale, Calif.). "It's a 'get down and do the heavy lifting' kind of problem."

Tomorrow, MontaVista will show off its muscles as it rolls out a reference implementation of a hard real-time kernel that it claims far exceeds the real-time capabilities of existing versions of Linux. During the coming months, the open-source software maker will take input and collaborate with the Linux community on issues surrounding the new technology. And while it does that, MontaVista is simultaneously laying plans to unveil a product based on its real-time kernel in early 2005.

"We are going to work with the open-source community, but we're not waiting for [formal adoption] before we take these technologies to market," Morgan said.

If embraced, the "Open Source Real-Time Linux Project," as MontaVista calls it, could open new avenues for Linux in mobile handsets and telecom switching equipment, where it typically must now reside alongside a separate real-time operating system (RTOS) in order to ensure real-time responsiveness. Moreover, the technology could have applications in other areas, such as factory automation, defense and aerospace, where real-time capability is considered critical."If they can provide a stable version of a hard real-time kernel, then they will definitely have an advantage over everyone else," said Daya Nadamuni, senior analyst for Gartner Dataquest Inc. (San Jose, Calif.). "Having real-time Linux only accelerates Linux's acceptance into the deeply embedded space."

Real-time roles
By trying to deliver real-time performance to the kernel of a Unix-like operating system, MontaVista follows in the footsteps of others that have attempted to do the same over the past three decades. Earlier Unix descendants, such as Silicon Graphics' Irix, Hewlett-Packard's HP-UX and Sun's Solaris, have been the target of efforts to improve responsiveness and determinism — with limited success. More recently, the open-source community delivered the 2.6 version of Linux, which offers some limited forms of real-time performance.

MontaVista, however, plans to take Linux's real-time performance to a higher plane, promising that its new effort will deliver two to three orders-of-magnitude improvement, enabling hard real-time performance with 100-microsecond response times. Though not strictly defined, "hard" real-time performance generally refers to a system's ability to meet its prescribed timing requirements without fail, every time. Most such systems offer response times of a few milliseconds or better. "Soft" real-time systems, meanwhile, meet their timing requirements most, but not all, of the time.

"This dramatically reduces the latencies and results in much better responsiveness," said Jacob Lehrbaum, product manager for mobile and wireless at MontaVista. "Eventually, our goal is to have real-time performance as good as the proprietary RTOSes."

MontaVista envisions the real-time software being used in dual-processor smart phones and single-processor low-end phones. The company's engineers also foresee its use in telecom switching and infrastructure equipment, where it could lower costs."We've seen strong traction for Linux in the communications market, on the phone side as well as the infrastructure side, so that's where we're going to focus," Lehrbaum said.

The effort to create a real-time Linux kernel, however, has implications far beyond the communications market. Real-time operating systems are considered critical in much of the industrial world, in factory automation and machine design, as well as in defense and aerospace. Moreover, developers have voiced a desire for a real-time version of Linux. In a study earlier this year by Venture Development Corp., engineers described real-time limitations as the most important factor inhibiting Linux adoption.

MontaVista engineers said that although they do not plan to target real-time Linux at markets outside of communications, they believe their "improved Linux kernel" could be applied in other arenas. "We're not talking about pushing Linux into smart bombs or any traditional embedded, super-real-time applications," Lehrbaum said. "But the technology could meet other industrial needs."

Up until now, Linux has not been viewed as a true, hard RTOS at any speed. Running software with precise timing and always meeting prescribed deadlines have consistently eluded it in the past.

To be sure, those who want real-time Linux have had an option: real-time extensions, such as those made by FSMLabs Inc. (Socorro, N.M.), which run Linux as a non-real-time thread of a real-time operating system. FSM has been successfully engineering such systems since 1996. Many engineers in the industrial world are openly skeptical of doing more. "People tried to make Solaris real-time," said Victor Yodaiken, chief executive officer of FSMLabs. "They tried to make HP-UX real-time; they tried to make SGI Irix real-time. And no one has been able to create a general-purpose Unix kernel that does all the things an RTOS is supposed to do, and do very well."MontaVista says, however, that it has reached that plateau, and hopes to rally community support. The keys to its effort center on two areas of architectural enhancement to the Linux kernel: multitasking and prioritization.

The company says it is improving Linux's multitasking capabilities so that real-time tasks can preempt any other tasks at any time. "If that real-time application needs to access a locked, critical region within the kernel, then the application in the locked region will get a priority upgrade, so that it can get its job done and get out of the way," Lehrbaum said. "That way, the real-time task can take over."

MontaVista engineers say they have also improved prioritization capabilities, so that hardware tasks can no longer preempt everything else. The new approach allows the system to schedule hardware events on a priority basis, so that a hardware task can no longer bump a more important real-time task.

MontaVista is rolling out the technology this week for community comment and expects to deliver products based on it sometime next year. The company also said it is working with customers to build the new technology into their embedded products. "Our strategy has been to push the technology to open-source, to encourage collaboration and adoption," Lehrbaum said. "When you start to add proprietary extensions at the core OS level, it leads to forks and single-vendor tie-ins. It's not the same as real open-source."

Experts acknowledge that true real-time performance could eventually reach Linux, but they say that such performance would require major innovations."There's some pretty smart guys who know Linux, and they've all decided to go the extension route," said Nat Frampton, president of Real Time Development Corp. (Pearl River, La.), a company that specializes in development of real-time software systems. "It would take a major rewrite of Linux to give it hard real-time performance. It's possible, but it's no small task."

Competitors also argue about the assumption on which MontaVista's technology was built, primarily that the 2.6 version of Linux provided some cursory level of real-time performance. Some, such as Yodaiken of FSMLabs, say that they were easily able to cause non-real-time delays of up to a half second in applications using that Linux edition. The reason for those delays, Yodaiken claimed, is that Linux still is inherently non-real-time. "It's hard to design a concrete mixer to do good cornering," Yodaiken said. "You either want a sports car or you want a concrete mixer. They can't be one and the same."

Competitors also question whether any software maker can push new technology into the open-source community. "Technologically, we're getting closer and closer to real-time," noted John Bruggeman, chief marketing officer for Wind River Systems Inc. (Alameda, Calif.). "But even if the technology is available, getting the open-source community to embrace and adopt it is a much, much harder problem. Several vendors have tried to champion technology and drive it into the community, and it has not been embraced. It's a 'show-me' community. They want to see it and use it first."

MontaVista executives, however, insist that similar efforts with HP-UX, Irix and Solaris struggled because they were aimed at servers and workstations, and not motivated by the necessities of the embedded marketplace. Furthermore, they say, they've collaborated with the open-source community for many years and know how it works. "We're taking advantage of previous open-source efforts in this area and adding our own vital components," said Morgan of MontaVista. "We've got the power of open source behind us."