Sun's Multithreaded Niagara Servers Flow

SUNNYVALE, Calif. — In a historic bid to win back a slice of the mainstream server market, Sun Microsystems Inc. will roll out the first two systems based on its multicore, multithreaded Niagara processor this week. Other computer makers likely will follow the lead Sun is taking with its CPU, which supports up to eight cores and 32 threads and will integrate PCI Express in its next iteration.

Sun said the new rack-mount systems provide more than twice the performance at half the power consumption of competing servers based on the current Intel Xeon processor. The systems deliver four times the performance/watt of IBM's p5 550 server based on the Power5+, the company said. A new Sun T2000 2U rack with a 1.2-GHz Niagara and 32 Gbytes of memory hit about 615 on the Spec J App server benchmark while consuming 325 W.

The design marks Sun's aggressive shift to using Moore's Law to drive down the cost of a compute thread rather than drive up the frequency of a single hot processor. "As the cost of a thread goes from $500 to $100 and someday to $20, the software developers will jump all over it, pushing the development of parallel apps," said Mike Splain, a Sun Technology fellow and distinguished engineer.

"Sun used to own the low-end server space before the X86 came in," said Nathan Brookwood, principal at market watcher Insight64 (Saratoga, Calif.). "Now they have a chance to reclaim lost territory. The chip has compelling performance, and even more compelling performance per watt which has become a huge issue recently.

"The question is whether Sun's advantage will be enough to convince people accustomed to X86 economics and broad software availability to move into the Sun camp with its relatively proprietary processor and operating system," he added. "Historically, the proprietary guys have been losing out to industry-standard approaches."Brookwood praised Sun for realizing the importance of thread-level parallelism in the transaction-intensive Internet era of computing. By contrast, competitors Intel and Hewlett-Packard developed their Itanium server microprocessor focusing on instruction-level parallelism at a time before the Web boom hit.

That's not necessarily a drawback, said Jerry Huck, a fellow in HP's server group who helped design the original single-threaded Itanium. Itanium has better single-thread performance and use of cache memory, he said. "We are finding to give up much on core performance is not cost-effective," he said. "It's better to take as much core performance as you can and then replicate processors."

With Niagara, "I can now have 30 threads that are basically thrashing the cache and come out way behind," he said.

Sun's new servers leverage as many mainstream server components as possible. The 1U T1000 and 2U T2000 use the same chassis, fans and power supplies as Sun's Opteron-based X86 servers. The 70-W Niagara processor, made in a 90-nm process by Texas Instruments, requires just three fans rather than six in the Opteron systems.

A Niagara processor board with up to eight DDR-2 DIMM slots links to Sun's existing J-Bus bridge chip on a separate I/O board that is otherwise populated with off-the-shelf PCI Express parts. The I/O board has two 8x Express slots and four Gbit Ethernet ports.System prices start at $2,995 for a T1000 using a six-core version of the CPU running at 1 GHz with 2 Gbytes of memory and no hard disks.

The system marked Sun's first foray into PCI Express, though PC servers have been shipping with the 2.5-Gbit/second serial interconnect for months. The Sun systems have been running at a handful of user data centers for several weeks.

Next year, Sun will roll out Niagara chips in server blade designs using both the ATCA chassis for telecom customers and a smaller form factor for commercial users. Ultimately Sun will sprinkle Niagara broadly among Web and application tier products.

Express integration

Some of the most interesting aspects of the Niagara rollout are the glimpses Sun allows into its next generation. Niagara-2 will use a 65-nm process to open up more die area, which Sun will fill with encryption and networking coprocessors and serial interfaces.Niagara-2 will double the single-thread performance of the first-generation device, in part by greater sharing of hardware and other tricks the company will not yet disclose. It will still have a maximum of eight cores, but those cores will run eight rather than four threads each, without creating significant resource contention.

"We are seeing linear scaling [on the 64-thread version]," Splain said.

The processor will also be Sun's first to go entirely serial. It will integrate an 8x PCI Express interface to eliminate the separate J-Bus chip. It will also integrate a new ground-up processor interconnect to open the door to systems with two Niagara chips.

Putting Express directly on a server processor is new, but "this will be a trend," Splain said. "Competitors are doing it, and you will see more of it."

Sun has access to 8x Express through LSI Logic, which supplied the intellectual property for the current J-Bus bridge chip. It also has an internally developed Express block.The resulting Niagara-2 board may have as few as half the 22 layers in the current pc board. The board also will be significantly smaller and less costly because not only the Express interconnect but a multiport Gbit Ethernet switch will be integrated in Niagara-2.

Also, Niagara-2 will include hardware support for bulk encryption algorithms including AES, 3DES and elliptical curve cryptography. By putting most of the I/O board components into the processor, Sun makes room for the two-chip system.

DIMMs get nod

The Niagara-2 server will also use fully buffered DIMMs. These emerging memories are initially expected to add power consumption and latency to server designs, but "this technology is the best solution to the memory expansion problems all server makers have," Splain said.

Sun acquired the Niagara design in July 2002 from startup Afara Websystems Inc., led by former Sparc designer Les Kohn. Most of Afara's 60 processor designers stayed on at Sun. Sun veteran Rick Hetherington became chief architect of Niagara.The Afara design, similar to the Niagara-2 plan, included networking, encryption and a proprietary interconnect to link multiple CPU cards in a chassis. Sun tore out much of the integration for the first-generation part in an effort to reduce die size, technology risks and time-to-market.

Encryption in the Afara device required a 3- to 4-mm2 block per core. "At 90 nm, that took a fairly large part of the die area," Splain said. As for Ethernet, "the value proposition was the threads, not networking," he added. The first-generation Niagara still has hardware acceleration for modular arithmetic to speed operations.

Sun also took off the PCI-X interface on the Afara device, choosing to use its J-Bus instead as a transition to Express. The company also removed the proprietary clustering interconnect, something it's keeping in a back pocket, potentially for Niagara-2. "We left the hardest parts alone: the core, threads, cache coherency, crossbar switch and load/store mechanisms," Splain said.

Afar had incomplete RTL code and a "fairly immature" design when it was acquired. "They were targeting a different fab as well, not TI," said Hetherington.