Web Application Servers

To find a bean-counter-approved solution, we invited BEA Systems, Borland Software, IBM, Microsoft, Novell, Oracle and Sun Microsystems to vie for a coveted spot within our infrastructure. Four products came our way: IBM WebSphere Application Server (WAS) Network Deployment Edition 6.0, Novell Extend 5.2, Microsoft Windows Server 2003 with .Net Framework 1.1, and Sun Microsystems' Java System Application Server 7 (AS7). BEA, Oracle and Borland ultimately declined our invitation. Oracle refused because we wouldn't supply our custom application code to its engineers before testing--standard policy for us--and Borland and BEA gave us no explanation for not participating.

After thorough testing--and retesting again and again--we confirmed what the vendors told us at the outset:

Server-clustering technology often is only a boon if you invest in a second--and a third--server. Although every application server can scale using clustering and load balancing, very few provide the increase in capacity we expected. If one application server can handle 500 messages per second, we hoped a second instance would handle 500 more. Perhaps that's unrealistic due to the resources consumed by the load-balancing daemons, but we anticipated at least some performance increase to justify the full cost of the license for a second instance. Novell's and Microsoft's application servers cleared the hurdle, but both servers ran at 100 percent CPU utilization to do so.

Although we'd prefer external hardware load balancers, our budget didn't permit this option. We simply had to choose from the products we tested. Therefore, we looked beyond performance, and, in fact, weighted that feature at only 10 percent in our report card. We considered features and management capabilities more important. The work involved in managing multiple servers becomes more complex with each additional instance. Managing multiple servers efficiently and selecting the right features are critical from the start. Speed can always be increased later.

We also wanted to make sure the application server could integrate with our infrastructure. A unified messaging infrastructure doesn't do much good if your app server can't communicate with that platform. The investment in an identity-management system is lost if you can't harness its capabilities from your chosen app server.Although Microsoft's Windows Server 2003 was the strongest performer, with a multitude of options for topology and deployment, it couldn't match IBM WAS's impressive management options, support for Web services and general ease of use. Microsoft didn't submit its AppCenter management solution, because it would've affected its product's overall price. We're confident that if AppCenter (with its broader management capabilities) had been included, Microsoft would have won this review handily, even with price increase. Considering all these factors, we give our NWC Inc. Choice award to IBM's product.

Management Minuses

You'd think any product that offers clustering would also feature centralized configuration management. Apparently, only IBM and Novell agree--and IBM's implementation is far more capable than Novell's.

We expected to manage a cluster's basic configuration, membership of application server instances and basic functions, such as deploying an application, but we were disappointed by most products' lack of such features. Frankly, in terms of management, monitoring and configuration of multiple instances, some of these products don't add diddly-squat. Most vendors, including IBM and Novell, provide little more than a shared repository, which, to maintain high availability, should be deployed on that third server we're waiting for.We also discovered that most products, with the exception of WAS, wouldn't let us share simple configuration pieces, such as our JDBC connections and data sources, across a cluster. This shortcoming made setting up each new instance feel like busywork.

IBM gets props for WAS's management capabilities. It's easy to deploy a single configuration across nodes in a cluster. Furthermore, the single, centralized console provides a slick method of managing multiple application servers, Web servers and even unmanaged application nodes. Although Novell handles application deployment across the cluster--a nice touch--no other product could share so much configuration with so little work as WAS.

No, we don't have an irrational fear of speed, but we're beginning to think app servers do. IBM's WAS and Sun's AS7 seemed to have endless performance problems when serving up our custom Java app. We began to wonder if it was our code's fault, but then Novell's Extend, a J2EE solution as well, chugged along and performed at least as well as Microsoft's C# product. We just wish Extend's price wasn't as high as its performance numbers.

We configured each product to serve J2EE applications, using the same JDBC connection pool to our Oracle9i database and the same data-source configuration, but only Novell handled our scenario without huffing and puffing. We expected a load-balanced configuration to provide improved performance, but it didn't, though Microsoft and Novell came close.

At this point, we determined that to double the performance of one instance, we'd need to add a third server for load balancing. We tested our theory by adding an F5 Networks Big-IP 5100 to our setup, to perform simple Layer 4 load balancing in front of two instances of Sun's AS7. In this configuration, AS7's performance on our micro-benchmark doubled, from 1,000 transactions per second to 2,000 TPS. In the dual-server configuration with Apache at the front end to facilitate the load balancing, however, performance actually degraded, to 620 TPS. This experiment convinced us these products' load-balancing components must, without a doubt, be deployed on their own servers. Plan to cough up an extra $3,000 to $5,000 for the additional hardware.Microsoft's Network Load Balancing (NLB), which load balances at Layer 2 only, performed best. Although performance didn't always jump twice as high going from a single to double configuration, it did increase--which is something only Microsoft's and Novell's products managed. Only Microsoft IIS was configured with an ODBC DSN (data source name), rather than JDBC, pointing to our Oracle9i database with the appropriate driver from Oracle.

Some of the tuning that can increase performance in both IBM WAS and Sun AS7 concerns itself with JVM settings and, in the case of our custom application, JDBC pool properties. Both products need some heavy tuning to boost their performance. During AS7's trial, we consistently had problems with JVM settings and our JDBC connection pool--most often a lack of available connections through which to service client requests. We also encountered problems with the number of concurrent connections to the application server and discovered that this was a prime contributing factor to decreased performance for both IBM WAS and Sun AS7. Both perform far better with fewer concurrent connections, which is consistent with our experience lab-testing Java HTTP servers. Performance starts to degrade sharply after 50 connections are made.

At least IBM's and Sun's GUI administrative consoles make it easy to tweak configuration settings. Making the same adjustments on Novell's and Microsoft's products frustrated us, as it was difficult to modify database-connection pool settings.

Web Services SupportWhile testing these four application servers, we examined their ability to support Web services. We were impressed with WAS's specific Web services management and configuration options in the administration console--specifically, its support for security standards, such as WS-Security configuration and bindings.

We used each product's development tool to build a custom application and generate a Web Services Definition Language file; then we tested the WSDL for WS-I Basic Profile compliance. We were not surprised when the products from Sun and Microsoft both passed with flying colors, nor when Novell's product failed--Novell isn't a core member of the Web Services-Interoperability effort. We were surprised, however, when IBM's WSDL failed, as IBM typically shows great dedication to open standards and interoperability.

The failures for both the Novell and IBM setups were minimal, however, and did not affect the ability of cross-platform, cross-language clients to interact with the services, so we didn't fail the two in our grading.

IBM's IDE (integrated development environment), WebSphere Application Developer 5.1.2, notified us that the WSDL it generated was not in compliance with the WS-I Basic Profile. Although we were glad it mentioned this, we were disappointed it generated a WS-I Basic Profile noncompliant WSDL in the first place.

So here's the bottom line: If you can, wait till you can implement a three-server configuration. Based on our testing, we don't recommend a high-availability, redundant configuration for these application servers with only two servers. If you're more concerned with redundancy to protect the availability of the service, and if you absolutely can't wait--as we couldn't--then move ahead.IBM WAS's performance wasn't as impressive as we'd have liked, but we can always add physical servers and instances of the application server later. WAS's above-par feature set, functionality and ability to manage multiple instances as easily as a single instance will alleviate the work--and the expense--of making those changes.

WAS is a delightful combination of powerful management tools and elegant configuration options that are certain to make an admin's life simpler. Its ability to share configurations across multiple instances and centrally manage instances of both the application and Web server more than compensated for the performance problems we found in our dual-server scenario.

WAS's agent-based management scheme makes centralized management of any number of application and Web servers a breeze, whether in a cluster or not. We were delighted to stop, start and deploy our three custom apps and server instances from a single console without worrying about batch files or separate tools. You can configure agents to communicate using SOAP or Java RMI (Remote Method Invocation; we chose SOAP, because we like to see products eat their own dog food). This was a marked difference from Sun's approach, which required manual configuration and management of individual instances.

New options specifically for Web services, such as WS Security bindings and configurations, are available directly from the administration console. IBM has updated the console with extensive navigation aids that walk you through configuration of options such as JDBC resources. We'd still like to see a task-based navigation scheme in all these products for commonly configured options, such as data sources.WAS uses the same load-balancing architecture as Sun's product, taking advantage of load-balancing plug-ins for Apache, IBM HTTP Server and other HTTP servers. This is a typical setup in which requests received by the HTTP server for specific URIs are handed off to a load-balancing listener on the application server, which determines where to route the traffic. The console makes it simple to add nodes, or instances, whether they are local to the deployment manager instance or remote. A load-balancing algorithm similar to weighted round robin is the default, with few buttons or knobs available. This algorithm uses server affinity to further modify the server chosen to service a request. New requests are serviced by a chosen server using weighted round robin, but the application server attempts to send subsequent requests from the same client to the server that previously serviced its request using session ID and IP information from the client.

At the application level, there are many knobs and buttons to play with, including those that control state and persistence and determine how applications are handled when served by a cluster. None of this applied to performance, however, and though we were pleased with the 600 TPS WAS supported in our micro-benchmarking tests, we were disappointed by the 15 TPS it handled with our custom application. As with Sun's implementation, clustering didn't improve.

WAS continues to provide the performance monitoring servlets of previous generations. Although we enjoy the granular detail from the performance viewer--which offers a view of statistics that is on par with the detail offered by Microsoft--IBM, like Sun, provides absolutely no eye candy. We'd appreciate a visual method of viewing specific statistics from both products. We don't need to see every stat visually represented, but a picture of the current load on application servers would be a nice start.

WebSphere Application Server 6.0, $4,000 per CPU license, unlimited number of servers. IBM, (800) IBM-4You. www.ibm.com

Microsoft's .Net Framework 1.1 coupled with Windows 2003 Server's NLB features provide an excellent environment for deploying Web services. There haven't been many changes in management or deployment of .Net applications since the appearance of the .Net Framework 1.0, but the deployment's stability has vastly improved with version 1.1.NLB and .Net 1.1 perform well as a team, taking a different approach to a typical clustered application server environment. If we had one, Microsoft would win the "most innovative use of a hub" award: It's configuration requirements specify using a hub on secondary NICs to provide load balancing and failover. To achieve stateful failover, all sessions, regardless of which server in a cluster the client has been directed to, must be mirrored across the entire cluster. Some setups, like Sun's, use a shared session database to provide this functionality. Others, such as Novell's, simply don't offer stateful failover. Microsoft doesn't use a shared database and instead exploits a hub's typical function to broadcast all packets received to all ports to mirror session data across the cluster.

This configuration required a second set of NICs and a $30 Linksys 10/100 Hub, but worked like a charm. The VIP (virtual IP) for the load-balanced cluster listens on the secondary NICs and routes incoming traffic to the appropriate server. Return traffic is routed out the primary NIC--much like the direct-server-return configuration available for hardware load balancing.

It takes a bit more work to replicate applications across Microsoft's cluster setup compared with doing the same thing with WAS, but as is often the case with Microsoft products, numerous options are available for implementing such an architecture. The simplest is to make applications available on a shared network drive, but this approach provides no failover or redundancy. We used Windows 2003 Distributed File System. which masks a file share as a virtual share on a virtual machine and can, if configured, replicate data across shares to provide a redundant environment. We configured a share on two machines, then created a DFS entry that replicated data bidirectionally and deployed our app to the share simply by copying the directory from the development machine to the server. Using IISM (Internet Information Services Manager), we created a virtual Web site whose home directory was the application directory on the DFS share.

The only negative to this approach is that the configuration within IIS had to be done manually on each server in the cluster. Microsoft said there are ways to configure all instances of IIS to point to the same configuration store, but it seemed reluctant to test the product with such a configuration. Regardless, the configuration worked just fine. Both servers picked up changes to applications within seconds of deployment.

Monitoring is, of course, separate from management and is available using the standard Windows Performance Monitor. Although the counters are numerous, and we could examine, in real time, just about every aspect of the .Net framework, we preferred Novell's performance monitor, which is integrated within its management console. Microsoft's performance monitoring is more extensive, but Novell's is easier to find and use. We wouldn't want Microsoft to reinvent the wheel, but we'd like to see some integration of performance monitoring within IISM.Our only beef with Microsoft IIS is that if you choose the platform, you lock yourself into its technology. Although every J2EE vendor has its own method of supporting Web services, none affects your base code. You can take the same code from one platform to another, regenerate the Web services and be on your way. If yours is a Microsoft shop and your staff has skills in C# or another Microsoft-specific technology, we have no problem suggesting you dive in and continue with Microsoft for your Web services deployments. If you're building Web services from scratch, you might also want to consider Microsoft's product set. From a development perspective, C# is Java, with a few minor changes--and it can serve you well. But if you're moving traditional J2EE client-server code, you'll likely want to stick with a J2EE technology or think hard about the amount of work needed to port to this nonportable technology.

Microsoft Windows Server 2003 Standard Edition, starts at $999. Microsoft Corp., (800) 426-9400, (425) 882-8080. www.microsoft.com

The core of Novell's Extend 5.2 suite hasn't changed much in the past two years. The noticeable changes lie in its Design tools, Director and Composer. Although the improvements are valuable, we'd like Novell to enhance its application server and management console.

Extend 5.2 still provides excellent performance and some sexy real-time monitoring, but we prefer WAS's Web administration console to Novell's Java-based, fat administration client. Some features, such as configurations that affected clustering and repositories, were difficult to manage effectively using the admin client.

Novell employs a shared repository for its clustering architecture. To configure multiple instances of Extend application servers, you must use Novell's SilverMaster database, then add those instances to a cluster. We discovered some of the admin console's problems when we installed two instances of Extend but didn't configure them to use a shared repository. There was no way to change this configuration in the administration client and attempts to modify the configuration file manually were unsuccessful. We ended up reinstalling the second instance of the application server and specifying the shared repository during the installation.Three other components are required to configure clustering and load balancing within Extend: LoadManager, which monitors instances; CacheManager, which indicates instances when configuration changes have occurred; and Dispatcher, the process that receives clients' requests and directs them to the appropriate instance of Extend based on a weighted round-robin load-balancing algorithm. Dispatcher uses HTTP 307 (redirect) codes to send the client to the appropriate server. Neither Novell nor we recommend this architecture. It's fine for small applications, but because the architecture necessitates a stateless failover model, we agree with Novell that hardware load balancing would better serve critical apps.

We easily deployed both our micro-benchmark code and our NWC Inc. application to Extend. Extend Designer, one of two IDEs Novell offers, creates a standard J2EE (1.3) WAR (Web archive) file and uses Ant scripts as the basis for deploying apps to its application server. The integration between Extend Designer and the application server is seamless. CacheManager immediately detected deployment of the apps to both one and two instances in a clustered environment.

Novell Extend 5.2, Profession Suite, $50,000 per CPU; Enterprise Suite, $120,000 per CPU. Novell, (888) 321-4272, (781) 464-8000. www.novell.com

AS7 hasn't changed its look much since we last tested it (ID# 1406f2), but this time we examined the enterprise edition instead of its base application server. By adding clustering and high-availability services and making changes to its IDE, Studio, Sun has turned this suite into a viable contender for Web services deployment. The most notable of these changes is the ability to use Java API for XML-based remote-procedure calls (JAX-RPC) 1.1, which offers doc/lit encoding for Web services on the platform and enables WS-I Basic Profile compliance.

Sun's flexible clustering architecture lets you deploy a stateful failover, shared-nothing environment over its high-availability database component. Configuring clusters is still a manual process that requires editing the Web server's configuration file under Sun's Java System Web Server, the architecture we chose for this test. AS7 also can be load-balanced by fronting it with an Apache or IIS Web server, or, like the other products we tested, using hardware load balancing. Load balancing between AS7 instances is accomplished with an industry-standard round-robin algorithm, much like Novell's implementation.Much of AS7's administration--including replication across the cluster of the WAR files containing our Web services implementation--is still accomplished using Ant scripts and command-line interfaces. Using the Web administration console for deployment is painless on a single instance, but would be intolerable for tens or hundreds of instances. We suggest automating this task using the command-line tools.

Performance monitoring is available by several methods--the easiest of which is an auto-refreshing Web page that contains just about every statistic you could dream of. SNMP is supported as well; in fact it is the only method of monitoring available for configuration over the administration console. We preferred Microsoft's implementation to both Novell's and Sun's offerings in this category--Microsoft's was the only one to continue reporting under heavy load.

We were disappointed with Sun's performance, but what really drove us up the wall was the number of defects that appeared in this released version. Customers expect core functionality, such as the load-balancing plug-in for its Sun Java System Web Server, to be complete when a product ships, and Sun's entry was not. We've seen signs of problems with released Sun products several times now, and we worry that the company's quality-control efforts have declined.

Sun Java System Application Server 7, Enterprise Edition, $10,000 per CPU, or with the Java Enterprise System at $100 per employee. Sun Microsystems, (800) 555-9Sun, (650) 960-1300. www.sun.com

Lori MacVittie is a Network Computing senior technology editor working in our Green Bay, Wis., labs. She has been a software developer, a network administrator and a member of the technical architecture team for a global transportation and logistics organization. Write to her at [email protected].Technically, the four products we tested support clustering in a dual-server configuration. Practically, though, our tests showed there's virtually no performance gain. The vendors that sent us their packages--IBM, Microsoft, Novell and Sun Microsystems--told us we'd need a hardware load balancer to make our application servers run the way servers should in a cluster, and they weren't kidding.

With a limited hardware budget for NWC Inc., our fictional retailer (for more on our 24/7 production environment and business applications lab, go to inc.networkcomputing.com), we set out to improve our app server's availability. So we're temporarily suppressing our need for speed and instead focusing on what we demand from any software: solid integration, good features and manageability. At least we'll get a small boost in availability and, eventually, we'll spring for that third piece of hardware.

We tested IBM WebSphere Application Server 6.0, Microsoft Windows Server 2003 with the .Net Framework 1.1, Novell Extend 5.2 and Sun Microsystems' Sun Java System Application Server 7, and found IBM's product has the right mix of management, support for Web services and ease of use. We gave it our NWC Inc. Choice award for these reasons.

We know from experience that XML processing makes short work of application servers, so we focused on clustering and load balancing as a means of scaling applications in our NWC Inc. business applications lab.

We installed two instances of each application server and deployed two applications: one designed to let us perform micro-benchmarks, based on interoperability testing, and a second, custom-written application providing order-entry operations for our business partners and requiring connectivity to our Oracle9i database. Each instance of the application servers from IBM, Novell and Sun was deployed on its own dual Xeon processor (2.6-GHz) server, with 1 GB of RAM, running Windows 2000 SP3. Microsoft's submission was deployed on the same hardware, but required Windows 2003. Each application server was configured to provide a data connection to the relational database over a connection pool using Oracle's thin-client driver, except for Microsoft's, whose ODBC-based connectivity over IIS required Oracle's thick driver.The process of configuring clustering and load balancing gave us a way to evaluate the management capabilities of each application server, while deployment gave us a way to examine the platforms' integration with development tools.

We tested performance using both the micro-benchmarks and our custom application. Both apps were tested using a single instance and in a load-balanced setup. Client load was generated using a Spirent Communications' Avalanche 2500.

Client request and responses varied in size. Our breakdown of files and percentage of traffic is shown in the table below.

The base code for all J2EE application servers was the same, with differences only in the code generated by each development environment to support a Web services deployment. Code for Microsoft IIS was developed in C# and was as syntactically close to the Java code as possible, allowing only for differences in types--not business logic.We wrote the code, just as you might write code for tests, because we wanted to see how each server performed using our applications, not a test app written to obtain pie-in-the-sky performance numbers. Web services were generated and deployed with the WS-I Basic Profile in mind, requiring doc/lit encoding.

WS-I compliance was verified through the use of Mindreef's SOAPScope 4.0. We required each platform to generate a WSDL for our custom application and ran that WSDL through SOAPScope to determine the level of conformance with the WS-I Basic Profile.

All Network Computing product reviews are conducted by current or former IT professionals in our Real-World Labs® or partner labs, according to our own test criteria. Vendor involvement is limited to assistance in configuration and troubleshooting. Network Computing schedules reviews based solely on our editorial judgment of reader needs, and we conduct tests and publish results without vendor influence.

R E V I E WWeb Application Servers

Sorry,
your browser
is not Java
enabled

Welcome to NETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.Click here for more information about our Interactive Report Card ®.