Empirix's FirstAct has been on the market for more than a year, but the need for such a tool is just beginning to grow. In version 2.0, Empirix has provided numerous enhancements to strengthen its offering.
FirstAct 2.0 is available for Microsoft Windows NT/2000 systems and requires that both Microsoft Office Web Components and Microsoft SOAP (Simple Object Access Protocol) Toolkit 2.0 be on the system before installation. FirstAct uses distributed agents to simulate virtual users for stress-testing services. It provides excellent support for WSDL (Web Services Description Language) and automatic test-case generation based on WSDL files. These enhancements let nontechnical users test Web services without substantial training. FirstAct 2.0 supports both the document/literal and RPC/encoded message formats. It also supports regression and performance testing of COM/DCOM objects as well as Web services.
On the downside, FirstAct 2.0's reporting could be better. You cannot create customized reports, and test-result displays are often awkward to navigate. Although the software supports multiple virtual users, you can view the results of only a single virtual user at a time. To make matters worse, there is no way to view aggregated results. Performance results, however, can be graphed against time, by concurrent virtual users (load) and by method. The performance data is reported at a high level, though there is no differentiation between time spent in transit versus time spent on the server processing requests. I'd like to see more flexibility in FirstAct's reporting capabilities.
FirstAct can meet your functional-verification, regression-testing and stress-testing requirements. You'll be able to verify functional correctness and determine what type of load your service can handle, but you won't be able to perform root-cause analysis or determine whether the server or your network is causing poor performance.
Cover Your Web Services
Support for WSDL and automated test-case generation are the most exciting improvements in FirstAct.
|
Good News
Automatically generates a variety of tests based on the WSDL file for a Web service.
Offers functional-verification and regression-testing support.
Has flexible support of Web service platforams.
Bad News
Performance data is not granular.
Requires installation of third-party software (Microsoft SOAP Toolkit and Office Web Components).
Error reporting not very intuitive.
|
The application can now read and parse the WSDL for a Web service and not only determine the syntactical correctness of the WSDL but use the information to determine what Web services are available for testing. And its test-case generation tools (Quick Cover, Profile Cover and Expert Cover) provide three test-built variations that can be used with little work on your part.
After coding a couple of simple Web services and deploying them on an Apache SOAP implementation, I was ready to begin testing them with FirstAct. Only a few simple steps are necessary to test a Web service. First, I pointed FirstAct at the WSDL file describing one of the services and selected the Greeter service for testing.
After you select Web services for testing, you must build one or more test cases. You can create test cases manually or use one of three automatic test-generation options. Quick Cover will generate a single test case including one remote call for each method of Web services selected for testing.
The Profile Cover generates a set of test cases based on a likelihood matrix, letting you define the order in which methods are typically called. For example, you would use a matrix to specify that a customer should call the "login" method before calling the "getAccountBalance" method. This test case is particularly useful for testing interactive Web services that may require two or more steps, such as when an initial authentication step is required before performing additional queries.
The Expert Cover option generates test cases based on user-configurable rule sets. This can include rules indicating the order of method calls and specification of relationships between methods. I generated a Quick Cover test for my service, then got down to customizing the test cases. FirstAct determines the parameter types from the WSDL file and can generate test data for each parameter based on that information. My Greeter service took as input a single parameter -- a string -- so FirstAct generated random string-based data as input for each test iteration. The data can be edited or imported manually from a flat file. Generated data is random, and though the data makes little sense (example of strings generated in my tests were Z, Zt and Agftd), it is valid from a syntactical view and can be used as the basis for a preliminary functionality test. For a real test, however, you'll need to edit the data manually so it matches the test cases you've prepared.
Testing Specifics
Once you create the test case, you can change the scenarios. The scenario lets you specify the total number of virtual users to create for the test as well as configure the reporting interval and number of test-case iterations per test round. Test results offer a view of total concurrent users but do not correlate this info to the number of successful or unsuccessful concurrent virtual users.
One option lets you collect the data returned from each method during testing. This is an excellent way to visually compare the data returned with the data expected for the functional-verification process. Also available is the option to compare automatically expected return values with the values returned by the Web service.
Once the test case is configured to your satisfaction, you can run the test immediately or schedule it to be run later. The scheduling mechanism uses the service within Windows. From FirstAct you can specify a single date and time at which to run the test, but if you'd like to perform the same test repeatedly, you'll need to modify the schedule from within Windows. This is an area I'd like to see enhanced in future revisions of FirstAct.
Remote Procedure Call (RPC)" Lets a remote client invoke an application or service on a distant server.
Simple Object Access Protocol (SOAP): An XML-based protocol that makes it possible for applications based on different platforms to communicate with each other over the Internet.
Web Services Description Language (WSDL): An XML-based language that describes the services available from a Web service and the parameters required to obtain it. |
Running the service opens a report window and provides almost real-time information regarding the performance of the tested service. Additionally, the window shows both a summary and a detailed report on the success/failure of the methods invoked during the test.
The performance data reports an average response time aggregated by virtual user. It cannot be drilled into and does not specify how the time breaks down across the network, server or client. Although this is not as relevant when testing on the LAN, it is valuable to break down the response time when testing remote Web services.
To test compatability with the myriad platforms used to deploy Web services, I used FirstAct to test a number of public Web services (gathered from XMethods' extensive list at www. xmethods.net). I had no issues with Web services deployed on Microsoft .Net, Borland platforms or Glue.
I ran into problems with Web services deployed on Apache SOAP and IBM's Web Services Toolkit platform. This was most likely due to a known interoperability issue with the deployment of services on platforms based on Apache SOAP. I had no problems testing my own Apache SOAP–based Web service that had been deployed with the proper deployment descriptor file to ensure interoperability.
Given the dearth of products in this market, FirstAct 2.0 is worth a look. But I'll be happy to see the Web services testing market mature and catch up in functionality and report granularity.
Technology editor Lori MacVittie has been a software developer and a network administrator. Most recently, she was a member of the technical architecture team for a global transportation and logistics organization. Send your comments on this article to her atlmacvittie@nwc.com.
|
RSS
