iPlanet Web Server,
Enterprise Edition 4.0
and
Stronghold 2.4.2
Performance Comparison

By Bruce Weiner
(PDF version, 68 KB)

February 7, 2000
Update to September 29,1999 version

White Paper Contents

Executive Summary
   Conclusions
   Mindcraft Certification
Analysis
Test Details
iWS Configuration
Stronghold
   Configuration

Disclosure

The Sun-Netscape Alliance sponsored the testing in this report. Mindcraft, Inc. conducted the performance tests described in this report at its test lab in Los Gatos, California. 

Acknowledgements

We thank the sales and technical support staff at C2Net for their responsiveness in dealing with SSL issues that arose during the tests.

Executive Summary

The iPlanet Web Server Java servlet performance is 3.9 times faster than Stronghold JServ's on a four-processor system and 4.3 times faster on a one-processor system

Mindcraft focused the performance tests for this white paper on Java servlets, Java Server Pages (JSP), SHTML, and CGI programs as alternatives for implementing Web-based applications. We conducted the tests on the Sun-Netscape Alliance iPlanet Web Server, Enterprise Edition 4.0 (iWS) and the C2Net Stronghold 2.4.2 (Stronghold) Web server

The programs we developed for this white paper were designed to test the speed at which a Web server starts up an application using each of the above alternatives. We implemented the same application both as a C CGI program and as a Java servlet. To determine performance, we looked at the average time to handle a request when a Web server is operating at its maximum number of requests/second (its peak performance). This time is reported as latency, the time from starting a connection to a Web server until the time the last byte of the response is received. Latency is measured at each client test system.

Figure 1 gives the latency for each of the tests we ran on both iWS and Stronghold on a four-processor system. In addition, it shows how much faster each iWS programming alternative is than the same one on Stronghold. The test results show that our Java servlet on iWS runs 3.9 times faster than on Stronghold with Apache JServFinally, the iWS latencies demonstrate that our Java servlet runs 2.7 times faster than our C CGI program. Figure 2 compares the iWS and Stronghold performance on a one-processor system.

Figure 1: iPlanet Web Server and Stronghold 4-CPU Latency at Peak Performance
(smaller numbers are better except for speed advantage)

Figure 2: iPlanet Web Server and Stronghold 1-CPU Latency at Peak Performance
(smaller numbers are better except for speed advantage)

The Analysis section of this white paper explains the importance of testing how quickly an application program starts up. You can understand the concept by imagining a race between two runners. If the faster runner takes longer to get out of the blocks at the start of a race, the slower runner effectively gets a head start. This makes it is possible for the slower runner to win the race depending on the length of its head start, how much faster the other runner is, and how long the race is. Similarly, fast Java servlet start-up, which is indicated by latency, may mean that a real-world Java application can outperform a C CGI application depending on how much of a head start it has, how long it runs, and how fast the CGI application runs.

Figure 3 shows the peak iWS and Stronghold performance on a four-processor system, measured in requests/second. Similarly, Figure 4 compares the performance on a one-processor system. There are no Stronghold results for the two e-commerce tests and for the Java Server Pages test. We could not get a valid run of the e-commerce tests because of an SSL handshaking issue between WebBench 3.0 and Stronghold. C2Net was quite supportive in making suggestions for workarounds but we could not resolve the issue in time for this report. 

As for the JSP test, the JServ Web site recommended using the GNU Java Server Pages software (gnujsp) for Apache, which is the same Web server that forms the foundation of Stronghold. However, gnujsp requires the Java 1.1 SDK instead of the Java 1.2 SDK that both JServ 1.0 and iWS use. We did not want any of the performance differences we might find to be based on using two different versions of Java for the JSP tests. Therefore, we decided not to test JSPs on Stronghold.

Figure 3: iPlanet Web Server and Stronghold 4-CPU Peak Performance Summary
(larger numbers are better)

Figure 4: iPlanet Web Server and Stronghold 1-CPU Peak Performance Summary
(larger numbers are better)

 

Test Environment

Mindcraft tested iWS and Stronghold using the Ziff-Davis Benchmark Operation WebBench 3.0 benchmark. We used  standard WebBench 3.0 tests and developed some of our own tests to show Java servlet and Java Server Page performance as well as to facilitate comparing the various Web server application interfaces. For example, the e-commerce tests we ran used the standard WebBench e-commerce test with the SSL dynamic and regular dynamic programs replaced by a Java servlet or CGI program. The Test Details section of this white paper provides full descriptions of the tests. 

Mindcraft used a Sun Enterprise 450 server with 4 x 400 MHz CPUs and 2 GB of memory to test both iWS and Stronghold. We disabled three of the CPUs for the one-processor test configurations. The system was running the Solaris 2.6 operating system.

Conclusions

  • The iPlanet Web Server lets you deploy high-performance applications as Java servlets instead of C CGI programs.
  • The iPlanet Web Server significantly outperforms Stronghold for all types of dynamic file access.

Mindcraft Certification

Mindcraft certifies that the benchmark results reported in this white paper accurately represent the performance of iPlanet Web Server, Enterprise Edition 4.0,  Stronghold 2.4.2, and Apache JServ 1.0 running on a Sun Enterprise 450 server configured and tested as specified herein.

Our test results should be reproducible by others using the same test lab configuration, the same Sun server configuration, and the same software configurations documented in this white paper.

Analysis and Test Details


Changes

  • Added test results for 1-CPU configurations

NOTICE:

The information in this publication is subject to change without notice.

MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR USE OF THIS MATERIAL.

This publication does not constitute an endorsement of the product or products that were tested. This test is not a determination of product quality or correctness, nor does it ensure compliance with any federal, state or local requirements.

Mindcraft is a registered trademark of Mindcraft, Inc.

The Mindcraft tests discussed herein were performed without independent verification by Ziff-Davis and Ziff-Davis makes no representations or warranties as to the results of the tests.

Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.

             
 
  Copyright 1999-2000. Mindcraft, Inc. All rights reserved.
Mindcraft is a registered trademark of Mindcraft, Inc.
Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective owners.
For more information, contact us at: info@mindcraft.com
Phone: +1 (408) 395-2404
Fax: +1 (408) 395-6324