iPlanet Web Server,
Enterprise Edition 4.0
and
Stronghold 2.4.2
Performance Comparison

By Bruce Weiner
(PDF version, 52 KB)

September 29,1999

White Paper Contents

Executive Summary
   Conclusions
   Mindcraft Certification
Analysis
Test Details
iWS Configuration
Stronghold
   Configuration

Disclosure

The Sun-Netscape Alliance sponsored the testing in this report. Mindcraft, Inc. conducted the performance tests described in this report at its test lab in Los Gatos, California. 

Acknowledgements

We thank the sales and technical support staff at C2Net for their responsiveness in dealing with SSL issues that arose during the tests.

Executive Summary

The iPlanet Web Server Java servlet performance is 2.7 times faster than its own CGI performance and is 3.9 times faster than the Stronghold JServ servlet performance

Mindcraft focused the performance tests for this white paper on Java servlets, Java Server Pages (JSP), SHTML, and CGI programs as alternatives for implementing Web-based applications. We conducted the tests on the Sun-Netscape Alliance iPlanet Web Server, Enterprise Edition 4.0 (iWS) and the C2Net Stronghold 2.4.2 (Stronghold) Web server. 

The programs we developed for this white paper were designed to test the speed at which a Web server starts up an application using each of the above alternatives. We implemented the same application both as a C CGI program and as a Java servlet. To determine performance, we looked at the average time to handle a request when a Web server is operating at its maximum number of requests/second (its peak performance). This time is reported as latency, the time from starting a connection to a Web server until the time the last byte of the response is received. Latency is measured at each client test system.

Figure 1 gives the latency for each of the tests we ran on both iWS and Stronghold. Also, Figure 1 shows how much faster each iWS programming alternative is than the same one on Stronghold. The test results show that our Java servlet on iWS runs 3.9 times faster than on Stronghold with Apache JServ. Finally, the iWS latencies demonstrate that our Java servlet runs 2.7 times faster than our C CGI program. 

Figure 1: iPlanet Web Server and Stronghold Latency at Peak Performance
(smaller numbers are better except for speed advantage)

The Analysis section of this white paper explains the importance of testing how quickly an application program starts up. You can understand the concept by imagining a race between two runners. If the faster runner takes longer to get out of the blocks at the start of a race, the slower runner effectively gets a head start. This makes it is possible for the slower runner to win the race depending on the length of its head start, how much faster the other runner is, and how long the race is. Similarly, fast Java servlet start-up, which is indicated by latency, may mean that a real-world Java application can outperform a C CGI application depending on how much of a head start it has, how long it runs, and how fast the CGI application runs.

Figure 2 shows the peak iWS and Stronghold performance measured in requests/second. There are no Stronghold results for the two e-commerce tests and for the Java Server Pages test. We could not get a valid run of the e-commerce tests because of an SSL handshaking issue between WebBench 3.0 and Stronghold. C2Net was quite supportive in making suggestions for workarounds but we could not resolve the issue in time for this report. 

As for the JSP test, the JServ Web site recommended using the GNU Java Server Pages software (gnujsp) for Apache, which is the same Web server that forms the foundation of Stronghold. However, gnujsp requires the Java 1.1 SDK instead of the Java 1.2 SDK that both JServ 1.0 and iWS use. We did not want any of the performance differences we might find to be based on using two different versions of Java for the JSP tests. Therefore, we decided not to test JSPs on Stronghold.

Figure 2: iPlanet Web Server and Stronghold Peak Performance Summary
(larger numbers are better)

Test Environment

Mindcraft tested iWS and Stronghold using the Ziff-Davis Benchmark Operation WebBench 3.0 benchmark. We used  standard WebBench 3.0 tests and developed some of our own tests to show Java servlet and Java Server Page performance as well as to facilitate comparing the various Web server application interfaces. For example, the e-commerce tests we ran used the standard WebBench e-commerce test with the SSL dynamic and regular dynamic programs replaced by a Java servlet or CGI program. The Test Details section of this white paper provides full descriptions of the tests. 

Mindcraft used a Sun Enterprise 450 server with 4 x 400 MHz CPUs and 2 GB of memory to test both iWS and Stronghold. The system was running the Solaris 2.6 operating system.

Conclusions

• The iPlanet Web Server lets you deploy high-performance applications as Java servlets instead of C CGI programs.
• The iPlanet Web Server significantly outperforms Stronghold for all types of dynamic file access.

Mindcraft Certification

Mindcraft certifies that the benchmark results reported in this white paper accurately represent the performance of iPlanet Web Server, Enterprise Edition 4.0,  Stronghold 2.4.2, and Apache JServ 1.0 running on a Sun Enterprise 450 server configured and tested as specified herein.

Our test results should be reproducible by others using the same test lab configuration, the same Sun server configuration, and the same software configurations documented in this white paper.

Analysis and Test Details

Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.