Netscape Application Server,
Version 4.0
Performance Audit

By Bruce Weiner
(PDF version, 69 KB)

February 11, 2000

Contents

Executive Summary
Product Configurations
Test Application
Conclusion
Mindcraft Certification

Disclosure

The Sun|Netscape Alliance performed the testing in this report. Mindcraft, Inc. was present during the tests described in this report and audited the test lab, test systems, the software tested, and the results.

The Netscape Application Server, Version 4.0, will evolve into and will be superceded by the iPlanet Application Server, Version 6.0.

Executive Summary

The performance tests reviewed in this audit report demonstrate that for a consumer shopping application the Netscape Application Server 4.0, Service Pack 1 (NAS) is 1.4 times faster than another popular Application Server, which we will call Green because of license restrictions. 

For the tests reported here, we call a transaction an HTTP GET or POST request and the associated response from the Web server. Figure 1 shows the transaction rates in Transactions per Minute (TPM) for NAS and Green running on a four-processor application server. The peak performance values are indicated by the markers. Figure 2 shows the corresponding average latency with markers for the average latency at peak performance and the maximum average latency. The reason that the Green curves go out through 80 client test threads is that we wanted to show that its peak performance had been reached. This was unnecessary for NAS since its peak occurred at 55 client test threads.

The "Number of Clients" on the X-axis in Figures 1 and 2 refers to how many simulated clients are running on the client test system. You should be careful not to correlate the number of clients to the number of real users either NAS or Green can support. The reason is that the client simulation did not include any time for a person to read a Web page, think about its content, and take the next action. In other words, each client made server requests as fast as possible. The client simulation was configured this way to maximize the server load while minimizing the number of client test systems needed to drive each application server to its peak performance.

Figure 1: NAS and Green Transaction Rates Under Load
(larger  numbers are better)

Figure 2: NAS and Green Average Latencies Under Load
(smaller numbers are better)

Product Configurations

In order to simulate a more realistic environment, Servlet HTTP Session data was made highly available in case of a Java virtual machine (JVM) failure. For example, if the JVM on which the HTTP session was initially created experienced a failure between requests within the session, then the next request would be directed to an available JVM and the HTTP session would be available in a manner that is transparent to the application.

Both NAS and Green were configured as similarly as possible for these tests:

• Both use the same Java servlet specification level (2.1) and a JDK 1.1.x Java environment.
• The Java servlet application source code was almost the same for both. The only differences were to accommodate the JDBC differences in the products (JDBC 2.0 in NAS and JDBC 1.0 in Green).
• NAS and Green were configured to use JDBC connection pooling.
• Both employ native Oracle 8.0.5 drivers under their respective JDBC drivers.
• NAS and Green were performance-tuned using each vendor's documented recommendations.
• Both were deployed on the same three-tier server environment.
• No special Solaris 2.6 operating system tuning was performed for either product.

To optimize performance on the four-processor application server used for these tests, each product was configured differently in the following areas, based on its inherent characteristics:

• NAS:
  • One NAS instance was deployed with two JVMs. NAS supports highly available sessions without requiring multiple instances of the application server. Hence, multi instance clustering is not required in NAS.
  • NAS has a servlet page caching capability that is very useful for speeding up content delivery that is fairly static across user sessions. For example, in a shopping application servlet page caching is very useful for catalog content. In a portal site, channel type information (news, weather, etc.) are also good candidates for servlet page caching.
  • In this test, the servlet page caching feature was applied in NAS to those servlets that deliver relatively static content. All servlets generating user or session-dependent information (such as shopping carts, ordering information, etc.) did not use servlet page caching.
    
• Green:
  • Four instances were deployed with one Green instance assigned to each CPU and each instance having its own JVM. Clustering was fully configured, which requires an IP address for each instance in cluster - even when running on a single machine.
  • Green does not have a servlet page caching feature so it could not be configured like NAS in this regard.

Test Application

The performance of any Application Server depends on the performance of the application used for testing as much as it does on the performance of the System Under Test (SUT). The application used for the tests reported here was a catalog-type store e-commerce simulation.

The application is implemented using Java servlets and JDBC-based access to a backend database for both NAS and Green. It simulates typical shopping experiences for several types of customers. Key aspects of the application are:

• Catalog and user data are stored in a relational DBMS on its own server.
• Form-based user authentication occurs against a database table.
• The core of the application is implemented using servlets to handle a simulated customer logging in, browsing for products, searching for products, displaying product information, managing a shopping cart, and ordering products.
• The simulated interactions are not transaction-oriented.
• Each simulated customer uses a different user ID and repeats the following shopping experience:
  • Login (authenticates against database and creates session in memory)
  • Displays products that are specially priced (read-only access to the database)
  • Browses for products based on various categories (read-only access to the database)
  • Iterates through following sequence four times, which uses the in-memory shopping cart:
    • Searches for products
    • Adds items to the shopping cart
  • Places an order (writes to the database)

Test Environment

The test environment consisted of a load generating client system networked through a load balancer to a three-level hierarchy of servers. Figure 3 shows how the systems were connected.

Figure 3: NAS and Green Test Bed

The systems used were:

• Load generator (client simulator): Sun Ultra 10
• Load balancer: 450Mhz Pentium III-based systems, 256 MB RAM, Resonate load balancing software
• Web Servers: Four 450Mhz Pentium III-based systems, 256 MB RAM
• Application Server: One Sun Enterprise 450 with 4 x 400MHz CPUs, 4 GB RAM
• Database Server: One Sun Enterprise 4500 with 12 x 336 MHz CPUs, 2 GB RAM, running Oracle 8.0.5.

All systems used 100Base-TX network interface cards and were connected to each other via a switch.

We validated that the load generator, load balancer, Web servers, database server and the network were not performance-limiting factors.

Conclusion

The Netscape Application Server 4.0 SP1 provides a higher performance, more responsive platform for implementing Java servlet-based applications than Green. The NAS feature set lets you implement high-availability, e-commerce applications without sacrificing performance.

Mindcraft Certification

Mindcraft certifies that the benchmark results reported in this white paper accurately represent the performance of the Netscape Application Server 4.0, Service Pack 1 and the Green Application Server configured and tested as specified herein.

NOTICE:

Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.