WebStone 2.x Benchmark Description

The WebStone benchmark tests were originally developed by Silicon Graphics to measure the performance of Web server software and hardware products. WebStone 2.0.1 is a more portable version of the original WebStone benchmark which added support to use Windows NT systems as client test systems.

Mindcraft, Inc. has acquired the rights to WebStone from Silicon Graphics. WebStone 2.5 is Mindcraft's enhancement to WebStone 2.0.1 to improve reliability and portability as well as to make tests more reproducible. WebStone 2.5 also offers new workloads for CGI and API tests (see below for more details). WebStone 2.5 provides performance-identical test results with WebStone 2.0.1 when using the same workloads.

How WebStone 2.x Works

WebStone creates a load on a Web server by simulating the activity of multiple clients, which are called Web clients and which can be thought of as users, Web browsers, or other software that retrieves files from a Web server. This simulation is carried out using multiple Web clients running on one or more computers. It is possible to run in excess of 100 simulated Web clients on a single computer.

In order to create large loads on a Web server, WebStone is able to distribute Web clients among client computers. The Webmaster is the program that controls all of the testing done by WebStone. It can be run on one of the client computers or on a separate computer. The Webmaster distributes the Web client software and test configuration files to the client computers. The Webmaster then starts a benchmark run and waits for the Web clients to report back the performance they measured. The Webmaster combines the performance results from all the Web clients into a single summary report.

The figure below shows the relationship between the various parts of a WebStone test lab:

What WebStone 2.x Tests

The performance measured by WebStone is dependent on the set of files used by the Web clients. The file set used by WebStone is based on a model of the Silicon Graphics Web site in 1995. It is possible to change the file set used by WebStone to one that better simulates a Web site of interest to you. The standard WebStone 2.0.1 run rules and the WebStone 2.5 run rules require the use of the standard file set to allow easy comparison between performance results for different systems. The standard WebStone file set is small so it typically will run out of a Web server's cache. This means that the WebStone benchmark tests Web server and operating system software as well as CPU and network speed, not the speed of disk access (except as related to process creation for CGI tests).

WebStone tests Web servers using three different types of access methods:

• HTML: These tests cause a Web server itself to return to a Web client a specified file, which is usually written in Hyper Text Markup Language. This access method is used to provide most Web pages.
  
  
• CGI: These tests cause the Web server to run another program, using the Common Gateway Interface protocol to pass to that program the parameters specified in the request. This method is commonly used to process forms and to provide responses to user input on Web pages.
  
  In WebStone 2.0.1, the CGI workload computes the data it returns by making a call to a random number function for each byte returned. This workload is compute intensive and may be more sensitive to CPU speed than to the efficiency of the CGI interface in the Web server software. In WebStone 2.5, there is an optional workload, the CGI2.5 workload, which gets a file name from the environment and returns the data in the file.
  
  
• API: These tests cause the Web server to pass the access request to its Application Programming Interface which will cause a program linked into the Web server to be called to respond to the request for a Web page. This method is typically a more efficient way to accomplish programmatic response to a request than the CGI method.
  
  In WebStone 2.0.1, the API workload computes the data it returns by making a call to a random number function for each byte returned. This workload is compute intensive and may be more sensitive to CPU speed than to the efficiency of the API interfaces in the Web server software. In WebStone 2.5, there is an optional workload, the API2.5 workload, which gets a file name from the requested URL and returns the data in the file. WebStone 2.0.1 comes with support for Netscape's API (NSAPI). Mindcraft has ported the API tests for Microsoft's Internet Information Server API (ISAPI) and has made those tests available at our Web site. WebStone 2.5 comes with support for both NSAPI and Microsoft's Internet Server API (ISAPI).

Warning

Do not compare WebStone 2.x results with WebStone 1.0 results. WebStone 2.x introduces significant changes in testing methodology and reporting.