Client Test Systems
Mindcraft® tested the performance of Baltimore Technologies
SelectAccess 3.1 using our iLOAD MVP™ tool to run
the AuthMark™ Benchmark
Login and Extranet Scenarios. In this
section, we describe these tools so that you will be able to understand
the performance results discussed in the Result
Analysis section below.
iLOAD MVP is a general-purpose,
script-driven capacity planning, benchmarking, and regression testing
tool. The major components of iLOAD
- A Control Center that manages client systems, controls test
script execution, and reports on test results.
- Multi-threaded client load generators that execute test scripts
to simulate users accessing a server.
- Test script generation programs.
- Test data generation programs.
iLOAD MVP provides the
capabilities needed to test high-performance servers with a small
number of client systems. Its capabilities include:
- The ability to simulate a large number of simultaneous user
sessions. The number of user sessions is limited only by the client
OS, the amount of memory and the performance of the client systems.
- Support for HTTP 1.0 and 1.1 as well as LDAP V3.
- Support for authentication and authorization.
- Support for SSL.
- Custom test scripts.
The AuthMark Benchmark is designed to test the performance of products
that provide authentication and authorization services in support
of Web servers. Authentication is the process of verifying
who a user is; it typically occurs when a user logs in. Authorization
is the process of verifying that an authenticated user is allowed
to see or to use a particular resource. In the case of a Web server
such resources include HTML files, graphic files, and programs that
generate Web pages dynamically.
AuthMark simulates a large number of users accessing Web servers
via their browsers. This approach permits AuthMark to test authentication
and authorization performance independent of the technology used
to provide those services.
AuthMark consists of several test scenarios to determine various
aspects of performance for authentication and authorization systems
under different circumstances. For the SelectAccess tests we used
the AuthMark Login and Extranet
The AuthMark Login Scenario focuses on testing authentication.
We call it the Login Scenario because authentication is done the
first time a user accesses a protected part of a Web site, just
like a login. The HTTP 1.0 and 1.1 protocols define the steps a
browser follows for authentication. Some of the steps are visible
to you and others are not. It is important to understand what happens
during a login in order to understand what the Login Scenario measurements
The following simplified sequence will walk you through the login
process to show you how it works using the HTTP 1.0 and 1.1 protocols:
- When you click on a link or enter a URL in your browser your
browser sends the requested URL to the Web server.
- The Web server determines that you must be authenticated before
it returns the resource at the requested URL. Typically, the authentication
requirement is specified as part of the Web server's configuration
or via an authentication/authorization product connected to the
- The Web server sends back a "401" HTTP response
to your browser indicating that you are not authorized to see
that requested resource.
- Your browser pops open a window and asks you to enter your user
ID and a password.
- After you enter your user ID and password, your browser stores
them in memory and associates them with the protected space
(called a realm ) containing the URL you requested.
- Your browser then resends a request for the same URL but this
time it includes an HTTP authorization header containing your
user ID and password.
- This time the Web server checks your user ID and password to
see if they match the authentication information in the authentication
system. If they do, you are authenticated.
- Now that you have been authenticated, the authorization system
checks whether or not you are authorized to access Web pages in
the realm. If you are authorized, the Web server sends the Web
page you requested.
Notice that the URL you clicked on or entered is actually sent
twice (in steps 1 and 6). This means that the authentication system
is used twice—first, it finds out that the requested URL requires
the user be authenticated, then it processes the authorization header
when the request is resent.
Once a user has been authenticated, the Web browser automatically
sends the authorization header whenever the user requests a URL
in the same realm requiring authentication.
Login Scenario Configuration
Table 1 shows the AuthMark Login Scenario
configuration parameters we used.
AuthMark Login Scenario Configuration Parameters
Number of users in the security database
Number of Organizational Units or security
Total number of user sessions per test
The number of user sessions active during a given test run is determined
by the length of the test and the number of logins. Sessions are
not logged out once created. Instead, each session remains quiescent
Running the Login Scenario
The basic steps for running the Login Scenario are:
Generate the data to fill the security database.
iLOAD MVP provides a tool to generate
realistic data for the LDAP V3 organizationalPerson object class
and Netscape's inetOrgPerson object class. It also includes
tools to load the same data into an LDAP directory, which was
used for this test.
Load the security database with the user data.
Generate the test scripts for the Login Scenario.
iLOAD MVP provides a tool to do
this. These scripts drive iLOAD MVP to simulate
user interaction with the Web server(s).
Load Web pages on the Web server(s). There
are 100 Web pages each of which is 14 KB in size for the Login
Load and configure the user management system
or authentication/authorization system.
Run the benchmark.
The Login Scenario test script selects users randomly
from the user database (see Table 1 for the
numbers we used for this test). The tester is free to select the
number of client test systems and the number of iLOAD MVP client threads to use.
These are called the load generators.
The tester selects the number of load generators to
get the highest performance possible from the authentication/authorization
system being tested. In order to obtain the peak performance from
an authentication/authorization system, the tester may need to use
multiple Web servers and database servers.
The tester is permitted, but not required, to do a
warm-up run of the test scenario in order to get the servers to
a state that would more likely represent the state they would be
in during normal operation. For this benchmark, we warmed-up the
servers by running the test script in its entirety.
The Extranet Scenario is intended to simulate an environment where
users must login to a Web site and where all access requests require
authorization. This scenario depicts a more complete and more realistic
usage pattern than the Login Scenario.
The Extranet Scenario test execution starts with the same operation
sequence as the Login Scenario (steps 1 - 6 above) and continues
with the following operations:
- The test client requests a resource.
- The authorization services check the validity of the user and
and that the user is authorized to have access to the resource.
- If the user is authorized, the resource is returned.
- The test client then requests additional resources.
SelectAccess checks the continuing validity of the authenticated
user each time a resource access request is made to ensure that
the user session has not been revoked. However, the user is
not re-authenticated. As a result, the user does not see a new login
request as long as the resources being accessed are in the Internet
domain in which the user has been authenticated.
The Extranet Scenario operation sequence consists of one login
followed by 10 authorizations yielding a total of 11 operations
per user session. We call these 11 operations an Extranet Sequence.
For the Extranet Scenario, we warmed-up the servers by running the
test script in its entirety.
This section analyzes the Login
and Extranet Scenario performance characteristics
of Baltimore Technologies SelectAccess 3.1 including its performance
scalability with different server configurations.
Table 2 summarizes the Login Scenario
performance as a function of the SelectAccess policy server
system(s) configuration. The Scaling Factor in Table 2 shows how
much faster a configuration is compared to a single system with
one CPU using one directory server, the smallest configuration.
We compute the logins/minute/total CPUs by dividing the logins/minute
by the total number of CPU used as shown in the column # of CPUs
by Server Type in Table 2. See the hardware
configurations for more details on the test environment.
Because the SelectAccess policy server is the control point for
all authentication and authorization, our tests were structured
to push the policy server systems as close as possible to 100% CPU
utilization. The CPU Utilization column in Table 2 shows
the average CPU utilizations by type of server. These show that
the LDAP directory server was barely used, so it was not a performance
The networks between the load generators and Web servers also were
not saturatedthe highest performance test had 3,423 logins
per second, which generated total network traffic of 777.6 Mbits/second
across four switches. The most bandwidth used in any switch was
222.2 Mbits/second, well below the switch maximum of 614.4 Mbits/second.
In addition, the most bandwidth in any switch port was 37 Mbits/second,
which is significantly below the bandwidth limit for a 100Base-TX
network segment. The network link between a policy server
and the Web servers was used about 30% at the highest login rate.
Looking at the policy server and the Web server CPU utilizations
together, we see that policy server CPU utilization dropped as policy
server CPUs were increased from three to four while the number of
Web servers and Web server CPUs stayed constant. The resulting Web
server CPU utilizations increased to 78% but we could not raise
it above that level. This indicates that if we had used more Web
server CPUs then we could have been able to increase the
policy server CPU utilization and might have achieved a higher login
Table 2: SelectAccess Login Performance
Scalability - 1,000,000 Users
|| 1 system, 1 CPU
1 system, 2 CPUs
|| 1 system, 3 CPUs
||1 system, 4 CPUs
2 systems, 2 CPUs
|| 2 systems, 4 CPUs
Figure 1 shows SelectAccess's Login
performance from Table 1 by server configuration.
SelectAccess Login Scalability for 1,000,000 Users
Table 3 shows the SelectAccess Extranet Scenario
performance for three configurations. The CPU utilizations shown
are the average for each type of server.
For the configuration with SelectAccess two policy servers
each having four CPUs, the policy servers CPU utilizations show
that the Extranet performance was not limited by available CPU cycles.
The network bandwidth for the three most heavily used switches between
the load generators and the Web servers used 61.6% of the available
switching bandwidth and about 63% of the bandwidth of each 100Base-TX
network segment. For the two policy server tests, one
of the policy servers handled six Web servers and the
other handled five. For the test using four CPUs per policy
server, the that served the six Web servers used about
57% of the available bandwidth on the network segment between it
and the Web servers.
Table 3: SelectAccess Extranet Performance
- 1,000,000 User Directory
These benchmark results lead us to conclude that:
- Baltimore Technologies SelectAccess 3.1 delivers
the highest overall AuthMark Login and Extranet performance we've
measured to date.
- SelectAccess 3.1 achieves
the highest Login performance per policy/security server CPU and
the best Login and Extranet performance per CPU, for all CPUs
used, that we have measured to date.
- SelectAccess delivers outstanding performance scaling as CPUs
and policy servers are added to a configuration.
Hardware Configurations Tested
Table 4 shows the configuration of the
Sun Enterprise 420R systems we used as Web, policy, and LDAP directory
Sun Enterprise 420R Server Configuration
||4 x 450 MHz UltraSPARC
Cache: L1: 16 KB I + 16 KB D; L2: 4 MB
||4 GB ECC
|| 2 x 18 GB SCSI;
one for Solaris and one for data
||2 x 100Base-TX
Figure 2 shows how the systems were configured
for the Login Scenario test using one Policy Server with one CPU.
Figure 3 gives the server configuration for
the Login and Extranet Scenario tests using one Policy Server with
two CPUs. Figure 4 presents the server configuration
for the Login Scenario tests of one Policy Servers each with three
and four CPUs. Figure 5 shows the system configuration
for all of the Login and Extranet Scenario tests that used two policy
servers. All of the systems were connected using 8-port Hewlett-Packard
ProCurve 408 10/100 Base-TX switches.
Server Configuration for 1 Policy Server with 1 CPU Login Test
Server Configuration for 1 Policy Server with 2 CPUs Login and Extranet
Server Configuration for 1 Policy Server with 3 and 4 CPUs Login
Server Configuration for all 2-Policy Server Login and Extranet
Server Software Configuration and Tuning
We used the following server software for these benchmark tests:
- Solaris 8 with all current recommended patches on all of the
- iPlanet Directory Server 4.13
- iPlanet Web Server, Enterprise Edition 4.1 Service Pack 3
- Baltimore SelectAccess 3.1
All software ran with default settings except for the following:
- For Netscape Directory Server:
Change in slapd.ldbm.conf:
index uid pres, eq, sub
index ou eq, sub
index o pres, eq
Changes in slapd.conf:
- Solaris (set using ndd -set /dev/tcp on all servers):
- Baltimore SelectAccess 3.1
Cached user names, passwords, and policies (only one policy
was used). Cache grows dynamically as needed.
Client Load Generator Systems
For all of the tests, we used Dell OptiPlex GX 150 client load
generator systems configured as shown in Table
Client Load Generator Systems Configuration
GX 150, 1 x 1 GHz Pentium III CPU
||256 MB SDRAM
||1 x 10 GB ATA/100
||1 x 100Base-TX
(integrated 3Com 3C920)
2000 Professional, Service Pack 1
The information in this publication is subject to
change without notice.
MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR OMISSIONS CONTAINED
HEREIN, NOR FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM
THE FURNISHING, PERFORMANCE, OR USE OF THIS MATERIAL.
This publication does not constitute an endorsement
of the product or products that were tested. This test is not a
determination of product quality or correctness, nor does it ensure
compliance with any federal, state or local requirements.