Mindcraft® tested the performance of
Netegrity SiteMinder 4.51 using our iLOAD MVP™ tool to run the
AuthMark ™ Benchmark Login
and Extranet Scenarios. In this section, we describe these
tools so that you will be able to understand the
performance results discussed in the Result
Analysis section below.
iLOAD MVP is a
general-purpose, script-driven capacity planning,
benchmarking, and regression testing tool. The major
components of iLOAD MVP are:
- A Control Center that manages client systems,
controls test script execution, and reports on
- Multi-threaded client load generators that
execute test scripts to simulate users accessing
- Test script generation programs.
- Test data generation programs.
iLOAD MVP provides
the capabilities needed to test high-performance servers
with a small number of client systems. Its capabilities
- The ability to simulate a large number of
simultaneous user sessions. The number of user
sessions is limited only by the client OS, the
amount of memory and the performance of the
- Support for HTTP 1.0 and 1.1 as well as LDAP V3.
- Support for authentication and authorization.
- Support for SSL.
- Custom test scripts.
The AuthMark Benchmark is designed to test the
performance of products that provide authentication and
authorization services in support of Web servers. Authentication
is the process of verifying who a user is; it
typically occurs when a user logs in. Authorization is
the process of verifying that an authenticated user is
allowed to see or to use a particular resource. In the
case of a Web server such resources include HTML files,
graphic files, and programs that generate Web pages
AuthMark simulates a large number of users accessing Web servers via their browsers. This approach permits
AuthMark to test authentication and authorization
performance independent of the technology used to
provide those services.
AuthMark consists of several test scenarios to
determine various aspects of performance for
authentication and authorization systems under different
circumstances. For the SiteMinder tests we used the
and Extranet Scenarios.
The AuthMark Login Scenario focuses on testing
authentication. We call it the Login Scenario because authentication is done the first time a user accesses a
protected part of a Web site, just like a login. The
HTTP 1.0 and 1.1 protocols define the steps a browser
follows for authentication. Some of the steps are
visible to you and others are not. It is important to
understand what happens during a login in
order to understand what the Login Scenario measurements
The following simplified sequence will walk you
through the login process to show you how it
works using the HTTP 1.0 and 1.1 protocols:
- When you click on a link or enter a URL in your
browser your browser sends the requested URL to
the Web server.
- The Web server determines that you must be
authenticated before it returns the resource at
the requested URL. Typically, the authentication
requirement is specified as part of the Web
server's configuration or via an
authentication/authorization product connected
to the Web server.
- The Web server sends back a "401" HTTP
response to your browser indicating that you are
not authorized to see that requested resource.
- Your browser pops open a window and asks you to
enter your user ID and a password.
- After you enter your user ID and password, your
browser stores them in memory and associates
them with the protected space (called a realm
) containing the URL you requested.
- Your browser then resends a request for the same
URL but this time it includes an HTTP
authorization header containing your user ID and
- This time the Web server checks your user ID and
password to see if they match the authentication
information in the authentication system. If
they do, you are authenticated.
- Now that you have been authenticated, the
authorization system checks whether or not you
are authorized to access Web pages in the realm.
If you are authorized, the Web server sends the
Web page you requested.
Notice that the URL you clicked on or entered is
actually sent twice (in steps 1 and 6). This means that
the authentication system is used twice—first, it
finds out that the requested URL requires the user be
authenticated, then it processes the authorization
header when the request is resent.
Once a user has been authenticated, the Web browser
automatically sends the authorization header whenever
the user requests a URL in the same realm requiring
Login Scenario Configuration
Table 1 shows the
AuthMark Login Scenario configuration parameters we
Table 1: AuthMark Login Scenario Configuration Parameters
Number of users
in the security database
Organizational Units or security groups
Total number of
user sessions per test
The number of user sessions active during a given
test run is determined by the length of the test and the
number of logins. Sessions are not logged out once
created. Instead, each session remains quiescent after
Running the Login Scenario
The basic steps for running the
Generate the data to fill the security
database. iLOAD MVP provides
a tool to generate realistic data for the
LDAP V3 organizationalPerson object class and
Netscape's inetOrgPerson object class. It also
includes tools to load the same data into an
LDAP directory, which was used for this
Load the security database with the user
Generate the test scripts for the Login
Scenario. iLOAD MVP provides
a tool to do this. These scripts drive iLOAD MVP to simulate
user interaction with the Web server(s).
Load Web pages on the Web server(s). There
are 100 Web pages each of which is 14 KB in
size for the Login Scenario.
Load and configure the user management
system or authentication/authorization
Run the benchmark.
The Login Scenario test script selects
users randomly from the user database (see Table 1 for the numbers
we used for this test). The tester is free to select the number of
client test systems and the number of iLOAD MVP client threads to use.
These are called the load generators.
The tester selects the number of load
generators to get the highest performance possible from
the authentication/authorization system being tested. In
order to obtain the peak performance from an
authentication/authorization system, the tester may need
to use multiple Web servers and database servers.
The tester is permitted, but not
required, to do a warm-up run of the test scenario in
order to get the servers to a state that would more
likely represent the state they would be in during
normal operation. For this benchmark, we warmed-up the
servers by running the test script in its entirety.
The Extranet Scenario is intended to simulate an environment where users must
login to a Web site and where all access requests
require authorization. This scenario depicts a more complete and
more realistic usage pattern than the Login Scenario.
The Extranet Scenario test execution starts with the same operation sequence as the
Login Scenario (steps 1 - 6 above) and
continues with the following operations:
- The test client requests a resource.
- The authorization services check the
validity of the user and and that the user is authorized to have
access to the resource.
- If the user is authorized, the resource
- The test client then requests additional
SiteMinder checks the continuing validity of the authenticated user each
time a resource access request is made to ensure
that the user session has not been revoked.
However, the user is not re-authenticated. As a
result, the user does not see a new login request as long
as the resources being accessed are in the Internet domain
in which the user has been authenticated.
The Extranet Scenario operation sequence consists of
one login followed by 10 authorizations yielding a total
of 11 operations per user session. We call these 11
operations an Extranet Sequence. For the Extranet
Scenario, we warmed-up the servers by running the test
script in its entirety.
This section analyzes the Login
and Extranet Scenario performance characteristics of Netegrity's SiteMinder
4.51 including its
performance scalability with different server
The SiteMinder Policy Server is the control
point for all authentication and authorization. Our
tests were structured to push the Policy Server systems
as close as possible to 100% CPU utilization. Table
2 summarizes the Login Scenario performance as a
function of the SiteMinder Policy Server system(s) configuration. The Scaling Factor in
Table 2 shows how much faster a configuration is
compared to a single system having one CPU.
2: SiteMinder Login Performance
Scalability - 1,000,000 Users
1 system, 1
1 system, 2 CPUs
1 system, 3 CPUs
2 systems, 1 CPU
2 systems, 2 CPUs
2 systems, 3 CPUs
The CPU utilizations for the Policy
Server configurations with one system, three CPUs and
with two systems, two and three CPUs show that we did not have enough Web
servers or fast enough ones to drive the Policy Servers
to full CPU utilization. This lack of Web servers showed
up for the test of the Policy Server configuration using
two systems with two CPUs each even though we added a
four-processor Enterprise 4500
Web server with 400 MHz CPUs to the ones we were using
for the other tests. And when we used an additional
Enterprise 450 with four 300 MHz CPUs for the test of
two Policy Servers with three CPUs each, we still had
insufficient Web server performance to drive the Policy
Servers to full CPU utilization. If the lab had enough Web servers
available, we fully expect that SiteMinder would have
achieved more logins per minute than it did.
During each of the tests, the LDAP directory server showed
only 10% to 15% CPU utilization on one CPU while its
other CPUs were idle. This shows that SiteMinder made
very efficient use of the LDAP directory server.
Figure 1 presents SiteMinder's performance from Table
2 by server configuration, with the gold columns showing
the results for a single Policy Server and the blue
column showing the result for dual Policy Servers.
1: SiteMinder Login Scalability for 1,000,000 Users
Table 3 compares the SiteMinder
Extranet Scenario performance to that of the Login
Scenario for the same
hardware configuration - one Policy Server with one
CPU. The results in
Table 3 demonstrate that the SiteMinder Policy Server
performs authorizations several times faster than authentications.
It is not possible to calculate the exact performance
difference because the CPU utilization of the Policy
Server CPU was 50% for the Extranet test and was 100%
for the Login test. This also means that the Extranet
performance would have been significantly higher if
there were enough Web servers available in the lab to
drive the Policy Server's CPU to full utilization.
Table 3: SiteMinder Extranet and Login
- 1 Policy Server with 1 CPU
These test results lead us to conclude
- Netegrity SiteMinder 4.51
outperforms all other products we've tested so far
for the AuthMark Login and Extranet Scenarios.
- SiteMinder 4.51
delivers the highest Login and Extranet performance per
policy/security server CPU of any product we have tested
- SiteMinder delivers outstanding performance
scaling as CPUs and Policy Servers are added to a
Mindcraft used a mix of systems for the Web servers. Table 4 shows the
Enterprise 450 Web server configuration we used. The
configurations we used for Web servers and the
SiteMinder Policy server are shown in Table 5.
Table 6 shows the Enterprise
4500 Web server configuration we used. Finally, Table
7 shows the Enterprise 4500 configuration we used
for the LDAP directory server.
4: Sun Enterprise 450 Web Server Configuration
x 300 MHz UltraSPARC II
Cache: L1: 16 KB I + 16 KB D; L2: 2 MB
4.2 GB SCSI; one for
Solaris and one for the Web data
x 100Base-TX NICs
5: Sun Ultra 80 Web and Policy Servers Configuration
x 450 MHz UltraSPARC II (we used the psradm command to
enable/disable processors in the Policy
Cache: L1: 16 KB I + 16 KB D; L2: 4 MB
|| 2 x
18 GB SCSI
x 100Base-TX NICs
6: Sun Enterprise 4500 Web Server Configurations
x 400 MHz UltraSPARC II (we used the psradm command to
disable 10 of the 14 processors in the system)
Cache: L1: 16 KB I + 16 KB D; L2: 4 MB
x 18 GB SCSI
x Quad 100Base-TX NIC (4 ports, only 2 used)
7: Sun Enterprise 4500 LDAP Directory Server
x 400 MHz UltraSPARC II
Cache: L1: 16 KB I + 16 KB D; L2: 4 MB
x 18 GB SCSI, no RAID
x Quad 100Base-TX NIC (4 ports, only 1 used)
Figure 2 shows how the
servers were configured for all of the Login Scenario tests using
one Policy Server as well as the test using two Policy
Servers with one CPU. Figure
3 gives the server configuration for the Login test using
two Policy Servers with two CPUs. Figure 4
presents the server configuration for the Login Scenario
test of two Policy Servers each
with three CPU and for the one Policy Server with one
CPU Extranet Scenario test. All of the systems were connected using
Netgear 8 port 10/100 Base-TX hubs.
2: Server Configuration for all 1 Policy Server and
2 Policy Server with 1 CPU Login Tests
3: Server Configuration for 2 Policy Servers
with 2 CPUs Login Test
4: Server Configuration for 2 Policy Servers
with 3 CPUs Login Test and
1 Policy Server with 1 CPU Extranet Test
Configuration and Tuning
We used the following server software for these
- Solaris 8 with all current recommended patches on all of the Sun systems
- Netscape Directory Server 4.12 (B00.193.0237)
- iPlanet Web Server, Enterprise Edition 4.1 Service Pack
- Netegrity SiteMinder 4.51, Service Pack 1
All software ran with default settings except for the
- For Netscape Directory Server:
Change in slapd.ldbm.conf:
index uid pres, eq, sub
index ou eq, sub
index o pres, eq
Changes in slapd.conf:
- For SiteMinder Policy Server:
- user cache size = all users
- 100% of all policies cached (there was
- session timeout = 2 hours
- For the SiteMinder Web Agent:
- User session cache disabled
- resource cache size = 1000
Client Test Systems
For all of the tests, we used Dell OptiPlex GX 110 client
test systems configured as shown in Table
8. The client test systems we used for each test
are shown in Figures 2 , 3
8: Client Test Systems Configuration
OptiPlex GX 110, 1 x 667 MHz Pentium III CPU
x 14.5 GB ATA/66
x 100Base-TX (3Com 3C905C-TX)
Windows NT 4.0 Workstation, Service Pack 5
The information in this publication is
subject to change without notice.
MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR
OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR
CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING,
PERFORMANCE, OR USE OF THIS MATERIAL.
This publication does not constitute
an endorsement of the product or products that were
tested. This test is not a determination of product
quality or correctness, nor does it ensure compliance
with any federal, state or local requirements.