Mindcraft Certified Performance

SiteMinder Version 3.51
AuthMark Login Performance

By Bruce Weiner

May 11,1999
Updated May 13, 1999


Executive Summary
Performance Analysis
Products Tested
Test Lab
Mindcraft Certification
iLOAD MVP Overview


Mindcraft, Inc. conducted the performance tests described in this report between April 14 and April 22, 1999. Netegrity, Inc. sponsored the testing in this report.


We thank Sun Microsystems for allowing ust to use their Enterprise Technology Center lab in Palo Alto for our tests.

Executive Summary

SiteMinder Version 3.51 Provides Fast, Scalable Performance for Web Site Logins

Mindcrafttested the performance of Netegrity's  SiteMinder Version 3.51 at Sun's Enterprise Technology Center lab running on Sun Enterprise 450 servers. For these tests we used Mindcraft's iLOAD MVP™  test tool running the AuthMark™ Benchmark Login Scenario to simulate 100,000 users logging into Web servers that used SiteMinder for authentications and authorizations. All tests were done using Netscape Directory Server 4.0 with 1,000,000 users in the directory. The details of the test are given in the Benchmark section. The detailed benchmark results are covered in the Performance Results section.

Figure 1 summarizes the peak login performance measured in logins per minute for a single SiteMinder server with one and two CPUs and for two SiteMinder servers with dual CPUs in a load-balancing configuration. Figure 2 shows the corresponding average time needed per login. The tests were done without caching in the SiteMinder Web agent in order to demonstrate a worst-case scenario of 100,000 different users accessing one Web page each. The AuthMark Login Scenario requires one user authentication and one access authorization to be processed through the Web agent to the SiteMinder server and in turn through to the directory server and the information to be returned via the same path


Figure 1 : SiteMinder Server AuthMark Login Scenario Peak Performance
(larger numbers are better)

 Logins per Minute


Figure 2: SiteMinder Server AuthMark Login Scenario Average Time per Login
(smaller numbers are better)

Average Time Per Login

Our tests demonstrate that SiteMinder is a high-performance, scalable user management system capable of satisfying the performance needs of the most demanding Web sites.

Performance Analysis

We suggest that you review the details of the AuthMark Login Scenario Benchmark specified below in order to have the context to understand the discussion in this section.

Login Performance Analysis

The performance measurements shown in Figure 1 are for the warmed-up Login Scenario tests. These tests show SiteMinder's performance on three different server configurations: a single server with one CPU, a single server with two CPUs, and two servers with two CPUs. Table 1 show the performance scaling we measured for SiteMinder.

Table 1 : SiteMinder Performance Scaling

Server Configuration Logins/Minute Performance Scaling
1 Server, 1 CPU 6,149 N/A
1 Server, 2 CPUs 10,444 1.7 times 1 CPU
2 Servers, 2 CPUs 21,687 2.1 times 1 server

The CPU utilization of the various servers will give you insight into how to configure an environment to get the maximum performance from SiteMinder. Table 2  shows the CPU utilization on each of the servers during the one server, 1 CPU test. It shows that the SiteMinder policy server is saturated while the other servers are lightly loaded. So, if you want to use a single SiteMinder policy server, use one with the fastest CPU available to you so that it will be able to handle the load you need.

Table 2: Server CPU Utilization for a Single CPU SiteMinder Policy Server

Server CPU Utilization
SiteMinder Policy Server 100%
Web Server #1 11% on both CPUs
Web Server #2 11% on both CPUs
Directory Server 15% on all CPUs

Table 3 shows the CPU utilization on each of the servers during the one server, two-CPU SiteMinder Policy server test. It shows that the SiteMinder policy server is fully utilized as performance scales up 70% from a single CPU server. This level of scalability means that you can economically justify adding a second CPU to a SiteMinder policy server.

Table 3 : Server CPU Utilization for a Two-CPU SiteMinder Policy Server

Server CPU Utilization
SiteMinder Policy Server 100% on both CPUs
Web Server #1 23% on both CPUs
Web Server #2 23% on both CPUs
Directory Server 25% on all CPUs

Table 4 shows the CPU utilization on each of the servers during the two server, two-CPU SiteMinder Policy server test. The SiteMinder policy server CPU utilization shows that SiteMinder is load balancing across both policy servers, although it is not sharing the load evenly. However, the 110% performance increase over a single dual-CPU policy server shows that SiteMinder is able to make better use of the more heavily loaded system when it is load balancing with another server. Thus, if you need the highest possible authentication performance at your Web site, use multiple, load-balanced SiteMinder policy servers.

Table 4 : Server CPU Utilization for Two dual-CPU SiteMinder Policy Servers

Server CPU Utilization
SiteMinder Policy Server #1 100% on both CPUs
SiteMinder Policy Server #2 60% on both CPUs
Web Server #1 50% on both CPUs
Web Server #2 50% on both CPUs
Directory Server 50% on all CPUs

Login Performance Conclusions

SiteMinder's performance for logins (authentications) scales extremely well as you increase the speed of the server CPU, add a CPU to the server, and add an additional server in a load-balancing configuration. The SiteMinder policy server configuration options let you balance the authentication server cost with the performance you need. 


The AuthMark Benchmark is designed to test the performance of products that provide authentication and authorization services in support of Web servers. Authentication is the process of verifying who a users is. Authorization is the process of verifying that an authenticated user is allowed to see or to use a particular resource; in the case of a Web server such resources include HTML files, graphic files, and programs that generate Web pages dynamically.

AuthMark simulates a large number of users accessing a Web server via their browsers. This approach permits AuthMark to test authentication and authorization performance independent of the technology used to provide those services.

AuthMark Login Scenario

The AuthMark Login Scenario focuses on testing authentication. We call it the Login Scenario because authentication is done the first time a user accesses a protected part of a Web site, just like a login. The HTTP 1.0 and 1.1 protocols define the steps a browser follows for authentication. Some of the steps are visible level to you and others are not. It is important to understand what happens during an authorization in order to understand what the Login Scenario measurements mean.

Authorization Process

The following simplified sequence will walk you through the authorization process and show you how what you see ties in with the authorization process:

  1. When you click on a link or enter a URL in your browser your browser sends the requested URL to the Web server.
  2. The Web server determines that you must be authenticated before it returns the resource at the requested URL. Typically, the authentication requirement is specified as part of the Web server's configuration or via an authentication/authorization product embedded in the Web server.
  3. The Web server sends back a "401" HTTP response to your browser indicating that you are not authorized to see that requested resource (generally a Web page).
  4. Your browser pops open a window and asks you to enter your user ID and a password.
  5. After you enter your user ID and password, your browser stores them in memory and associates them with the protected space (called a realm ) containing the URL you requested.
  6. Your browser then resends a request for the same URL but this time it includes an HTTP authorization header containing your user ID and password.
  7. This time the Web server checks your user ID and password to see if they match the authentication information in the authentication system. If they do, you are authenticated.
  8. Now that you have been authenticated, the authorization system checks whether or not you are authorized to access Web pages in the realm. If you are authorized, the Web server sends the Web page you requested.

Notice that the URL you clicked on or entered is actually sent twice (in steps 1 and 6). This means that the authentication system is used twice—first, it finds out that the requested URL requires the user be authenticated, then it processes the authorization header when the request is resent.

After the user is authenticated, the request is checked to see if the user is authorized to use the requested resource. The authorization check does not require any contact with the Web browser.

Once a user has been authenticated, the Web browser automatically sends the authorization header whenever the user requests a URL in a realm requiring authentication. So, the user will not see a window open up asking him or her to login again to a realm that has already been visited.

Login Scenario Parameters

The Login Scenario is based on the iLOAD MVP configuration parameter values in Table 5.

Table 5: AuthMark Login Scenario Configuration Parameters



Number of users in security directory/data base 1,000,000
Number of organizationalUnits or security groups 10
Total number of user sessions 100,000
Number of HTTP GETs per user session 1
(authentication actually causes 2 GETs to be issued)

Running the Login Scenario

The basic steps for running the Login Scenario are:

  1. Generate the data to fill the security directory/database. iLOAD MVP provides a tool to generate realistic data for the LDAP V3 orgalizationalPerson schema and Netscape's inetOrgPerson schema.
  2. Load the security directory/database with the user data.
  3. Generate the test scripts for the Login Scenario. iLOAD MVP provides a tool to do this. These scripts drive iLOAD MVP to simulate user interaction with the Web server(s).
  4. Load Web pages on the Web server(s). There are 100 Web pages each of which is 14 KB in size for the Login Scenario.
  5. Load and configure the user management system or authentication/authorization system.
  6. Run the benchmark.

The Login Scenario test script uses 100,000 users randomly selected from the directory/database of 1,000,000 users. The tester is free to select the number of client test systems and the number of iLOAD MVP client threads to use. These are called the load generators .

The tester selects the number of load generators to get the highest performance possible from the authentication/authorization system being tested. In order to obtain the peak performance from an authentication/authorization system, the tester may need to use multiple Web servers and directory/database servers.

The tester is permitted, but not required, to do a warm-up run of the test scenario in order to get the servers to a state that would more likely represent the state they would be in during normal operation. The test report must state whether the servers were warmed-up or not.

Products Tested

Server Configurations

Mindcraft used Sun Enterprise 450 servers for the Web servers, SiteMinder policy servers and the LDAP directory servers. Table 6 shows the server configurations we used.

Table 6: Sun Enterprise 450 Configurations



CPU 4 x 400 MHz UltraSPARC II (we used the psradm command to enable/disable processors)
Cache: L1: 16 KB I + 16 KB D; L2: 4 MB
Disk Web servers : 3 x 9 GB

SiteMinder policy servers : 3 x 9 GB

Directory server: One StorEdge 3000 configured as RAID 0 with
10 x 9 GB disks, 64 KB stripe
Networks Web and SiteMinder policy servers: 2 x 100Base-TX Sun NICs

Directory server: 1 x 100Base-TX integrated NIC

Hardware Configuration

  • Used the Solaris psradm command to disable/enable processors to test uniprocessor and dual-processor configurations

Software Configuration

Solaris 7 Configuration

  • Default configuration for the hardware configurations used except for the following:

    • On the SiteMinder policy servers and the directory server:
      Used ndd to set tcp_time_wait_interval to 60 seconds
    • On the Web servers:
      Used ndd to set tcp_time_wait_interval to 60 seconds

SiteMinder Configuration

  • Used SiteMinder Version 3.51
  • The Web agent configuration file we used is as follows:
    # WebAgent.conf - configuration file for SiteMinder Web Agent
    # Web Agent Version=351
    # Web Agent InstallPath=
       /opt # Do not remove or edit the preceding lines,
    or   you may not # be able to perform
  • Policy server configuration:
    • Policies were bound to each organizationalUnit.

    • Policy store cache was on. Policy information that is normally stored in LDAP or SQL database can be cached in RAM. Any admininistrative changes to the policy information automatically update policy store caches on all policy servers.

    • Authorization cache was on. This dynamic cache tracks the relationship between resources and policies.

    • Verbose logging was off.

    • Error logging was on.

Netscape Enterprise Server (Web Server) Configuration

  • Used Enterprise Server 3.61
  • Default configuration used

Netscape Directory Server Configuration

  • Used Directory Server 4.0
  • Used inetOrgPerson schema with all attribute indexes off except for the following:
    • dn: equality and wildcard
    • cn: equality (this attribute was used as a unique ID)

The Test Lab Configuration

Mindcraft ran these tests using four identically configured Dell Optiplex GX1 client test systems (six systems were available to be used). Table 7 shows the configuration of the client systems.

Table 7: Configuration of the Client Test Systems



CPU 1 x 450 MHz Pentium II, 100 MHz front side bus; 512 KB L2 cache
RAM 128 MB 60 ns SDRAM
Disk 6.4 GB Ultra DMA/ATA 33
Network Integrated 10/100Base-TX 3Com Parallel Tasking II Fast EtherLink XL II
Operating System Windows NT Server 4 with Service Pack 4

Changed registry setting: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\
Tcpip\Parameters\MaxUserPort to 60,000

Switches were used for all network connections. Figure 3 shows a diagram of the complete test lab. We did not use all of the systems in the diagram.

    Figure 3: Test Lab Configuration
    (larger diagram in 93 KB pdf file)


    Test Lab Diagram


    Mindcraft Certification

    Mindcraft certifies that the results reported accurately represent the performance of SiteMinder 3.51 running on Sun Enterprise 450 servers configured as specified herein and as measured by AuthMark benchmark using its Login Scenario.

    Our test results should be reproducible by others using the same test lab configuration, the same Sun computers, and the software configurations documented in this report.

    iLOAD MVP Overview

    iLOAD MVP is a general-purpose, script-driven capacity planning, benchmarking, and regression testing tool. The major components of iLOAD MVP are:

    • A Control Center that manages client systems, controls test script execution, and reports on test results.
    • Multi-threaded client load generators that execute test scripts to simulate users accessing a server.
    • Test script generation programs.
    • Test data generation programs.

    iLOAD MVP provides the capabilities needed to test high-performance servers with a small number of client systems. Its capabilities include:

    • The ability to simulate a large number of simultaneous user sessions; the number of user sessions is limited only by the client OS, the amount of memory and the performance of the client systems.
    • Support for HTTP 1.0 and 1.1 as well as LDAP V3.
    • Support for authentication and authorization.
    • Support for SSL.
    • Custom test script generation.

    Specifications of iLOAD MVP subject to change without notice.

    Change Information

    1. May 13, 1999: Added Figure 2, Average Time Per Login. Renumbered the test lab configuration diagram to be Figure 3.


    The information in this publication is subject to change without notice.


    This publication does not constitute an endorsement of the product or products that were tested. This test is not a determination of product quality or correctness, nor does it ensure compliance with any federal, state or local requirements.

    Mindcraft is a registered trademark of Mindcraft, Inc. iLOAD MVP and AuthMark are trademarks of Mindcraft, Inc.

    Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.



    Copyright 1997-99. Mindcraft, Inc. All rights reserved.
    Mindcraft is a registered trademark of Mindcraft, Inc.
    Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective owners.
    For more information, contact usat: info@mindcraft.com
    Phone: +1 (408) 395-2404
    Fax: +1 (408) 395-6324