Netegrity Inc. sponsored
the testing in this report. Mindcraft, Inc. conducted
the performance tests described in this report at
Sun’s test lab in Menlo Park,
We thank Sun for providing the systems
used for the tests and the support staff who helped
configure the servers.
Netegrity SiteMinder 4.51 delivers
112,202 logins per minute and 20,179 Extranet
Sequences (221,969 operations) per minute for 1,000,000
users, achieving the highest overall and per CPU performance
that we've seen
Netegrity SiteMinder 4.51 delivers dramatic
performance improvement over earlier
versions and sets a new performance standard for
authentication/authorization products. Throughout our tests, SiteMinder
outperformed all other similar products we've tested
both in overall performance and in performance per policy/security server
tested Netegrity SiteMinder 4.51 running on
a mix of Sun Enterprise servers. For these tests, we used Mindcraft’s
iLOAD MVP™ test
tool running the AuthMark™
Login and Extranet
The Login Scenario represents the type of load
commonly seen at portal sites. It simulates users accessing protected resources via Web servers.
Scenario assumes that 10% of a portal's user population
logs in concurrently to use portal resources. All tests were
done using a 1,000,000-user directory with 100,000 active
The Login Scenario measures the combination of one user
authentication and one authorization for access to a
resource (called a Login). The Result Analysis section in the
second part of this white
paper explains the benchmark results.
The SiteMinder Policy Server is the control
point for all authentication and authorization. Our
tests were structured to push the Policy Server systems
as closely as possible to 100% CPU utilization. Table 1
summarizes the Login Scenario performance as a
function of the SiteMinder Policy Server system(s) configuration. The Scaling Factor in
Table 1 shows how much faster a configuration is
compared to a single system with one CPU using one
directory server, the smallest configuration.
Table 1: SiteMinder Login Performance
Scalability - 1,000,000 Users
1 system, 1
1 system, 2 CPUs
1 system, 3 CPUs
2 systems, 1 CPU
2 systems, 2 CPUs
2 systems, 3 CPUs
The CPU utilizations for the Policy
Server configurations with one system, three CPUs and
with two systems, two and three CPUs show that we did not have enough
load generators and Web
servers, or fast enough ones, to drive the Policy Servers
to full CPU utilization. If the lab had enough load
generators (which ran at 70% CPU utilization for the
highest performance tests) and Web servers
available, we fully expect that SiteMinder would have
achieved more logins per minute than it did.
Figure 1 shows
SiteMinder's Login performance from Table
1 by server configuration.
1: SiteMinder Login Scalability for 1,000,000 Users
The Extranet Scenario
measures the combination of one user authentication and
10 authorizations for access to resources (these 11 operations
constitute one Extranet sequence). The Extranet Scenario, because it uses
a more realistic mix of operations than the Login
Scenario, provides a better basis for capacity planning
Table 2 compares the SiteMinder
Extranet Scenario performance to that of the Login
Scenario for the same
hardware configuration - one Policy Server with one
CPU. The results in
Table 2 demonstrate that the SiteMinder Policy Server
performs authorizations several times faster than authentications.
It is not possible to calculate the exact performance
difference because the CPU utilization of the Policy
Server CPU was 50% for the Extranet test and was 100%
for the Login test. The CPUs in the Web servers averaged
95% utilization while the load generator CPUs averaged
70% utilization. This means that the Extranet
performance would have been significantly higher, if
there were enough load generators and Web servers available in the lab to
drive the Policy Server's CPU to full utilization.
Table 2: SiteMinder Extranet and Login
- 1 Policy Server with 1 CPU
The benchmark results lead us to
- Netegrity SiteMinder 4.51
outperforms all other products we've tested so far
for the AuthMark Login and Extranet Scenarios.
- SiteMinder 4.51
delivers the highest Login and Extranet performance per
policy/security server CPU of any product we have tested
- SiteMinder delivers outstanding performance
scaling as CPUs and Policy Servers are added to a
Mindcraft certifies that the results reported
accurately represent the performance of Netegrity
SiteMinder 4.51 running on Sun Enterprise servers configured as
specified herein and as measured by AuthMark benchmark.
Our test results should be reproducible by others
using the same test lab configuration, the same Sun
server configurations, and the same software
configurations documented in this white paper.
and Test Details
- Added load generator system CPU utilization to the Login and
Extranet Scenario commentary.
- Added lack of additional load generator systems as another
reason that the Policy Servers CPUs were not fully utilized.
The information in this publication is
subject to change without notice.
MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR
OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR
CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING,
PERFORMANCE, OR USE OF THIS MATERIAL.
This publication does not constitute
an endorsement of the product or products that were
tested. This test is not a determination of product
quality or correctness, nor does it ensure compliance
with any federal, state or local requirements.