Netegrity Inc. sponsored
the testing in this report. Mindcraft, Inc. conducted
the performance tests described in this report at
Sun’s test lab in Menlo Park,
We thank Sun for providing the systems
used for the tests and the support staff who helped
configure the servers.
Netegrity SiteMinder 4.51 delivers
112,202 logins per minute and 20,179 Extranet
Sequences (221,969 operations) per minute for 1,000,000
users, achieving the highest overall and per CPU performance
that we've seen
Netegrity SiteMinder 4.51 delivers dramatic
performance improvement over earlier
versions and sets a new performance standard for
authentication/authorization products. Throughout our tests, SiteMinder
outperformed all other similar products we've tested
both in overall performance and in performance per policy/security server
tested Netegrity SiteMinder 4.51 running on
a mix of Sun Enterprise servers. For these tests, we used Mindcraft’s
iLOAD MVP™ test
tool running the AuthMark™
Login and Extranet
The Login Scenario represents the type of load
commonly seen at portal sites. It simulates users accessing protected resources via Web servers.
Scenario assumes that 10% of a portal's user population
logs in concurrently to use portal resources. All tests were
done using a 1,000,000-user directory with 100,000 active
The Login Scenario measures the combination of one user
authentication and one authorization for access to a
resource (called a Login). The Result Analysis section in the
second part of this white
paper explains the benchmark results.
The SiteMinder Policy Server is the control
point for all authentication and authorization. Our
tests were structured to push the Policy Server systems
as closely as possible to 100% CPU utilization. Table 1
summarizes the Login Scenario performance as a
function of the SiteMinder Policy Server system(s) configuration. The Scaling Factor in
Table 1 shows how much faster a configuration is
compared to a single system with one CPU using one
directory server, the smallest configuration.
Table 1: SiteMinder Login Performance
Scalability - 1,000,000 Users
1 system, 1
1 system, 2 CPUs
1 system, 3 CPUs
2 systems, 1 CPU
2 systems, 2 CPUs
2 systems, 3 CPUs
The CPU utilizations for the 1 system, 3 CPUs and the 2 systems,
2 and 3 CPUs configurations show that we did not have enough Web
servers or fast enough ones to drive the Policy Servers to full CPU
utilization. If the lab had enough Web servers available, we fully
expect that SiteMinder would have achieved more logins per minute
than it did.
Figure 1 shows
SiteMinder's Login performance from Table
1 by server configuration.
1: SiteMinder Login Scalability for 1,000,000 Users
The Extranet Scenario
measures the combination of one user authentication and
10 authorizations for access to resources (these 11 operations
constitute one Extranet sequence). The Extranet Scenario, because it uses
a more realistic mix of operations than the Login
Scenario, provides a better basis for capacity planning
2 compares the SiteMinder Extranet Scenario performance to that
of the Login Scenario for the same hardware configuration - one
Policy Server with one CPU. The results in Table 2 demonstrate that
the SiteMinder Policy Server performs authorizations several times
faster than authentications. It is not possible to calculate the
exact performance difference because the CPU utilization of the
Policy Server CPU was 50% for the Extranet test and was 100% for the
Login test. This also means that the Extranet performance would have
been significantly higher if there were enough Web servers available
in the lab to drive the Policy Server's CPU to full utilization.
Table 2: SiteMinder Extranet and Login
- 1 Policy Server with 1 CPU
The benchmark results lead us to
- Netegrity SiteMinder 4.51
outperforms all other products we've tested so far
for the AuthMark Login and Extranet Scenarios.
- SiteMinder 4.51
delivers the highest Login and Extranet performance per
policy/security server CPU of any product we have tested
- SiteMinder delivers outstanding performance
scaling as CPUs and Policy Servers are added to a
Mindcraft certifies that the results reported
accurately represent the performance of Netegrity
SiteMinder 4.51 running on Sun Enterprise servers configured as
specified herein and as measured by AuthMark benchmark.
Our test results should be reproducible by others
using the same test lab configuration, the same Sun
server configurations, and the same software
configurations documented in this white paper.
and Test Details
The information in this publication is
subject to change without notice.
MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR
OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR
CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING,
PERFORMANCE, OR USE OF THIS MATERIAL.
This publication does not constitute
an endorsement of the product or products that were
tested. This test is not a determination of product
quality or correctness, nor does it ensure compliance
with any federal, state or local requirements.