Compaq, Microsoft, and OpenNetwork Technologies
sponsored the testing in this report. Mindcraft, Inc. conducted
the performance tests described in this report at OpenNetwork Technologies
in Clearwater, Florida.
OpenNetwork Technologies DirectorySmart
4.7 with Microsoft Windows 2000 Active Directory running on Compaq
Proliant ML570 systems delivers outstanding performance at low cost
using 1,000,000-user and 15,000,000-user directories.
Mindcraft's tests show that DirectorySmart with Active
Directory provides enterprise-class performance. Our test results
- For a 1,000,000-user directory, 309,227 Extranet
operations/minute and 14,056 Extranet operations/minute/CPU, the
highest we've measured by 19%.
- For a 15,000,000-user directory, 294,390 Extranet operations/minute
and 13,381 Extranet operations/minute/CPU.
Performance is an important consideration when evaluating which
product to buy. However, performance comes at at cost. So, in this
paper we present not only performance measurements but also normalized
performance metrics based on the total cost of ownership for each
We used the Mindcraft iLOAD
MVP™ test tool running the AuthMark™
Login and Extranet
Scenarios to test performance of the access control and identity
management solutions covered in this paper.
Total Cost of Ownership (TCO) represents the costs to acquire,
install, maintain, and use a solution. TCO includes the following
- The cost of buying all of the hardware used including computers
and networks (we excluded cable costs and the cost of the load
generator systems, which are not part of the solution).
- The cost of licensing all of the software used.
- Training class costs.
- Hardware maintenance cost for the evaluation time period.
- Software maintenance cost for the evaluation time period.
- Personnel costs for training, installation, and time spent supporting
the solution for the evaluation time period.
In this paper, we evaluate TCO for a three-year period. Also, we
use two metrics to help you compare these results to others and
to make purchasing decisions and project justifications: TCO/Performance
and Annual TCO/User.
TCO/Performance is a price/performance metric that is useful for
comparing performance results of different solutions because it
normalizes performance based on the cost to own the solution. A
lower TCO/Performance metric is better than a higher one because
the solution with the lower metric costs you less per unit of performance
than one with a higher metric.
Annual TCO/User is simply the annualized TCO (TCO divided by three)
divided by the number of users in the directory. Using Annual TCO/User
metrics based on the same number of users, you can make informed
purchasing decisions and Extranet project justifications. Annual
TCO/User should be used only with solutions that meet your performance
requirements. A lower Annual TCO/User metric means that the solution
costs you less per user each year than one with a higher metric.
So a smaller Annual TCO/User is better.
There is a caveat to using the Annual TCO/User metric: it is affected
significantly by the number of users in the directory being tested.
Therefore, when comparing Annual TCO/User metrics be sure that they
were based on the same number of users.
The TCO spreadsheet for these tests
shows how we arrived at the TCO and calculates the TCO/Performance
and Annual TCO/User metrics. We have set up the TCO spreadsheet
so that you can enter your own costs and even evaluate the TCO/Performance
and Annual TCO/User for other solutions.
Extranet Scenario Results Summary
The Extranet Scenario simulates customers or suppliers logging
into a private Web site and obtaining information they are authorized
to get. It measures the combination of one user authentication and
10 authorizations for access to resources (these 11 Extranet operations
constitute one Extranet sequence). We report the total operations
per minute. The Extranet Scenario, because it uses a more realistic
mix of operations than the Login Scenario, provides a better basis
for comparing access control and identity management solutions.
You can find a more complete description of the Extranet
Scenario in the detailed paper.
Figure 1 shows the Extranet Scenario
performance of DirectorySmart 4.7 with Active Directory for tests
with 1,000,000-user and 15,000,000-user directories. The X-axis
shows the total number of CPUs used in all of the servers. We did
not test a six-CPU configuration with a 15,000,000-user directory.
1: DirectorySmart 4.7
with Active Directory Extranet Performance
Table 1 shows the Extranet Scenario performance
and TCO metrics. The per CPU performance is the highest we've measured
to date by almost 20%. The 1,000,000-user directory test results
- Doubling the number of servers doubles performance, demonstrating
this solution scales linearly.
- Doubling the number of Web and directory servers improves TCO/Performance
more than 42%, giving you more bang for your money.
- Doubling the number of servers increases the Annual TCO/User
metric only 15%, making this a cost-effective way to double performance.
Table 1: DirectorySmart 4.7 with Active
Extranet Performance and TCO Metrics
(Lower TCO metrics are better)
The performance of the 15,000,000-user directory test is notable
because Active Directory had to access disk for each user authentication,
whereas all of the user authentication data was cached in memory
for the 1,000,000-user directory tests.
The detailed paper provides additional
information on these and other Extranet tests.
Login Scenario Results Summary
The Login Scenario simulates users requesting and receiving the
first Web page at a protected Web site. It measures the combination
of one user authentication and one authorization for access to a
protected resource (called a Login). We report Logins/minute. The
Login Scenario assumes that 10% of the user population in a directory
logs in concurrently to use resources. So, for the tests with a
1,000,000-user directory, 100,000 users did a Login. For the test
with a 15,000,000-user directory, 1,500,000 users did a Login. Login
Scenario performance results should be considered best-case performance.
The detailed paper explains the Login
Scenario more thoroughly.
Figure 2 shows the Login Scenario performance
of DirectorySmart 4.7 with Active Directory for tests with 1,000,000-user
and 15,000,000-user directories. The X-axis shows the total number
of CPUs used in all of the servers. We did not test a 32-CPU configuration
with a 15,000,000-user directory.
2: DirectorySmart 4.7 with
Active Directory Extranet Performance
Table 2 presents the Login Scenario performance
and TCO metrics. For this scenario, the 1,000,000-user directory
- Linear scaling; performance doubles as the number of servers
- TCO/Performance improves more than 40% by doubling the number
of Web and directory servers, giving you more bang for your money.
- Annual TCO/User increases only 17% when the number of servers
doubles, making this a cost-effective solution for doubling performance.
Table 2: DirectorySmart 4.7 with Active
Login Performance and TCO Metrics
(Lower TCO metrics are better)
For this scenario too, the performance of the 15,000,000-user directory
test was impacted because Active Directory had to access disk for
each user authentication.
on these and other Login tests is available in the detailed part
of this paper.
These results lead us to conclude that:
- OpenNetwork Technologies DirectorySmart 4.7 with
Microsoft Windows 2000 Active Directory on Compaq ProLiant ML570
servers sets the TCO/Performance and Annual TCO/User standards
against which other Web access control and identity management
solutions will be measured.
- OpenNetwork Technologies DirectorySmart 4.7 with
Microsoft Active Directory delivers the highest AuthMark Extranet
Scenario performance per CPU that we have measured to date.
- DirectorySmart with Active Directory on ML570
servers offers a low cost-per-user access control and identity
management solution for 1,000,000 users, 15,000,000 users, and
- DirectorySmart with Active Directory delivers outstanding performance
Mindcraft certifies that the reported results accurately represent
the performance of OpenNetwork Technologies DirectorySmart 4.7 using
Microsoft Active Directory and Windows 2000 Server running on Compaq
ML570 systems configured as specified herein and as measured by
the AuthMark Benchmark.
Our test results should be reproducible by others using the same
test lab configuration, the same server configurations, and the
same software configurations documented in this white paper.
The information in this publication is subject to
change without notice.
MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR OMISSIONS CONTAINED
HEREIN, NOR FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM
THE FURNISHING, PERFORMANCE, OR USE OF THIS MATERIAL.
This publication does not constitute an endorsement
of the product or products that were tested. This test is not a
determination of product quality or correctness, nor does it ensure
compliance with any federal, state or local requirements.
Mindcraft is a registered trademark of Mindcraft,
Product and corporate names mentioned herein are
trademarks and/or registered trademarks of their respective companies.