DirectorySmart 4.7 with
Active Directory:
Total Cost of Ownership and Performance Details

By Bruce Weiner
(PDF version, 197 KB)
September 19, 2001

Contents

Executive Summary
TCO
Extranet Results
Login Results
Full-Cache Results
Conclusions
Test Methodology
   iLOAD MVP
    AuthMark
Server Hardware
Server Software
Load Generators
 

Disclosure

Compaq, Microsoft, and OpenNetwork Technologies sponsored the testing in this report. Mindcraft, Inc. conducted the performance tests described in this report at OpenNetwork Technologies in Clearwater, Florida.

Mindcraft's tests show that DirectorySmart with Active Directory provides enterprise-class performance. Our test results include:

• For a 1,000,000-user directory, 309,227 Extranet operations/minute and 14,056 Extranet operations/minute/CPU, the highest we've measured by 19%.
  
  
• For a 15,000,000-user directory, 294,390 Extranet operations/minute and 13,381 Extranet operations/minute/CPU.

Performance is an important consideration when evaluating which product to buy. However, performance comes at at cost. So, in this paper we present not only performance measurements but also normalized performance metrics based on the total cost of ownership for each solution tested.

The Test Methodology section describes how we tested the performance of DirectorySmart with Active Directory and explains the Extranet and Login performance metrics.

Total Cost of Ownership

Total Cost of Ownership (TCO) represents the costs to acquire, install, maintain, and use a solution. TCO includes the following costs:

• The cost of buying all of the hardware used including computers and networks (we excluded cable costs and the cost of the load generator systems, which are not part of the solution).
• The cost of licensing all of the software used.
• Training class costs.
• Hardware maintenance cost for the evaluation time period.
• Software maintenance cost for the evaluation time period.
• Personnel costs for training, installation, and time spent supporting the solution for the evaluation time period.

In this paper, we evaluate TCO for a three-year period. Also, we use two metrics to help you compare these results to others and to make purchasing decisions and project justifications: TCO/Performance and Annual TCO/User.

TCO/Performance is a price/performance metric that is useful for comparing performance results of different solutions because it normalizes performance based on the cost to own the solution. A lower TCO/Performance metric is better than a higher one because the solution with the lower metric costs you less per unit of performance than one with a higher metric.

Annual TCO/User is simply the annualized TCO (TCO divided by three) divided by the number of users in the directory. Using Annual TCO/User metrics based on the same number of users, you can make informed purchasing decisions and Extranet project justifications. Annual TCO/User should be used only with solutions that meet your performance requirements. A lower Annual TCO/User metric means that the solution costs you less per user each year than one with a higher metric. So a smaller Annual TCO/User is better.

There is a caveat to using the Annual TCO/User metric: it is affected significantly by the number of users in the directory being tested. Therefore, when comparing Annual TCO/User metrics be sure that they were based on the same number of users.

The TCO spreadsheet for these tests shows how we arrived at the TCO and calculates the TCO/Performance and Annual TCO/User metrics. We have set up the TCO spreadsheet so that you can enter your own costs and even evaluate the TCO/Performance and Annual TCO/User for other solutions.

Detailed Extranet Results

The Extranet Scenario simulates customers or suppliers logging into a private Web site and obtaining information they are authorized to get. It measures the combination of one user authentication and 10 authorizations for access to resources (these 11 operations constitute one Extranet sequence). We report the total operations per minute. The Extranet Scenario, because it uses a more realistic mix of operations than the Login Scenario, provides a better basis for comparing access control and identity management solutions. You can find a more complete description of the Extranet Scenario below.

All of the TCO-oriented Extranet tests were done with the DirectorySmart cache configured as if the servers were situated in a network DMZ. In this "Normal" cache configuration policies and user status information are cached but user access rights and passwords are not cached. We did an additional Extranet test with the DirectorySmart cache configured as it could be if the servers were on the inside of a private network. See Full-Cache Results for more details.

Figure 1 shows the Extranet Scenario performance of DirectorySmart 4.7 with Active Directory for tests with 1,000,000-user and 15,000,000-user directories. The X-axis shows the total number of CPUs used in all of the servers. We did not test a six-CPU configuration with a 15,000,000-user directory.

Figure 1: DirectorySmart 4.7 with Active Directory Extranet Performance

Table 1 shows the Extranet Scenario performance and TCO metrics. The per CPU performance is the highest we've measured to date by almost 20%. Table 1 shows performance with a 1,000,000-user directory scaled linearly as the number of CPUs doubled. The corresponding TCO/Performance improved by over 42% with the additional CPUs. Also, the Annual TCO/User for the configuration with more CPUs was reduced by over 15%.

Table 1 shows the Extranet Scenario performance and TCO metrics. The per CPU performance is the highest we've measured to date by almost 20%. The 1,000,000-user directory test results show:

• Doubling the number of servers doubles performance, demonstrating this solution scales linearly.
• Doubling the number of Web and directory servers improves TCO/Performance more than 42%, giving you more bang for your money.
• Doubling the number of servers increases the Annual TCO/User metric only 15%, making this a cost-effective way to double performance.

Table 1: DirectorySmart 4.7 with Active Directory
Extranet Performance and TCO Metrics
(Lower TCO metrics are better)

The performance of the 15,000,000-user directory test is notable because Active Directory had to access disk for each user authentication, whereas all of the user authentication data was cached in memory for the 1,000,000-user directory tests.

Table 2 shows the CPU utilizations for the Active Directory and Web servers during the Extranet tests. During all of these tests the load generator systems had CPU utilizations between 30% and 35%. The reason that the server CPU utilizations were so close for all of the Extranet tests is that available network bandwidth between the Web servers and the load generators severely limited performance. For each authorization, which makes up 10 out of every 11 Extranet operations, a 14,000-byte file is transferred from a Web server to a load generator. This means that even for the lowest performance test, Test #1, each network segment used 87.2 Mbits/second just for sending the file data. Unfortunately, we were not able to measure directly the network bandwidth used. However, assuming a total network overhead of 10%, the effective bandwidth used was 96 Mbits/second out of a possible 100 Mbits/second. Adding the bandwidth needed for each authentication, it is clear that the network was saturated. Obviously, if more network bandwidth were available between the Web servers and the load generators, performance would have been significantly higher.

Table 2: DirectorySmart 4.7 with Active Directory Extranet CPU Utilization

Detailed Login Results

The Login Scenario simulates users requesting and receiving the first Web page at a protected Web site. It measures the combination of one user authentication and one authorization for access to a protected resource (called a Login). We report Logins/minute. The Login Scenario assumes that 10% of the user population in a directory logs in concurrently to use resources. So, for the tests with a 1,000,000-user directory, 100,000 users did a Login. For the test with a 15,000,000-user directory, 1,500,000 users did a Login. Login Scenario performance results should be considered best-case performance. See the Login Scenario for more details on the test.

All of the TCO-oriented Login tests were done using DirectorySmart's normal cache configuration, just like the Extranet tests. We did an additional Login test using DirectorySmart's full-cache configuration.

Figure 2 shows the Login Scenario performance of DirectorySmart 4.7 with Active Directory for tests with 1,000,000-user and 15,000,000-user directories. The X-axis shows the total number of CPUs used in all of the servers. We did not test a 32-CPU configuration with a 15,000,000-user directory.

Figure 2: DirectorySmart 4.7 with Active Directory Extranet Performance

Table 3 presents the Login Scenario performance and TCO metrics. For this scenario, the 1,000,000-user directory tests show:

• Linear scaling; performance doubles as the number of servers doubles.
• TCO/Performance improves more than 40% by doubling the number of Web and directory servers, giving you more bang for your money.
• Annual TCO/User increases only 17% when the number of servers doubles, making this a cost-effective solution for doubling performance.

Table 3: DirectorySmart 4.7 with Active Directory
Login Performance and TCO Metrics
(Lower TCO metrics are better)

For this scenario too, the performance of the 15,000,000-user directory test was impacted because Active Directory had to access disk for each user authentication.

The CPU utilizations for the Login Scenario test are shown in Table 4. For Tests #4 and #6, the load generator CPUs were 95% utilized. During Test #5, the load generator CPU utilization was 65%.

Because each Login does an authorization, there is a 14,000-byte file is transferred from a Web server to a load generator. So, Tests #4 and #6 used about 73 Mbits/second of the bandwidth between each load generator system and Web server. Because each network segment was in full-duplex mode, this bandwidth usage level was not enough to limit performance. For Tests #4 and #6, the 95% CPU utilization on the load generators did contribute to limiting overall performance. The amount of disk activity on the directory servers was a performance-limiting factor also. The lower CPU utilization in Test #5, which used a RAID to store the directory for 15,000,000 users, is the result of heavy disk activity to get the login credentials for each user.

Table 4: DirectorySmart 4.7 with Active Directory Login CPU Utilization

Detailed Full-Cache Results

The tests reported in this section were done with the DirectorySmart cache configured as it could be if the servers were on the inside of a private network. This means that DirectorySmart was configured to cache configuration policies, user status information, user access rights, and passwords.

Table 5 presents the Extranet and Login Scenario performance using Active Directory and DirectorySmart's full-cache configuration. The Extranet performance is the highest we've measured to date. Additionally, the Extranet operations/minute/CPU is more than double that of any solution we have tested. With Active Directory, DirectorySmart's full-cache Login performance also exceeds that of any product we have tested by 36%. In addition, its Login operations/minute/CPU is more than 2.6 times that of any other product.

Table 5: DirectorySmart 4.7 with Active Directory
Full-Cache Extranet and Login Performance

In Table 6, you can easily see the benefit of using the full-caching capabilities of DirectorySmart: the CPU utilization for the Active Directory server is 0% for both the Extranet and Login Scenario tests, Tests #7 and #8, respectively. During the Extranet Test #8, the load generator systems had 75% CPU utilizations. As with the other Extranet tests, the performance-limiting factor was the network bandwidth between the Web servers and the clients. Extranet performance would have been significantly higher if more network bandwidth were available between the Web servers and the load generators. The Login performance was limited by the load generator systems, which ran at 100% CPU utilization.

Table 6: DirectorySmart 4.7 with Active Directory
Full-Cache Extranet and Login CPU Utilization

Conclusions

The results lead us to conclude that:

• OpenNetwork Technologies DirectorySmart 4.7 with Microsoft Windows 2000 Active Directory on Compaq ProLiant ML570 servers sets the TCO/Performance and Annual TCO/User standards against which other Web access control and identity management solutions will be measured.
• OpenNetwork Technologies DirectorySmart 4.7 with Microsoft Active Directory produced the highest overall and per CPU Extranet and Login Scenario performance that we have measured.
• DirectorySmart 4.7 with Active Directory delivers the highest AuthMark Extranet Scenario performance per CPU to date.
• DirectorySmart with Active Directory on ML570 servers offers a low cost-per-user access control and identity management solution for 1,000,000 users, 15,000,000 users, and more.
• DirectorySmart with Active Directory delivers outstanding performance scaling.

Test Methodology

Mindcraft^® tested the performance of DirectorySmart 4.7 with Active Directory using our iLOAD MVP™ tool to run the AuthMark™ Benchmark Login and Extranet Scenarios. In this section, we describe these tools so that you will be able to understand our performance measurements. 

iLOAD MVP Overview

iLOAD MVP is a general-purpose, script-driven capacity planning, benchmarking, and regression testing tool. The major components of iLOAD MVP are:

• A Control Center that manages load generator systems, controls test script execution, and reports on test results.
• Multi-threaded client load generators that execute test scripts to simulate users accessing a server.
• Test script generation programs.
• Test data generation programs.

iLOAD MVP provides the capabilities needed to test high-performance servers with a small number of load generator systems. Its capabilities include:

• The ability to simulate a large number of simultaneous user sessions. The number of user sessions is limited only by the load generator OS, the amount of memory and the performance of the load generator systems.
• Support for HTTP 1.0 and 1.1.
• Support for authentication and authorization.
• Support for SSL.
• Custom test scripts.

The AuthMark Benchmark

The AuthMark Benchmark is designed to test the performance of products that provide authentication and authorization services in support of Web servers. Authentication is the process of verifying who a user is; it typically occurs when a user logs in. Authorization is the process of verifying that an authenticated user is allowed to see or to use a particular resource. In the case of a Web server such resources include HTML files, graphic files, and programs that generate Web pages dynamically.

AuthMark simulates a large number of users accessing Web servers via their browsers. This approach permits AuthMark to test authentication and authorization performance independent of the technology used to provide those services.

AuthMark consists of several test scenarios to determine various aspects of performance for authentication and authorization systems under different circumstances. For the DirectorySmart with Active Directory tests we used the AuthMark Login and Extranet Scenarios.

AuthMark Login Scenario

The AuthMark Login Scenario focuses on testing authentication. It simulates users requesting and receiving the first Web page at a protected Web site. The Login Scenario measures the combination of one user authentication and one authorization for access to a protected resource (called a Login). We report Logins/minute. Understanding what happens during a login will help you understand what the Login Scenario measurements mean.

Login Process

The following simplified sequence will walk you through the login process to show you how it works using the DirectorySmart login process (which differs from the HTTP 1.0 and 1.1 protocols in that a form is used to send the user name and password):

1. The iLOAD load generator (a thread of a script-driven program that emulates a web browser) sends a request to the Web server for a protected resource.
2. The Web server returns the login form (in these tests, the login credentials are username and password) and a cookie from the DirectorySmart Web server plug-in.
3. Using an HTTP POST operation, the load generator returns the login credentials and the DirectorySmart cookie to the Web server, which forwards them to the DirectorySmart Web server plug-in.
4. The DirectorySmart Web server plug-in checks the credentials against those in the LDAP directory to validate the user name and password. If authenticated, authorization is checked in the next step. Otherwise, an an authentication error is returned returned.
5. DirectorySmart checks if the user is authorized to access the requested resource. If so, the resource, which in this case is one of the 14 KB Web pages, is returned to the load generator along with an encrypted session cookie.

Once it has been authenticated, the iLOAD load generator automatically sends the encrypted session cookie in each header whenever it requests a URL in the same realm, just like a Web browser does.

Login Scenario Configuration

Table 7 shows the AuthMark Login Scenario configuration parameters we used.

Table 7: AuthMark Login Scenario Configuration Parameters

The number of user sessions active during a given test run is determined by the length of the test and the number of logins. Sessions are not logged out once created. Instead, each session remains quiescent after login. 

Running the Login Scenario

The basic steps for running the Login Scenario are:

1. Generate the data to fill the security directory. iLOAD MVP provides a tool to generate realistic data for the LDAP V3 organizationalPerson object class and Netscape's inetOrgPerson object class. It also includes tools to load the same data into an LDAP directory such as Active Directory, which were used for this test.
2. Load the security directory with the user data.
3. Generate the test scripts for the Login Scenario. iLOAD MVP provides a tool to do this. These scripts drive iLOAD MVP to simulate user interaction with the Web servers.
4. Load Web pages on the Web servers. There are 100 Web pages each of which is 14 KB in size for the Login Scenario.
5. Load and configure the authentication/authorization system.
6. Run the benchmark.

The Login Scenario test script selects users randomly from the user database (see Table 7 for the numbers we used for these tests). The tester is free to select the number of load generator systems and the number of iLOAD MVP load generator threads to use.

The tester selects the number of load generators to get the highest performance possible from the authentication/authorization system being tested. In order to obtain the peak performance from an authentication/authorization system, the tester may need to use multiple Web servers and directory servers.

The tester is permitted, but not required, to do a warm-up run of the test scenario in order to get the servers to a state that would more likely represent the state they would be in during normal operation. For this benchmark, we warmed-up the servers by running the test script in its entirety.

Extranet Scenario

The Extranet Scenario simulates customers or suppliers logging into a private Web site and obtaining information they are authorized to get. It measures the combination of one user authentication and 10 authorizations for access to resources (these 11 Extranet operations constitute one Extranet sequence). We report the total operations per minute. The Extranet Scenario depicts a more complete and more realistic usage pattern than the Login Scenario.

The Extranet Scenario test execution starts with the same operation sequence as the Login Scenario (steps 1 - 5 above) and continues with the following operations:

6. A load generator requests a resource, sending the encrypted session cookie along with the request.
7. The authorization services check the validity of the user and that the user is authorized to have access to the resource.
8. If the user is authorized, the resource is returned.
9. The load generator then returns to Step 6 eight more times, for a total of 10 resources requested and returned.

DirectorySmart checks the continued validity of the authenticated user each time a resource access request is made to ensure that the user session has not been revoked.  However, the user is not re-authenticated. As a result, the user does not see a new login request as long as the resources being accessed are in the Internet domain in which the user has been authenticated.

For the Extranet Scenario, we warmed-up the servers by running the test script in its entirety.

Hardware Configurations Tested

We used Compaq ProLiant ML570 servers for all tests. Table 8 shows the detailed Compaq ProLiant ML570 configuration. Table 9 shows by test the number of servers and how many CPUs each had.

We used a Cisco Catalyst 2900XL switch to network the servers and load generator systems.

Table 8: Compaq ProLiant ML570 Servers

Table 9: Server Configurations by Test

Figure 3: Test #1 - Extranet, 1 Million Users
1 Directory Server with 2 CPUs and 3 Web Servers with 3 CPUs

Figure 4: Test #2 and #3 - Extranet, 1 Million and 15 Million Users
2 Directory Servers with 2 CPUs (internal RAID for Test #3) and 6 Web Servers with 3 CPUs

Figure 5: Test #4 and #5 - Login, 1 Million and 15 Million Users
2 Directory Servers with 4 CPUs (internal RAID for Test #5) and 2 Web Servers with 4 CPUs

Figure 6: Test #6 - Login, 1 Million Users
4 Directory Servers with 4 CPUs and 4 Web Servers with 4 CPUs

Figure 7: Test #7 and #8 - Full-Cache Extranet and Login, 1 Million Users,
1 Directory Server with 1 CPU and 7 Web Servers with 2 CPUs

Server Software Configuration and Tuning

We used the following software on the Compaq Proliant ML570 servers:

• Microsoft Windows 2000 Server, Service Pack 2
• Microsoft Active Directory
• Microsoft IIS 5
• OpenNetwork Technologies DirectorySmart 4.7

All software ran with default settings except for the following:

• For Windows 2000 Server on all systems:
  
  On all servers, disabled unneeded services, which left the following services running:
  
      DHCP Client
      Distributed File System
      DNS Client
      Event Log
      IIS Admin Service
      IPSEC Policy Agent
      Network Connections
      Plug and Play
      Protected Storage
      Remote Access Connection Manager
      Remote Procedure Call (RPC)
      Remote Registry Service
      Security Accounts Manager
      Server
      TCP/IP NetBIOS Helper Service
      Telephony
      Windows Management Instrumentation
      Workstation
      World Wide Web Publishing Service
  
  On all servers, set the network interface cards (NICs) to:

      100 Mbit/full-duplex (explicity set, default=auto)
      Receive Buffers = 768 (default=48)
      Coalesce Buffers = 32 (default=8)
      TransmitControlBlocks = 64 (default=32)

  On all servers, set the Registry key
  HKLM/HKLM/SYSTEM/CurrentControlSet/Services/Tcpip/Parameters:

      TcpWindowSize = 0xffff
      MaxUserPort = 0xfffe
  
  

• On all Web servers:
  
  Set the Registry key HKLM/HKLM/SYSTEM/CurrentControlSet/Services/Tcpip/Parameters:
  
      MaxFreeTWTcbs = 0x10c110
      MaxHashTableSize = 0x10000
      NumTcbTablePartitions = 0x40
      TcpTimedWaitDelay = 0x3c
  
  

• On all Active Directory servers:

  Boot with "/3GB" switch to allow directory process to use more memory for cache

  Set directories to "Native Mode" (default is "Mixed Mode"). Mixed Mode means you have both NT4 and Windows 2000 domain controllers. Native Mode means you have only Windows 2000 domain controllers.

  Use "ntdsutil" to set Active Directory parameters:

          MaxPoolThreads = 10 (default=4)
          MaxDatagramRecv = 65535 (default=1024)
          MaxConnections = 10000 (default=5000)
          MaxActiveQueries = 80 (default=20)
          MaxPageSize = 4096 (default=1000)
  
  Change ACLs for all objects in directory:

          Remove the "read all properties" permission for
          Everyone.
  
          Remove the following "pre-Windows 2000 Compatible
          Access" permissions to reduce the size of each in-
          memory object. This enables 100,000 active users
          to be cached in memory during the warm-up phase:

              List Contents
              Special
              Special (it does appears twice)
              Read Permission
  

• For DirectorySmart 4.7:
  
  POST_TO_LOGIN="Y"
  CONNECTION_POOL_SIZE="60"
  CONNECTION_POOL_MAX_CHECKS="120"
  WEB_SERVICE_CACHE_INITIAL_SIZE="20"
  USER_CACHE_SIZE="100002"
  USER_CACHE_REBALANCE_EVERY_N="99998"
  

User Data

User data was loaded into Active Directory using an LDIF file that was generated using a tool provided by OpenNetwork Technologies. The LDIF file had the following characteristics:

• All users have CNs of the form "USR-#". Examples are "USR-1" and "USR-63285".
  
  
• All users had the same password, "newPassword".

Here is a sample entry in the LDIF file:

dn: cn=USR-0,ou=People,dc=domain2,dc=com
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectClass: user
uid: USR-0
cn: USR-0
sn: Number0
givenname: Person0
sAMAccountName: USR-0
useraccountcontrol: 512
unicodePwd::IgBuAGUAdwBQAGEAcwBzAHcAbwByAGQAIgA=
organizationdn: organizationuid=ORG-0,ou=Organizations,dc=domain2,dc=com
roledn: roleuid=ROLE-0-0,ou=Roles,dc=domain2,dc=com
telephonenumber: 9999999999
accessstatus: Active
startdate: 20000101
enddate: 20041231

Client Test Systems

For all of the tests, we used personal computers for the load generator systems configured as shown in Table 10. The number of load generator systems we used for each test are shown in Figures 3 through 7.

Table 10: PC Load Generator Systems Configuration

NOTICE:

The information in this publication is subject to change without notice.

This publication does not constitute an endorsement of the product or products that were tested. This test is not a determination of product quality or correctness, nor does it ensure compliance with any federal, state or local requirements.

Mindcraft is a registered trademark of Mindcraft, Inc.

Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.