Mindcraft Certified Performance Comparison Report

Netscape Directory Server 3.0
Netscape Directory Server 1.0
and
Novell LDAP Services for NDS 1.0

Contents

Executive Summary 
Server Environments Tested
Test Lab
Test Procedures
Performance Analysis
Mindcraft's Certification
Glossary

 

Executive Summary

Mindcraft used our LDAP server benchmark, DirectoryMark, to measure the performance of Netscape Directory Server 3.0, Netscape Directory Server 1.0, and  Novell LDAP Services for NDS 1.0. Figure 1 summarizes the peak performance measured in search operations/second. It shows that Netscape Directory Server 3.0 (NSDS3) is 4.8 times faster than its predecessor, Netscape Directory Server 1.0 (NSDS1), and that NSDS3 is 229.5 times faster than Novell LDAP Services for NDS 1.0 (LDAP for NDS). We ran the tests on a Hewlett-Packard NetServer LX Pro system using a database of 10,000 entries with random, realistic values in each data item.

Figure 1: Peak Performance

peakperf.gif (7613 bytes)

Figure 2 shows the average response times for these LDAP servers. NSDS3 responded to queries an average of 5.2 times faster than NSDS1 and 254 times faster than LDAP for NDS.

Figure 2: Average Response Times

avgresptimes.gif (15523 bytes)

Server Environments Tested

Table 1 shows the detailed server computer configuration. Tables 2, 3, and 4 give the software configurations for NSDS3, NSDS1, and LDAP for NDS, respectively.

Table 1: HP NetServer LX Pro Configuration

HP NetServer LX Pro

System
CPU 2 x 200 MHz Intel Pentium Pro; only one CPU was used for testing
Cache L1: 16 KB (8KB I + 8 KB D)
L2: 512 KB
RAM 512 MB ECC
Disk NetRAID controller with 4 MB of RAM;  6 Ultra/Wide SCSI Hot Swappable 4.2GB drives
Network Built-in 100Base-TX adapter; half-duplex

 

Table 2: Netscape Directory Server 3.0 Configuration

Netscape Directory Server 3.0

Operating
System
Windows NT Server 4.0, Service Pack 3; on C: drive; data on D: drive
Configuration Parameters:
   Application Performance Boost=Maximum
   /numprocs=1 in boot.ini
Directory Server Software Netscape Directory Server 3.0
Configuration Parameters:
   Log Level=0
   Track Modifies=Yes
   Schema Check=Yes
   Index of attributes=default
   Max Entries in Cache=500,000
   Max DB Cache Size=400,000,000

 

Table 3: Netscape Directory Server 1.0 Configuration

Netscape Directory Server 1.0

Operating
System
Windows NT Server 4.0, Service Pack 3; on C: drive; data on D: drive
Configuration Parameters:
   Application Performance Boost=Maximum
   /numprocs=1 in boot.ini
Directory Server Software Netscape Directory Server 1.03
Configuration Parameters:
   Log Level=0
   Track Modifies=Yes
   Schema Check=Yes
   Index of attributes=default
   Max Entries in Cache=500,000
   Max DB Cache Size=400,000,000

Table 4: Novell LDAP Services for NDS Configuration

Novell LDAP Services for NDS 1.0

Operating
System
IntranetWare 4.11 with the following updates:
   landrv32: 6/23/97 update
   dskdrv: 6/23/97 update
   IW Support Pack 4: 10/24/97
   LDAP update 1.03
Configuration Parameters:
   Operating system and data on C:
   SMP not installed
   NDS inactivity synchronization interval=1440 min

   Directory Cache Allocation Wait Time=0.5 sec
   Max Directory Cache Buffers=20,000
   Min Directory Cache Buffers=8,000
   Max Number of Internal Directory Handles=1000
   Max Number of Directory Handles=1000
Directory Server Software LDAP Services for NDS 1.03
Configuration Parameters:
   All parameters were set to their defaults

 

Test Lab

Figure 3 shows the test lab configuration we used. We needed only one client test system, configured as shown in Table 5, to load the LDAP servers.

Figure 3: LDAP Server Test Lab Configuration

LDAP Server Test Lab Configuration (10100 bytes)

Table 5: Client Test System

Client Test System

System
CPU 200 MHz Pentium Pro; Intel VS440FX motherboard
Cache L1: 16 KB (8KB I + 8KB D)
L2: 256 KB
RAM 64 MB EDO
Disk 2GB EIDE
Operating System Windows NT Server 4.0, Service Pack 3 and Post SP3 TCP/IP Hot Fix installed
Network 100Base-TX 3Com 3C905-TX Network Interface Card; half-duplex

We used a dedicated 100Base-TX network for all tests.

Test Procedures

Mindcraft used the DirectoryMark benchmark for these tests. It is made up of four major programs:

  1. dbgen: The database generation program. It generates an LDIF file containing information for the specified number of people.
  2. scriptgen: The test script generation program. It generates test scripts that use various LDAP operations. The frequency for each type of operation is specified by a percentage. The LDAP operations, and their associated parameters, used in the testing for this report were:
  3.  

    Parameter Meaning
    -u Unique ID search. This search uses the UID attribute in the database. This type of search is an exact match on an attribute with a unique value. UID searches would be used, for example, by an e-mail program to get information associated with an e-mail address.
    -s General search. General searches are further subdivided in the following mix:
    10% Searches resulting in a "not found" status
    20% Wildcard search on CN attribute with * at the beginning of the value searched for
    30% Exact match search on CN attribute
    40% Exact match search on an attribute randomly selected from: CN, SN, GN, UID.

     

  4. dm_master: The master controller for the DirectoryMark benchmark. It starts the dm_client program, collects test results, and formats a report.
  5. dm_client:The client program. For the duration of the test time it loops through the test script issuing the specified operations against the LDAP server under test.

Table 6 shows the DirectoryMark configuration parameters we used.

Table 6: DirectoryMark Configuration

DirectoryMark Benchmark Configuration

Parameter

Value Comment
Database Size

10,000

Represents 10,000 people

Number of dm_client processes used for testing 1 Only one process is used on the client test system
Number of dm_client threads used for testing

4

Each thread issues its own LDAP operations
Number of operations in test script

10,000

Script is repeated when a dm_client thread reaches the end

scriptgen -s value 25 25% of operations are general searches
scriptgen -u value 75 75% of operations look up a UID

For the Netscape tests, we booted Windows NT 4.0 on the NetServer directly from its hard disks. For the Novell tests, we first booted MS-DOS on the NetServer from a floppy disk and then started the IntranetWare server from the C: hard disk.

We ran each test with a 15 minute warm-up time to fill the server's cache and then a 10 minute test time. The test was done by automatically restarting the dm_client at the end of the warm-up time.

We ran perfmon on the NetServer during the Netscape tests and had it log processor, memory, and physical disk performance information to disk every 5 seconds. During the Novell tests we ran monitor.nlm to track performance on the server and logged LDAP transaction summaries to a file.

Other than connection and anonymous LDAP BIND operations, we used only search operations for our testing. This was done because searches account for almost all of the LDAP operations encountered by real servers. We chose the particular mix of general searches vs. UID searches to simulate a forward-looking scenario where e-mail and other programs will be large users of LDAP servers.

We determined via preliminary testing that a single client test system would be sufficient to load the LDAP servers we tested. When we looked at the client test system CPU utilization we found it averaged 75% for the NSDS3 tests, 16% for the NSDS1 tests, and less than 2% for the LDAP for NDS tests. At the same time we were able to drive the NetServer's CPU to full utilization for the Netscape products and to 95% utilization for LDAP for NDS.

Performance Analysis

These tests were constructed to facilitate comparing the performance of NSDS3, NSDS1, and LDAP for NDS. We used the same computer system as the server and the same client test system for all tests. However, the Netscape and Novell products run on different operating systems. The overall performance we measured is a function of both the LDAP server and its underlying operating system. This combined performance is just what users of LDAP servers will experience across a network (impacted by network traffic, of course).

The four primary factors that limit performance of LDAP servers are:

  1. The server's CPU and  memory;
  2. The disk subsystem;
  3. The network; and
  4. The software, including the operating system and the LDAP server.

We'll examine each factor individually.

Server CPU and Memory Performance

When we tested both NSDS3 and NSDS1, the NetServer's CPU was 100% utilized. For the LDAP for NDS test, the NetServer's CPU was 95% utilized. Since the client test system's CPU utilization was low to moderate for all tests, we conclude that the performance we measured in our tests of NSDS3 and NSDS1 was limited by the server system's CPU, not the performance of the client test system. Because LDAP for NDS did not saturate the NetServer's CPU, another factor must have limited its performance. CPU utilization alone does not tell us if the CPU was being used efficiently by the software; we will examine that below.

Memory was not a performance-limiting factor for any of the products. Perfmon showed that the Netscape products used less than 100 MB of memory. Similarly, LDAP for NDS used less than 100 MB.

Disk Subsystem Performance

The performance monitoring we did for NSDS3 and NSDS1 showed an average of less than one disk access per second once the cache was filled. We could not log disk activity for LDAP for NDS, however, the disk activity lights on the drives indicated very few disk accesses. These are reasonable observations given that the entire 10,000 person LDAP database can easily fit into memory on the NetServer system we used. Thus, we conclude that the disk subsystem was not a performance limiting factor.

Network Performance

The network was not a limiting factor for any of the LDAP servers. Network bandwidth utilization averaged under 40% in all cases because the operation rate was low and so little data was transferred for each operation.

Operating System and LDAP Software Performance

The performance of LDAP server software is affected by the overall performance of the operating system on which it is running. However, because we were doing "black box" testing, we could not readily separate the performance of the operating system from that of the LDAP server we tested. So, our analysis will focus on the LDAP servers themselves.

NSDS3 showed significant performance improvements over the pervious version, NSDS1. During the 10-minute test, NSDS3 handled 110,169 transcations while NSDS1 handled 23,060 transactions. Because the NetServer's CPU was 100% utilized and because the same operating system configuration was used for both Netscape Directory Server products, we conclude that NSDS3 makes more efficient use of the available resources than NSDS1. Even though the performance of both NSDS3 and NSDS1 will increase as the speed of the server's CPU increases, you will get more of a bang for your buck with NSDS3.

The performance of LDAP for NDS is poor - it did only 474 transactions for the 10-minute test period. While the test was running, we looked at the CPU utilization in the scheduling option of the monitor.nlm. It showed that all of the LDAP processes or threads combined used under 1% of the CPU. In fact, all of the monitored threads used under 5% of the CPU, yet the overall CPU utilization averaged about 95%. What caused this to happen?

We believe that most of the overhead is in searching the NDS data structures. We could not find any documentation for Novell LDAP Services for NDS 1.0 that specified how to create different types of attribute indexing nor were there indexing options available in the LDAP or NDS management programs. This leads us to conclude that some types of searches, such as wildcard searches, may go through the entire LDAP database sequentially to resolve the search because more optimal indexing techniques are not available. An additional inefficiency of LDAP for NDS comes from its mapping LDAP attributes to NDS attributes, although we couldn't measure how much overhead this mapping contributed.

Conclusion

The performance of the Netscape Directory Server 3.0 is a substantial improvement over that of Netscape Directory Server 1.0 and is high enough that it can be used in very demanding applications. Netscape Directory Server 3.0 can take full advantage of high-speed CPUs.

Regardless of the cause, search times for Novell LDAP Services for NDS are too long and its performance is unacceptably low.

Mindcraft's Certification

Mindcraft, Inc. conducted the performance tests described in this report between October 30 and December 12, 1997, and April 24 to 27, 1998.

Mindcraft certifies that the results reported herein fairly represent the performance of systems tested as measured by the DirectoryMark benchmark configured as specified. Our test results should be reproducible by others who use the same test lab configuration as well as the computer and software configurations and modifications documented in this report.

Glossary

LDAP
An acronym for Lightweight Directory Access Protocol.
LDIF
An acronym for LDAP Data Interchange Format. It is an ASCII format for specifying values for LDAP schema attributes.
NDS
An acronym for Novell Directory Services.
Thread
Short for "thread of control in a process." As used in this report, it means the number of DirectoryMark requestors threads simultaneously making requests of an LDAP server.
 

Copyright 1997-98. Mindcraft, Inc. All rights reserved.
Mindcraft is a registered trademark of Mindcraft, Inc.
For more information, contact us at: info@mindcraft.com
Phone: +1 (408) 395-2404
Fax: +1 (408) 395-6324