dirmark-logo.gif (20645 bytes)

The LDAP Performance Benchmark

 

Overview

DirectoryMark is a new benchmark that measures the LDAP performance of different server platforms and implementations under real world situations. DirectoryMark simulates several clients which simultaneously perform LDAP transactions on the server. The clients measure the server's response time and throughput and a summary of the results is generated. To simulate real client loads, each client runs its own script of transactions on the LDAP server. The mixture of transaction types can be configured to simulate different operation loads.

DirectoryMark is written to be independent of the server platform and the software running on it. DirectoryMark should be able to test any fully LDAP compliant server.  It has been used successfully to test Netscape Directory Server 1.0x and 3.0, Microsoft Exchange 5.0, AltaVista Directory 1.0, and UMich slapd-3.3.

In order to test Microsoft Exchange 5.0, DirectoryMark uses Microsoft's Directory API (DAPI), which is their proprietary Directory access protocol. This was done because Exchange 5.0 only supports LDAP search operations. All other operations on Exchange must be done using DAPI.

Performance Measures

DirectoryMark measures the latency and throughput for the common LDAP operations. The latency is the time it takes for an operation to be completed measured in milliseconds.  The throughput is the number of operations completed over a unit of time, measured in operations/second.

The LDAP operations which DirectoryMark performs are:

The ratios of each operation can be configured.


 

Testbed

The testbed consists of at least two machines. One machine running an LDAP server, the other running the DirectoryMark benchmark. It could be extended further to have clients running on multiple machines if necessary.  The testbed consists of the server configuration as well as the client benchmark load.

Server Configurations

The following parameters must be known about the server portion of the testbed for meaningful comparisons to be made. DirectoryMark treats the server as a black box, and these parameters must be specified by the server administrator.

Server Settings

Database Configurations

The standard databases are generated by an DirectoryMark tool called dbgen, which outputs a data file in LDIF format.  The default object class assumed by DirectoryMark is organizationalPerson  as defined by IETF RFC 2256.  DirectoryMark also supports the proposed inetOrgPerson object class as well.  The LDIF files generated by the DirectoryMark dbgen tool look similar to this:
 

Transaction scripts for the client can be created for any database as they are generated from an LDIF file.

Client Configurations

The following parameters of the load can be specified:


Setting Up A Directorymark Test

In order to run DirectoryMark you must first edit the config file.  The config file contains DirectoryMark configuration settings including the hostname and port of the LDAP server, the number of clients to run, and the scripts the clients should use. The DirectoryMark distribution will include several different configurations to run which can be used to test the server with different sized databases and query loads.

A sample config file:

 
# This is where the LDAPServer is
# Example: SERVERNAME:localhost
# Example: SERVERNAME:192.168.0.1
SERVERNAME:dirserver.test.com

# LDAP port
# Example: SERVERPORT:portnumber
SERVERPORT:389

# Number of client processes
# Example: NUMCLIENTS:number of clients
NUMCLIENTS:1

# number of threads on each client process. Note that you must have generated
# enough scripts with scriptgen.pl. For example, if you generated 7 scripts
# then you can use a value of 7 or less here.
# Example: THREADSPERCLIENT:number of threads
THREADSPERCLIENT:1

# number of ms each client will wait between operations
# Example: WAITBETWEENOPERATIONS:0
WAITBETWEENOPERATIONS:0

# Number of seconds to wait for search results.
SEARCHTIMEOUT: 60

# set to 1 if you want each client to bind once as Directory Manager
# set to 0 if you want to bind before each operation
# Example :BINDONCE:0
BINDONCE:0

# The basename of the client scripts. The first thread will use
# the script named basename0, the second thread will use the script named
# basename1, and so on...
# Unix Example: CLIENTBASESCRIPTS:Scripts/address.client0.script
# Windows NT Example: CLIENTBASESCRIPTS:Scripts\messaging.client0.script
#
# If you want to run multiple DirectoryMark client processes, then
# you'll need to generate enough scripts (see the "-c" option of
# scriptgen) and specify the client script bases here. The first
# process # will use the first basename specified, the second process
# will take # the second basename specified, and so on. For example:
# CLIENTBASESCRIPTS:Scripts/try.client0.script:Scripts/try.client1.script
#
CLIENTBASESCRIPTS:Scripts/try.client0.script

# This is the root DN, and root password.
# Example:BINDDN:"cn=Directory Manager, o=DirMark Inc."
# Example:BINDPW:unrestricted
BINDDN:"cn=Administrator, dc=foo ,dc=com"
BINDPW:password

# Where the statistics and html files will go when we're done
# Unix Example: STATDIR:Results/
# Windows NT Example: STATDIR:Results\
STATDIR:Results/

# The name of the server software.
# This is only used for reporting results.
# Example: SERVERSOFTWARE:"Netscape Communications Directory 3.0"
SERVERSOFTWARE:Directory Server

# The hardware the server is running on.
# This is only used for reporting results.
# Example:MACHINETYPE:"Sun UltraSparc1"
MACHINETYPE:VAX 1170

# The index settings on the server.
# This is only used for reporting results.
# Example:INDEXSETTINGS:"default"
INDEXSETTINGS:"default"

# The name of the ldif file of the database
# This is only used for reporting results.
# Example:LDIFFILE:ldapstone10000.ldif
#LDIFFILE:airius10k.ldif

# The number of entries in the database
# This is only used for reporting results.
# Example:DBSIZE:10000 Entries
DBSIZE:10000

# The cache settings for the server
# This is only used for reporting results.
# Example: CACHESETTINGS:default
CACHESETTINGS:cachesize default, dbcachesize default

# The maximum run time in seconds.
# If you use -1 then each script file will be run exactly once.
# Otherwise if time remains when a client thread gets to the end
# of its script file then it will start over from the beginning.
# Example: MAXRUNTIME:600
# Example: MAXRUNTIME:-1
MAXRUNTIME:600

# Use Microsoft DAPI (works only on NT)
# Example:USEDAPI:0
USEDAPI:0

# Exchange specific stuff (ignore if not testing Exchange)
# Example: EXCHANGEBASEPOINT:"/o=DirMark Inc./ou=DirMark Inc."
EXCHANGEBASEPOINT:"/o=DirMark Inc./ou=DirMark Inc."

# More Exchange specific stuff (ignore if not testing Exchange)
# Example:EXCHANGECONTAINER:"Recipients"
EXCHANGECONTAINER:"Recipients"

# ore Exchange specific stuff (ignore if not testing Exchange)
# Example:EXCHANGECONTAINER:"Recipients"
# EXCHANGESITEOU:"/ou=DirMark Inc."
EXCHANGESITEOU:"/ou=DirMark Inc."

#Other options:
#
# Uncomment VERBOSE: if you want the clients to print the LDAP operation
# and the time it took to the screen for each operation
#
# Uncomment DEBUG: if you want all the VERBOSE: output plus the
# arguments to each LDAP function
#
#DEBUG:1
#VERBOSE: 1

When DirectoryMark is finished, it creates an html file that contains a summary of the test run. This file is called Masterstat.html and is located in the directory that corresponds to the number of client threads used. The summary lists the number and type of operations performed, and the average times it took for each type of operation. The whole test run is summarized in the DirectoryMark rating, which is in terms of operations/second.
 

The Object Class

DirectoryMark supports the standard organizationalPerson object class.   See IETF RFC 2256 for a description of this object class.   DirectoryMark uses and generates data for the following attributes of organizationalPerson:
 

UID and givenName

Unfortunately, the organizationalPerson class doesn't include uid (user ID) or givenName (a person's first name) attributes. We feel these attributes are important and necessary for our proposed  Address and Messaging test scenarios. Rather than creating our own new object class we have decided to use  the organizationalPerson object class and use the seeAlso and description attributes to contain the givenName and UID values, respectively.

For example, in the sample LDIF entry above ("Kristi Vaserfirer") you can see that the value of the description attribute is the givenName which is "Kristi".  Similarly, the seeAlso attribute has Krist's UID which is "Kristi_Vaserfirer".

The -G and -U options of scriptgen and dbgen can be used to override using the seeAlso and description attributes to hold the givenName and UID values.  For example, if your object class  has an attribute called firstName then  you can invoke the dbgen tool with:

and dbgen will put the givenName values into the FirstName attributes.  You can then call scriptgen with the same option and it will create scripts that search on the firstName attribute rather than seeAlso.

If you are using the default values, that is seeAlso and description, then you should enable your directory server to index those values.  This will yield improved performance.
 

inetOrgPerson

DirectoryMark also supports the proposed inetOrgPerson object class (IETF draft "draft-smith-ldap-inetorgperson-00").  To use  inetOrgPerson, just use the -i option when invoking the dbgen and scriptgen tools.  DirectoryMark uses the following attributes of  inetOrgPerson:
   

Running DirectoryMark on University of Michigan LDAP-3.3

Because the University of Michigan LDAP-3.3 server does not support ACL settings, the following lines should be included in the etc/slapd.conf file: These settings will allow any client to read the database, but a user may only modify his or her own attributes after authenticating.

Architecture

The DirectoryMark performance benchmark consists of three components: The DirectoryMark Master controls the DirectoryMark clients. The DirectoryMark Master forks its clients and passes them the name of the DirectoryMark Transaction script each should run. The clients perform the operations specified in their Transaction Plan and report performance metrics back to the DirectoryMark Master when finished. Each client can use a different Transaction script to simulate real life load situations. Each client can also be configured by the DirectoryMark Master to wait a certain amount of time between transactions.

Transaction Scripts

Specification: Transaction scripts are a flat file that specify the operations the clients are to perform on the database. The transaction scripts are generated by the DirectoryMark script generation tool before the clients are started. Each client will be passed the pathname of its transaction plan by the DirectoryMark master. The following sections detail the layout of the flat file for each transaction type.
 


 

Search

A search transaction requires the following parameters be specified in the script file.

ldap_search

base scope filter attrs attrsonly Each entry is specified on a separate line in the script file with new lines between searches.
Example:

Add

Add requires the following parameters in the flat file:

ldap_add

dn attrs
  Example:

Delete

The delete operation  requires the following parameters specified in the script file:

ldap_delete

dn Each entry is specified on a separate line in the script file with new lines between searches
Example:

Modify

The modify operation requires the following attributes to be specified in the script file:

ldap_modify

dn changetype: modify mods Each entry is specified on a separate line in the script file with new lines between modification commands.
  i.e:

Modrdn

Modifying the rdn requires the following parameters in the script file:

dn

newrdn deleteoldrdn  Example:

Compare

The compare operation requires the following parameters be specified in the script file:

dn

attr value Example:

LDAP Master


LDAP Clients


DirectoryMark Tools

DirectoryMark comes with the following tools which may be useful in creating new tests.

dbgen.pl

scriptgen.pl

Running Scriptgen

The scriptgen tool is used to generate scripts that the client will run against the Directory Server. The usage is:
perl scriptgen.pl [options] output_name ldif_file number_of_transactions
A typical usage is:
perl scriptgen.pl -v -i -t 4 -S messaging ..\scripts\scr ..\ldif\db.ldif 10000
 
scriptgen Option Explanation
-A attribute list Comma-seperated list of attributes to return for searches.  Default is all attributes.
-a ratio Ratio of ldap_add commands in the generated script. 
-b Bind as a random person.  The default is to bind anonymously.
-B number The number of operations to do per bind command.  For example, -B 6 will do a bind followed by six LDAP operations (search, add, delete, etc), another bind followed by six more operations, etc.  Use -1 (negative 1) to do a single bind before all commands.
-c ratio Ratio of ldap_compare commands in the generated script.
-C ratio Ratio of "Netscape Communicator" searches in the generated script.
-d ratio Ratio of ldap_delete commands in the generated script.
-g ratio Ratio of exact searches on the gn attribute.
-G attribute Use this attribute to hold the givenName value.
-i New adds will follow the  inetOrgPerson object class.  The default is organizationalPerson.
-m ratio Ratio of ldap_modify commands in the generated script.
-n ratio  Ratio of exact searches on the cn attribute.
-O Turn on the attributes-only flag for searches.
-p number Number of processes.  The default is 1.
-r ratio Ratio of ldap_modrdn commands in the generated script.
-s ratio Ratio of exact searches on the sn attribute.
-S address or 
-S messaging
Generate scripts that follow the "address" or "messaging" scenarios described in this paper.  If this option is used then the -B, -c, -C, -d, -g, -m, -n, -r, -s, -u, and -w options are ignored.
-t number Number of threads that will be used to run commands against the Directory Server.  scriptgen will generate a script for each thread.
-u ratio Ratio of exact searches on the uid attribute.
-U attribute Use this attribute to hold the UID value
-v Turn on verbose output.
-w ratio Ratio of wildcard (eg, containing "*") searches on the cn attribute.
-W Use this option if you are testing Microsoft's Active Directory..
-x ratio Ratio of searches that won't match any entries in the database.
-z number Seed for the random number generator.
-Z scope Set the search scope. Acceptable values are LDAP_SCOPE_SUBTREE , LDAP_SCOPE_BASE, and LDAP_SCOPE_ONELEVE
- DirectoryMark is a trademark of Mindcraft, Inc. and Netscape Communications Corporation.