The LDAP Performance Benchmark

Overview

DirectoryMark is a new benchmark that measures the LDAP performance of different server platforms and implementations under real world situations. DirectoryMark simulates several clients which simultaneously perform LDAP transactions on the server. The clients measure the server's response time and throughput and a summary of the results is generated. To simulate real client loads, each client runs its own script of transactions on the LDAP server. The mixture of transaction types can be configured to simulate different operation loads.

DirectoryMark is written to be independent of the server platform and the software running on it. DirectoryMark should be able to test any fully LDAP compliant server. It has been used successfully to test Netscape Directory Server 1.0x and 3.0, Microsoft Exchange 5.0, AltaVista Directory 1.0, and UMich slapd-3.3.

In order to test Microsoft Exchange 5.0, DirectoryMark uses Microsoft's Directory API (DAPI), which is their proprietary Directory access protocol. This was done because Exchange 5.0 only supports LDAP search operations. All other operations on Exchange must be done using DAPI.

Performance Measures

DirectoryMark measures the latency and throughput for the common LDAP operations. The latency is the time it takes for an operation to be completed measured in milliseconds. The throughput is the number of operations completed over a unit of time, measured in operations/second.

The LDAP operations which DirectoryMark performs are:

search
modify
add
delete
compare
modrdn

The ratios of each operation can be configured.

Testbed

The testbed consists of at least two machines. One machine running an LDAP server, the other running the DirectoryMark benchmark. It could be extended further to have clients running on multiple machines if necessary. The testbed consists of the server configuration as well as the client benchmark load.

Server Configurations

The following parameters must be known about the server portion of the testbed for meaningful comparisons to be made. DirectoryMark treats the server as a black box, and these parameters must be specified by the server administrator.

Server Settings

Server Software
Server platform (Hardware, OS, amount of RAM, etc...)
Cache settings (cachesize, dbcachesize)
Index settings

Database Configurations

The standard databases are generated by an DirectoryMark tool called dbgen, which outputs a data file in LDIF format. The default object class assumed by DirectoryMark is organizationalPerson as defined by IETF RFC 2256. DirectoryMark also supports the proposed inetOrgPerson object class as well. The LDIF files generated by the DirectoryMark dbgen tool look similar to this:

Transaction scripts for the client can be created for any database as they are generated from an LDIF file.

Client Configurations

The following parameters of the load can be specified:

Number of clients
Type of operation mix for each client (i.e. 10% modify 90% search) -- in transaction scripts
Time to wait between operations
Option to bind once as the Directory Administrator for the entire test run, or as random users before each operation

Setting Up A Directorymark Test

In order to run DirectoryMark you must first edit the config file. The config file contains DirectoryMark configuration settings including the hostname and port of the LDAP server, the number of clients to run, and the scripts the clients should use. The DirectoryMark distribution will include several different configurations to run which can be used to test the server with different sized databases and query loads.

A sample config file:

# This is where the LDAPServer is
# Example: SERVERNAME:localhost
# Example: SERVERNAME:192.168.0.1
SERVERNAME:dirserver.test.com
# LDAP port
# Example: SERVERPORT:portnumber
SERVERPORT:389

# Number of client processes
# Example: NUMCLIENTS:number of clients
NUMCLIENTS:1

# number of threads on each client process. Note that you must have generated
# enough scripts with scriptgen.pl. For example, if you generated 7 scripts
# then you can use a value of 7 or less here.
# Example: THREADSPERCLIENT:number of threads
THREADSPERCLIENT:1

# number of ms each client will wait between operations
# Example: WAITBETWEENOPERATIONS:0
WAITBETWEENOPERATIONS:0

# Number of seconds to wait for search results.
SEARCHTIMEOUT: 60

# set to 1 if you want each client to bind once as Directory Manager
# set to 0 if you want to bind before each operation
# Example :BINDONCE:0
BINDONCE:0

# The basename of the client scripts. The first thread will use
# the script named basename0, the second thread will use the script named
# basename1, and so on...
# Unix Example: CLIENTBASESCRIPTS:Scripts/address.client0.script
# Windows NT Example: CLIENTBASESCRIPTS:Scripts\messaging.client0.script
#
# If you want to run multiple DirectoryMark client processes, then
# you'll need to generate enough scripts (see the "-c" option of
# scriptgen) and specify the client script bases here. The first
# process # will use the first basename specified, the second process
# will take # the second basename specified, and so on. For example:
# CLIENTBASESCRIPTS:Scripts/try.client0.script:Scripts/try.client1.script
#
CLIENTBASESCRIPTS:Scripts/try.client0.script

# This is the root DN, and root password.
# Example:BINDDN:"cn=Directory Manager, o=DirMark Inc."
# Example:BINDPW:unrestricted
BINDDN:"cn=Administrator, dc=foo ,dc=com"
BINDPW:password

# Where the statistics and html files will go when we're done
# Unix Example: STATDIR:Results/
# Windows NT Example: STATDIR:Results\
STATDIR:Results/

# The name of the server software.
# This is only used for reporting results.
# Example: SERVERSOFTWARE:"Netscape Communications Directory 3.0"
SERVERSOFTWARE:Directory Server

# The hardware the server is running on.
# This is only used for reporting results.
# Example:MACHINETYPE:"Sun UltraSparc1"
MACHINETYPE:VAX 1170

# The index settings on the server.
# This is only used for reporting results.
# Example:INDEXSETTINGS:"default"
INDEXSETTINGS:"default"

# The name of the ldif file of the database
# This is only used for reporting results.
# Example:LDIFFILE:ldapstone10000.ldif
#LDIFFILE:airius10k.ldif

# The number of entries in the database
# This is only used for reporting results.
# Example:DBSIZE:10000 Entries
DBSIZE:10000

# The cache settings for the server
# This is only used for reporting results.
# Example: CACHESETTINGS:default
CACHESETTINGS:cachesize default, dbcachesize default

# The maximum run time in seconds.
# If you use -1 then each script file will be run exactly once.
# Otherwise if time remains when a client thread gets to the end
# of its script file then it will start over from the beginning.
# Example: MAXRUNTIME:600
# Example: MAXRUNTIME:-1
MAXRUNTIME:600

# Use Microsoft DAPI (works only on NT)
# Example:USEDAPI:0
USEDAPI:0

# Exchange specific stuff (ignore if not testing Exchange)
# Example: EXCHANGEBASEPOINT:"/o=DirMark Inc./ou=DirMark Inc."
EXCHANGEBASEPOINT:"/o=DirMark Inc./ou=DirMark Inc."

# More Exchange specific stuff (ignore if not testing Exchange)
# Example:EXCHANGECONTAINER:"Recipients"
EXCHANGECONTAINER:"Recipients"

# ore Exchange specific stuff (ignore if not testing Exchange)
# Example:EXCHANGECONTAINER:"Recipients"
# EXCHANGESITEOU:"/ou=DirMark Inc."
EXCHANGESITEOU:"/ou=DirMark Inc."

#Other options:
#
# Uncomment VERBOSE: if you want the clients to print the LDAP operation
# and the time it took to the screen for each operation
#
# Uncomment DEBUG: if you want all the VERBOSE: output plus the
# arguments to each LDAP function
#
#DEBUG:1
#VERBOSE: 1

When DirectoryMark is finished, it creates an html file that contains a summary of the test run. This file is called Masterstat.html and is located in the directory that corresponds to the number of client threads used. The summary lists the number and type of operations performed, and the average times it took for each type of operation. The whole test run is summarized in the DirectoryMark rating, which is in terms of operations/second.

The Object Class

DirectoryMark supports the standard organizationalPerson object class. See IETF RFC 2256 for a description of this object class. DirectoryMark uses and generates data for the following attributes of organizationalPerson:

cn
sn
description
facsimileTelephoneNumber
l
ou
postalAddress
telephoneNumber
title
userPassword
seeAlso

UID and givenName

Unfortunately, the organizationalPerson class doesn't include uid (user ID) or givenName (a person's first name) attributes. We feel these attributes are important and necessary for our proposed Address and Messaging test scenarios. Rather than creating our own new object class we have decided to use the organizationalPerson object class and use the seeAlso and description attributes to contain the givenName and UID values, respectively.

For example, in the sample LDIF entry above ("Kristi Vaserfirer") you can see that the value of the description attribute is the givenName which is "Kristi". Similarly, the seeAlso attribute has Krist's UID which is "Kristi_Vaserfirer".

The -G and -U options of scriptgen and dbgen can be used to override using the seeAlso and description attributes to hold the givenName and UID values. For example, if your object class has an attribute called firstName then you can invoke the dbgen tool with:

-G FirstName

and dbgen will put the givenName values into the FirstName attributes. You can then call scriptgen with the same option and it will create scripts that search on the firstName attribute rather than seeAlso.

If you are using the default values, that is seeAlso and description, then you should enable your directory server to index those values. This will yield improved performance.

inetOrgPerson

DirectoryMark also supports the proposed inetOrgPerson object class (IETF draft "draft-smith-ldap-inetorgperson-00"). To use inetOrgPerson, just use the -i option when invoking the dbgen and scriptgen tools. DirectoryMark uses the following attributes of inetOrgPerson:

cn
sn
description
facsimileTelephoneNumber
l
ou
postalAddress
telephoneNumber
title
userPassword
uid
givenName
mail
carLicense
departmentNumber
employeeType
homePhone
initials
mobile
pager
manager
secretary
roomNumber

Running DirectoryMark on University of Michigan LDAP-3.3

Because the University of Michigan LDAP-3.3 server does not support ACL settings, the following lines should be included in the etc/slapd.conf file:

default access read
access to * by self write

These settings will allow any client to read the database, but a user may only modify his or her own attributes after authenticating.

Architecture

The DirectoryMark performance benchmark consists of three components:

DirectoryMark clients: perform operations against the server
DirectoryMark master: Runs benchmark monitoring clients
DirectoryMark tools: database generator, client script generator, DirectoryMark configuration Wizard

The DirectoryMark Master controls the DirectoryMark clients. The DirectoryMark Master forks its clients and passes them the name of the DirectoryMark Transaction script each should run. The clients perform the operations specified in their Transaction Plan and report performance metrics back to the DirectoryMark Master when finished. Each client can use a different Transaction script to simulate real life load situations. Each client can also be configured by the DirectoryMark Master to wait a certain amount of time between transactions.

Transaction Scripts

Specification: Transaction scripts are a flat file that specify the operations the clients are to perform on the database. The transaction scripts are generated by the DirectoryMark script generation tool before the clients are started. Each client will be passed the pathname of its transaction plan by the DirectoryMark master. The following sections detail the layout of the flat file for each transaction type.

Search

A search transaction requires the following parameters be specified in the script file.

ldap_search

This is the command that tells the DirectoryMark client to perform a search operation. base

Distinguished name (DN) of the entry that serves as the starting point for the search. For example, setting base to "o=Airius.com", c=US" restricts the search to entries at Airius.com in the United States. The scriptgen tool will use the suffix specified in the LDIF file (or via its "-x" command-line option) as the base. scope

LDAP_SCOPE_BASE -- searches the entry specified by base.

LDAP_SCOPE_ONELEVEL -- searches all entries one level beneath the entry specified by base.

LDAP_SCOPE_SUBTREE -- searches the entry specified by base and all entries at all levels beneath the entry specified by base.

filter

(attributetype=attributevalue)

attrs

A comma-separated list of attribute types to return from entries that match filter. If you specify "ALL" then all attributes will be returned. attrsonly

0 -- (zero) specifies that both attribute types and attribute values are returned.

1 -- (one) specifies that only attribute types are returned.

Each entry is specified on a separate line in the script file with new lines between searches.
Example:

bind_as_root

ldap_search

o=DirMark Inc., c=US

LDAP_SCOPE_SUBTREE

(cn=Allen Tom)

ALL

0

Add

Add requires the following parameters in the flat file:

ldap_add

This is the command that tells the DirectoryMark client to perform an add operation. dn

Distinguished name (DN) of the entry to add. With the exception of the leftmost component, all components of the distinguished name (for example, o=organization or c=country) must already exist. attrs

dn: `dn`

attr1:`values one`

attr2: `value two`

Example:

bind_as_root

ldap_add

dn: `cn=Steve Ross, ou=Human Resources, o=Airius.com`

objectclass: `top` `organizationalperson` `inetorgperson`

sn: `Ross`

cn: `Steve Ross`

Delete

The delete operation requires the following parameters specified in the script file:

ldap_delete

This is the command that tells the DirectoryMark client to perform a delete operation. dn

Each entry is specified on a separate line in the script file with new lines between searches
Example:

bind_as_root

ldap_delete

"cn=Steve Ross, ou=Engineering, o=Netscape, c=US"

Modify

The modify operation requires the following attributes to be specified in the script file:

ldap_modify

This is the command that tells the DirectoryMark client to perform a modify operation. dn

Distinguished name (DN) of the entry to modify. changetype: modify

This line is required to appear exactly like this. mods

The attribute to modify and value to place in that attribute. Use a line containing only a dash character ("-") to specify more than one modification. Each entry is specified on a separate line in the script file with new lines between modification commands.

bind_instructions

ldap_modify

changetype: modify

mod_op

mod_values

i.e:

bind_as_root

ldap_modify

cn=Allen Tom, ou=Product Development, o=Airius.com

changetype: modify

replace: title

title: DirMark Czar

Modrdn

Modifying the rdn requires the following parameters in the script file:

The distinguished name of the entry that you want to change. newrdn

The new RDN for the entry that you want to modify. Note that this is an RDN, not a DN so you only specify the CN portion of the DN. The RDNs in the scripts created by the Directorymark scriptgen tool are just the old CN with the word "ldap_modrdn" prepended. deleteoldrdn

Specifies whether or not the existing RDN should be deleted. To delete the existing RDN, set this argument to 1. To leave the existing RDN as an attribute of the entry, set this argument to 0 Example:

bind_root

modrdn

cn=Allen Tom, ou = Engineering, o=Airius.com

cn=Allen Lee Tom

1

Compare

The compare operation requires the following parameters be specified in the script file:

Distinguished name (DN) of the entry used in the comparison. attr

Attribute used in the comparison. value

Value to compare against the entry. Example:

ldap_compare:

"cn=Allen Tom, ou=Engineering, o=Netscape, c=US"

"mail"

"atom@netscape.com"

LDAP Master

Responsibilities:

Initial Configuration,
Managing Clients,
Collecting/Reporting Client Results

LDAP Clients

Responsibilities:

Perform Transactions against server,
report results

DirectoryMark Tools

DirectoryMark comes with the following tools which may be useful in creating new tests.

dbgen.pl

Dbgen can be used to create databases in LDIF format of arbitrary size. The resulting database will contain inetOrgPersons with unique, realistic-looking names. scriptgen.pl

Scriptgen is used to create client scripts for DirectoryMark clients. The ratio of each operation (add, delete, modify, search, compare, modrdn) can be specified on the command line, along with a random seed to use. Scriptgen takes the LDIF file created by dbgen (or any LDIF file) and creates operations based on the data in the database.

Running Scriptgen

The scriptgen tool is used to generate scripts that the client will run against the Directory Server. The usage is:

perl scriptgen.pl [options] output_name ldif_file number_of_transactions

A typical usage is:

perl scriptgen.pl -v -i -t 4 -S messaging ..\scripts\scr ..\ldif\db.ldif 10000

scriptgen Option	Explanation
-A attribute list	Comma-seperated list of attributes to return for searches. Default is all attributes.
-a ratio	Ratio of ldap_add commands in the generated script.
-b	Bind as a random person. The default is to bind anonymously.
-B number	The number of operations to do per bind command. For example, -B 6 will do a bind followed by six LDAP operations (search, add, delete, etc), another bind followed by six more operations, etc. Use -1 (negative 1) to do a single bind before all commands.
-c ratio	Ratio of ldap_compare commands in the generated script.
-C ratio	Ratio of "Netscape Communicator" searches in the generated script.
-d ratio	Ratio of ldap_delete commands in the generated script.
-g ratio	Ratio of exact searches on the gn attribute.
-G attribute	Use this attribute to hold the givenName value.
-i	New adds will follow the inetOrgPerson object class. The default is organizationalPerson.
-m ratio	Ratio of ldap_modify commands in the generated script.
-n ratio	Ratio of exact searches on the cn attribute.
-O	Turn on the attributes-only flag for searches.
-p number	Number of processes. The default is 1.
-r ratio	Ratio of ldap_modrdn commands in the generated script.
-s ratio	Ratio of exact searches on the sn attribute.
-S address or -S messaging	Generate scripts that follow the "address" or "messaging" scenarios described in this paper. If this option is used then the -B, -c, -C, -d, -g, -m, -n, -r, -s, -u, and -w options are ignored.
-t number	Number of threads that will be used to run commands against the Directory Server. scriptgen will generate a script for each thread.
-u ratio	Ratio of exact searches on the uid attribute.
-U attribute	Use this attribute to hold the UID value
-v	Turn on verbose output.
-w ratio	Ratio of wildcard (eg, containing "*") searches on the cn attribute.
-W	Use this option if you are testing Microsoft's Active Directory..
-x ratio	Ratio of searches that won't match any entries in the database.
-z number	Seed for the random number generator.
-Z scope	Set the search scope. Acceptable values are LDAP_SCOPE_SUBTREE , LDAP_SCOPE_BASE, and LDAP_SCOPE_ONELEVE

™ - DirectoryMark is a trademark of Mindcraft, Inc. and Netscape Communications Corporation.