File Server Comparison:

Microsoft Windows NT Server 4.0
on a Compaq ProLiant 3000
and
Sun Solaris 2.6 and Syntax TotalNET 5.2
on a Sun Ultra Enterprise 450

Contents

Executive Summary
Performance Analysis
How NetBench 5.01 Works
What Are the Bottlenecks?
Price/Performance
Products Tested
Test Lab
Mindcraft Certification
Overall NetBench Results

Executive Summary

Windows NT Server 4.0 Has 4.2 Times Faster File-Server Performance and 11.2 Times Better Price/Performance Than Solaris 2.6 with TotalNET 5.2

Mindcraft tested the file-server performance of Microsoft Windows NT Server 4.0 on a Compaq ProLiant 3000 and SunSoft Solaris 2.6 on a Sun Ultra Enterprise 450. We tested the Server Message Block (SMB) file-sharing protocol on top of TCP/IP for both servers. The Sun server ran TotalNET 5.2 from Syntax to provide the SMB protocol. We used TotalNET because Sun ships an evaluation copy with its Ultra Enterprise 450. Table 1 shows the peak throughput measured for each system in megabytes per second (MB/S), the price of the systems tested, and the price/performance in dollars per MB/S.

Table 1: Result Summary
(larger numbers are better for Peak Throughput, smaller numbers are better for the others)

Mindcraft tested these file servers with the Ziff-Davis Benchmark Operation NetBench 5.01 benchmark using its standard disk mix test. The price/performance calculations are presented in the Price/Performance section.

The benchmark results show that peak file-server performance of Microsoft Windows NT Server 4.0 on a Compaq ProLiant 3000 is over 4.2 times that of Solaris 2.6 and TotalNET 5.2 on a Sun Ultra Enterprise 450. The Microsoft solution also offers 11.2 times better price/performance than the Sun system, making it a much more cost-effective solution.

Performance Analysis

Looking at the Results

The NetBench 5.01 benchmark measures file server performance. Its primary performance metric is throughput in bytes per second. The NetBench documentation defines throughput as "The number of bytes a client transferred to and from the server each second. NetBench measures throughput by dividing the number of bytes moved by the amount of time it took to move them. NetBench reports throughput as bytes per second." We report throughput in megabytes per second to make the charts easier to read.

We tested the Server Message Block (SMB) file sharing protocol on both the Windows NT Server and Solaris-TotalNET platforms using the standard NetBench NBDM_60.TST test suite. Figure 1 shows the throughput we measured plotted against the number of test systems that participated in each data point. Note, while the test suite attempted to use the same number of test systems for a given data point for both servers, the Sun server could only support up to 28 test systems; the rest stopped participating in the test because of file-sharing errors.

Figure 1: SMB File Server Throughput Performance (larger numbers are better)

How NetBench 5.01 Works

You need to know how NetBench 5.01 works in order to understand what the NetBench throughput measurement means. NetBench is designed to stress a file server by using a number of test systems to read and write files on it. Specifically, a NetBench test suite is made up of a number of mixes. A mix is a particular configuration of NetBench parameters, including the number of test systems used to load the server. Typically, each mix increases the load on the server by increasing the number of test systems involved while keeping the rest of the parameters the same. The NBDM_60.TST test suite we used has the parameters shown in Table 2.

Table 2: NBDM_60.TST Parameters

NetBench does a good job of testing a file server under heavy load. To do this, each NetBench test system (called a client in the NetBench documentation) executes a script that specifies a file access pattern. As the number of test systems is increased, the load on a server is increased. You need to be careful, however, not to correlate the number of NetBench test systems participating in a test mix with the number of simultaneous users that a file server can support. This is because each NetBench test system represents more of a load than a single user would generate. NetBench was designed to behave this way in order to do benchmarking with as few test systems as possible while still generating large enough loads on a server to saturate it.

When comparing NetBench results, be sure to look at the configurations of the test systems because they have a significant effect on the measurements that NetBench makes. For example, the test system operating system may cache some or all of the workspace in its own RAM causing the NetBench test program not to go over the network to the file server as frequently as expected. This can significantly increase the reported throughput. In some cases, we’ve seen reported results that are 75% above the available network bandwidth. If the same test systems and network components are used to test multiple servers with the same test suite configuration, you can make a fair comparison of the servers.

With this background, let us look at the results in Figure 1 (the supporting details for this chart are in Overall NetBench Results). It is obvious that the throughput of the Compaq ProLiant 3000 was significantly higher than that of the Sun Ultra Enterprise 450 when four or more test systems were used. The ProLiant 3000 had over 4.2 times the throughput of the Ultra Enterprise 450 at each server’s peak performance.

For the Windows NT Server/ProLiant platform, all of the test systems specified for each data point participated. But the Solaris/Ultra Enterprise platform had test systems stop participating in each data point after 20 systems and it never had more than 28 test systems participating in any mix. This means that under moderate load people are likely to see a significant number of errors if they use SMB to access files stored a Solaris/Ultra Enterprise platform. These errors could result in lost data.

What Are the Bottlenecks?

The readily measured factors that limit performance of a file server are:

1. The server’s CPU and memory;
2. The disk subsystem;
3. The network; and
4. The operating system and file server software.

We’ll examine each factor individually.

Performance Monitoring Tools

We ran the standard Windows NT performance-monitoring tool, perfmon, on the ProLiant 3000 during the tests to gather performance statistics. Perfmon allows you to select which performance statistics you want to monitor and lets you see them in a real-time chart as well as save them in a log file for later analysis. We logged the processor, memory, network interface, and disk subsystem performance counters for these tests.

To collect performance data on the Ultra Enterprise 450 during the test, we ran vmstat for memory-related statistics and mpstat for processor-related statistics. These programs output a fixed set of performance statistics that can be displayed or saved in a file.

Server CPU Performance

Each of the ProLiant 3000’s CPUs was 45% utilized at peak performance. NetBench performance was not CPU-limited on the ProLiant 3000.

At peak performance, the Ultra Enterprise 450 CPUs spent 29% of their time in system and user time. In addition, they were waiting 70% of the time and idle only 1%. The Ultra Enterprise 450 CPUs were 99% utilized at peak performance. The large wait time indicates that something other than processing was causing the high CPU utilization. We’ll look at other factors below to understand the CPU bottleneck.

Memory was not a performance limitation for either system as shown by monitoring programs. Throughout the test the ProLiant 3000 had less than 30 MB committed for all uses (except for file system cache) out of the 512 MB on the system. The Ultra Enterprise 450 used less than 256 MB of memory. Other factors contributed to limit the performance of both systems.

Disk Subsystem Performance

Both systems were configured with the operating system and paging/swap space on one disk and the NetBench data on another. The third disk in each system was not used for these tests because NetBench does not readily support splitting its test data across multiple disks.

The perfmon "% Disk Time" counter shows the percentage of elapsed time that the selected disk drive is busy servicing read or write requests. Table 3 shows the % Disk Time information for the two disks involved in NetBench testing. The high disk utilization on the D: drive clearly indicates that NetBench performance was disk-limited on the ProLiant 3000.

Table 3: Windows NT Disk Utilization

The Ultra Enterprise 450 disk with the operating system and swap partitions averaged 4 operations per second. The disk with the NetBench data averaged 115 operations per second from the mix on. It would not go above this average even though the test system load increased. So the disk subsystem was a performance-limiting factor for the Ultra Enterprise 450 as well.

We believe that had we used a hardware-based RAID to store the NetBench data, the performance of the ProLiant 3000 would have been significantly higher because the effective disk access time would have been lower. We did not use a hardware RAID because we wanted to have a fair comparison with the Sun system and there was no hardware-based RAID available for it.

Network Performance

The network did not limit the performance of the ProLiant 3000. At peak performance, NetBench throughput was 64 Mbits/second. This represents heavy use of a single 100Base-TX network. Because the ProLiant 3000 had two such networks, there was sufficient bandwidth available to support higher throughput.

Similarly, the network did not limit the performance of the Ultra Enterprise 450 because its peak throughput was only 14.9 Mbits/second on both of its networks.

Operating System and File Server Software Performance

SMB file sharing is integrated into the core of Windows NT Server 4.0. So how much time the CPUs are in privileged mode is a good indicator of how much time is spent on file sharing support. During all mixes, privileged mode execution time accounted for 100% of the CPU utilization. Because the CPUs were only 45% utilized, the operating system and SMB file sharing were not limiting the performance of the ProLiant 3000.

The Ultra Enterprise 450 provides SMB file sharing via TotalNET. By analyzing CPU performance during the test, we can understand why TotalNET performed poorly. From the peak performance mix onward, the CPUs did 1,500 to 3,100 context switches per second and had to process 10,000 to 30,000 system calls per second. These are high figures and occur in large part because TotalNET is not integrated with Solaris 2.6. The extra work that Solaris needs to do to support TotalNET is an essential part of why the CPUs were idle less than 3% of the time after the second mix. Thus, both Solaris 2.6 and TotalNET combined to limit the performance of the Ultra Enterprise 450.

Conclusion

Windows NT Server 4.0 on a Compaq ProLiant 3000 offers high-performance file sharing. Comparably configured, its performance is over 4.2 times that of Solaris 2.6 with TotalNET on a Sun Ultra Enterprise 450. The price/performance of a Windows NT Server/ProLiant 3000 platform is 11.2 times better than a Solaris/TotalNET/Ultra Enterprise 450 platform.

Solaris 2.6 on an Ultra Enterprise 450 is performance-limited for SMB file sharing by a combination of its disk subsystem, operating system, and TotalNET. Solaris 2.6 on an Ultra Enterprise 450 with TotalNET is not appropriate for high-volume SMB file sharing.

Price/Performance

We calculated price/performance by dividing the street price of the servers and software tested by the peak throughput measured in megabytes per second. We obtained the street price of the ProLiant 3000 shown in Table 4 by requesting a quote from a Compaq value-added reseller (VAR). Likewise, the street price of the Ultra Enterprise 450 was obtained from a Sun VAR quote; we added to that the cost of the TotalNET software for 60 users based on a quote we received from Syntax, Inc.

Table 4: Compaq ProLiant 3000 Pricing

Table 5: Sun Ultra Enterprise 450 Pricing

Products Tested

Configurations and Tuning

We configured the Compaq and Sun hardware comparably, and tuned the software on each system to maximize performance. Table 6 shows the configuration of the Compaq ProLiant 3000 we tested. Table 7 describes the Sun Ultra Enterprise 450 configuration we used, including the TotalNET configuration.

Table 6: Compaq ProLiant 3000 Configuration

Table 7: Sun Ultra Enterprise 450 Configuration

Test Lab

The Test Systems and Network Configuration

Mindcraft ran these tests using a total of 60 test systems configured as shown in Table 8.

Table 8: Test System Configurations

We balanced the networks by grouping the test systems so that one system on each hub would be added for each mix after the second mix, which uses four test systems. Figure 2 shows the test lab configuration.

Figure 2: Test Lab Configuration

Mindcraft Certification

Mindcraft, Inc. conducted the performance tests described in this report between April 14 and May 1, 1998. Mindcraft used the NetBench 5.01 benchmark to measure performance with the standard NetBench NBDM_60.TST test suite.

Mindcraft certifies that the results reported herein represent the performance of Microsoft Windows NT Server 4.0 on a Compaq ProLiant 3000 as measured by NetBench 5.01. Mindcraft also certifies that the results reported herein represent the performance of Sun Solaris 2.6 and Syntax TotalNET on a Sun Ultra Enterprise 450 as measured by NetBench 5.01.

Our test results should be reproducible by others who use the same test lab configuration as well as the computer and software configurations and modifications documented in this report.

Overall NetBench Results

Compaq ProLiant 3000 Detailed Benchmark Results

Sun Ultra Enterprise 450 Detailed Benchmark Results

NOTICE:

Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.