Summary Rebuttal to Novell's Criticism
of
Mindcraft's File Server
Comparison
By Bruce Weiner
October 6, 1998
Updated: October 7, 1998
(The Detailed Rebuttal and PDF version of this Summary Rebuttal)
To keep this summary short, we have links to the relevant parts of our detailed rebuttal. We strongly suggest that you look at the details to understand any claims we make and for more explanations. The structure of this summary is based on that of Novell's criticism.
Dubious Origin
Mindcraft declined to answer who sponsored the testing for our report because our client had not authorize us to disclose their name. We have now been authorized to disclose that Mindcraft was commissioned by Microsoft Corporation to produce an independent and unbiased assessment of the file server performance of Windows NT Server 4.0 and NetWare 5 using NetBench 5.01. Contrary to Novell's implication, who our client is makes no difference as to how we run a test. I'd be willing to bet that Novell thinks that the BorderManager report they hired us to produce is fair, accurate and unbiased.
Unprofessional Methods
(1) Standard Benchmarking Practices Were Violated. We want you to know that had we used the standard NetBench NBDM_60.TST test suite, we never would have seen the peak performance of NetWare 5 on the server we tested!
(2) The benchmark test parameters used were known to give faulty results. Mindcraft had several discussions with two Novell technical experts about the best way to configure NetWare 5 to maximize its performance. Both had helped PC Magazine for their First Look at NetWare 5. We used all of the NetWare 5 server and client tuning parameters they recommended, except for opportunistic locking. Mindcraft used iterations for test duration rather than time because it allows for fair comparisons between different servers.
(3) The server hardware configuration was contrived to put NT in it's best light. Novell clearly misunderstood our discussion about tuning the NetFlex 3 Network Interface Card. We have never seen a "blue screen" on Windows NT Server 4.0 when using NetBench despite Novell's implication. Novell's contention that we fixed the disk configuration to favor Windows NT Server 4.0 over NetWare 5 is totally without merit.
(4) Major operational characteristics were applied unevenly. Novell's discussion of opportunistic locking is a very clever act of misdirection. We used each vendor's default settings for opportunistic locking because most users would go with a vendor's recommendation. This was contrary to what Novell and Microsoft wanted us to do, which was to use special settings on the clients that are not enabled by default and that would require setting registry keys on every client system connected to the server. We thought that would mislead our readers, so we didn't do it.
Obviously Biased Results
The numbering continues as it was in Novell's criticism.
(1) The benchmark results are contrary to other published results. Why does the First Look article show that Windows NT Server 4.0 outperforms NetWare 4.11 when tested with NetBench?
(2) The Price/Performance metric cited is extremely misleading.
A price/performance metric is affected by both of its components. We fully disclosed that we used pricing for both operating systems based on a quotation from a VAR who we asked to give us prices for the same discount level and licensing circumstances. As we show in the detailed rebuttal, our measurements were unbiased.
Comments on Novell's Benchmark Tests
The NetBench performance results Novell published for a single- and dual-processor system with opportunistic locking show very little scaling, contrary to Novell's claim, and do not meet the NetBench disclosure requirements.
Summary
The attacks and criticisms that Novell made are unfounded. Mindcraft also would welcome the opportunity to work with Novell to resolve any of the issues raised by our testing and to produce another honest and impartial product comparison.
NOTICE:
The information in this publication is subject to change without notice.
MINDCRAFT, INC. SHALL NOT BE LIABLE FOR ERRORS OR OMISSIONS CONTAINED HEREIN, NOR FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES RESULTING FROM THE FURNISHING, PERFORMANCE, OR USE OF THIS MATERIAL.
This publication does not constitute an endorsement of the product or products that were tested. This test is not a determination of product quality or correctness, nor does it ensure compliance with any federal, state or local requirements.
The Mindcraft tests discussed herein were performed without independent verification by Ziff-Davis and Ziff-Davis makes no representations or warranties as to the results of the tests.
Mindcraft is a registered trademark of Mindcraft, Inc.
Product and corporate names mentioned herein are trademarks and/or registered trademarks of their respective companies.
