Transcript
December 2007
www.veritest.com •
[email protected]
Built In EV-DO Comparison for Dell, Lenovo and HP
Test report prepared under contract from Dell Inc.
Executive Summary
Dell Inc. commissioned VeriTest, the testing service of Lionbridge Technologies Inc., to conduct a competitive analysis comparing the performance of built-in EV-DO (Rev-A) mobile broadband modules using Verizon Wireless BroadbandAccess. VeriTest conducted a series of tests to find which notebook had the best performance in terms of overall data throughput speed and performance. The notebooks tested in this study include the following Business Class models. • Business Class o Dell Latitude D630 o HP 6910p o Lenovo T61
Key Findings
Average Download Speed
The Dell Latitude’s EV-DO Rev-A module is manufactured by Novatel. The HP and Lenovo EV-DO Rev-A modules Chart 1. Average Kbps for FTP and Web Downloading. are made by Sierra Wireless. Each of the notebooks tested had comparable On average, the Dell D630 download speeds were 36% configurations (e.g., processor speed, faster than the HP 6910p and 88% faster than the memory, and system OS). All notebooks Lenovo ThinkPad T61. included Intel Core 2 Duo processors as well as 2 GB (2 x 1GB) of random access memory (RAM). For a list of complete specifications for each notebook, please refer to Appendix A. In order to best replicate the end user experience, each notebook’s operating system type was installed according to what is used the most from a business perspective. Business class models included Microsoft Windows XP Professional with Service Pack 2.
Integrated EV-DO Comparison for Dell, Lenovo and HP
1
Tests were conducted in the following US cities over a 29 day period using Verizon Wireless BroadbandAccess. • • • San Diego, California New York, New York Chicago, Illinois
Three test locations were used in each city. All notebooks were tested on the same day for each location. Each location was tested on separate days to account for network variability. The final result for each test was calculated by averaging the encountered results from multiple test iterations for each location. Six (6) to ten (10) iterations were conducted for each test. The following tests were conducted: A. Web Downloading Test - to measure throughput when downloading 1MB and 2MB web files via HTTP protocol. B. FTP Send and Receive Test - to measure throughput when transferring 1MB and 2MB files to and from web servers via FTP protocol Upon completion, the study findings show that on average, the Dell Latitude D630 provided 36% more throughput than the HP 6910p and 88% more throughput than the Lenovo ThinkPad T61 for download speed. The Dell Latitude D630 had an average throughput speed averaged across the FTP and Web Downloading tests of 1023 Kbps. The HP 6910p had an average throughput of 753 Kbps and the Lenovo T61 had an average throughput of 545 Kbps. Please refer to the Test Methodology section of this report for complete details of how testing was conducted. All units were tested using default factory settings. Lionbridge confirmed that all notebook computers were new and unopened prior to testing and ensured all tests were completed with impartiality.
Test Results
The following section details the results from testing.
Web Download Throughput
Goal: Benchmark throughput when performing web downloads via HTTP using a low traffic server hosted by a local internet service provider. Summary of Results: The Dell Latitude D630 was on average 43% faster than the HP 6910p and 74% faster than the Lenovo T61 when downloading a 1MB zip file from the web. The Dell D630 was on average 27% faster than the HP 6910p and 80% faster than the Lenovo T61 when downloading a 2MB zip file from the web.
Integrated EV-DO Comparison for Dell, Lenovo and HP
2
Web Download Comparison
1100 1000 900 Kilobits Per Second 800 700 600 500 400 300 200 100 0 1MB 2MB
Dell HP Lenovo
Chart 2: Average Kbps for all test run averages for the Web Downloading Test.
Notebook DELL D630 HP 6910p LENOVO T61
1MB(Kbps)
2MB(Kbps)
1012.66 707.29 582.48
989.18 780.47 550.79
Table A. Average Throughput of Web Downloading Test
FTP Test
Goal: Benchmark throughput when using FTP to transfer files. Summary of Results: The Dell D630 was on average 43% faster than the HP 6910p and 94% faster than the Lenovo T61 when downloading a 1 MB zip file via FTP. The Dell D630 was on average 32% faster than the HP 6910p and 106% faster than the Lenovo T61 when downloading a 2 MB zip file via FTP. Upload results displayed minimal variance in performance. The Dell D630 was on average 10% faster than the HP 6910p and 5% faster than the Lenovo T61 when uploading a 1 MB zip file via FTP. The Dell D630 was on average 13% faster than the HP 6910p and 3% faster than the Lenovo T61 when uploading a 2 MB zip file via FTP.
Integrated EV-DO Comparison for Dell, Lenovo and HP
3
FTP Throughput Comparison
1100 1000 900 Kilobits Per Second 800 700 600 500 400 300 200 100 0 1MB snd 2MB snd 1MB rcv 2MB rcv
Dell HP Lenovo
Chart 3: Average throughput for all test run averages for the FTP Test.
Notebook DELL D630 HP 6910p LENOVO T61
1MB(Kbps)
2MB(Kbps)
1066.27 746.47 549.36
1MB(Kbps)
1024.86 776.21 497.78
2MB(Kbps)
Table B. Average Download Speed of FTP Test
Notebook DELL D630 HP 6910p LENOVO T61
631.13 571.36 601.45
623.39 549.68 602.39
Table C. Average Upload Speed of FTP Test
Test Methodology
Dell Inc. commissioned VeriTest to conduct a competitive analysis comparing performance of built-in EV-DO (Rev A) mobile broadband modules. VeriTest conducted a series of throughput tests to measure which notebook had the highest performance.
Notebook Configurations
Test conditions were designed for the notebooks to be as close to an out-of-the-box configuration as possible, but some third party software, such as firewalls and virus protection, had to be disabled. This ensured that some of this software did not affect download speed.
Integrated EV-DO Comparison for Dell, Lenovo and HP
4
The following software was required for testing which needed to be installed: o Wget for DOS (an application distributed under the GNU license) for the non-interactive Web Downloading Test.
Test Locations
The test locations included areas with low, marginal, and strong coverage. Locations used were deemed typical for users of built-in mobile broadband and included hotels and coffee shops. Some locations provided unreliable connections resulting in data with such large standard deviations as to be statistically useless. Data from these locations was not used in this report. Several test runs would take 8 hours or more requiring the availability of AC power to keep the notebooks charged, however, notebooks tested were not connected to AC power during testing.
Test Run Conditions
To keep the testing notebooks from possessing any advantages over each other based on location, orientation or configuration, the test design called for the following conditions to be met before testing could begin: o o o o o o Each computer must be placed in the same point of origin and facing the same direction Each LCD monitor must be tilted back at approximately 110 degrees Each computer must be running on battery only, to simulate a typical user experience Power settings must be set to Always On All brightness settings must be maximized Device radios are to be powered off unless when under test
There are many factors affecting actual throughput analysis (e.g. structures, latency, and network traffic). Because these factors were out of our control and testing was not done in a controlled lab the results may not be reproducible. The tests were designed to be objective by ensuring that each notebook was affected by the same or similar environmental factors.
Test Run Details
After test setup was complete and all test run conditions were met, all notebook tests were repeated from six (6) to ten (10) iterations. A completed iteration is defined as a single test executed once on all three notebooks. Interleaving the tests from notebook to notebook helped ensure that any network traffic or signal fluctuations would affect all notebooks equally minimizing factors that can cause network variance such as network traffic. Only one computer was connected to the Verizon Wireless BroadbandAccess service at a time. Only one notebook module was powered on at a time. The test run was complete when all iterations were complete for all tests.
Calculating Average
Once a test run was complete, the recorded data points for each test location were averaged. These results can be found in Appendix C: Detailed Results.
Test Details
The following describes each test in detail and includes information on test development, description of execution and data gathering.
Integrated EV-DO Comparison for Dell, Lenovo and HP
5
A. Web Downloading Test Test Description Wget for DOS was used to download a 1 megabyte and 2 megabyte zip file from a web site. Zip files were used to eliminate the possibility of file transfer compression affecting test results. Test Development The Wget application was installed on each notebook. A batch file executed the Wget command with the logging option to download the two zip files and log the results. The zip files were located on a low traffic web server from a local ISP. Test Execution and Data Gathering Double clicking the batch file would run the test. A log file was created on the desktop upon completion. The start and stop time data as well as the Kb/s from each download were recorded into an Excel spreadsheet to be averaged. B. FTP Test Test Description FTP is one of the most common methods for transferring large files over the internet. To test throughput speeds via FTP, Microsoft XP’s integrated FTP client was used to download and upload a 1 megabyte zip file and a 2 megabyte zip file. Zip files were used to eliminate the possibility of file transfer compression affecting test results. Test Development A batch file was written to log onto an FTP server and run put/get commands which would send and receive the files via FTP then log the results to a log file on the Desktop. The FTP server used for this test was provided by a low traffic local ISP. Test Execution and Data Gathering Double clicking the batch file would run the test. Upon completion a log file was created on the desktop. The time in seconds it took for each download and upload to complete as well as the Kb/s were copied into an Excel document in order to be averaged.
Integrated EV-DO Comparison for Dell, Lenovo and HP
6
Appendix A: System Configurations
System Information Dell Latitude D630 Processor/speed Intel® Core™ 2 Duo 2.00GHz System Ram/type/number of slots 2 GB (2x1GB) Mother board manufacturer Dell Latitude D630 Built-in EV-DO Rev A Module Manufacturer Novatel OEM OS Win XP + SP2
Table D. System configuration for Dell Latitude D630
System Information HP Compaq 6910p Processor/speed Intel® Core™ 2 Duo 1.80GHz System Ram/type/number of slots 2 GB (2x1GB) Mother board manufacturer HP Compaq 6910p Built-in EV-DO Rev A Module Manufacturer Sierra Wireless OEM OS Win XP + SP2
Table E. System configuration for HP Compaq 6910p
System Information Lenovo ThinkPad T61 Processor/speed Intel® Core™ 2 Duo 2.00GHz System Ram/type/number of slots 2 GB (2x1GB) Mother board manufacturer Lenovo ThinkPad T61 Built-in EV-DO Rev A Module Manufacturer Sierra Wireless OEM OS Win XP + SP2
Table F. System configuration for Lenovo ThinkPad T61
Appendix B: Testing Software Information
Software Microsoft File Transfer Program Version 5.1.2600.2180 Associated Test FTP Test Description This is an application integrated into Windows XP and was used to test throughput via FTP protocol. This is an application which is distributed under the GNU General Public license and was used to retrieve files (non interactive) via HTTP protocol.
Wget
1.8.2
Web Downloading Test
Table G. Testing software information
Integrated EV-DO Comparison for Dell, Lenovo and HP
7
Appendix C: Detailed Results
The following section details the results from each test run. All numbers contained herein are averages from all the test iterations executed during that run as described in the Test Methodology section. Please refer to that section for more details.
San Diego, CA – Test Location 1:
Testing Location: Tuesday, November 6, 2007 – Starbucks (Mission Beach) – San Diego, CA Address: 4343 Mission Blvd # 104, San Diego, CA 92109 Details: Testing took place inside on the southwest side of the lobby area. This area included a nearby power outlet on the wall. The nearest Verizon Wireless tower is approximately 1.5 miles north from this location. This location was observed as a low to moderate signal reception area.
HTTP Download DELL HP LENOVO
AVG STDEV AVG STDEV AVG STDEV 1MB Down 344.46 68.72 263.95 91.31 435.28 54.75 2MB Down 326.62 70.10 278.00 75.46 472.73 76.42 1MB Up 347.84 89.81 420.50 209.87 428.27 53.72 2MB Up 385.13 97.99 436.98 167.03 397.18 119.03
FTP Test
1MB Down 323.84 81.49 376.30 148.26 346.48 118.58 2MB Down 330.62 94.24 306.24 105.27 285.73 77.71
San Diego, CA – Test Location 2:
Testing Location: Thursday, November 8, 2007 – Archstone La Jolla Colony – San Diego, CA Address: 7205 Charmant Dr. San Diego, CA 92122 Details: Testing took place in a covered outdoor area with access to AC power. This is located in the southeast area of the complex. This location was deemed as a moderate to good signal coverage area.
HTTP Download DELL HP LENOVO
AVG STDEV AVG STDEV AVG STDEV 1MB Down 1427.52 332.10 976.45 288.86 136.87 136.87 2MB Down 1333.84 324.52 1143.38 233.16 883.42 86.45 1MB Up 658.61 44.76 578.63 53.61 582.68 26.24 2MB Up 653.16 30.67 642.08 22.69 629.12 48.97
FTP Test
1MB Down 1771.57 322.88 1346.77 611.42 975.55 136.00 2MB Down 1667.83 349.09 1514.65 450.60 882.99 107.59
Integrated EV-DO Comparison for Dell, Lenovo and HP
8
San Diego, CA – Test Location 3:
Testing Location: Saturday, November 10, 2007 – Westfield Shopping Town – San Diego, CA Address: 1640 Camino Del Rio N, San Diego, CA 92108 Details: Testing took place in the Westfield Mall in Mission Valley. This location is near the National University Marine Corps Recruit Depot and DeVry University. There is an in-building Verizon Wireless tower at this location. Notebooks were tested in the food court area of the mall with nearby access to AC power.
HTTP Download
1MB Down 2MB Down 746.74 159.17 393.45 82.00 603.27 106.55 1MB Up 564.87 62.17 519.53 171.89 456.27 130.64 2MB Up 567.12 66.65 415.81 138.27 414.37 121.14 AVG STDEV AVG STDEV AVG STDEV 840.49 192.40 481.37 124.68 547.27 132.03
FTP Test
1MB Down 1214.65 642.26 374.65 112.55 393.81 88.02 2MB Down 1074.07 649.68 449.80 128.28 348.11 72.73
DELL HP LENOVO
New York, NY – Test Location 1:
Testing Location: Wednesday, November 14, 2007 – Radisson Martinique Hotel – New York nd Address: 49 W.32 Street, New York, NY 10001 Details: Testing took place in the restaurant area of the hotel. This is on the first floor of the hotel (street level). The Radisson Hotel is approximately ¼ mile from the nearest Verizon Wireless tower, but there is a fair amount of structural interference as well as a high amount of network traffic. This location was deemed as a good coverage area based upon preliminary throughput tests.
HTTP Download
1MB Down 2MB Down 1432.54 136.05 780.89 97.59 462.20 24.63 1MB Up 711.12 98.96 550.29 113.41 740.67 30.16 2MB Up 670.11 88.48 535.67 155.01 761.16 22.88 AVG STDEV AVG STDEV AVG STDEV 1458.48 158.09 665.04 203.25 441.43 48.67
FTP Test
1MB Down 1262.25 93.93 767.00 186.36 487.35 38.10 2MB Down 1295.68 98.13 733.20 189.97 452.16 46.74
DELL HP LENOVO
Integrated EV-DO Comparison for Dell, Lenovo and HP
9
New York, NY – Test Location 2:
Testing Location: Friday, November 16, 2007 – Staten Island Ferry Station – New York Address: South Ferry – State Street & Whitehall Street, New York, NY Details: Testing took place in the main entrance area. No nearby AC power was available. This area is approximately 1/2 mile south of Wall Street as well as the Castle Clinton National Monument. This location was observed to be a moderate to good signal coverage area.
HTTP Download
1MB Down 2MB Down 709.07 54.57 627.23 70.52 469.89 50.72 1MB Up 791.07 9.18 660.57 70.72 713.45 70.69 2MB Up 792.79 23.62 675.19 25.61 752.29 46.45 AVG STDEV AVG STDEV AVG STDEV 863.52 114.48 617.80 149.79 458.72 56.43
FTP Test
1MB Down 983.31 163.33 646.03 177.53 562.65 33.62 2MB Down 840.55 64.65 638.35 46.80 468.00 25.60
DELL HP LENOVO
New York, NY – Test Location 3:
Testing Location: Sunday, November 18, 2007 – Starbucks – New York th Address: 378 6 Ave, New York, NY 10011 Details: Testing took place in the lobby area at the northwest corner of the building near the entrance. This location is next to New York University and Washington Square Park. The nearest Verizon Wireless tower is approximately 1 ¼ mile west of the location. This location was observed as a low to moderate signal coverage area with average network traffic.
HTTP Download
1MB Down 2MB Down 1031.90 146.48 607.28 91.21 510.27 45.61 1MB Up 416.32 66.21 301.09 149.78 302.61 106.79 2MB Up 426.20 69.70 316.08 140.30 324.88 94.66 AVG STDEV AVG STDEV AVG STDEV 1135.12 202.93 559.24 150.86 518.20 44.55
FTP Test
1MB Down 1122.55 136.38 537.23 116.94 519.24 27.72 2MB Down 1101.01 110.92 588.31 225.21 482.77 31.53
DELL HP LENOVO
Integrated EV-DO Comparison for Dell, Lenovo and HP
10
Chicago, IL – Test Location 1:
Testing Location: Thursday, November 29, 2007 – Starbucks – Chicago Address: 15 S Prospect Ave, Park Ridge, IL 60068 Details: Testing took place in the lobby area of the building with close access to AC power. There are approximately ten (10) schools within a ¼ mile radius of the location. The nearest Verizon Wireless tower is approximately 200 yards east of this location. This location was observed as a moderate to good coverage area.
HTTP Download
1MB Down 2MB Down 1164.27 41.93 1113.49 36.73 572.89 27.94 1MB Up 807.44 4.30 739.00 24.99 751.28 19.36 2MB Up 794.71 16.38 748.48 26.83 764.73 8.12 AVG STDEV AVG STDEV AVG STDEV 1135.12 79.94 1063.45 49.15 725.35 255.44
FTP Test
1MB Down 1152.08 45.08 1065.63 138.48 595.07 32.00 2MB Down 1172.11 45.79 1087.89 143.36 594.44 19.81
DELL HP LENOVO
Chicago, IL – Test Location 2:
Testing Location: Saturday, December 1, 2007 – Radisson Hotel – Chicago Address: 1450 East Touhy Avenue, Des Plaines, IL 60018 Details: Testing took place in the lobby area of the hotel in the east part of the building. This area is approximately 2 miles from the nearest Verizon Wireless tower. This location was observed as a moderate coverage area.
HTTP Download
1MB Down 2MB Down 901.94 70.00 708.01 40.27 535.97 57.89 1MB Up 766.36 10.68 776.55 9.40 757.03 24.51 2MB Up 753.89 13.71 758.25 23.17 760.37 17.72 AVG STDEV AVG STDEV AVG STDEV 955.09 106.24 647.75 66.41 605.44 21.32
FTP Test
1MB Down 1069.35 52.71 641.60 99.34 580.03 8.16 2MB Down 979.45 62.62 658.92 67.57 551.95 19.71
DELL HP LENOVO
Integrated EV-DO Comparison for Dell, Lenovo and HP
11
Chicago, IL – Test Location 3:
Testing Location: Monday, December 3, 2007 – Airport, O’Hare – Chicago Address: Not Available Details: Testing took place in the upper level main area of Terminal 3. There was no access to AC power. This area was observed as a low to moderate coverage area with high network traffic.
HTTP Download
1MB Down 2MB Down 1242.37 179.74 1067.24 302.28 412.19 59.03 1MB Up 616.59 48.52 596.04 194.51 680.81 38.21 2MB Up 567.37 79.00 418.59 255.73 617.40 75.77 AVG STDEV AVG STDEV AVG STDEV 1081.57 341.52 873.25 269.49 449.87 53.34
FTP Test
1MB Down 696.81 208.18 963.03 407.09 484.11 35.17 2MB Down 762.47 233.18 1008.51 348.34 413.91 89.10
DELL HP LENOVO
Integrated EV-DO Comparison for Dell, Lenovo and HP
12
All references, in part or in full, to test results presented in this report must directly reference or link to this complete report. All references should be made to this report which will be posted in the competitive analysis test section of the Lionbridge web site.
Disclaimer of Warranties; Limitation of Liability:
VERITEST HAS MADE REASONABLE EFFORTS TO ENSURE THE ACCURACY AND VALIDITY OF ITS TESTING, HOWEVER, VERITEST SPECIFICALLY DISCLAIMS ANY WARRANTY, EXPRESSED OR IMPLIED, RELATING TO THE TEST RESULTS AND ANALYSIS, THEIR ACCURACY, COMPLETENESS OR QUALITY, INCLUDING ANY IMPLIED WARRANTY OF FITNESS FOR ANY PARTICULAR PURPOSE. ALL PERSONS OR ENTITIES RELYING ON THE RESULTS OF ANY TESTING DO SO AT THEIR OWN RISK, AND AGREE THAT VERITEST, ITS EMPLOYEES AND ITS SUBCONTRACTORS SHALL HAVE NO LIABILITY WHATSOEVER FROM ANY CLAIM OF LOSS OR DAMAGE ON ACCOUNT OF ANY ALLEGED ERROR OR DEFECT IN ANY TESTING PROCEDURE OR RESULT. IN NO EVENT SHALL VERITEST BE LIABLE FOR INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH ITS TESTING, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT SHALL VERITEST'S LIABILITY, INCLUDING FOR DIRECT DAMAGES, EXCEED THE AMOUNTS PAID IN CONNECTION WITH VERITEST'S TESTING. VERITEST AND THE VERITEST LOGO ARE REGISTERED TRADEMARKS OF LIONBRIDGE TECHNOLOGIES. DELL, THE DELL LOGO, AND LATITUDE ARE REGISTERED TRADEMARKS OF DELL INC. OTHER COMPANIES’ NAMES ARE USED HEREIN FOR IDENTIFICATION PURPOSES ONLY AND BELONG TO THEIR RESPECTIVE OWNERS.
Integrated EV-DO Comparison for Dell, Lenovo and HP
13