Related Articles
Now playing:
Watch this:
4G networks: How they’re tested
1:53
At 10:30 in the morning a few weeks ago I hopped into a plain white van, a mysterious briefcase carefully laid out on the passenger’s seat abuzz with phones.
Into the afternoon, the driver answered my probing questions about long hours on the road, awkward run-ins with the cops, and flashed guns. This was no police surveillance ride along; my driver was merely sharing a day in the life of an independent cell phone network tester.
Meet RootMetrics
A few miles away from Microsoft, a small company called RootMetrics (formerly Root Wireless) sends out a tiny squadron of testers — called scouts — to independently test carrier networks, and they do it using little more than a Garmin GPS unit and a briefcase full of phones.
Inside the briefcase, one phone per carrier runs through the battery of tests. Every 5 minutes, the phone tracks calls, upload and download speeds, and texting. After the weeklong-testing period (or longer), RootMetrics crunches the data. Twice a year, the company compiles a larger report on the health of each network around the country.
Together, the scouts who form the backbone of the operation clock 20,000 hours of testing and calculate information from 100,000,000 data points in a single year. The bottom line: network testing is serious stuff. Here are five things you probably didn’t know about the voice and data testing that goes on in cities across America, maybe even your city, maybe even right now.
Editors’ note: CNET partners with RootMetrics to provide coverage maps on our mobile reviews site.
1. The Gettysburg Address plays a role
RootMetrics hasn’t been around for four score and seven years, but President Abraham Lincoln’s iconic speech is a bedrock element for RootMetrics’ network tests, albeit indirectly.
Inside the scout’s black portfolios are six or seven phones, each one fixed with Velcro and powered by a large, all-day battery. Each phone runs through the series of tests: outbound calls first, followed by inbound calls and then photo, video, and music uploads. Next come the downloads and finally the text message. A master phone connected through Bluetooth to the other five or six runs the whole batch.
Jessica Dolcourt/CNET
Now back to Lincoln. RootMetrics uses Twilio, a San Francisco-based company that lets developers integrate voice and text to their apps, as the server that these test phones ping. After Twilio’s server answers the phone call, it plays back the Gettysburg Address for a total of 1 minute and 40 seconds.
The entire testing cycle is over in 4 minutes and starts fresh every 5 minutes, which means that within the testing briefcase of five phones, the Gettysburg Address plays at least five times every 5 minutes, 60 times every hour, and 480 times every 8-hour shift.
2. Android forms the backbone
I mentioned that a RootMetrics scout carries a total of six or seven phones in the briefcase. These aren’t just any phones. Each of the company’s 12 portfolios are identical, and RootMetrics chooses the testing handsets with care.
Right now the devices are all Android phones, because the app runs on Google’s mobile operating system. RootMetrics recently performed a sample test on the iPhone, but it required five scouts in one car to manually run through the test every 5 minutes — one scout per network in this outlier case.
In San Francisco, RootMetrics tests AT&T, MetroPCS, Sprint, T-Mobile, and Verizon, but in other markets, the company will add or replace carriers based on the provider’s strength in a given market. For example, scouts might test Cricket in Tucson and San Diego in addition to the standard Big Four carriers, and add Virgin Mobile and U.S. Cellular in Chicago and Milwaukee.
#nlrText {float:right; width:150px;padding:20px;margin:20px 0px 20px 10px;background:#EEEEEE;-webkit-box-shadow:inset 0px 1px 1px 0px rgba(0, 0, 0, .25);box-shadow:inset 0px 1px 1px 0px rgba(0, 0, 0, .25);-webkit-border-radius: 3px;border-radius: 3px;}#nlrText p {font-family:helvetica;font-size: 14px;line-height:18px;border-bottom:1px solid rgba(0, 0, 0, .1);-webkit-box-shadow: 0px 1px 0px 0px rgba(256, 256, 256, .4);box-shadow: 0px 1px 0px 0px rgba(256, 256, 256, .4);padding:10px 0;margin:0;}#nlrText b {font-family:franklin-gothic-urw-cond, Helvetica; font-size:15px;font-weight:700;color:#39434C;}
#nlrText h4{font-family:franklin-gothic-urw-cond, Helvetica;font-size:24px;color:#860000;margin:0;font-weight:700;}
By the numbers
Testing carriers’ voice and data networks is all about statistics.
13 scouts
Just over a dozen people and a handful of phones can test the nation’s voice, 3G, and 4G networks.
150 miles
Scouts drive about 150 miles per 8-hour shift, and shifts span all hours of the day.
153 cities
RootMetrics tested 75 urban centers in the first half of the 2012 and will 78 in the second half.
480 times
The Gettysburg Address plays over and over throughout a testing shift.
20,000 hours
Together, scouts spend roughly 20,000 hours on the road every year.
200,000 total miles
Over the course of a year, scouts collectively drive a car into the ground.
100,000,000 data points
RootMetrics analysts sift through millions of data points to evaluate network strength.
Before finding a permanent home in the testing kit, a phone has to pass initiation through extensive field testing trials. If the phone can run the testing software, stay connected to the master phone, and deliver data, it stays in the running.
RootMetrics selects higher-end phones for each carrier, like the HTC Rezound for Verizon, the Samsung Galaxy S II Skyrocket for AT&T, and the Samsung Galaxy Attain 4G for Metro PCS. Why? Because these handsets should capture the best network results across the board for all networks, which will paint a clearer picture of the network’s capability. If the test results are poor, chances are it’s because of the network and not due to complications with a sub-par phone.
As such, RootMetrics is auditioning a handset to replace Sprint’s Samsung Epic 4G, which supports 4G over WiMax, with an LTE-ready phone. RootMetrics wouldn’t say which phone is the frontrunner, but they’ll buy 12 units of the winner off the shelf to add to their testing kits.
3. Network testing runs on gas
Scouts may be chosen for their technical aptitude and troubleshooting skills, said my RootMetrics guides, Josh Doehne, senior marketing manager, and Tom Ingarson, field operations manager, but they spend between 60 and 70 percent of their 8-hour shift in the car. That’s a lot of driving, around 150 miles per shift and collectively about 200,000 miles a year.
RootMetrics’s house-written desktop mapping software randomly generates the testing route, which scouts then download onto the Garmin. Even though itineraries are selected at random, they’re based on the city’s urban, suburban, and rural geography. Moreover, the route-generator abides by statistical definitions of validity, which means that RootMetrics is making sure it’s conducting enough data and voice network tests in areas where people live, drive, and shop.
“We’re doing enough sampling out there,” says RootMetrics’ president and CEO, Bill Moore, “that we’re properly characterizing the entire marketplace.”
Phone tests are mostly automated; the solo scouts start the data-collection app at the beginning of their shift and turn it off at the end. If all goes well, they won’t have to disturb the phones, except to carry them indoors for more testing and then back out to the rental car. However, in the rare event something goes wrong, scouts are trained to troubleshoot.
“Don’t the scouts get bored?” I asked my guides. Ingarson, who started off scouting and still goes on training and monitoring rides, says the time flies by. Between keeping an eye on the routes and phones, and logging stops as part of a note taking protocol, there’s plenty to do.
4. Scouts are all around you
See a man or woman clutching an oversize briefcase and walking the aisles of your local supermarket, and you may be looking at a RootMetrics scout. They could be in the car next to you on the road, portfolio open on the passenger seat, eyes glued to the Garmin. As much as it’s possible to blend in with a large black briefcase at their side, they do.
While scouts spend the majority of the time behind the wheel, RootMetrics tests indoor locations, too. In the course of each shift, there are 10 waypoints that a scout must hit along his or her route, and for each of these 10 are 3 indoor locations that a scout can use to test the portfolio phones indoors.
Scouts are encouraged to takes tests indoors at each waypoint, but use their discretion. Indoor tests last 20 minutes, or four testing cycles, during which scouts anonymously walk the aisles of supermarkets, convenience stores, and book shops, sipping coffee, scanning nutrition labels, and thumbing magazines while the master phone and its worker phones probe the radio waves.
Out of RootMetrics’ growing pool of 65 employees, 13 are scouts. However, no more than 7 are currently in the field at a time, and no one tests more than 7 consecutive days. Schedules vary tremendously, and larger markets require more scouting power.
Here’s how it works. Ever the statisticians, RootMetrics defines markets by U.S. census boundaries, so San Francisco also includes a larger circle of cities to the north, east, and south (go west and you hit the ocean.) Based on population size, RootMetrics calculates that San Francisco is a seven-day market, which one scout can handle on his or her own (Doehne says scouts are evenly gendered.)
Jessica Dolcourt/CNET
In contrast, New York is a 35-day market, requiring five scouts scouring the greater metropolitan area around the clock. The smallest markets warrant five days of testing. During even a five-day scouting session, each hour of the clock is covered at least once, with an emphasis on the daylight hours that draw the majority of call, texts, and data activity. (London, which RootMetrics plans to test, is a 50-day market, and collecting data in Berlin will take a whopping 80 days.)
For instance, a scout’s shift could run from 9:00 a.m. to 5:00 p.m. one day, and from 10:00 p.m. to 6:00 a.m. the next. In larger markets like New York, the smaller hours of the morning would see proportionally more testing time than in San Francisco, but fewer than the peak times of calling, testing, and data activity.
RootMetrics keeps its scouts busy. They cover 75 U.S. cities in the first half of the year, and tackle 78 cities in the second half. “They’re our front line in data collection,” Senior Marketing Manager Doehne said.
After the day is done, scouts power down devices and recharge them, then upload notes on the drive — how was traffic, what was the weather like, and did stores seem crowded? Along the route, they also snap photos inside and outside of an indoor location, and log the GPS coordinates of each stop. At the end of the shift, scouts also upload the GPS tracks to the company database.
5. It can get awkward
Indoors, scouts stay out of the way as much as they can, but with the testing kit in tow, there are only so many places to hide.
“It can get a little awkward walking around a Safeway at 3 a.m. with a portfolio,” RootMetrics’ field operations manager Tom Ingarson said.
Furthermore, scouts are instructed to skedaddle or scrap the indoor plan if the environs seem unsafe. When prodded about any danger, Ingarson told me he once had a gun flashed at him in Kansas City, then quickly backed off the topic. After all, he doesn’t want to make his work seem dangerous or unsavory.
Jessica Dolcourt/CNET
There are the small accidents that come with steering a vehicle on the road for a third of a day — the bumps, scrapes, and fender benders, but Ingarson insisted that there’s never any violence, and scouts are told to avoid or vacate any situation that feels unsafe, no matter what the Garmin says.
Read more Smartphones Unlocked
- Smartphone innovation: Where we’re going next
- 6 things I want from NFC
- What it really takes to make a flexible phone
- How to sell your phone for cash
- Your smartphone’s secret afterlife
- Smartphone batteries: Problems and fixes
- 5 things you didn’t know about data testing
- The ABCs of smartphone screens
- Why more camera megapixels aren’t better
Sometimes it’s the scouts who appear shady. Every scout has gotten pulled over by law enforcement at least once for their strange-seeming behavior, Ingarson told me. It’s almost a rite of passage. When his Garmin led him to a naval base in Baltimore in the dead of night, suspicious base cops grilled Ingarson for 5 or 10 minutes, but a quick phone call to support staff back in Bellevue quickly cleared things up.
While chats with the local authorities are the height of most confrontations a scout sees while testing carrier networks, Ingarson hinted that scouts get an eyeful of humanity’s more shadowy side. “You see a lot of stuff,” he said.
Where do the numbers go?
It takes about a month for RootMetrics analysts to crunch the numbers and compile them into meaningful reports. Free reports appear online every six months for consumers, and the numbers also feed into free apps that you can download for Android and iPhone. In addition to drawing from RootMetrics’ database, the apps also combine crowdsourced results from phone owners’ real-time tests.
Revenue comes from syndicated reports that the device-makers and carriers buy. The four major national carriers are all RootMetrics clients, and use the tests to compare their strengths and weaknesses against their competitors. “We offer carriers the opportunity to see what they get in terms of network performance and signal strength,” said RootMetrics’ Doehne.
“T-Mobile has a long-standing relationship with RootMetrics,” a T-Mobile spokesperson told CNET in an e-mail, “and has worked with them for many years to validate benchmarking methodologies and develop strategies to measure customer experience.”
As I mentioned above, the company also partners with third parties, like CNET, to provide interactive coverage maps.
In the future, RootMetrics plans to move into device-specific tests, perhaps for individual tablets, and may license their testing software directly to phone-makers and carriers so they can perform their own in-house testing using some of RootMetrics’ methods.
Screenshot by Jessica Dolcourt/CNET
Testing is huge
More than ever, a carrier’s quality — especially 4G data speed — is a critical factor in choosing one provider over another. CNET’s cell phone editors do our best to test call quality and network speeds in a variety of situations for each phone, but we can’t be everywhere at once and even with time and transportation on our side, testing is limited to a small portion of our respective cities.
The two urban hubs where CNET editors test, San Francisco and New York, are especially challenging. Our cities are riddled with radio-blocking concrete and hills (S.F.), too-few towers, unexpected dead zones, and heavy congestion. We’re often the last to see a new data network technology like LTE, which can make it difficult when our readers ask how fast Sprint’s new 4G LTE network operates (it hasn’t launched in either spot at the time of this story) or T-Mobile’s speeds compared to AT&T’s and Verizon’s in the midwest.
Companies like RootMetrics, although they’re not the only ones in the game, offer a useful service to consumers seeking independent information about carrier performance in their cities and towns.
CNET
Smartphones Unlocked is a monthly column that dives deep into the inner workings of your trusty smartphone.
