OIT_RFP2017000243_AttachmentC_InitialMarketAnalysis_FINAL_PartII.pdf

SiGNALS Analytics, LLC

III. Survey Methodology and Results

Survey Methodology and Results

• Introduction

– Objectives

– Background

– List of Deliverables

• Insights and Analysis

– Key Insights

– Numerical Results

– Non-Numerical Results

• Demographics of Respondents

• Appendices

– Data Cleaning Methodology

– Data Cleaning Audit History

Objectives

• This survey had two objectives:

– Sample the population of first responders in Colorado to determine key information that will be important in planning the FirstNet network

– Conduct a census of PSEs in Colorado to gather important contact information that has not previously been assembled

• The survey was launched in May of 2015. We encourage the reader to read this report, to study the numerical results, and to study the detailed responses for visibility into the anecdotal thinking of respondents.

• The reader should note that the survey produces a rich collection of contract information (embedded in the merged responses) that will be important for many types of initiatives in the future.

Background

• FirstNet Colorado initiated a survey of public safety entities in May of 2015, with the objective of collecting the information requested by FirstNet from each state and territory.

• The survey exists in “full” and “short” formats. The short survey captures a subset of the information.

• 147 entities have responded, from May 29th, 2015 to September 8th, 2015. 106 parties completed the full survey; 41 completed the short. The responses represented 12,233 first responders.

• Each survey was modified slightly during fielding. There are therefore 4 versions of the survey instrument (full rev1, full rev2, short rev1, and short rev2).

• The four versions have been merged into a single data set. Not every question was answered by every respondent, because of the multiple formats.

Key Insights

Data Usage per Device per Month (GBs)

0.

1.

2.

2.

3.

3.

2.

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

M2M Module Smartphone 4G Smartphone Tablet 4G Tablet Laptop Colorado FIRST

RESPONDERS

U

SA

G E pe r D ev ic e pe r M on th (G

Bs , W id e Ar ea N et w or k)

Colorado First Responder Usage vs. Consumer Usage

Insights on Usage

• First responders have a very distinct and essential mission.

• Interestingly, when it comes to measured data usage, first responders look surprisingly similar to consumers:

– First responders have a mixture of smart phones (voice + data), computers (data), tablets (data), and other devices

– First responders will increasingly transmit and consume video – for situational awareness and other essential needs

– Consumers also transmit and receive video (over 50% of all mobile internet traffic), primarily for entertainment

– Data consumption is influenced by the capabilities of devices and the capabilities of the network

– On a weighted average basis the total usage of first responders is currently indistinguishable from that of consumers

• First responders need service in remote areas. This impacts the required geographic footprint of the LTE network, but not its aggregate capacity.

Cost of Service: Public Safety vs. Consumer ($/month)

$4 6.

$5 4.

$3 3.

$4 7.

$10.00

$20.00

$30.00

$40.00

$50.00

$60.00

Consumers (All Devices, 2014) Colorado PSEs - Voice and Data Colorado PSEs - Data Only Colorado PSEs - Composite

Av er ag e Se rv ic e Re ve nu e (A

RP

U ) p er M on th

Consumers vs. Public Safety

Insights on Pricing

• The survey captures the average monthly cost of service, before taxes and other fees. This is known in the cellular industry as Average Revenue Per User (ARPU). The survey captures pricing two distinct categories:

– Voice and data devices (e.g. smart phones)

– Data only devices (computers, tablets, M2M devices, etc.)

• The survey also captures the total number of devices in each category.

• We can calculate the blended monthly ARPU for first responders in Colorado and compare it to the average ARPU of mobile operators nationwide.

• Interestingly, the two numbers are nearly identical:

– Colorado PSEs: $47.12

– US Consumers: $46.64

• The next page discusses factors that might explain this result.

Pricing: Positive and Negative Factors

Factor Driving LOWER Pricing Factors Driving HIGHER Pricing

• Some PSEs are large organizations, with hundreds or thousands of devices. Large entities (corporations or governments) should be able to negotiate attractive volume discounts.

• Some PSEs, like large corporate clients, might not ask mobile operators to subsidize their devices. Other PSEs might prefer bundled solutions, where they pay for service, inclusive of a hardware subsidy. Others might prefer the opposite, paying for hardware, inclusive of service. Removing all subsidies would lower service pricing relative to consumer pricing.

• First responders represent a very “low churn” customer segment for operators. This greatly increases the lifetime value of a customer to the operator, and should – if recognized - result in more favorable monthly pricing.

• First responders demand high quality service and are likely to purchase service from the best provider, regardless of price.

• Rural areas, that represent much of Colorado, are often served by fewer mobile operators.

This lack of competitive choice is likely to increase pricing.

• PSEs may be receiving technical support or other services in place of lower service pricing.

• Consumers are increasingly choosing to pay full price for their device, in some cases in cash, in other cases by a separate financing arrangement with the mobile operator or with a third party. Either way, the device subsidy that has historically been important to consumers and a sizable cost to operators is going away.

Staffing Ratios

0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

TOTAL Staff FULL TIME Staff PART TIME Staff VOLUNTEERS Vehicles

St af fin g Ra tio s (

ot al S ta ff

Device Ratios

5%

7%

1%

0%

20%

40%

60%

80%

100%

120%

140%

DEVICES /

Person

ENTITY

Voice + Data

DEVICES /

Person

ENTITY Data Only

DEVICES /

Person

ENTITY

Other

DEVICES /

Person

M2M

DEVICES /

Person

Individual V&D

DEVICES /

Person

Person

Vehicle

DEVICES /

Vehicle

De vi ce R at io s (

ot al S ta ff, U nl es s O th er w is e In di ca te d)

Insights on Devices

• We need to know the number of devices per first responder to estimate total demand. Devices include:

– PSE-provided devices for individuals

– PSE-provided devices for vehicles

– Devices personally owned by individuals

• There is approximately one device per first responder, if we consider only PSE provided devices. If we add personally owned devices the ratio of devices to people rises to just over 1.2

• Since the vast majority of PSEs allow personal devices to be used, the same entities will probably allow Band 14 enabled personal devices to be used.

This is particularly relevant to organizations with large volunteer staffs.

• The graph on the previous page shows PSE-provided devices and personally-owned devices in different colors.

Applications: Ranked by Importance (daily + weekly)

0%

0%

9%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Q: Which of the following is a barriers to adoption?

46%

39%

77%

21%

61%

54%

60%

23%

76%

39%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cost Reliability Security Coverage Expertise

YES, a Barrier

NO, Not a Barrier

Statistical Margin of Error

• The statistical margin or error, commonly reported in the media for consumer surveys, identifies the 95% confidence threshold implied by the sample size. It typically assume a “0% to 100%” question format.

• Using this common approach the survey has the following margin of error:

– Questions answered by all 147 respondents: +/- 8.1%

– Questions answered by 50% of respondents: +/- 11.4%

– Questions answered by 30% of respondents: +/- 14.8%

• In reality, many other factors influence the margin of error in a survey, notably:

– Errors in survey design (e.g. ambiguous questions)

– Biases among respondents (people unwilling or hesitant to provide certain information)

– A “less than infinite” population being surveyed (true in our situation)

• Some conclusions are drawn – as they are in many pieces of market research – from indices representing calculations (sums, ratios, etc.) of particular responses.

Calculating the error in such instances is much more complex.

Notes on Methodology with Numerical Results

• The survey instrument had no required fields. Therefore, the response rate within completed surveys varied by question.

• In many cases we determined that the safest way to handle numerical questions was to calculate the average of the responses for each question then to show ratios among numerical questions based on these averages, rather than as a ratio of the sum of the responses for each question.

If every question had been required (forcing the respondent to complete it or abandon the survey), then these two methodologies would produce identical results.

• In analyzing employees we added the average full time, part time and volunteer staff to arrive at an average “total” staff, to ensure that the ratios added to exactly 100%.

The average reported total staff (which was subject to quantization error, as was each “between A and B” response) was extremely close to this calculated value.

• Finally, we calculated weighted averages for usage per subscriber and for price points. These attributed greater weight to organizations with a larger number of devices, reflecting the methodology behind the financials of any public firm.

Key Insights by Discipline

Usage per Device by Discipline

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Police Fire EMS Other COMPOSITE

De vi ce s p er P er so n U sa ge (G

Bs

ev ic e M on th

Devices / Person

Usage (GBs) / Device / Month

Distribution of Devices by Discipline

0.5

1.0

1.5

2.0

2.5

3.0

Police Fire EMS Other Composite

De vi ce s p er P er so n

Personally-Owned Devices

PSE-Provided Device to VEHICLE

PSE-Provided Devices to PERSON

Key Numerical Results

Volunteers 21% 31% 15% 5% 17% TOTAL Staff 100% 100% 100% 100% 100%

Devices / Total Staff 2.54 2.08 1.20 0.40 1.23

GBs / Device / Month 2.95 3.28 3.05 0.56 2.64

Numerical Results (Ratios)

Conclusions

• Devices per first responder:

– Police / Law Enforcement has the greatest number of devices per first responder

– Fire has the next greatest number of devices per first responder

– Emergency medical services appears to have the third greatest number of devices per first responder, although the EMS sample size is so small that it is difficult describe the segment with confidence

– Other PSEs (including those for whom the survey response reflected multiple entities with different missions) have proportionately fewer devices

• Fire departments appear to make the greatest use of vehicle-mounted solutions. The result is fewer devices per person than law enforcement, but slightly greater usage per device.

• Each type of organization has similar usage per device: 2.64 GBs/device/month.

• The usage calculation is a weighted average (devices per PSE times usage per device). Therefore:

– Urban first responders (who typically have excellent coverage and represent the greatest number of first responders) greatly influence the result

– If some rural PSEs reported reduced usage as a result of poor coverage the impact of this artifact on the calculated average is modest, because rural first responders represent a minority of users

– The estimate of current usage should therefore be indicative of what one might expect to see on a Band 14 network if one were available today and priced competitively with current commercial offerings

Sample Size and Statistical Reliability

Important: the statistical reliability of subsamples deteriorates quickly in inverse relation to subsample size. Our level of confidence around the results by discipline is low because of the small sub-sample sizes

* Entities other than police, fire, and EMS plus and entities that responded on behalf of multiple disciplines

** The statistical margin or error, commonly reported in the media for consumer surveys, identifies the 95% confidence threshold implied by the sample size. It typically assume a “0% to 100%” question format.

Police Fire EMS Other / Combined* TOTAL/Composite

Average 66% 65% 67% 70% 67% Minimum 31% 23% 31% 39% 31% Maximum 98% 98% 100% 97% 97%

Sample Size / Statistical Reliability

Two Important Questions

• One should ask two questions:

– Does it really matter what usage differences exist between first responder disciplines?

Arguably:

• The radio network should be sensitive to total usage (users x devices per user x GBs per device)

• The radio network should be relatively insensitive to the category of user generating the usage

– Do we believe that there will be a significant difference, long term, between the usage per device and/or the usage per first responder by function?

• If so, then research seeking to quantify those differences is valuable

• It is equally possible that differences between urban and rural users or between large PSEs and small PSE is between categories of devices is more material

• In the end, the network economics are driven by total amount of usage (number of devices x usage per device) in a geographic area

An Important Dimension: Adoption

• The analysis of the survey results and the discussion of demand are silent on an critical dimension: adoption.

– Each examines either the current market or the addressable market and simply inserts placeholders for penetration and market share (see II. Utilization of the Sample Network)

– Neither seeks to estimate what percentage of PSEs and first responders with personally owned devices will actually purchase Band 14 service in what timeframe

• We omitted this dimension because it is dependent upon decisions that have not yet been made:

– How much geography will the CO-RAN provide?

– How will services be priced?

– How effective will FirstNet be in encouraging LTE device manufacturers and LTE chip manufacturers to support Band 14?

– How easy will it be for non full-time users with personal devices (volunteer fire fighters, teachers, and others) to purchase Band 14 service?

– How quickly will the network be built?

– How will existing contracts and device replacement cycles influence adoption?

• These questions must be answered to arrive at an accurate estimate of adoption and a corresponding estimate of demand.

Numerical Questions

Number and Distribution of PSE-Provide Devices

5,925, 54%

1,412, 13%

505, 5%

1,157, 10%

1,458, 13%

567, 5%

PSE-Provided Devices: Voice + Data vs. Data

7,082, 64%

2,870, 26%

1,072, 10%

Other

PSE-Provided Devices: Persons vs. Vehicles

7,842, 71%

3,182, 29%

Wireless Data Applications and Frequency of Use

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Not Used

Not Used but Desired

Infrequently

Weekly

Daily

Non-Numerical Questions

Q: What is the contract duration of your current plan in years?

29%

1%

20%

1%

29%

7%

13%

Unknown

Q: Are employees allowed to use personally owned devices to perform any type of work functions?

18%

55%

1%

26%

No

Yes

Q: Is use of personal devices mandatory?

93%

7%

No Yes

Q: Are personally owned devices allowed to connect to any official applications/services i.e. email, internal databases, etc.?

21%

79%

No Yes

Q: Is any compensation provided to employees for use of personally owned devices?

2%

24%

74%

Based on Usage Fixed None

Q: Are there any near term planned changes that could impact these device counts significantly i.e. deployment of cellular enabled tablets?

80%

20%

0%

Q: Does your agency contract with multiple service providers?

62%

38%

No

Yes, with brief explanation

Brands Mentioned in Comments on Service Providers

22, 28%

10, 13%

10, 13%

9, 11%

7, 9%

2, 3%

2, 3%

2, 3%

2, 3%

1, 1% 1, 1%

1, 1% 1, 1%

1, 1%

1, 1% 1, 1% Verizon

AT&T

Century Link

Comcast

Sprint

Jade Communications

LCSO

Sky Beam

Spillman

Agate Mutual Telephone

Baja Broadband

Blue Host

Cedar Networks

Charter

Deaplydigital

Dish

Fire Manager

IBM

Ouraynet

ProVelocity

TDS

Viero

Weld Regional Communications NOTE: this list of brands includes many categories of products beyond mobile operators (e.g. fixed broadband operators, software products, etc.)

Q: Does your agency use a commercial or shared paging system?

54%

46% No

Yes, with brief

Q: What procurement process was utilized

37%

29%

4%

5%

25% Based on Price Quotes

Local RFP/Bid

Master Contract (GSA/Federal) Master Contract (Others) Master Contract (State/WSCA)

Q: Are other procurement options available?

51%

49%

No

Yes, with brief

Q: Does your agency have a direct sales/account relationship with the provider?

40%

60%

No

Yes, with brief

Q: Does your agency have a direct technical support relationship with the provider?

60%

40% No

Yes, with brief

Q: Are any additional or specialized services provided by the provider?

89%

11%

No

Yes, with brief

Q: How important is having a fixed rate plan?

31%

69%

problem

Q: Who is responsible for payment of the invoice?

85%

11%

4%

Agency Liable (agency pays)

Q: What are the monthly data limits on the plan?

68%

18%

14%

0%

Unlimited Capped/Metered Pooled

Q: What is the contract duration of your current plan in years?

29%

1%

20%

1%

29%

7%

13%

Unknown

Demographics of Respondents

Counties and Tribes Participating

County or Tribe Responses County or Tribe Responses

Adams 1 Kiowa 1

Baca 2 Lake 1 Boulder 4 Larimer 6 Broomfield 3 Lincoln 2 Chaffee 1 Logan 1 Clear Creek 1 Mesa 11 Conejos 2 Moffat 1 Costilla 3 Montrose 1 Custer 2 Otero 1 Delta 12 Ouray 4 Denver 3 Pitkin 2 Douglas 2 Prowers 1

Elbert 7 Routt 12 Fremont 3 Saguache 1

Grand 5 Summit 2 Gunnison 1 Washington 1 Hinsdale 3 Weld 2 Jefferson 9 Yuma 1

Total Responses: 147

Department

51, 35%

50, 34%

10, 7%

1, 0%

12, 8%

1, 1% 2, 1%

3, 2% 7, 5%

8, 5%

Fire Service

Emergency Medical Services

Corrections, Courts and Security

Emergency Management

Highway and DOT

Hospitals and Medical Facilities

Public Health

Public Safety Communications

Other (list in field below)

Type of Entity

36%

17%

35%

1%

3% 1%

7%

County Municipal Special District Tribal State Federal Other (list in field below)