Attachment 8 - Air Force Meter Data Management Plan.pdf

Air Force Meter Data Management Plan

(MDMP)

Updated April 2022

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from the Air Force.

However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately

United States Air Force

TABLE OF CONTENTS

1. Requirement 3

2. Executive Summary 4

3. Background 5

4. Current Status 7

5. Cyber Secure Requirement 8

6. Meter Data Management & Use 9

7. Non-AMRS Bases 10

8. AMRS Sustainment 11

9. Summary 12

10. Figure 1 – Air Force AMRS Accreditation Boundary 13

11. Figure 2 – Statutory Cost Effective Criteria 14

12. Figure 3 – AMRS Meter Specifications 15

Requirement

This report is being provided to the Deputy Under Secretary of Defense for Installations and Environment in response to the Utility Meter Policy Memorandum dated April 16, 2013.

No later than 1 year after publication of this policy, Components shall develop a MDMP, to be approved by the Deputy Under Secretary of Defense for Installations and Environment DUSD(I&E), which describes how each Component plans to develop advance meters/AMS and intends to use their meter data to manage energy and water use. MDMPs shall be informed through a bottom up review of installations and shall inexplicitly establish a Component-specific metering goal for electricity and natural gas, based on distinctive attributes of Component buildings and operations.

The Air Force meter program is based in several laws:

10 U.S. Code § 2911 42 U.S. Code § 8253

Department of Defense:

DoDI 4170.11

Air Force:

AFMAN 32-1061

Executive Summary

The Air Force Meter Data Management Plan (MDMP) provides a lasting framework supporting the enterprise execution of advanced meters, advanced meter reading systems, and meter data usage. This plan enables the Air Force to effectively meet Office of the Under Secretary of Defense (OSD) and Air Force metering policies. Metering utilities is foundational to installation management and mission sustainment across the Air Force. It is designed to allow near real time monitoring of utilities and enable Energy Assurance.

Through the Advanced Meter Reading System (AMRS) program, the Air Force will install a complete meter system at all Active Duty installations. These systems include installation of cyber secure advanced “smart” meters, meter reading system software, and a secure industrial control system at targeted installations to capture critical electric, water, steam and natural gas consumption. The focus was initially on the highest consuming installations for the deployment of this system. The program was centrally managed and funded.

The focus is now on installations with critical missions as outlined by OSD where the initial install scope is centrally managed and funded, with full system expansion to be funded and executed by each installation. The Air Force will evaluate mission needs and potential energy savings associated with AMRS to determine if the scope of the centrally funded program shall be expanded or restricted. New missions are continuously being evaluated as it is recognized that mission critical designation is not static.

Additionally, the AMRS program management office (PMO) is the AMRS system owner, and leads system tool and metric development to analyze the meter data. The goal is to set Air Force level standards, and determine the best approaches to manage both the facilities and their energy use. These tools will enable bases with and without AMRS to manage their energy use in accordance with statutes, OSD, and Air Force policy.

Background

Energy Policy Act of 2005 was passed by Congress on July 29, 2005. This law required metering of electric on all federal buildings. The law required the U.S. Department of Energy (DOE) to establish guidelines for agencies to meter their facilities. The Energy Independence and Security Act of 2007 further established that each agency shall provide equivalent metering of natural gas and steam on their facilities. It is further noted that all MILCON projects as well as O&M renovations over $200K must include advanced metering by statute. Laws passed in the FY20 and 21 NDAAs specifically noting the need to use cyber-secure, sampling rates, need to mirror crucial energy needs, and need for advanced meters to monitor mission critical buildings, and their systems.

The Office of the Under Secretary of Defense established policy in 2013 requiring service components to accurately capture a minimum of 60% of electrical and natural gas use with a goal of 85% thru use of an AMS (Advanced Meter System). Additionally, it outlined requirements for the metering of all steam systems, data centers, communication nodes, and water intensive facilities. Furthermore, this is at the building level as well as sub-metering major sub-systems as needed to ensure monitoring for mission assurance. The Air Force further refined this thru targets in AFPAM 32-1061 with compliance monitored under AFI 90-201 self-assessment checklists for each installation. These directions also required the creation of a MDMP to codify explicitly how the Air Force will meet both statutes as well as policy targets.

The Undersecretary of the Air Force for installations is updating expectations for metering and cybersecurity. The core goal remains on using metering as the backbone for near real time monitoring of utility status and condition. This directly empowers the ability to assure energy supplies to key missions as outlined in the Air Force Energy Flight Plan.

Further policy changes are expected and this document is reviewed 2x/year and updated appropriately based on statutory, OSD and Air Force direction. Designers of facilities are recommended to reference this document to ensure compliance not to exceed one year out of project award to ensure compliance. Guidance is this document goes well beyond existing UFC and UFGSs to ensure compliance with all requirements. Current Air Force position is value engineering meters out of MILCON or SRM projects over $200K is not acceptable unless an approved advanced meter already exists. Connecting these meters to existing CE COINE network will fall into the cost effective evaluation criteria.

Shadow metering on privatized utility systems is also allowed under the above noted statutes.

AFCEC/CIU and AFCEC/CNP actively work to deconflict potential issues is this is needed.

Installations with metering issues on privatized systems should contact AFCEC/CNP. Use of privatized meters or 3rd party meters is not allowed as none have been shown to meet requirements for data collection, islanding, cybersecurity and cost effectiveness.

Currently, the advanced meter way forward is based on a centralized asset management approach executed by a PMO within the Air Force Civil Engineer Center (AFCEC) Energy Directorate at Tyndall AFB, FL. The AMRS PMO requirement is to accomplish the initial system install of the Air Force’s advanced meters. The initial install centers on installation defined critical buildings, crucial infrastructure nodes, and real property systems supporting those missions and nodes. Under this the Air Force is using meters as a system to allow the near real time monitoring of utility status and condition to ensure key utilities are of the quality and quantity to ensure mission success. Installations are expected to continue system expansion thru regular SRM project and MILCON investment. Compliance with statutes, policy and guidance is monitored under the AFI 90-201 self-inspection checklist.

The Air Force’s AMRS ability is the principle driver in data automation to better inform investments. The Installation Resilience Operations Center (IROC) uses AMRS as a cornerstone as the Air Force moves into the ability to use data to automate decisions and centrally manage the ocean of CE Control System data.

Current Status

AMRS is currently the Air Force standard for smart metering. The AMRS PMO centralized and standardized on this platform to ensure cyber security requirements, software maintenance, network standards, and proper personnel for execution are maintained. A critical component of the AMRS system is the installation of the CE COIN (formerly CE VLAN). This independent communication path provides a cyber-secure backbone for the AMRS system. The Air Force installed this cyber secure network at multiple installations. Air Force is now migrating to the next generation of a secure environment under COIN2.

The central install program secures funding under the Air Force’s central SRM Integrated Priority List (IPL). Bases are prioritized then validated by AFCEC and SAF leadership.

Projects are awarded thru the 772nd Enterprise Sourcing Squadron (ESS) at Tyndall AFB. A Contracting Officer Representative is needed for each install with technical reach back provided by the AMRS team. A base install effort averages 100-200 meters, sever and laptops. The AMRS devices are considered Real Property Installed Equipment (RIPE) with servers and laptops considered Facility Operations Tech Investment (FO-TI) items. After install is accomplished the system receives final commissioning and formal system training by the AMRS team.

The AMRS servers and laptop are transferred to each installation’s property books.

AFCEC/CNP will maintain the licenses as well as system ATO. They will also advocate for resources needed for server and laptop technical refresh. If centralized resources are not provided for server or laptop replacements, installations are expected to require needed resources thru AFIMSC/IZ under the FO-TI funding call.

System and network expansion beyond the initial installation is with the installation for funding.

Installation leadership is asked to contact the Energy Directorate (AFCEC/CN) if a follow on installation project is needed due to mission change or some other factor. Each request is justified by that installation and evaluated against Air Force strategic priorities and available resources.

AMRS approved devices are capable of varying sampling rates and providing data to multiple CE Control Systems. The approved device listing and commissioning guides are under constant review and updating. The ARMS ATO and system configuration are also the Air Force’s only centrally managed control system but used for base management.

Furthermore, the AMRS PMO in conjunction with the Measurements and Analysis division (AFCEC/CNA) will evaluate and analyze the energy data for trends, benchmarking, identifying anomalies, and other indicators enabling energy driven, asset management decisions for energy assurance, cost containment, and consumption reduction. Accomplishing these actions will greatly benefit the Air Force by increasing resiliency while managing valuable commodities.

Cyber Security Requirements

ARMS is centrally managed and controlled. This is unique in all Air Force control systems.

AMRS system owner is AFCEC/CNP. The system’s ISSM and ISSO are also centrally located at Tyndall with AFCEC/CNP. The system’s configuration and accreditation are also centrally managed. Installations are not responsible for these factors but any system risk, denial of service or interconnection should be forwarded to required authorities with a Cc to AFCEC/CNP team.

Advanced Metering Infrastructure must maintain compliance with DoD Information Assurance and Cyber Security DoDI 8500 series requirements and Air Force Instruction 33210, The Risk Management Framework (RMF) for Air Force Information Technology (IT).

All AMRS components will utilize the CE COIN platform IT system for transmitting data from the advanced meter to a centralized server on the installation (Figure 1).

Baseline configuration changes shall occur as events dictate, such as changes due to USCYBERCOM tactical orders / directives or cyber-attacks (Control Correlation Identifier (CCI) 000297). This policy shall be reviewed annually and updated as required (CCIs 000289, 000292).

Configurations will be strictly maintained for each AMRS core component (CCIs 001737, 001503). Currently, core AMRS components are comprised of the following items:

1) AMRS Host / Physical Machine

2) AMRS Web Server Virtual Machine

3) AMRS Laptop

The requirement for patches to the baseline system images for incorporation into the AMRS will be reviewed by the AMRS information systems security manager (ISSM) on at least a quarterly basis or within the timeframe required by the associated directive.

Patches that are minor in nature (such as those mandated by USCYBERCOM) and do not affect operation on the system should be promptly implemented under the direction of the ISSM. The AMRS ISSM will bring all remaining patches to the configuration control board (CCB) for evaluation and consideration for deployment. After CCB approval, patches shall be incorporated into AMRS and an appropriate version number will be assigned according to CCB procedures. Baseline system images shall be patched at least every 12 months as required to sustain cybersecurity and functionality of AMRS.

The Air Force AMRS accreditation boundary is presented in Figure 1 of this document.

Meter Data Management and Use

The Air Force plan has two current tracks:

1) AMRS installed systems and bases

2) Non-AMRS installed systems and bases.

For bases scheduled for AMRS, this system comes complete with software that enables the base to effectively see critical information about utility status, facility issues, and energy quality.

This includes, but is not limited to determining power factor, peak power, total consumption over time, and energy use. Sampling rate is initially set for 15 minute intervals but may go as low as 0.1 seconds for electrical meters to meet mission needs. The system also allows for additional reports and analysis as determined by AFCEC Subject Matter Specialists or the installation. This basic information allows for bases to perform:

1) Power quality

2) Microgrid operation and load shedding

3) Error and outage reporting

4) Demand management programs

5) Benchmarking of facility and facility system health

6) Facility Energy intensity comparison

7) Unit energy use comparison

Islanded microgrid management, demand side management and utility monitoring for mission assurance is core for meter data. Metering data is often the initial or only indicator of a facility with an issue which culminates in mission impact. This system identifies problems for Civil Engineers where equipment may require tune ups or load management. It will allow them to focus diminishing resources on maintenance, repair, or equipment replacement on those areas using excessive energy having the savings potential 5-15% of utility costs, as well as countless man hours troubleshooting utility system issues. It also provides critical data for facility retro-commissioning and implementation of other energy conservation opportunities.

The meter data is owned wholly by the Air Force and is a valuable asset for potential vendors.

Sharing of any AMRS data with outside organizations or vendors should include data restrictions and retirement requirements. Final disposition of AMRS data is under review, along with all other CE control system data, for applicability under AF PD 33-3 and AFI 33-322. No classification guide exists for AMRS data but it should be considered Sensitive, and for limited distribution.

The AMRS system is for real property assets and benefit of Civil Engineers. Industrial processes, mission systems, and privatized entities could benefit from the meter data also.

These users should work thru AFCEC/CNP to validate if a cost sharing model is required before data is provided. For larger potential uses of AMRS data, AFCEC/CN, AFCEC/OEA and AFCEC/CO are actively looking at potential uses.

Non-AMRS Bases

For bases that are not currently scheduled to receive AMRS there are two levels of action that need to be addressed:

1) 1-5 years

2) 6-10 years

For the one to five year outlook, bases not currently projected for AMRS must comply with OSD and Air Force metering policies for renovation and new construction. See CE Dash and AFMAN 32-1061 for criteria. All new meter acquisitions should be approved by the AMRS PMO to ensure compliance with cyber security, software and hardware requirements. The AMRS PMO will continue to be the Air Force center of excellence for metering standards and deployment.

During the six to ten year outlook, the AMRS PMO will validate scope for that installation based on mission needs, and gain a vector if the centralized install will support the install at that base.

Bases with metering systems that are non-AMRS compliant shall look for opportunities to bring their metering system into compliance. This should include, but not be limited to installing CE COINE or appropriate systems, repair by replacement of all non-compliant meters with AMRS compliant meters, coverage of the Air Force Network on key infrastructure nodes and ensure all new facilities and major renovations install AMRS compliant meters consistent with Air Force requirements. Expansion of the network to support CE COINE installation is done under the FO TI (Facility Operations Technology Investment) program as part of AFIMSC/IZ’s portfolio.

Installations should look at having pre-priced options in their SABER or other MATOC contracts approved meters to simplify installation.

AMRS Sustainment

The Advanced Meter Reading System (AMRS) initial installation is centralized in order to generate an enterprise solution for metering software, hardware, and cyber security. Beyond the initial install; a long term sustainment plan was necessary to ensure proper funding and personnel for the support.

Current Air Force position is centralized Air Force funding of the initial system to include needed servers, laptops and devices. All items are singed over to the installation upon system commissioning. The initial system install does not meet all statutory and policy directives but is done based on current Air Force position for cost effective metering.

The system is monitored day to day by installation Civil Engineer and Communications personnel.

This is done under the same MOU for all CE Control Systems. The local communications squadron is responsible for local network trouble shooting, Civil Engineers are responsible for trouble shooting when a meter or system device is not communicating. AFCEC/CNP will act as deep reach back and provide in person field support, as resources allow, only when all local troubleshooting efforts fail.

AFCEC/CNP is targeting centralized system troubleshooting and updating using remote methods.

Installations should understand this is not currently in place but using remote capability is a major initiative to maximize system up time for installations.

Additional training for new personnel is available on a first come/first served basis at other bases formal training. TDY fees are the responsibility of the requesting organization. AFCEC does offer some familiarization training and will continue to offer phone support as resources allowed. Virtual training is being looked into.

AFCEC/CNP also retains system experts on a support contract. Requests for specialized configurations, reports or unknown issues may be referenced to these vendors. Installations may request support from these experts but final allotment of resources is designated by AFCEC/CNP.

Summary

The Air Force MDMP is a comprehensive plan to bring the Air Force into compliance with OSD and congressional direction. This plan encompasses a systematic enterprise approach that provides a pathway for compliance with energy and cyber security policy. This plan should be reviewed, updated and approved at least annually to ensure the program is tracking towards the intended goals.

Figure 1 – Air Force AMRS Accreditation Boundary

U.S. Air Force Page 14 of 18

Figure 2 – Statutory Cost Effective Criteria

(Installed Costs + Maintenance Costs) < (5% x Annual Utility Cost x 10 years) + Other Savings

1) Initial installation cost of metering to include meter, software, and communications

2) $300/year maintenance cost over 10 year meter life cycle.

3) Minimum energy savings due to increased awareness and improved operations

4) Annual cost of energy through the meter

5) Life cycle period of 10 years

6) Other documentable savings (reduced cost of operation and maintenance, increased potential for energy management/savings, and energy efficiency improvement

U.S. Air Force Page 15 of 18

Figure 3 – AMRS Meter Specifications

AMRS Electric Meter Specifications The AMRS Compliant Electric Meter shall meet Department of Defense cybersecurity in addition to the following requirements:

• Electric meters shall meet or exceed the ANSI C12.20-2015, 0.2 accuracy class standard

• Current transformers (CTs) shall meet or exceed the accuracy class of their associated electrical meter

• Onboard memory

• Onboard Ethernet communications, 10/100 base with RJ45 connector or receptacle

• Modbus/TCP communication protocol is required

• Front display with ability to display all measured values

• Minimum of one (1) external RS485 serial communications port

• Minimum of two inputs that shall count pulses from other devices

• If the electric meter does not have the following capabilities on-board, additional modules or devices listed below shall be required within the electric meter enclosure:

Stand-alone step-down transformer compatible with the system voltage that is being monitored Stand-alone DC power supply for external devices Input/output terminal blocks

AMRS Compliant Meter List The following list of meters were evaluated, meet the AMRS meter specification and shall function within the AMRS platform. All networked devices, their integrated firmware and software will be approved by the PMO for use in the AMRS. PMO approval is contingent upon the device, in its fielded configuration, passing USAF cyber and AMRS compatibility testing.

• Schneider Electric PM8000

• Schneider Electric ION 8650

• Schneider Electric ION 9000

• Schneider Electric ION 9000T

• Siemens 9410

• Siemens 9810

• Schneider Electric PM5560 (currently connected to AMRS only, no new installs)

• Schneider Electric ION 8600 (currently connected to AMRS only, no new installs)

• Electro Industries Nexus 1272 (currently connected to AMRS only, no new installs)

• Electro Industries Nexus 1262 (currently connected to AMRS only, no new installs)

• Schneider Electric PM800 (currently connected to AMRS only, no new installs)

• Veris Hawkeye 8100 series (currently connected to AMRS only, no new installs)

U.S. Air Force Page 16 of 18

Meter Installation Guidelines

• Contractor shall assume all existing equipment is working properly. If during the post award survey portion of the SOW an existing piece of equipment is found to be malfunctioning, then the contractor shall bring it to the government's attention to determine how to proceed. An RFI shall be submitted. This may require a contract change to delete this meter from the scope or to add a new meter.

• All due diligence to survey the existing site conditions is the responsibility of the contractor.

• New electric meter enclosures shall be located 5 feet above finished floor level and as close to the main distribution panel as possible. All metering device labels shall be white on black phenolic no smaller than 1” x 3”

• New meter installations shall include the installation of new, properly sized, split or solid core CTs. Accurate sizing of the new CTs is the responsibility of the contractor.

Existing, unused CTs are to be removed if possible. If removal is not possible, existing CTs must be shorted out.

• If CT installation method includes disturbing the bonds of existing cables or wiring within the gear, before and after IR scans shall be performed before removing and after torqueing the connections to ensure the integrity of the conductor bonds.

• Meter enclosures shall include fused, finger safe voltage disconnects and CT shorting blocks. Instructions explaining how to operate the CT shorting blocks must be visible inside each meter enclosure.

• The meter must not require utility outages for servicing.

• Meters shall have control power regardless of breaker position.

• All meter enclosures shall be fed by a load break 3-pole fusible disconnect switch that is external of the meter assembly. This safety disconnect switch shall be marked as follows “Meter Disconnect NOT Service Equipment” per the NEC

• New electric meter supply voltages shall be acquired by means of voltage taps from the bus or system breakers for the voltage system that the AMRS meter is monitoring.

Insulation piercing connectors are not an approved method and energized electrical work will not be permitted.

• Electric meters with gas or water meters connected shall have two internal registers configured - one for a raw pulse count and one for a weighted pulse count based on the scaling of the connected gas or water meter.

• In the event communications equipment and/or a metering device has to be located in an exterior location, the meter enclosure must be a lockable NEMA 4 enclosure. All new interior communications equipment and metering devices are required to be enclosed in a lockable NEMA3 or NEMA3R enclosure

U.S. Air Force Page 17 of 18

Cabling Installation Guidelines

• New meter installations shall be connected via CAT6 cable from the meter, to a wall outlet in the electrical room, to a patch panel in the communications room, and then to the AFNet switch in the communications room (base communication squadron will make the last connection to the AFNet switch). CAT6 cable runs are not to exceed 328 feet including patch cords.

• Preferred technical approach to network communication cabling in excess of 328 feet is:

fiber optic cable and fiber media converters with lockable enclosures and dedicated power outlets.

• TIAA compliant fiber media converter shall be installed in the communications room as close as possible to the network switch and as close as possible to the electric meter enclosure.

• Communication receptacles accessible to the general public must be installed in lockable enclosures.

• Cable labels shall be permanent, wrap-around and self-laminating with text generated by a mechanical device. Labels shall be applied at both ends of the cable, visible during normal maintenance of the infrastructure and resistant to environmental conditions.

• All data cable test results (including fiber) shall be completed only after the cable has been terminated in its permanent location, temporary cable test results will not be accepted.

• Surge protection is required on all communication cabling extending from the exterior to the interior of a facility. This includes gas and water meter communication cables from the exterior to the interior of the building.

• Surge suppression devices shall be installed w/in 3 feet of the building entrance inside the facility. Devices shall be easily accessible and labeled to aide in location during future troubleshooting efforts.

• Surge protection devices shall be Power over Ethernet compatible and housed within a lockable enclosure that allows space for a proper cable bend radius

• Fiber optic cable strands shall be terminated on patch panel bulkheads or within a fiber optic wall outlet assembly; unsecured connectors are not acceptable.

AMRS Compatible Gas Meter Specifications AMRS compatible gas metering solutions consist of a componential system to achieve the goal of delivering gas consumption data into the AMRS. To accurately scale output pulses the contractor shall obtain building gas usage from base personnel, taking into account peak demand when sizing the components that are necessary as well as determining peak pulse rate as to not saturate the receiving device and risk losing captured pulse data.

• Pulse kit must be able to produce a two-wire (Form A) pulse output via RS485 cable

• Gas meter or attached pulse kit must include a visual register or dial

• All new gas meter assemblies shall include upstream strainers

• All meter labels shall be white on black phenolic

U.S. Air Force Page 18 of 18

AMRS Compatible Water Meter Specifications AMRS compatible water metering solutions consist of a componential system to achieve the goal of delivering water consumption data into the AMRS. To accurately scale output pulses the contractor shall obtain building water usage from base personnel, taking into account peak demand when sizing the components that are necessary as well as determining peak pulse rate as to not saturate the receiving device and risk losing captured pulse data.

• Pulse kit must be able to produce a two-wire (Form A) pulse output

• Water meter or attached pulse kit must include a visual register or dial

• All new water mater assemblies shall include upstream strainers

• All meter labels shall be white on black phenolic

AMRS Compatible Translation Devices and Accumulators The following devices are compatible with the AMRS platform, have been tested and passed cybersecurity requirements. Please note that they will only be approved for use on a case-by-case basis when dictated by unique existing site conditions. Translation devices and accumulators require lockable enclosures and dedicated power outlets.

• Schneider Electric Link 150 Ethernet Gateway

• Schneider Electric ComX 210 Data Logger

• Perle SDS1P Device Server (currently connected to AMRS only, no new installs)

• Perle IOLAN DS1 (currently connected to AMRS only, no new installs)

• Advantec ADAM-6060 I/O Module (currently connected to AMRS only, no new installs)

• Advantec ADAM-6051 CE Version (currently connected to AMRS only, no new installs)