TE-22_MRF_v7.0_Operations_Manual.pdf

Joint Land Component

Constructive Training Capability

Multi-Resolution Federation

(JLCCTC MRF)

Federation Manager’s Operations

Manual

Version 7.0

June 12, 2014

JLCCTC MRF v7.0

Federation Manager’s Operations Manual ii

Table of Contents

1 Purpose

2 Organization

3 JLCCTC Federation Overview

3.1 MRF v7.0 Federation Architecture

3.2 JLCCTC MRF Components

4 Federation Technical Guidance

5 Federation Setup

5.1 Network Topology Effects on the RTI.RID

5.1.1 Distributor Nodes

5.2 Configuring System Time

5.3 Verify Network Connectivity and Federate RTI Parameters

5.3.1 Check TCP Connections

5.3.2 Duplicate IP Address

5.4 Configuring Maximum Transmission Unit (MTU)

5.4.1 Verifying MTU

5.4.2 Changing MTU in Windows Vista or Windows 7

5.4.3 Changing MTU in Linux

5.5 Federation Data Products

5.5.1 Universe of Entities (UOE) Files

5.5.2 Fixed Sites XML File

5.5.3 Detailed Supply Mapping File (DSMF)

5.6 Publish Federation Configuration Information

5.6.1 JIS Website

5.6.2 FMT-Lite Webpage

5.6.3 Remote FMT-R Client

6 Federation Execution Operations

6.1 Start JIS Infrastructure

6.2 Initialize the Federation

6.3 Save the Federation

6.4 Restore the Federation

6.4.1 Restore from Ported Save

6.5 Pause the Federation

6.6 Resume the Federation

6.7 Adjust the Rate of Time Advance

6.8 Refresh a Federate’s Subscribed Object Classes

6.9 Build or Edit a Refresh Macro

6.10 Execute a Refresh Macro

6.11 Late Join a Federate

6.12 Shutdown the Federation

7 Troubleshooting Guidance

7.1 FMT-R as a First Responder

7.2 Using the RTI Console to Identify and Resolve Federate Issues iii

7.2.1 RTI Logging

7.3 Scenario Resolution

7.4 Interpreting RTI Error Messages

Appendix A. Flowchart Key ................................................................................................... A-1

Appendix B. Important Federate RTI Parameters ...................................................................B-1

Appendix C. How to use ssh/scp without Passwords ..............................................................C-1

Appendix D. How to Burn a CD/DVD from JLCCTC Lockdown......................................... D-1

Appendix E. How to Unlock Locked User Account ............................................................... E-2

Appendix F. Federation Metrics.............................................................................................. F-3

Appendix G. Sample Federation Metric logs ......................................................................... G-1

Appendix H. Troubleshooting Flowcharts .............................................................................. H-1

List of Figures

Figure 1. MRF Generic Target Federation Architecture - High Level

Figure 2. JTTI+K Federation Architecture

Figure 3. Pre-Federation Tasks Flowchart

Figure 4. Sample Topology without Distributor Nodes

Figure 5. Sample Topology with Distributor Nodes

Figure 6. Example MRF Federation Layout Using Distributor Nodes

Figure 7. Start JIS Infrastructure

Figure 8. Federation Initialization Flowchart

Figure 9. Federation Save Flowchart

Figure 10. Federation Restore Flowchart

Figure 11. Federation Pause Flowchart

Figure 12. Federation Resume Flowchart

Figure 13. Federation Game Rate Adjustment Flowchart

Figure 14. Federate Refresh Flowchart

Figure 15. Build/Edit Refresh Macro Flowchart

Figure 16. Execute Refresh Macro Flowchart

Figure 17. Late Join Federate Flowchart

Figure 18. Federation Shutdown Flowchart

Figure 19. Federation time not advancing

Figure 20. Flowchart Key .......................................................................................................... A-1

Figure 21. Federation Join Issues Resolution Flowchart ........................................................... H-1

Figure 22. Federation Save Issues Resolution Flowchart .......................................................... H-2

Figure 23. Time Advancement Issues Resolution Flowchart (1/2) ........................................... H-3

Figure 24. Time Advancement Issues Resolution Flowchart (2/2) ........................................... H-4

Figure 25. Restore Issues Resolution Flowchart (1/2) ............................................................... H-5

Figure 26. Restore Issues Resolution Flowchart (2/2) ............................................................... H-6

Figure 27. FMT-R Not Updating Resolution Flowchart ........................................................... H-7 iv

List of Tables

Table 1. Typical Packet Sizes

Table 2. Federation Initialization Steps

Table 3. Federation Save Steps

Table 4. Federation Restore Steps

Table 5. FMT-R Steps to Restore from Ported Save

Table 6. RTIExec Steps to Restore from Ported Save

Table 7. SITH as Surrogate Steps to Restore from Ported Save

Table 8. Pause the Federation Steps

Table 9. Resume the Federation Steps

Table 10. Adjust Game-rate Steps

Table 11. Federate Refresh Steps

Table 12. Build/Edit Refresh Macro Steps

Table 13. Execute Refresh Macro Steps

Table 14. Late Join Federate Steps

Table 15. Federation Shutdown Steps

Table 16. Important FMT-R Columns

Table 17. Console Debugging Checklist

Table 18. Troubleshooting Scenarios and Resolutions

Table 19. Interpreting Common RTI Error Messages

Table 20. Sample Federation Save Time Log ............................................................................ G-1

Table 21. Sample Federate Late Join Refresh Time Log ........................................................... G-2

Table 22. Sample Late Join Enclave Refresh Time Log ........................................................... G-3

Table 23. Sample Federation Restore Time Log ....................................................................... G-4

Table 24. Sample Federate Availability Log ............................................................................. G-5

Table 25. Sample Federate Testing Availability Log ................................................................ G-6

Table 26. Sample Integration Availability Log ......................................................................... G-7

1 Purpose

This document provides a technical control operator with information necessary to setup, start, operate, shutdown, and troubleshoot a Joint Land Component Constructive Training Capability

Multi-Resolution Federation (JLCCTC MRF).

2 Organization

The remainder of this document is divided into the following sections:

JLCCTC Federation Overview

Federation technical guidance

Federation setup

Federation execution operations

Troubleshooting guidance

3 JLCCTC Federation Overview

The Joint Land Component Constructive Training Capability (JLCCTC) is a modeling and simulation capability that supports Army and Joint training by providing the appropriate levels of model and simulation resolution as well as the fidelity needed to support both Army and Joint training requirements.

MRF is a federated set of constructive simulation software components that are supported by commercial software and Commercial Off-the-Shelf (COTS) hardware used to train commanders and their staffs in maneuver, logistics, intelligence, air defense and artillery. The federated simulations are connected by a combination of standard High-Level Architecture (HLA) Run-

Time Infrastructure (RTI), Distributed Interactive Simulation (DIS), custom interfaces, and other

Point-to-Point (PTP) interfaces. MRF is a Command Post Exercise (CPX) driver designed to train Army commanders and their staffs at division through echelons above corps. MRF provides the simulated operational environment in which computer-generated forces stimulate and respond to the command and control processes of the commanders and staffs. The JLCCTC simulations provide full training functionality for leader and battle staff for the Army and the

Joint, Intergovernmental and Multinational spectrum. MRF provides an interface to Army Battle

Command System (ABCS) equipment allowing commanders and their staffs to train with their

"go-to-war" systems.

3.1 MRF v7.0 Federation Architecture

Figure 1 depicts a high level view of the MRF v7.0 Federation Architecture.

Core Federation Site

LOWER Enclave

Compound Federates

Logistics Site

LOWER Enclave

RTI

Distributor

JDLM

RTI (MRF v7.0 FOM)

WARSIM

Bridge

JECC-L

FMT-RJNEM

RTI (WARSIM FOM) RTI (MRF v7.0 FOM)

RTI

Distributor

AARS/ISM

C2 Network

C2 Network

MIDDLE Enclave UPPER Enclave

RTIC2 Network

RTIRTI (WARSIM FOM) RTIRTI (WARSIM FOM)

UE WAN

Rialto

1 2

AARS

WARSIM-

WIM (ME)

TCSP

WARSIM-

WIM (UE)

Rialto2Rialto 1 Radiant

Mercury

Distr. Site

LOWER Enclave

C2 Network

WARSIM

Tech Fwd Enhanced

BE Server

MUSE

DIS Network

WARSIM-

WIM (LE)

Figure 1. MRF Generic Target Federation Architecture - High Level

Figure 2 depicts the high level view of the MRF-based JTTI+K federation, which includes the

Korean Simulation Systems (KSIMS).

KBSC KASC

LE (Operating at SECRET-RELROK)

Compound Federates

LESD

Operating at SECRET-RELROK

JDLM

RTI (MRF v7.0 FOM)

WARSIM

Bridge

JECC-L

FMT-RJNEM

RTI (WARSIM FOM) RTI (MRF v7.0 FOM)

AARS/ISM

C2 Network

C2 Network

ME (Operating at US SECRET) UE (Operating at TS/SCI)

RTIC2 Network

RTIRTI (WARSIM FOM) RTIRTI (WARSIM FOM)

UE WAN

Rialto

AARS

WARSIM-

WIM (ME)

TCSP

WARSIM-

WIM (UE)

Rialto2Rialto 1 Radiant

Mercury

RESA

CJ21

CJB

KFMT

CSSM

CH

Operating at SECRET RELROK

CG

Schriever

Operating at SECRET RELROK

MDST

RTI (MRF v7.0 FOM)

C2 Network

ACE-IOS

DIS Network

RTI (MRF v7.0 FOM)

C2 Network

STAAR

RTI

Distributor

RTI

Distributor

RTI

Distributor

RTI

Distributor

WARSIM-

WIM (LE)

AFMSTT

MUSE /

AFSERS

Figure 2. JTTI+K Federation Architecture

3.2 JLCCTC MRF Components

JLCCTC MRF is a system of systems that supports training as a standalone Army federation as well as serving as the foundation of several Joint federations. In addition, MRF itself is primarily composed of models that are known collectively as jlcctc. The major components of JLCCTC

MRF are identified below.

jlcctc is the core component of JLCCTC MRF and consists of the following Army components:

AARS (After Action Review System) provides commanders and staffs with the means to evaluate training and processes.

ISM (Independent Stimulation Module) allows the injection of Master Scenario Events

List (MSEL) actions into the federation, provides the exercise controller interface for

JNEM and manages information flow between the JNEM and live MC systems.

JDLM (Joint Deployment and Logistics Model), aka LOGFED, provides high-fidelity combat service support (CSS) information for the training audience. In MRF, JDLM can optionally share CSS functions with WARSIM. WARSIM executes unit-level CSS, and

JDLM executes direct support (DS) and general support (GS)-level CSS. The exchange of sustainment information between the two simulations is analogous to the link between unit and DS level of support. JDLM only models US forces.

JNEM (Joint Non-kinetic Effects Model) models the changing satisfaction levels of civilian population groups as the simulation exercise unfolds. Population groups may represent a political faction, tribe, or clan, business organization, or non-governmental or international organizations, such as the Red Cross or United Nations. JNEM monitors events and object updates produced by the combat simulations and uses this data to adjust simulated satisfaction levels for the civilian population groups. JNEM publishes satisfaction objects, which are used by ISM to construct reports to the Training Audience.

JNEM can also cause events to occur in the combat models.

RIALTO (including Radiant Mercury, a cross domain solution) is the software that bridges two federations operating at different levels of classification. RIALTO allows

HLA federation coordination messages to pass both high to low and low to high. In MRF v7.0, simulation data is only allowed to pass low to high.

TCSP (Tactical Communications Support Processor) is a modified version of the CSP used by the intelligence community at unified, specified and major commands worldwide. The modifications included in TCSP focus on making the CSP system transportable and capable of interfacing with other tactical systems in the field.

WARSIM (Warfighters’ Simulation) trains Army and Joint commanders and their staffs for wartime, stability operations and peacetime. It portrays post-employment operations like war termination and post-conflict operations including restoring order, supplementing civilian government, providing humanitarian assistance, redeployment, reconstitution and demobilization. It supports planning, decision-making and operational execution of commanders and staffs from battalion through theater-level.

WIM (WARSIM Intelligence Module) is the Intelligence component of WARSIM. It simulates intelligence asset behavior and provides training to the Intelligence training audience

MRF is the base federation used to support Army simulation-stimulated training and consists of the following components:

jlcctc, as defined above

Archer is an after action review tool that provides commanders and staffs with the means to evaluate training and processes. Archer is a “Friend of the federation”, which is not a fielded MRF component.

BE Server (Basic Encyclopedia Server), aka ACEIOS, maintains the state of fixed sites within the federation.

FMT-R (Federation Management Tool-Reloaded) is used by technical control personnel to start, stop, and manage the federation. It provides a global perspective on the evolution of federation time, the health and status of the communications channels, and

HLA object information.

IEWTPT (Intelligence and Electronic Warfare Tactical Proficiency Trainer) trains commanders and battle staffs to focus intelligence assets that the commander deems most important; trains intelligence units and intelligence warfighters on the ability to understand the commander's needs and on how to use the collectors and processors available to provide the commander with the required information. IEWTPT is a “Friend of the federation”, which is not a fielded MRF component.

MUSE (Multiple Unified Simulation Environment) simulates unmanned aerial vehicles. In MRF, it is driven by feeds from BEServer and feeds state data back to

ASCCE and MC.

JECCL (JLCCTC Enumerations Cross Checker – Live) is used by technical control personnel to monitor other federates’ adherence to the agreed upon data standard set forth in the Master Enumerations List (MEL). Whenever a federate sends data which is invalid, JECCL flags it and provides reports reviewable by tech control and developers via a web interface.

JTTI is a federation formed from a Memorandum of Agreement (MOA) between the Army and

Air Force, and consists of the following components:

MRF, as defined above

AFMSTT (Air Force Modeling and Simulation Training Toolkit) provides the authoritative representation of air, space, and cyber power for U.S. Title 10 training exercises and mission rehearsals conducted jointly in all major commands around the world. AFMSTT is comprised of:

o ACEIOS, which provides intelligence and BDA reporting, models fixed sites and air defense radar, and other information operations modeling. In JTTI it is also used to provide an entity feed for MUSE.

o AFMSTT Bridge, which provides translation between the internal Air Force federation and the JLCCTC MRF federation.

o AWSIM, which trains Air Force and Joint commanders and their staffs in air and space operations.

o AirbaseSim, which models the logistics needed for airbases.

JTTI+ is a federation formed of US simulations to support SECRET US ONLY exercises in

Korea and SECRET REL-J exercises in Japan, and consists of the following components:

JTTI, as defined above

MDST (Missile Defense and Space warning Tool), which provides detailed modeling of TBM and ICBM launch detection, flyouts, and predicted impact points.

RESA (Research, Evaluation and System Analysis Simulation) simulates naval warfare. It models all Navy objects (surface, subsurface, and air) and all of their threats and targets. All naval warfare areas are simulated.

JTTI+K is a federation formed of US and Republic of Korea (ROK) simulations to support

SECRET RELROK exercises in Korea, and consists of the following components:

JTTI+, as defined above

CG (Chang Gong) is a simulation of air operations used to support training of ROK forces.

CH (Cheong Hae) is a simulation of naval activities used to support training of ROK forces.

CJ21 (Chang Jo 21) is a simulation of ground operations used to support training of

ROK forces.

CJB (Cheon Ja Bong) is a simulation of marine corps activities used to support training of ROK forces.

CSSM (Combat Service Support Model) is a simulation of combat service support used to support training of ROK forces.

KFMT (Korean Federation Management Tool) is a federation management tool used to support training of ROK forces.

STAAR (System for Theater-level After Action Review) is an after-action tool used to support training of ROK forces.

4 Federation Technical Guidance

A MRF federation manager uses several software applications that are included in the JIS

Software Installation Package to initialize, execute, operate and maintain a healthy federation.

Most operations are performed from FMT-R; however, when problems arise, a federation manager should also be able to utilize output from the RTIExec window, understand how and when to use the RTI Console, be aware of key RTI RID parameters and environment variables, be able to employ the SITH, and possess basic Linux network debugging skills. A Federation

Manager’s job includes the following:

Distributing federation initialization information and cross-federate data configuration files before exercise execution

Creating and destroying a federation execution with respect to its existence and membership.

Managing federation execution participation via federation joins and resigns.

Coordinating federation-wide operations including federation saves, restores, and synchronization.

Monitoring FMT-R and the RTIExec window for signs of federation abnormalities.

Resolving federation abnormalities with minimal impact to time advancement, federation membership, and operational activities.

Debugging infrastructure issues, both at the federate and federation levels.

5 Federation Setup

This section identifies tasks that should be completed prior to federation initialization.

Identify Network Time Server.

Request all operators configure box to use identified time server.

Request federates set their

FederateEndpoint in either RTI.rid or environment variable.

Become familiar with network and comms architecture.

Record IP Addresses, VOIP numbers, usernames/ passwords. Verify connectivity from federates to License Serve, RTIExec, and Distributor Nodes, and between Distributor Nodes. Plan to use distributor nodes?

Configure FederationInterconnect section of RTI.rid Yes

Identify IPAddress of VIRTC License Server and RTIExec box.

Install JIS Software.

Post “data products” to JIS website.

Inform federates data is available

Request all Federates configure their MTU to 1400.

No

Start RTIExec and FMT-R. Verify federates can join the federation.

Provide IPAdresses to network personnel at each site to create firewall exceptions.

Use the FMT-R’s “RTI Version” column to verify all federate’s are using the same RTI version.

Pre-Federation Tasks Flowchart

Figure 3. Pre-Federation Tasks Flowchart

Review/record network and communications architecture. Gather a list of federates and their associated IP addresses, including RTIExec, Distributor Nodes, and VIRTC License

Manager.

Provide IP Addresses to network personnel to add firewall exceptions o Each federate will require network communication to

Every other federate (unless distributor nodes are used, then only to every other federate served by the same distributor node)

The RTIExec

RTI License Manager on ports ranging from 1024 – 65535.

o Federates will require access to port 8080 on the FMT-R.

o Federates should permit ICMP/ping/traceroute from every host.

o Federates should permit access to ssh/scp (port 22) from the RTIExec (for retrieval of LRC Save files, if performing local saves).

If the event is distributed (i.e., federates execute at geographically distributed locations) the RTI.RID file may be modified to enable the use of distributor nodes.

Verify federate-to-federate network connectivity and federate RTI parameter definitions

(e.g. FederateEndpoint).

Configure each federate system to use a time server.

Post common data products on the JIS website.

Request each federate system reduce MTU settings from default of 1500 to 1400 (when network architecture includes remote sites connected via TACLANE.

Publish the JIS website address and other federation configuration information.

Request each federate system set FederateEndpoint.

5.1 Network Topology Effects on the RTI.RID

5.1.1 Distributor Nodes

This section is network topology dependent. If all federates are on the same LAN, then skip this section. If the event is distributed over a WAN, and more than one site contains multiple federates, then using the HierarchicalEventChannels federation interconnect strategy (instead of the default CollocatedEventChannel) eliminates redundant messages over WAN links, helps alleviate the load of propagating reliable data from the local federate, and may help reduce the amount of work required to configure network port/ipaddress filtering.

Comment [BDY1]: Add diagram of normal use case star topology

Figure 4. Sample Topology without Distributor Nodes

Figure 5. Sample Topology with Distributor Nodes

Changing federation interconnect strategies requires additional configuration steps in the

RTI.RID file, and an additional dedicated computer (loaded with the RTI software) to serve as the distributor node at each distributed site. A distributor is not required at the site running the

RTI Exec process. When the RTI Exec is contacted by an RTI distributor at a distant site, it will automatically start/spawn its local distributor process.

1. Edit the RTI.rid file

Change the StrategyToUse to HeirarchicalEventChannels and define all distributors, listing the computers that will connect to each distributor as children. Create a topology that mirrors the underlying physical network WAN links taking into account the highest bandwidth/lowest latency links. The computer where the RTIExec will run must be a host and will run a launcher for the rest of the distributors defined as hosts.

Examples of the modifications for this section are listed below:

2 site topology. One site has a specific host and a subnet range as child federates. The other site has only a subnet range for child federates.

(FederationInterconnect (StrategyToUse HierarchicalEventChannels) (HierarchicalEventChannelsOptions (Topology (NSC_Site (ParentNode null) (Host <Machine IP or name of Host running Distributor>) (Children (Child1 20.20.10.2/32) ;; Specific host is a child of NSC (Child2 20.21.1.0/24) ;; All hosts on subnet 20.21.1.x are also children of NSC

(Hurlburt_Site (ParentNode null) (Host <Machine IP or name of Host running Distributor>) (Children (Foobar1 30.20.10.0/24) ;; All hosts on subnet 30.20.10.x are children of Hurlburt

3 site, flat topology. This looks the same as the previous example except a third site that has another subnet range for child federates.

(HierarchicalEventChannelsOptions

(Topology (NSC_Site (ParentNode null) (Host <Machine IP or name of Host running Distributor>) (Children (Child1 20.20.10.2/32) ;; Specific host is a child of NSC (Child2 20.21.1.0/24) ;; All hosts on subnet 20.21.1.x are also children of NSC

(Hurlburt_Site (ParentNode null) (Host <Machine IP or name of Host running Distributor>) (Children (Foobar1 30.20.10.0/24) ;; All hosts on subnet 30.20.10.x are children of Hurlburt

(FtLee_Site (ParentNode null) (Host <Machine IP or name of Host running Distributor>) (Children (Foobar1 40.20.0.0/16) ;; All hosts on subnet 40.20.x.x are children of Ft Lee

3 site, tree topology. This looks the same as the previous example except the ParentNode values are different.

(HierarchicalEventChannelsOptions (Topology (NSC_Site (ParentNode null) (Host <Machine IP or name of Host running Distributor>) (Children (Child1 20.20.10.2/32) ;; Specific host is a child of NSC (Child2 20.21.1.0/24) ;; All hosts on subnet 20.21.1.x are also children of NSC

(Hurlburt_Site (ParentNode NSC_Site) (Host <Machine IP or name of Host running Distributor>)

(Children (Foobar1 30.20.10.0/24) ;; All hosts on subnet 30.20.10.x are children of Hurlburt

(FtLee_Site (ParentNode NSC_Site) (Host <Machine IP or name of Host running Distributor>) (Children (Foobar1 40.20.0.0/16) ;; All hosts on subnet 40.20.x.x are children of Ft Lee

2. Start the RTI Exec Process.

3. Start the RTI Distributors defined in the RTI.rid file at each remote site by entering the following command into a terminal:

runlauncher

You will receive confirmation that the distributor is sitting there waiting.

4. Join federates using normal join procedures.

When you join the first federate at your site, your distributor will show that it is initializing the site and then you will see it entering the event loop to process federate requests.

5. A couple of tips:

If a federate from a distant site tries to join the federation and fails but joins successfully on a second attempt then the federation time stops advancing, use the RTI Console LBTS

Master print_state command to see if any LBTS nodes have associated CORBA errors.

If so, use the check_node_consistency command to clear out the bad node.

To see the distributor federates, use the RTI Console list_federation_nodes -tree –s command.

Figure 6. Example MRF Federation Layout Using Distributor Nodes

5.2 Configuring System Time

Each system should be configured to use a network time server before being federated. An alternative is to set the system’s time before being federated. If a system’s time is changed after the system is federated, federation time will stop advancing. Verifying correct time settings at startup prevents having to debug a difficult and often fatal federation time issue later in the event. Time zone and daylight savings time changes do not affect the RTI.

5.3 Verify Network Connectivity and Federate RTI Parameters

Several hard-to-diagnose federation issues can be avoided if a federate-to-federate connectivity check is done prior to federation initialization. While performing this check, also verify that each federate has defined the FederateEndpoint variable (refer to Appendix B of this document for more information.)

To perform a federate-to-federate network connectivity check, use the list of IPAddresses gathered earlier, and ping each IPAddress from each federate computer. If there is a problem, contact the network administrator.

5.3.1 Check TCP Connections

Some networks disable ping and traceroute, making it difficult to verify network connectivity. It is also possible to verify network connectivity by connecting to another open port, such as HTTP

(80) or ssh(22).

Debugging federate-to-federate communications issues can be difficult, especially when one of the federates is behind a firewall that blocks ping/traceroute.

5.3.1.1 Debug Federate-to-Federate Connection

Each federate maintains a TCP connection to every other federate and the RTIExec. Verifying that a federate process has a TCP connection to every other federate’s IP Address can narrow down the amount of debug time. In UNIX, everything is a file including socket connections (IP

Address:Port). Identify the federate’s process id by entering the following into a terminal

(replacing fedex (fedex is for verifying TCP connections from the RTIExec) with the federate process name):

ps –aef | grep –I fedex

To view the TCP connections that process ID owns, enter the following into a terminal:

/usr/sbin/lsof –p <fedex process id>

Look for the inbound and outbound IP addresses, specifically ones that appear incorrect.

5.3.1.2 Debug Blocked Ports

Note: netcat (nc) is a great tool for verifying port availability. Configure netcat as a server on one machine to listen on a specific port :

nc –l <port>

Use netcat as a client to connect to the server on ipaddress listening on port:

nc <ipaddress> <port>

*Note: tcpdump is great for viewing inbound and outbound network traffic; however, it may not be available on some computers due to operating system lockdown restrictions.

5.3.2 Duplicate IP Address

On rare occasions, strange behavior may be an indication that a duplicate IP Address exists on the network. Symptoms to watch for are:

SSH sessions stop responding

SSH warning about keys changing

FMT-R clients stop responding (if duplicate IP Address is FMT-R machine)

Federation time advancement issues

If possible, shut off the machine with suspected issues and attempt to ping the IP Address to confirm the existence of a duplicate IP Address.

5.4 Configuring Maximum Transmission Unit (MTU)

Every network link has a maximum packet size called the link's MTU (Maximum Transmission

Unit). The full path from one computer to another may travel across many links with different

MTUs. The smallest MTU for all the links in a path is the path MTU. If a packet starts out on a network segment with a large MTU, it may arrive at a link with a smaller MTU and be too big to fit – dropping the packet, causing the loss of federation data.

The TACLANE encryption devices that connect NSC and LESD have a MTU set to 1500 bytes.

The value 1500 is also an industry standard for point-to-point network connections. However, TACLANEs add data to the packets that are sent between the sites - overrunning the 1500 limit.

In order to accommodate the additional data in the packet, the MTU for MRF components must be set to a lower value that allows for the additional data bytes. The way forward is to push to have this system level setting assigned to 1400 as part of the lockdown, removing the need to manually set it on every computer.

An MTU mismatch on a network interface (possibly crypto devices, assuming Ethernet encryptors) introduces the potential for data packets to be split on the transmitting side but then not reassembled at the other end because the other network device assumes the packet is whole and not split. In JLCCTC MRF, a path MTU of 1400 has been selected as a value that has been demonstrated to work with network security.

Note that for exercises involving components run at NEMOC/Ft. Bragg, additional encryption headers now necessitate using an MTU of 1250.

5.4.1 Verifying MTU

1. Verify the MTU value by sending an unfragmentable ping to each remote federate. You are looking for the smallest size that doesn’t require fragmentation, and works for all federates.

a. Linux: "ping -c 4 -s 1472 xxx.xxx.xxx.xxx".

b. Windows: “ping xxx.xxx.xxx.xxx –f –l 1472”.

2. Lower the size the packet in increments of 10 (e.g. 1472, 1462, 1440, 1400) until you have a packet size that does not fragment.

3. Begin increasing the packet size from this number in small increments until you find the largest size that does not fragment. Add 28 (or 48 if using PPPoE) to that number

(IP/ICMP headers) to get the optimal MTU setting. For example, if the largest packet size from ping tests is 1462, add 28 to 1462 to get a total of 1490 which is the optimal MTU setting.

Table 1. Typical Packet Sizes

Packet Size Description

1500 The biggest-sized IP packet that can normally traverse the Internet without getting fragmented.

Typical MTU for non-PPPoE, non-VPN connections.

1492 The maximum MTU recommended for Internet PPPoE implementations.

1472 The maximum ping data payload before fragmentation errors are received on non-PPPoE, non-

Packet Size Description

VPN connections.

1460 TCP Data size (MSS) when MTU is 1500 and not using PPPoE.

1464 The maximum ping data payload before fragmentation errors are received when using a PPPoE-connected machine.

1452 TCP Data size (MSS) when MTU is 1492 and using PPPoE.

576 Typically recommended as the MTU for dial-up type applications, leaving 536 bytes of TCP data.

76 TACLANE header size.

48 The sum of IP, TCP and PPPoE headers.

40 The sum of IP and TCP headers.

28 The sum of IP and ICMP headers.

5.4.2 Changing MTU in Windows Vista or Windows 7

1. To see which interfaces your computer has, type:

o netsh interface ipv4 show subinterfaces

2. To change the MTU, type o netsh interface ipv4 set subinterface "Local Area Connection" mtu=nnnn store=persistent

“Local Area Connection” is the name of the network connection on your computer, from the list obtained above.

nnnn is the desired value for MTU.

3. Reboot after making the change.

5.4.3 Changing MTU in Linux

On the Linux computers, simply use neat – a setting in the lower left corner of the tab where the

IP's are set.

5.5 Federation Data Products

Prior to actual exercise start, before the initial instantiation of the federation, federation-managed data products should be placed on the JIS Website (see section 5.6.1) for easy distribution and configuration management. The Federation Data Manager’s Manual contains more information about these federation data products. This section identifies cross-federate information sharing, including the producers and consumers of the data.

5.5.1 Universe of Entities (UOE) Files

The Universe of Entities files are a collection of nine files containing class and instance data for a scenario that WARSIM plans to use. It provides WARSIM scenario-specific initialization data used by other federates. These files should be provided prior to an event, and are also posted to

FMT-R Web Server.

Produced by: WARSIM

Consumed by: ISM /AARS, JDLM

5.5.2 Fixed Sites XML File

The Fixed Sites XML and Fixed Sites XLS files are generated by the Fixed Sites Correlation Tool

(FSCT). They contain a set of fixed sites that have been selected by the training audience or exercise control for modeling across the federation.

Produced by: FSCT

Consumed by: ACE-IOS, JNEM, JDLM, MUSE, WARSIM

5.5.3 Detailed Supply Mapping File (DSMF)

The Detailed Supply Mapping File contains supply information data exchanged by JDLM and

WARSIM. The file is developed and maintained by the JLCCTC Data Working Group.

Produced by: JLCCTC Data Working Group

Consumed by: JDLM, WARSIM

5.6 Publish Federation Configuration Information

The technical control operator should disseminate the following information to all participating federates:

The JIS Website HTTP address (e.g. http://ipaddress:8080/JLCCTC/)

The start date of the exercise

5.6.1 JIS Website

The JIS website is the main location where federates obtain federation configuration information.

It contains settings and files that federates need to configure their models to be able to join and interoperate with the federation, such as:

Confederation Name (e.g. Confederation_1)

VIRTC License File (e.g. 27000@ipadddress)

FOM (e.g. OMT, FED, etc.)

RTI.RID

Data Products (e.g. UOE, DSMF, Fixed Sites)

The website is typically run from the same computer as FMT-R and is accessed via a web browser at http://ipaddress:8080/JLCCTC/. To add files to the website, simply copy them into

/home/jisuser/JIS/Java/jboss-5.1.0.GA/server/default/deploy/JLCCTC.war/ directory.

5.6.2 FMT-Lite Webpage

The JIS website contains fmt-lite.jsp, which provides a lightweight, web-based summary of the federation’s current state available to any host with a web browser on the simulation WAN.

Although much less capable than the Remote FMT-R Client, it also uses far fewer resources and http://ipaddress:8080/JLCCTC/ http://ipaddress:8080/JLCCTC/ bandwidth than the Remote FMT-R Client. This is the preferred tool for remote sites interested in monitoring the state of the federation.

5.6.3 Remote FMT-R Client

The JIS website contains FMT.jnlp, which provides remote access to a FMT-R client interface, enabling others to monitor the status of the federation via the FMT-R. The computer will require a JRE or JDK to execute the .jnlp file.

1. From a Java Web Start (JWS) plug-in enabled browser, type the following address into the browser: http://</FMTR machine ip number>:8080/JLCCTC/FMT-R.jnlp.

a. If the browser is not JWS plug-in enabled, you will prompted to locate an application that is capable of handling the .jnlp extension. Locate javaws in either a JDK or JRE’s bin directory.

2. Once the FMT-R client has launched, left-click the FMT menu, and select Connection

Information.

3. Select Yes from the Federation Connect dialog box.

4. Type the host name or IP where the FMT-R server is running and press ENTER.

5. Select the federation to connect to, and click OK.

Note that the interchange of data between the FMT-R remote client and the server requires a non-trivial amount of bandwidth (measured at about 250 kb/sec). For this reason, the number of

Remote FMT-R Clients should be kept to a minimum and most sites and users should instead use the FMT-Lite webpage instead.

6 Federation Execution Operations

Please refer to Appendix A for a flowchart key.

6.1 Start JIS Infrastructure

Perform steps applicable to your operational environment (e.g. not all environments use

ADAPTORs).

Figure 7. Start JIS Infrastructure

Step Action Command(s) Expected Outcome

1 Notify all federates to prepare for initialization or restore, but do not join until notified.

2 Start the license server. Note: a typical

JLCCTC MRF installation will most likely be configured to use the RTI License

Server installed and executed by

WARSIM. In such cases, you can skip this step.

Enter: runLM The License Manager is launched.

3 At the computer where you will run the

RTI, start the RTIExec in a new terminal

Enter: runrti The RTI is launched.

4 On the computer running the RTIExec process, open a new terminal and start the

RTI Console.

Enter: console The RTI console is launched.

5 At each computer hosting a distributor node, start the distributor node in a new terminal.

Enter: runlauncher The distributor node is launched.

6 On the computer where you will run the

ALSP ADAPTOR, open a new terminal, and start the ALSP ADAPTOR for RESA.

Enter:

runADAPTOR –c 1

–a 2 –n RESA -v

Once the MRP is started, it should show that two connections have been established, and the ADAPTOR window should also show “connected to actor” and “connected to MRP”.

7 On the computer where you will run FMT-

R, open a new terminal, and start FMT-R.

Enter: runFMTR The FMT-R GUI appears.

6.2 Initialize the Federation

Figure 8. Federation Initialization Flowchart

Table 2. Federation Initialization Steps

Step Action Command(s) Expected Outcome

1 Notify all federates to prepare for full federation initialization, but do not join until notified.

2 Start the JIS

Infrastructure

3 Request WARSIM

Technical Control to prepare for initialization, and join the Federation

Bridge when ready.

WARSIM federate appears in the FMT-R

Summary tab.

WARNING: After a federate joins the federation, they may NOT change their system clock time – it will cause the federate to stop advancing time.

4 Wait until the federation bridge has joined the federation

Step Action Command(s) Expected Outcome

5 Verify that WARSIM is ready for object registration.

Even though the federation bridge has joined, not all WARSIM federates may be joined to the WARSIM federation.

6 When WARSIM tech control indicates they are ready, initiate a Register

Objects from the FMT-R window.

On the FMT-R window, select the

ALSP pull down menu and select

Register Objects.

The Register Objects Signal pop-up window appears.

7 Send the Register

Objects signal to the federation.

Select OK on the Register Objects

Signal pop-up window.

Register Objects signal is sent to the

JLCCTC side of the federation.

8 Notify WARSIM

Technical Control that a

Register Objects signal has been sent.

WARSIM Technical Control operators need to know when the Register Objects signal has been sent so they can enter the scenario date and begin sending objects to the JLCCTC federation.

9 Wait for WARSIM to indicate object registration is complete.

10 Ask WARSIM to indicate when a save can be taken.

11 Perform federation save procedures.

Successful WARSIM and FMT-R federation save complete.

12 Request operators for all remaining federates to join the federation for initialization.

All joined federates appear in the FMT-R

Summary tab.

13 Wait for federates to appear in FMT-R

Summary tab.

14 Initiate an object registration from the

FMT-R

Select FMT pull down menu, select Register Synchronization

Point, Enter

REGISTER_OBJECTS, click OK.

REGISTER_OBJECTS synchronization point is sent to the federation.

15 Wait for federates to finish object registration.

16 Perform Federation

Refresh.

Federation refresh is complete, game is moving forward at a positive ratio (e.g.1:1).

17 Ask WARSIM to join middle and upper enclaves, and indicate when WARSIM internal refresh is complete.

WARSIM internal refresh is complete.

18 Perform federation save procedures.

Successful federation save complete.

6.3 Save the Federation

The federation stops advancing time while the save is executing.

It is important to understand the save scheme defined the in the RTI.RID file to know where the

LRC save files are stored, and where the save files need to be located for a successful restore.

The RTI.RID file allows the save scheme to be changed before the federation is created (this parameter is checked for consistency) by changing the FederationSection.Federation-

Executive.SaveAndRestoreLocally parameter to either Yes or No.

A value of “Yes”, also referred to as “Distributed Saves”, indicates that federates will store their

LRC files on their local hard drives. In an environment where federates are geographically distributed across a WAN, “Distributed Saves” execute in roughly 1-3 minutes (federation time stops advancing during a save). This scheme allows faster saves and less simulation downtime, but each federate’s LRC save file must be manually collected after each save in order to restore the federation after a catastrophic node failure (network becomes unavailable, building loses power, etc.).

A value of “No”, also referred to as “Centralized Saves”, indicates that the LRC save file of each federate is automatically transferred to the machine running the RTIExec. In an environment where federates are geographically distributed across a WAN, “Centralized Saves” execute in roughly 3-10 minutes (federation time stops advancing during a save). This scheme increases federation unavailability, but automatically transfers each of the federates’ LRC save files as part of the saving process.

NOTE: The size of a LRC save file is determined by several factors, but is typically larger than

100 MB. It is important to be aware of the amount of remaining disk space on each federate’s hard drive as an event progresses. If a federate has insufficient disk space to store the LRC save file, the save will fail.

For JLCCTC MRF v7.0, the default setting is to use Local Saves and Restores.

Figure 9. Federation Save Flowchart

Table 3. Federation Save Steps

Step Action Command(s) Expected Outcome

1 Insure that

WARSIM is more than 10 minutes past initialization (i.e.

Run State) and that it is not currently late joining any federates and/or middle or upper enclaves.

2 Schedule a federation save from FMT-R.

From the FMT-R menu, select ‘Schedule

Federation Save’.

The schedule federation save window appears.

3 Enter a time and a save label.

Enter a time and a save label into the dialog.

The current simulation time can be found on the upper left of the

FMT-R window. The save label should not contain characters that are not valid file names for the Linux and Windows operating systems.

Click the OKAY button.

The next scheduled save appears above the tabs in the FMT-R application.

Step Action Command(s) Expected Outcome

4 Verify the save is complete.

When the simulation time for the save is accomplished, a

‘SAVING’ banner will flash across FMT-R. The banner disappears when the save is complete. The status pane at the bottom of the FMT-R Summary tab displays ‘Federation Save

Complete’.

5 (Optional) If this save will be ported for restore elsewhere, export the federation save file from

FMT-R.

In FMT-R, click on the

FMT menu item and select Export

Federation Save.

The saved file has the naming convention of

~/JIS/run/FMTR/federationSaveName_SAVE.xml

*NOTE: If a federation is being created from another federation's save, and the new federation will use a different federation execution name (Confederation_1 vs.

Confederation_3), then an xml node must be edited in the

*_SAVE.xml file. Open the xml file in your favorite text editor and look for the xml <entry> node with the name attribute of FederatonName, it will look similar to <entry name="FederationName" value="Confederation_1">. Replace the value="..." attribute with the new federation execution name such as value="Confederation_2".

6.4 Restore the Federation

Figure 10. Federation Restore Flowchart

Table 4. Federation Restore Steps

Step Action Command(s) Expected Outcome

1 Notify all federates to prepare for full federation restore but do not join until notified. The JDLM operators require the save label to prepare for the federation restore.

2 Start the JIS Infrastructure

3 Click on the Save Summary tab in

FMT-R.

Save Summary tab shows

4 Select the save to restore via the

‘Save Names’ tree.

Step Action Command(s) Expected Outcome

5 Request WARSIM Technical

Control to prepare for a restore without run-up and join the federation bridge when ready.

WARSIM federate appears in the FMT-R

Summary tab.

6 Wait for WARSIM Technical

Control to notify that they are in the

“Run” state. Ensure BEServer is time regulating.

WARNING! Failure to do this step could result in starting the restore process over again.

NOTE: WARSIM joins the federation with a time of 10 seconds. An indication that

WARSIM is in the run state is when time advances past 10 seconds. When time has advanced, ask WARSIM if they are in the

“Run” state.

7 Request operators of the Non-

ADAPTOR federates to join the federation for a restore.

All Non-ADAPTOR federates appear in the

FMT-R Summary tab.

NOTE: LOGFED will be unable to successfully restore if the federation advances past 10 minutes of simulation time. If the restore cannot be issued before 10 minutes of simulation time has elapsed, ask the

WARSIM operator to pause until the restore can be executed.

8 Join the ALSP ADAPTOR for

RESA

Enter: join in the

ADAPTOR terminal

WARNING! RESA should NOT be attached to the ADAPTOR during the restore process.

9 From the Summary Tab, go to the

FMT menu and select “Schedule

Federation Restore”.

Federation Restore dialog window appears.

10 Select the desired save to restore. In the Federation

Restore dialog, select the save label to restore and click the

OK Button.

Flashing text indicates the restore is in progress.

NOTE: If a list_federation_nodes is performed in the RTI Console during a restore, it is not uncommon to see a single star next to federates.

11 Start the MRP and Instruct the

RESA operator to join the federation

12 Verify JNEM operator has set a positive game ratio after the restore since they restore to a zero game ratio.

13 Verify restore is complete. If restore appears to be complete, and the federation bridge is not advancing time then verify with

WARSIM Technical Control that they have skipped “Run-up” (or have started the internal WARSIM run-up).

Restore banner stops flashing and status window shows restore was successful.

6.4.1 Restore from Ported Save

Restoring a federation from ported saves can be a great way to test the suitability of a save for potential restore, debug federation problems in a test environment, replicate issues observed in the production federation, etc. All federates that were in the federation during a save must be present for the restore.

If any federates are missing, and their LRC save files can be obtained, a surrogate can be used to decoy the federate’s presence. Each federate needs at least the LRC save file, and probably the federate’s save file. This section describes how to restore a federation from another federation’s save, from the perspective of the tech control operator.

These instructions are for performing a restore for a distributed save scheme (Refer to section 6.3 for a description of the save schemes). If performing centralized saves, simply place the LRC file of each federate in the ~/JIS/run/RTI/ directory of the computer running the RTIExec process to perform a restore from a ported save.

Table 5. FMT-R Steps to Restore from Ported Save

Step Action Command(s) Expected Outcome

1 Export the federation save file from FMT-

R.

In FMT-R, click on the FMT menu item and select Export Federation

Save.

The saved file has the naming convention of

/opt/JIS/run/FMTR/federationSaveName_SAVE.x ml

*NOTE: If a federation is being created from another federation's save, and the new federation will use a different federation execution name

(Confederation_1 vs. Confederation_3), then an xml node must be edited in the *_SAVE.xml file.

Open the xml file in your favorite text editor and look for the xml <entry> node with the name attribute of FederatonName, it will look similar to

<entry name="FederationName" value="Confederation_1">. Replace the value="..." attribute with the new federation execution name such as value="Confederation_2".

2 Obtain the FMT-

R’s LRC save file.

/opt/JIS/run/FMTR/federation-

ExecutionName_saveName_-federateName_federateHandle.save

(e.g. Confederation_1_-

GameSave020130_FMT-R_1.save)

3 Copy both the save, and the xml files to the new FMT-R’s

/opt/JIS/run/FM

TR directory.

4 Run the new

FMT-R.

Enter runFMTR in a new terminal window.

Step Action Command(s) Expected Outcome

5 Import the federation save xml file.

Click the FMT menu item, and select Import Federation Save.

Select the .xml file that was exported from the old federation and click the Open button.

6 After all other federates are ready, proceed with normal restore.

NOTE: The RTI Build Type of the restoring federate MUST be the same as the RTI Build Type of the saving federation. Gotchas include Linux

vs. Windows, and 32bit vs. 64bit.

Table 6. RTIExec Steps to Restore from Ported Save

Step Action Command(s) Expected

Outcome

1 Obtain and transfer the RTIExec save files.

/opt/JIS/run/RTI/federationExecutionName_saveName_fedex.save (e.g.

Confederation_1_GameSave020130_fedex.save)

And

/opt/JIS/run/RTI/federationExecutionname_saveName_fedex_om.save

(e.g. Confederation_1_GameSave020130_fedex_om.save)

2 Place both of the save files in the new federation’s

/opt/JIS/run/RTI/ directory.

6.4.1.1 SITH as a Surrogate

The SITH can be used in place of a real federate for a restore if the federate’s LRC save file is available. Typically, the SITH is used as a surrogate if one or more federates fail to restore from a save, or if a federate that was part of the…