The file's text, extracted by GovTribe without its formatting.
Stratagem BAA Q&A
| BAA: | FA8750-20-S-7007 |
| Date: | November 8th, 2019 |
| Focus: | Focus Area 1 |
General Questions
| Question 1: | What is the primary focus of this BAA (and Focus Area 1)? What is considered a success for the Stratagem program? |
| Answer 1: | The focus of the overall announcement is to work toward harnessing the recent state of the art advances in AI game playing and apply it to military relevant scenarios. Focus Area 1 is particularly geared toward developing an AI player that can meet or exceed expert human play in a game based around operational-level wargaming/planning and could be used a Red Adversary as well as a Blue Assistant. |
The ultimate goal of the program is for a human to use this as a vetting source for a selection of COAs against an adversary that might not necessarily use traditional, expected means. Shifting from static planning and wargaming to dynamic planning. Go against a smart adversary that may do things that are unexpected or cunning.
Success for a performer under Focus Area 1 is achieving a Red AI Player that can beat an expert human planner and force that expert human planner to think in unconventional ways while taking advantage the benefits offered by a Blue AI Player assistant. Success is the realization of a Red AI Player that can truly challenge an operational planner/wargamer.
| Question 2: | From your perspective, what are some of the most challenging aspects specific to Focus Area 1? |
| Answer 2: | From the Technology Challenges listed in the BAA: 1) representation of the game, 2) the lack of existing training data and replays (limited data), and 3) developing a long term strategy. Additionally, there is the inherent complexity of multi-domain operations, how to model them and integrate them. |
| Question 2a: | What is the performer developing vs. what AFRL is providing? |
| Answer 2a: | The performer would create a human-expert-level AI player for the Stratagem Wargame (Red and/or Blue). AFRL is providing the Stratagem Wargame codebase, which includes example heuristic AI players and example scenarios. |
| Question 3: | What is the Stratagem Wargame supposed to do? What is its purpose? |
| Answer 3: | It provides answers to the following questions: |
· What does it take to win against a given configuration of Red Force (AI Player / Order of Battle)?
· For existing plans/COAs, what is the rough likelihood of meeting minimum/target success criteria?
· How much will it (relatively) cost to achieve objectives, to win?
· Casualties and equipment lost
· Supplies expended
· Time required
· What is the delta outcome of modifying an existing plan? What is the delta outcome of using different forces or having forces arrive later than intended?
| Question 4: | Will AFRL provide performers with the game scenarios (threat laydown, terrain, etc.), or are they just giving us the game and leaving us to develop the scenarios? |
| Answer 4: | The Government will provide initial scenarios and newer scenarios as the game evolves over time. The scenarios will be provided along with the Stratagem wargame code base via a DI2E BitBucket repository. |
| Question 5: | Will multiple runs of the game provide different results? Will a random seed be used to force consistent results across runs? |
| Answer 5: | Yes, many modules within the game model use stochastic elements. As well, some AI players may make different decisions across multiple runs if they are not deterministic. The intent is to see the range and frequency of the results that are generated over the course of a large number of runs (hundreds, thousands) for a single scenario input parameter configuration. |
The Government is looking at incorporating a random seed to force some level of determinism, at least at the model level (not taking into account possible non-determinism in AI players).
| Question 6: | Is AFRL expecting/desiring the performer to help create or inform the AI Unity Engine-based war game design or interfaces? |
| Answer 6: | Not at this stage for the wargame. The Government is looking for Red/Blue AI Player development for the time being. |
| Question 7: | Will AFRL provide a facility for performing Monte-Carlo runs for training the AI system? |
| Answer 7: | No. AFRL will provide the wargame and the API, whereas performers would be responsible for setting up the infrastructure necessary for their specific approaches for training/developing their AI players. |
| Question 8: | Are the “rules of the game” static? If they can change, can they change during an episode (a play of the game) or only between game episodes? |
| Answer 8: | The rules of the game would stay the same for each episode. Of course, the rules are based largely on a particular scenario, which defines: force configuration and placement, force objectives (i.e., for each player), and terrain configuration. A “scenario” in the Stratagem Wargame could be comparable to a “map” in the game of StarCraft 2. |
| Question 9: | Can the mission objectives (utility/payoff function) change during the course of play within a single game? |
| Answer 9: | No. |
| Question 10: | Given the nature of expert knowledge capture, is the Government also interested in the development of Cognitive Task Analysis (CTA) for measuring human performance? |
| Answer 10: | No. |
| Question 11: | Would we be allowed to manipulate the provided game to fit our representation? |
| Answer 11: | The game provides a player interface that exposes what a player should/can know about the environment and the other players acting within that environment. That game state data can be transformed/adapted by you after you receive the game state data to better fit the representation needed by your specific solution implementation. |
| Question 12: | What is the anticipated award timeframe for FY20 Focus Area 1? |
| Answer 12: | Awards are anticipated starting in Spring 2020 for FY20 Focus Area 1. |
| Question 13: | Do we need to develop our own wargame to get started? |
| Answer 13: | No. |
| Question 14: | What will human interaction with the game be like? |
| Answer 14: | The human user will have access to a GUI developed in Unity3D. This GUI will show the information that is contained within the standard wargame state information (detailed under the Wargame Questions section of this document), like terrain info, general game info, player info, units info, and so on. |
| The human user will see a hex grid with unit cards placed within appropriate cells. At a glance, the user will be able to tell the name, type, and combat factors of the unit. The user can hover over a unit to get information on the unit such as its strength, fatigue, disorganization, and its current set of orders. |
| When displayed, the orders for each unit will be depicted as game objects drawn over the hex map. These game objects will show what the order is when it’s supposed to occur. |
| The human user can add to these orders (an order chain), edit them, or remove elements from them using the mouse. |
| When the game proceeds into execution and after action phases, the human user can observe the effects of all the units carrying out their assigned orders. During the after action phase, human users can hover their mouse over battle/combat game objects that reveals the details of all the combat that occurred during the preceding execution phase. |
| Question 15: | Is it a requirement to have a license for a professional version of Unity? |
| Answer 15: | Yes. |
| Question 16: | Will AFRL also provide the human player interface along with the Stratagem Wargame? |
| Answer 16: | It is already part of the Stratagem Wargame codebase that will be provided. |
It should be noted that every element of the wargame’s development is being kept as modular as possible (Model-View-Controller). Thus, there is not one single visualization or GUI component to the game. So, for example, it’s possible that two human players could be using two different GUI/clients to connect to the game – so long as all clients utilize the same provided interfaces. One of the things that the Government set out to do at the beginning was to make sure that if someone wanted to create a client that could play the game via the command line interface, they certainly could.
| Question 17: | What will you be keeping track of in game replays for both human and AI players? |
| Answer 17: | For each turn/iteration of the game, the Government will keep track of all of the orders created during the planning phase, and all of the outcomes/events generated during the execution phase. |
| What the Government won’t be keeping track of is what the player was looking at or inspecting during planning and after action phases, at least not initially (i.e., similar to tracking where the camera was positioned throughout the game for a player in a StarCraft 2 replay). |
| Question 18: | Will contract performers have access to the game source code? |
| Answer 18: | Yes, performers will be granted read access to our DI2E BitBucket repository for the Stratagem game. Performers are free to develop their own scenarios for training purposes. While it would be possible for performers to modify the Stratagem game locally, it should be noted that any Red/Blue agents that are developed will have to be evaluated on our version of the codebase. |
| Question 19: | Assume multiple awards? Thoughts on options? Guidance on scope? |
| Answer 19: | In accordance with Part II, Section II.1.a of the BAA, individual awards will not normally exceed 24 months with dollar values normally ranging from $300,000 to $2M. In accordance with Part II, Section II.3, it is anticipated that only one award will be made under Focus Area 2 and one award under Focus Area 3. Multiple awards are anticipated under Focus Area 1. |
Options would certainly be welcome.
| Question 20: | Are you going to integrate with AFSIM or some other simulation at some point? |
| Answer 20: | AFSIM is not an integration candidate at this point. However, through Focus Area 3, it may be a target environment for Transfer Learning. |
| Question 21: | There are six tech challenges listed in the BAA. Do we need to address all six? |
| Answer 21: | No, these were more provided as things to take into consideration, but not necessarily challenges that had to be addressed for any given proposed technical approach. |
| Question 22: | Does proposing to or getting an award in Focus Area 1 prevent us from looking at Focus Areas 2 and 3 when they become available for whitepaper submission? |
| Answer 22: | No. |
| Question 23: | Do you eventually plan to ingest existing COAs, OPLANs, and TPFDDs? |
| Answer 23: | Not at the current stage. This will certainly be revisited at later stages of the Stratagem program. |
| Question 24: | From a Design of Experiments perspective, what would you vary? |
| Answer 24: | Anything that a SME or operational planner/wargamer might be interested in changing. For example: transportation schedules (when do troops arrive?), supply schedules, etc. Number of troops. Troop quality. Troop readiness. This could apply to both Blue and Red forces. |
| Question 25: | Is it possible to repeat a run, to turn off variance? Achieve determinism? |
| Answer 25: | The Government is looking at incorporating a random seed to force some level of determinism/repeatability, at least at the model level (not taking into account non-determinism in AI players). |
| Question 26: | What is the current TRL for the Stratagem program? |
| Answer 26: | The Stratagem Program is currently at TRL 2, with a goal of reaching TRL 3. The Stratagem Program is an early 6.2 research program. |
| Question 27: | What is the expectation with respect to interaction between performers? |
| Answer 27: | Free and open level of discussion between the performers. Free release of information would be required. There will be regular PI meetings 2-4 times a year. |
| Question 28: | Why wasn’t there a Stratagem Industry Day? |
| Answer 28: | The Government did not want to unduly constrain offerors’ creativity, rather that all ideas be encouraged and evaluated during the white paper process. |
| Question 29: | Is the Stratagem program related to the Flyleaf program? |
| Answer 29: | The Government envisions a future symbiotic potential, but the programs are independent and separate at this time. |
AI Player Questions
| Question 30: | How do we determine the relative quality of an AI player? |
| Answer 30: | An AI player is graded based on its ability to achieve its objectives in less (simulated/game) time, while reducing losses. How fast, how much land covered, what did we destroy? An AI player would also be judged on its flexibility to apply to a larger number of scenarios instead of only playing well in a smaller, focused number of scenarios. |
As well, efficiency for execution rate would be another metric, as the Government wants AI players that can rapidly play tens or perhaps hundreds of thousands of games in quick succession (Monte Carlo) so that we may perform experiments, analyze the robustness of COAs, compare COAs, find more optimal COAs, and so on. To that end, the Government needs a Red (and Blue) AI Player that can execute rapidly. As well, ultimately, a Blue AI Player should be able to examine the current state of an ongoing game and quickly game out the remainder of the game in order to quickly generate a number of diverse recommended COAs for a human player.
One of the underlying ideas here is that the Government doesn’t want a single learning agent that can only apply effectively to a single environment/scenario; rather, the Government prefers to have an agent that can play over multiple different exercises and scenarios. The agent should be flexible, not tied to a single static engagement/scenario.
The Government isn’t looking for a reactive system, but an active planning agent.
| Question 31: | Will performers be provided expert and/or novice player example playback results? |
| Answer 31: | No, a library of game replays is something that Stratagem will need to build over time. These results would be more a product of the program than an input. |
| Question 32: | How are you dealing with the lack of replay data like the kind that Deepmind AlphaStar had available? |
| Answer 32: | The Government will start off with generating replays with various baseline heuristic (AI) players. The goal is to build up a catalog/database of replays. |
| Question 33: | Is the Government also interested in the concept of how explainable artificial intelligence can be used to support the understanding of how Machine Learning can be developed into learning objectives? |
| Answer 33: | No. |
| Question 34: | Is the Government interested in the use of cloud-based systems to train machine learning agents such as AWS or Azure? |
| Answer 34: | Not critical or necessary, but is acceptable. However, it should be possible to easily detach/divorce a training regime solution from the cloud so that it can be instantiated locally. |
| Question 35: | Is the Government interested in 'purposeful mistakes' so that the Government can capture expert insights when they are dealing with error? |
| Answer 35: | No. |
| Question 36: | Are there any constraints on the AI behavior? |
| Answer 36: | The AI player (Red, Blue, etc.) will have to follow the rules of the game. It will only have access to the Player API in terms of what it should access to (game state information, available actions). Thus, for example, it should not be allowed for an agent to open an external socket to communicate with another agent outside of the confines of the wargame. |
| As well, when the game runs in “real-time” execution mode, while not a hard constraint, if the agent takes too long during execution to make a decision, it could very well miss the window in which its decision might have been timely/appropriate/effective. |
| Question 37: | Do we need to utilize the ml-agents package within Unity3D? |
| Answer 37: | The Government is not requiring any performer to utilize the Unity-specific ml-agents package. Different technical approaches may require a training solution other than what may be possible/feasible in the Unity ml-agents package. |
| Question 38: | Games are built with rules and heuristics that look like AI to the player but the model gives the designer a lot more control over the experience than machine learning. The goal of game AI is not to win but give the player a scalable level of challenges. Early levels teach the player the rules and over time the level of difficulty increases by combining the rules to provide more complex challenges. Is the goal of this project to build a tool where scenarios can be crafted and tweaked by a designer in a predictable, measurable way or is this an exploration of applying machine learning to a game? |
| Answer 38: | The majority of video/computer games have AI players with the end goal of providing fun for the players – in that the humans playing the game ultimately are allowed to be victorious. The ideal Red AI Player for that Stratagem program should not let any opponent (human or otherwise) win. |
The Government is interested in varying levels of AI difficulty and play styles (aggressive, early attack, scorched earth, defensive hunker-down, etc.), but the primary focus would ultimately be on developing an advanced Red AI player that is capable of beating human-expert-level planners/wargamers.
| Question 39: | The Government should be aware that potential adversaries know that we’re looking into machine learning, and they’re doing machine learning as well. What is the Government doing with that in mind? What are the Government doing to prevent an enemy from trying to exploit the weaknesses of any potential approach, machine learning or otherwise? |
| Answer 39: | No potential technological solution should be discarded due to potential exploitation. |
| Question 40: | Is there any interest in game theoretical techniques? |
| Answer 40: | Yes. |
| Question 41: | What about approaches outside of machine learning and game theory? |
| Answers 41: | Yes, the Government is interested in approaches other than those in machine learning and game theory, as well. The Stratagem BAA Synopsis has been updated to reflect interest in approaches outside of machine learning and game theory. |
| Question 42: | How important is it that any agent that we develop is able to detect/determine what COA the enemy/adversary is pursuing? |
| Answer 42: | This is inherent or complicit in developing a good red/blue strategy. Any effective agent definitely needs to determine what opponent is trying to achieve in order to be able to exploit it. |
| Question 43: | Will our agents know (i.e., be given) the objectives of the other players? |
| Answer 43: | By default this is the case, yes, all players will be aware of each other’s objectives. There will also be some experiments/scenarios where this is not the case. |
Wargame Questions Disclaimer: The wargame may change as operational partner requirements evolve.
General
| Question 44: | Can you briefly describe the essence and purpose of the Stratagem wargame? |
| Answer 44: | It is an operational-level, hybrid turn-based and real-time strategy game played on a hex-based grid. It is a high-level, low fidelity, effects-based model. It is meant to incorporate multiple domains, starting with land and air. It is heavily inspired by concepts from Avalon Hill’s tabletop wargame series, most notably their Stalingrad and Midway titles. |
| Question 45: | Will the Stratagem wargame be provided to all contract performers? When would it be made available? |
| Answer 45: | Yes. Day one of the contract award/start. |
| Question 46: | What is the Stratagem wargame developed in? |
| Answer 46: | The model layer for the game is developed in pure C#. The visualization and controller layers for human players are developed in Unity3D. That said, AI players could potentially connect without any Unity3D-based elements being involved. |
| Question 47: | The announcement mentioned StarCraft 2 and Civilization. Do you see this being a multi-player game or single-player, or potentially both? |
| Answer 47: | The game will always have at least two players, either of which could be AI or human. In many cases, the game will have two players where Red is always AI and Blue is AI or human. Anticipate the possibility of players other than Red and Blue that may be friendly, neutral, or hostile. |
| Question 48: | Are you going to have a specific taxonomy as to what units can be? |
| Answer 48: | There is not going to be a specific taxonomy. It will be possible for scenario developers to create new, arbitrary units in scenarios that vary in terms of combat factor, movement factor, and other unit characteristics described within this document. Part of the reason why this is allowed is because we are modeling the abstract effect of the unit, not the physics of the unit. |
The default land unit is going to be a Brigade Combat Team. However, what we’re most interested in here is the combat factor – the effect – of the unit. That is, the Government is more concerned that a unit has a attack factor value of three than what the unit is called/labelled.
| Question 49: | Will probability tables be known in advance? Combat, movement, etc. |
| Answer 49: | Yes. |
| Question 50: | Does the game have stochastic elements? |
| Answer 50: | Yes. Most notably, combat. Whenever one unit attacks another, there are combat odds involved. The combat odds are the likelihood that the attacker will make a successful attack. These combat odds are based on the attacking unit’s attack factor divided by the defending unit’s defense factor. If the combat odds are (8:1), then out of nine combat instances, the attacker will probably win eight of them. |
| Question 51: | Is Stratagem a text-based game? |
| Answer 51: | No. |
Although inadvisable, it is possible to create a human player interface to play the game solely from a command line.
Combat
| Question 52: | How are combat odds computed? |
| Answer 52: | Combat odds are the odds that an attacker will make a successful attack against a defender. This probability is calculated by taking the total summed attack factor of the attackers and dividing it by the total summed defense factor of the defenders. Battle always takes place in the defender’s hex cell. |
| Question 53: | What does the outcome of a battle look like? |
| Answer 53: | Both the attacker and the defender suffer losses to their capable forces, which in turn reduces their attack factor, defense factor, and movement factor. Depending on the degree of loss, it will also increase fatigue and disorganization. If the battle is successful enough, the attacker may advance into the hex where the battle occurred and the defender must retreat out of it. Moreover, if any force suffers enough casualties, it will gradually stop responding to orders from players and may even flee the area of operations altogether. |
| Question 54: | What is the Zone Of Control (ZOC) for a unit? |
| Answer 54: | The Zone Of Control (ZOC) is the area (the central and six neighboring hexes) that a unit is able to exert its influence or control in way of its combat factors. If any unit crosses into an adversary ZOC, they must stop movement and engage in battle. |
| Question 55: | For the execution phase of the game, what decides which troop fights first in combat? What decides which troop moves first during movement phases? |
| Answer 55: | A unit’s troop quality attribute determines this. Troop quality is largely decided by a unit’s fatigue, disorganization, and supply state values. Thus, if a unit is highly fatigued through sustained movement, highly disorganized through numerous combat losses, and undersupplied, it will certainly go last during the execution turn phase in the game. |
| Question 56: | Are artillery fire missions and air strike missions separate from standard land combat instances? |
| Answer 56: | Yes. And as with regular combat, the order in which these may occur on a per unit basis is determined on the unit’s troop quality attribute. These also occur during the execution phase of the game. |
| Question 57: | Can we assume that a force could have SAM sites? Can we specifically target those sites? |
| Answer 57: | Yes, although this is handled in an abstract fashion as to whether or not air attrition is suffered. You can not directly target SAM sites, although you can utilize your land forces to claim ownership of any particular hex cell. Aircraft that fly through hex cells that are enemy owned and have high levels of enemy exertion of control (classic tabletop wargame Zone of Control concept) will suffer greater levels of attrition. |
Modeling / Game
| Question 58: | What does the structure of a game turn look like? |
| Answer 58: | A complete turn is split into three phases: planning, execution, and after action. Players give their units orders in the planning phase. The units execute those orders in the execution phase (movement and combat ensues). Players may inspect the results of the execution phase in the after action phase. |
| Question 59: | Is the wargame’s model effects-based or physics-based? |
| Answer 59: | Effects based. For each unit in the game, the Government is looking at the high-level effect that it is introducing into the area of operations. All units will provide attack factors and defense factors, although some units will provide some additional special capabilities. For example, having a unit that has artillery components will be able to perform automatic barrage attacks to weaken an enemy at long range (low fidelity, probabilistic, table lookup). Also, for example, having a unit that has an engineer/engineering component will allow the unit to gain an additional defense factor bonus (modifier) when holding/defending a position. |
| Question 60: | The game is at the effects-based operational level with tactical elements abstracted away. Can you give some examples of tactical elements that have been abstracted away? What does our AI player not have to take into account? |
| Answer 60: | No communication models. “Instantly” shared intelligence and vision. No physics modeling or formations taken into account. No acceleration, rotation. No flight or driving models. The Government is not modeling sensors or sensor types. No weapon models. There is no notion of a brigade facing in a given direction and thus could be flanked from the side. There is no notion of a vision/sensor cone being projected out in a focused direction. |
| Question 61: | How does the game operate between the real-time and turn-based execution modes? |
| Answer 61: | In turn-based mode, the game does not move to the next phase until all players have signaled they are done/ready. For example, the game will not move to the execution phase until all players have signaled that they are done with the planning phase. |
In real-time mode, the planning and after action phases auto-end after an arbitrarily set period of time – some interval of milliseconds depending on the scenario/experiment. If a player didn’t get an order to a specific unit in time for one particular game turn, it will be entered and acted upon in the execution phase for the next game turn.
| Question 62: | Who controls the speed of the real-time execution mode? Can the game be paused? |
| Answer 62: | These parameters could vary on an experiment by experiment or scenario by scenario basis. There are a few approaches that the Government intends on supporting: |
· The game executes at the slowest requested speed across all players.
· The game executes at a fixed rate regardless of requests to change the rate.
· One player is given control over the rate of execution (this will usually be a human player) for the entire game.
· Pausing is supported feature for real-time execution mode, although it could optionally be shut off depending on the scenario or type of experiment.
| Question 63: | Are land units the primary domain with other domains as support? |
| Answer 63: | Yes, initially the Government is starting with the land domain with air as in a supportive/augmenting role. The Government will add in more domains (sea, space, and so on) once we’ve gotten the foundation established with land and air. |
| Question 64: | What can air units do? Can air units perform air vs. air combat? |
| Answer 64: | Right now, air units can augment attack/defense factors of land units that they are ordered to support. As well, air units can perform strike missions that can soften/degrade red force units. If red and blue air units come into contact while supporting land units or performing strike missions, they can fight and suffer attrition. Air units can also suffer losses from coming into contact with ground units that have anti-air capabilities. The Government is investigating the idea of separating out combat factors (attack/defense) into separate domains (e.g., a single unit having a ground attack factor of 1, with an air attack factor of 3). |
| Question 65: | Will there be Fog of War or elements of imperfect information? |
| Answer 65: | Initially, no. The assumption at the beginning is that ISR missions are effective at determining the locations and attributes of all blue and red (and other, if they exist in a scenario) forces. And, of course, no agent will be able to examine red intent or orders from the state info that is provided to players/agents. |
Future, more advanced versions of the wargame may contain limited Fog of War elements if there is interest from operational partners for these features. It would be possible to apply fuzziness to some of the intelligence gained on enemy units, where lower degrees of intelligence gathering success would result in more uncertain values.
| Question 66: | What level of fidelity will the weather be modeled at, will it be constant weather across the map, or varied across the terrain? |
| Answer 66: | Very low fidelity. The weather will be the same across the map and change over multiple days. This is a lower priority feature for the Stratagem Wargame. It will mostly impact ground conditions (slowing movement of ground forces) and cloud conditions (degrading air combat factors). |
| Question 67: | What level of fidelity will the units be modeled at? E.g., will a battalion of soldiers be a number of soldiers with the same health and weaponry, or will soldier sub-types be applicable, such as riflemen or grenadiers? |
| Answer 67: | The Government is looking at low fidelity, effects-based modeling for units in the wargame. The Government is abstracting away the tactical elements of combat, like utilizing certain weapons for certain troops. Instead, the Government is looking more at the effect, the combat factor, of applying a certain unit in the game. For example, the Government may say a certain unit may have an attack factor of 3.2 and a defense factor of 2.4, whereas another unit may have an attack factor of 4.0 and a defense factor of 3.7. |
| Question 68: | Is the wargame going to run single server or distributed? |
| Answer 68: | Single server. |
The game is very lightweight and low fidelity so that it is possible to run hundreds of thousands of runs in order to train agents. A single run should run on a single server. However, multiple parallel runs could run on a single server (ideally) or on multiple servers. It should be noted that if a single instance of a game has to run in a distributed fashion across multiple servers, then it’s likely too heavy to effectively train agents with and defeats the purpose of this approach.
| Question 69: | You mention that units have a certain amount of supply on hand (in tons). Do you break these up into separate categories like fuel, ammo, food, etc.? |
| Answer 69: | The Government does not presently do this. The supply gets consumed at an abstract level through existence, movement, and combat. The Government will revisit this abstraction in the coming months. |
| Question 70: | What does your supply line modeling look like? |
| Answer 70: | Each of the units require supply or their effectiveness degrades. Depending on what units do, they consume their supply on hand. Their supply on hand is replenished by supply nodes that they are attached to (base, HQ, supply drop, etc.). The amount of supply they can receive per day is impacted by the distance that the unit is from the supply node and how much the supply node can furnish on a day to day basis. Remember, any given supply node might have to service multiple units, all having varying levels of consumption rates based on their activity. |
| Question 71: | Can we apply air to impact/attack/degrade red supply lines? |
| Answer 71: | Yes. If you order your air units to operate in an area that red supply lines run through, then those supply lines will be proportionately degraded. |
| Question 72: | What kind of hex types will there be? |
| Answer 72: | Hex cells can be at various elevations (giving a potential advantage during ground combat). Hex cells will have different surface/terrain types that will impact different movement classes for units (e.g., tracked, wheels, foot). The main types are: wooded, clear, city, village, water, river, marsh, road, hills, mountains, rubble, factory, water edge. |
Player API
| Question 73: | How would the game state be accessed? Through API or screen pixels? |
| Answer 73: | An agent would have access to a player API that exposes all state information (known to the player per its perspective) and provides the ability to perform actions. It would be in many ways similar to the StarCraft II API’s default approach. However, this does not preclude the use of raw pixels, if desired. |
| Question 74: | What does the state space look like for an AI player? |
| Answer 74: | There are state space information snapshots which are given to each AI player at the beginning of each turn phase (planning, execution, and after action). As well, events are generated during execution which are fed to AI players in the order that these events occur. |
There are three main categories of state space information snapshots:
The first is hex grid information: this reveals to the player the attributes of every hex cell that represents the area of operations. These hex cell attributes include position in hex grid, the surface/terrain type (impacts movement cost), elevation, size of the hex (in km), the units contained within, and the exerted control by each player.
The second category is general game information, where the agent learns what the current turn is, what the current turn phase is, who the other players are, and so on.
The third category is a full list of known units, where the agent can find out the values to attributes such as position in the grid, the domain (e.g., ground, air), the player that owns the unit, the name of the unit, the unit type (e.g., BCT, squadron of F15s), the components underneath this unit (e.g., battalion of artillery), the HQ the unit is attached to, the capable count, wounded count, killed count, list of current orders (move and support over time), attack factor, defense factor, movement factor, supply state, tons of supply on hand, troop quality, disorganization, fatigue, readiness, and so on.
The supported events in the Stratagem wargame currently include: CombatEvent, GameEvent, GameReloadEvent, MoveEvent, ObjectiveEvent, PhaseEvent, TurnEvent, and StoppedInZOCEvent.
CombatEvents tell us for each instance of combat, which units were the attackers, which units were the defenders, what were the combat odds, who won the engagement, what the suffered losses were, whether the attacking units advanced, whether the defending units had to retreat.
A GameReloadEvent tells us that the game was rewound back to a previous saved state of the game.
A MoveEvent tells us that an observed unit has moved from one hex cell to another hex cell.
An ObjectiveEvent tells us that one of the players had achieved or failed one of their assigned objectives.
A PhaseEvent tells us that a new turn phase has begun (planning, execution, after action).
A TurnEvent tells us that a new game turn has begun.
A StoppedInZOCEvent tells us that one of our units has stopped moving during the execution phase because it ran into an enemy Zone Of Control. This means the unit entered into a hex cell that an enemy unit has exerted control into.
| Question 75: | What does the action space look like for an AI player? |
| Answer 75: | You will have the ability give orders to each of your units. These orders can outline an avenue of advance or support that must be given to friendly unit. For setting an avenue of advance, you can describe locations that you want a given unit to reach in time (no earlier than, no later than). Your unit will attempt to achieve the order to the best of its ability as the situation evolves from turn to turn. Additionally, you may set a posture/mode to the avenue of advance that describes the type of move (strategic, tactical, aggressive, defensive, evasive, etc.) where the mode will impact how quickly the unit moves and how it will engage enemy units within range (that is, within a zone of control). |
| Question 76: | When our agents submit an action (make an order for a unit), how do we know the outcome of the activity? How do we get feedback? |
| Answer 76: | An agent (AI player) submits actions during the planning phase of the game. Then the game allows the units to execute these orders during the execution phase of the game. The agent can inspect the results (movement, combat, etc.) during the after action phase of the game. |
| Question 77: | Can the agent query the game to get extra information? |
| Answer 77: | No, the agent will be provided all the information that can be known via the Player API. |
| Question 78: | Would agents have direct access to the hidden features of the wargame model? |
| Answer 78: | No, agents would only have access to the game state information that is directly provided to them, which is based on their perspective/knowledge of the environment, especially if/when fog of war features are enabled. |
| In a nutshell, the AI has access to the same amount of exposed information that a human player would through the game’s GUI visualization. |
| Question 79: | Is the agent based around centralized or decentralized control? |
| Answer 79: | Centralized. A single agent will control a single Player object in the Stratagem Wargame. That player object will be able to give orders to a collection of units that fall under that player. This is a similar to the approach used in real-time strategy games like StarCraft II or Command & Conquer. |
| Question 80: | What programming languages can I use to create an agent to play the wargame? |
| Answer 80: | The Government is starting with C#. Out of the box, under the Unity MLAgents framework, it would be possible to utilize Python. |
Scale
| Question 81: | How many players? How many units per player? |
| Answer 81: | In most potential scenarios, there would be two players: red and blue. Upwards of 100 units per player for early scenarios. |
| Question 82: | How large are the units? |
| Answer 82: | A typical land-based unit would be a Brigade Combat Team (BCT), which consists of between 4000-5000 soldiers. A typical air-based unit would be a squadron of some air asset. A typical sea-based unit would be a single ship or Surface Action Group (SAG). It should be noted that you only give orders to the unit as a whole, not to any of the lower level components of that unit. |
| It should be noted, however, that for a given unit, the Government is more concerned with the fact that its attack factor is, say, a value of 3.7 rather than the fact that the unit has 2000 soldiers armed with a certain rifle with special ammunition and grenade launcher attachments. |
| Question 83: | How much time is represented by a single turn? How many decision points could there be in the run of a single game? |
| Answer 83: | By default, one turn will represent a single day. However, it is expected that in some scenarios it will be possible to have a day’s time split up into 4-6 turns or a single turn perhaps representing as many as three days. The number of decision points will be a function of the number of days of operation and the number of turns per day. (In Avalon Hill’s PC version of Stalingrad, players had between 3-6 moves per days depending on the size of the map.) |
| Question 84: | What period of time could a scenario cover? |
| Answer 84: | Generally, between 100 and 200 days. Although some specialized scenarios (based on requirements from operational partners) may lower or raise this number. |
| Question 85: | What defines the “scale” of a scenario? |
| Answer 85: | The scale of the scenario is defined by: length of time (days), the number of forces present in the scenario, and the size of the operating environment (hex grid). A “smaller scale” scenario would be around 100 days, 3-5 units, and on a 100x100 grid. A “larger scale” scenario would be more on the order of 200-300 days, 10-25 units, and a 700x700 grid. Or even larger. |
| Question 86: | How do you account for or contend with the very different time scales amongst the different domains (land, sea, air, subsurface, cyber, and space)? |
| Answer 86: | To begin with in the game, the Government is focusing on land with the other domains providing a supporting/augmenting role. The Government is starting off simple. The Government is investigating how to further integrate other domains, however this is not on the immediate critical path. |
Scenarios
| Question 87: | Will a collection of scenarios be provided with the AFRL-produced Stratagem wargame? Will performers have the ability to create their own scenarios for training/testing purposes? |
| Answer 87: | Yes to both. For the latter, the API will expose the functionality necessary to create new scenarios. |
| Question 88: | What is known about the setup of a new game or scenario? What is revealed in the initial game state information? |
| Answer 88: | At the beginning of the game, the initial state info object will reveal the hex grid details, the current objectives for each player, as well as the location and attributes for each unit. The location for each unit may also include the time or time range when they will arrive within the environment. |
| Question 89: | What are some example objectives? |
| Answer 89: | Some example objectives include: take over hex {x,y} and hold for n number of days, prevent opponents from achieving their objectives for n number of days, continue to hold hex {x,y} to day #, hold some % of total land by day #. |
| Question 90: | Can objectives have sub-objectives? |
| Answer 90: | Yes. |
Other
| Question 91: | What coordinate system are you using for the wargame’s hex grid? |
| Answer 91: | Offset odd-r hex coordinates. Please see: https://www.redblobgames.com/grids/hexagons/#coordinates |
