Not a secret that I was a pilot and that I have worked on several military aircraft, making me aware about weapons, electronic warfare and Theater of Operations. If you read between the lines, I wrote "faster decisions" and not "better decisions". Time will unveil the outcome of a decision. 33 engines on a rocket instead of 5? Crazy decision at first isn't it (SpaceX)?
Pilots are trained at making rapid decisions and what is the process underneath that speed? How does that relate to Risk Based Testing, Pairing, Context Driven Testing, Test Campaign Management, Resources Management, Data management, Communication management?
My training goes around a very realistic Air Combat flight Simulator that is powered by AI, including briefing, threat analysis and planning tools for goals settings. For AI and my own performance evaluation I use a debriefing tool to perform a retro.
The rig is designed around those software and hardware:
The briefing is a structured presentation used to convey information, provide instructions, or facilitate decision-making. It is essential for ensuring that all personnel are informed, aligned, and prepared for their roles and responsibilities. Charts, maps, diagrams, and slides are often used to enhance the presentation and help illustrate key points. These aids make complex information more accessible and easier to comprehend.
Based on the previous briefing I will evaluate the "quality" of the information presented, assess the Red's data fusion capabilities. How many troops do they have, where are they, can they detect me, what is the range of their weapons, where the supplies are coming from, what are the risks and how can I contain those risks? What is the range of the my squadron, how much fuel versus weapons can we carry, do I have to "specialize" one aircraft for a part of the mission?
An example in the process of decision making:
If I need to use a AGM 154 (gliding bomb) then I will have to account for wind, a Paveway laser guided bomb will require some visibility but then I will have to "sacrifice" one pod to carry the laser designator/InfraRed camera. If the day is too hot in the desert I will have a poor IR image. Too cold and then the heat from jet engines is going to attract their IR missile seekers. How fast the Red are going to react? Maybe I can send part of my squadron ahead to wipe out their radar or defense system (ground attack) and then make another round why my squadron.
Focus is on the details of the specific mission or operation. It includes information on objectives, tactics, logistics, ROE (Rules Of Engagement) and contingencies.
ROE are covering the following aspects (feel free to translate some terms to match software/hardware engineering while reading):
I will now implement the mission in the system. The mission comprises a couple of steps that would resemble Jira tickets assigned to different teams. Those tickets are linked together because of dependencies. In my rig, those are 'triggers' with sophisticated logic (If that part of the mission (the ticket) scored more than X then do that, else to this). Each ticket will contribute to the mission overall score. When the overall score reaches a certain number of 'points', the mission will be declared as a success and then I will issue the "return to base" command to my squadron.
Used to coordinate activities and share information among staff members. These are transparent tasks in the game, but basically all teams are aware about the communication frequencies to use, the codes for the lasers encoding, where to deploy assets.
After usual pre-flight checklists, it is time to fire up the FA18 Hornet's turbines (or whatever aircraft was deemed to be suitable for the mission).
AI was programmed by humans, trained on some scenarios created by humans, so I am expecting some "human predictable" reactions. This is where the Cynefin framework used by military folks enters the game. Clearly I am now entering the Complex domain. The Probe-Sense-Respond approach is suitable. The second approach I need to introduce at the same time is data fusion from my onboard sensors. I had published something about data fusion in jet fighters a couple of months ago here. This will be an indicator about the quality of the threat assessment and mission planning.
In short I could Gherkin (simplified to avoid the TL;DR) something like:
Scenario: Fly below Line Of Sight with Radar off so I remain undetected.
Given I am in my FA18 aircraft
And my aircraft is below 6000 feet above flat ground
And the RED's radar is located at more than 176.18962149 kms away (Line Of Sight)
When my radar is on STANDBY
Then RED's radar cannot detect me.
Great! That works, because I see the other RED aircraft circling around, not heading towards me. That information is provided by DATALINK from the AWACS (airborne radar) that I have deployed during the mission preparation.
Next step is to split the squadron, send one group 45 degrees off track while I draw enemy's attention on me. For this I will pair with three wingmen. Climbing on afterburners to 14000 feet and upon reaching turning on our radars and jammers. That way the AI should think that we could be a large group of aircrafts flying towards the RED.
Great! That works because the RED are flying towards my small group and not the spilt group.
Next step is making AI believe that I will continue flying that route! At the same time I will fly directly at them. When sure RED are tracking us I will then make a 90 degrees turn so I can notch their radar.
Great! That works because they continue their route while I fly 90 degrees offtrack.
The mission will continue till the end or until I issue the "Return to base" command. Along the mission I would be aware of all major events thanks to a strict wording exchanged over the radios. Those data would feed me for resources management and mission progress. When an event occurs I will have to reassess the situation, resources, goals and take proper decisions.
Evaluation of the mission is done using TACVIEW. Evaluation of the AI will be based on the information (telemetry) available in TACVIEW.
Some examples where AI does not work properly:
Upon returning to base and landing on the carrier or the airfield, priority is given to damaged aircrafts of aircraft with low fuel conditions. This is not performed.
Some aircrafts are way too heavy to land on the carrier, they should have jettison some weapons before landing but they did not. This is not performed.
In close combat, my wingmen are often too close to me at same altitude while they should be higher to cover me in case another aircraft tries to be in my 6 (back).
What base do I use to evaluate AI then?
Aerial combats have been refined over the years by different airmen. Does Topgun ring a bell to you? I have learned and practiced many hours the basics, then the different tactics around air combat. I mix those information with what I know about technical capabilities of radar, airframes, aerodynamics, weapons capabilities. I could prepare and fly the same mission several times and it would unfold differently each time, forcing me to make decisions, sometimes snap decision when getting closer to the RED at 686 meters per second or when a missile is heading towards me at 1715 meters per second closing speed.
In the end I have to answer those questions during the debrief or the retro:
How much effort did we deployed during that mission and how much did we achieved. One would say "How was our ROI (Return On Investment) ?". How many team members did we lose? Did we changed the Rules Of Engagement to adapt to the situation? How could I have improved on that close combat ( flying slower, drop weapons to be lighter and have a more agile aircraft) .
While I have a large expertise in testing complex and complicated systems I would state exactly what was written on my business card when I was on the Boeing 777: Test and Evaluation pilot.
In other words, one should be able to tests part of AI but you can only evaluate AI as a whole.
Transposition of knowledge/competence is largely underestimated by Human Resources. Just ask yourself why some ex Marines are preferred for some positions in US based companies.
Aviation/Aerospace and military projects are leading the world in terms of progress in technologies. This is achieved by having the right brains in their teams but also by having well established and proven methodologies.