h. Test CFT4

= 1. Introduction =

This experiment validates multiple human-machine teaming technologies in Urban Search and Rescue (USAR) operations. Four operational modules simulate a full operational storyline across two days: wide area assessment, full-area reconnaissance with health monitoring, indoor drone-assisted search, and precision inspection in confined spaces.

4

5

The modules test system functions from five use cases, and aim to quantify effects on safety, situation awareness (SA), physical workload, mission effectiveness, and decision-making quality. Results will be compared against expected performance without these technologies, based on either baseline team data, observer input, or known solution benchmarks.

----

= 2. Method =

== 2.1 Participants ==

12

13

Approximately 24–30 international first responders, organized in teams. Each team rotates across the four modules. Roles include responders, team leaders, drone/robot operators, analysts, medics, and safety officers.

14

15

== 2.2 Experimental Design ==

16

17

A **within-subject design** is used where all teams go through the four modules. Performance is compared across modules and against predefined baseline criteria. Observers collect data in real time; surveys and biometric data are used to validate subjective and objective measurements.

----

== 2.3 Tasks (Per Module) ==

----

=== **Module 1 – Wide Area Assessment** ===

26

27

**Use Case**: UC03.0

28

**Scenario**: Teams arrive at a simulated disaster zone. Structures are unstable. Drone support is requested for external mapping and hazard detection.

29

30

**Tested Functions**:

31

32

* Drone feed provides real-time visuals to field teams and command

33

* Zoom-ins allow inspection of rooftops and entry points

34

* Footage used to mark safe approach routes

**Measured Claims**:

* CL1: Improved external SA

39

* CL2: Safer movement planning

40

* CL3: Faster planning cycle

41

* CL4: Reduced mental workload for recon

42

* CL5: Improved coordination (shared SA)

43

44

**Quantifiable Success Factors**:

45

46

* ≥80% of hazards correctly marked on the map (based on preset dummy hazards)

47

* ≥90% agreement in SA between team and command (map match)

48

* Average planning time ≤ 10 minutes from drone launch

49

* NASA-TLX workload score ≤ 50 (moderate) for command roles

**How to Measure**:

* Observer logs & stopwatch for planning time

54

* Map test: Compare team-drawn vs. actual map (SAGAT-lite)

55

* Count number of correctly identified hazards from drone feed

56

* NASA-TLX filled by drone operator and team lead

57

* Post-module survey: "How useful was the drone in forming your plan?" (1–5)

----

=== **Module 2 – Health Monitoring & Reconnaissance** ===

62

63

**Use Cases**: UC01.1 (Fire) and UC01.2 (USAR)

64

**Scenario**: Team performs full-area recon. Wearables measure heart rate, hydration, and simulated gas exposure. Simulated fatigue and alerts escalate to medics or team leads.

65

66

**Tested Functions**:

67

68

* Alerts for fatigue/gas exposure

69

* Remote dashboard monitoring by safety officer

70

* Escalation protocols for health interventions

71

* Logging and after-action review

**Measured Claims**:

* CL1–CL2: Prevent overexertion and increase responder awareness

76

* CL3–CL4: Enable remote intervention and informed medical decision

77

* CL5: Enable better rotation/rest planning

78

* CL6: Debrief uses health logs

79

* CL7: Improve mission success

80

81

**Quantifiable Success Factors**:

82

83

* ≥90% of health alerts acknowledged within 1 minute

84

* ≥80% of interventions judged "timely" in AAR interviews

85

* ≥50% of teams adjust tactics or rest cycles based on health data

86

* ≥1 health-based lesson identified per team in debrief

87

* ≤2 simulated incidents due to unmanaged fatigue/gas exposure

**How to Measure**:

* Log alert timings vs. response time

92

* Observer notes + medic reports on intervention

93

* Exit survey: "Did alerts help prevent fatigue/injury?"

94

* Use of wearable dashboard during debrief (Yes/No)

95

* NASA-TLX for responders

----

=== **Module 3 – Indoor Drone Search (Barracks)** ===

100

101

**Use Cases**: UC02.1 and UC02.2

102

**Scenario**: Collapsed barracks building. Indoor drone used for autonomous scan. Analyst tags victims, hazards, and updates C3I map. Drone does close inspection on request.

103

104

**Tested Functions**:

105

106

* Pre-entry thermal scan

107

* Hazard/victim detection

108

* Analyst-supported interpretation and tagging

109

* Entry planning based on drone data

**Measured Claims**:

* CL1: Heightened SA before entry

114

* CL2: Increased safety (less exposure)

115

* CL3–CL5: Faster, more accurate victim detection

116

* CL6: Trust in drone data

117

* CL7: Increased mission efficiency

118

119

**Quantifiable Success Factors**:

120

121

* ≥90% of dummy victims detected by drone+analyst

122

* ≥2 new hazards marked per team from drone feed

123

* Average time-to-first victim ≤ 3 minutes

124

* ≥80% of responders rate drone info as “trustworthy” (score ≥4/5)

125

* ≤1 injury due to unknown hazard in follow-up entry

**How to Measure**:

* Victim tags placed in known positions for ground truth

130

* Observer logs: detection times and analyst confirmations

131

* Team SA quiz: "How many victims? Where were they located?"

132

* Trust survey: “I would act on this drone data” (1–5)

133

* Entry path compared to drone hazard map

----

=== **Module 4 – Precision Inspection with ANYMAL/SNAKE** ===

138

139

**Use Case**: UC04.0

140

**Scenario**: Teams reach unstable voids. Robots are deployed to inspect inaccessible areas. SNAKE arm is used to look into cracks. Results update team maps and entry plans.

141

142

**Tested Functions**:

143

144

* Autonomous or manual ANYMAL movement

145

* Void inspection using flexible arm

146

* Victim/hazard confirmation

147

* Decision-making based on robot visuals

**Measured Claims**:

* CL1: Access without risk

152

* CL2: Detection in confined space

153

* CL3: Safer routing

154

* CL4: Trust in robot-assessed visuals

155

* CL5: Faster room clearing

156

157

**Quantifiable Success Factors**:

158

159

* ≥2 hazards or victims confirmed via SNAKE per team

160

* ≥80% of voids scanned without human entry

161

* ≥70% of teams adjust route based on robot findings

162

* ≥80% of participants rate robot visuals as “clear and usable”

163

* Average inspection time ≤ 8 minutes per room

**How to Measure**:

* Observer log: robot path vs. human path

168

* Detection log compared to known hidden items

169

* Survey: “Did robot findings improve your plan?” (Yes/No)

170

* Video review of time-per-room

171

* Trust in visuals scale (1–5)

== 2.4 Measures ==

This section describes how each claim will be measured during each module, using a combination of objective logging, observer annotations, post-task surveys, and scenario-based evaluation.

----

=== **Module 1 – Wide Area Assessment (UC03.0)** ===

182

183

|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**

184

|CL1 – Improved SA|Number of hazards correctly identified on team maps|SAGAT-lite: Pre/post map-drawing task + verbal hazard recall|≥80% match with ground-truth hazard list

185

|CL2 – Safer planning|Number of hazard zones avoided during later entry|Observer logs cross-referenced with hazard map|100% of marked hazards avoided

186

|CL3 – Faster planning|Time from drone launch to team briefing|Stopwatch & observer notes|≤10 minutes total

187

|CL4 – Reduced workload|Mental workload score of command & drone operator|NASA-TLX (short form)|≤50 average score

188

|CL5 – Shared SA|Consistency between team and command in map data|Comparison of annotations across roles|≥90% agreement on key features

----

=== **Module 2 – Health Monitoring & Reconnaissance (UC01.1 / UC01.2)** ===

195

196

|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**

197

|CL1 – Prevent overload|HR trend + alert timing vs. pause/extraction|Wearable logs + observer notes|≥90% alerts followed by correct action within 1 minute

198

|CL2 – Responder awareness|Survey response on self-adjustment|Post-task Likert: “The alert helped me act”|≥80% rate 4 or 5

199

|CL3 – Remote escalation|Alert-to-medic contact time|System log + stopwatch|≤1 minute average

200

|CL4 – Medical support|Alignment of alerts with medical assessment|Medic forms + sensor log correlation|≥80% concordance

201

|CL5 – Operational planning|Number of rest/rotation decisions based on dashboard|Observer logs + team lead AAR|≥50% of teams adapt plan

202

|CL6 – AAR use of health data|Was biometric data used during debrief?|Debrief analysis|Yes, per team

203

|CL7 – Mission effectiveness|Task time + incidents avoided|Stopwatch + incident log|Task time not slower than baseline; 0 uncontrolled fatigue/gas incidents

----

=== **Module 3 – Indoor Drone Search (UC02.1 / UC02.2)** ===

210

211

|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**

212

|CL1 – Heightened SA|SA questionnaire + map task|Pre/post: victims, layout, hazard count|≥80% correct recall post-drone

213

|CL2 – Increased safety|Hazard zone avoidance rate|Observer vs. ground truth map|≥90% of flagged areas avoided

214

|CL3 – Faster victim detection|Time to first detection|Stopwatch from drone entry|≤3 minutes

215

|CL4 – Accuracy of detection|Victim detection rate|Drone log vs. planted victims|≥90% detected

216

|CL5 – Trust in results|Survey: “I trust the drone data for decision-making”|1–5 Likert scale|≥80% rate 4 or 5

217

|CL6 – Efficiency|Entry time after drone plan vs. without drone|Stopwatch; compare with baseline data|10–20% faster planning phase

----

=== **Module 4 – Robot-Based Precision Inspection (UC04.0)** ===

224

225

|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**

226

|CL1 – Extended reach|Percentage of voids explored by robot not human|Observer log + inspection plan|≥80% of voids scanned by robot

227

|CL2 – Detection in small spaces|Victim/hazard detection in hidden locations|Camera log vs. planted markers|≥2 findings per team

228

|CL3 – Safer routing|Route changes based on robot input|Pre/post plan comparison + observer notes|≥70% of teams adapt plan

229

|CL4 – Trust in visuals|Survey on clarity and trust in robot data|Likert: “The robot data was sufficient for decisions”|≥80% rate 4 or 5

230

|CL5 – Room clearing speed|Time per room before vs. after robot scout|Stopwatch log|≤8 minutes per room avg.

----

== 2.5 Procedure ==

All modules follow a similar four-part procedure, tailored per use case.

239

240

=== **General Daily Timeline** ===

241

242

* **08:30 – 09:00**: Morning briefing, safety, tech setup

243

* **09:00 – 12:00**: First module rotation (two parallel teams)

244

* **13:00 – 16:00**: Second module rotation (two parallel teams)

245

* **16:00 – 17:00**: Shared after-action review

246

247

Each module runs with the following structure:

248

249

=== **Per Module Procedure** ===

250

251

1. (((

252

**Briefing (10–15 min)**

253

254

* Explain objectives, scenario, roles, safety, success factors

255

* Introduce technology and expectations

256

)))

257

1. (((

258

**Execution Phase (45–60 min)**

259

260

* Scenario runs in real time

261

* Observer logs events, actions, communications

262

* System logs recorded (drone, robot, wearables)

263

)))

264

1. (((

265

**Measurement Phase (15–20 min)**

266

267

* Paper or tablet surveys: SA, trust, NASA-TLX

268

* Sensor data downloaded to central system

269

* Short interview or checklist with operator and team lead

270

)))

271

1. (((

272

**Debrief (15–20 min)**

273

274

* Team reflects on use of technology, decision-making

275

* Facilitator prompts discussion of claims (trust, effectiveness, awareness)

276

* Recorded notes for final reporting

277

)))

278

279

For cross-checking performance without the tech, one team per module may be assigned a simplified "control" version of the scenario, using conventional tools only (where feasible).

----

== 2.6 Material ==

Each module requires scenario-specific equipment, environmental props, logging tools, and survey forms:

286

287

=== **Common Materials (all modules)** ===