Skip to Content
Last modified by Mark Rinse van Koningsveld on 2026/07/27 10:06
Hide last authors
Tjalling Haije 1.1 1 = 1. Introduction =
2
Mark Rinse van Koningsveld 7.1 3 This experiment validates multiple human-machine teaming technologies in Urban Search and Rescue (USAR) operations. Four operational modules simulate a full operational storyline across two days: wide area assessment, full-area reconnaissance with health monitoring, indoor drone-assisted search, and precision inspection in confined spaces.
Tjalling Haije 1.1 4
Mark Rinse van Koningsveld 7.1 5 The modules test system functions from five use cases, and aim to quantify effects on safety, situation awareness (SA), physical workload, mission effectiveness, and decision-making quality. Results will be compared against expected performance without these technologies, based on either baseline team data, observer input, or known solution benchmarks.
Tjalling Haije 1.1 6
Mark Rinse van Koningsveld 7.1 7 ----
Rosa Van Tuijn 6.1 8
Mark Rinse van Koningsveld 7.1 9 = 2. Method =
Rosa Van Tuijn 6.1 10
Mark Rinse van Koningsveld 7.1 11 == 2.1 Participants ==
Rosa Van Tuijn 6.1 12
Mark Rinse van Koningsveld 7.1 13 Approximately 24–30 international first responders, organized in teams. Each team rotates across the four modules. Roles include responders, team leaders, drone/robot operators, analysts, medics, and safety officers.
Rosa Van Tuijn 6.1 14
Mark Rinse van Koningsveld 7.1 15 == 2.2 Experimental Design ==
Rosa Van Tuijn 6.1 16
Mark Rinse van Koningsveld 7.1 17 A **within-subject design** is used where all teams go through the four modules. Performance is compared across modules and against predefined baseline criteria. Observers collect data in real time; surveys and biometric data are used to validate subjective and objective measurements.
Rosa Van Tuijn 6.1 18
Mark Rinse van Koningsveld 7.1 19 ----
Rosa Van Tuijn 6.1 20
Mark Rinse van Koningsveld 7.1 21 == 2.3 Tasks (Per Module) ==
Rosa Van Tuijn 6.1 22
Mark Rinse van Koningsveld 7.1 23 ----
Rosa Van Tuijn 6.1 24
Mark Rinse van Koningsveld 7.1 25 === **Module 1 – Wide Area Assessment** ===
Rosa Van Tuijn 6.1 26
Mark Rinse van Koningsveld 7.1 27 **Use Case**: UC03.0
28 **Scenario**: Teams arrive at a simulated disaster zone. Structures are unstable. Drone support is requested for external mapping and hazard detection.
Rosa Van Tuijn 6.1 29
Mark Rinse van Koningsveld 7.1 30 **Tested Functions**:
Rosa Van Tuijn 6.1 31
Mark Rinse van Koningsveld 7.1 32 * Drone feed provides real-time visuals to field teams and command
33 * Zoom-ins allow inspection of rooftops and entry points
34 * Footage used to mark safe approach routes
Rosa Van Tuijn 6.1 35
Mark Rinse van Koningsveld 7.1 36 **Measured Claims**:
Rosa Van Tuijn 6.1 37
Mark Rinse van Koningsveld 7.1 38 * CL1: Improved external SA
39 * CL2: Safer movement planning
40 * CL3: Faster planning cycle
41 * CL4: Reduced mental workload for recon
42 * CL5: Improved coordination (shared SA)
Tjalling Haije 1.1 43
Mark Rinse van Koningsveld 7.1 44 **Quantifiable Success Factors**:
Tjalling Haije 1.1 45
Mark Rinse van Koningsveld 7.1 46 * ≥80% of hazards correctly marked on the map (based on preset dummy hazards)
47 * ≥90% agreement in SA between team and command (map match)
48 * Average planning time ≤ 10 minutes from drone launch
49 * NASA-TLX workload score ≤ 50 (moderate) for command roles
Tjalling Haije 1.1 50
Mark Rinse van Koningsveld 7.1 51 **How to Measure**:
Tjalling Haije 1.1 52
Mark Rinse van Koningsveld 7.1 53 * Observer logs & stopwatch for planning time
54 * Map test: Compare team-drawn vs. actual map (SAGAT-lite)
55 * Count number of correctly identified hazards from drone feed
56 * NASA-TLX filled by drone operator and team lead
57 * Post-module survey: "How useful was the drone in forming your plan?" (1–5)
Tjalling Haije 1.1 58
Mark Rinse van Koningsveld 7.1 59 ----
Tjalling Haije 1.1 60
Mark Rinse van Koningsveld 7.1 61 === **Module 2 – Health Monitoring & Reconnaissance** ===
Tjalling Haije 1.1 62
Mark Rinse van Koningsveld 7.1 63 **Use Cases**: UC01.1 (Fire) and UC01.2 (USAR)
64 **Scenario**: Team performs full-area recon. Wearables measure heart rate, hydration, and simulated gas exposure. Simulated fatigue and alerts escalate to medics or team leads.
Tjalling Haije 1.1 65
Mark Rinse van Koningsveld 7.1 66 **Tested Functions**:
67
68 * Alerts for fatigue/gas exposure
69 * Remote dashboard monitoring by safety officer
70 * Escalation protocols for health interventions
71 * Logging and after-action review
72
73 **Measured Claims**:
74
75 * CL1–CL2: Prevent overexertion and increase responder awareness
76 * CL3–CL4: Enable remote intervention and informed medical decision
77 * CL5: Enable better rotation/rest planning
78 * CL6: Debrief uses health logs
79 * CL7: Improve mission success
80
81 **Quantifiable Success Factors**:
82
83 * ≥90% of health alerts acknowledged within 1 minute
84 * ≥80% of interventions judged "timely" in AAR interviews
85 * ≥50% of teams adjust tactics or rest cycles based on health data
86 * ≥1 health-based lesson identified per team in debrief
87 * ≤2 simulated incidents due to unmanaged fatigue/gas exposure
88
89 **How to Measure**:
90
91 * Log alert timings vs. response time
92 * Observer notes + medic reports on intervention
93 * Exit survey: "Did alerts help prevent fatigue/injury?"
94 * Use of wearable dashboard during debrief (Yes/No)
95 * NASA-TLX for responders
96
97 ----
98
99 === **Module 3 – Indoor Drone Search (Barracks)** ===
100
101 **Use Cases**: UC02.1 and UC02.2
102 **Scenario**: Collapsed barracks building. Indoor drone used for autonomous scan. Analyst tags victims, hazards, and updates C3I map. Drone does close inspection on request.
103
104 **Tested Functions**:
105
106 * Pre-entry thermal scan
107 * Hazard/victim detection
108 * Analyst-supported interpretation and tagging
109 * Entry planning based on drone data
110
111 **Measured Claims**:
112
113 * CL1: Heightened SA before entry
114 * CL2: Increased safety (less exposure)
115 * CL3–CL5: Faster, more accurate victim detection
116 * CL6: Trust in drone data
117 * CL7: Increased mission efficiency
118
119 **Quantifiable Success Factors**:
120
121 * ≥90% of dummy victims detected by drone+analyst
122 * ≥2 new hazards marked per team from drone feed
123 * Average time-to-first victim ≤ 3 minutes
124 * ≥80% of responders rate drone info as “trustworthy” (score ≥4/5)
125 * ≤1 injury due to unknown hazard in follow-up entry
126
127 **How to Measure**:
128
129 * Victim tags placed in known positions for ground truth
130 * Observer logs: detection times and analyst confirmations
131 * Team SA quiz: "How many victims? Where were they located?"
132 * Trust survey: “I would act on this drone data” (1–5)
133 * Entry path compared to drone hazard map
134
135 ----
136
137 === **Module 4 – Precision Inspection with ANYMAL/SNAKE** ===
138
139 **Use Case**: UC04.0
140 **Scenario**: Teams reach unstable voids. Robots are deployed to inspect inaccessible areas. SNAKE arm is used to look into cracks. Results update team maps and entry plans.
141
142 **Tested Functions**:
143
144 * Autonomous or manual ANYMAL movement
145 * Void inspection using flexible arm
146 * Victim/hazard confirmation
147 * Decision-making based on robot visuals
148
149 **Measured Claims**:
150
151 * CL1: Access without risk
152 * CL2: Detection in confined space
153 * CL3: Safer routing
154 * CL4: Trust in robot-assessed visuals
155 * CL5: Faster room clearing
156
157 **Quantifiable Success Factors**:
158
159 * ≥2 hazards or victims confirmed via SNAKE per team
160 * ≥80% of voids scanned without human entry
161 * ≥70% of teams adjust route based on robot findings
162 * ≥80% of participants rate robot visuals as “clear and usable”
163 * Average inspection time ≤ 8 minutes per room
164
165 **How to Measure**:
166
167 * Observer log: robot path vs. human path
168 * Detection log compared to known hidden items
169 * Survey: “Did robot findings improve your plan?” (Yes/No)
170 * Video review of time-per-room
171 * Trust in visuals scale (1–5)
172
173
174
Tjalling Haije 1.1 175 == 2.4 Measures ==
176
Mark Rinse van Koningsveld 7.1 177 This section describes how each claim will be measured during each module, using a combination of objective logging, observer annotations, post-task surveys, and scenario-based evaluation.
Tjalling Haije 1.1 178
Mark Rinse van Koningsveld 7.1 179 ----
180
181 === **Module 1 – Wide Area Assessment (UC03.0)** ===
182
183 |=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
184 |CL1 – Improved SA|Number of hazards correctly identified on team maps|SAGAT-lite: Pre/post map-drawing task + verbal hazard recall|≥80% match with ground-truth hazard list
185 |CL2 – Safer planning|Number of hazard zones avoided during later entry|Observer logs cross-referenced with hazard map|100% of marked hazards avoided
186 |CL3 – Faster planning|Time from drone launch to team briefing|Stopwatch & observer notes|≤10 minutes total
187 |CL4 – Reduced workload|Mental workload score of command & drone operator|NASA-TLX (short form)|≤50 average score
188 |CL5 – Shared SA|Consistency between team and command in map data|Comparison of annotations across roles|≥90% agreement on key features
189
190
191
192 ----
193
194 === **Module 2 – Health Monitoring & Reconnaissance (UC01.1 / UC01.2)** ===
195
196 |=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
197 |CL1 – Prevent overload|HR trend + alert timing vs. pause/extraction|Wearable logs + observer notes|≥90% alerts followed by correct action within 1 minute
198 |CL2 – Responder awareness|Survey response on self-adjustment|Post-task Likert: “The alert helped me act”|≥80% rate 4 or 5
199 |CL3 – Remote escalation|Alert-to-medic contact time|System log + stopwatch|≤1 minute average
200 |CL4 – Medical support|Alignment of alerts with medical assessment|Medic forms + sensor log correlation|≥80% concordance
201 |CL5 – Operational planning|Number of rest/rotation decisions based on dashboard|Observer logs + team lead AAR|≥50% of teams adapt plan
202 |CL6 – AAR use of health data|Was biometric data used during debrief?|Debrief analysis|Yes, per team
203 |CL7 – Mission effectiveness|Task time + incidents avoided|Stopwatch + incident log|Task time not slower than baseline; 0 uncontrolled fatigue/gas incidents
204
205
206
207 ----
208
209 === **Module 3 – Indoor Drone Search (UC02.1 / UC02.2)** ===
210
211 |=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
212 |CL1 – Heightened SA|SA questionnaire + map task|Pre/post: victims, layout, hazard count|≥80% correct recall post-drone
213 |CL2 – Increased safety|Hazard zone avoidance rate|Observer vs. ground truth map|≥90% of flagged areas avoided
214 |CL3 – Faster victim detection|Time to first detection|Stopwatch from drone entry|≤3 minutes
215 |CL4 – Accuracy of detection|Victim detection rate|Drone log vs. planted victims|≥90% detected
216 |CL5 – Trust in results|Survey: “I trust the drone data for decision-making”|1–5 Likert scale|≥80% rate 4 or 5
217 |CL6 – Efficiency|Entry time after drone plan vs. without drone|Stopwatch; compare with baseline data|10–20% faster planning phase
218
219
220
221 ----
222
223 === **Module 4 – Robot-Based Precision Inspection (UC04.0)** ===
224
225 |=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
226 |CL1 – Extended reach|Percentage of voids explored by robot not human|Observer log + inspection plan|≥80% of voids scanned by robot
227 |CL2 – Detection in small spaces|Victim/hazard detection in hidden locations|Camera log vs. planted markers|≥2 findings per team
228 |CL3 – Safer routing|Route changes based on robot input|Pre/post plan comparison + observer notes|≥70% of teams adapt plan
229 |CL4 – Trust in visuals|Survey on clarity and trust in robot data|Likert: “The robot data was sufficient for decisions”|≥80% rate 4 or 5
230 |CL5 – Room clearing speed|Time per room before vs. after robot scout|Stopwatch log|≤8 minutes per room avg.
231
232
233
234 ----
235
Tjalling Haije 1.1 236 == 2.5 Procedure ==
237
Mark Rinse van Koningsveld 7.1 238 All modules follow a similar four-part procedure, tailored per use case.
Tjalling Haije 1.1 239
Mark Rinse van Koningsveld 7.1 240 === **General Daily Timeline** ===
241
242 * **08:30 – 09:00**: Morning briefing, safety, tech setup
243 * **09:00 – 12:00**: First module rotation (two parallel teams)
244 * **13:00 – 16:00**: Second module rotation (two parallel teams)
245 * **16:00 – 17:00**: Shared after-action review
246
247 Each module runs with the following structure:
248
249 === **Per Module Procedure** ===
250
251 1. (((
252 **Briefing (10–15 min)**
253
254 * Explain objectives, scenario, roles, safety, success factors
255 * Introduce technology and expectations
256 )))
257 1. (((
258 **Execution Phase (45–60 min)**
259
260 * Scenario runs in real time
261 * Observer logs events, actions, communications
262 * System logs recorded (drone, robot, wearables)
263 )))
264 1. (((
265 **Measurement Phase (15–20 min)**
266
267 * Paper or tablet surveys: SA, trust, NASA-TLX
268 * Sensor data downloaded to central system
269 * Short interview or checklist with operator and team lead
270 )))
271 1. (((
272 **Debrief (15–20 min)**
273
274 * Team reflects on use of technology, decision-making
275 * Facilitator prompts discussion of claims (trust, effectiveness, awareness)
276 * Recorded notes for final reporting
277 )))
278
279 For cross-checking performance without the tech, one team per module may be assigned a simplified "control" version of the scenario, using conventional tools only (where feasible).
280
281 ----
282
Tjalling Haije 1.1 283 == 2.6 Material ==
284
Mark Rinse van Koningsveld 7.1 285 Each module requires scenario-specific equipment, environmental props, logging tools, and survey forms:
Tjalling Haije 1.1 286
Mark Rinse van Koningsveld 7.1 287 === **Common Materials (all modules)** ===
Tjalling Haije 1.1 288
Mark Rinse van Koningsveld 7.1 289 * Observer logbooks (standardized per module)
290 * Stopwatch or time-tracking app
291 * Participant role badges and checklists
292 * Data collection station with tablets/laptops
293 * Printed Likert-scale surveys (SA, trust, workload)
294 * SAGAT-lite map templates
Tjalling Haije 1.1 295
Mark Rinse van Koningsveld 7.1 296 === **Module-Specific Materials** ===
Tjalling Haije 1.1 297
Mark Rinse van Koningsveld 7.1 298 **Module 1 – Wide Area Assessment**
Tjalling Haije 1.1 299
Mark Rinse van Koningsveld 7.1 300 * Outdoor drones with RTK GPS and live zoom cameras
301 * Command screen with drone feed
302 * Large printed site maps with hazard zones (for scoring)
303 * Structural hazard props (collapsed façades, signs)
Rosa Van Tuijn 2.1 304
Mark Rinse van Koningsveld 7.1 305 **Module 2 – Health Monitoring**
306
307 * Wearable sensors (HR, hydration, gas; real or simulated)
308 * Dashboard software for live feed + logging
309 * Incident trigger devices (e.g., CO2 canisters, alarms)
310 * Medic checklist sheets
311 * Alert simulation software (optional)
312
313 **Module 3 – Indoor Drone Search**
314
315 * Thermal indoor drone with autonomous mode
316 * C3I-compatible map annotation system
317 * Dummy victims with heat packs or QR markers
318 * Printed room layouts for SA testing
319 * Indoor hazard props (rubble, fake smoke, blocked doors)
320
321 **Module 4 – ANYMAL and SNAKE**
322
323 * ANYMAL robot (legged) and SNAKE articulated arm
324 * Confined space mockups (voids, crawlspaces, stairs)
325 * Hidden hazard/victim tags inside small cavities
326 * Robot operator station + external monitor
327 * Scenario map with route overlays
Mark Rinse van Koningsveld 8.1 328
329 = 3. Results =
330
331 = 4. Discussion =
332
333 = 5. Conclusions =