Skip to Content
Last modified by Mark Rinse van Koningsveld on 2026/07/27 10:06
From version 5.1
edited by Rosa Van Tuijn
on 2025/07/09 08:29
Change comment: There is no comment for this version
To version 8.1
edited by Mark Rinse van Koningsveld
on 2025/07/13 22:32
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.RosaVanTuijn
1 +XWiki.MarkVanKoningsveld
Content
... ... @@ -1,35 +1,333 @@
1 1  = 1. Introduction =
2 2  
3 -//<include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case>//
3 +This experiment validates multiple human-machine teaming technologies in Urban Search and Rescue (USAR) operations. Four operational modules simulate a full operational storyline across two days: wide area assessment, full-area reconnaissance with health monitoring, indoor drone-assisted search, and precision inspection in confined spaces.
4 4  
5 +The modules test system functions from five use cases, and aim to quantify effects on safety, situation awareness (SA), physical workload, mission effectiveness, and decision-making quality. Results will be compared against expected performance without these technologies, based on either baseline team data, observer input, or known solution benchmarks.
5 5  
7 +----
8 +
6 6  = 2. Method =
7 7  
8 -
9 9  == 2.1 Participants ==
10 10  
13 +Approximately 24–30 international first responders, organized in teams. Each team rotates across the four modules. Roles include responders, team leaders, drone/robot operators, analysts, medics, and safety officers.
11 11  
12 -== 2.2 Experimental design ==
15 +== 2.2 Experimental Design ==
13 13  
17 +A **within-subject design** is used where all teams go through the four modules. Performance is compared across modules and against predefined baseline criteria. Observers collect data in real time; surveys and biometric data are used to validate subjective and objective measurements.
14 14  
15 -== 2.3 Tasks ==
19 +----
16 16  
21 +== 2.3 Tasks (Per Module) ==
17 17  
23 +----
24 +
25 +=== **Module 1 – Wide Area Assessment** ===
26 +
27 +**Use Case**: UC03.0
28 +**Scenario**: Teams arrive at a simulated disaster zone. Structures are unstable. Drone support is requested for external mapping and hazard detection.
29 +
30 +**Tested Functions**:
31 +
32 +* Drone feed provides real-time visuals to field teams and command
33 +* Zoom-ins allow inspection of rooftops and entry points
34 +* Footage used to mark safe approach routes
35 +
36 +**Measured Claims**:
37 +
38 +* CL1: Improved external SA
39 +* CL2: Safer movement planning
40 +* CL3: Faster planning cycle
41 +* CL4: Reduced mental workload for recon
42 +* CL5: Improved coordination (shared SA)
43 +
44 +**Quantifiable Success Factors**:
45 +
46 +* ≥80% of hazards correctly marked on the map (based on preset dummy hazards)
47 +* ≥90% agreement in SA between team and command (map match)
48 +* Average planning time ≤ 10 minutes from drone launch
49 +* NASA-TLX workload score ≤ 50 (moderate) for command roles
50 +
51 +**How to Measure**:
52 +
53 +* Observer logs & stopwatch for planning time
54 +* Map test: Compare team-drawn vs. actual map (SAGAT-lite)
55 +* Count number of correctly identified hazards from drone feed
56 +* NASA-TLX filled by drone operator and team lead
57 +* Post-module survey: "How useful was the drone in forming your plan?" (1–5)
58 +
59 +----
60 +
61 +=== **Module 2 – Health Monitoring & Reconnaissance** ===
62 +
63 +**Use Cases**: UC01.1 (Fire) and UC01.2 (USAR)
64 +**Scenario**: Team performs full-area recon. Wearables measure heart rate, hydration, and simulated gas exposure. Simulated fatigue and alerts escalate to medics or team leads.
65 +
66 +**Tested Functions**:
67 +
68 +* Alerts for fatigue/gas exposure
69 +* Remote dashboard monitoring by safety officer
70 +* Escalation protocols for health interventions
71 +* Logging and after-action review
72 +
73 +**Measured Claims**:
74 +
75 +* CL1–CL2: Prevent overexertion and increase responder awareness
76 +* CL3–CL4: Enable remote intervention and informed medical decision
77 +* CL5: Enable better rotation/rest planning
78 +* CL6: Debrief uses health logs
79 +* CL7: Improve mission success
80 +
81 +**Quantifiable Success Factors**:
82 +
83 +* ≥90% of health alerts acknowledged within 1 minute
84 +* ≥80% of interventions judged "timely" in AAR interviews
85 +* ≥50% of teams adjust tactics or rest cycles based on health data
86 +* ≥1 health-based lesson identified per team in debrief
87 +* ≤2 simulated incidents due to unmanaged fatigue/gas exposure
88 +
89 +**How to Measure**:
90 +
91 +* Log alert timings vs. response time
92 +* Observer notes + medic reports on intervention
93 +* Exit survey: "Did alerts help prevent fatigue/injury?"
94 +* Use of wearable dashboard during debrief (Yes/No)
95 +* NASA-TLX for responders
96 +
97 +----
98 +
99 +=== **Module 3 – Indoor Drone Search (Barracks)** ===
100 +
101 +**Use Cases**: UC02.1 and UC02.2
102 +**Scenario**: Collapsed barracks building. Indoor drone used for autonomous scan. Analyst tags victims, hazards, and updates C3I map. Drone does close inspection on request.
103 +
104 +**Tested Functions**:
105 +
106 +* Pre-entry thermal scan
107 +* Hazard/victim detection
108 +* Analyst-supported interpretation and tagging
109 +* Entry planning based on drone data
110 +
111 +**Measured Claims**:
112 +
113 +* CL1: Heightened SA before entry
114 +* CL2: Increased safety (less exposure)
115 +* CL3–CL5: Faster, more accurate victim detection
116 +* CL6: Trust in drone data
117 +* CL7: Increased mission efficiency
118 +
119 +**Quantifiable Success Factors**:
120 +
121 +* ≥90% of dummy victims detected by drone+analyst
122 +* ≥2 new hazards marked per team from drone feed
123 +* Average time-to-first victim ≤ 3 minutes
124 +* ≥80% of responders rate drone info as “trustworthy” (score ≥4/5)
125 +* ≤1 injury due to unknown hazard in follow-up entry
126 +
127 +**How to Measure**:
128 +
129 +* Victim tags placed in known positions for ground truth
130 +* Observer logs: detection times and analyst confirmations
131 +* Team SA quiz: "How many victims? Where were they located?"
132 +* Trust survey: “I would act on this drone data” (1–5)
133 +* Entry path compared to drone hazard map
134 +
135 +----
136 +
137 +=== **Module 4 – Precision Inspection with ANYMAL/SNAKE** ===
138 +
139 +**Use Case**: UC04.0
140 +**Scenario**: Teams reach unstable voids. Robots are deployed to inspect inaccessible areas. SNAKE arm is used to look into cracks. Results update team maps and entry plans.
141 +
142 +**Tested Functions**:
143 +
144 +* Autonomous or manual ANYMAL movement
145 +* Void inspection using flexible arm
146 +* Victim/hazard confirmation
147 +* Decision-making based on robot visuals
148 +
149 +**Measured Claims**:
150 +
151 +* CL1: Access without risk
152 +* CL2: Detection in confined space
153 +* CL3: Safer routing
154 +* CL4: Trust in robot-assessed visuals
155 +* CL5: Faster room clearing
156 +
157 +**Quantifiable Success Factors**:
158 +
159 +* ≥2 hazards or victims confirmed via SNAKE per team
160 +* ≥80% of voids scanned without human entry
161 +* ≥70% of teams adjust route based on robot findings
162 +* ≥80% of participants rate robot visuals as “clear and usable”
163 +* Average inspection time ≤ 8 minutes per room
164 +
165 +**How to Measure**:
166 +
167 +* Observer log: robot path vs. human path
168 +* Detection log compared to known hidden items
169 +* Survey: “Did robot findings improve your plan?” (Yes/No)
170 +* Video review of time-per-room
171 +* Trust in visuals scale (1–5)
172 +
173 +
174 +
18 18  == 2.4 Measures ==
19 19  
177 +This section describes how each claim will be measured during each module, using a combination of objective logging, observer annotations, post-task surveys, and scenario-based evaluation.
20 20  
179 +----
180 +
181 +=== **Module 1 – Wide Area Assessment (UC03.0)** ===
182 +
183 +|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
184 +|CL1 – Improved SA|Number of hazards correctly identified on team maps|SAGAT-lite: Pre/post map-drawing task + verbal hazard recall|≥80% match with ground-truth hazard list
185 +|CL2 – Safer planning|Number of hazard zones avoided during later entry|Observer logs cross-referenced with hazard map|100% of marked hazards avoided
186 +|CL3 – Faster planning|Time from drone launch to team briefing|Stopwatch & observer notes|≤10 minutes total
187 +|CL4 – Reduced workload|Mental workload score of command & drone operator|NASA-TLX (short form)|≤50 average score
188 +|CL5 – Shared SA|Consistency between team and command in map data|Comparison of annotations across roles|≥90% agreement on key features
189 +
190 +
191 +
192 +----
193 +
194 +=== **Module 2 – Health Monitoring & Reconnaissance (UC01.1 / UC01.2)** ===
195 +
196 +|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
197 +|CL1 – Prevent overload|HR trend + alert timing vs. pause/extraction|Wearable logs + observer notes|≥90% alerts followed by correct action within 1 minute
198 +|CL2 – Responder awareness|Survey response on self-adjustment|Post-task Likert: “The alert helped me act”|≥80% rate 4 or 5
199 +|CL3 – Remote escalation|Alert-to-medic contact time|System log + stopwatch|≤1 minute average
200 +|CL4 – Medical support|Alignment of alerts with medical assessment|Medic forms + sensor log correlation|≥80% concordance
201 +|CL5 – Operational planning|Number of rest/rotation decisions based on dashboard|Observer logs + team lead AAR|≥50% of teams adapt plan
202 +|CL6 – AAR use of health data|Was biometric data used during debrief?|Debrief analysis|Yes, per team
203 +|CL7 – Mission effectiveness|Task time + incidents avoided|Stopwatch + incident log|Task time not slower than baseline; 0 uncontrolled fatigue/gas incidents
204 +
205 +
206 +
207 +----
208 +
209 +=== **Module 3 – Indoor Drone Search (UC02.1 / UC02.2)** ===
210 +
211 +|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
212 +|CL1 – Heightened SA|SA questionnaire + map task|Pre/post: victims, layout, hazard count|≥80% correct recall post-drone
213 +|CL2 – Increased safety|Hazard zone avoidance rate|Observer vs. ground truth map|≥90% of flagged areas avoided
214 +|CL3 – Faster victim detection|Time to first detection|Stopwatch from drone entry|≤3 minutes
215 +|CL4 – Accuracy of detection|Victim detection rate|Drone log vs. planted victims|≥90% detected
216 +|CL5 – Trust in results|Survey: “I trust the drone data for decision-making”|1–5 Likert scale|≥80% rate 4 or 5
217 +|CL6 – Efficiency|Entry time after drone plan vs. without drone|Stopwatch; compare with baseline data|10–20% faster planning phase
218 +
219 +
220 +
221 +----
222 +
223 +=== **Module 4 – Robot-Based Precision Inspection (UC04.0)** ===
224 +
225 +|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
226 +|CL1 – Extended reach|Percentage of voids explored by robot not human|Observer log + inspection plan|≥80% of voids scanned by robot
227 +|CL2 – Detection in small spaces|Victim/hazard detection in hidden locations|Camera log vs. planted markers|≥2 findings per team
228 +|CL3 – Safer routing|Route changes based on robot input|Pre/post plan comparison + observer notes|≥70% of teams adapt plan
229 +|CL4 – Trust in visuals|Survey on clarity and trust in robot data|Likert: “The robot data was sufficient for decisions”|≥80% rate 4 or 5
230 +|CL5 – Room clearing speed|Time per room before vs. after robot scout|Stopwatch log|≤8 minutes per room avg.
231 +
232 +
233 +
234 +----
235 +
21 21  == 2.5 Procedure ==
22 22  
238 +All modules follow a similar four-part procedure, tailored per use case.
23 23  
240 +=== **General Daily Timeline** ===
241 +
242 +* **08:30 – 09:00**: Morning briefing, safety, tech setup
243 +* **09:00 – 12:00**: First module rotation (two parallel teams)
244 +* **13:00 – 16:00**: Second module rotation (two parallel teams)
245 +* **16:00 – 17:00**: Shared after-action review
246 +
247 +Each module runs with the following structure:
248 +
249 +=== **Per Module Procedure** ===
250 +
251 +1. (((
252 +**Briefing (10–15 min)**
253 +
254 +* Explain objectives, scenario, roles, safety, success factors
255 +* Introduce technology and expectations
256 +)))
257 +1. (((
258 +**Execution Phase (45–60 min)**
259 +
260 +* Scenario runs in real time
261 +* Observer logs events, actions, communications
262 +* System logs recorded (drone, robot, wearables)
263 +)))
264 +1. (((
265 +**Measurement Phase (15–20 min)**
266 +
267 +* Paper or tablet surveys: SA, trust, NASA-TLX
268 +* Sensor data downloaded to central system
269 +* Short interview or checklist with operator and team lead
270 +)))
271 +1. (((
272 +**Debrief (15–20 min)**
273 +
274 +* Team reflects on use of technology, decision-making
275 +* Facilitator prompts discussion of claims (trust, effectiveness, awareness)
276 +* Recorded notes for final reporting
277 +)))
278 +
279 +For cross-checking performance without the tech, one team per module may be assigned a simplified "control" version of the scenario, using conventional tools only (where feasible).
280 +
281 +----
282 +
24 24  == 2.6 Material ==
25 25  
285 +Each module requires scenario-specific equipment, environmental props, logging tools, and survey forms:
26 26  
27 -= 3. Results =
287 +=== **Common Materials (all modules)** ===
28 28  
289 +* Observer logbooks (standardized per module)
290 +* Stopwatch or time-tracking app
291 +* Participant role badges and checklists
292 +* Data collection station with tablets/laptops
293 +* Printed Likert-scale surveys (SA, trust, workload)
294 +* SAGAT-lite map templates
29 29  
30 -= 4. Discussion =
296 +=== **Module-Specific Materials** ===
31 31  
298 +**Module 1 – Wide Area Assessment**
32 32  
33 -= 5. Conclusions =
300 +* Outdoor drones with RTK GPS and live zoom cameras
301 +* Command screen with drone feed
302 +* Large printed site maps with hazard zones (for scoring)
303 +* Structural hazard props (collapsed façades, signs)
34 34  
35 -
305 +**Module 2 – Health Monitoring**
306 +
307 +* Wearable sensors (HR, hydration, gas; real or simulated)
308 +* Dashboard software for live feed + logging
309 +* Incident trigger devices (e.g., CO2 canisters, alarms)
310 +* Medic checklist sheets
311 +* Alert simulation software (optional)
312 +
313 +**Module 3 – Indoor Drone Search**
314 +
315 +* Thermal indoor drone with autonomous mode
316 +* C3I-compatible map annotation system
317 +* Dummy victims with heat packs or QR markers
318 +* Printed room layouts for SA testing
319 +* Indoor hazard props (rubble, fake smoke, blocked doors)
320 +
321 +**Module 4 – ANYMAL and SNAKE**
322 +
323 +* ANYMAL robot (legged) and SNAKE articulated arm
324 +* Confined space mockups (voids, crawlspaces, stairs)
325 +* Hidden hazard/victim tags inside small cavities
326 +* Robot operator station + external monitor
327 +* Scenario map with route overlays
328 +
329 += 3. Results =
330 +
331 += 4. Discussion =
332 +
333 += 5. Conclusions =
XWiki.XWikiComments[0]
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.MarkVanKoningsveld
Comment
... ... @@ -1,0 +1,25 @@
1 += 1. Introduction[[Edit>>url:https://wisce.synergise-project.eu/bin/edit/3.%20Evaluation/h.%20Test/WebHome?section=1&form_token=ThxE4oHo2FgcrNOPAkA8sw]] =
2 +
3 +//<include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case>//
4 +
5 +Claims uit [[UC01.1: Health and environmental monitoring (Firefighters)>>url:https://wisce.synergise-project.eu/bin/view/2.%20Specification/b.%20Use%20Cases/UC01.0%3A%20Health%20Sensors/Usecase%3A%20Health%20sensors%20%28Firefighters%29/]] die getest worden:
6 +
7 +* Weten ze waar tocix gas is?
8 +* commander weet wat de situatie van zijn personeel is?
9 +* Reddingsmedewerk heeft SA over hun eigen status (genoeg dat ze optijd kunnen reageren) (d.m.v. communicatie met commander of d.m.v. trillen sensor)?
10 +* HQ krijgt voldoende (en op het juiste moment) informatie over de situatie in het veld om ondersteuning te kunnen bieden?
11 +
12 +Claims uit UC01.2:Health and environmental monitoring (USAR) die getest worden:
13 +
14 +* Zelfde als hierboven maar dan iets aangepast voor USAR
15 +
16 +Claims uit UC02.2: Indoor Drone Exploration and Victim Detection (USAR) die getest worden:
17 +
18 +* Weten first responders (genoeg) wat er binnen is om veilig naar binnen te gaan? hebben ze verhoogde SA van de binnenkant van een gebouw? SA/reliance
19 +* Kunnen er beter en sneller victims worden gevonden? > speed, task performance
20 +* Kunnen er meer betrouwbare analyses worden gemaakt van de binnenkant van een gebouw door bijv. een plan maken voor een veilige/ begaanbare route > SA
21 +* Task performance: kunnen er sneller en meer gestroomlijnd victim reports worden gemaakt en gedeeld (essentie = gaat victim assessement beter)?
22 +
23 +Claims uit UC02.1: Indoor Drone Exploration and Victim Detection (Firefighters) die getest worden:
24 +
25 +* zelfde als hierboven maar dan iets meer aangepast voor Firefighters
Date
... ... @@ -1,0 +1,1 @@
1 +2025-07-13 22:23:32.283