Skip to Content

Wiki source code of j. SFT1 evaluation templates

Version 25.6 by Tjalling Haije on 2026/05/04 14:21
Show last authors
1 = General =
2
3 [[image:1776255113603-367.png]]
4
5
6 |(% colspan="3" %)[[image:1776259393121-951.png]]
7 |**Baseline phases**|**Who fills it**|**Main purpose**
8 |//Briefing//|-|-
9 |//Estimate building integrity//|-|-
10 |//Shoring//|Particpants|(make estimation about shoring time)
11 |//Before sending in FRs (DM)//|Participant|Ground truth/ baseline SA
12 |//Send in FRs, identify voids, DM//|Evaluation leader|Objective performance, safety, timing, decision making
13 |//FRs exit building//|Participant|Subjective SA, trust, workload, decision making
14 |//Hotwash//| |
15
16 |(% colspan="3" style="width:414px" %)[[image:1776259356197-904.png]]
17 |(% style="width:414px" %)**SYNERGISE phases**|(% style="width:245px" %)**Who fills it**|(% style="width:690px" %)**Main purpose**
18 |(% style="width:414px" %)//Briefing//|(% style="width:245px" %)-|(% style="width:690px" %)-
19 |(% style="width:414px" %)//Estimate building integrity//|(% style="width:245px" %)-|(% style="width:690px" %)-
20 |(% style="width:414px" %)//Send in robots, identify voids with robots//|(% style="width:245px" %)-|(% style="width:690px" %)-
21 |(% style="width:414px" %)//Decision Making (DM)//|(% style="width:245px" %)Participants (Squad/ team leader)|(% style="width:690px" %)Ground truth/ baseline SA
22 |(% style="width:414px" %)//Send in FRs ?//|(% style="width:245px" %)Evaluation leader|(% style="width:690px" %)Objective performance, safety, timing, decision making
23 |(% style="width:414px" %)//After test//|(% style="width:245px" %)Paricipant|(% style="width:690px" %)Subjective SA, trust, workload, decision making
24 |(% style="width:414px" %)//Hotwash//|(% style="width:245px" %) |(% style="width:690px" %)
25
26 == List of ground truths needed for comparison: ==
27
28 * Health ranges
29 * hazards/ victims in buildings
30 * Safest path within buildings
31 * State of automation (tech question)
32
33
34
35 = Templates per role =
36
37 (% class="table-bordered" %)
38 |(% colspan="2" %)(((
39 == Template: Evaluation leader (Field) ==
40 )))
41 |(% colspan="2" %)(((
42 **General: **
43
44 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
45 )))
46 |(% style="width:296px" %)(((
47 System & mission readiness
48
49 //(Tick list based on predefined acceptable ranges. Cross out if not applicable.)//
50 )))|(% style="width:1053px" %)(((
51 ☐ C3I operational, describe: .......................................................................................................................................................................................................................
52
53 ☐ Health & location sensors connected, describe: .....................................................................................................................................................................................
54
55 ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................
56
57 ☐ OWL operational, describe: .....................................................................................................................................................................................................................
58
59 ☐ ANYmal operational, describe: ................................................................................................................................................................................................................
60
61 ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................
62
63 ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
64 )))
65 |(% colspan="2" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
66 |(% style="width:296px" %)Hand out questionnaires|(% style="width:1053px" %)Each role their own question for this section
67 |(% style="width:296px" %) |(% style="width:1053px" %)
68 |(% style="width:296px" %)(((
69 **Threshold breached, alert and intervention record **
70 )))|(% style="width:1053px" %)(((
71 Health threshold breached (timestamp + action taken)
72
73 Alert detection by system (timestamp + action taken)
74
75 Intervention started (timestamp + action taken)
76 )))
77 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
78 |(% style="width:296px" %)(((
79 **Time within building/ hazard zone (humans)**
80 )))|(% style="width:1053px" %)(((
81 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
82 )))
83 |(% style="width:296px" %)(((
84 **Time within building/ hazard zone (robots)**
85 )))|(% style="width:1053px" %)(((
86 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
87 )))
88 |(% style="width:296px" %)(((
89 **Safety observations **
90 )))|(% style="width:1053px" %)(((
91 ☐ No unsafe situations observed
92 ☐ Unsafe / near‑incident observed, describe (amount and type) :..................................................................................................................................................................
93
94 .........................................................................................................................................................................................................................................................................
95 )))
96 |(% style="width:296px" %)(((
97 **Path deviations observed**
98
99 //(Check if the entry team takes the route that is discussed before entering)//
100 )))|(% style="width:1053px" %)(((
101 ☐ None ☐ Minor ☐ Major
102
103 Notes:
104 )))
105 |(% style="width:296px" %) |(% style="width:1053px" %)
106 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
107 |(% style="width:296px" %)Give every role their questionnaire|(% style="width:1053px" %)
108 |(% style="width:296px" %)Plenary questions: |(% style="width:1053px" %)...??
109
110 (% class="table-bordered" %)
111 |(% colspan="2" %)(((
112 == Template: Squad/ Team leader (Field) ==
113 )))
114 |(% colspan="2" %)(((
115 **General: **
116
117 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
118 )))
119 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
120 |(% style="width:296px" %)(((
121 (CL01) Safety Plan Confirmation
122 )))|(% style="width:1053px" %)(((
123 Mark chosen entry path on provided building map. //Sketch or describe entry path. (Sketch space)//
124
125
126
127
128
129 )))
130 |(% style="width:296px" %)(CL02) Situation Awareness (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
131 List expected hazards and victims. //(Bullets and/ or sketch)//
132
133
134
135
136 )))
137 |(% style="width:296px" %)(CL02) Situation Awareness interior (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
138 Sketch/ Describe expected interior layout. //(Sketch space)//
139
140
141
142
143
144 )))
145 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
146 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
147 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
148 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
149 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
150 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
151 )))
152 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
153 |(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)|(% style="width:1053px" %)(((
154 Where are you team members now and what hazards are near them? //(Sketch space)//
155
156
157
158
159
160 )))
161 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
162 |(% style="width:296px" %)(((
163 (CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):
164 )))|(% style="width:1053px" %)(((
165 |(% style="width:587px" %)Statement|(% style="width:138px" %)1 (LOW)|(% style="width:60px" %)2|(% style="width:57px" %)3|(% style="width:59px" %)4|(% style="width:58px" %)5|(% style="width:60px" %)6|(% style="width:147px" %)7 (HIGH)
166 |(% style="width:587px" %)(((
167 I had a clear understanding of the mission objectives, my role, and the intended plan before execution.
168 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
169 |(% style="width:587px" %)(((
170 During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.
171 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
172 |(% style="width:587px" %)(((
173 The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.
174 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
175 |(% style="width:587px" %)(((
176 Team coordination and communication were effective and supported successful task execution.
177 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
178 |(% style="width:587px" %)(((
179 The tools, systems, or support aids available during the mission effectively supported mission execution.
180 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
181 |(% style="width:587px" %)(((
182 Based on this exercise, the team is better prepared to perform similar missions in the future.
183 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
184
185 (% class="box warningmessage" %)
186 (((
187 OR (open questions better?)
188 )))
189
190 |(% style="width:587px" %)Open questions
191 |(% style="width:587px" %)(((
192 What was the plan/intent for the mission, and what did you personally aim to achieve?// (1–2 sentences)//
193
194
195
196 )))
197 |(% style="width:587px" %)(((
198 What actually happened? List the 2–3 most important events or turning points. //(bullets)//
199
200
201
202 )))
203 |(% style="width:587px" %)(((
204 What went well and should be sustained? Give one concrete example. //(example + why it worked)//
205
206
207
208 )))
209 |(% style="width:587px" %)(((
210 What did not go as intended? Give one concrete example and its impact. //(example + consequence)//
211
212
213
214 )))
215 |(% style="width:587px" %)(((
216 Why did it happen? What were the main contributing factors? //(e.g., information/SA, communication, coordination, timing, tools/technology, environment)//
217
218
219
220 )))
221 |(% style="width:587px" %)(((
222 What are the top 2 actionable improvements for next time (specific, doable)? //(Action + owner/role + when)//
223
224
225
226 )))
227 )))
228 |(% style="width:296px" %)(((
229 (CL01) Safety Error Self-Assessment
230 )))|(% style="width:1053px" %)(((
231 List any unsafe or mistaken decisions you recognize from this mission (if any). //(Provide brief descriptions)//
232
233
234
235 )))
236 |(% style="width:296px" %)(((
237 (CL02) Situation Awareness (Post-Scenario SAGAT)
238 )))|(% style="width:1053px" %)(((
239 From memory, sketch the interior layout and mark all hazards and victims encountered. //(Sketch space and bullets)//
240
241
242
243
244 )))
245 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
246 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
247 Give a score between:
248
249 0 (very low) - 100 (very high)
250 )))
251 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
252 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
253 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
254 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
255 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
256 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
257 )))
258 |(% style="width:296px" %)(((
259 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
260 )))|(% style="width:1053px" %)(((
261 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
262 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
263 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
264 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
265 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
266 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
267 )))
268
269 == ==
270
271 (% class="table-bordered" %)
272 |(% colspan="2" %)(((
273 == Template: Robot operator ==
274 )))
275 |(% colspan="2" %)(((
276 **General: **
277
278 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
279 )))
280 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
281 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). |(% style="width:1053px" %)(((
282 Before entering, list the hazards or victims you expect to encounter, based on briefing info. //(Bullets)//
283
284 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
285 |The system is deceptive (misleading or deceiving).|☐|☐|☐|☐|☐|☐|☐
286 |The system behaves in an underhanded (shady or dishonest) manner.|☐|☐|☐|☐|☐|☐|☐
287 |I am suspicious of the system's intent, action, or outputs.|☐|☐|☐|☐|☐|☐|☐
288 |I am wary (cautious or careful) of the system.|☐|☐|☐|☐|☐|☐|☐
289 |The system's actions will have a harmful or injurious outcome.|☐|☐|☐|☐|☐|☐|☐
290 |I am confident in the system.|☐|☐|☐|☐|☐|☐|☐
291 |The system provides security.|☐|☐|☐|☐|☐|☐|☐
292 |The system has integrity (honesty or honor). |☐|☐|☐|☐|☐|☐|☐
293 |The system is dependable.|☐|☐|☐|☐|☐|☐|☐
294 |The system is reliable.|☐|☐|☐|☐|☐|☐|☐
295 |I can trust the system.|☐|☐|☐|☐|☐|☐|☐
296 |I am familiar with the system.|☐|☐|☐|☐|☐|☐|☐
297 )))
298 |(% style="width:296px" %)(((
299 (CL02) Situation Awareness Robot Reconnaissance data understanding (OPTIONAL; is more important for the analyst)
300 )))|(% style="width:1053px" %)(((
301 After the robot’s initial run but before humans enter, list any //hazards, obstacles, or victims// that you (as operator) detected or suspect from the robot’s feed. //(Open list)//
302
303
304
305
306 )))
307 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
308 |(% style="width:296px" %)(CL02/ CL04) Operational notes|(% style="width:1053px" %)(((
309 Logging of manual control or override the robot's/ system's suggestions (with timestamp) as well as any alerts ignored.
310
311
312
313 )))
314 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
315 |(% style="width:296px" %)(((
316 (CL01) SafetyTeam Decision Appraisal:
317 )))|(% style="width:1053px" %)(((
318 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
319 |How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
320 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
321 )))
322 |(% style="width:296px" %)(((
323 (CL01) Safety Error Reflection
324 )))|(% style="width:1053px" %)(((
325 List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
326
327
328
329
330 )))
331 |(% style="width:296px" %)(((
332 (CL02) Situation Awareness (Post-Scenario SAGAT)
333 )))|(% style="width:1053px" %)(((
334 Describe what the robot observed/ found during the mission (e.g. hazards, obstacles) for navigation purposes. //(Sketch space and bullets)//
335
336
337
338
339 )))
340 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
341 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
342 Give a score between:
343
344 0 (very low) - 100 (very high)
345 )))
346 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
347 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
348 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
349 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
350 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
351 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
352 )))
353 |(% style="width:296px" %)(((
354 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
355 )))|(% style="width:1053px" %)(((
356 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
357 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
358 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
359 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
360 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
361 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
362 )))
363
364
365
366 (% class="table-bordered" %)
367 |(% colspan="2" %)(((
368 == Template: Robot analyst ==
369 )))
370 |(% colspan="2" %)(((
371 **General: **
372
373 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
374 )))
375 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
376 |(% style="width:445px" %)(((
377 (CL02) Situation Awareness Robot Reconnaissance Assessment (SAGAT expected situation report)
378 )))|(% style="width:1071px" %)(((
379 Based on the robot’s pre-entry scan, list the hazards, victims, and notable features you have identified or expect inside. //(Bullets and/or sketch)//
380
381
382
383
384 )))
385 |(% style="width:296px" %)(((
386 (CL04) Trust baseline
387 )))|(% style="width:1053px" %)(((
388 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
389 |(((
390 The system’s data is trustworthy
391 )))|☐|☐|☐|☐|☐|☐|☐
392 |(((
393 can rely on the robot to detect hazards I might miss
394 )))| | | | | | |
395 |....?| | | | | | |
396 )))
397 |(% colspan="2" style="width:445px" %)**POST EXIT PHASE (hot wash)**
398 |(% style="width:296px" %)(((
399 (CL01) SafetyTeam Decision Appraisal:
400 )))|(% style="width:1053px" %)(((
401 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
402 |How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
403 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
404 )))
405 |(% style="width:296px" %)(((
406 (CL01) Safety Error Reflection
407 )))|(% style="width:1053px" %)(((
408 List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
409
410
411
412
413 )))
414 |(% style="width:296px" %)(((
415 (CL02) Situation Awareness (Post-Scenario SAGAT)
416 )))|(% style="width:1053px" %)(((
417 Now that the mission is over, summarize the final floor layout, and list all hazards and victims found or confirmed. //(Sketch space and bullets)//
418
419
420
421
422 )))
423 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
424 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
425 Give a score between:
426
427 0 (very low) - 100 (very high)
428 )))
429 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
430 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
431 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
432 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
433 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
434 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
435 )))
436 |(% style="width:296px" %)(((
437 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000)
438 )))|(% style="width:1053px" %)(((
439 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
440 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
441 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
442 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
443 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
444 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
445 )))
446
447
448
449 (% class="table-bordered" %)
450 |(% colspan="2" %)(((
451 == Template: Entry team member ==
452 )))
453 |(% colspan="2" %)(((
454 **General: **
455
456 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
457 )))
458 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
459 |(% style="width:296px" %)(((
460 (CL02) Situation Awareness (Pre-mission knowledge check):
461 )))|(% style="width:1053px" %)(((
462 Before entering, list the hazards or victims you expect to encounter, based on briefing info. //(Bullets)//
463
464
465
466 )))
467 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
468 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
469 |I am confident in the robot analyst's capabilities.|☐|☐|☐|☐|☐|☐|☐
470 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
471 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
472 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
473 )))
474 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
475 |(% style="width:296px" %)(((
476 (CL01) Safety - Team Decision Appraisal:
477 )))|(% style="width:1053px" %)(((
478 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
479 |How would you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
480 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
481 )))
482 |(% style="width:296px" %)(((
483 (CL02) Situation Awareness Post Operation Recall
484 )))|(% style="width:1053px" %)(((
485 Describe the layout and list all hazards/victims that were found. //(Sketch and describe)//
486
487
488
489
490
491 )))
492 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
493 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
494 Give a score between:
495
496 0 (very low) - 100 (very high)
497 )))
498 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
499 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
500 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
501 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
502 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
503 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
504 )))
505 |(% style="width:296px" %)(((
506 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
507 )))|(% style="width:1053px" %)(((
508 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
509 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
510 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
511 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
512 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
513 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
514 )))
515
516
517
518 (% class="table-bordered" %)
519 |(% colspan="2" %)(((
520 == Template: Base of Operations (LEMA) ==
521 )))
522 |(% colspan="2" %)(((
523 **General: **
524
525 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
526 )))
527 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
528 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
529 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
530 |I am confident in the robot analyst's capabilities.|☐|☐|☐|☐|☐|☐|☐
531 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
532 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
533 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
534 )))
535 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
536 |(% style="width:296px" %)(CL01, CL03-05) Safety and Effectiveness|(% style="width:1053px" %)(((
537 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
538 |From the command (strategic) perspective, rate the overall mission outcome on safety and effectiveness.|☐|☐|☐|☐|☐|☐|☐
539 )))
540 |(% style="width:296px" %)(((
541 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
542 )))|(% style="width:1053px" %)(((
543 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
544 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
545 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
546 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
547 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
548 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
549 )))
550 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
551 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
552 Give a score between:
553
554 0 (very low) - 100 (very high)
555 )))
556 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
557 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
558 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
559 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
560 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
561 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
562 )))
563
564 === Before humans enter a building/ hazard ===
565
566 |**Claim**|**Measurement**|**What is recorded**|**Who records**
567 |CL01 Safety|Ground-truth safe paths|predefined safe/ suboptimal / unsafe paths|//Evaluation leader//
568 |(((
569 CL01 Safety
570 )))|(((
571 Chosen entry path
572 )))|(((
573 Intended entry path selection
574 )))|//Squad/ Team leader (informed by Robot analyst)//
575 |(((
576 CL02 Situation Awareness
577 )))|(((
578 Reported hazards, victims, layout (pre)
579 )))|(((
580 Sketch / map / description of interior
581 )))|//Squad/ Team leader, Entry team, Robot analyst (separately)//
582 |(((
583 CL02 SA
584 )))|(((
585 Robot‑based expectations
586 )))|(((
587 Expected hazards / victims from robot data
588 )))|//Robot analyst//
589 |CL05 Trustworthy SA|Baseline trust (optional)|Initial trust in system/ robot|//Squad/ Team leader, Robot operator, Robot analyst//
590 |(((
591 CL06 Health
592 )))|(((
593 Acceptable health ranges
594 )))|(((
595 HR, temp thresholds
596 )))|//Evaluation leader//
597 |(((
598 CL07 Efficiency
599 )))|(((
600 Mission start timestamp
601 )))|(((
602 End briefing → start
603 )))|//Evaluation leader//
604
605 During humans/robots are inside building/ hazard
606
607 |**Claim**|**Measurement**|**What is recorded**|**Who records**
608 |(((
609 CL01 Safety
610 )))|(((
611 Near‑incidents
612 )))|(((
613 Unsafe situations / close calls
614 )))|//Evaluation leader//
615 |(((
616 CL01 Safety
617 )))|(((
618 Hazard avoidance
619 )))|(((
620 Avoided predefined hazards
621 )))|//Evaluation leader//
622 |(((
623 CL02 SA
624 )))|(((
625 Executed path
626 )))|(((
627 Actual path taken
628 )))|//Evaluation leader, Engage system/ location sensors (?)//
629 |(((
630 CL02 SA
631 )))|(((
632 Path deviations
633 )))|(((
634 Deviations + cause (something for after instead?)
635 )))|//Squad/ Team leader, Entry team//
636 |(((
637 CL02 SA
638 )))|(((
639 Interpretations of robot images
640 )))|(((
641 ...?
642 )))|//Robot analyst (and Base or Operations/ LEMA ?)//
643 |(((
644 CL02 SA (Optional)
645 )))|(((
646 Robot control actions
647 )))|(((
648 Overrides, manual interventions
649 )))|//Robot operator//
650 |(((
651 CL06 Health
652 )))|(((
653 Threshold breach
654 )))|(((
655 Breach moment (HR, temp, stress (?))
656 )))|//Evaluation leader//
657 |(((
658 CL06 Health
659 )))|(((
660 Detection & intervention
661 )))|(((
662 Detection with corresponding action
663 )))|//Evaluation leader//
664 |(((
665 CL07 Efficiency
666 )))|(((
667 Entry timestamp
668 )))|(((
669 Entry team enters OR robot enters
670 )))|//Evaluation leader//
671 |(((
672 CL07 Efficiency
673 )))|(((
674 Exit timestamp
675 )))|Entry team exits OR robot exits |//Evaluation leader//
676
677 === After humans/robots exited building/hazard ===
678
679 |**Claim**|**Measurement**|**What is recorded**|**Who records**
680 |(((
681
682
683 CL01 Safety
684 )))|(((
685
686
687 Decision quality (retro)
688 )))|(((
689
690
691 “How good were decisions?”
692 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
693 |(((
694 CL01 Safety
695 )))|(((
696 Error reflection
697 )))|(((
698 Reflected unsafe choices
699 )))|//Squad/ Team leader, Robot operator, Robot analyst//
700 |(((
701 CL02 SA
702 )))|(((
703 Reported hazards, victims, layout (post)
704 )))|(((
705 Sketch / map / description
706 )))|//Squad/ Team leader, Entry team, Robot analyst//
707 |CL03 mission effectiveness|Decision and path quality|Route of FRs, decisions made.|//Squad/ Team leader, Entry team, Robot analyst//
708 |(((
709 CL04 acceptable workload
710 )))|(((
711 Workload
712 )))|(((
713 Short NASA‑TLX
714 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
715 |(((
716 CL05 Trustworthy SA
717 )))|(((
718 Trust
719 )))|(((
720 Trust survey / interview
721 )))|//Squad/ Team leader, Entry team (?), Robot operator, Robot analyst, Baes of Operations/ LEMA//
722 |(((
723 CL06 Health
724 )))|(((
725 Health issue handling
726 )))|(((
727 Adequacy & timeliness
728 )))|//Evaluation leader (+ expert)//
729 |(((
730 CL07 Efficiency
731 )))|(((
732 Total mission duration
733 )))|(((
734 Start (end briefing) till end (?)
735 )))|//Evaluation leader//
736
737