Skip to Content

Wiki source code of j. SFT1 evaluation templates

Version 23.2 by Rosa Van Tuijn on 2026/04/20 15:36
Show last authors
1 = General =
2
3 [[image:1776255113603-367.png]]
4
5
6 |(% colspan="3" %)[[image:1776259393121-951.png]]
7 |**Baseline phases**|**Who fills it**|**Main purpose**
8 |//Briefing//|-|-
9 |//Estimate building integrity//|-|-
10 |//Shoring//|Particpants|(make estimation about shoring time)
11 |//Before sending in FRs (DM)//|Participant|Ground truth/ baseline SA
12 |//Send in FRs, identify voids, DM//|Evaluation leader|Objective performance, safety, timing, decision making
13 |//FRs exit building//|Participant|Subjective SA, trust, workload, decision making
14 |//Hotwash//| |
15
16 |(% colspan="3" style="width:414px" %)[[image:1776259356197-904.png]]
17 |(% style="width:414px" %)**SYNERGISE phases**|(% style="width:245px" %)**Who fills it**|(% style="width:690px" %)**Main purpose**
18 |(% style="width:414px" %)//Briefing//|(% style="width:245px" %)-|(% style="width:690px" %)-
19 |(% style="width:414px" %)//Estimate building integrity//|(% style="width:245px" %)-|(% style="width:690px" %)-
20 |(% style="width:414px" %)//Send in robots, identify voids with robots//|(% style="width:245px" %)-|(% style="width:690px" %)-
21 |(% style="width:414px" %)//Decision Making (DM)//|(% style="width:245px" %)Participants (Squad/ team leader)|(% style="width:690px" %)Ground truth/ baseline SA
22 |(% style="width:414px" %)//Send in FRs ?//|(% style="width:245px" %)Evaluation leader|(% style="width:690px" %)Objective performance, safety, timing, decision making
23 |(% style="width:414px" %)//After test//|(% style="width:245px" %)Paricipant|(% style="width:690px" %)Subjective SA, trust, workload, decision making
24 |(% style="width:414px" %)//Hotwash//|(% style="width:245px" %) |(% style="width:690px" %)
25
26 (% class="wikigeneratedid" %)
27 List of ground truths needed for comparison:
28
29 * Health ranges
30 * hazards/ victims in buildings
31 * Safest path within buildings
32 * State of automation (tech question)
33
34
35
36 === Before humans enter a building/ hazard ===
37
38 |**Claim**|**Measurement**|**What is recorded**|**Who records**
39 |CL01 Safety|Ground-truth safe paths|predefined safe/ suboptimal / unsafe paths|//Evaluation leader//
40 |(((
41 CL01 Safety
42 )))|(((
43 Chosen entry path
44 )))|(((
45 Intended entry path selection
46 )))|//Squad/ Team leader (informed by Robot analyst)//
47 |(((
48 CL02 Situation Awareness
49 )))|(((
50 Reported hazards, victims, layout (pre)
51 )))|(((
52 Sketch / map / description of interior
53 )))|//Squad/ Team leader, Entry team, Robot analyst (separately)//
54 |(((
55 CL02 SA
56 )))|(((
57 Robot‑based expectations
58 )))|(((
59 Expected hazards / victims from robot data
60 )))|//Robot analyst//
61 |(((
62 CL03–05 Mission effectiveness
63 )))|(((
64 Baseline trust (optional)
65 )))|(((
66 Initial trust in system/ robot
67 )))|//Squad/ Team leader, Robot operator, Robot analyst//
68 |(((
69 CL06 Health
70 )))|(((
71 Acceptable health ranges
72 )))|(((
73 HR, temp thresholds
74 )))|//Evaluation leader//
75 |(((
76 CL07 Efficiency
77 )))|(((
78 Mission start timestamp
79 )))|(((
80 End briefing → start
81 )))|//Evaluation leader//
82
83 During humans/robots are inside building/ hazard
84
85 |**Claim**|**Measurement**|**What is recorded**|**Who records**
86 |(((
87 CL01 Safety
88 )))|(((
89 Near‑incidents
90 )))|(((
91 Unsafe situations / close calls
92 )))|//Evaluation leader//
93 |(((
94 CL01 Safety
95 )))|(((
96 Hazard avoidance
97 )))|(((
98 Avoided predefined hazards
99 )))|//Evaluation leader//
100 |(((
101 CL02 SA
102 )))|(((
103 Executed path
104 )))|(((
105 Actual path taken
106 )))|//Evaluation leader, Engage system/ location sensors (?)//
107 |(((
108 CL02 SA
109 )))|(((
110 Path deviations
111 )))|(((
112 Deviations + cause (something for after instead?)
113 )))|//Squad/ Team leader, Entry team//
114 |(((
115 CL02 SA
116 )))|(((
117 Interpretations of robot images
118 )))|(((
119 ...?
120 )))|//Robot analyst (and Base or Operations/ LEMA ?)//
121 |(((
122 CL02 SA (Optional)
123 )))|(((
124 Robot control actions
125 )))|(((
126 Overrides, manual interventions
127 )))|//Robot operator//
128 |(((
129 CL06 Health
130 )))|(((
131 Threshold breach
132 )))|(((
133 Breach moment (HR, temp, stress (?))
134 )))|//Evaluation leader//
135 |(((
136 CL06 Health
137 )))|(((
138 Detection & intervention
139 )))|(((
140 Detection with corresponding action
141 )))|//Evaluation leader//
142 |(((
143 CL07 Efficiency
144 )))|(((
145 Entry timestamp
146 )))|(((
147 Entry team enters OR robot enters
148 )))|//Evaluation leader//
149 |(((
150 CL07 Efficiency
151 )))|(((
152 Exit timestamp
153 )))|Entry team exits OR robot exits |//Evaluation leader//
154
155 === After humans/robots exited building/hazard ===
156
157 |**Claim**|**Measurement**|**What is recorded**|**Who records**
158 |(((
159
160
161 CL01 Safety
162 )))|(((
163
164
165 Decision quality (retro)
166 )))|(((
167
168
169 “How good were decisions?”
170 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
171 |(((
172 CL01 Safety
173 )))|(((
174 Error reflection
175 )))|(((
176 Reflected unsafe choices
177 )))|//Squad/ Team leader, Robot operator, Robot analyst//
178 |(((
179 CL02 SA
180 )))|(((
181 Reported hazards, victims, layout (post)
182 )))|(((
183 Sketch / map / description
184 )))|//Squad/ Team leader, Entry team, Robot analyst//
185 |(((
186 CL03 Effectiveness
187 )))|(((
188 Workload
189 )))|(((
190 Short NASA‑TLX
191 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
192 |(((
193 CL04 Effectiveness
194 )))|(((
195 Trust
196 )))|(((
197 Trust survey / interview
198 )))|//Squad/ Team leader, Entry team (?), Robot operator, Robot analyst, Baes of Operations/ LEMA//
199 |(((
200 CL05 Effectiveness
201 )))|(((
202 Decision confidence
203 )))|(((
204 Confidence in own judgments
205 )))|//Squad/ Team leader, Robot operator, Robot analyst//
206 |(((
207 CL06 Health
208 )))|(((
209 Health issue handling
210 )))|(((
211 Adequacy & timeliness
212 )))|//Evaluation leader (+ expert)//
213 |(((
214 CL07 Efficiency
215 )))|(((
216 Total mission duration
217 )))|(((
218 Start (end briefing) till end (?)
219 )))|//Evaluation leader//
220
221
222
223 = Templates per role =
224
225 (% class="table-bordered" %)
226 |(% colspan="2" %)(((
227 == Template: Evaluation leader (Field) ==
228 )))
229 |(% colspan="2" %)//**Before start scenario**//
230 |(% colspan="2" %)(((
231 **General: **
232
233 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
234 )))
235 |(% style="width:296px" %)(((
236 **System & mission readiness**
237
238 //(Tick list based on predefined acceptable ranges. Cross out if not applicable.)//
239 )))|(% style="width:1053px" %)(((
240 ☐ C3I operational, describe: .......................................................................................................................................................................................................................
241
242 ☐ Health & location sensors connected, describe: .....................................................................................................................................................................................
243
244 ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................
245
246 ☐ OWL operational, describe: .....................................................................................................................................................................................................................
247
248 ☐ ANYmal operational, describe: ................................................................................................................................................................................................................
249
250 ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................
251
252 ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
253 )))
254 |(% colspan="2" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
255 |(% style="width:296px" %)**Let Team leader (and entry team?) sketch/ describe hazard, victims, layout**|(% style="width:1053px" %)(((
256 Notes:
257
258
259
260 )))
261 |(% style="width:296px" %)(((
262 **Threshold breached, alert and intervention record **
263 )))|(% style="width:1053px" %)(((
264 Health threshold breached (timestamp + action taken)
265
266 Alert detection by system (timestamp + action taken)
267
268 Intervention started (timestamp + action taken)
269 )))
270 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
271 |(% style="width:296px" %)(((
272 **Time within building/ hazard zone (humans)**
273 )))|(% style="width:1053px" %)(((
274 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
275 )))
276 |(% style="width:296px" %)(((
277 **Time within building/ hazard zone (robots)**
278 )))|(% style="width:1053px" %)(((
279 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
280 )))
281 |(% style="width:296px" %)(((
282 **Safety observations **
283 )))|(% style="width:1053px" %)(((
284 ☐ No unsafe situations observed
285 ☐ Unsafe / near‑incident observed, describe (amount and type) :..................................................................................................................................................................
286
287 .........................................................................................................................................................................................................................................................................
288 )))
289 |(% style="width:296px" %)(((
290 **Path deviations observed**
291
292 //(Check if the entry team takes the route that is discussed before entering)//
293 )))|(% style="width:1053px" %)(((
294 ☐ None ☐ Minor ☐ Major
295
296 Notes:
297 )))
298 |(% style="width:296px" %) |(% style="width:1053px" %)
299 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
300 |(% style="width:296px" %)Give every role their questionnaire|(% style="width:1053px" %)
301 |(% style="width:296px" %)Plenary questions: |(% style="width:1053px" %)...??
302
303 (% class="table-bordered" %)
304 |(% colspan="2" %)(((
305 == Template: Squad/ Team leader (Field) ==
306 )))
307 |(% colspan="2" %)(((
308 **General: **
309
310 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
311 )))
312 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
313 |(% style="width:296px" %)(((
314 (CL01) Safety Plan Confirmation
315 )))|(% style="width:1053px" %)(((
316 Mark chosen entry path on provided building map. //Sketch or describe entry path. (Sketch space)//
317
318
319
320
321
322 )))
323 |(% style="width:296px" %)(CL02) Situation Awareness (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
324 List expected hazards and victims. //(Bullets and/ or sketch)//
325
326
327
328
329 )))
330 |(% style="width:296px" %)(CL02) Situation Awareness interior (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
331 Sketch/ Describe expected interior layout. //(Sketch space)//
332
333
334
335
336
337 )))
338 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
339 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
340 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
341 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
342 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
343 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
344 )))
345 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
346 |(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)|(% style="width:1053px" %)(((
347 Where are you team members now and what hazards are near them? //(Sketch space)//
348
349
350
351
352
353 )))
354 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
355 |(% style="width:296px" %)(((
356 (CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):
357 )))|(% style="width:1053px" %)(((
358 |(% style="width:587px" %)Statement|(% style="width:138px" %)1 (LOW)|(% style="width:60px" %)2|(% style="width:57px" %)3|(% style="width:59px" %)4|(% style="width:58px" %)5|(% style="width:60px" %)6|(% style="width:147px" %)7 (HIGH)
359 |(% style="width:587px" %)(((
360 I had a clear understanding of the mission objectives, my role, and the intended plan before execution.
361 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
362 |(% style="width:587px" %)(((
363 During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.
364 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
365 |(% style="width:587px" %)(((
366 The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.
367 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
368 |(% style="width:587px" %)(((
369 Team coordination and communication were effective and supported successful task execution.
370 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
371 |(% style="width:587px" %)(((
372 The tools, systems, or support aids available during the mission effectively supported mission execution.
373 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
374 |(% style="width:587px" %)(((
375 Based on this exercise, the team is better prepared to perform similar missions in the future.
376 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
377
378 (% class="box warningmessage" %)
379 (((
380 OR (open questions better?)
381 )))
382
383 |(% style="width:587px" %)Open questions
384 |(% style="width:587px" %)(((
385 What was the plan/intent for the mission, and what did you personally aim to achieve?// (1–2 sentences)//
386
387
388
389 )))
390 |(% style="width:587px" %)(((
391 What actually happened? List the 2–3 most important events or turning points. //(bullets)//
392
393
394
395 )))
396 |(% style="width:587px" %)(((
397 What went well and should be sustained? Give one concrete example. //(example + why it worked)//
398
399
400
401 )))
402 |(% style="width:587px" %)(((
403 What did not go as intended? Give one concrete example and its impact. //(example + consequence)//
404
405
406
407 )))
408 |(% style="width:587px" %)(((
409 Why did it happen? What were the main contributing factors? //(e.g., information/SA, communication, coordination, timing, tools/technology, environment)//
410
411
412
413 )))
414 |(% style="width:587px" %)(((
415 What are the top 2 actionable improvements for next time (specific, doable)? //(Action + owner/role + when)//
416
417
418
419 )))
420 )))
421 |(% style="width:296px" %)(((
422 (CL01) Safety Error Self-Assessment
423 )))|(% style="width:1053px" %)(((
424 List any unsafe or mistaken decisions you recognize from this mission (if any). //(Provide brief descriptions)//
425
426
427
428 )))
429 |(% style="width:296px" %)(((
430 (CL02) Situation Awareness (Post-Scenario SAGAT)
431 )))|(% style="width:1053px" %)(((
432 From memory, sketch the interior layout and mark all hazards and victims encountered. //(Sketch space and bullets)//
433
434
435
436
437 )))
438 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
439 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
440 Give a score between:
441
442 0 (very low) - 100 (very high)
443 )))
444 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
445 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
446 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
447 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
448 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
449 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
450 )))
451 |(% style="width:296px" %)(((
452 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
453 )))|(% style="width:1053px" %)(((
454 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
455 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
456 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
457 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
458 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
459 )))
460 |(% style="width:296px" %)(((
461 (CL05) Decision Confidence:
462 )))|(% style="width:1053px" %)(((
463 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
464 Give a score between:
465
466 0 (very low) - 100 (very high)
467 )))
468 |(% style="width:861px" %)How confident are you in the decisions you personally made during this mission? |(% style="width:305px" %)[[image:1776690776402-253.png]]
469 |(% style="width:861px" %)(asked per key decision; e.g. path choice  ?? )|(% style="width:305px" %)[[image:1776690777988-282.png]]
470 )))
471
472 == ==
473
474 (% class="table-bordered" %)
475 |(% colspan="2" %)(((
476 == Template: Robot operator ==
477 )))
478 |(% colspan="2" %)(((
479 **General: **
480
481 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
482 )))
483 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
484 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). |(% style="width:1053px" %)(((
485 Before entering, list the hazards or victims you expect to encounter, based on briefing info. //(Bullets)//
486
487 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
488 |The system is deceptive (misleading or deceiving).|☐|☐|☐|☐|☐|☐|☐
489 |The system behaves in an underhanded (shady or dishonest) manner.|☐|☐|☐|☐|☐|☐|☐
490 |I am suspicious of the system's intent, action, or outputs.|☐|☐|☐|☐|☐|☐|☐
491 |I am wary (cautious or careful) of the system.|☐|☐|☐|☐|☐|☐|☐
492 |The system's actions will have a harmful or injurious outcome.|☐|☐|☐|☐|☐|☐|☐
493 |I am confident in the system.|☐|☐|☐|☐|☐|☐|☐
494 |The system provides security.|☐|☐|☐|☐|☐|☐|☐
495 |The system has integrity (honesty or honor). |☐|☐|☐|☐|☐|☐|☐
496 |The system is dependable.|☐|☐|☐|☐|☐|☐|☐
497 |The system is reliable.|☐|☐|☐|☐|☐|☐|☐
498 |I can trust the system.|☐|☐|☐|☐|☐|☐|☐
499 |I am familiar with the system.|☐|☐|☐|☐|☐|☐|☐
500 )))
501 |(% style="width:296px" %)(((
502 (CL02) Situation Awareness Robot Reconnaissance data understanding (OPTIONAL; is more important for the analyst)
503 )))|(% style="width:1053px" %)(((
504 After the robot’s initial run but before humans enter, list any //hazards, obstacles, or victims// that you (as operator) detected or suspect from the robot’s feed. //(Open list)//
505
506
507
508
509 )))
510 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
511 |(% style="width:296px" %)(CL02/ CL04) Operational notes|(% style="width:1053px" %)(((
512 Logging of manual control or override the robot's/ system's suggestions (with timestamp) as well as any alerts ignored.
513
514
515
516 )))
517 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
518 |(% style="width:296px" %)(((
519 (CL01) SafetyTeam Decision Appraisal:
520 )))|(% style="width:1053px" %)(((
521 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
522 |How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
523 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
524 )))
525 |(% style="width:296px" %)(((
526 (CL01) Safety Error Reflection
527 )))|(% style="width:1053px" %)(((
528 List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
529
530
531
532
533 )))
534 |(% style="width:296px" %)(((
535 (CL02) Situation Awareness (Post-Scenario SAGAT)
536 )))|(% style="width:1053px" %)(((
537 Describe what the robot observed/ found during the mission (e.g. hazards, obstacles) for navigation purposes. //(Sketch space and bullets)//
538
539
540
541
542 )))
543 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
544 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
545 Give a score between:
546
547 0 (very low) - 100 (very high)
548 )))
549 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
550 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
551 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
552 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
553 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
554 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
555 )))
556 |(% style="width:296px" %)(((
557 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
558 )))|(% style="width:1053px" %)(((
559 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
560 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
561 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
562 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
563 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
564 )))
565 |(% style="width:296px" %)(((
566 (CL05) Confidence in Robot-Related Decisions:
567 )))|(% style="width:1053px" %)(((
568 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
569 |How confident are you that you made the correct decisions in operating and intervening with the robot|☐|☐|☐|☐|☐|☐|☐
570 | | | | | | | |
571 )))
572
573
574
575 (% class="table-bordered" %)
576 |(% colspan="2" %)(((
577 == Template: Robot analyst ==
578 )))
579 |(% colspan="2" %)(((
580 **General: **
581
582 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
583 )))
584 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
585 |(% style="width:445px" %)(((
586 (CL02) Situation Awareness Robot Reconnaissance Assessment (SAGAT expected situation report)
587 )))|(% style="width:1071px" %)(((
588 Based on the robot’s pre-entry scan, list the hazards, victims, and notable features you have identified or expect inside. //(Bullets and/or sketch)//
589
590
591
592
593 )))
594 |(% style="width:296px" %)(((
595 (CL04) Trust baseline
596 )))|(% style="width:1053px" %)(((
597 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
598 |(((
599 The system’s data is trustworthy
600 )))|☐|☐|☐|☐|☐|☐|☐
601 |(((
602 can rely on the robot to detect hazards I might miss
603 )))| | | | | | |
604 |....?| | | | | | |
605 )))
606 |(% colspan="2" style="width:445px" %)**DURING ENTRY PHASE**
607 |(% style="width:445px" %)(((
608 CL02 Data Interpretation Check (real-time observation)
609 )))|(% style="width:1071px" %)Observation by evaluation leader
610 |(% colspan="2" style="width:445px" %)**POST EXIT PHASE (hot wash)**
611 |(% style="width:296px" %)(((
612 (CL01) SafetyTeam Decision Appraisal:
613 )))|(% style="width:1053px" %)(((
614 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
615 |How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
616 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
617 )))
618 |(% style="width:296px" %)(((
619 (CL01) Safety Error Reflection
620 )))|(% style="width:1053px" %)(((
621 List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
622
623
624
625
626 )))
627 |(% style="width:296px" %)(((
628 (CL02) Situation Awareness (Post-Scenario SAGAT)
629 )))|(% style="width:1053px" %)(((
630 Now that the mission is over, summarize the final floor layout, and list all hazards and victims found or confirmed. //(Sketch space and bullets)//
631
632
633
634
635 )))
636 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
637 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
638 Give a score between:
639
640 0 (very low) - 100 (very high)
641 )))
642 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
643 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
644 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
645 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
646 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
647 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
648 )))
649 |(% style="width:296px" %)(((
650 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000)
651 )))|(% style="width:1053px" %)(((
652 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
653 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
654 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
655 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
656 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
657 )))
658 |(% style="width:296px" %)(((
659 (CL05) Confidence in Robot-Related Decisions:
660 )))|(% style="width:1053px" %)(((
661 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
662 |How confident are you that you made the correct decisions in operating and intervening with the robot|☐|☐|☐|☐|☐|☐|☐
663 | | | | | | | |
664 )))
665
666
667
668 (% class="table-bordered" %)
669 |(% colspan="2" %)(((
670 == Template: Entry team member ==
671 )))
672 |(% colspan="2" %)(((
673 **General: **
674
675 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
676 )))
677 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
678 |(% style="width:296px" %)(((
679 (CL02) Situation Awareness (Pre-mission knowledge check):
680 )))|(% style="width:1053px" %)(((
681 Before entering, list the hazards or victims you expect to encounter, based on briefing info. //(Bullets)//
682
683
684
685 )))
686 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
687 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
688 |I am confident in the robot analyst's capabilities.|☐|☐|☐|☐|☐|☐|☐
689 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
690 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
691 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
692 )))
693 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
694 |(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)|(% style="width:1053px" %)(((
695 Any new hazads? //(Bullets)//
696
697
698
699
700 )))
701 |(% style="width:296px" %)**POST EXIT PHASE (hot wash)**|(% style="width:1053px" %)
702 |(% style="width:296px" %)(((
703 (CL01) Safety - Team Decision Appraisal:
704 )))|(% style="width:1053px" %)(((
705 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
706 |How would you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
707 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
708 )))
709 |(% style="width:296px" %)(((
710 (CL02) Situation Awareness Post Operation Recall
711 )))|(% style="width:1053px" %)(((
712 Describe the layout and list all hazards/victims that were found. //(Sketch and describe)//
713
714
715
716
717
718 )))
719 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
720 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
721 Give a score between:
722
723 0 (very low) - 100 (very high)
724 )))
725 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
726 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
727 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
728 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
729 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
730 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
731 )))
732 |(% style="width:296px" %)(((
733 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
734 )))|(% style="width:1053px" %)(((
735 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
736 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
737 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
738 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
739 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
740 )))
741 |(% style="width:296px" %)(((
742 (CL05) Decision Confidence
743 )))|(% style="width:1053px" %)(((
744 |(% style="width:858px" %)Statement|(% style="width:308px" %)(((
745 Give a score between:
746
747 0 (very low) - 100 (very high)
748 )))
749 |(% style="width:858px" %)How confident are you in the actions you personally took?|(% style="width:308px" %)[[image:1776690776402-253.png]]
750 )))
751
752 (% class="table-bordered" style="width:929px" %)
753 |(% colspan="2" style="width:892px" %)(((
754 == Template: Base of Operations (LEMA) ==
755 )))|(% style="width:30px" %) |
756 |(% colspan="4" %)(((
757 **General: **
758
759 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
760 )))
761 |(% colspan="4" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
762 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:697px" %)((
763 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
764 |I am confident in the robot analyst's capabilities.|☐|☐|☐|☐|☐|☐|☐
765 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
766 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
767 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
768 ))
769 |(% style="width:296px" %) |(% style="width:697px" %)
770
771