Skip to Content

Wiki source code of j. SFT1 evaluation templates

Version 24.5 by Rosa Van Tuijn on 2026/04/20 15:53
Show last authors
1 = General =
2
3 [[image:1776255113603-367.png]]
4
5
6 |(% colspan="3" %)[[image:1776259393121-951.png]]
7 |**Baseline phases**|**Who fills it**|**Main purpose**
8 |//Briefing//|-|-
9 |//Estimate building integrity//|-|-
10 |//Shoring//|Particpants|(make estimation about shoring time)
11 |//Before sending in FRs (DM)//|Participant|Ground truth/ baseline SA
12 |//Send in FRs, identify voids, DM//|Evaluation leader|Objective performance, safety, timing, decision making
13 |//FRs exit building//|Participant|Subjective SA, trust, workload, decision making
14 |//Hotwash//| |
15
16 |(% colspan="3" style="width:414px" %)[[image:1776259356197-904.png]]
17 |(% style="width:414px" %)**SYNERGISE phases**|(% style="width:245px" %)**Who fills it**|(% style="width:690px" %)**Main purpose**
18 |(% style="width:414px" %)//Briefing//|(% style="width:245px" %)-|(% style="width:690px" %)-
19 |(% style="width:414px" %)//Estimate building integrity//|(% style="width:245px" %)-|(% style="width:690px" %)-
20 |(% style="width:414px" %)//Send in robots, identify voids with robots//|(% style="width:245px" %)-|(% style="width:690px" %)-
21 |(% style="width:414px" %)//Decision Making (DM)//|(% style="width:245px" %)Participants (Squad/ team leader)|(% style="width:690px" %)Ground truth/ baseline SA
22 |(% style="width:414px" %)//Send in FRs ?//|(% style="width:245px" %)Evaluation leader|(% style="width:690px" %)Objective performance, safety, timing, decision making
23 |(% style="width:414px" %)//After test//|(% style="width:245px" %)Paricipant|(% style="width:690px" %)Subjective SA, trust, workload, decision making
24 |(% style="width:414px" %)//Hotwash//|(% style="width:245px" %) |(% style="width:690px" %)
25
26 (% class="wikigeneratedid" %)
27 List of ground truths needed for comparison:
28
29 * Health ranges
30 * hazards/ victims in buildings
31 * Safest path within buildings
32 * State of automation (tech question)
33
34
35
36 === Before humans enter a building/ hazard ===
37
38 |**Claim**|**Measurement**|**What is recorded**|**Who records**
39 |CL01 Safety|Ground-truth safe paths|predefined safe/ suboptimal / unsafe paths|//Evaluation leader//
40 |(((
41 CL01 Safety
42 )))|(((
43 Chosen entry path
44 )))|(((
45 Intended entry path selection
46 )))|//Squad/ Team leader (informed by Robot analyst)//
47 |(((
48 CL02 Situation Awareness
49 )))|(((
50 Reported hazards, victims, layout (pre)
51 )))|(((
52 Sketch / map / description of interior
53 )))|//Squad/ Team leader, Entry team, Robot analyst (separately)//
54 |(((
55 CL02 SA
56 )))|(((
57 Robot‑based expectations
58 )))|(((
59 Expected hazards / victims from robot data
60 )))|//Robot analyst//
61 |(((
62 CL03–05 Mission effectiveness
63 )))|(((
64 Baseline trust (optional)
65 )))|(((
66 Initial trust in system/ robot
67 )))|//Squad/ Team leader, Robot operator, Robot analyst//
68 |(((
69 CL06 Health
70 )))|(((
71 Acceptable health ranges
72 )))|(((
73 HR, temp thresholds
74 )))|//Evaluation leader//
75 |(((
76 CL07 Efficiency
77 )))|(((
78 Mission start timestamp
79 )))|(((
80 End briefing → start
81 )))|//Evaluation leader//
82
83 During humans/robots are inside building/ hazard
84
85 |**Claim**|**Measurement**|**What is recorded**|**Who records**
86 |(((
87 CL01 Safety
88 )))|(((
89 Near‑incidents
90 )))|(((
91 Unsafe situations / close calls
92 )))|//Evaluation leader//
93 |(((
94 CL01 Safety
95 )))|(((
96 Hazard avoidance
97 )))|(((
98 Avoided predefined hazards
99 )))|//Evaluation leader//
100 |(((
101 CL02 SA
102 )))|(((
103 Executed path
104 )))|(((
105 Actual path taken
106 )))|//Evaluation leader, Engage system/ location sensors (?)//
107 |(((
108 CL02 SA
109 )))|(((
110 Path deviations
111 )))|(((
112 Deviations + cause (something for after instead?)
113 )))|//Squad/ Team leader, Entry team//
114 |(((
115 CL02 SA
116 )))|(((
117 Interpretations of robot images
118 )))|(((
119 ...?
120 )))|//Robot analyst (and Base or Operations/ LEMA ?)//
121 |(((
122 CL02 SA (Optional)
123 )))|(((
124 Robot control actions
125 )))|(((
126 Overrides, manual interventions
127 )))|//Robot operator//
128 |(((
129 CL06 Health
130 )))|(((
131 Threshold breach
132 )))|(((
133 Breach moment (HR, temp, stress (?))
134 )))|//Evaluation leader//
135 |(((
136 CL06 Health
137 )))|(((
138 Detection & intervention
139 )))|(((
140 Detection with corresponding action
141 )))|//Evaluation leader//
142 |(((
143 CL07 Efficiency
144 )))|(((
145 Entry timestamp
146 )))|(((
147 Entry team enters OR robot enters
148 )))|//Evaluation leader//
149 |(((
150 CL07 Efficiency
151 )))|(((
152 Exit timestamp
153 )))|Entry team exits OR robot exits |//Evaluation leader//
154
155 === After humans/robots exited building/hazard ===
156
157 |**Claim**|**Measurement**|**What is recorded**|**Who records**
158 |(((
159
160
161 CL01 Safety
162 )))|(((
163
164
165 Decision quality (retro)
166 )))|(((
167
168
169 “How good were decisions?”
170 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
171 |(((
172 CL01 Safety
173 )))|(((
174 Error reflection
175 )))|(((
176 Reflected unsafe choices
177 )))|//Squad/ Team leader, Robot operator, Robot analyst//
178 |(((
179 CL02 SA
180 )))|(((
181 Reported hazards, victims, layout (post)
182 )))|(((
183 Sketch / map / description
184 )))|//Squad/ Team leader, Entry team, Robot analyst//
185 |(((
186 CL03 Effectiveness
187 )))|(((
188 Workload
189 )))|(((
190 Short NASA‑TLX
191 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
192 |(((
193 CL04 Effectiveness
194 )))|(((
195 Trust
196 )))|(((
197 Trust survey / interview
198 )))|//Squad/ Team leader, Entry team (?), Robot operator, Robot analyst, Baes of Operations/ LEMA//
199 |(((
200 CL05 Effectiveness
201 )))|(((
202 Decision confidence
203 )))|(((
204 Confidence in own judgments
205 )))|//Squad/ Team leader, Robot operator, Robot analyst//
206 |(((
207 CL06 Health
208 )))|(((
209 Health issue handling
210 )))|(((
211 Adequacy & timeliness
212 )))|//Evaluation leader (+ expert)//
213 |(((
214 CL07 Efficiency
215 )))|(((
216 Total mission duration
217 )))|(((
218 Start (end briefing) till end (?)
219 )))|//Evaluation leader//
220
221
222
223 = Templates per role =
224
225 (% class="table-bordered" %)
226 |(% colspan="2" %)(((
227 == Template: Evaluation leader (Field) ==
228 )))
229 |(% colspan="2" %)(((
230 **General: **
231
232 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
233 )))
234 |(% style="width:296px" %)(((
235 System & mission readiness
236
237 //(Tick list based on predefined acceptable ranges. Cross out if not applicable.)//
238 )))|(% style="width:1053px" %)(((
239 ☐ C3I operational, describe: .......................................................................................................................................................................................................................
240
241 ☐ Health & location sensors connected, describe: .....................................................................................................................................................................................
242
243 ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................
244
245 ☐ OWL operational, describe: .....................................................................................................................................................................................................................
246
247 ☐ ANYmal operational, describe: ................................................................................................................................................................................................................
248
249 ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................
250
251 ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
252 )))
253 |(% colspan="2" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
254 |(% style="width:296px" %)Hand out questionnaires|(% style="width:1053px" %)Each role their own question for this section
255 |(% style="width:296px" %) |(% style="width:1053px" %)
256 |(% style="width:296px" %)(((
257 **Threshold breached, alert and intervention record **
258 )))|(% style="width:1053px" %)(((
259 Health threshold breached (timestamp + action taken)
260
261 Alert detection by system (timestamp + action taken)
262
263 Intervention started (timestamp + action taken)
264 )))
265 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
266 |(% style="width:296px" %)(((
267 **Time within building/ hazard zone (humans)**
268 )))|(% style="width:1053px" %)(((
269 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
270 )))
271 |(% style="width:296px" %)(((
272 **Time within building/ hazard zone (robots)**
273 )))|(% style="width:1053px" %)(((
274 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
275 )))
276 |(% style="width:296px" %)(((
277 **Safety observations **
278 )))|(% style="width:1053px" %)(((
279 ☐ No unsafe situations observed
280 ☐ Unsafe / near‑incident observed, describe (amount and type) :..................................................................................................................................................................
281
282 .........................................................................................................................................................................................................................................................................
283 )))
284 |(% style="width:296px" %)(((
285 **Path deviations observed**
286
287 //(Check if the entry team takes the route that is discussed before entering)//
288 )))|(% style="width:1053px" %)(((
289 ☐ None ☐ Minor ☐ Major
290
291 Notes:
292 )))
293 |(% style="width:296px" %) |(% style="width:1053px" %)
294 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
295 |(% style="width:296px" %)Give every role their questionnaire|(% style="width:1053px" %)
296 |(% style="width:296px" %)Plenary questions: |(% style="width:1053px" %)...??
297
298 (% class="table-bordered" %)
299 |(% colspan="2" %)(((
300 == Template: Squad/ Team leader (Field) ==
301 )))
302 |(% colspan="2" %)(((
303 **General: **
304
305 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
306 )))
307 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
308 |(% style="width:296px" %)(((
309 (CL01) Safety Plan Confirmation
310 )))|(% style="width:1053px" %)(((
311 Mark chosen entry path on provided building map. //Sketch or describe entry path. (Sketch space)//
312
313
314
315
316
317 )))
318 |(% style="width:296px" %)(CL02) Situation Awareness (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
319 List expected hazards and victims. //(Bullets and/ or sketch)//
320
321
322
323
324 )))
325 |(% style="width:296px" %)(CL02) Situation Awareness interior (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
326 Sketch/ Describe expected interior layout. //(Sketch space)//
327
328
329
330
331
332 )))
333 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
334 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
335 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
336 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
337 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
338 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
339 )))
340 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
341 |(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)|(% style="width:1053px" %)(((
342 Where are you team members now and what hazards are near them? //(Sketch space)//
343
344
345
346
347
348 )))
349 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
350 |(% style="width:296px" %)(((
351 (CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):
352 )))|(% style="width:1053px" %)(((
353 |(% style="width:587px" %)Statement|(% style="width:138px" %)1 (LOW)|(% style="width:60px" %)2|(% style="width:57px" %)3|(% style="width:59px" %)4|(% style="width:58px" %)5|(% style="width:60px" %)6|(% style="width:147px" %)7 (HIGH)
354 |(% style="width:587px" %)(((
355 I had a clear understanding of the mission objectives, my role, and the intended plan before execution.
356 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
357 |(% style="width:587px" %)(((
358 During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.
359 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
360 |(% style="width:587px" %)(((
361 The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.
362 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
363 |(% style="width:587px" %)(((
364 Team coordination and communication were effective and supported successful task execution.
365 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
366 |(% style="width:587px" %)(((
367 The tools, systems, or support aids available during the mission effectively supported mission execution.
368 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
369 |(% style="width:587px" %)(((
370 Based on this exercise, the team is better prepared to perform similar missions in the future.
371 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
372
373 (% class="box warningmessage" %)
374 (((
375 OR (open questions better?)
376 )))
377
378 |(% style="width:587px" %)Open questions
379 |(% style="width:587px" %)(((
380 What was the plan/intent for the mission, and what did you personally aim to achieve?// (1–2 sentences)//
381
382
383
384 )))
385 |(% style="width:587px" %)(((
386 What actually happened? List the 2–3 most important events or turning points. //(bullets)//
387
388
389
390 )))
391 |(% style="width:587px" %)(((
392 What went well and should be sustained? Give one concrete example. //(example + why it worked)//
393
394
395
396 )))
397 |(% style="width:587px" %)(((
398 What did not go as intended? Give one concrete example and its impact. //(example + consequence)//
399
400
401
402 )))
403 |(% style="width:587px" %)(((
404 Why did it happen? What were the main contributing factors? //(e.g., information/SA, communication, coordination, timing, tools/technology, environment)//
405
406
407
408 )))
409 |(% style="width:587px" %)(((
410 What are the top 2 actionable improvements for next time (specific, doable)? //(Action + owner/role + when)//
411
412
413
414 )))
415 )))
416 |(% style="width:296px" %)(((
417 (CL01) Safety Error Self-Assessment
418 )))|(% style="width:1053px" %)(((
419 List any unsafe or mistaken decisions you recognize from this mission (if any). //(Provide brief descriptions)//
420
421
422
423 )))
424 |(% style="width:296px" %)(((
425 (CL02) Situation Awareness (Post-Scenario SAGAT)
426 )))|(% style="width:1053px" %)(((
427 From memory, sketch the interior layout and mark all hazards and victims encountered. //(Sketch space and bullets)//
428
429
430
431
432 )))
433 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
434 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
435 Give a score between:
436
437 0 (very low) - 100 (very high)
438 )))
439 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
440 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
441 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
442 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
443 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
444 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
445 )))
446 |(% style="width:296px" %)(((
447 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
448 )))|(% style="width:1053px" %)(((
449 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
450 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
451 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
452 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
453 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
454 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
455 )))
456 |(% style="width:296px" %)(((
457 (CL05) Decision Confidence:
458 )))|(% style="width:1053px" %)(((
459 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
460 Give a score between:
461
462 0 (very low) - 100 (very high)
463 )))
464 |(% style="width:861px" %)How confident are you in the decisions you personally made during this mission? |(% style="width:305px" %)[[image:1776690776402-253.png]]
465 |(% style="width:861px" %)(asked per key decision; e.g. path choice  ?? )|(% style="width:305px" %)[[image:1776690777988-282.png]]
466 )))
467
468 == ==
469
470 (% class="table-bordered" %)
471 |(% colspan="2" %)(((
472 == Template: Robot operator ==
473 )))
474 |(% colspan="2" %)(((
475 **General: **
476
477 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
478 )))
479 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
480 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). |(% style="width:1053px" %)(((
481 Before entering, list the hazards or victims you expect to encounter, based on briefing info. //(Bullets)//
482
483 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
484 |The system is deceptive (misleading or deceiving).|☐|☐|☐|☐|☐|☐|☐
485 |The system behaves in an underhanded (shady or dishonest) manner.|☐|☐|☐|☐|☐|☐|☐
486 |I am suspicious of the system's intent, action, or outputs.|☐|☐|☐|☐|☐|☐|☐
487 |I am wary (cautious or careful) of the system.|☐|☐|☐|☐|☐|☐|☐
488 |The system's actions will have a harmful or injurious outcome.|☐|☐|☐|☐|☐|☐|☐
489 |I am confident in the system.|☐|☐|☐|☐|☐|☐|☐
490 |The system provides security.|☐|☐|☐|☐|☐|☐|☐
491 |The system has integrity (honesty or honor). |☐|☐|☐|☐|☐|☐|☐
492 |The system is dependable.|☐|☐|☐|☐|☐|☐|☐
493 |The system is reliable.|☐|☐|☐|☐|☐|☐|☐
494 |I can trust the system.|☐|☐|☐|☐|☐|☐|☐
495 |I am familiar with the system.|☐|☐|☐|☐|☐|☐|☐
496 )))
497 |(% style="width:296px" %)(((
498 (CL02) Situation Awareness Robot Reconnaissance data understanding (OPTIONAL; is more important for the analyst)
499 )))|(% style="width:1053px" %)(((
500 After the robot’s initial run but before humans enter, list any //hazards, obstacles, or victims// that you (as operator) detected or suspect from the robot’s feed. //(Open list)//
501
502
503
504
505 )))
506 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
507 |(% style="width:296px" %)(CL02/ CL04) Operational notes|(% style="width:1053px" %)(((
508 Logging of manual control or override the robot's/ system's suggestions (with timestamp) as well as any alerts ignored.
509
510
511
512 )))
513 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
514 |(% style="width:296px" %)(((
515 (CL01) SafetyTeam Decision Appraisal:
516 )))|(% style="width:1053px" %)(((
517 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
518 |How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
519 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
520 )))
521 |(% style="width:296px" %)(((
522 (CL01) Safety Error Reflection
523 )))|(% style="width:1053px" %)(((
524 List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
525
526
527
528
529 )))
530 |(% style="width:296px" %)(((
531 (CL02) Situation Awareness (Post-Scenario SAGAT)
532 )))|(% style="width:1053px" %)(((
533 Describe what the robot observed/ found during the mission (e.g. hazards, obstacles) for navigation purposes. //(Sketch space and bullets)//
534
535
536
537
538 )))
539 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
540 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
541 Give a score between:
542
543 0 (very low) - 100 (very high)
544 )))
545 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
546 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
547 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
548 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
549 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
550 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
551 )))
552 |(% style="width:296px" %)(((
553 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
554 )))|(% style="width:1053px" %)(((
555 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
556 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
557 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
558 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
559 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
560 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
561 )))
562 |(% style="width:296px" %)(((
563 (CL05) Confidence in Robot-Related Decisions:
564 )))|(% style="width:1053px" %)(((
565 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
566 |How confident are you that you made the correct decisions in operating and intervening with the robot|☐|☐|☐|☐|☐|☐|☐
567 | | | | | | | |
568 )))
569
570
571
572 (% class="table-bordered" %)
573 |(% colspan="2" %)(((
574 == Template: Robot analyst ==
575 )))
576 |(% colspan="2" %)(((
577 **General: **
578
579 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
580 )))
581 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
582 |(% style="width:445px" %)(((
583 (CL02) Situation Awareness Robot Reconnaissance Assessment (SAGAT expected situation report)
584 )))|(% style="width:1071px" %)(((
585 Based on the robot’s pre-entry scan, list the hazards, victims, and notable features you have identified or expect inside. //(Bullets and/or sketch)//
586
587
588
589
590 )))
591 |(% style="width:296px" %)(((
592 (CL04) Trust baseline
593 )))|(% style="width:1053px" %)(((
594 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
595 |(((
596 The system’s data is trustworthy
597 )))|☐|☐|☐|☐|☐|☐|☐
598 |(((
599 can rely on the robot to detect hazards I might miss
600 )))| | | | | | |
601 |....?| | | | | | |
602 )))
603 |(% colspan="2" style="width:445px" %)**DURING ENTRY PHASE**
604 |(% style="width:445px" %)(((
605 CL02 Data Interpretation Check (real-time observation)
606 )))|(% style="width:1071px" %)Observation by evaluation leader
607 |(% colspan="2" style="width:445px" %)**POST EXIT PHASE (hot wash)**
608 |(% style="width:296px" %)(((
609 (CL01) SafetyTeam Decision Appraisal:
610 )))|(% style="width:1053px" %)(((
611 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
612 |How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
613 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
614 )))
615 |(% style="width:296px" %)(((
616 (CL01) Safety Error Reflection
617 )))|(% style="width:1053px" %)(((
618 List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
619
620
621
622
623 )))
624 |(% style="width:296px" %)(((
625 (CL02) Situation Awareness (Post-Scenario SAGAT)
626 )))|(% style="width:1053px" %)(((
627 Now that the mission is over, summarize the final floor layout, and list all hazards and victims found or confirmed. //(Sketch space and bullets)//
628
629
630
631
632 )))
633 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
634 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
635 Give a score between:
636
637 0 (very low) - 100 (very high)
638 )))
639 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
640 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
641 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
642 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
643 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
644 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
645 )))
646 |(% style="width:296px" %)(((
647 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000)
648 )))|(% style="width:1053px" %)(((
649 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
650 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
651 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
652 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
653 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
654 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
655 )))
656 |(% style="width:296px" %)(((
657 (CL05) Confidence in Robot-Related Decisions:
658 )))|(% style="width:1053px" %)(((
659 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
660 |How confident are you that you made the correct decisions in operating and intervening with the robot|☐|☐|☐|☐|☐|☐|☐
661 | | | | | | | |
662 )))
663
664
665
666 (% class="table-bordered" %)
667 |(% colspan="2" %)(((
668 == Template: Entry team member ==
669 )))
670 |(% colspan="2" %)(((
671 **General: **
672
673 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
674 )))
675 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
676 |(% style="width:296px" %)(((
677 (CL02) Situation Awareness (Pre-mission knowledge check):
678 )))|(% style="width:1053px" %)(((
679 Before entering, list the hazards or victims you expect to encounter, based on briefing info. //(Bullets)//
680
681
682
683 )))
684 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
685 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
686 |I am confident in the robot analyst's capabilities.|☐|☐|☐|☐|☐|☐|☐
687 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
688 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
689 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
690 )))
691 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
692 |(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)|(% style="width:1053px" %)(((
693 Any new hazads? //(Bullets)//
694
695
696
697
698 )))
699 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
700 |(% style="width:296px" %)(((
701 (CL01) Safety - Team Decision Appraisal:
702 )))|(% style="width:1053px" %)(((
703 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
704 |How would you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?|☐|☐|☐|☐|☐|☐|☐
705 |Did your team correctly decide when to send the robot versus humans?|☐|☐|☐|☐|☐|☐|☐
706 )))
707 |(% style="width:296px" %)(((
708 (CL02) Situation Awareness Post Operation Recall
709 )))|(% style="width:1053px" %)(((
710 Describe the layout and list all hazards/victims that were found. //(Sketch and describe)//
711
712
713
714
715
716 )))
717 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
718 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
719 Give a score between:
720
721 0 (very low) - 100 (very high)
722 )))
723 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
724 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
725 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
726 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
727 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
728 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
729 )))
730 |(% style="width:296px" %)(((
731 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
732 )))|(% style="width:1053px" %)(((
733 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
734 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
735 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
736 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
737 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
738 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
739 )))
740 |(% style="width:296px" %)(((
741 (CL05) Decision Confidence
742 )))|(% style="width:1053px" %)(((
743 |(% style="width:858px" %)Statement|(% style="width:308px" %)(((
744 Give a score between:
745
746 0 (very low) - 100 (very high)
747 )))
748 |(% style="width:858px" %)How confident are you in the actions you personally took?|(% style="width:308px" %)[[image:1776690776402-253.png]]
749 )))
750
751
752
753 (% class="table-bordered" %)
754 |(% colspan="2" %)(((
755 == Template: Base of Operations (LEMA) ==
756 )))
757 |(% colspan="2" %)(((
758 **General: **
759
760 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
761 )))
762 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE (after briefing; before actual start of scenario)**
763 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO|(% style="width:1053px" %)(((
764 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
765 |I am confident in the robot analyst's capabilities.|☐|☐|☐|☐|☐|☐|☐
766 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
767 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
768 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
769 )))
770 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
771 |(% style="width:296px" %)CL07 Efficiency monitor (if possible)|(% style="width:1053px" %)(((
772 Log mission timeline (e.g. entry started 10.45, all-clear at 11.30), usually logged by evaluation leader but the base might independently be tracking time for their operational picture.
773
774 //Timestamp//: .......................................  //Action/ mission step//: .......................................
775
776 //Timestamp//: .......................................  //Action/ mission step//: .......................................
777
778 //Timestamp//: .......................................  //Action/ mission step//: .......................................
779
780 //Timestamp//: .......................................  //Action/ mission step//: .......................................
781 )))
782 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE (hot wash)**
783 |(% style="width:296px" %)(CL01, CL03-05) Safety and Effectiveness|(% style="width:1053px" %)(((
784 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
785 |From the command (strategic) perspective, rate the overall mission outcome on safety and effectiveness.|☐|☐|☐|☐|☐|☐|☐
786 )))
787 |(% style="width:296px" %)(((
788 (CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
789 )))|(% style="width:1053px" %)(((
790 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
791 |I would trust this robot system in future missions.|☐|☐|☐|☐|☐|☐|☐
792 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
793 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
794 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
795 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
796 )))
797 |(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)|(% style="width:1053px" %)(((
798 |(% style="width:861px" %)Statement|(% style="width:305px" %)(((
799 Give a score between:
800
801 0 (very low) - 100 (very high)
802 )))
803 |(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?|(% style="width:305px" %)[[image:1776690776402-253.png]]
804 |(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?|(% style="width:305px" %)[[image:1776690777988-282.png]]
805 |(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic|(% style="width:305px" %)[[image:1776690778813-812.png]]
806 |(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?|(% style="width:305px" %)[[image:1776690779609-960.png]]
807 |(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?|(% style="width:305px" %)[[image:1776690780664-359.png]]
808 |(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?|(% style="width:305px" %)[[image:1776690781527-833.png]]
809 )))
810 |(% style="width:296px" %)(((
811 (CL05) Decision Confidence
812 )))|(% style="width:1053px" %)(((
813 |(% style="width:858px" %)Statement|(% style="width:308px" %)(((
814 Give a score between:
815
816 0 (very low) - 100 (very high)
817 )))
818 |(% style="width:858px" %)How confident are you in the key decisions made at the command level (strategic)?|(% style="width:308px" %)[[image:1776690776402-253.png]]
819 )))
820
821
822
823