Skip to Content

Wiki source code of j. SFT1 evaluation templates

Version 17.1 by Rosa Van Tuijn on 2026/04/20 12:55
Show last authors
1 = General =
2
3 [[image:1776255113603-367.png]]
4
5
6 |(% colspan="3" %)[[image:1776259393121-951.png]]
7 |**Baseline phases**|**Who fills it**|**Main purpose**
8 |//Briefing//|-|-
9 |//Estimate building integrity//|-|-
10 |//Shoring//|Particpants|(make estimation about shoring time)
11 |//Before sending in FRs (DM)//|Participant|Ground truth/ baseline SA
12 |//Send in FRs, identify voids, DM//|Evaluation leader|Objective performance, safety, timing, decision making
13 |//FRs exit building//|Participant|Subjective SA, trust, workload, decision making
14 |//Hotwash//| |
15
16 |(% colspan="3" style="width:414px" %)[[image:1776259356197-904.png]]
17 |(% style="width:414px" %)**SYNERGISE phases**|(% style="width:245px" %)**Who fills it**|(% style="width:690px" %)**Main purpose**
18 |(% style="width:414px" %)//Briefing//|(% style="width:245px" %)-|(% style="width:690px" %)-
19 |(% style="width:414px" %)//Estimate building integrity//|(% style="width:245px" %)-|(% style="width:690px" %)-
20 |(% style="width:414px" %)//Send in robots, identify voids with robots//|(% style="width:245px" %)-|(% style="width:690px" %)-
21 |(% style="width:414px" %)//Decision Making (DM)//|(% style="width:245px" %)Participants (Squad/ team leader)|(% style="width:690px" %)Ground truth/ baseline SA
22 |(% style="width:414px" %)//Send in FRs ?//|(% style="width:245px" %)Evaluation leader|(% style="width:690px" %)Objective performance, safety, timing, decision making
23 |(% style="width:414px" %)//After test//|(% style="width:245px" %)Paricipant|(% style="width:690px" %)Subjective SA, trust, workload, decision making
24 |(% style="width:414px" %)//Hotwash//|(% style="width:245px" %) |(% style="width:690px" %)
25
26 (% class="wikigeneratedid" %)
27 List of ground truths needed for comparison:
28
29 * Health ranges
30 * hazards/ victims in buildings
31 * Safest path within buildings
32 * State of automation (tech question)
33
34
35
36 === Before humans enter a building/ hazard ===
37
38 |**Claim**|**Measurement**|**What is recorded**|**Who records**
39 |CL01 Safety|Ground-truth safe paths|predefined safe/ suboptimal / unsafe paths|//Evaluation leader//
40 |(((
41 CL01 Safety
42 )))|(((
43 Chosen entry path
44 )))|(((
45 Intended entry path selection
46 )))|//Squad/ Team leader (informed by Robot analyst)//
47 |(((
48 CL02 Situation Awareness
49 )))|(((
50 Reported hazards, victims, layout (pre)
51 )))|(((
52 Sketch / map / description of interior
53 )))|//Squad/ Team leader, Entry team, Robot analyst (separately)//
54 |(((
55 CL02 SA
56 )))|(((
57 Robot‑based expectations
58 )))|(((
59 Expected hazards / victims from robot data
60 )))|//Robot analyst//
61 |(((
62 CL03–05 Mission effectiveness
63 )))|(((
64 Baseline trust (optional)
65 )))|(((
66 Initial trust in system/ robot
67 )))|//Squad/ Team leader, Robot operator, Robot analyst//
68 |(((
69 CL06 Health
70 )))|(((
71 Acceptable health ranges
72 )))|(((
73 HR, temp thresholds
74 )))|//Evaluation leader//
75 |(((
76 CL07 Efficiency
77 )))|(((
78 Mission start timestamp
79 )))|(((
80 End briefing → start
81 )))|//Evaluation leader//
82
83 During humans/robots are inside building/ hazard
84
85 |**Claim**|**Measurement**|**What is recorded**|**Who records**
86 |(((
87 CL01 Safety
88 )))|(((
89 Near‑incidents
90 )))|(((
91 Unsafe situations / close calls
92 )))|//Evaluation leader//
93 |(((
94 CL01 Safety
95 )))|(((
96 Hazard avoidance
97 )))|(((
98 Avoided predefined hazards
99 )))|//Evaluation leader//
100 |(((
101 CL02 SA
102 )))|(((
103 Executed path
104 )))|(((
105 Actual path taken
106 )))|//Evaluation leader, Engage system/ location sensors (?)//
107 |(((
108 CL02 SA
109 )))|(((
110 Path deviations
111 )))|(((
112 Deviations + cause (something for after instead?)
113 )))|//Squad/ Team leader, Entry team//
114 |(((
115 CL02 SA
116 )))|(((
117 Interpretations of robot images
118 )))|(((
119 ...?
120 )))|//Robot analyst (and Base or Operations/ LEMA ?)//
121 |(((
122 CL02 SA (Optional)
123 )))|(((
124 Robot control actions
125 )))|(((
126 Overrides, manual interventions
127 )))|//Robot operator//
128 |(((
129 CL06 Health
130 )))|(((
131 Threshold breach
132 )))|(((
133 Breach moment (HR, temp, stress (?))
134 )))|//Evaluation leader//
135 |(((
136 CL06 Health
137 )))|(((
138 Detection & intervention
139 )))|(((
140 Detection with corresponding action
141 )))|//Evaluation leader//
142 |(((
143 CL07 Efficiency
144 )))|(((
145 Entry timestamp
146 )))|(((
147 Entry team enters OR robot enters
148 )))|//Evaluation leader//
149 |(((
150 CL07 Efficiency
151 )))|(((
152 Exit timestamp
153 )))|Entry team exits OR robot exits |//Evaluation leader//
154
155 === After humans/robots exited building/hazard ===
156
157 |**Claim**|**Measurement**|**What is recorded**|**Who records**
158 |(((
159
160
161 CL01 Safety
162 )))|(((
163
164
165 Decision quality (retro)
166 )))|(((
167
168
169 “How good were decisions?”
170 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
171 |(((
172 CL01 Safety
173 )))|(((
174 Error reflection
175 )))|(((
176 Reflected unsafe choices
177 )))|//Squad/ Team leader, Robot operator, Robot analyst//
178 |(((
179 CL02 SA
180 )))|(((
181 Reported hazards, victims, layout (post)
182 )))|(((
183 Sketch / map / description
184 )))|//Squad/ Team leader, Entry team, Robot analyst//
185 |(((
186 CL03 Effectiveness
187 )))|(((
188 Workload
189 )))|(((
190 Short NASA‑TLX
191 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
192 |(((
193 CL04 Effectiveness
194 )))|(((
195 Trust
196 )))|(((
197 Trust survey / interview
198 )))|//Squad/ Team leader, Entry team (?), Robot operator, Robot analyst, Baes of Operations/ LEMA//
199 |(((
200 CL05 Effectiveness
201 )))|(((
202 Decision confidence
203 )))|(((
204 Confidence in own judgments
205 )))|//Squad/ Team leader, Robot operator, Robot analyst//
206 |(((
207 CL06 Health
208 )))|(((
209 Health issue handling
210 )))|(((
211 Adequacy & timeliness
212 )))|//Evaluation leader (+ expert)//
213 |(((
214 CL07 Efficiency
215 )))|(((
216 Total mission duration
217 )))|(((
218 Start (end briefing) till end (?)
219 )))|//Evaluation leader//
220
221
222
223 = Templates per role =
224
225
226 (% class="table-bordered" %)
227 |(% colspan="2" %)(((
228 == Template: Base of Operations LEMA (BoO/ HQ) ==
229 )))
230 |(% colspan="2" %)(((
231 **General: **
232
233
234 //Participants roles (encircle)//: BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team  |  //Trial type/ Building/ scenario//: ............................................  |  //Date/ Time//: ..........................................
235 )))
236 |(% style="width:296px" %)(((
237
238 )))|(% style="width:1053px" %)(((
239
240 )))
241
242 (% class="table-bordered" %)
243 |(% colspan="2" %)(((
244 == Template: Evaluation leader (Field) ==
245 )))
246 |(% colspan="2" %)//**Before start scenario**//
247 |(% colspan="2" %)(((
248 **General: **
249
250
251 //Participants roles (encircle)//: BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team  |  //Trial type/ Building/ scenario//: ............................................  |  //Date/ Time//: ..........................................
252 )))
253 |(% style="width:296px" %)(((
254 **System & mission readiness**
255
256 //(Tick list based on predefined acceptable ranges. Cross out if not applicable.)//
257 )))|(% style="width:1053px" %)(((
258 ☐ C3I operational, describe: .......................................................................................................................................................................................................................
259
260 ☐ Health & location sensors connected, describe: .....................................................................................................................................................................................
261
262 ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................
263
264 ☐ OWL operational, describe: .....................................................................................................................................................................................................................
265
266 ☐ ANYmal operational, describe: ................................................................................................................................................................................................................
267
268 ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................
269
270 ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
271 )))
272 |(% colspan="2" %)//**During start scenario**//
273 |(% style="width:296px" %)**Let Team leader (and entry team?) sketch/ describe hazard, victims, layout**|(% style="width:1053px" %)(((
274 Notes:
275
276
277
278 )))
279 |(% style="width:296px" %)(((
280 **Threshold breached, alert and intervention record **
281 )))|(% style="width:1053px" %)(((
282 Health threshold breached (timestamp + action taken)
283
284 Alert detection by system (timestamp + action taken)
285
286 Intervention started (timestamp + action taken)
287 )))
288 |(% colspan="2" style="width:296px" %)//**When humans enter a building/ hazard zone**//
289 |(% style="width:296px" %)(((
290 **Time within building/ hazard zone**
291 )))|(% style="width:1053px" %)(((
292 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
293 )))
294 |(% style="width:296px" %)(((
295 **Safety observations **
296 )))|(% style="width:1053px" %)(((
297 ☐ No unsafe situations observed
298 ☐ Unsafe / near‑incident observed, describe (amount and type) :..................................................................................................................................................................
299
300 .........................................................................................................................................................................................................................................................................
301 )))
302 |(% style="width:296px" %)(((
303 **Path deviations observed**
304
305 //(Check if the entry team takes the route that is discussed before entering)//
306 )))|(% style="width:1053px" %)(((
307 ☐ None ☐ Minor ☐ Major
308
309 Notes:
310 )))
311 |(% style="width:296px" %) |(% style="width:1053px" %)
312 |(% colspan="2" style="width:296px" %)//**When humans have exited the building/ hazard zone**//
313 |(% style="width:296px" %)Better decision overall|(% style="width:1053px" %)Notes:
314
315 (% class="table-bordered" %)
316 |(% colspan="2" %)(((
317 == Template: Squad/ Team leader (Field) - WIHT SYNERGISE TECH. ==
318 )))
319 |(% colspan="2" %)(((
320 **General: **
321
322 //Trial type/ Building/ scenario ID//: .................................  |  //Date/ Time//: .......................................  |  //Condition (encircle)//: Baseline / SYNERGISE-tech supported
323 )))
324 |(% colspan="2" style="width:296px" %)**PRE-ENTRY PHASE**
325 |(% style="width:296px" %)(((
326 (CL01) Mark chosen entry path on provided building map . //Sketch or describe entry path.//
327 )))|(% style="width:1053px" %)(((
328 //(Sketch space)//
329
330
331
332
333
334 )))
335 |(% style="width:296px" %)(CL02) List expected hazards (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
336
337
338
339 )))
340 |(% style="width:296px" %)(CL02) List expected victims (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
341
342
343
344 )))
345 |(% style="width:296px" %)(CL02) Sketch/ Describe expected interior layout (SAGAT; Endsley 1988):|(% style="width:1053px" %)(((
346 //(Sketch space)//
347
348
349
350
351
352 )))
353 |(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000)|(% style="width:1053px" %)(((
354 |Statement|1 (LOW)|2|3|4|5|6|7 (HIGH)
355 |I am confident in the robot/system’s abilities.|☐|☐|☐|☐|☐|☐|☐
356 |The robot/system will act in our best interest.|☐|☐|☐|☐|☐|☐|☐
357 |The robot/system is reliable.|☐|☐|☐|☐|☐|☐|☐
358 |I trust the system’s information.|☐|☐|☐|☐|☐|☐|☐
359 )))
360 |(% colspan="2" style="width:296px" %)**DURING ENTRY PHASE**
361 |(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)|(% style="width:1053px" %)(((
362 Where are you team members now and what hazards are near them? //(Sketch space)//
363
364
365
366
367
368 )))
369 |(% colspan="2" style="width:296px" %)**POST EXIT PHASE**
370 |(% style="width:296px" %)(((
371 (CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):
372 )))|(% style="width:1053px" %)(((
373 |(% style="width:587px" %)Statement|(% style="width:138px" %)1 (LOW)|(% style="width:60px" %)2|(% style="width:57px" %)3|(% style="width:59px" %)4|(% style="width:58px" %)5|(% style="width:60px" %)6|(% style="width:147px" %)7 (HIGH)
374 |(% style="width:587px" %)(((
375 had a clear understanding of the mission objectives, my role, and the intended plan before execution.
376 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
377 |(% style="width:587px" %)(((
378 During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.
379 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
380 |(% style="width:587px" %)(((
381 The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.
382 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
383 |(% style="width:587px" %)(((
384 Team coordination and communication were effective and supported successful task execution.
385 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
386 |(% style="width:587px" %)(((
387 The tools, systems, or support aids available during the mission effectively supported mission execution.
388 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
389 |(% style="width:587px" %)(((
390 Based on this exercise, the team is better prepared to perform similar missions in the future.
391 )))|(% style="width:138px" %)☐|(% style="width:60px" %)☐|(% style="width:57px" %)☐|(% style="width:59px" %)☐|(% style="width:58px" %)☐|(% style="width:60px" %)☐|(% style="width:147px" %)☐
392
393 OR (open questions better?)
394
395
396 |(% style="width:587px" %)Open questions
397 |(% style="width:587px" %)(((
398 What was the plan/intent for the mission, and what did you personally aim to achieve?// (1–2 sentences)//
399
400
401
402 )))
403 |(% style="width:587px" %)(((
404 What actually happened? List the 2–3 most important events or turning points. //(bullets)//
405
406
407
408 )))
409 |(% style="width:587px" %)(((
410 What went well and should be sustained? Give one concrete example. //(example + why it worked)//
411
412
413
414 )))
415 |(% style="width:587px" %)(((
416 What did not go as intended? Give one concrete example and its impact. //(example + consequence)//
417
418
419
420 )))
421 |(% style="width:587px" %)(((
422 Why did it happen? What were the main contributing factors? //(e.g., information/SA, communication, coordination, timing, tools/technology, environment)//
423
424
425
426 )))
427 |(% style="width:587px" %)(((
428 What are the top 2 actionable improvements for next time (specific, doable)? //(Action + owner/role + when)//
429
430
431
432 )))
433 )))
434 |(% style="width:296px" %)(((
435 (CL01) Safety Error Self-Assessment
436 )))|(% style="width:1053px" %)(((
437 List any unsafe or mistaken decisions you recognize from this mission (if any). //(Provide brief descriptions)//
438
439
440
441 )))
442 |(% style="width:296px" %) |(% style="width:1053px" %)
443
444 == ==
445
446 == Template: Robot operator (Field) ==
447
448
449
450 == Template: Robot analyst (Field) ==
451
452
453
454 == Template: Entry team (Field) ==