Skip to Content

Wiki source code of j. SFT1 evaluation templates

Version 15.1 by Rosa Van Tuijn on 2026/04/15 14:21
Show last authors
1 = General =
2
3 [[image:1776255113603-367.png]]
4
5
6 |(% colspan="3" %)[[image:1776259393121-951.png]]
7 |**Baseline phases**|**Who fills it**|**Main purpose**
8 |//Briefing//|-|-
9 |//Estimate building integrity//|-|-
10 |//Shoring//|Particpants|(make estimation about shoring time)
11 |//Before sending in FRs (DM)//|Participant|Ground truth/ baseline SA
12 |//Send in FRs, identify voids, DM//|Evaluation leader|Objective performance, safety, timing, decision making
13 |//FRs exit building//|Participant|Subjective SA, trust, workload, decision making
14 |//Hotwash//| |
15
16 |(% colspan="3" style="width:414px" %)[[image:1776259356197-904.png]]
17 |(% style="width:414px" %)**SYNERGISE phases**|(% style="width:245px" %)**Who fills it**|(% style="width:690px" %)**Main purpose**
18 |(% style="width:414px" %)//Briefing//|(% style="width:245px" %)-|(% style="width:690px" %)-
19 |(% style="width:414px" %)//Estimate building integrity//|(% style="width:245px" %)-|(% style="width:690px" %)-
20 |(% style="width:414px" %)//Send in robots, identify voids with robots//|(% style="width:245px" %)-|(% style="width:690px" %)-
21 |(% style="width:414px" %)//Decision Making (DM)//|(% style="width:245px" %)Participants (Squad/ team leader)|(% style="width:690px" %)Ground truth/ baseline SA
22 |(% style="width:414px" %)//Send in FRs ?//|(% style="width:245px" %)Evaluation leader|(% style="width:690px" %)Objective performance, safety, timing, decision making
23 |(% style="width:414px" %)//After test//|(% style="width:245px" %)Paricipant|(% style="width:690px" %)Subjective SA, trust, workload, decision making
24 |(% style="width:414px" %)//Hotwash//|(% style="width:245px" %) |(% style="width:690px" %)
25
26 (% class="wikigeneratedid" %)
27 List of ground truths needed for comparison:
28
29 * Health ranges
30 * hazards/ victims in buildings
31 * Safest path within buildings
32 * State of automation (tech question)
33
34
35
36 === Before humans enter a building/ hazard ===
37
38 |**Claim**|**Measurement**|**What is recorded**|**Who records**
39 |CL01 Safety|Ground-truth safe paths|predefined safe/ suboptimal / unsafe paths|//Evaluation leader//
40 |(((
41 CL01 Safety
42 )))|(((
43 Chosen entry path
44 )))|(((
45 Intended entry path selection
46 )))|//Squad/ Team leader (informed by Robot analyst)//
47 |(((
48 CL02 Situation Awareness
49 )))|(((
50 Reported hazards, victims, layout (pre)
51 )))|(((
52 Sketch / map / description of interior
53 )))|//Squad/ Team leader, Entry team, Robot analyst (separately)//
54 |(((
55 CL02 SA
56 )))|(((
57 Robot‑based expectations
58 )))|(((
59 Expected hazards / victims from robot data
60 )))|//Robot analyst//
61 |(((
62 CL03–05 Mission effectiveness
63 )))|(((
64 Baseline trust (optional)
65 )))|(((
66 Initial trust in system/ robot
67 )))|//Squad/ Team leader, Robot operator, Robot analyst//
68 |(((
69 CL06 Health
70 )))|(((
71 Acceptable health ranges
72 )))|(((
73 HR, temp thresholds
74 )))|//Evaluation leader//
75 |(((
76 CL07 Efficiency
77 )))|(((
78 Mission start timestamp
79 )))|(((
80 End briefing → start
81 )))|//Evaluation leader//
82
83 During humans/robots are inside building/ hazard
84
85 |**Claim**|**Measurement**|**What is recorded**|**Who records**
86 |(((
87 CL01 Safety
88 )))|(((
89 Near‑incidents
90 )))|(((
91 Unsafe situations / close calls
92 )))|//Evaluation leader//
93 |(((
94 CL01 Safety
95 )))|(((
96 Hazard avoidance
97 )))|(((
98 Avoided predefined hazards
99 )))|//Evaluation leader//
100 |(((
101 CL02 SA
102 )))|(((
103 Executed path
104 )))|(((
105 Actual path taken
106 )))|//Evaluation leader, Engage system/ location sensors (?)//
107 |(((
108 CL02 SA
109 )))|(((
110 Path deviations
111 )))|(((
112 Deviations + cause (something for after instead?)
113 )))|//Squad/ Team leader, Entry team//
114 |(((
115 CL02 SA
116 )))|(((
117 Interpretations of robot images
118 )))|(((
119 ...?
120 )))|//Robot analyst (and Base or Operations/ LEMA ?)//
121 |(((
122 CL02 SA (Optional)
123 )))|(((
124 Robot control actions
125 )))|(((
126 Overrides, manual interventions
127 )))|//Robot operator//
128 |(((
129 CL06 Health
130 )))|(((
131 Threshold breach
132 )))|(((
133 Breach moment (HR, temp, stress (?))
134 )))|//Evaluation leader//
135 |(((
136 CL06 Health
137 )))|(((
138 Detection & intervention
139 )))|(((
140 Detection with corresponding action
141 )))|//Evaluation leader//
142 |(((
143 CL07 Efficiency
144 )))|(((
145 Entry timestamp
146 )))|(((
147 Entry team enters OR robot enters
148 )))|//Evaluation leader//
149 |(((
150 CL07 Efficiency
151 )))|(((
152 Exit timestamp
153 )))|Entry team exits OR robot exits |//Evaluation leader//
154
155 === After humans/robots exited building/hazard ===
156
157 |**Claim**|**Measurement**|**What is recorded**|**Who records**
158 |(((
159
160
161 CL01 Safety
162 )))|(((
163
164
165 Decision quality (retro)
166 )))|(((
167
168
169 “How good were decisions?”
170 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
171 |(((
172 CL01 Safety
173 )))|(((
174 Error reflection
175 )))|(((
176 Reflected unsafe choices
177 )))|//Squad/ Team leader, Robot operator, Robot analyst//
178 |(((
179 CL02 SA
180 )))|(((
181 Reported hazards, victims, layout (post)
182 )))|(((
183 Sketch / map / description
184 )))|//Squad/ Team leader, Entry team, Robot analyst//
185 |(((
186 CL03 Effectiveness
187 )))|(((
188 Workload
189 )))|(((
190 Short NASA‑TLX
191 )))|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
192 |(((
193 CL04 Effectiveness
194 )))|(((
195 Trust
196 )))|(((
197 Trust survey / interview
198 )))|//Squad/ Team leader, Entry team (?), Robot operator, Robot analyst, Baes of Operations/ LEMA//
199 |(((
200 CL05 Effectiveness
201 )))|(((
202 Decision confidence
203 )))|(((
204 Confidence in own judgments
205 )))|//Squad/ Team leader, Robot operator, Robot analyst//
206 |(((
207 CL06 Health
208 )))|(((
209 Health issue handling
210 )))|(((
211 Adequacy & timeliness
212 )))|//Evaluation leader (+ expert)//
213 |(((
214 CL07 Efficiency
215 )))|(((
216 Total mission duration
217 )))|(((
218 Start (end briefing) till end (?)
219 )))|//Evaluation leader//
220
221
222
223 = Templates per role =
224
225
226 (% class="table-bordered" %)
227 |(% colspan="2" %)(((
228 == Template: Base of Operations LEMA (BoO/ HQ) ==
229 )))
230 |(% colspan="2" %)(((
231 **General: **
232
233
234 //Participants roles (encircle)//: BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team  |  //Trial type/ Building/ scenario//: ............................................  |  //Date/ Time//: ..........................................
235 )))
236 |(% style="width:296px" %)(((
237
238 )))|(% style="width:1053px" %)(((
239
240 )))
241
242 (% class="table-bordered" %)
243 |(% colspan="2" %)(((
244 == Template: Evaluation leader (Field) ==
245 )))
246 |(% colspan="2" %)//**Before start scenario**//
247 |(% colspan="2" %)(((
248 **General: **
249
250
251 //Participants roles (encircle)//: BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team  |  //Trial type/ Building/ scenario//: ............................................  |  //Date/ Time//: ..........................................
252 )))
253 |(% style="width:296px" %)(((
254 **System & mission readiness**
255
256 //(Tick list based on predefined acceptable ranges. Cross out if not applicable.)//
257 )))|(% style="width:1053px" %)(((
258 ☐ C3I operational, describe: .......................................................................................................................................................................................................................
259
260 ☐ Health & location sensors connected, describe: .....................................................................................................................................................................................
261
262 ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................
263
264 ☐ OWL operational, describe: .....................................................................................................................................................................................................................
265
266 ☐ ANYmal operational, describe: ................................................................................................................................................................................................................
267
268 ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................
269
270 ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
271 )))
272 |(% colspan="2" %)//**During start scenario**//
273 |(% style="width:296px" %)**Let Team leader (and entry team?) sketch/ describe hazard, victims, layout**|(% style="width:1053px" %)(((
274 Notes:
275
276
277
278 )))
279 |(% style="width:296px" %)(((
280 **Threshold breached, alert and intervention record **
281 )))|(% style="width:1053px" %)(((
282 Health threshold breached (timestamp + action taken)
283
284 Alert detection by system (timestamp + action taken)
285
286 Intervention started (timestamp + action taken)
287 )))
288 |(% colspan="2" style="width:296px" %)//**When humans enter a building/ hazard zone**//
289 |(% style="width:296px" %)(((
290 **Time within building/ hazard zone**
291 )))|(% style="width:1053px" %)(((
292 Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................
293 )))
294 |(% style="width:296px" %)(((
295 **Safety observations **
296 )))|(% style="width:1053px" %)(((
297 ☐ No unsafe situations observed
298 ☐ Unsafe / near‑incident observed, describe (amount and type) :..................................................................................................................................................................
299
300 .........................................................................................................................................................................................................................................................................
301 )))
302 |(% style="width:296px" %)(((
303 **Path deviations observed**
304
305 //(Check if the entry team takes the route that is discussed before entering)//
306 )))|(% style="width:1053px" %)(((
307 ☐ None ☐ Minor ☐ Major
308
309 Notes:
310 )))
311 |(% style="width:296px" %) |(% style="width:1053px" %)
312 |(% colspan="2" style="width:296px" %)//**When humans have exited the building/ hazard zone**//
313 |(% style="width:296px" %)Better decision overall|(% style="width:1053px" %)Notes:
314
315 (% class="table-bordered" %)
316 |(% colspan="2" %)(((
317 == Template: Squad/ Team leader (Field) ==
318 )))
319 |(% colspan="2" %)(((
320 **General: **
321
322
323 //Participants roles (encircle)//: BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team  |  //Trial type/ Building/ scenario//: ............................................  |  //Date/ Time//: ..........................................
324 )))
325 |(% style="width:296px" %)(((
326 **Pre-entry situation awareness sketch (participant)**
327
328 Sketch or describe //hazards, victims, layout// as you currently understand them. Predict victims, triage.
329 )))|(% style="width:1053px" %)(((
330 //(Sketch space)//
331
332
333
334
335
336 )))
337
338 == ==
339
340 == Template: Robot operator (Field) ==
341
342
343
344 == Template: Robot analyst (Field) ==
345
346
347
348 == Template: Entry team (Field) ==
349
350
351