Wiki source code of j. SFT1 evaluation templates

Last modified by Rosa Van Tuijn on 2026/05/05 11:58

version	line-number	content
8.3	1	= General =
	2
26.1	3	[[image:1777962330814-818.png]]
9.2	4
	5
14.1	6	\|(% colspan="3" %)[[image:1776259393121-951.png]]
9.2	7	\|Baseline phases\|Who fills it\|Main purpose
	8	\|//Briefing//\|-\|-
	9	\|//Estimate building integrity//\|-\|-
	10	\|//Shoring//\|Particpants\|(make estimation about shoring time)
	11	\|//Before sending in FRs (DM)//\|Participant\|Ground truth/ baseline SA
	12	\|//Send in FRs, identify voids, DM//\|Evaluation leader\|Objective performance, safety, timing, decision making
	13	\|//FRs exit building//\|Participant\|Subjective SA, trust, workload, decision making
9.1	14	\|//Hotwash//\| \|
8.3	15
14.1	16	\|(% colspan="3" style="width:414px" %)[[image:1776259356197-904.png]]
9.2	17	\|(% style="width:414px" %)SYNERGISE phases\|(% style="width:245px" %)Who fills it\|(% style="width:690px" %)Main purpose
	18	\|(% style="width:414px" %)//Briefing//\|(% style="width:245px" %)-\|(% style="width:690px" %)-
	19	\|(% style="width:414px" %)//Estimate building integrity//\|(% style="width:245px" %)-\|(% style="width:690px" %)-
	20	\|(% style="width:414px" %)//Send in robots, identify voids with robots//\|(% style="width:245px" %)-\|(% style="width:690px" %)-
	21	\|(% style="width:414px" %)//Decision Making (DM)//\|(% style="width:245px" %)Participants (Squad/ team leader)\|(% style="width:690px" %)Ground truth/ baseline SA
	22	\|(% style="width:414px" %)//Send in FRs ?//\|(% style="width:245px" %)Evaluation leader\|(% style="width:690px" %)Objective performance, safety, timing, decision making
	23	\|(% style="width:414px" %)//After test//\|(% style="width:245px" %)Paricipant\|(% style="width:690px" %)Subjective SA, trust, workload, decision making
	24	\|(% style="width:414px" %)//Hotwash//\|(% style="width:245px" %) \|(% style="width:690px" %)
	25
25.3	26	== List of ground truths needed for comparison: ==
8.3	27
	28	* Health ranges
26.2	29	* hazards/ victims in buildings + Safest path within buildings : Safety observations + path observations >> have a premade map with everything and note down the route. Have an expert rate it after the fact.
	30	* Decision quality: have expert (or team leaders) rate the path of the FRs inside
9.1	31	* State of automation (tech question)
8.3	32
13.2	33	= Templates per role =
2.1	34
13.2	35	(% class="table-bordered" %)
	36	\|(% colspan="2" %)(((
9.3	37	== Template: Evaluation leader (Field) ==
	38	)))
12.1	39	\|(% colspan="2" %)(((
	40	General:
	41
19.9	42	//Trial type/ Building/ scenario ID//: ................................. \| //Date/ Time//: ....................................... \| //Condition (encircle)//: Baseline / SYNERGISE-tech supported
12.1	43	)))
13.1	44	\|(% style="width:296px" %)(((
24.5	45	System & mission readiness
12.1	46
	47	//(Tick list based on predefined acceptable ranges. Cross out if not applicable.)//
13.1	48	)))\|(% style="width:1053px" %)(((
	49	☐ C3I operational, describe: .......................................................................................................................................................................................................................
12.1	50
13.1	51	☐ Health & location sensors connected, describe: .....................................................................................................................................................................................
12.1	52
13.1	53	☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................
12.1	54
13.1	55	☐ OWL operational, describe: .....................................................................................................................................................................................................................
12.1	56
13.1	57	☐ ANYmal operational, describe: ................................................................................................................................................................................................................
12.1	58
13.1	59	☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................
12.1	60
13.1	61	☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
12.1	62	)))
22.1	63	\|(% colspan="2" %)PRE-ENTRY PHASE (after briefing; before actual start of scenario)
24.5	64	\|(% style="width:296px" %)Hand out questionnaires\|(% style="width:1053px" %)Each role their own question for this section
13.1	65	\|(% style="width:296px" %)(((
26.2	66	Start time of scenario
13.1	67	)))\|(% style="width:1053px" %)(((
26.2	68	Timestamp of start of the scenario:.....................................................................................
	69	)))
22.1	70	\|(% colspan="2" style="width:296px" %)DURING ENTRY PHASE
13.3	71	\|(% style="width:296px" %)(((
26.2	72	(if squad leader gets a questionnaire after DM, before human entry; pause timer or enter timestamps)
13.3	73	)))\|(% style="width:1053px" %)(((
	74	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
	75	)))
	76	\|(% style="width:296px" %)(((
26.2	77	Time within building/ hazard zone (humans)
19.9	78	)))\|(% style="width:1053px" %)(((
	79	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
26.2	80
	81	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
19.9	82	)))
	83	\|(% style="width:296px" %)(((
26.2	84	Time within building/ hazard zone (robots)
13.3	85	)))\|(% style="width:1053px" %)(((
26.2	86	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
8.2	87
26.2	88	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
13.3	89	)))
	90	\|(% style="width:296px" %)(((
26.2	91	Events with timestamp
13.3	92	)))\|(% style="width:1053px" %)(((
26.2	93	Timestamp of victim detected:......................................................................... \| Timestamp of victim detected:.............................................................................................
13.3	94
26.2	95	Timestamp of victim detected:......................................................................... \| Timestamp of victim detected:.............................................................................................
	96
	97
26.3	98	Timestamp of manual robot control takeover:.................................................. \| Timestamp of manual robot control takeover:....................................................................
	99
	100	Timestamp of manual robot control takeover:.................................................. \| Timestamp of manual robot control takeover:....................................................................
	101
	102
	103	Timestamp of health sensor notification:......................................................... \| Timestamp of health sensor notification:............................................................................
	104
	105	Timestamp of health sensor notification:......................................................... \| Timestamp of health sensor notification:............................................................................
13.3	106	)))
26.2	107	\|(% style="width:296px" %)End time of scenario\|(% style="width:1053px" %)Timestamp of finishing the scenario:........................................................................................................
19.9	108	\|(% colspan="2" style="width:296px" %)POST EXIT PHASE (hot wash)
	109	\|(% style="width:296px" %)Give every role their questionnaire\|(% style="width:1053px" %)
26.2	110	\|(% style="width:296px" %)Plenary questions: \|(% style="width:1053px" %)(((
	111	\|(% colspan="2" style="width:587px" %)Open questions
	112	\|(% colspan="2" style="width:587px" %)(((
	113	What was the plan/intent for the mission, and what did you personally aim to achieve?// (1–2 sentences)//
13.3	114
26.2	115
	116
	117	)))
	118	\|(% colspan="2" style="width:587px" %)(((
	119	What went well and should be sustained? Give one concrete example. //(example + why it worked)//
	120
	121
	122
	123	)))
	124	\|(% colspan="2" style="width:587px" %)(((
	125	What did not go as intended? Give one concrete example and its impact. //(example + consequence)//
	126
	127
	128
	129	)))
	130	\|(% colspan="2" style="width:587px" %)(((
	131	Why did it happen? What were the main contributing factors? //(e.g., information/SA, communication, coordination, timing, tools/technology, environment)//
	132
	133
	134
	135	)))
	136	\|(% colspan="2" style="width:587px" %)(((
	137	What are the top 2 actionable improvements for next time (specific, doable)? //(Action + owner/role + when)//
	138
	139
	140
	141	)))
	142
	143
	144	)))
	145
13.2	146	(% class="table-bordered" %)
	147	\|(% colspan="2" %)(((
19.4	148	== Template: Squad/ Team leader (Field) ==
13.2	149	)))
	150	\|(% colspan="2" %)(((
	151	General:
9.3	152
16.1	153	//Trial type/ Building/ scenario ID//: ................................. \| //Date/ Time//: ....................................... \| //Condition (encircle)//: Baseline / SYNERGISE-tech supported
13.2	154	)))
22.1	155	\|(% colspan="2" style="width:296px" %)PRE-ENTRY PHASE (after briefing; before actual start of scenario)
26.2	156	\|(% style="width:296px" %)Baseline robot/ automation experience\|(% style="width:1053px" %)(((
	157	\|Statement
	158	\|(((
28.1	159	I have experience with robot(s)/ automation in my daily work (encircle): Yes / No
26.2	160
	161	Write down which (optional)
	162
	163
	164
	165	)))
	166	\|(((
28.1	167	I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No
26.2	168
	169	Write down which (optional)
	170
	171
	172
	173	)))
	174	)))
13.2	175	\|(% style="width:296px" %)(((
18.2	176	(CL01) Safety Plan Confirmation
13.2	177	)))\|(% style="width:1053px" %)(((
18.2	178	Mark chosen entry path on provided building map. //Sketch or describe entry path. (Sketch space)//
8.3	179
	180
13.2	181
	182
	183
	184	)))
18.2	185	\|(% style="width:296px" %)(CL02) Situation Awareness (SAGAT; Endsley 1988):\|(% style="width:1053px" %)(((
26.2	186	List expected hazards and victims with details (e.g. location, gender, age, status). //(Bullets and/ or sketch)//
13.2	187
16.1	188
19.5	189
16.1	190
	191	)))
26.2	192	\|(% style="width:296px" %)(((
	193	(CL02) Situation Awareness interior (SAGAT; Endsley 1988):
	194
	195
	196	)))\|(% style="width:1053px" %)(((
18.2	197	Sketch/ Describe expected interior layout. //(Sketch space)//
16.1	198
	199
	200
	201
	202
	203	)))
19.5	204	\|(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO\|(% style="width:1053px" %)(((
16.1	205	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
	206	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	207	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	208	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	209	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	210	)))
26.3	211	\|(% colspan="2" style="width:296px" %)HUMAN ENTRY PHASE
16.1	212	\|(% style="width:296px" %)(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)\|(% style="width:1053px" %)(((
16.2	213	Where are you team members now and what hazards are near them? //(Sketch space)//
16.1	214
	215
	216
16.2	217
16.1	218
	219	)))
19.4	220	\|(% colspan="2" style="width:296px" %)POST EXIT PHASE (hot wash)
16.1	221	\|(% style="width:296px" %)(((
	222	(CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):
	223	)))\|(% style="width:1053px" %)(((
16.2	224	\|(% style="width:587px" %)Statement\|(% style="width:138px" %)1 (LOW)\|(% style="width:60px" %)2\|(% style="width:57px" %)3\|(% style="width:59px" %)4\|(% style="width:58px" %)5\|(% style="width:60px" %)6\|(% style="width:147px" %)7 (HIGH)
	225	\|(% style="width:587px" %)(((
22.1	226	I had a clear understanding of the mission objectives, my role, and the intended plan before execution.
16.2	227	)))\|(% style="width:138px" %)☐\|(% style="width:60px" %)☐\|(% style="width:57px" %)☐\|(% style="width:59px" %)☐\|(% style="width:58px" %)☐\|(% style="width:60px" %)☐\|(% style="width:147px" %)☐
16.3	228	\|(% style="width:587px" %)(((
	229	During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.
	230	)))\|(% style="width:138px" %)☐\|(% style="width:60px" %)☐\|(% style="width:57px" %)☐\|(% style="width:59px" %)☐\|(% style="width:58px" %)☐\|(% style="width:60px" %)☐\|(% style="width:147px" %)☐
	231	\|(% style="width:587px" %)(((
	232	The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.
	233	)))\|(% style="width:138px" %)☐\|(% style="width:60px" %)☐\|(% style="width:57px" %)☐\|(% style="width:59px" %)☐\|(% style="width:58px" %)☐\|(% style="width:60px" %)☐\|(% style="width:147px" %)☐
	234	\|(% style="width:587px" %)(((
	235	Team coordination and communication were effective and supported successful task execution.
	236	)))\|(% style="width:138px" %)☐\|(% style="width:60px" %)☐\|(% style="width:57px" %)☐\|(% style="width:59px" %)☐\|(% style="width:58px" %)☐\|(% style="width:60px" %)☐\|(% style="width:147px" %)☐
	237	\|(% style="width:587px" %)(((
	238	The tools, systems, or support aids available during the mission effectively supported mission execution.
	239	)))\|(% style="width:138px" %)☐\|(% style="width:60px" %)☐\|(% style="width:57px" %)☐\|(% style="width:59px" %)☐\|(% style="width:58px" %)☐\|(% style="width:60px" %)☐\|(% style="width:147px" %)☐
	240	\|(% style="width:587px" %)(((
	241	Based on this exercise, the team is better prepared to perform similar missions in the future.
	242	)))\|(% style="width:138px" %)☐\|(% style="width:60px" %)☐\|(% style="width:57px" %)☐\|(% style="width:59px" %)☐\|(% style="width:58px" %)☐\|(% style="width:60px" %)☐\|(% style="width:147px" %)☐
18.1	243	)))
17.1	244	\|(% style="width:296px" %)(((
	245	(CL01) Safety Error Self-Assessment
	246	)))\|(% style="width:1053px" %)(((
	247	List any unsafe or mistaken decisions you recognize from this mission (if any). //(Provide brief descriptions)//
16.3	248
17.1	249
16.3	250
	251	)))
18.2	252	\|(% style="width:296px" %)(((
	253	(CL02) Situation Awareness (Post-Scenario SAGAT)
	254	)))\|(% style="width:1053px" %)(((
26.2	255	From memory, sketch the interior layout and mark all hazards and victims encountered with list of details (e.g. location, gender, age, status). //(Sketch space and bullets)//
18.3	256
	257
	258
	259
18.2	260	)))
18.3	261	\|(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)\|(% style="width:1053px" %)(((
19.2	262	\|(% style="width:861px" %)Statement\|(% style="width:305px" %)(((
18.3	263	Give a score between:
	264
	265	0 (very low) - 100 (very high)
	266	)))
19.2	267	\|(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?\|(% style="width:305px" %)[[image:1776690776402-253.png]]
	268	\|(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?\|(% style="width:305px" %)[[image:1776690777988-282.png]]
	269	\|(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic\|(% style="width:305px" %)[[image:1776690778813-812.png]]
	270	\|(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?\|(% style="width:305px" %)[[image:1776690779609-960.png]]
	271	\|(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?\|(% style="width:305px" %)[[image:1776690780664-359.png]]
	272	\|(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?\|(% style="width:305px" %)[[image:1776690781527-833.png]]
18.3	273	)))
19.1	274	\|(% style="width:296px" %)(((
19.5	275	(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
19.3	276	)))\|(% style="width:1053px" %)(((
	277	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
24.5	278	\|I would trust this robot system in future missions.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.3	279	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	280	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	281	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	282	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	283	)))
16.1	284
13.2	285	== ==
	286
19.6	287	(% class="table-bordered" %)
	288	\|(% colspan="2" %)(((
	289	== Template: Robot operator ==
	290	)))
	291	\|(% colspan="2" %)(((
	292	General:
8.2	293
19.6	294	//Trial type/ Building/ scenario ID//: ................................. \| //Date/ Time//: ....................................... \| //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
	295	)))
22.1	296	\|(% colspan="2" style="width:296px" %)PRE-ENTRY PHASE (after briefing; before actual start of scenario)
26.2	297	\|(% style="width:296px" %)Baseline robot/ automation experience\|(% style="width:1053px" %)(((
	298	\|Statement
	299	\|(((
28.1	300	I have experience with robot(s)/ automation in my daily work (encircle): Yes / No
26.2	301
	302	Write down which (optional)
	303
	304
	305
	306	)))
	307	\|(((
28.1	308	I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No
26.2	309
	310	Write down which (optional)
	311
	312
	313
	314	)))
	315	)))
19.6	316	\|(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). \|(% style="width:1053px" %)(((
	317	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
26.3	318	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	319	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	320	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	321	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.6	322	)))
26.3	323	\|(% colspan="2" style="width:296px" %)AFTER ROBOT entry
19.9	324	\|(% style="width:296px" %)(((
19.11	325	(CL02) Situation Awareness Robot Reconnaissance data understanding (OPTIONAL; is more important for the analyst)
19.9	326	)))\|(% style="width:1053px" %)(((
26.3	327	After the robot’s initial run but before humans enter, list any //hazards, obstacles, or victims// that you (as operator) detected or suspect from the robot’s feed with list of details (e.g. location, gender, age, status). //(Open list)//
19.9	328
	329
	330
	331
	332	)))
23.1	333	\|(% colspan="2" style="width:296px" %)POST EXIT PHASE (hot wash)
19.9	334	\|(% style="width:296px" %)(((
	335	(CL01) SafetyTeam Decision Appraisal:
	336	)))\|(% style="width:1053px" %)(((
	337	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
	338	\|How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	339	\|Did your team correctly decide when to send the robot versus humans?\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	340	)))
19.11	341	\|(% style="width:296px" %)(((
	342	(CL01) Safety Error Reflection
	343	)))\|(% style="width:1053px" %)(((
	344	List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
8.2	345
	346
8.3	347
19.11	348
	349	)))
	350	\|(% style="width:296px" %)(((
	351	(CL02) Situation Awareness (Post-Scenario SAGAT)
	352	)))\|(% style="width:1053px" %)(((
	353	Describe what the robot observed/ found during the mission (e.g. hazards, obstacles) for navigation purposes. //(Sketch space and bullets)//
8.3	354
19.4	355
	356
19.11	357
	358	)))
	359	\|(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)\|(% style="width:1053px" %)(((
	360	\|(% style="width:861px" %)Statement\|(% style="width:305px" %)(((
	361	Give a score between:
	362
	363	0 (very low) - 100 (very high)
	364	)))
	365	\|(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?\|(% style="width:305px" %)[[image:1776690776402-253.png]]
	366	\|(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?\|(% style="width:305px" %)[[image:1776690777988-282.png]]
	367	\|(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic\|(% style="width:305px" %)[[image:1776690778813-812.png]]
	368	\|(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?\|(% style="width:305px" %)[[image:1776690779609-960.png]]
	369	\|(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?\|(% style="width:305px" %)[[image:1776690780664-359.png]]
	370	\|(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?\|(% style="width:305px" %)[[image:1776690781527-833.png]]
	371	)))
	372	\|(% style="width:296px" %)(((
	373	(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
	374	)))\|(% style="width:1053px" %)(((
	375	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
24.5	376	\|I would trust this robot system in future missions.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.11	377	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	378	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	379	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	380	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	381	)))
	382
	383
	384
19.4	385	(% class="table-bordered" %)
	386	\|(% colspan="2" %)(((
19.11	387	== Template: Robot analyst ==
	388	)))
	389	\|(% colspan="2" %)(((
	390	General:
	391
	392	//Trial type/ Building/ scenario ID//: ................................. \| //Date/ Time//: ....................................... \| //Condition (encircle)//: --Baseline --/ SYNERGISE-tech supported
	393	)))
22.1	394	\|(% colspan="2" style="width:296px" %)PRE-ENTRY PHASE (after briefing; before actual start of scenario)
26.2	395	\|(% style="width:296px" %)Baseline robot/ automation experience\|(% style="width:1053px" %)(((
	396	\|Statement
	397	\|(((
28.1	398	I have experience with robot(s)/ automation in my daily work (encircle): Yes / No
26.2	399
	400	Write down which (optional)
	401
	402
	403
	404	)))
	405	\|(((
28.1	406	I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No
26.2	407
	408	Write down which (optional)
	409
	410
	411
	412	)))
	413	)))
27.1	414	\|(% style="width:296px" %)(((
	415	(CL01) Safety Plan Confirmation
	416	)))\|(% style="width:1053px" %)(((
	417	Mark chosen entry path on provided building map. //Sketch or describe entry path. (Sketch space)//
19.11	418
	419
	420
27.1	421
19.11	422
	423	)))
27.1	424	\|(% style="width:296px" %)(CL02) Situation Awareness (SAGAT; Endsley 1988):\|(% style="width:1053px" %)(((
	425	List expected hazards and victims with details (e.g. location, gender, age, status). //(Bullets and/ or sketch)//
	426
	427
	428
	429
	430	)))
26.3	431	\|(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). \|(% style="width:1053px" %)(((
23.1	432	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
26.3	433	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	434	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	435	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	436	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
23.1	437	)))
	438	\|(% colspan="2" style="width:445px" %)POST EXIT PHASE (hot wash)
	439	\|(% style="width:296px" %)(((
	440	(CL01) SafetyTeam Decision Appraisal:
	441	)))\|(% style="width:1053px" %)(((
	442	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
	443	\|How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	444	\|Did your team correctly decide when to send the robot versus humans?\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	445	)))
	446	\|(% style="width:296px" %)(((
	447	(CL01) Safety Error Reflection
	448	)))\|(% style="width:1053px" %)(((
	449	List any mistakes or unsafe decisions you think occurred regarding robot deployment. //(Open question)//
19.11	450
	451
	452
23.1	453
	454	)))
	455	\|(% style="width:296px" %)(((
	456	(CL02) Situation Awareness (Post-Scenario SAGAT)
	457	)))\|(% style="width:1053px" %)(((
	458	Now that the mission is over, summarize the final floor layout, and list all hazards and victims found or confirmed. //(Sketch space and bullets)//
19.11	459
23.1	460
	461
	462
	463	)))
	464	\|(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)\|(% style="width:1053px" %)(((
	465	\|(% style="width:861px" %)Statement\|(% style="width:305px" %)(((
	466	Give a score between:
	467
	468	0 (very low) - 100 (very high)
	469	)))
	470	\|(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?\|(% style="width:305px" %)[[image:1776690776402-253.png]]
	471	\|(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?\|(% style="width:305px" %)[[image:1776690777988-282.png]]
	472	\|(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic\|(% style="width:305px" %)[[image:1776690778813-812.png]]
	473	\|(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?\|(% style="width:305px" %)[[image:1776690779609-960.png]]
	474	\|(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?\|(% style="width:305px" %)[[image:1776690780664-359.png]]
	475	\|(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?\|(% style="width:305px" %)[[image:1776690781527-833.png]]
	476	)))
	477	\|(% style="width:296px" %)(((
	478	(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000)
	479	)))\|(% style="width:1053px" %)(((
	480	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
24.5	481	\|I would trust this robot system in future missions.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
23.1	482	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	483	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	484	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	485	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	486	)))
	487
	488
	489
19.11	490	(% class="table-bordered" %)
	491	\|(% colspan="2" %)(((
19.5	492	== Template: Entry team member ==
19.4	493	)))
	494	\|(% colspan="2" %)(((
	495	General:
	496
	497	//Trial type/ Building/ scenario ID//: ................................. \| //Date/ Time//: ....................................... \| //Condition (encircle)//: Baseline / SYNERGISE-tech supported
	498	)))
22.1	499	\|(% colspan="2" style="width:296px" %)PRE-ENTRY PHASE (after briefing; before actual start of scenario)
26.2	500	\|(% style="width:296px" %)Baseline robot/ automation experience\|(% style="width:1053px" %)(((
	501	\|Statement
	502	\|(((
28.1	503	I have experience with robot(s)/ automation in my daily work (encircle): Yes / No
26.2	504
	505	Write down which (optional)
	506
	507
	508
	509	)))
	510	\|(((
28.1	511	I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No
26.2	512
	513	Write down which (optional)
	514
	515
	516
	517	)))
	518	)))
19.4	519	\|(% style="width:296px" %)(((
27.1	520	(CL01) Safety Plan Confirmation
19.5	521	)))\|(% style="width:1053px" %)(((
27.1	522	Mark chosen entry path on provided building map. //Sketch or describe entry path. (Sketch space)//
19.5	523
	524
27.1	525
	526
19.5	527
	528	)))
27.1	529	\|(% style="width:296px" %)(CL02) Situation Awareness (SAGAT; Endsley 1988):\|(% style="width:1053px" %)(((
	530	List expected hazards and victims with details (e.g. location, gender, age, status). //(Bullets and/ or sketch)//
	531
	532
	533
	534
	535	)))
19.5	536	\|(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO\|(% style="width:1053px" %)(((
	537	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
22.1	538	\|I am confident in the robot analyst's capabilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.5	539	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	540	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	541	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	542	)))
24.3	543	\|(% colspan="2" style="width:296px" %)POST EXIT PHASE (hot wash)
19.5	544	\|(% style="width:296px" %)(((
22.1	545	(CL01) Safety - Team Decision Appraisal:
19.5	546	)))\|(% style="width:1053px" %)(((
	547	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
22.1	548	\|How would you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.9	549	\|Did your team correctly decide when to send the robot versus humans?\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.5	550	)))
	551	\|(% style="width:296px" %)(((
	552	(CL02) Situation Awareness Post Operation Recall
	553	)))\|(% style="width:1053px" %)(((
27.1	554	Describe the layout and list all hazards/victims that were found with details (e.g. location, gender, age, status). //(Sketch and describe)//
19.5	555
	556
	557
	558
	559
	560	)))
	561	\|(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)\|(% style="width:1053px" %)(((
	562	\|(% style="width:861px" %)Statement\|(% style="width:305px" %)(((
	563	Give a score between:
	564
	565	0 (very low) - 100 (very high)
	566	)))
	567	\|(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?\|(% style="width:305px" %)[[image:1776690776402-253.png]]
	568	\|(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?\|(% style="width:305px" %)[[image:1776690777988-282.png]]
	569	\|(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic\|(% style="width:305px" %)[[image:1776690778813-812.png]]
	570	\|(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?\|(% style="width:305px" %)[[image:1776690779609-960.png]]
	571	\|(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?\|(% style="width:305px" %)[[image:1776690780664-359.png]]
	572	\|(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?\|(% style="width:305px" %)[[image:1776690781527-833.png]]
	573	)))
	574	\|(% style="width:296px" %)(((
	575	(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
	576	)))\|(% style="width:1053px" %)(((
	577	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
24.5	578	\|I would trust this robot system in future missions.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
19.5	579	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	580	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	581	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	582	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	583	)))
	584
19.9	585
24.2	586
	587	(% class="table-bordered" %)
	588	\|(% colspan="2" %)(((
23.2	589	== Template: Base of Operations (LEMA) ==
24.2	590	)))
	591	\|(% colspan="2" %)(((
19.9	592	General:
	593
	594	//Trial type/ Building/ scenario ID//: ................................. \| //Date/ Time//: ....................................... \| //Condition (encircle)//: Baseline / SYNERGISE-tech supported
	595	)))
27.1	596	\|(% colspan="2" style="width:296px" %)RE-ENTRY PHASE (after briefing; before actual start of scenario)
26.2	597	\|(% style="width:296px" %)Baseline robot/ automation experience\|(% style="width:1053px" %)(((
	598	\|Statement
	599	\|(((
28.1	600	I have experience with robot(s)/ automation in my daily work (encircle): Yes / No
26.2	601
	602	Write down which (optional)
	603
	604
	605
	606	)))
	607	\|(((
28.1	608	I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No
26.2	609
	610	Write down which (optional)
	611
	612
	613
	614	)))
	615	)))
24.2	616	\|(% style="width:296px" %)(CL04) Trust in automation (Baseline (% style="background-color:transparent" %)Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO\|(% style="width:1053px" %)(((
23.2	617	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
	618	\|I am confident in the robot analyst's capabilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	619	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	620	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	621	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
24.2	622	)))
24.3	623	\|(% colspan="2" style="width:296px" %)POST EXIT PHASE (hot wash)
24.5	624	\|(% style="width:296px" %)(((
	625	(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO
	626	)))\|(% style="width:1053px" %)(((
	627	\|Statement\|1 (LOW)\|2\|3\|4\|5\|6\|7 (HIGH)
	628	\|I would trust this robot system in future missions.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	629	\|I am confident in the robot/system’s abilities.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	630	\|The robot/system will act in our best interest.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	631	\|The robot/system is reliable.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	632	\|I trust the system’s information.\|☐\|☐\|☐\|☐\|☐\|☐\|☐
	633	)))
	634	\|(% style="width:296px" %)(CL03) Workload (NASA-TLX; Hart & Staveland 1988)\|(% style="width:1053px" %)(((
	635	\|(% style="width:861px" %)Statement\|(% style="width:305px" %)(((
	636	Give a score between:
24.3	637
24.5	638	0 (very low) - 100 (very high)
	639	)))
	640	\|(% style="width:861px" %)Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?\|(% style="width:305px" %)[[image:1776690776402-253.png]]
	641	\|(% style="width:861px" %)Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?\|(% style="width:305px" %)[[image:1776690777988-282.png]]
	642	\|(% style="width:861px" %)Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic\|(% style="width:305px" %)[[image:1776690778813-812.png]]
	643	\|(% style="width:861px" %)Effort - How hard did you work mentally and physically to accomplish your level of performance?\|(% style="width:305px" %)[[image:1776690779609-960.png]]
	644	\|(% style="width:861px" %)Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?\|(% style="width:305px" %)[[image:1776690780664-359.png]]
	645	\|(% style="width:861px" %)Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?\|(% style="width:305px" %)[[image:1776690781527-833.png]]
	646	)))
24.3	647
25.2	648	=== Before humans enter a building/ hazard ===
24.3	649
25.2	650	\|Claim\|Measurement\|What is recorded\|Who records
	651	\|CL01 Safety\|Ground-truth safe paths\|predefined safe/ suboptimal / unsafe paths\|//Evaluation leader//
	652	\|(((
	653	CL01 Safety
	654	)))\|(((
	655	Chosen entry path
	656	)))\|(((
	657	Intended entry path selection
	658	)))\|//Squad/ Team leader (informed by Robot analyst)//
	659	\|(((
	660	CL02 Situation Awareness
	661	)))\|(((
	662	Reported hazards, victims, layout (pre)
	663	)))\|(((
	664	Sketch / map / description of interior
	665	)))\|//Squad/ Team leader, Entry team, Robot analyst (separately)//
	666	\|(((
	667	CL02 SA
	668	)))\|(((
	669	Robot‑based expectations
	670	)))\|(((
	671	Expected hazards / victims from robot data
	672	)))\|//Robot analyst//
25.6	673	\|CL05 Trustworthy SA\|Baseline trust (optional)\|Initial trust in system/ robot\|//Squad/ Team leader, Robot operator, Robot analyst//
25.2	674	\|(((
	675	CL06 Health
	676	)))\|(((
	677	Acceptable health ranges
	678	)))\|(((
	679	HR, temp thresholds
	680	)))\|//Evaluation leader//
	681	\|(((
	682	CL07 Efficiency
	683	)))\|(((
	684	Mission start timestamp
	685	)))\|(((
	686	End briefing → start
	687	)))\|//Evaluation leader//
24.5	688
25.2	689	During humans/robots are inside building/ hazard
24.5	690
25.2	691	\|Claim\|Measurement\|What is recorded\|Who records
	692	\|(((
	693	CL01 Safety
	694	)))\|(((
	695	Near‑incidents
	696	)))\|(((
	697	Unsafe situations / close calls
	698	)))\|//Evaluation leader//
	699	\|(((
	700	CL01 Safety
	701	)))\|(((
	702	Hazard avoidance
	703	)))\|(((
	704	Avoided predefined hazards
	705	)))\|//Evaluation leader//
	706	\|(((
	707	CL02 SA
	708	)))\|(((
	709	Executed path
	710	)))\|(((
	711	Actual path taken
	712	)))\|//Evaluation leader, Engage system/ location sensors (?)//
	713	\|(((
	714	CL02 SA
	715	)))\|(((
	716	Path deviations
	717	)))\|(((
	718	Deviations + cause (something for after instead?)
	719	)))\|//Squad/ Team leader, Entry team//
	720	\|(((
	721	CL02 SA
	722	)))\|(((
	723	Interpretations of robot images
	724	)))\|(((
	725	...?
	726	)))\|//Robot analyst (and Base or Operations/ LEMA ?)//
	727	\|(((
	728	CL02 SA (Optional)
	729	)))\|(((
	730	Robot control actions
	731	)))\|(((
	732	Overrides, manual interventions
	733	)))\|//Robot operator//
	734	\|(((
	735	CL06 Health
	736	)))\|(((
	737	Threshold breach
	738	)))\|(((
	739	Breach moment (HR, temp, stress (?))
	740	)))\|//Evaluation leader//
	741	\|(((
	742	CL06 Health
	743	)))\|(((
	744	Detection & intervention
	745	)))\|(((
	746	Detection with corresponding action
	747	)))\|//Evaluation leader//
	748	\|(((
	749	CL07 Efficiency
	750	)))\|(((
	751	Entry timestamp
	752	)))\|(((
	753	Entry team enters OR robot enters
	754	)))\|//Evaluation leader//
	755	\|(((
	756	CL07 Efficiency
	757	)))\|(((
	758	Exit timestamp
	759	)))\|Entry team exits OR robot exits \|//Evaluation leader//
	760
	761	=== After humans/robots exited building/hazard ===
	762
	763	\|Claim\|Measurement\|What is recorded\|Who records
	764	\|(((
19.9	765
25.2	766
	767	CL01 Safety
	768	)))\|(((
	769
	770
	771	Decision quality (retro)
	772	)))\|(((
	773
	774
	775	“How good were decisions?”
	776	)))\|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
	777	\|(((
	778	CL01 Safety
	779	)))\|(((
	780	Error reflection
	781	)))\|(((
	782	Reflected unsafe choices
	783	)))\|//Squad/ Team leader, Robot operator, Robot analyst//
	784	\|(((
25.5	785	CL02 SA
25.2	786	)))\|(((
	787	Reported hazards, victims, layout (post)
	788	)))\|(((
	789	Sketch / map / description
	790	)))\|//Squad/ Team leader, Entry team, Robot analyst//
25.5	791	\|CL03 mission effectiveness\|Decision and path quality\|Route of FRs, decisions made.\|//Squad/ Team leader, Entry team, Robot analyst//
25.2	792	\|(((
25.4	793	CL04 acceptable workload
25.2	794	)))\|(((
	795	Workload
	796	)))\|(((
	797	Short NASA‑TLX
	798	)))\|//Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA//
	799	\|(((
25.4	800	CL05 Trustworthy SA
25.2	801	)))\|(((
	802	Trust
	803	)))\|(((
	804	Trust survey / interview
	805	)))\|//Squad/ Team leader, Entry team (?), Robot operator, Robot analyst, Baes of Operations/ LEMA//
	806	\|(((
	807	CL06 Health
	808	)))\|(((
	809	Health issue handling
	810	)))\|(((
	811	Adequacy & timeliness
	812	)))\|//Evaluation leader (+ expert)//
	813	\|(((
	814	CL07 Efficiency
	815	)))\|(((
	816	Total mission duration
	817	)))\|(((
	818	Start (end briefing) till end (?)
	819	)))\|//Evaluation leader//
	820
	821

Wiki source code of j. SFT1 evaluation templates

Welcome

Navigation

Recently Modified