j. SFT1 evaluation templates

Last modified by Rosa Van Tuijn on 2026/05/05 11:58

General


Baseline phases	Who fills it	Main purpose
Briefing	-	-
Estimate building integrity	-	-
Shoring	Particpants	(make estimation about shoring time)
Before sending in FRs (DM)	Participant	Ground truth/ baseline SA
Send in FRs, identify voids, DM	Evaluation leader	Objective performance, safety, timing, decision making
FRs exit building	Participant	Subjective SA, trust, workload, decision making
Hotwash


SYNERGISE phases	Who fills it	Main purpose
Briefing	-	-
Estimate building integrity	-	-
Send in robots, identify voids with robots	-	-
Decision Making (DM)	Participants (Squad/ team leader)	Ground truth/ baseline SA
Send in FRs ?	Evaluation leader	Objective performance, safety, timing, decision making
After test	Paricipant	Subjective SA, trust, workload, decision making
Hotwash

List of ground truths needed for comparison:

Health ranges
hazards/ victims in buildings + Safest path within buildings : Safety observations + path observations >> have a premade map with everything and note down the route. Have an expert rate it after the fact.
Decision quality: have expert (or team leaders) rate the path of the FRs inside
State of automation (tech question)

Templates per role

Template: Evaluation leader (Field)

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

System & mission readiness

(Tick list based on predefined acceptable ranges. Cross out if not applicable.)

☐ C3I operational, describe: .......................................................................................................................................................................................................................

☐ Health & location sensors connected, describe: .....................................................................................................................................................................................

☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: .....................................................................................................................................

☐ OWL operational, describe: .....................................................................................................................................................................................................................

☐ ANYmal operational, describe: ................................................................................................................................................................................................................

☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................

☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

Hand out questionnaires

Each role their own question for this section

Start time of scenario

Timestamp of start of the scenario:.....................................................................................

DURING ENTRY PHASE

(if squad leader gets a questionnaire after DM, before human entry; pause timer or enter timestamps)

Timestamp of entering:..................................................................................... | Timestamp of exiting:...........................................................................................................

Time within building/ hazard zone (humans)

Time within building/ hazard zone (robots)

Events with timestamp

Timestamp of victim detected:......................................................................... | Timestamp of victim detected:.............................................................................................

Timestamp of manual robot control takeover:.................................................. | Timestamp of manual robot control takeover:....................................................................

Timestamp of health sensor notification:......................................................... | Timestamp of health sensor notification:............................................................................

End time of scenario

Timestamp of finishing the scenario:........................................................................................................

POST EXIT PHASE (hot wash)

Give every role their questionnaire

Plenary questions:

Open questions
What was the plan/intent for the mission, and what did you personally aim to achieve? (1–2 sentences)
What went well and should be sustained? Give one concrete example. (example + why it worked)
What did not go as intended? Give one concrete example and its impact. (example + consequence)
Why did it happen? What were the main contributing factors? (e.g., information/SA, communication, coordination, timing, tools/technology, environment)
What are the top 2 actionable improvements for next time (specific, doable)? (Action + owner/role + when)

Template: Squad/ Team leader (Field)

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

Baseline robot/ automation experience

Statement

I have experience with robot(s)/ automation in my daily work (encircle): Yes / No

Write down which (optional)

I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No

Write down which (optional)

(CL01) Safety Plan Confirmation

Mark chosen entry path on provided building map. Sketch or describe entry path. (Sketch space)

(CL02) Situation Awareness (SAGAT; Endsley 1988):

List expected hazards and victims with details (e.g. location, gender, age, status). (Bullets and/ or sketch)

(CL02) Situation Awareness interior (SAGAT; Endsley 1988):

Sketch/ Describe expected interior layout. (Sketch space)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

HUMAN ENTRY PHASE

(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)

Where are you team members now and what hazards are near them? (Sketch space)

POST EXIT PHASE (hot wash)

(CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I had a clear understanding of the mission objectives, my role, and the intended plan before execution.	☐	☐	☐	☐	☐	☐	☐
During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.	☐	☐	☐	☐	☐	☐	☐
The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.	☐	☐	☐	☐	☐	☐	☐
Team coordination and communication were effective and supported successful task execution.	☐	☐	☐	☐	☐	☐	☐
The tools, systems, or support aids available during the mission effectively supported mission execution.	☐	☐	☐	☐	☐	☐	☐
Based on this exercise, the team is better prepared to perform similar missions in the future.	☐	☐	☐	☐	☐	☐	☐

(CL01) Safety Error Self-Assessment

List any unsafe or mistaken decisions you recognize from this mission (if any). (Provide brief descriptions)

(CL02) Situation Awareness (Post-Scenario SAGAT)

From memory, sketch the interior layout and mark all hazards and victims encountered with list of details (e.g. location, gender, age, status). (Sketch space and bullets)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I would trust this robot system in future missions.	☐	☐	☐	☐	☐	☐	☐
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

Template: Robot operator

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): ~~Baseline~~ / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

Baseline robot/ automation experience

Statement

I have experience with robot(s)/ automation in my daily work (encircle): Yes / No

Write down which (optional)

I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No

Write down which (optional)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000).

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

AFTER ROBOT entry

(CL02) Situation Awareness Robot Reconnaissance data understanding (OPTIONAL; is more important for the analyst)

After the robot’s initial run but before humans enter, list any hazards, obstacles, or victims that you (as operator) detected or suspect from the robot’s feed with list of details (e.g. location, gender, age, status). (Open list)

POST EXIT PHASE (hot wash)

(CL01) SafetyTeam Decision Appraisal:

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?	☐	☐	☐	☐	☐	☐	☐
Did your team correctly decide when to send the robot versus humans?	☐	☐	☐	☐	☐	☐	☐

(CL01) Safety Error Reflection

List any mistakes or unsafe decisions you think occurred regarding robot deployment. (Open question)

(CL02) Situation Awareness (Post-Scenario SAGAT)

Describe what the robot observed/ found during the mission (e.g. hazards, obstacles) for navigation purposes. (Sketch space and bullets)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I would trust this robot system in future missions.	☐	☐	☐	☐	☐	☐	☐
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

Template: Robot analyst

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): ~~Baseline~~ / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

Baseline robot/ automation experience

Statement

I have experience with robot(s)/ automation in my daily work (encircle): Yes / No

Write down which (optional)

I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No

Write down which (optional)

(CL01) Safety Plan Confirmation

Mark chosen entry path on provided building map. Sketch or describe entry path. (Sketch space)

(CL02) Situation Awareness (SAGAT; Endsley 1988):

List expected hazards and victims with details (e.g. location, gender, age, status). (Bullets and/ or sketch)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000).

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

POST EXIT PHASE (hot wash)

(CL01) SafetyTeam Decision Appraisal:

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?	☐	☐	☐	☐	☐	☐	☐
Did your team correctly decide when to send the robot versus humans?	☐	☐	☐	☐	☐	☐	☐

(CL01) Safety Error Reflection

List any mistakes or unsafe decisions you think occurred regarding robot deployment. (Open question)

(CL02) Situation Awareness (Post-Scenario SAGAT)

Now that the mission is over, summarize the final floor layout, and list all hazards and victims found or confirmed. (Sketch space and bullets)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000)

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I would trust this robot system in future missions.	☐	☐	☐	☐	☐	☐	☐
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

Template: Entry team member

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

Baseline robot/ automation experience

Statement

I have experience with robot(s)/ automation in my daily work (encircle): Yes / No

Write down which (optional)

I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No

Write down which (optional)

(CL01) Safety Plan Confirmation

Mark chosen entry path on provided building map. Sketch or describe entry path. (Sketch space)

(CL02) Situation Awareness (SAGAT; Endsley 1988):

List expected hazards and victims with details (e.g. location, gender, age, status). (Bullets and/ or sketch)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot analyst's capabilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

POST EXIT PHASE (hot wash)

(CL01) Safety - Team Decision Appraisal:

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
How would you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?	☐	☐	☐	☐	☐	☐	☐
Did your team correctly decide when to send the robot versus humans?	☐	☐	☐	☐	☐	☐	☐

(CL02) Situation Awareness Post Operation Recall

Describe the layout and list all hazards/victims that were found with details (e.g. location, gender, age, status). (Sketch and describe)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I would trust this robot system in future missions.	☐	☐	☐	☐	☐	☐	☐
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

Template: Base of Operations (LEMA)

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

RE-ENTRY PHASE (after briefing; before actual start of scenario)

Baseline robot/ automation experience

Statement

I have experience with robot(s)/ automation in my daily work (encircle): Yes / No

Write down which (optional)

I have experience with robot(s)/ automation within the SYNERGISE project (encircle): Yes / No

Write down which (optional)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot analyst's capabilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

POST EXIT PHASE (hot wash)

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I would trust this robot system in future missions.	☐	☐	☐	☐	☐	☐	☐
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

Before humans enter a building/ hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Ground-truth safe paths	predefined safe/ suboptimal / unsafe paths	Evaluation leader
CL01 Safety	Chosen entry path	Intended entry path selection	Squad/ Team leader (informed by Robot analyst)
CL02 Situation Awareness	Reported hazards, victims, layout (pre)	Sketch / map / description of interior	Squad/ Team leader, Entry team, Robot analyst (separately)
CL02 SA	Robot‑based expectations	Expected hazards / victims from robot data	Robot analyst
CL05 Trustworthy SA	Baseline trust (optional)	Initial trust in system/ robot	Squad/ Team leader, Robot operator, Robot analyst
CL06 Health	Acceptable health ranges	HR, temp thresholds	Evaluation leader
CL07 Efficiency	Mission start timestamp	End briefing → start	Evaluation leader

During humans/robots are inside building/ hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Near‑incidents	Unsafe situations / close calls	Evaluation leader
CL01 Safety	Hazard avoidance	Avoided predefined hazards	Evaluation leader
CL02 SA	Executed path	Actual path taken	Evaluation leader, Engage system/ location sensors
CL02 SA	Path deviations	Deviations + cause (something for after instead?)	Squad/ Team leader, Entry team
CL02 SA	Interpretations of robot images	...?	Robot analyst (and Base or Operations/ LEMA ?)
CL02 SA (Optional)	Robot control actions	Overrides, manual interventions	Robot operator
CL06 Health	Threshold breach	Breach moment (HR, temp, stress )	Evaluation leader
CL06 Health	Detection & intervention	Detection with corresponding action	Evaluation leader
CL07 Efficiency	Entry timestamp	Entry team enters OR robot enters	Evaluation leader
CL07 Efficiency	Exit timestamp	Entry team exits OR robot exits	Evaluation leader

After humans/robots exited building/hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Decision quality (retro)	“How good were decisions?”	Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA
CL01 Safety	Error reflection	Reflected unsafe choices	Squad/ Team leader, Robot operator, Robot analyst
CL02 SA	Reported hazards, victims, layout (post)	Sketch / map / description	Squad/ Team leader, Entry team, Robot analyst
CL03 mission effectiveness	Decision and path quality	Route of FRs, decisions made.	Squad/ Team leader, Entry team, Robot analyst
CL04 acceptable workload	Workload	Short NASA‑TLX	Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA
CL05 Trustworthy SA	Trust	Trust survey / interview	Squad/ Team leader, Entry team , Robot operator, Robot analyst, Baes of Operations/ LEMA
CL06 Health	Health issue handling	Adequacy & timeliness	Evaluation leader (+ expert)
CL07 Efficiency	Total mission duration	Start (end briefing) till end	Evaluation leader

j. SFT1 evaluation templates

General

List of ground truths needed for comparison:

Templates per role

Template: Evaluation leader (Field)

Template: Squad/ Team leader (Field)

Template: Robot operator

Template: Robot analyst

Template: Entry team member

Template: Base of Operations (LEMA)

Before humans enter a building/ hazard

After humans/robots exited building/hazard

Welcome

Navigation

Recently Modified