j. SFT1 evaluation templates

Version 22.1 by Rosa Van Tuijn on 2026/04/20 14:32

General


Baseline phases	Who fills it	Main purpose
Briefing	-	-
Estimate building integrity	-	-
Shoring	Particpants	(make estimation about shoring time)
Before sending in FRs (DM)	Participant	Ground truth/ baseline SA
Send in FRs, identify voids, DM	Evaluation leader	Objective performance, safety, timing, decision making
FRs exit building	Participant	Subjective SA, trust, workload, decision making
Hotwash


SYNERGISE phases	Who fills it	Main purpose
Briefing	-	-
Estimate building integrity	-	-
Send in robots, identify voids with robots	-	-
Decision Making (DM)	Participants (Squad/ team leader)	Ground truth/ baseline SA
Send in FRs ?	Evaluation leader	Objective performance, safety, timing, decision making
After test	Paricipant	Subjective SA, trust, workload, decision making
Hotwash

List of ground truths needed for comparison:

Health ranges
hazards/ victims in buildings
Safest path within buildings
State of automation (tech question)

Before humans enter a building/ hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Ground-truth safe paths	predefined safe/ suboptimal / unsafe paths	Evaluation leader
CL01 Safety	Chosen entry path	Intended entry path selection	Squad/ Team leader (informed by Robot analyst)
CL02 Situation Awareness	Reported hazards, victims, layout (pre)	Sketch / map / description of interior	Squad/ Team leader, Entry team, Robot analyst (separately)
CL02 SA	Robot‑based expectations	Expected hazards / victims from robot data	Robot analyst
CL03–05 Mission effectiveness	Baseline trust (optional)	Initial trust in system/ robot	Squad/ Team leader, Robot operator, Robot analyst
CL06 Health	Acceptable health ranges	HR, temp thresholds	Evaluation leader
CL07 Efficiency	Mission start timestamp	End briefing → start	Evaluation leader

During humans/robots are inside building/ hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Near‑incidents	Unsafe situations / close calls	Evaluation leader
CL01 Safety	Hazard avoidance	Avoided predefined hazards	Evaluation leader
CL02 SA	Executed path	Actual path taken	Evaluation leader, Engage system/ location sensors
CL02 SA	Path deviations	Deviations + cause (something for after instead?)	Squad/ Team leader, Entry team
CL02 SA	Interpretations of robot images	...?	Robot analyst (and Base or Operations/ LEMA ?)
CL02 SA (Optional)	Robot control actions	Overrides, manual interventions	Robot operator
CL06 Health	Threshold breach	Breach moment (HR, temp, stress )	Evaluation leader
CL06 Health	Detection & intervention	Detection with corresponding action	Evaluation leader
CL07 Efficiency	Entry timestamp	Entry team enters OR robot enters	Evaluation leader
CL07 Efficiency	Exit timestamp	Entry team exits OR robot exits	Evaluation leader

After humans/robots exited building/hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Decision quality (retro)	“How good were decisions?”	Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA
CL01 Safety	Error reflection	Reflected unsafe choices	Squad/ Team leader, Robot operator, Robot analyst
CL02 SA	Reported hazards, victims, layout (post)	Sketch / map / description	Squad/ Team leader, Entry team, Robot analyst
CL03 Effectiveness	Workload	Short NASA‑TLX	Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA
CL04 Effectiveness	Trust	Trust survey / interview	Squad/ Team leader, Entry team , Robot operator, Robot analyst, Baes of Operations/ LEMA
CL05 Effectiveness	Decision confidence	Confidence in own judgments	Squad/ Team leader, Robot operator, Robot analyst
CL06 Health	Health issue handling	Adequacy & timeliness	Evaluation leader (+ expert)
CL07 Efficiency	Total mission duration	Start (end briefing) till end	Evaluation leader

Templates per role

Template: Evaluation leader (Field)
Before start scenario
General: Trial type/ Building/ scenario ID: ................................. \| Date/ Time: ....................................... \| Condition (encircle): Baseline / SYNERGISE-tech supported
System & mission readiness (Tick list based on predefined acceptable ranges. Cross out if not applicable.)	☐ C3I operational, describe: ....................................................................................................................................................................................................................... ☐ Health & location sensors connected, describe: ..................................................................................................................................................................................... ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: ..................................................................................................................................... ☐ OWL operational, describe: ..................................................................................................................................................................................................................... ☐ ANYmal operational, describe: ................................................................................................................................................................................................................ ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................ ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
PRE-ENTRY PHASE (after briefing; before actual start of scenario)
Let Team leader (and entry team?) sketch/ describe hazard, victims, layout	Notes:
Threshold breached, alert and intervention record	Health threshold breached (timestamp + action taken) Alert detection by system (timestamp + action taken) Intervention started (timestamp + action taken)
DURING ENTRY PHASE
Time within building/ hazard zone (humans)	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
Time within building/ hazard zone (robots)	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
Safety observations	☐ No unsafe situations observed ☐ Unsafe / near‑incident observed, describe (amount and type) :.................................................................................................................................................................. .........................................................................................................................................................................................................................................................................
Path deviations observed (Check if the entry team takes the route that is discussed before entering)	☐ None ☐ Minor ☐ Major Notes:

POST EXIT PHASE (hot wash)
Give every role their questionnaire
Plenary questions:	...??

Template: Squad/ Team leader (Field)

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

(CL01) Safety Plan Confirmation

Mark chosen entry path on provided building map. Sketch or describe entry path. (Sketch space)

(CL02) Situation Awareness (SAGAT; Endsley 1988):

List expected hazards and victims. (Bullets and/ or sketch)

(CL02) Situation Awareness interior (SAGAT; Endsley 1988):

Sketch/ Describe expected interior layout. (Sketch space)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

DURING ENTRY PHASE

(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)

Where are you team members now and what hazards are near them? (Sketch space)

POST EXIT PHASE (hot wash)

(CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I had a clear understanding of the mission objectives, my role, and the intended plan before execution.	☐	☐	☐	☐	☐	☐	☐
During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.	☐	☐	☐	☐	☐	☐	☐
The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.	☐	☐	☐	☐	☐	☐	☐
Team coordination and communication were effective and supported successful task execution.	☐	☐	☐	☐	☐	☐	☐
The tools, systems, or support aids available during the mission effectively supported mission execution.	☐	☐	☐	☐	☐	☐	☐
Based on this exercise, the team is better prepared to perform similar missions in the future.	☐	☐	☐	☐	☐	☐	☐

OR (open questions better?)

Open questions

What was the plan/intent for the mission, and what did you personally aim to achieve? (1–2 sentences)

What actually happened? List the 2–3 most important events or turning points. (bullets)

What went well and should be sustained? Give one concrete example. (example + why it worked)

What did not go as intended? Give one concrete example and its impact. (example + consequence)

Why did it happen? What were the main contributing factors? (e.g., information/SA, communication, coordination, timing, tools/technology, environment)

What are the top 2 actionable improvements for next time (specific, doable)? (Action + owner/role + when)

(CL01) Safety Error Self-Assessment

List any unsafe or mistaken decisions you recognize from this mission (if any). (Provide brief descriptions)

(CL02) Situation Awareness (Post-Scenario SAGAT)

From memory, sketch the interior layout and mark all hazards and victims encountered. (Sketch space and bullets)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

(CL05) Decision Confidence:

Statement	Give a score between: 0 (very low) - 100 (very high)
How confident are you in the decisions you personally made during this mission?
(asked per key decision; e.g. path choice ?? )

Template: Robot operator

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): ~~Baseline~~ / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000).

Before entering, list the hazards or victims you expect to encounter, based on briefing info. (Bullets)

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
The system is deceptive (misleading or deceiving).	☐	☐	☐	☐	☐	☐	☐
The system behaves in an underhanded (shady or dishonest) manner.	☐	☐	☐	☐	☐	☐	☐
I am suspicious of the system's intent, action, or outputs.	☐	☐	☐	☐	☐	☐	☐
I am wary (cautious or careful) of the system.	☐	☐	☐	☐	☐	☐	☐
The system's actions will have a harmful or injurious outcome.	☐	☐	☐	☐	☐	☐	☐
I am confident in the system.	☐	☐	☐	☐	☐	☐	☐
The system provides security.	☐	☐	☐	☐	☐	☐	☐
The system has integrity (honesty or honor).	☐	☐	☐	☐	☐	☐	☐
The system is dependable.	☐	☐	☐	☐	☐	☐	☐
The system is reliable.	☐	☐	☐	☐	☐	☐	☐
I can trust the system.	☐	☐	☐	☐	☐	☐	☐
I am familiar with the system.	☐	☐	☐	☐	☐	☐	☐

(CL02) Situation Awareness Robot Reconnaissance data understanding (OPTIONAL; is more important for the analyst)

After the robot’s initial run but before humans enter, list any hazards, obstacles, or victims that you (as operator) detected or suspect from the robot’s feed. (Open list)

DURING ENTRY PHASE

(CL02/ CL04) Operational notes

Logging of manual control or override the robot's/ system's suggestions (with timestamp) as well as any alerts ignored.

POST EXIT PHASE (hot wash)

(CL01) SafetyTeam Decision Appraisal:

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
How could you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?	☐	☐	☐	☐	☐	☐	☐
Did your team correctly decide when to send the robot versus humans?	☐	☐	☐	☐	☐	☐	☐

(CL01) Safety Error Reflection

List any mistakes or unsafe decisions you think occurred regarding robot deployment. (Open question)

(CL02) Situation Awareness (Post-Scenario SAGAT)

Describe what the robot observed/ found during the mission (e.g. hazards, obstacles) for navigation purposes. (Sketch space and bullets)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

(CL05) Confidence in Robot-Related Decisions:

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
How confident are you that you made the correct decisions in operating and intervening with the robot	☐	☐	☐	☐	☐	☐	☐

Template: Robot analyst
General: Trial type/ Building/ scenario ID: ................................. \| Date/ Time: ....................................... \| Condition (encircle): ~~Baseline~~ / SYNERGISE-tech supported
PRE-ENTRY PHASE (after briefing; before actual start of scenario)
(CL02) Situation Awareness Robot Reconnaissance Assessment (SAGAT expected situation report)	Based on the robot’s pre-entry scan, list the hazards, victims, and notable features you have identified or expect inside. (Bullets and/or sketch)

Template: Entry team member

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

PRE-ENTRY PHASE (after briefing; before actual start of scenario)

(CL02) Situation Awareness (Pre-mission knowledge check):

Before entering, list the hazards or victims you expect to encounter, based on briefing info. (Bullets)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000). IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot analyst's capabilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

DURING ENTRY PHASE

(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)

Any new hazads? (Bullets)

POST EXIT PHASE (hot wash)

(CL01) Safety - Team Decision Appraisal:

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
How would you rate the team's decisions during the mission (including the robot) in terms of safety and effectiveness?	☐	☐	☐	☐	☐	☐	☐
Did your team correctly decide when to send the robot versus humans?	☐	☐	☐	☐	☐	☐	☐

(CL02) Situation Awareness Post Operation Recall

Describe the layout and list all hazards/victims that were found. (Sketch and describe)

(CL03) Workload (NASA-TLX; Hart & Staveland 1988)

Statement	Give a score between: 0 (very low) - 100 (very high)
Mental demand - How much mental and perceptual activity was required - thinking, deciding, calculating, searching? Was the task easy or demanding, simple or complex, exacting or forgiving?
Physical demand - How much physical activity was required - pushing, pulling, turning? Was the task easy or demanding, slow or brisk, restful or laborious?
Temporal demand - How much time pressure did you feel? Was the pace slow and leisurely or rapid and frantic
Effort - How hard did you work mentally and physically to accomplish your level of performance?
Performance - How successful did you feel in accomplishing the goals of the task? How satisfied were you with your performance?
Frustration level - How discouraged, stressed, irritated, and annoyed versus gratified, content, and relaxed did you feel during the task?

(CL04) Trust (Post-Use Trust in Automation scale; Jian et al. 2000) IF SYNERGISE TECH SCENARIO

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

(CL05) Decision Confidence

Statement	Give a score between: 0 (very low) - 100 (very high)
How confident are you in the actions you personally took?

Template: Base of Oparations (LEMA)
Before start scenario
General: Trial type/ Building/ scenario ID: ................................. \| Date/ Time: ....................................... \| Condition (encircle): Baseline / SYNERGISE-tech supported

j. SFT1 evaluation templates

General

Before humans enter a building/ hazard

After humans/robots exited building/hazard

Templates per role

Template: Evaluation leader (Field)

Template: Squad/ Team leader (Field)

Template: Robot operator

Template: Robot analyst

Template: Entry team member

Template: Base of Oparations (LEMA)

Welcome

Navigation

Recently Modified