j. SFT1 evaluation templates

Version 16.3 by Rosa Van Tuijn on 2026/04/20 12:53

General


Baseline phases	Who fills it	Main purpose
Briefing	-	-
Estimate building integrity	-	-
Shoring	Particpants	(make estimation about shoring time)
Before sending in FRs (DM)	Participant	Ground truth/ baseline SA
Send in FRs, identify voids, DM	Evaluation leader	Objective performance, safety, timing, decision making
FRs exit building	Participant	Subjective SA, trust, workload, decision making
Hotwash


SYNERGISE phases	Who fills it	Main purpose
Briefing	-	-
Estimate building integrity	-	-
Send in robots, identify voids with robots	-	-
Decision Making (DM)	Participants (Squad/ team leader)	Ground truth/ baseline SA
Send in FRs ?	Evaluation leader	Objective performance, safety, timing, decision making
After test	Paricipant	Subjective SA, trust, workload, decision making
Hotwash

List of ground truths needed for comparison:

Health ranges
hazards/ victims in buildings
Safest path within buildings
State of automation (tech question)

Before humans enter a building/ hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Ground-truth safe paths	predefined safe/ suboptimal / unsafe paths	Evaluation leader
CL01 Safety	Chosen entry path	Intended entry path selection	Squad/ Team leader (informed by Robot analyst)
CL02 Situation Awareness	Reported hazards, victims, layout (pre)	Sketch / map / description of interior	Squad/ Team leader, Entry team, Robot analyst (separately)
CL02 SA	Robot‑based expectations	Expected hazards / victims from robot data	Robot analyst
CL03–05 Mission effectiveness	Baseline trust (optional)	Initial trust in system/ robot	Squad/ Team leader, Robot operator, Robot analyst
CL06 Health	Acceptable health ranges	HR, temp thresholds	Evaluation leader
CL07 Efficiency	Mission start timestamp	End briefing → start	Evaluation leader

During humans/robots are inside building/ hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Near‑incidents	Unsafe situations / close calls	Evaluation leader
CL01 Safety	Hazard avoidance	Avoided predefined hazards	Evaluation leader
CL02 SA	Executed path	Actual path taken	Evaluation leader, Engage system/ location sensors
CL02 SA	Path deviations	Deviations + cause (something for after instead?)	Squad/ Team leader, Entry team
CL02 SA	Interpretations of robot images	...?	Robot analyst (and Base or Operations/ LEMA ?)
CL02 SA (Optional)	Robot control actions	Overrides, manual interventions	Robot operator
CL06 Health	Threshold breach	Breach moment (HR, temp, stress )	Evaluation leader
CL06 Health	Detection & intervention	Detection with corresponding action	Evaluation leader
CL07 Efficiency	Entry timestamp	Entry team enters OR robot enters	Evaluation leader
CL07 Efficiency	Exit timestamp	Entry team exits OR robot exits	Evaluation leader

After humans/robots exited building/hazard

Claim	Measurement	What is recorded	Who records
CL01 Safety	Decision quality (retro)	“How good were decisions?”	Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA
CL01 Safety	Error reflection	Reflected unsafe choices	Squad/ Team leader, Robot operator, Robot analyst
CL02 SA	Reported hazards, victims, layout (post)	Sketch / map / description	Squad/ Team leader, Entry team, Robot analyst
CL03 Effectiveness	Workload	Short NASA‑TLX	Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA
CL04 Effectiveness	Trust	Trust survey / interview	Squad/ Team leader, Entry team , Robot operator, Robot analyst, Baes of Operations/ LEMA
CL05 Effectiveness	Decision confidence	Confidence in own judgments	Squad/ Team leader, Robot operator, Robot analyst
CL06 Health	Health issue handling	Adequacy & timeliness	Evaluation leader (+ expert)
CL07 Efficiency	Total mission duration	Start (end briefing) till end	Evaluation leader

Templates per role

Template: Base of Operations LEMA (BoO/ HQ)
General: Participants roles (encircle): BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team \| Trial type/ Building/ scenario: ............................................ \| Date/ Time: ..........................................

Template: Evaluation leader (Field)
Before start scenario
General: Participants roles (encircle): BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team \| Trial type/ Building/ scenario: ............................................ \| Date/ Time: ..........................................
System & mission readiness (Tick list based on predefined acceptable ranges. Cross out if not applicable.)	☐ C3I operational, describe: ....................................................................................................................................................................................................................... ☐ Health & location sensors connected, describe: ..................................................................................................................................................................................... ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: ..................................................................................................................................... ☐ OWL operational, describe: ..................................................................................................................................................................................................................... ☐ ANYmal operational, describe: ................................................................................................................................................................................................................ ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................ ☐ ANYmal with SNAKE, describe: ..............................................................................................................................................................................................................
During start scenario
Let Team leader (and entry team?) sketch/ describe hazard, victims, layout	Notes:
Threshold breached, alert and intervention record	Health threshold breached (timestamp + action taken) Alert detection by system (timestamp + action taken) Intervention started (timestamp + action taken)
When humans enter a building/ hazard zone
Time within building/ hazard zone	Timestamp of entering:..................................................................................... \| Timestamp of exiting:...........................................................................................................
Safety observations	☐ No unsafe situations observed ☐ Unsafe / near‑incident observed, describe (amount and type) :.................................................................................................................................................................. .........................................................................................................................................................................................................................................................................
Path deviations observed (Check if the entry team takes the route that is discussed before entering)	☐ None ☐ Minor ☐ Major Notes:

When humans have exited the building/ hazard zone
Better decision overall	Notes:

Template: Squad/ Team leader (Field) - WIHT SYNERGISE TECH.

General:

Trial type/ Building/ scenario ID: ................................. | Date/ Time: ....................................... | Condition (encircle): Baseline / SYNERGISE-tech supported

PRE-ENTRY PHASE

(CL01) Mark chosen entry path on provided building map . Sketch or describe entry path.

(Sketch space)

(CL02) List expected hazards (SAGAT; Endsley 1988):

(CL02) List expected victims (SAGAT; Endsley 1988):

(CL02) Sketch/ Describe expected interior layout (SAGAT; Endsley 1988):

(Sketch space)

(CL04) Trust in automation (Baseline Scale of Trust in Automated Systems; Jian et al. 2000)

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
I am confident in the robot/system’s abilities.	☐	☐	☐	☐	☐	☐	☐
The robot/system will act in our best interest.	☐	☐	☐	☐	☐	☐	☐
The robot/system is reliable.	☐	☐	☐	☐	☐	☐	☐
I trust the system’s information.	☐	☐	☐	☐	☐	☐	☐

DURING ENTRY PHASE

(CL02 optional; if pause is feasable) Situation Awareness (mid-mission SAGAT probe; Endsley 1988)

Where are you team members now and what hazards are near them? (Sketch space)

POST EXIT PHASE

(CL01) Safety Decision Quality Rating (AAR-based; U.S. Army, 1993):

Statement	1 (LOW)	2	3	4	5	6	7 (HIGH)
had a clear understanding of the mission objectives, my role, and the intended plan before execution.	☐	☐	☐	☐	☐	☐	☐
During the mission, I had sufficient situational awareness to understand what was happening and how the situation was evolving.	☐	☐	☐	☐	☐	☐	☐
The decisions made (by myself and/or the team) during the mission were timely and appropriate given the situation.	☐	☐	☐	☐	☐	☐	☐
Team coordination and communication were effective and supported successful task execution.	☐	☐	☐	☐	☐	☐	☐
The tools, systems, or support aids available during the mission effectively supported mission execution.	☐	☐	☐	☐	☐	☐	☐
Based on this exercise, the team is better prepared to perform similar missions in the future.	☐	☐	☐	☐	☐	☐	☐

OR (open questions better?)

Open questions

What was the plan/intent for the mission, and what did you personally aim to achieve? (1–2 sentences)

What actually happened? List the 2–3 most important events or turning points. (bullets)

What went well and should be sustained? Give one concrete example. (example + why it worked)

What did not go as intended? Give one concrete example and its impact. (example + consequence)

Why did it happen? What were the main contributing factors? (e.g., information/SA, communication, coordination, timing, tools/technology, environment)

What are the top 2 actionable improvements for next time (specific, doable)? (Action + owner/role + when)

j. SFT1 evaluation templates

General

Before humans enter a building/ hazard

After humans/robots exited building/hazard

Templates per role

Template: Base of Operations LEMA (BoO/ HQ)

Template: Evaluation leader (Field)

Template: Squad/ Team leader (Field) - WIHT SYNERGISE TECH.

Template: Robot operator (Field)

Template: Robot analyst (Field)

Template: Entry team (Field)

Welcome

Navigation

Recently Modified