Project homej. SFT1 evaluation templates
General
 | ||
| Baseline phases | Who fills it | Main purpose |
| Briefing | - | - |
| Estimate building integrity | - | - |
| Shoring | Particpants | (make estimation about shoring time) |
| Before sending in FRs (DM) | Participant | Ground truth/ baseline SA |
| Send in FRs, identify voids, DM | Evaluation leader | Objective performance, safety, timing, decision making |
| FRs exit building | Participant | Subjective SA, trust, workload, decision making |
| Hotwash | ||
 | ||
| SYNERGISE phases | Who fills it | Main purpose |
| Briefing | - | - |
| Estimate building integrity | - | - |
| Send in robots, identify voids with robots | - | - |
| Decision Making (DM) | Participants (Squad/ team leader) | Ground truth/ baseline SA |
| Send in FRs ? | Evaluation leader | Objective performance, safety, timing, decision making |
| After test | Paricipant | Subjective SA, trust, workload, decision making |
| Hotwash | ||
List of ground truths needed for comparison:
- Health ranges
- hazards/ victims in buildings
- Safest path within buildings
- State of automation (tech question)
Before humans enter a building/ hazard
| Claim | Measurement | What is recorded | Who records |
| CL01 Safety | Ground-truth safe paths | predefined safe/ suboptimal / unsafe paths | Evaluation leader |
CL01 Safety | Chosen entry path | Intended entry path selection | Squad/ Team leader (informed by Robot analyst) |
CL02 Situation Awareness | Reported hazards, victims, layout (pre) | Sketch / map / description of interior | Squad/ Team leader, Entry team, Robot analyst (separately) |
CL02 SA | Robot‑based expectations | Expected hazards / victims from robot data | Robot analyst |
CL03–05 Mission effectiveness | Baseline trust (optional) | Initial trust in system/ robot | Squad/ Team leader, Robot operator, Robot analyst |
CL06 Health | Acceptable health ranges | HR, temp thresholds | Evaluation leader |
CL07 Efficiency | Mission start timestamp | End briefing → start | Evaluation leader |
During humans/robots are inside building/ hazard
| Claim | Measurement | What is recorded | Who records |
CL01 Safety | Near‑incidents | Unsafe situations / close calls | Evaluation leader |
CL01 Safety | Hazard avoidance | Avoided predefined hazards | Evaluation leader |
CL02 SA | Executed path | Actual path taken | Evaluation leader, Engage system/ location sensors  |
CL02 SA | Path deviations | Deviations + cause (something for after instead?) | Squad/ Team leader, Entry team |
CL02 SA | Interpretations of robot images | ...? | Robot analyst (and Base or Operations/ LEMA ?) |
CL02 SA (Optional) | Robot control actions | Overrides, manual interventions | Robot operator |
CL06 Health | Threshold breach | Breach moment (HR, temp, stress | Evaluation leader |
CL06 Health | Detection & intervention | Detection with corresponding action | Evaluation leader |
CL07 Efficiency | Entry timestamp | Entry team enters OR robot enters | Evaluation leader |
CL07 Efficiency | Exit timestamp | Entry team exits OR robot exits | Evaluation leader |
After humans/robots exited building/hazard
| Claim | Measurement | What is recorded | Who records |
CL01 Safety | Decision quality (retro) | “How good were decisions?” | Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA |
CL01 Safety | Error reflection | Reflected unsafe choices | Squad/ Team leader, Robot operator, Robot analyst |
CL02 SA | Reported hazards, victims, layout (post) | Sketch / map / description | Squad/ Team leader, Entry team, Robot analyst |
CL03 Effectiveness | Workload | Short NASA‑TLX | Squad/ Team leader, Entry team, Robot operator, Robot analyst, Baes of Operations/ LEMA |
CL04 Effectiveness | Trust | Trust survey / interview | Squad/ Team leader, Entry team , Robot operator, Robot analyst, Baes of Operations/ LEMA |
CL05 Effectiveness | Decision confidence | Confidence in own judgments | Squad/ Team leader, Robot operator, Robot analyst |
CL06 Health | Health issue handling | Adequacy & timeliness | Evaluation leader (+ expert) |
CL07 Efficiency | Total mission duration | Start (end briefing) till end | Evaluation leader |
Templates per role
Template: Base of Operations LEMA (BoO/ HQ) | |
General: Participants roles (encircle): BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team | Trial type/ Building/ scenario: ............................................ | Date/ Time: .......................................... | |
|
|
Template: Evaluation leader (Field) | |
| Before start scenario | |
General: Participants roles (encircle): BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team | Trial type/ Building/ scenario: ............................................ | Date/ Time: .......................................... | |
System & mission readiness (Tick list based on predefined acceptable ranges. Cross out if not applicable.) | ☐ C3I operational, describe: ....................................................................................................................................................................................................................... ☐ Health & location sensors connected, describe: ..................................................................................................................................................................................... ☐ Acceptable/ baseline health ranges defined (HR, temperature, etc.), describe: ..................................................................................................................................... ☐ OWL operational, describe: ..................................................................................................................................................................................................................... ☐ ANYmal operational, describe: ................................................................................................................................................................................................................ ☐ ANYmal with robot arm operational, describe: ........................................................................................................................................................................................ ☐ ANYmal with SNAKE, describe: .............................................................................................................................................................................................................. |
| During start scenario | |
| Let Team leader (and entry team?) sketch/ describe hazard, victims, layout | Notes:
|
Threshold breached, alert and intervention record | Health threshold breached (timestamp + action taken) Alert detection by system (timestamp + action taken) Intervention started (timestamp + action taken) |
| When humans enter a building/ hazard zone | |
Time within building/ hazard zone | Timestamp of entering:..................................................................................... | Timestamp of exiting:........................................................................................................... |
Safety observations | ☐ No unsafe situations observed ......................................................................................................................................................................................................................................................................... |
Path deviations observed (Check if the entry team takes the route that is discussed before entering) | ☐ None ☐ Minor ☐ Major Notes: |
| When humans have exited the building/ hazard zone | |
| Better decision overall | Notes: |
Template: Squad/ Team leader (Field) | |
General: Participants roles (encircle): BoO/ Suad leader / Team leader / Robot operator / Robot analyst / Entry team | Trial type/ Building/ scenario: ............................................ | Date/ Time: .......................................... | |
Pre-entry situation awareness sketch (participant) Sketch or describe hazards, victims, layout as you currently understand them. Predict victims, triage. | (Sketch space)
|
Template: Robot operator (Field)
Template: Robot analyst (Field)
Template: Entry team (Field)