Skip to Content

h. Test SFT1: System baseline test

Last modified by Rosa Van Tuijn on 2026/04/13 12:47

Experiment title
System baseline test

2. Objective / Research Question
Do the SYNERGISE technologies together improve the SYNERGISE Objectives (stated below) without negative effects?

  1. Mission effectiveness
  2. Mission efficiency
  3. FR Health
  4. FR and victim safety

3. Hypotheses / Expectations (optional)
See the claims from UC04.3: Detailed indoor exploration with ANYMAL (USAR), being: 

  • CL01: Improve safety for responders
  • CL02: Increased SA
  • CL03: Improved mission effectiveness
  • CL04: acceptable workload
  • CL05: trustworthy SA
  • CL06: improved FR health
  • CL07: degraded mission efficiency

4. Scenario / Context
ASR3: Detailed indoor exploration

Preconditions from UC04.3: Detailed indoor exploration with ANYMAL (USAR) : 

  • Structure is flagged as (relatively?) safe for direct human entry; sector/ worksite allocated.
  • FRs wearables distributed, connected, and calibrated.
  • Specific to scenario with ANYmal:
    • ANYmal is mission-ready (battery, controls, and payload checked).
    • C3I is online, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); 5G pods are installed for connection. 

5. Participants

  • FR team ROBOT:
    • 1x Squad leaders
    • 1x Robot operator 
    • 1x Robot analyst
    • 2x First responder
  • FR team HUMAN:
    • 1x Squad leaders
    • 2x First reponder 
  • Base of Operations:
    • 1x Safety officer / medical specialist
    • Team lead

Number of participants: 9 
 

6. System(s) Under Test

TechnologyDescription
ANYmal
  • Robust mobility over uneven terrain (stairs, debris, slopes). 
  • Multi-sensor payload (video, LIDAR, thermal??)
  • Reliable wireless communication/ connection in degraded environments. 
  • Support for tele‑operation with clear operator feedback.
  • Link with C3I, such as adding victims as pin on map. 
  • NO autonomy.
C3I (BoO and tablet version)
  • Interoperability with health and location sensors. 
  • Robust operation under limited connectivity (sync when possible).
  • Usable tablet interface for field conditions (gloves, stress, time pressure).
Health sensors
  • Accurate and reliable physiological sensing (e.g. heart rate, exertion proxies). 
  • Comfortable, non‑intrusive form factor suitable for long missions.
  • Secure wireless data transmission to C3I systems. 
  • Real‑time alerting and trend monitoring. 
Location sensors
  • Indoor and outdoor localisation capability. 
  • Sufficient accuracy and update rate for safety‑critical use.
  • Integration with C3I and mapping systems. 
  • Robustness to environmental interference (dust, debris, structures, height differences). 

7. Tasks

UC04.3: Detailed indoor exploration with ANYMAL (USAR) 

UC04.4: Detailed indoor exploration without ANYMAL (USAR) 
 

8. Experimental Conditions / Variants

Two buildings: 

  • Building A: Easier location, obstacles and environmental conditions. (clear visibility, ..)
  • Building B: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).

Trial 1: 

  • FR Team ROBOT does Building A
  • FR team HUMAN does Building B

Trial 2: 

  • FR Team ROBOT does Building B
  • FR team HUMAN does Building A

9. Measurements

See the measurements from UC04.3: Detailed indoor exploration with ANYMAL (USAR). Compare outcomes between the two conditions teams if applicable. See j. SFT1 evaluation templates  for more details about the measurements during a test.

Claim: Safety [CL01]Measurment(s)How to test 
Safer decision before entrySafe path chosen
  • Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
  • Score paths beforehand: safe/ suboptimal/ unsafe
Better decision overall"How good were you decisions looking back?"
  • Hotwash reflection question (likert + open) 
  • Observations of errors (e.g. near-incidents, unsafe choices)
Fewer dangerous situationsNear-incident reports
  • Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not 
Claim: Situation Awareness [CL02]Measurment(s)How to test 
Before entry 

Correctness of reported hazards, victims, layout

  • Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth

 

During entry

FR path logs

  • Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information

 

After mission

Correctness of reported hazards, victims, layout

  • Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
Claim: Mission effectiveness [CL03] [CL04] [CL05]Measurment(s)How to test 

Acceptable workload

 

NASA-TLX

  • Shortened post-trial NASA-TLX questionnaire and analyze score (discuss if these with experts to check if these are "acceptable")

 

Appropriate trust

 

Trust survey / interviews

  •  

Decision confidence

 

Retrospective decision quality rating

 
Claim: Health [CL06]Measurment(s)How to test 

Health remains acceptable

 

Health indicators

  • Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these ranges

Issues handled in time

 

Health issues tackled in acceptable time

  • Timestamp when acceptable ranges are breached till moment detection and intervention moment (compare this with predefined response thresholds)
Claim: Mission Efficiency [CL07]Measurment(s)How to test 
Faster mission executionTotal mission completion time
  • Timestamp of start (after briefing) till all located victims have been extracted
Faster inside performanceFirst responder time inside
  • Timestamp of entry building till exist building (last located victim is exstracted)

10. Procedure (Step-by-Step)

  1. Team preparation:
    1. FR team ROBOT is trained in operating the robot, analysing the camera stream, and pinning information on C3I
    2. Squad leader and BoO are trained for both teams in interpreting the health and location sensor data.
    3. Experiment leader knows where hazards and victims are in both buildings. 
    4. Anymal robot is tested with network in target buildings for experiment (without FR themselves seeing building!)
  2. Experiment preparation:
    1. Building is prepped with victims, hazards, etc.
  3. Briefing:
    1. Same for both teams for both building types: Building has been inspected from outside and is estimated safe for entry. Situation inside unknown. Victims possibly present. Goal: find, assess, and extract victims while avoiding and mapping hazards.
    2. One person is notified that their health sensor will be triggered during the Building B (difficult building) situation.
  4. Execution:
    1. For FR team ROBOT, see: UC04.3: Detailed indoor exploration with ANYMAL (USAR) 
    2. For FR team HUMAN, see: UC04.4: Detailed indoor exploration without ANYMAL (USAR) 
    3. During execution of both teams, performance will be measured by experiment leader:
      1. Mission completion time
      2. Health indicators stay within acceptable range
      3. Time for tackling any health alerts
      4. Track FR path
  5. Hotwash:
    1. Questions (specified per role, couple questions per topic) on:
      1. Near-incident report
      2. Path inside
      3. FR location and status of hazards, victims, and layout
      4. Mission effectiveness: e.g. self reported effectiveness of decisions
      5. Self-reported workload
      6. Trust survey
    2. Group discussion with feedback

11. Planning:

  • Team preparation: Day before. 1 hour.
  • Trial 1:
    • Setup scenario: Test day 1. 09:00 - 09:30
    • Setup tech and perform checks: Test day 1. 09:00 - 09:30
    • Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
    • Mission execution: Test day 1. 09:30 - 10:30
    • Hot wash: Test day 1. 10:30-11:00
    • Scenario reset: test day 1. 10:30-11:00
  • Teams switch location: Test day 1. 11:00-11:15
  • Trial 2: 
    • (Setup scenario: Test day 1. 12:30 - 13:00)
    • Setup tech and perform checks: Test day 1. 12:30 - 13:00
    • Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
    • Mission execution: Test day 1. 13:00 - 14:00
    • Hot wash: Test day 1. 14:00-14:30
  • Cleanup area and store tech. 14:00-14:30

12. Materials

  • For FRs:
    • 1x Anymal with teleop setup and networking
    • 2x Tablet with C3I for squad leader (one spare)
    • 1x Base of Operations with computer with C3I 
    • 4x Health sensors
    • 4x Location sensors
    • 8x walky talky with two channels help (for FR team and BoO). 
  • For usecase:
    • 2x training building (~100m2?)
    • Enviromental challenges: smoke machine, obstacles, etc. 
    • ~8x Actors or dummies for victims.
  • 8x Clipboards with papers and pens for hotwash and questionnaires

13. Dependencies

  • Availability of materials
  • Availability and functioning of ANYmal robot with teleoperation and network in target buildings
  • Availability of functioning network between robot, FRs, and BoO.
  • Availability of:
    • Experiment participants
    • People willing to play victim
    • Experiment leaders

14. Success Criteria
When mission(s) completed by both teams, and we can compare their performance using the metrics.