Changes for page g. Test CFT4 and the coming IFT's

Last modified by Mark Rinse van Koningsveld on 2026/07/27 10:06

From 6.1 to 5.1 From 11.2 to 11.1

From version 11.1

edited by Mark Rinse van Koningsveld
on 2025/09/13 15:46

Change comment: There is no comment for this version

To version 6.1

edited by Rosa Van Tuijn
on 2025/07/09 10:51

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (3 modified, 0 added, 0 removed)
Objects (0 modified, 0 added, 1 removed)
- XWiki.XWikiComments[0]

Details

Page properties

Title

@@ -1,1 +1,1 @@
--g. Test CFT4 and the coming IFT's
++h. Test CFT4

Author

@@ -1,1 +1,1 @@
--XWiki.MarkVanKoningsveld
++XWiki.RosaVanTuijn

Content

@@ -1,333 +1,64 @@
  = 1. Introduction =
--This experiment validates multiple human-machine teaming technologies in Urban Search and Rescue (USAR) operations. Four operational modules simulate a full operational storyline across two days: wide area assessment, full-area reconnaissance with health monitoring, indoor drone-assisted search, and precision inspection in confined spaces.
++//<include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case>//
--The modules test system functions from five use cases, and aim to quantify effects on safety, situation awareness (SA), physical workload, mission effectiveness, and decision-making quality. Results will be compared against expected performance without these technologies, based on either baseline team data, observer input, or known solution benchmarks.
++Claims uit [[UC01.1: Health and environmental monitoring (Firefighters)>>doc:2\. Specification.b\. Use Cases.UC01\.0\: Health Sensors.Usecase\: Health sensors (Firefighters).WebHome]] die getest worden:
------
++* Weten ze waar tocix gas is?
++* commander weet wat de situatie van zijn personeel is?
++* Reddingsmedewerk heeft SA over hun eigen status (genoeg dat ze optijd kunnen reageren) (d.m.v. communicatie met commander of d.m.v. trillen sensor)?
++* HQ krijgt voldoende (en op het juiste moment) informatie over de situatie in het veld om ondersteuning te kunnen bieden?
--= 2. Method =
++Claims uit UC01.2:Health and environmental monitoring (USAR) die getest worden:
--== 2.1 Participants ==
++* Zelfde als hierboven maar dan iets aangepast voor USAR
--Approximately 24–30 international first responders, organized in teams. Each team rotates across the four modules. Roles include responders, team leaders, drone/robot operators, analysts, medics, and safety officers.
--== 2.2 Experimental Design ==
--A **within-subject design** is used where all teams go through the four modules. Performance is compared across modules and against predefined baseline criteria. Observers collect data in real time; surveys and biometric data are used to validate subjective and objective measurements.
++Claims uit UC02.2: Indoor Drone Exploration and Victim Detection (USAR) die getest worden:
------
++* Weten first responders (genoeg) wat er binnen is om veilig naar binnen te gaan? hebben ze verhoogde SA van de binnenkant van een gebouw? SA/reliance
++* Kunnen er beter en sneller victims worden gevonden? > speed, task performance
++* Kunnen er meer betrouwbare analyses worden gemaakt van de binnenkant van een gebouw door bijv. een plan maken voor een veilige/ begaanbare route > SA
++* Task performance: kunnen er sneller en meer gestroomlijnd victim reports worden gemaakt en gedeeld (essentie = gaat victim assessement beter)?
--== 2.3 Tasks (Per Module) ==
------
--=== **Module 1 – Wide Area Assessment** ===
++Claims uit UC02.1: Indoor Drone Exploration and Victim Detection (Firefighters) die getest worden:
--**Use Case**: UC03.0
--**Scenario**: Teams arrive at a simulated disaster zone. Structures are unstable. Drone support is requested for external mapping and hazard detection.
++* zelfde als hierboven maar dan iets meer aangepast voor Firefighters
--**Tested Functions**:
--* Drone feed provides real-time visuals to field teams and command
--* Zoom-ins allow inspection of rooftops and entry points
--* Footage used to mark safe approach routes
--**Measured Claims**:
--* CL1: Improved external SA
--* CL2: Safer movement planning
--* CL3: Faster planning cycle
--* CL4: Reduced mental workload for recon
--* CL5: Improved coordination (shared SA)
++= 2. Method =
--**Quantifiable Success Factors**:
--* ≥80% of hazards correctly marked on the map (based on preset dummy hazards)
--* ≥90% agreement in SA between team and command (map match)
--* Average planning time ≤ 10 minutes from drone launch
--* NASA-TLX workload score ≤ 50 (moderate) for command roles
++== 2.1 Participants ==
--**How to Measure**:
--* Observer logs & stopwatch for planning time
--* Map test: Compare team-drawn vs. actual map (SAGAT-lite)
--* Count number of correctly identified hazards from drone feed
--* NASA-TLX filled by drone operator and team lead
--* Post-module survey: "How useful was the drone in forming your plan?" (1–5)
++== 2.2 Experimental design ==
------
--=== **Module 2 – Health Monitoring & Reconnaissance** ===
++== 2.3 Tasks ==
--**Use Cases**: UC01.1 (Fire) and UC01.2 (USAR)
--**Scenario**: Team performs full-area recon. Wearables measure heart rate, hydration, and simulated gas exposure. Simulated fatigue and alerts escalate to medics or team leads.
--**Tested Functions**:
--
--* Alerts for fatigue/gas exposure
--* Remote dashboard monitoring by safety officer
--* Escalation protocols for health interventions
--* Logging and after-action review
--
--**Measured Claims**:
--
--* CL1–CL2: Prevent overexertion and increase responder awareness
--* CL3–CL4: Enable remote intervention and informed medical decision
--* CL5: Enable better rotation/rest planning
--* CL6: Debrief uses health logs
--* CL7: Improve mission success
--
--**Quantifiable Success Factors**:
--
--* ≥90% of health alerts acknowledged within 1 minute
--* ≥80% of interventions judged "timely" in AAR interviews
--* ≥50% of teams adjust tactics or rest cycles based on health data
--* ≥1 health-based lesson identified per team in debrief
--* ≤2 simulated incidents due to unmanaged fatigue/gas exposure
--
--**How to Measure**:
--
--* Log alert timings vs. response time
--* Observer notes + medic reports on intervention
--* Exit survey: "Did alerts help prevent fatigue/injury?"
--* Use of wearable dashboard during debrief (Yes/No)
--* NASA-TLX for responders
--
------
--
--=== **Module 3 – Indoor Drone Search (Barracks)** ===
--
--**Use Cases**: UC02.1 and UC02.2
--**Scenario**: Collapsed barracks building. Indoor drone used for autonomous scan. Analyst tags victims, hazards, and updates C3I map. Drone does close inspection on request.
--
--**Tested Functions**:
--
--* Pre-entry thermal scan
--* Hazard/victim detection
--* Analyst-supported interpretation and tagging
--* Entry planning based on drone data
--
--**Measured Claims**:
--
--* CL1: Heightened SA before entry
--* CL2: Increased safety (less exposure)
--* CL3–CL5: Faster, more accurate victim detection
--* CL6: Trust in drone data
--* CL7: Increased mission efficiency
--
--**Quantifiable Success Factors**:
--
--* ≥90% of dummy victims detected by drone+analyst
--* ≥2 new hazards marked per team from drone feed
--* Average time-to-first victim ≤ 3 minutes
--* ≥80% of responders rate drone info as “trustworthy” (score ≥4/5)
--* ≤1 injury due to unknown hazard in follow-up entry
--
--**How to Measure**:
--
--* Victim tags placed in known positions for ground truth
--* Observer logs: detection times and analyst confirmations
--* Team SA quiz: "How many victims? Where were they located?"
--* Trust survey: “I would act on this drone data” (1–5)
--* Entry path compared to drone hazard map
--
------
--
--=== **Module 4 – Precision Inspection with ANYMAL/SNAKE** ===
--
--**Use Case**: UC04.0
--**Scenario**: Teams reach unstable voids. Robots are deployed to inspect inaccessible areas. SNAKE arm is used to look into cracks. Results update team maps and entry plans.
--
--**Tested Functions**:
--
--* Autonomous or manual ANYMAL movement
--* Void inspection using flexible arm
--* Victim/hazard confirmation
--* Decision-making based on robot visuals
--
--**Measured Claims**:
--
--* CL1: Access without risk
--* CL2: Detection in confined space
--* CL3: Safer routing
--* CL4: Trust in robot-assessed visuals
--* CL5: Faster room clearing
--
--**Quantifiable Success Factors**:
--
--* ≥2 hazards or victims confirmed via SNAKE per team
--* ≥80% of voids scanned without human entry
--* ≥70% of teams adjust route based on robot findings
--* ≥80% of participants rate robot visuals as “clear and usable”
--* Average inspection time ≤ 8 minutes per room
--
--**How to Measure**:
--
--* Observer log: robot path vs. human path
--* Detection log compared to known hidden items
--* Survey: “Did robot findings improve your plan?” (Yes/No)
--* Video review of time-per-room
--* Trust in visuals scale (1–5)
--
--
--
  == 2.4 Measures ==
--This section describes how each claim will be measured during each module, using a combination of objective logging, observer annotations, post-task surveys, and scenario-based evaluation.
------
--
--=== **Module 1 – Wide Area Assessment (UC03.0)** ===
--
--|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
--|CL1 – Improved SA|Number of hazards correctly identified on team maps|SAGAT-lite: Pre/post map-drawing task + verbal hazard recall|≥80% match with ground-truth hazard list
--|CL2 – Safer planning|Number of hazard zones avoided during later entry|Observer logs cross-referenced with hazard map|100% of marked hazards avoided
--|CL3 – Faster planning|Time from drone launch to team briefing|Stopwatch & observer notes|≤10 minutes total
--|CL4 – Reduced workload|Mental workload score of command & drone operator|NASA-TLX (short form)|≤50 average score
--|CL5 – Shared SA|Consistency between team and command in map data|Comparison of annotations across roles|≥90% agreement on key features
--
--
--
------
--
--=== **Module 2 – Health Monitoring & Reconnaissance (UC01.1 / UC01.2)** ===
--
--|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
--|CL1 – Prevent overload|HR trend + alert timing vs. pause/extraction|Wearable logs + observer notes|≥90% alerts followed by correct action within 1 minute
--|CL2 – Responder awareness|Survey response on self-adjustment|Post-task Likert: “The alert helped me act”|≥80% rate 4 or 5
--|CL3 – Remote escalation|Alert-to-medic contact time|System log + stopwatch|≤1 minute average
--|CL4 – Medical support|Alignment of alerts with medical assessment|Medic forms + sensor log correlation|≥80% concordance
--|CL5 – Operational planning|Number of rest/rotation decisions based on dashboard|Observer logs + team lead AAR|≥50% of teams adapt plan
--|CL6 – AAR use of health data|Was biometric data used during debrief?|Debrief analysis|Yes, per team
--|CL7 – Mission effectiveness|Task time + incidents avoided|Stopwatch + incident log|Task time not slower than baseline; 0 uncontrolled fatigue/gas incidents
--
--
--
------
--
--=== **Module 3 – Indoor Drone Search (UC02.1 / UC02.2)** ===
--
--|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
--|CL1 – Heightened SA|SA questionnaire + map task|Pre/post: victims, layout, hazard count|≥80% correct recall post-drone
--|CL2 – Increased safety|Hazard zone avoidance rate|Observer vs. ground truth map|≥90% of flagged areas avoided
--|CL3 – Faster victim detection|Time to first detection|Stopwatch from drone entry|≤3 minutes
--|CL4 – Accuracy of detection|Victim detection rate|Drone log vs. planted victims|≥90% detected
--|CL5 – Trust in results|Survey: “I trust the drone data for decision-making”|1–5 Likert scale|≥80% rate 4 or 5
--|CL6 – Efficiency|Entry time after drone plan vs. without drone|Stopwatch; compare with baseline data|10–20% faster planning phase
--
--
--
------
--
--=== **Module 4 – Robot-Based Precision Inspection (UC04.0)** ===
--
--|=**Claim**|=**Metric**|=**Method/Tool**|=**Success Threshold**
--|CL1 – Extended reach|Percentage of voids explored by robot not human|Observer log + inspection plan|≥80% of voids scanned by robot
--|CL2 – Detection in small spaces|Victim/hazard detection in hidden locations|Camera log vs. planted markers|≥2 findings per team
--|CL3 – Safer routing|Route changes based on robot input|Pre/post plan comparison + observer notes|≥70% of teams adapt plan
--|CL4 – Trust in visuals|Survey on clarity and trust in robot data|Likert: “The robot data was sufficient for decisions”|≥80% rate 4 or 5
--|CL5 – Room clearing speed|Time per room before vs. after robot scout|Stopwatch log|≤8 minutes per room avg.
--
--
--
------
--
  == 2.5 Procedure ==
--All modules follow a similar four-part procedure, tailored per use case.
++Ttijdens debriefing vragen of ze bepaalde dingen hebben gemerkt
--=== **General Daily Timeline** ===
--
--* **08:30 – 09:00**: Morning briefing, safety, tech setup
--* **09:00 – 12:00**: First module rotation (two parallel teams)
--* **13:00 – 16:00**: Second module rotation (two parallel teams)
--* **16:00 – 17:00**: Shared after-action review
--
--Each module runs with the following structure:
--
--=== **Per Module Procedure** ===
--
--1. (((
--**Briefing (10–15 min)**
--
--* Explain objectives, scenario, roles, safety, success factors
--* Introduce technology and expectations
--)))
--1. (((
--**Execution Phase (45–60 min)**
--
--* Scenario runs in real time
--* Observer logs events, actions, communications
--* System logs recorded (drone, robot, wearables)
--)))
--1. (((
--**Measurement Phase (15–20 min)**
--
--* Paper or tablet surveys: SA, trust, NASA-TLX
--* Sensor data downloaded to central system
--* Short interview or checklist with operator and team lead
--)))
--1. (((
--**Debrief (15–20 min)**
--
--* Team reflects on use of technology, decision-making
--* Facilitator prompts discussion of claims (trust, effectiveness, awareness)
--* Recorded notes for final reporting
--)))
--
--For cross-checking performance without the tech, one team per module may be assigned a simplified "control" version of the scenario, using conventional tools only (where feasible).
--
------
--
  == 2.6 Material ==
--Each module requires scenario-specific equipment, environmental props, logging tools, and survey forms:
--=== **Common Materials (all modules)** ===
--
--* Observer logbooks (standardized per module)
--* Stopwatch or time-tracking app
--* Participant role badges and checklists
--* Data collection station with tablets/laptops
--* Printed Likert-scale surveys (SA, trust, workload)
--* SAGAT-lite map templates
--
--=== **Module-Specific Materials** ===
--
--**Module 1 – Wide Area Assessment**
--
--* Outdoor drones with RTK GPS and live zoom cameras
--* Command screen with drone feed
--* Large printed site maps with hazard zones (for scoring)
--* Structural hazard props (collapsed façades, signs)
--
--**Module 2 – Health Monitoring**
--
--* Wearable sensors (HR, hydration, gas; real or simulated)
--* Dashboard software for live feed + logging
--* Incident trigger devices (e.g., CO2 canisters, alarms)
--* Medic checklist sheets
--* Alert simulation software (optional)
--
--**Module 3 – Indoor Drone Search**
--
--* Thermal indoor drone with autonomous mode
--* C3I-compatible map annotation system
--* Dummy victims with heat packs or QR markers
--* Printed room layouts for SA testing
--* Indoor hazard props (rubble, fake smoke, blocked doors)
--
--**Module 4 – ANYMAL and SNAKE**
--
--* ANYMAL robot (legged) and SNAKE articulated arm
--* Confined space mockups (voids, crawlspaces, stairs)
--* Hidden hazard/victim tags inside small cavities
--* Robot operator station + external monitor
--* Scenario map with route overlays
--
  = 3. Results =
++
  = 4. Discussion =
++
  = 5. Conclusions =
++
++

XWiki.XWikiComments[0]

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.MarkVanKoningsveld

Comment

@@ -1,25 +1,0 @@
--= 1. Introduction[[Edit>>url:https://wisce.synergise-project.eu/bin/edit/3.%20Evaluation/h.%20Test/WebHome?section=1&form_token=ThxE4oHo2FgcrNOPAkA8sw]] =
--
--//<include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case>//
--
--Claims uit [[UC01.1: Health and environmental monitoring (Firefighters)>>url:https://wisce.synergise-project.eu/bin/view/2.%20Specification/b.%20Use%20Cases/UC01.0%3A%20Health%20Sensors/Usecase%3A%20Health%20sensors%20%28Firefighters%29/]] die getest worden:
--
--* Weten ze waar tocix gas is?
--* commander weet wat de situatie van zijn personeel is?
--* Reddingsmedewerk heeft SA over hun eigen status (genoeg dat ze optijd kunnen reageren) (d.m.v. communicatie met commander of d.m.v. trillen sensor)?
--* HQ krijgt voldoende (en op het juiste moment) informatie over de situatie in het veld om ondersteuning te kunnen bieden?
--
--Claims uit UC01.2:Health and environmental monitoring (USAR) die getest worden:
--
--* Zelfde als hierboven maar dan iets aangepast voor USAR
--
--Claims uit UC02.2: Indoor Drone Exploration and Victim Detection (USAR) die getest worden:
--
--* Weten first responders (genoeg) wat er binnen is om veilig naar binnen te gaan? hebben ze verhoogde SA van de binnenkant van een gebouw? SA/reliance
--* Kunnen er beter en sneller victims worden gevonden? > speed, task performance
--* Kunnen er meer betrouwbare analyses worden gemaakt van de binnenkant van een gebouw door bijv. een plan maken voor een veilige/ begaanbare route > SA
--* Task performance: kunnen er sneller en meer gestroomlijnd victim reports worden gemaakt en gedeeld (essentie = gaat victim assessement beter)?
--
--Claims uit UC02.1: Indoor Drone Exploration and Victim Detection (Firefighters) die getest worden:
--
--* zelfde als hierboven maar dan iets meer aangepast voor Firefighters

Date

...	...	@@ -1,1 +1,0 @@
1		-2025-07-13 22:23:32.283

Changes for page g. Test CFT4 and the coming IFT's

Summary

Details

Welcome

Navigation

Recently Modified