Skip to Content
Last modified by Rosa Van Tuijn on 2026/04/13 12:47
From version 1.1
edited by Tjalling Haije
on 2026/03/23 13:19
Change comment: There is no comment for this version
To version 2.4
edited by Rosa Van Tuijn
on 2026/03/24 16:16
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -h. Test SFT1
1 +h. Test SFT1: System baseline test
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.TjallingHaije
1 +XWiki.RosaVanTuijn
Content
... ... @@ -1,0 +1,259 @@
1 +**Experiment title**
2 +System baseline test
3 +
4 +
5 +**2. Objective / Research Question**
6 +Do the SYNERGISE technologies together improve the SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] (stated below) without negative effects?
7 +
8 +1. Mission effectiveness
9 +1. Mission efficiency
10 +1. FR Health
11 +1. FR and victim safety
12 +
13 +**3. Hypotheses / Expectations (optional)**
14 +See the claims from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]], being:
15 +
16 +* CL01: Improve safety for responders
17 +* CL02: Increased SA
18 +* CL03: Improved mission effectiveness
19 +* CL04: acceptable workload
20 +* CL05: trustworthy SA
21 +* CL06: improved FR health
22 +* CL07: degraded mission efficiency
23 +
24 +**4. Scenario / Context**
25 +ASR3: Detailed indoor exploration
26 +
27 +Preconditions from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] :
28 +
29 +* Structure is flagged as (relatively?) **safe for direct human entry**; sector/ worksite allocated.
30 +* **FRs wearables** distributed, connected, and calibrated.
31 +* Specific to scenario with ANYmal:
32 +** **ANYmal **is mission-ready (battery, controls, and payload checked).
33 +** **C3I is online**, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); **5G pods** are installed for connection.
34 +
35 +**5. Participants**
36 +
37 +* FR team ROBOT:
38 +** 1x Squad leaders
39 +** 1x Robot operator
40 +** 1x Robot analyst
41 +** 2x First responder
42 +* FR team HUMAN:
43 +** 1x Squad leaders
44 +** 2x First reponder
45 +* Base of Operations:
46 +** 1x Safety officer / medical specialist
47 +** Team lead
48 +
49 +Number of participants: 9 
50 +
51 +
52 +**6. System(s) Under Test**
53 +
54 +|(% style="width:267px" %)Technology|(% style="width:1250px" %)Description
55 +|(% style="width:267px" %)ANYmal|(% style="width:1250px" %)(((
56 +* Robust mobility over uneven terrain (stairs, debris, slopes).
57 +* Multi-sensor payload (video, LIDAR, thermal??)
58 +* Reliable wireless communication/ connection in degraded environments.
59 +* Support for tele‑operation with clear operator feedback.
60 +* Link with C3I, such as adding victims as pin on map.
61 +* NO autonomy.
62 +)))
63 +|(% style="width:267px" %)C3I (BoO and tablet version)|(% style="width:1250px" %)(((
64 +* Interoperability with health and location sensors.
65 +* Robust operation under limited connectivity (sync when possible).
66 +* Usable tablet interface for field conditions (gloves, stress, time pressure).
67 +)))
68 +|(% style="width:267px" %)Health sensors|(% style="width:1250px" %)(((
69 +* Accurate and reliable physiological sensing (e.g. heart rate, exertion proxies).
70 +* Comfortable, non‑intrusive form factor suitable for long missions.
71 +* Secure wireless data transmission to C3I systems.
72 +* Real‑time alerting and trend monitoring.
73 +)))
74 +|(% style="width:267px" %)Location sensors|(% style="width:1250px" %)(((
75 +* Indoor and outdoor localisation capability.
76 +* Sufficient accuracy and update rate for safety‑critical use.
77 +* Integration with C3I and mapping systems.
78 +* Robustness to environmental interference (dust, debris, structures, height differences).
79 +)))
80 +
81 +
82 +
83 +**7. Tasks**
84 +
85 +[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
86 +
87 +[[UC04.4: Detailed indoor exploration without ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYmal (USAR).WebHome]] 
88 +
89 +
90 +**8. Experimental Conditions / Variants**
91 +
92 +Two buildings:
93 +
94 +* Building A: Easier location, obstacles and environmental conditions. (clear visibility, ..)
95 +* Building B: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
96 +
97 +Trial 1:
98 +
99 +* FR Team ROBOT does Building A
100 +* FR team HUMAN does Building B
101 +
102 +Trial 2:
103 +
104 +* FR Team ROBOT does Building B
105 +* FR team HUMAN does Building A
106 +
107 +**9. Measurements**
108 +
109 +See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]. Compare outcomes between the two conditions teams if applicable.
110 +
111 +|**Claim: Safety**|**Measurment(s)**|**How to test **
112 +|Safer decision before entry|Safe path chosen|(((
113 +* Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
114 +* Score paths beforehand: safe/ suboptimal/ unsafe
115 +)))
116 +|Better decision overall|"How good were you decisions looking back?"|(((
117 +* Hotwash reflection question (likert + open)
118 +* Observations of errors (e.g. near-incidents, unsafe choices)
119 +)))
120 +|Fewer dangerous situations|Near-incident reports|(((
121 +* Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not
122 +)))
123 +
124 +|**Claim: Efficiency**|**Measurment(s)**|**How to test **
125 +|Faster mission execution|Total mission completion time|(((
126 +* Timestamp of start (after briefing) till all located victims have been extracted
127 +)))
128 +|Faster inside performance|First responder time inside|(((
129 +* Timestamp of entry building till exist building (last located victim is exstracted)
130 +)))
131 +
132 +|**Claim: Situation Awareness**|**Measurment(s)**|**How to test **
133 +|Before entry |(((
134 +Correctness of reported hazards, victims, layout
135 +)))|(((
136 +* Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth
137 +
138 +(((
139 +
140 +)))
141 +)))
142 +|During entry|(((
143 +FR path logs
144 +)))|(((
145 +* Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information
146 +
147 +(((
148 +
149 +)))
150 +)))
151 +|After mission|(((
152 +Correctness of reported hazards, victims, layout
153 +)))|(((
154 +* Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
155 +)))
156 +
157 +|**Claim: Health**|**Measurment(s)**|**How to test **
158 +|(((
159 +Health remains acceptable
160 +)))|(((
161 +
162 +
163 +Health indicators
164 +)))|(((
165 +* Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these parapeter ranges
166 +)))
167 +|(((
168 +Issues handled in time
169 +)))|(((
170 +
171 +
172 +Health issues tackled in acceptable time
173 +)))|(((
174 +* Log when
175 +)))
176 +
177 +
178 +
179 +**10. Procedure (Step-by-Step)**
180 +
181 +1. __Team preparation__:
182 +11. FR team ROBOT is trained in operating the robot, analysing the camera stream, and pinning information on C3I
183 +11. Squad leader and BoO are trained for both teams in interpreting the health and location sensor data.
184 +11. Experiment leader knows where hazards and victims are in both buildings.
185 +11. Anymal robot is tested with network in target buildings for experiment (without FR themselves seeing building!)
186 +1. __Experiment preparation:__
187 +11. Building is prepped with victims, hazards, etc.
188 +1. __Briefing:__
189 +11. Same for both teams for both building types: Building has been inspected from outside and is estimated safe for entry. Situation inside unknown. Victims possibly present. Goal: find, assess, and extract victims while avoiding and mapping hazards.
190 +11. One person is notified that their health sensor will be triggered during the Building B (difficult building) situation.
191 +1. __Execution:__
192 +11. For FR team ROBOT, see: [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
193 +11. For FR team HUMAN, see: [[UC04.4: Detailed indoor exploration without ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYmal (USAR).WebHome]]
194 +11. During execution of both teams, performance will be measured by experiment leader:
195 +111. Mission completion time
196 +111. Health indicators stay within acceptable range
197 +111. Time for tackling any health alerts
198 +111. Track FR path
199 +1. __Hotwash__:
200 +11. Questions (specified per role, couple questions per topic) on:
201 +111. Near-incident report
202 +111. Path inside
203 +111. FR location and status of hazards, victims, and layout
204 +111. Mission effectiveness: e.g. self reported effectiveness of decisions
205 +111. Self-reported workload
206 +111. Trust survey
207 +11. Group discussion with feedback
208 +
209 +**~11. Planning:**
210 +
211 +* Team preparation: Day before. 1 hour.
212 +* Trial 1:
213 +** Setup scenario: Test day 1. 09:00 - 09:30
214 +** Setup tech and perform checks: Test day 1. 09:00 - 09:30
215 +** Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
216 +** Mission execution: Test day 1. 09:30 - 10:30
217 +** Hot wash: Test day 1. 10:30-11:00
218 +** Scenario reset: test day 1. 10:30-11:00
219 +* Teams switch location: Test day 1. 11:00-11:15
220 +* Trial 2: 
221 +** (Setup scenario: Test day 1. 12:30 - 13:00)
222 +** Setup tech and perform checks: Test day 1. 12:30 - 13:00
223 +** Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
224 +** Mission execution: Test day 1. 13:00 - 14:00
225 +** Hot wash: Test day 1. 14:00-14:30
226 +* Cleanup area and store tech. 14:00-14:30
227 +
228 +
229 +
230 +**12. Materials**
231 +
232 +* For FRs:
233 +** 1x Anymal with teleop setup and networking
234 +** 2x Tablet with C3I for squad leader (one spare)
235 +** 1x Base of Operations with computer with C3I
236 +** 4x Health sensors
237 +** 4x Location sensors
238 +** 8x walky talky with two channels (?) (for FR team and BoO).
239 +* For usecase:
240 +** 2x training building (~~100m2?)
241 +** Enviromental challenges: smoke machine, obstacles, etc.
242 +** ~~8x Actors or dummies for victims.
243 +* 8x Clipboards with papers and pens for hotwash and questionnaires
244 +
245 +**13. Dependencies**
246 +
247 +* Availability of materials
248 +* Availability and functioning of ANYmal robot with teleoperation and network in target buildings
249 +* Availability of functioning network between robot, FRs, and BoO.
250 +* Availability of:
251 +** Experiment participants
252 +** People willing to play victim
253 +** Experiment leaders
254 +
255 +**14. Success Criteria**
256 +When mission(s) completed by both teams, and we can compare their performance using the metrics.
257 +
258 +
259 +