Skip to Content
Last modified by Rosa Van Tuijn on 2026/04/13 12:47
From version 3.1
edited by Rosa Van Tuijn
on 2026/03/24 16:26
Change comment: There is no comment for this version
To version 1.6
edited by Tjalling Haije
on 2026/03/23 13:57
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -h. Test SFT1: System baseline test
1 +h. Test SFT1
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.RosaVanTuijn
1 +XWiki.TjallingHaije
Content
... ... @@ -1,9 +1,9 @@
1 1  **Experiment title**
2 -System baseline test
2 +[Short, descriptive title]
3 3  
4 4  
5 5  **2. Objective / Research Question**
6 -Do the SYNERGISE technologies together improve the SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] (stated below) without negative effects?
6 +Do the SYNERGISE technologies together improve the below SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] without negative effects?
7 7  
8 8  1. Mission effectiveness
9 9  1. Mission efficiency
... ... @@ -10,6 +10,7 @@
10 10  1. FR Health
11 11  1. FR and victim safety
12 12  
13 +
13 13  **3. Hypotheses / Expectations (optional)**
14 14  See the claims from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]], being:
15 15  
... ... @@ -21,25 +21,19 @@
21 21  * CL06: improved FR health
22 22  * CL07: degraded mission efficiency
23 23  
25 +
24 24  **4. Scenario / Context**
25 -ASR3: Detailed indoor exploration
27 +[Short description of the operational or simulated scenario in which the experiment runs]
28 +
26 26  
27 -Preconditions from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] :
28 -
29 -* Structure is flagged as (relatively?) **safe for direct human entry**; sector/ worksite allocated.
30 -* **FRs wearables** distributed, connected, and calibrated.
31 -* Specific to scenario with ANYmal:
32 -** **ANYmal **is mission-ready (battery, controls, and payload checked).
33 -** **C3I is online**, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); **5G pods** are installed for connection.
34 -
35 35  **5. Participants**
36 36  
37 -* FR team ROBOT:
32 +* FR team 1:
38 38  ** 1x Squad leaders
39 39  ** 1x Robot operator
40 40  ** 1x Robot analyst
41 41  ** 2x First responder
42 -* FR team HUMAN:
37 +* FR team 2:
43 43  ** 1x Squad leaders
44 44  ** 2x First reponder
45 45  * Base of Operations:
... ... @@ -51,35 +51,11 @@
51 51  
52 52  **6. System(s) Under Test**
53 53  
54 -|(% style="width:267px" %)Technology|(% style="width:1250px" %)Description
55 -|(% style="width:267px" %)ANYmal|(% style="width:1250px" %)(((
56 -* Robust mobility over uneven terrain (stairs, debris, slopes).
57 -* Multi-sensor payload (video, LIDAR, thermal??)
58 -* Reliable wireless communication/ connection in degraded environments.
59 -* Support for tele‑operation with clear operator feedback.
60 -* Link with C3I, such as adding victims as pin on map.
61 -* NO autonomy.
62 -)))
63 -|(% style="width:267px" %)C3I (BoO and tablet version)|(% style="width:1250px" %)(((
64 -* Interoperability with health and location sensors.
65 -* Robust operation under limited connectivity (sync when possible).
66 -* Usable tablet interface for field conditions (gloves, stress, time pressure).
67 -)))
68 -|(% style="width:267px" %)Health sensors|(% style="width:1250px" %)(((
69 -* Accurate and reliable physiological sensing (e.g. heart rate, exertion proxies).
70 -* Comfortable, non‑intrusive form factor suitable for long missions.
71 -* Secure wireless data transmission to C3I systems.
72 -* Real‑time alerting and trend monitoring.
73 -)))
74 -|(% style="width:267px" %)Location sensors|(% style="width:1250px" %)(((
75 -* Indoor and outdoor localisation capability.
76 -* Sufficient accuracy and update rate for safety‑critical use.
77 -* Integration with C3I and mapping systems.
78 -* Robustness to environmental interference (dust, debris, structures, height differences).
79 -)))
49 +* Hardware / robot / software: [Describe system(s)]
50 +* Version / configuration: [Describe]
51 +* Relevant capabilities and limitations: [Overview]
52 +
80 80  
81 -
82 -
83 83  **7. Tasks**
84 84  
85 85  [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
... ... @@ -88,201 +88,48 @@
88 88  
89 89  
90 90  **8. Experimental Conditions / Variants**
62 +[Condition A vs B, baseline vs intervention, with/without transparency, etc.]
91 91  
92 -Two buildings:
93 93  
94 -* Building A: Easier location, obstacles and environmental conditions. (clear visibility, ..)
95 -* Building B: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
96 -
97 -Trial 1:
98 -
99 -* FR Team ROBOT does Building A
100 -* FR team HUMAN does Building B
101 -
102 -Trial 2:
103 -
104 -* FR Team ROBOT does Building B
105 -* FR team HUMAN does Building A
106 -
107 107  **9. Measurements**
108 108  
109 -See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]. Compare outcomes between the two conditions teams if applicable.
67 +* Objective measures: [E.g., task completion time, errors, navigation metrics]
68 +* Subjective measures: [E.g., trust, workload, clarity, usability questionnaires]
69 +* Logs / sensors: [E.g., robot logs, video, physiological sensors]
110 110  
111 -|**Claim: Safety [CL01]**|**Measurment(s)**|**How to test **
112 -|Safer decision before entry|Safe path chosen|(((
113 -* Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
114 -* Score paths beforehand: safe/ suboptimal/ unsafe
115 -)))
116 -|Better decision overall|"How good were you decisions looking back?"|(((
117 -* Hotwash reflection question (likert + open)
118 -* Observations of errors (e.g. near-incidents, unsafe choices)
119 -)))
120 -|Fewer dangerous situations|Near-incident reports|(((
121 -* Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not
122 -)))
123 123  
124 -|**Claim: Situation Awareness [CL02]**|**Measurment(s)**|**How to test **
125 -|Before entry |(((
126 -Correctness of reported hazards, victims, layout
127 -)))|(((
128 -* Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth
72 +**10. Procedure (Step-by-Step)**
129 129  
130 -(((
131 -
132 -)))
133 -)))
134 -|During entry|(((
135 -FR path logs
136 -)))|(((
137 -* Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information
74 +1. [Describe preparation]
75 +1. [Briefing]
76 +1. [Execution]
77 +1. [Debrief / Hotwash]
138 138  
139 -(((
140 -
141 -)))
142 -)))
143 -|After mission|(((
144 -Correctness of reported hazards, victims, layout
145 -)))|(((
146 -* Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
147 -)))
148 148  
149 -|**Claim: Mission effectiveness [CL03] [CL04]**|**Measurment(s)**|**How to test **
150 -|(((
151 -Acceptable workload
152 -)))|(((
80 +**~11. Materials / Setup Components**
81 +[List equipment, environment layout, props, simulation assets]
153 153  
154 154  
155 -NASA-TLX
156 -)))|(((
157 -* Shortened post-trial NASA-TLX questionnaire and analyze score (discuss if these with experts to check if these are "acceptable")
84 +**12. Safety & Ethical Considerations**
158 158  
159 -(((
160 -
161 -)))
162 -)))
163 -|(((
164 -Appropriate trust
165 -)))|(((
86 +* Risks: [Describe]
87 +* Mitigations: [Describe]
88 +* Informed consent: [Yes/No + method]
89 +* Data handling: [Describe]
166 166  
167 167  
168 -Trust survey / interviews
169 -)))|(((
170 -*
171 -)))
172 -|(((
173 -Decision confidence
174 -)))|(((
92 +**13. Location & Duration**
175 175  
94 +* Location: [Where does it take place?]
95 +* Duration: [Total time per run + preparation]
176 176  
177 -Retrospective decision quality rating
178 -)))|
179 179  
180 -|**Claim: Health [CL06]**|**Measurment(s)**|**How to test **
181 -|(((
182 -Health remains acceptable
183 -)))|(((
98 +**14. Dependencies**
99 +[Other WPs, required assets, system availability, permissions]
184 184  
185 185  
186 -Health indicators
187 -)))|(((
188 -* Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these ranges
189 -)))
190 -|(((
191 -Issues handled in time
192 -)))|(((
102 +**15. Success Criteria**
103 +[How do you know the experiment worked as intended?]
193 193  
194 194  
195 -Health issues tackled in acceptable time
196 -)))|(((
197 -* Timestamp when acceptable ranges are breached till moment detection and intervention moment (compare this with predefined response thresholds)
198 -)))
199 -
200 -|**Claim: Mission Efficiency [CL07]**|**Measurment(s)**|**How to test **
201 -|Faster mission execution|Total mission completion time|(((
202 -* Timestamp of start (after briefing) till all located victims have been extracted
203 -)))
204 -|Faster inside performance|First responder time inside|(((
205 -* Timestamp of entry building till exist building (last located victim is exstracted)
206 -)))
207 -
208 -**10. Procedure (Step-by-Step)**
209 -
210 -1. __Team preparation__:
211 -11. FR team ROBOT is trained in operating the robot, analysing the camera stream, and pinning information on C3I
212 -11. Squad leader and BoO are trained for both teams in interpreting the health and location sensor data.
213 -11. Experiment leader knows where hazards and victims are in both buildings.
214 -11. Anymal robot is tested with network in target buildings for experiment (without FR themselves seeing building!)
215 -1. __Experiment preparation:__
216 -11. Building is prepped with victims, hazards, etc.
217 -1. __Briefing:__
218 -11. Same for both teams for both building types: Building has been inspected from outside and is estimated safe for entry. Situation inside unknown. Victims possibly present. Goal: find, assess, and extract victims while avoiding and mapping hazards.
219 -11. One person is notified that their health sensor will be triggered during the Building B (difficult building) situation.
220 -1. __Execution:__
221 -11. For FR team ROBOT, see: [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
222 -11. For FR team HUMAN, see: [[UC04.4: Detailed indoor exploration without ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYmal (USAR).WebHome]]
223 -11. During execution of both teams, performance will be measured by experiment leader:
224 -111. Mission completion time
225 -111. Health indicators stay within acceptable range
226 -111. Time for tackling any health alerts
227 -111. Track FR path
228 -1. __Hotwash__:
229 -11. Questions (specified per role, couple questions per topic) on:
230 -111. Near-incident report
231 -111. Path inside
232 -111. FR location and status of hazards, victims, and layout
233 -111. Mission effectiveness: e.g. self reported effectiveness of decisions
234 -111. Self-reported workload
235 -111. Trust survey
236 -11. Group discussion with feedback
237 -
238 -**~11. Planning:**
239 -
240 -* Team preparation: Day before. 1 hour.
241 -* Trial 1:
242 -** Setup scenario: Test day 1. 09:00 - 09:30
243 -** Setup tech and perform checks: Test day 1. 09:00 - 09:30
244 -** Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
245 -** Mission execution: Test day 1. 09:30 - 10:30
246 -** Hot wash: Test day 1. 10:30-11:00
247 -** Scenario reset: test day 1. 10:30-11:00
248 -* Teams switch location: Test day 1. 11:00-11:15
249 -* Trial 2: 
250 -** (Setup scenario: Test day 1. 12:30 - 13:00)
251 -** Setup tech and perform checks: Test day 1. 12:30 - 13:00
252 -** Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
253 -** Mission execution: Test day 1. 13:00 - 14:00
254 -** Hot wash: Test day 1. 14:00-14:30
255 -* Cleanup area and store tech. 14:00-14:30
256 -
257 -
258 -
259 -**12. Materials**
260 -
261 -* For FRs:
262 -** 1x Anymal with teleop setup and networking
263 -** 2x Tablet with C3I for squad leader (one spare)
264 -** 1x Base of Operations with computer with C3I
265 -** 4x Health sensors
266 -** 4x Location sensors
267 -** 8x walky talky with two channels (?) (for FR team and BoO).
268 -* For usecase:
269 -** 2x training building (~~100m2?)
270 -** Enviromental challenges: smoke machine, obstacles, etc.
271 -** ~~8x Actors or dummies for victims.
272 -* 8x Clipboards with papers and pens for hotwash and questionnaires
273 -
274 -**13. Dependencies**
275 -
276 -* Availability of materials
277 -* Availability and functioning of ANYmal robot with teleoperation and network in target buildings
278 -* Availability of functioning network between robot, FRs, and BoO.
279 -* Availability of:
280 -** Experiment participants
281 -** People willing to play victim
282 -** Experiment leaders
283 -
284 -**14. Success Criteria**
285 -When mission(s) completed by both teams, and we can compare their performance using the metrics.
286 -
287 -
288 288