Skip to Content
Last modified by Rosa Van Tuijn on 2026/04/13 12:47
From version 2.4
edited by Rosa Van Tuijn
on 2026/03/24 16:16
Change comment: There is no comment for this version
To version 1.7
edited by Tjalling Haije
on 2026/03/23 14:15
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -h. Test SFT1: System baseline test
1 +h. Test SFT1
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.RosaVanTuijn
1 +XWiki.TjallingHaije
Content
... ... @@ -1,9 +1,9 @@
1 1  **Experiment title**
2 -System baseline test
2 +[Short, descriptive title]
3 3  
4 4  
5 5  **2. Objective / Research Question**
6 -Do the SYNERGISE technologies together improve the SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] (stated below) without negative effects?
6 +Do the SYNERGISE technologies together improve the below SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] without negative effects?
7 7  
8 8  1. Mission effectiveness
9 9  1. Mission efficiency
... ... @@ -10,6 +10,7 @@
10 10  1. FR Health
11 11  1. FR and victim safety
12 12  
13 +
13 13  **3. Hypotheses / Expectations (optional)**
14 14  See the claims from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]], being:
15 15  
... ... @@ -21,6 +21,7 @@
21 21  * CL06: improved FR health
22 22  * CL07: degraded mission efficiency
23 23  
25 +
24 24  **4. Scenario / Context**
25 25  ASR3: Detailed indoor exploration
26 26  
... ... @@ -32,6 +32,7 @@
32 32  ** **ANYmal **is mission-ready (battery, controls, and payload checked).
33 33  ** **C3I is online**, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); **5G pods** are installed for connection.
34 34  
37 +
35 35  **5. Participants**
36 36  
37 37  * FR team ROBOT:
... ... @@ -91,169 +91,59 @@
91 91  
92 92  Two buildings:
93 93  
94 -* Building A: Easier location, obstacles and environmental conditions. (clear visibility, ..)
95 -* Building B: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
97 +* Building A: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
98 +* Building B: Easier location, obstacles and environmental conditions. (clear visibility, ..)
96 96  
97 -Trial 1:
98 98  
99 -* FR Team ROBOT does Building A
100 -* FR team HUMAN does Building B
101 101  
102 -Trial 2:
102 +Trial1:
103 103  
104 -* FR Team ROBOT does Building B
105 -* FR team HUMAN does Building A
104 +* FR Team ROBOT does easy
106 106  
107 -**9. Measurements**
106 +Team A does first [[UC04.3>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
108 108  
109 -See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]. Compare outcomes between the two conditions teams if applicable.
108 +[Condition A vs B, baseline vs intervention, with/without transparency, etc.]
110 110  
111 -|**Claim: Safety**|**Measurment(s)**|**How to test **
112 -|Safer decision before entry|Safe path chosen|(((
113 -* Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
114 -* Score paths beforehand: safe/ suboptimal/ unsafe
115 -)))
116 -|Better decision overall|"How good were you decisions looking back?"|(((
117 -* Hotwash reflection question (likert + open)
118 -* Observations of errors (e.g. near-incidents, unsafe choices)
119 -)))
120 -|Fewer dangerous situations|Near-incident reports|(((
121 -* Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not
122 -)))
123 123  
124 -|**Claim: Efficiency**|**Measurment(s)**|**How to test **
125 -|Faster mission execution|Total mission completion time|(((
126 -* Timestamp of start (after briefing) till all located victims have been extracted
127 -)))
128 -|Faster inside performance|First responder time inside|(((
129 -* Timestamp of entry building till exist building (last located victim is exstracted)
130 -)))
111 +**9. Measurements**
131 131  
132 -|**Claim: Situation Awareness**|**Measurment(s)**|**How to test **
133 -|Before entry |(((
134 -Correctness of reported hazards, victims, layout
135 -)))|(((
136 -* Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth
113 +* Objective measures: [E.g., task completion time, errors, navigation metrics]
114 +* Subjective measures: [E.g., trust, workload, clarity, usability questionnaires]
115 +* Logs / sensors: [E.g., robot logs, video, physiological sensors]
137 137  
138 -(((
139 -
140 -)))
141 -)))
142 -|During entry|(((
143 -FR path logs
144 -)))|(((
145 -* Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information
146 146  
147 -(((
148 -
149 -)))
150 -)))
151 -|After mission|(((
152 -Correctness of reported hazards, victims, layout
153 -)))|(((
154 -* Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
155 -)))
118 +**10. Procedure (Step-by-Step)**
156 156  
157 -|**Claim: Health**|**Measurment(s)**|**How to test **
158 -|(((
159 -Health remains acceptable
160 -)))|(((
120 +1. [Describe preparation]
121 +1. [Briefing]
122 +1. [Execution]
123 +1. [Debrief / Hotwash]
161 161  
162 162  
163 -Health indicators
164 -)))|(((
165 -* Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these parapeter ranges
166 -)))
167 -|(((
168 -Issues handled in time
169 -)))|(((
126 +**~11. Materials / Setup Components**
127 +[List equipment, environment layout, props, simulation assets]
170 170  
171 171  
172 -Health issues tackled in acceptable time
173 -)))|(((
174 -* Log when
175 -)))
130 +**12. Safety & Ethical Considerations**
176 176  
132 +* Risks: [Describe]
133 +* Mitigations: [Describe]
134 +* Informed consent: [Yes/No + method]
135 +* Data handling: [Describe]
177 177  
178 178  
179 -**10. Procedure (Step-by-Step)**
138 +**13. Location & Duration**
180 180  
181 -1. __Team preparation__:
182 -11. FR team ROBOT is trained in operating the robot, analysing the camera stream, and pinning information on C3I
183 -11. Squad leader and BoO are trained for both teams in interpreting the health and location sensor data.
184 -11. Experiment leader knows where hazards and victims are in both buildings.
185 -11. Anymal robot is tested with network in target buildings for experiment (without FR themselves seeing building!)
186 -1. __Experiment preparation:__
187 -11. Building is prepped with victims, hazards, etc.
188 -1. __Briefing:__
189 -11. Same for both teams for both building types: Building has been inspected from outside and is estimated safe for entry. Situation inside unknown. Victims possibly present. Goal: find, assess, and extract victims while avoiding and mapping hazards.
190 -11. One person is notified that their health sensor will be triggered during the Building B (difficult building) situation.
191 -1. __Execution:__
192 -11. For FR team ROBOT, see: [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
193 -11. For FR team HUMAN, see: [[UC04.4: Detailed indoor exploration without ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYmal (USAR).WebHome]]
194 -11. During execution of both teams, performance will be measured by experiment leader:
195 -111. Mission completion time
196 -111. Health indicators stay within acceptable range
197 -111. Time for tackling any health alerts
198 -111. Track FR path
199 -1. __Hotwash__:
200 -11. Questions (specified per role, couple questions per topic) on:
201 -111. Near-incident report
202 -111. Path inside
203 -111. FR location and status of hazards, victims, and layout
204 -111. Mission effectiveness: e.g. self reported effectiveness of decisions
205 -111. Self-reported workload
206 -111. Trust survey
207 -11. Group discussion with feedback
140 +* Location: [Where does it take place?]
141 +* Duration: [Total time per run + preparation]
208 208  
209 -**~11. Planning:**
210 210  
211 -* Team preparation: Day before. 1 hour.
212 -* Trial 1:
213 -** Setup scenario: Test day 1. 09:00 - 09:30
214 -** Setup tech and perform checks: Test day 1. 09:00 - 09:30
215 -** Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
216 -** Mission execution: Test day 1. 09:30 - 10:30
217 -** Hot wash: Test day 1. 10:30-11:00
218 -** Scenario reset: test day 1. 10:30-11:00
219 -* Teams switch location: Test day 1. 11:00-11:15
220 -* Trial 2: 
221 -** (Setup scenario: Test day 1. 12:30 - 13:00)
222 -** Setup tech and perform checks: Test day 1. 12:30 - 13:00
223 -** Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
224 -** Mission execution: Test day 1. 13:00 - 14:00
225 -** Hot wash: Test day 1. 14:00-14:30
226 -* Cleanup area and store tech. 14:00-14:30
144 +**14. Dependencies**
145 +[Other WPs, required assets, system availability, permissions]
227 227  
228 228  
148 +**15. Success Criteria**
149 +[How do you know the experiment worked as intended?]
229 229  
230 -**12. Materials**
231 231  
232 -* For FRs:
233 -** 1x Anymal with teleop setup and networking
234 -** 2x Tablet with C3I for squad leader (one spare)
235 -** 1x Base of Operations with computer with C3I
236 -** 4x Health sensors
237 -** 4x Location sensors
238 -** 8x walky talky with two channels (?) (for FR team and BoO).
239 -* For usecase:
240 -** 2x training building (~~100m2?)
241 -** Enviromental challenges: smoke machine, obstacles, etc.
242 -** ~~8x Actors or dummies for victims.
243 -* 8x Clipboards with papers and pens for hotwash and questionnaires
244 -
245 -**13. Dependencies**
246 -
247 -* Availability of materials
248 -* Availability and functioning of ANYmal robot with teleoperation and network in target buildings
249 -* Availability of functioning network between robot, FRs, and BoO.
250 -* Availability of:
251 -** Experiment participants
252 -** People willing to play victim
253 -** Experiment leaders
254 -
255 -**14. Success Criteria**
256 -When mission(s) completed by both teams, and we can compare their performance using the metrics.
257 -
258 -
259 259