Skip to Content
Last modified by Rosa Van Tuijn on 2026/04/13 12:47
From version 1.7
edited by Tjalling Haije
on 2026/03/23 14:15
Change comment: There is no comment for this version
To version 5.1
edited by Rosa Van Tuijn
on 2026/04/13 07:54
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -h. Test SFT1
1 +h. Test SFT1: System baseline test
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.TjallingHaije
1 +XWiki.RosaVanTuijn
Content
... ... @@ -1,9 +1,9 @@
1 1  **Experiment title**
2 -[Short, descriptive title]
2 +System baseline test
3 3  
4 4  
5 5  **2. Objective / Research Question**
6 -Do the SYNERGISE technologies together improve the below SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] without negative effects?
6 +Do the SYNERGISE technologies together improve the SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] (stated below) without negative effects?
7 7  
8 8  1. Mission effectiveness
9 9  1. Mission efficiency
... ... @@ -10,7 +10,6 @@
10 10  1. FR Health
11 11  1. FR and victim safety
12 12  
13 -
14 14  **3. Hypotheses / Expectations (optional)**
15 15  See the claims from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]], being:
16 16  
... ... @@ -22,7 +22,6 @@
22 22  * CL06: improved FR health
23 23  * CL07: degraded mission efficiency
24 24  
25 -
26 26  **4. Scenario / Context**
27 27  ASR3: Detailed indoor exploration
28 28  
... ... @@ -34,7 +34,6 @@
34 34  ** **ANYmal **is mission-ready (battery, controls, and payload checked).
35 35  ** **C3I is online**, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); **5G pods** are installed for connection.
36 36  
37 -
38 38  **5. Participants**
39 39  
40 40  * FR team ROBOT:
... ... @@ -94,59 +94,198 @@
94 94  
95 95  Two buildings:
96 96  
97 -* Building A: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
98 -* Building B: Easier location, obstacles and environmental conditions. (clear visibility, ..)
94 +* Building A: Easier location, obstacles and environmental conditions. (clear visibility, ..)
95 +* Building B: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
99 99  
97 +Trial 1:
100 100  
99 +* FR Team ROBOT does Building A
100 +* FR team HUMAN does Building B
101 101  
102 -Trial1:
102 +Trial 2:
103 103  
104 -* FR Team ROBOT does easy
104 +* FR Team ROBOT does Building B
105 +* FR team HUMAN does Building A
105 105  
106 -Team A does first [[UC04.3>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
107 +**9. Measurements**
107 107  
108 -[Condition A vs B, baseline vs intervention, with/without transparency, etc.]
109 +See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]. Compare outcomes between the two conditions teams if applicable. See [[j. SFT1 evaluation templates >>3\. Evaluation.g\. Prototype .WebHome]] for more details about the measurements during a test.
109 109  
111 +|**Claim: Safety [CL01]**|**Measurment(s)**|**How to test **
112 +|Safer decision before entry|Safe path chosen|(((
113 +* Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
114 +* Score paths beforehand: safe/ suboptimal/ unsafe
115 +)))
116 +|Better decision overall|"How good were you decisions looking back?"|(((
117 +* Hotwash reflection question (likert + open)
118 +* Observations of errors (e.g. near-incidents, unsafe choices)
119 +)))
120 +|Fewer dangerous situations|Near-incident reports|(((
121 +* Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not
122 +)))
110 110  
111 -**9. Measurements**
124 +|**Claim: Situation Awareness [CL02]**|**Measurment(s)**|**How to test **
125 +|Before entry |(((
126 +Correctness of reported hazards, victims, layout
127 +)))|(((
128 +* Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth
112 112  
113 -* Objective measures: [E.g., task completion time, errors, navigation metrics]
114 -* Subjective measures: [E.g., trust, workload, clarity, usability questionnaires]
115 -* Logs / sensors: [E.g., robot logs, video, physiological sensors]
130 +(((
131 +
132 +)))
133 +)))
134 +|During entry|(((
135 +FR path logs
136 +)))|(((
137 +* Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information
116 116  
139 +(((
140 +
141 +)))
142 +)))
143 +|After mission|(((
144 +Correctness of reported hazards, victims, layout
145 +)))|(((
146 +* Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
147 +)))
117 117  
118 -**10. Procedure (Step-by-Step)**
149 +|**Claim: Mission effectiveness [CL03] [CL04] [CL05]**|**Measurment(s)**|**How to test **
150 +|(((
151 +Acceptable workload
152 +)))|(((
153 +
119 119  
120 -1. [Describe preparation]
121 -1. [Briefing]
122 -1. [Execution]
123 -1. [Debrief / Hotwash]
155 +NASA-TLX
156 +)))|(((
157 +* Shortened post-trial NASA-TLX questionnaire and analyze score (discuss if these with experts to check if these are "acceptable")
124 124  
159 +(((
160 +
161 +)))
162 +)))
163 +|(((
164 +Appropriate trust
165 +)))|(((
166 +
125 125  
126 -**~11. Materials / Setup Components**
127 -[List equipment, environment layout, props, simulation assets]
168 +Trust survey / interviews
169 +)))|(((
170 +*
171 +)))
172 +|(((
173 +Decision confidence
174 +)))|(((
175 +
128 128  
177 +Retrospective decision quality rating
178 +)))|
129 129  
130 -**12. Safety & Ethical Considerations**
180 +|**Claim: Health [CL06]**|**Measurment(s)**|**How to test **
181 +|(((
182 +Health remains acceptable
183 +)))|(((
184 +
131 131  
132 -* Risks: [Describe]
133 -* Mitigations: [Describe]
134 -* Informed consent: [Yes/No + method]
135 -* Data handling: [Describe]
186 +Health indicators
187 +)))|(((
188 +* Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these ranges
189 +)))
190 +|(((
191 +Issues handled in time
192 +)))|(((
193 +
136 136  
195 +Health issues tackled in acceptable time
196 +)))|(((
197 +* Timestamp when acceptable ranges are breached till moment detection and intervention moment (compare this with predefined response thresholds)
198 +)))
137 137  
138 -**13. Location & Duration**
200 +|**Claim: Mission Efficiency [CL07]**|**Measurment(s)**|**How to test **
201 +|Faster mission execution|Total mission completion time|(((
202 +* Timestamp of start (after briefing) till all located victims have been extracted
203 +)))
204 +|Faster inside performance|First responder time inside|(((
205 +* Timestamp of entry building till exist building (last located victim is exstracted)
206 +)))
139 139  
140 -* Location: [Where does it take place?]
141 -* Duration: [Total time per run + preparation]
208 +**10. Procedure (Step-by-Step)**
142 142  
210 +1. __Team preparation__:
211 +11. FR team ROBOT is trained in operating the robot, analysing the camera stream, and pinning information on C3I
212 +11. Squad leader and BoO are trained for both teams in interpreting the health and location sensor data.
213 +11. Experiment leader knows where hazards and victims are in both buildings.
214 +11. Anymal robot is tested with network in target buildings for experiment (without FR themselves seeing building!)
215 +1. __Experiment preparation:__
216 +11. Building is prepped with victims, hazards, etc.
217 +1. __Briefing:__
218 +11. Same for both teams for both building types: Building has been inspected from outside and is estimated safe for entry. Situation inside unknown. Victims possibly present. Goal: find, assess, and extract victims while avoiding and mapping hazards.
219 +11. One person is notified that their health sensor will be triggered during the Building B (difficult building) situation.
220 +1. __Execution:__
221 +11. For FR team ROBOT, see: [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
222 +11. For FR team HUMAN, see: [[UC04.4: Detailed indoor exploration without ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYmal (USAR).WebHome]]
223 +11. During execution of both teams, performance will be measured by experiment leader:
224 +111. Mission completion time
225 +111. Health indicators stay within acceptable range
226 +111. Time for tackling any health alerts
227 +111. Track FR path
228 +1. __Hotwash__:
229 +11. Questions (specified per role, couple questions per topic) on:
230 +111. Near-incident report
231 +111. Path inside
232 +111. FR location and status of hazards, victims, and layout
233 +111. Mission effectiveness: e.g. self reported effectiveness of decisions
234 +111. Self-reported workload
235 +111. Trust survey
236 +11. Group discussion with feedback
143 143  
144 -**14. Dependencies**
145 -[Other WPs, required assets, system availability, permissions]
238 +**~11. Planning:**
146 146  
240 +* Team preparation: Day before. 1 hour.
241 +* Trial 1:
242 +** Setup scenario: Test day 1. 09:00 - 09:30
243 +** Setup tech and perform checks: Test day 1. 09:00 - 09:30
244 +** Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
245 +** Mission execution: Test day 1. 09:30 - 10:30
246 +** Hot wash: Test day 1. 10:30-11:00
247 +** Scenario reset: test day 1. 10:30-11:00
248 +* Teams switch location: Test day 1. 11:00-11:15
249 +* Trial 2: 
250 +** (Setup scenario: Test day 1. 12:30 - 13:00)
251 +** Setup tech and perform checks: Test day 1. 12:30 - 13:00
252 +** Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
253 +** Mission execution: Test day 1. 13:00 - 14:00
254 +** Hot wash: Test day 1. 14:00-14:30
255 +* Cleanup area and store tech. 14:00-14:30
147 147  
148 -**15. Success Criteria**
149 -[How do you know the experiment worked as intended?]
150 150  
151 151  
259 +**12. Materials**
260 +
261 +* For FRs:
262 +** 1x Anymal with teleop setup and networking
263 +** 2x Tablet with C3I for squad leader (one spare)
264 +** 1x Base of Operations with computer with C3I
265 +** 4x Health sensors
266 +** 4x Location sensors
267 +** 8x walky talky with two channels (?) (for FR team and BoO).
268 +* For usecase:
269 +** 2x training building (~~100m2?)
270 +** Enviromental challenges: smoke machine, obstacles, etc.
271 +** ~~8x Actors or dummies for victims.
272 +* 8x Clipboards with papers and pens for hotwash and questionnaires
273 +
274 +**13. Dependencies**
275 +
276 +* Availability of materials
277 +* Availability and functioning of ANYmal robot with teleoperation and network in target buildings
278 +* Availability of functioning network between robot, FRs, and BoO.
279 +* Availability of:
280 +** Experiment participants
281 +** People willing to play victim
282 +** Experiment leaders
283 +
284 +**14. Success Criteria**
285 +When mission(s) completed by both teams, and we can compare their performance using the metrics.
286 +
287 +
152 152