Skip to Content
Last modified by Rosa Van Tuijn on 2026/04/13 12:47
From version 1.11
edited by Tjalling Haije
on 2026/03/23 15:10
Change comment: There is no comment for this version
To version 6.1
edited by Rosa Van Tuijn
on 2026/04/13 12:47
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -h. Test SFT1
1 +h. Test SFT1: System baseline test
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.TjallingHaije
1 +XWiki.RosaVanTuijn
Content
... ... @@ -1,9 +1,9 @@
1 1  **Experiment title**
2 -[Short, descriptive title]
2 +System baseline test
3 3  
4 4  
5 5  **2. Objective / Research Question**
6 -Do the SYNERGISE technologies together improve the below SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] without negative effects?
6 +Do the SYNERGISE technologies together improve the SYNERGISE [[Objectives>>2\. Specification.Objectives.WebHome]] (stated below) without negative effects?
7 7  
8 8  1. Mission effectiveness
9 9  1. Mission efficiency
... ... @@ -10,7 +10,6 @@
10 10  1. FR Health
11 11  1. FR and victim safety
12 12  
13 -
14 14  **3. Hypotheses / Expectations (optional)**
15 15  See the claims from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]], being:
16 16  
... ... @@ -22,7 +22,6 @@
22 22  * CL06: improved FR health
23 23  * CL07: degraded mission efficiency
24 24  
25 -
26 26  **4. Scenario / Context**
27 27  ASR3: Detailed indoor exploration
28 28  
... ... @@ -34,7 +34,6 @@
34 34  ** **ANYmal **is mission-ready (battery, controls, and payload checked).
35 35  ** **C3I is online**, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); **5G pods** are installed for connection.
36 36  
37 -
38 38  **5. Participants**
39 39  
40 40  * FR team ROBOT:
... ... @@ -96,7 +96,6 @@
96 96  
97 97  * Building A: Easier location, obstacles and environmental conditions. (clear visibility, ..)
98 98  * Building B: Challenging location, obstacles and environmental conditions for humans and robots (e.g. smoke and obstacles).
99 -*
100 100  
101 101  Trial 1:
102 102  
... ... @@ -108,24 +108,120 @@
108 108  * FR Team ROBOT does Building B
109 109  * FR team HUMAN does Building A
110 110  
111 -
112 112  **9. Measurements**
113 113  
114 -See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
109 +See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]. Compare outcomes between the two conditions teams if applicable. See [[j. SFT1 evaluation templates >>3\. Evaluation.g\. Prototype .WebHome]] for more details about the measurements during a test.
115 115  
111 +|**Claim: Safety [CL01]**|**Measurment(s)**|**How to test **
112 +|Safer decision before entry|Safe path chosen|(((
113 +* Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
114 +* Score paths beforehand: safe/ suboptimal/ unsafe
115 +)))
116 +|Better decision overall|"How good were you decisions looking back?"|(((
117 +* Hotwash reflection question (likert + open)
118 +* Observations of errors (e.g. near-incidents, unsafe choices)
119 +)))
120 +|Fewer dangerous situations|Near-incident reports|(((
121 +* Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not
122 +)))
116 116  
124 +|**Claim: Situation Awareness [CL02]**|**Measurment(s)**|**How to test **
125 +|Before entry |(((
126 +Correctness of reported hazards, victims, layout
127 +)))|(((
128 +* Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth
129 +
130 +(((
131 +
132 +)))
133 +)))
134 +|During entry|(((
135 +FR path logs
136 +)))|(((
137 +* Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information
138 +
139 +(((
140 +
141 +)))
142 +)))
143 +|After mission|(((
144 +Correctness of reported hazards, victims, layout
145 +)))|(((
146 +* Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
147 +)))
148 +
149 +|**Claim: Mission effectiveness [CL03] [CL04] [CL05]**|**Measurment(s)**|**How to test **
150 +|(((
151 +Acceptable workload
152 +)))|(((
153 +
154 +
155 +NASA-TLX
156 +)))|(((
157 +* Shortened post-trial NASA-TLX questionnaire and analyze score (discuss if these with experts to check if these are "acceptable")
158 +
159 +(((
160 +
161 +)))
162 +)))
163 +|(((
164 +Appropriate trust
165 +)))|(((
166 +
167 +
168 +Trust survey / interviews
169 +)))|(((
170 +*
171 +)))
172 +|(((
173 +Decision confidence
174 +)))|(((
175 +
176 +
177 +Retrospective decision quality rating
178 +)))|
179 +
180 +|**Claim: Health [CL06]**|**Measurment(s)**|**How to test **
181 +|(((
182 +Health remains acceptable
183 +)))|(((
184 +
185 +
186 +Health indicators
187 +)))|(((
188 +* Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these ranges
189 +)))
190 +|(((
191 +Issues handled in time
192 +)))|(((
193 +
194 +
195 +Health issues tackled in acceptable time
196 +)))|(((
197 +* Timestamp when acceptable ranges are breached till moment detection and intervention moment (compare this with predefined response thresholds)
198 +)))
199 +
200 +|**Claim: Mission Efficiency [CL07]**|**Measurment(s)**|**How to test **
201 +|Faster mission execution|Total mission completion time|(((
202 +* Timestamp of start (after briefing) till all located victims have been extracted
203 +)))
204 +|Faster inside performance|First responder time inside|(((
205 +* Timestamp of entry building till exist building (last located victim is exstracted)
206 +)))
207 +
117 117  **10. Procedure (Step-by-Step)**
118 118  
119 -1. Team training:
210 +1. __Team preparation__:
120 120  11. FR team ROBOT is trained in operating the robot, analysing the camera stream, and pinning information on C3I
121 121  11. Squad leader and BoO are trained for both teams in interpreting the health and location sensor data.
122 122  11. Experiment leader knows where hazards and victims are in both buildings.
123 -1. Usecase preparation
214 +11. Anymal robot is tested with network in target buildings for experiment (without FR themselves seeing building!)
215 +1. __Experiment preparation:__
124 124  11. Building is prepped with victims, hazards, etc.
125 -1. Briefing:
217 +1. __Briefing:__
126 126  11. Same for both teams for both building types: Building has been inspected from outside and is estimated safe for entry. Situation inside unknown. Victims possibly present. Goal: find, assess, and extract victims while avoiding and mapping hazards.
127 127  11. One person is notified that their health sensor will be triggered during the Building B (difficult building) situation.
128 -1. Execution:
220 +1. __Execution:__
129 129  11. For FR team ROBOT, see: [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
130 130  11. For FR team HUMAN, see: [[UC04.4: Detailed indoor exploration without ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYmal (USAR).WebHome]]
131 131  11. During execution of both teams, performance will be measured by experiment leader:
... ... @@ -133,47 +133,64 @@
133 133  111. Health indicators stay within acceptable range
134 134  111. Time for tackling any health alerts
135 135  111. Track FR path
136 -1. Hotwash:
137 -11. Questions on:
228 +1. __Hotwash__:
229 +11. Questions (specified per role, couple questions per topic) on:
138 138  111. Near-incident report
139 139  111. Path inside
140 140  111. FR location and status of hazards, victims, and layout
141 -111. Mission effectiveness
233 +111. Mission effectiveness: e.g. self reported effectiveness of decisions
142 142  111. Self-reported workload
143 143  111. Trust survey
236 +11. Group discussion with feedback
144 144  
145 -Planning:
238 +**~11. Planning:**
146 146  
147 -* Team training:  Day before. 1 hour.
240 +* Team preparation: Day before. 1 hour.
148 148  * Trial 1:
149 -** Setup usecase: Test day 1. 09:00 - 09:30
242 +** Setup scenario: Test day 1. 09:00 - 09:30
150 150  ** Setup tech and perform checks: Test day 1. 09:00 - 09:30
151 151  ** Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
152 152  ** Mission execution: Test day 1. 09:30 - 10:30
153 153  ** Hot wash: Test day 1. 10:30-11:00
154 -** Usecase reset: test day 1. 10:30-11:00
247 +** Scenario reset: test day 1. 10:30-11:00
155 155  * Teams switch location: Test day 1. 11:00-11:15
156 156  * Trial 2: 
157 -** Setup usecase: Test day 1. 12:30 - 13:00
250 +** (Setup scenario: Test day 1. 12:30 - 13:00)
158 158  ** Setup tech and perform checks: Test day 1. 12:30 - 13:00
159 159  ** Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
160 160  ** Mission execution: Test day 1. 13:00 - 14:00
161 161  ** Hot wash: Test day 1. 14:00-14:30
162 -** Cleanup area and store tech. 14:00-14:30
255 +* Cleanup area and store tech. 14:00-14:30
163 163  
164 164  
165 165  
259 +**12. Materials**
166 166  
167 -**~11. Materials / Setup Components**
168 -[List equipment, environment layout, props, simulation assets]
261 +* For FRs:
262 +** 1x Anymal with teleop setup and networking
263 +** 2x Tablet with C3I for squad leader (one spare)
264 +** 1x Base of Operations with computer with C3I
265 +** 4x Health sensors
266 +** 4x Location sensors
267 +** 8x walky talky with two channels (?) (for FR team and BoO).
268 +* For usecase:
269 +** 2x training building (~~100m2?)
270 +** Enviromental challenges: smoke machine, obstacles, etc.
271 +** ~~8x Actors or dummies for victims.
272 +* 8x Clipboards with papers and pens for hotwash and questionnaires
169 169  
274 +**13. Dependencies**
170 170  
171 -**14. Dependencies**
172 -[Other WPs, required assets, system availability, permissions]
276 +* Availability of materials
277 +* Availability and functioning of ANYmal robot with teleoperation and network in target buildings
278 +* Availability of functioning network between robot, FRs, and BoO.
279 +* Availability of:
280 +** Experiment participants
281 +** People willing to play victim
282 +** Experiment leaders
173 173  
284 +**14. Success Criteria**
285 +When mission(s) completed by both teams, and we can compare their performance using the metrics.
174 174  
175 -**15. Success Criteria**
176 -[How do you know the experiment worked as intended?]
177 177  
178 -
179 179