Skip to Content
Last modified by Rosa Van Tuijn on 2026/04/13 12:47
From version 3.3
edited by Rosa Van Tuijn
on 2026/03/25 14:36
Change comment: There is no comment for this version
To version 1.15
edited by Tjalling Haije
on 2026/03/23 15:25
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -h. Test SFT1: System baseline test
1 +h. Test SFT1
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.RosaVanTuijn
1 +XWiki.TjallingHaije
Content
... ... @@ -1,5 +1,5 @@
1 1  **Experiment title**
2 -System baseline test
2 +[Short, descriptive title]
3 3  
4 4  
5 5  **2. Objective / Research Question**
... ... @@ -10,6 +10,7 @@
10 10  1. FR Health
11 11  1. FR and victim safety
12 12  
13 +
13 13  **3. Hypotheses / Expectations (optional)**
14 14  See the claims from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]], being:
15 15  
... ... @@ -21,6 +21,7 @@
21 21  * CL06: improved FR health
22 22  * CL07: degraded mission efficiency
23 23  
25 +
24 24  **4. Scenario / Context**
25 25  ASR3: Detailed indoor exploration
26 26  
... ... @@ -32,6 +32,7 @@
32 32  ** **ANYmal **is mission-ready (battery, controls, and payload checked).
33 33  ** **C3I is online**, role-based views are configurated (Operator/ Analyst/ Team leader/ Safety officer); **5G pods** are installed for connection.
34 34  
37 +
35 35  **5. Participants**
36 36  
37 37  * FR team ROBOT:
... ... @@ -104,107 +104,12 @@
104 104  * FR Team ROBOT does Building B
105 105  * FR team HUMAN does Building A
106 106  
110 +
107 107  **9. Measurements**
108 108  
109 -See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]. Compare outcomes between the two conditions teams if applicable.
113 +See the measurements from [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
110 110  
111 -|**Claim: Safety [CL01]**|**Measurment(s)**|**How to test **
112 -|Safer decision before entry|Safe path chosen|(((
113 -* Define ground-truth safe path(s) beforehand so that they can be compared to chosen entry path during the trial
114 -* Score paths beforehand: safe/ suboptimal/ unsafe
115 -)))
116 -|Better decision overall|"How good were you decisions looking back?"|(((
117 -* Hotwash reflection question (likert + open)
118 -* Observations of errors (e.g. near-incidents, unsafe choices)
119 -)))
120 -|Fewer dangerous situations|Near-incident reports|(((
121 -* Define hazards in buildings beforehand and check (with observations of during hotwash) if these hazards are avoided or not
122 -)))
123 123  
124 -|**Claim: Situation Awareness [CL02]**|**Measurment(s)**|**How to test **
125 -|Before entry |(((
126 -Correctness of reported hazards, victims, layout
127 -)))|(((
128 -* Before entry let squad leader and entry team (seperatly) sketch/map or describe inside of building (harzards, victims, layout) > compare these with the ground-truth
129 -
130 -(((
131 -
132 -)))
133 -)))
134 -|During entry|(((
135 -FR path logs
136 -)))|(((
137 -* Compare plannend vs. executed path (how many or how big were the deviations? and were they acceptable?) and identify detours caused by misjudgment or new information
138 -
139 -(((
140 -
141 -)))
142 -)))
143 -|After mission|(((
144 -Correctness of reported hazards, victims, layout
145 -)))|(((
146 -* Repeat sketching/mapping or descriptive task and compare the outcomes (improvement/ deterioration vs pre-entry)
147 -)))
148 -
149 -|**Claim: Mission effectiveness [CL03] [CL04] [CL05]**|**Measurment(s)**|**How to test **
150 -|(((
151 -Acceptable workload
152 -)))|(((
153 -
154 -
155 -NASA-TLX
156 -)))|(((
157 -* Shortened post-trial NASA-TLX questionnaire and analyze score (discuss if these with experts to check if these are "acceptable")
158 -
159 -(((
160 -
161 -)))
162 -)))
163 -|(((
164 -Appropriate trust
165 -)))|(((
166 -
167 -
168 -Trust survey / interviews
169 -)))|(((
170 -*
171 -)))
172 -|(((
173 -Decision confidence
174 -)))|(((
175 -
176 -
177 -Retrospective decision quality rating
178 -)))|
179 -
180 -|**Claim: Health [CL06]**|**Measurment(s)**|**How to test **
181 -|(((
182 -Health remains acceptable
183 -)))|(((
184 -
185 -
186 -Health indicators
187 -)))|(((
188 -* Define acceptable ranges (e.g. HR, temperature, etc.) beforehand and log these ranges
189 -)))
190 -|(((
191 -Issues handled in time
192 -)))|(((
193 -
194 -
195 -Health issues tackled in acceptable time
196 -)))|(((
197 -* Timestamp when acceptable ranges are breached till moment detection and intervention moment (compare this with predefined response thresholds)
198 -)))
199 -
200 -|**Claim: Mission Efficiency [CL07]**|**Measurment(s)**|**How to test **
201 -|Faster mission execution|Total mission completion time|(((
202 -* Timestamp of start (after briefing) till all located victims have been extracted
203 -)))
204 -|Faster inside performance|First responder time inside|(((
205 -* Timestamp of entry building till exist building (last located victim is exstracted)
206 -)))
207 -
208 208  **10. Procedure (Step-by-Step)**
209 209  
210 210  1. __Team preparation__:
... ... @@ -226,33 +226,32 @@
226 226  111. Time for tackling any health alerts
227 227  111. Track FR path
228 228  1. __Hotwash__:
229 -11. Questions (specified per role, couple questions per topic) on:
137 +11. Questions on:
230 230  111. Near-incident report
231 231  111. Path inside
232 232  111. FR location and status of hazards, victims, and layout
233 -111. Mission effectiveness: e.g. self reported effectiveness of decisions
141 +111. Mission effectiveness
234 234  111. Self-reported workload
235 235  111. Trust survey
236 -11. Group discussion with feedback
237 237  
238 238  **~11. Planning:**
239 239  
240 240  * Team preparation: Day before. 1 hour.
241 241  * Trial 1:
242 -** Setup scenario: Test day 1. 09:00 - 09:30
149 +** Setup usecase: Test day 1. 09:00 - 09:30
243 243  ** Setup tech and perform checks: Test day 1. 09:00 - 09:30
244 244  ** Briefing of each team individually (outside building): Test day 1: 09:00 - 09:30
245 245  ** Mission execution: Test day 1. 09:30 - 10:30
246 246  ** Hot wash: Test day 1. 10:30-11:00
247 -** Scenario reset: test day 1. 10:30-11:00
154 +** Usecase reset: test day 1. 10:30-11:00
248 248  * Teams switch location: Test day 1. 11:00-11:15
249 249  * Trial 2: 
250 -** (Setup scenario: Test day 1. 12:30 - 13:00)
157 +** Setup usecase: Test day 1. 12:30 - 13:00
251 251  ** Setup tech and perform checks: Test day 1. 12:30 - 13:00
252 252  ** Briefing of each team individually (outside building): Test day 1: 12:30 - 13:00
253 253  ** Mission execution: Test day 1. 13:00 - 14:00
254 254  ** Hot wash: Test day 1. 14:00-14:30
255 -* Cleanup area and store tech. 14:00-14:30
162 +** Cleanup area and store tech. 14:00-14:30
256 256  
257 257  
258 258  
... ... @@ -271,6 +271,7 @@
271 271  ** ~~8x Actors or dummies for victims.
272 272  * 8x Clipboards with papers and pens for hotwash and questionnaires
273 273  
181 +
274 274  **13. Dependencies**
275 275  
276 276  * Availability of materials
... ... @@ -281,6 +281,7 @@
281 281  ** People willing to play victim
282 282  ** Experiment leaders
283 283  
192 +
284 284  **14. Success Criteria**
285 285  When mission(s) completed by both teams, and we can compare their performance using the metrics.
286 286