Skip to Content
Last modified by Tjalling Haije on 2026/07/27 16:18
From version 4.4
edited by Tjalling Haije
on 2026/07/27 15:03
Change comment: There is no comment for this version
To version 5.1
edited by Tjalling Haije
on 2026/07/27 16:18
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -121,33 +121,51 @@
121 121  |Baseline|USAR Team Leader|1
122 122  |Run 3|USAR Team Member|3
123 123  |Run 3|USAR Team Leader|1
124 +|Run 1|USAR Team Member|3
125 +|Run 1|USAR Team Leader|1
124 124  
125 -[[image:1785164456543-963.png||height="376" width="752"]]
126 126  
127 -[[image:1785164494950-855.png||height="348" width="756"]]
128 128  
129 -[[image:1785164507435-933.png||height="253" width="745"]]
129 +[[image:1785167968429-388.png||height="390" width="821"]]
130 130  
131 +[[image:1785167981571-844.png||height="419" width="815"]]
131 131  
133 +[[image:1785167994552-876.png||height="329" width="767"]]
134 +
135 +
136 +These results show that in run 1 workload, trust and situation awareness were significantly lower compared to the baseline, with decision making and safety confidence differing only slightly from the baseline. 
137 +Run 3 showed an improvement in decision making (specified as right info was available and decisions were timely) compared to the baseline and run 1.
138 +
139 +On average both runs had a lower workload (-25%), trust (-33%), situation awareness (-11%), slightly lower safety confidence (-4%) and improved decision making (+8%).
140 +
141 +Note: Run 3 also functioned as a demonstration, which increased speed of decisions and the scenario as a whole. Also, participants performed incident 2 in 3 runs in the same teamcomposition and building layout, such that participants giving them an unfair advantage. These factors might have influenced the results.
142 +
143 +
144 +=== 5. Results connected to Design Pattern measures ===
145 +
132 132  Note: The measures are defined in [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]  and [[x. Measures>>3\. Evaluation.Measures.WebHome]]
133 133  
134 134  |(% style="width:376px" %)**Measure**|(% style="width:225px" %)**Results**|**Connected Claim**|**Claim proven?**
135 -|(% style="width:376px" %)Safe path chosen|(% style="width:225px" %) |CL01 Improve safety for responders |
136 -|(% style="width:376px" %)Decision quality|(% style="width:225px" %) |CL01 Improve safety for responders |
149 +|(% style="width:376px" %)Safe path chosen|(% style="width:225px" %)No FR path data available.|CL01 Improve safety for responders |Unknown
150 +|(% style="width:376px" %)Decision quality|(% style="width:225px" %)Can we use decision making questions from questionnaire for this?  |CL01 Improve safety for responders |Unknown
137 137  |(% style="width:376px" %)Near-incident and incident reports|(% style="width:225px" %)No incidents in baseline or run. 
138 138  No near-incidents data available.|CL01 Improve safety for responders |Unknown
139 139  |(% style="width:376px" %)Reported hazards, victims, layout correctness, based on robot intel before FR entry|(% style="width:225px" %)No ground truth data available.|CL02 Increased SA|Unknown
140 140  |(% style="width:376px" %)How optimal was the FR path|(% style="width:225px" %)No FR path or ground truth data available.|CL02 Increased SA|Unknown
141 141  |(% style="width:376px" %)Reported hazards, victims, layout correctness, after FR entry|(% style="width:225px" %)No FR path or ground truth data available.|CL02 Increased SA|Unknown
142 -|(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %) |CL03 improved mission effectiveness|
143 -|(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %) |CL04 acceptable workload|
144 -|(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %) |CL04 acceptable workload|
145 -|(% style="width:376px" %)?|(% style="width:225px" %) |CL05 Trustworthy SA|
146 -|(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %) |CL06 Improved FR health|
147 -|(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %) |CL06 Improved FR health|
148 -|(% style="width:376px" %)Total mission completion time|(% style="width:225px" %) |CL07 Degraded mission efficiency|
149 -|(% style="width:376px" %)First responder time inside|(% style="width:225px" %) |CL07 Degraded mission efficiency|
156 +|(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %)Questionnaire indicates increased workload of 25%. But not literature reviewed questionnaire and runs not clean. |CL03 improved mission effectiveness|Indication says strong no, but data unreliable.
157 +|(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %)Can we use trust questions from questionnaire for this? They did not target robots specifically.|CL04 acceptable workload|Unknown
158 +|(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %)Can we use decision making questions from questionnaire for this?  |CL04 acceptable workload|Unknown
159 +|(% style="width:376px" %)Trust in SA|(% style="width:225px" %)Questionnaire indicates decreased trust of -33%. But not literature reviewed questionnaire and runs not clean|CL05 Trustworthy SA|Indication says strong no, but data unreliable.
160 +|(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %)No health data available.|CL06 Improved FR health|Unknown
161 +|(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %)Do we have this?|CL06 Improved FR health|Unknown
162 +|(% style="width:376px" %)Total mission completion time|(% style="width:225px" %)Do we have this? Compared to baseline?|CL07 Degraded mission efficiency|Unknown
163 +|(% style="width:376px" %)First responder time inside|(% style="width:225px" %)Do we have this? Compared to baseline?|CL07 Degraded mission efficiency|Unknown
150 150  
165 +(% class="wikigeneratedid" %)
166 +(Reliable) data missing to evaluate all claims for the usecase and as such team design pattern.
167 +
168 +
151 151  === 5. Pros and cons for TDP ===
152 152  
153 153  [[TDP2 — Robots First, Then Humans Protocol>>2\. Specification.Requirements.TDP2 — Robots First, Then Humans Protocol.WebHome]]
1785167243750-606.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +149.0 KB
Content
1785167259751-744.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +178.2 KB
Content
1785167267780-382.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +115.0 KB
Content
1785167968429-388.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +161.2 KB
Content
1785167981571-844.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +211.8 KB
Content
1785167994552-876.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +168.8 KB
Content