Skip to Content
Last modified by Tjalling Haije on 2026/07/27 16:18
From version 5.1
edited by Tjalling Haije
on 2026/07/27 16:18
Change comment: There is no comment for this version
To version 3.1
edited by Tjalling Haije
on 2026/07/27 13:37
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -65,13 +65,7 @@
65 65  === 2. Incident 2 tech evaluation ===
66 66  
67 67  |(% style="width:247px" %)**Tech**|(% style="width:319px" %)**How was it used**|(% style="width:329px" %)**Issues**|(% style="width:329px" %)**Did it work sufficiently?**
68 -|(% style="width:247px" %)[[ASTRAL RTK Drone>>Main.c\. Technology.ASTRAL RTK Drone.WebHome]] (outdoor)|(% style="width:319px" %)Baseline & Run 1-3: The drone team conducts an aerial assessment of the collapsed buildings, identifying structural risks, access points, and possible victim locations. |(% style="width:329px" %)(((
69 -Run 1 USAR 1: Drone had connection issues which delayed its deployment.
70 -
71 -{{warning}}
72 -TODO: expand these issues with questionnaire answers
73 -{{/warning}}
74 -)))|(% style="width:329px" %)Yes
68 +|(% style="width:247px" %)[[ASTRAL RTK Drone>>Main.c\. Technology.ASTRAL RTK Drone.WebHome]] (outdoor)|(% style="width:319px" %)Baseline & Run 1-3: The drone team conducts an aerial assessment of the collapsed buildings, identifying structural risks, access points, and possible victim locations. |(% style="width:329px" %)Run 1 USAR 1: Drone had connection issues which delayed its deployment.|(% style="width:329px" %)Yes
75 75  |(% style="width:247px" %)[[OWL Indoor Drone>>Main.c\. Technology.OWL Indoor Drone.WebHome]] |(% style="width:319px" %)(((
76 76  Run 2 USAR 2: used to explore building and search for objects and victims
77 77  
... ... @@ -79,9 +79,7 @@
79 79  )))|(% style="width:329px" %)(((
80 80  Run 2 USAR 1: OWL missed victim and entire floor. Object detection initially disabled.
81 81  
82 -{{warning}}
83 -TODO: expand these issues with questionnaire answers
84 -{{/warning}}
76 +
85 85  )))|(% style="width:329px" %)Unreliable
86 86  |(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[Robot Arm (for ANYmal)>>Main.c\. Technology.Robot Arm (for ANYmal).WebHome]] |(% style="width:319px" %)(((
87 87  Run 1 USAR 1: remove beam blocking door
... ... @@ -89,10 +89,8 @@
89 89  Run 3 USAR 1: Used to close a valve
90 90  
91 91  
92 -)))|(% style="width:329px" %){{warning}}TODO: expand these issues with questionnaire answers{{/warning}}(((
93 -
94 -)))|(% style="width:329px" %)Yes
95 -|(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[SNAKE>>Main.c\. Technology.SNAKE.WebHome]] |(% style="width:319px" %)Run 3 USAR 1: Used to search for victims in a narrow space|(% style="width:329px" %){{warning}}TODO: expand these issues with questionnaire answers{{/warning}}|(% style="width:329px" %)?
84 +)))|(% style="width:329px" %) |(% style="width:329px" %)Yes
85 +|(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[SNAKE>>Main.c\. Technology.SNAKE.WebHome]] |(% style="width:319px" %)Run 3 USAR 1: Used to search for victims in a narrow space|(% style="width:329px" %) |(% style="width:329px" %)?
96 96  
97 97  === 3. Incident 2 way of working ===
98 98  
... ... @@ -111,61 +111,26 @@
111 111  
112 112  == 4. Raw results ==
113 113  
114 -=== Results for Incident 2, Run 3, USAR team 1 VS baseline USAR team 1 ===
104 +**Results for Incident 2, Run 1, USAR team 1**
115 115  
116 -Analysis of questionnaire results for members with ID: UT13, UT11, UT12, UTL1
117 -
118 -
119 -|=Scenario|=Role|=Participants
120 -|Baseline|USAR Team Member|3
121 -|Baseline|USAR Team Leader|1
122 -|Run 3|USAR Team Member|3
123 -|Run 3|USAR Team Leader|1
124 -|Run 1|USAR Team Member|3
125 -|Run 1|USAR Team Leader|1
126 -
127 -
128 -
129 -[[image:1785167968429-388.png||height="390" width="821"]]
130 -
131 -[[image:1785167981571-844.png||height="419" width="815"]]
132 -
133 -[[image:1785167994552-876.png||height="329" width="767"]]
134 -
135 -
136 -These results show that in run 1 workload, trust and situation awareness were significantly lower compared to the baseline, with decision making and safety confidence differing only slightly from the baseline. 
137 -Run 3 showed an improvement in decision making (specified as right info was available and decisions were timely) compared to the baseline and run 1.
138 -
139 -On average both runs had a lower workload (-25%), trust (-33%), situation awareness (-11%), slightly lower safety confidence (-4%) and improved decision making (+8%).
140 -
141 -Note: Run 3 also functioned as a demonstration, which increased speed of decisions and the scenario as a whole. Also, participants performed incident 2 in 3 runs in the same teamcomposition and building layout, such that participants giving them an unfair advantage. These factors might have influenced the results.
142 -
143 -
144 -=== 5. Results connected to Design Pattern measures ===
145 -
146 146  Note: The measures are defined in [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]  and [[x. Measures>>3\. Evaluation.Measures.WebHome]]
147 147  
148 148  |(% style="width:376px" %)**Measure**|(% style="width:225px" %)**Results**|**Connected Claim**|**Claim proven?**
149 -|(% style="width:376px" %)Safe path chosen|(% style="width:225px" %)No FR path data available.|CL01 Improve safety for responders |Unknown
150 -|(% style="width:376px" %)Decision quality|(% style="width:225px" %)Can we use decision making questions from questionnaire for this?  |CL01 Improve safety for responders |Unknown
151 -|(% style="width:376px" %)Near-incident and incident reports|(% style="width:225px" %)No incidents in baseline or run. 
152 -No near-incidents data available.|CL01 Improve safety for responders |Unknown
153 -|(% style="width:376px" %)Reported hazards, victims, layout correctness, based on robot intel before FR entry|(% style="width:225px" %)No ground truth data available.|CL02 Increased SA|Unknown
154 -|(% style="width:376px" %)How optimal was the FR path|(% style="width:225px" %)No FR path or ground truth data available.|CL02 Increased SA|Unknown
155 -|(% style="width:376px" %)Reported hazards, victims, layout correctness, after FR entry|(% style="width:225px" %)No FR path or ground truth data available.|CL02 Increased SA|Unknown
156 -|(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %)Questionnaire indicates increased workload of 25%. But not literature reviewed questionnaire and runs not clean. |CL03 improved mission effectiveness|Indication says strong no, but data unreliable.
157 -|(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %)Can we use trust questions from questionnaire for this? They did not target robots specifically.|CL04 acceptable workload|Unknown
158 -|(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %)Can we use decision making questions from questionnaire for this?  |CL04 acceptable workload|Unknown
159 -|(% style="width:376px" %)Trust in SA|(% style="width:225px" %)Questionnaire indicates decreased trust of -33%. But not literature reviewed questionnaire and runs not clean|CL05 Trustworthy SA|Indication says strong no, but data unreliable.
160 -|(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %)No health data available.|CL06 Improved FR health|Unknown
161 -|(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %)Do we have this?|CL06 Improved FR health|Unknown
162 -|(% style="width:376px" %)Total mission completion time|(% style="width:225px" %)Do we have this? Compared to baseline?|CL07 Degraded mission efficiency|Unknown
163 -|(% style="width:376px" %)First responder time inside|(% style="width:225px" %)Do we have this? Compared to baseline?|CL07 Degraded mission efficiency|Unknown
109 +|(% style="width:376px" %)Safe path chosen|(% style="width:225px" %) |CL01 Improve safety for responders |
110 +|(% style="width:376px" %)Decision quality|(% style="width:225px" %) |CL01 Improve safety for responders |
111 +|(% style="width:376px" %)Near-incident and incident reports|(% style="width:225px" %) |CL01 Improve safety for responders |
112 +|(% style="width:376px" %)Reported hazards, victims, layout correctness, based on robot intel before FR entry|(% style="width:225px" %) |CL02 Increased SA|
113 +|(% style="width:376px" %)How optimal was the FR path|(% style="width:225px" %) |CL02 Increased SA|
114 +|(% style="width:376px" %)Reported hazards, victims, layout correctness, after FR entry|(% style="width:225px" %) |CL02 Increased SA|
115 +|(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %) |CL03 improved mission effectiveness|
116 +|(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %) |CL04 acceptable workload|
117 +|(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %) |CL04 acceptable workload|
118 +|(% style="width:376px" %)?|(% style="width:225px" %) |CL05 Trustworthy SA|
119 +|(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %) |CL06 Improved FR health|
120 +|(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %) |CL06 Improved FR health|
121 +|(% style="width:376px" %)Total mission completion time|(% style="width:225px" %) |CL07 Degraded mission efficiency|
122 +|(% style="width:376px" %)First responder time inside|(% style="width:225px" %) |CL07 Degraded mission efficiency|
164 164  
165 -(% class="wikigeneratedid" %)
166 -(Reliable) data missing to evaluate all claims for the usecase and as such team design pattern.
167 -
168 -
169 169  === 5. Pros and cons for TDP ===
170 170  
171 171  [[TDP2 — Robots First, Then Humans Protocol>>2\. Specification.Requirements.TDP2 — Robots First, Then Humans Protocol.WebHome]]
1785164456543-963.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -146.4 KB
Content
1785164494950-855.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -162.9 KB
Content
1785164507435-933.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -108.5 KB
Content
1785167243750-606.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -149.0 KB
Content
1785167259751-744.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -178.2 KB
Content
1785167267780-382.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -115.0 KB
Content
1785167968429-388.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -161.2 KB
Content
1785167981571-844.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -211.8 KB
Content
1785167994552-876.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.TjallingHaije
Size
... ... @@ -1,1 +1,0 @@
1 -168.8 KB
Content