Skip to Content
Last modified by Tjalling Haije on 2026/07/27 16:18
From version 2.20
edited by Tjalling Haije
on 2026/07/27 13:24
Change comment: There is no comment for this version
To version 5.1
edited by Tjalling Haije
on 2026/07/27 16:18
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -i. Design Patterns Evaluation SFT1
1 +k. Design Patterns Evaluation SFT1
Content
... ... @@ -65,7 +65,13 @@
65 65  === 2. Incident 2 tech evaluation ===
66 66  
67 67  |(% style="width:247px" %)**Tech**|(% style="width:319px" %)**How was it used**|(% style="width:329px" %)**Issues**|(% style="width:329px" %)**Did it work sufficiently?**
68 -|(% style="width:247px" %)[[ASTRAL RTK Drone>>Main.c\. Technology.ASTRAL RTK Drone.WebHome]] (outdoor)|(% style="width:319px" %)Baseline & Run 1-3: The drone team conducts an aerial assessment of the collapsed buildings, identifying structural risks, access points, and possible victim locations. |(% style="width:329px" %)Run 1 USAR 1: Drone had connection issues which delayed its deployment.|(% style="width:329px" %)Yes
68 +|(% style="width:247px" %)[[ASTRAL RTK Drone>>Main.c\. Technology.ASTRAL RTK Drone.WebHome]] (outdoor)|(% style="width:319px" %)Baseline & Run 1-3: The drone team conducts an aerial assessment of the collapsed buildings, identifying structural risks, access points, and possible victim locations. |(% style="width:329px" %)(((
69 +Run 1 USAR 1: Drone had connection issues which delayed its deployment.
70 +
71 +{{warning}}
72 +TODO: expand these issues with questionnaire answers
73 +{{/warning}}
74 +)))|(% style="width:329px" %)Yes
69 69  |(% style="width:247px" %)[[OWL Indoor Drone>>Main.c\. Technology.OWL Indoor Drone.WebHome]] |(% style="width:319px" %)(((
70 70  Run 2 USAR 2: used to explore building and search for objects and victims
71 71  
... ... @@ -73,7 +73,9 @@
73 73  )))|(% style="width:329px" %)(((
74 74  Run 2 USAR 1: OWL missed victim and entire floor. Object detection initially disabled.
75 75  
76 -
82 +{{warning}}
83 +TODO: expand these issues with questionnaire answers
84 +{{/warning}}
77 77  )))|(% style="width:329px" %)Unreliable
78 78  |(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[Robot Arm (for ANYmal)>>Main.c\. Technology.Robot Arm (for ANYmal).WebHome]] |(% style="width:319px" %)(((
79 79  Run 1 USAR 1: remove beam blocking door
... ... @@ -81,8 +81,10 @@
81 81  Run 3 USAR 1: Used to close a valve
82 82  
83 83  
84 -)))|(% style="width:329px" %) |(% style="width:329px" %)Yes
85 -|(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[SNAKE>>Main.c\. Technology.SNAKE.WebHome]] |(% style="width:319px" %)Run 3 USAR 1: Used to search for victims in a narrow space|(% style="width:329px" %) |(% style="width:329px" %)?
92 +)))|(% style="width:329px" %){{warning}}TODO: expand these issues with questionnaire answers{{/warning}}(((
93 +
94 +)))|(% style="width:329px" %)Yes
95 +|(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[SNAKE>>Main.c\. Technology.SNAKE.WebHome]] |(% style="width:319px" %)Run 3 USAR 1: Used to search for victims in a narrow space|(% style="width:329px" %){{warning}}TODO: expand these issues with questionnaire answers{{/warning}}|(% style="width:329px" %)?
86 86  
87 87  === 3. Incident 2 way of working ===
88 88  
... ... @@ -90,36 +90,72 @@
90 90  |(% style="width:166px" %)Run 1, USAR team 1|(% style="width:442px" %)(((
91 91  The outdoor drone provides the Team Leader with an initial assessment of the incident scene. The Team Leader uses this information to deploy USAR teams and robotic assets; ANYmal supports by investigating and clearing hazards, after which the USAR teams gain access, stabilize the structure, locate victims, and perform rescues while continuously reporting progress to the Team Leader.
92 92  )))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] 
93 -Although the robot seems to have had a very limited role here and not have explored much.
103 +Although the robot didn't explore inside.
94 94  |(% style="width:166px" %)Run 2, USAR team 2|(% style="width:442px" %)(((
95 95  The OWL indoor drone first explores the collapsed structure and streams its findings to the Team Leader. Based on this reconnaissance, the USAR team enters, searches for victims, responds to emergencies, and conducts rescues while the Team Leader maintains situational awareness through the digital support system.
96 96  )))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
97 97  |(% style="width:166px" %)Run 3, USAR team 1|(% style="width:442px" %)(((
98 98  The outdoor drone assesses the scene, SNAKE inspects the interior, and ANYmal handles hazardous intervention tasks. Using the information gathered by these robotic systems, the USAR team enters, locates victims, and performs the rescue while the Team Leader maintains overall situational awareness and coordination.
99 -)))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]
109 +)))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] 
110 +Although the robot didn't explore inside.
100 100  
101 101  == 4. Raw results ==
102 102  
103 -**Results for Incident 2, Run 1, USAR team 1**
114 +=== Results for Incident 2, Run 3, USAR team 1 VS baseline USAR team 1 ===
104 104  
116 +Analysis of questionnaire results for members with ID: UT13, UT11, UT12, UTL1
117 +
118 +
119 +|=Scenario|=Role|=Participants
120 +|Baseline|USAR Team Member|3
121 +|Baseline|USAR Team Leader|1
122 +|Run 3|USAR Team Member|3
123 +|Run 3|USAR Team Leader|1
124 +|Run 1|USAR Team Member|3
125 +|Run 1|USAR Team Leader|1
126 +
127 +
128 +
129 +[[image:1785167968429-388.png||height="390" width="821"]]
130 +
131 +[[image:1785167981571-844.png||height="419" width="815"]]
132 +
133 +[[image:1785167994552-876.png||height="329" width="767"]]
134 +
135 +
136 +These results show that in run 1 workload, trust and situation awareness were significantly lower compared to the baseline, with decision making and safety confidence differing only slightly from the baseline. 
137 +Run 3 showed an improvement in decision making (specified as right info was available and decisions were timely) compared to the baseline and run 1.
138 +
139 +On average both runs had a lower workload (-25%), trust (-33%), situation awareness (-11%), slightly lower safety confidence (-4%) and improved decision making (+8%).
140 +
141 +Note: Run 3 also functioned as a demonstration, which increased speed of decisions and the scenario as a whole. Also, participants performed incident 2 in 3 runs in the same teamcomposition and building layout, such that participants giving them an unfair advantage. These factors might have influenced the results.
142 +
143 +
144 +=== 5. Results connected to Design Pattern measures ===
145 +
105 105  Note: The measures are defined in [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]]  and [[x. Measures>>3\. Evaluation.Measures.WebHome]]
106 106  
107 107  |(% style="width:376px" %)**Measure**|(% style="width:225px" %)**Results**|**Connected Claim**|**Claim proven?**
108 -|(% style="width:376px" %)Safe path chosen|(% style="width:225px" %) |CL01 Improve safety for responders |
109 -|(% style="width:376px" %)Decision quality|(% style="width:225px" %) |CL01 Improve safety for responders |
110 -|(% style="width:376px" %)Near-incident and incident reports|(% style="width:225px" %) |CL01 Improve safety for responders |
111 -|(% style="width:376px" %)Reported hazards, victims, layout correctness, based on robot intel before FR entry|(% style="width:225px" %) |CL02 Increased SA|
112 -|(% style="width:376px" %)How optimal was the FR path|(% style="width:225px" %) |CL02 Increased SA|
113 -|(% style="width:376px" %)Reported hazards, victims, layout correctness, after FR entry|(% style="width:225px" %) |CL02 Increased SA|
114 -|(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %) |CL03 improved mission effectiveness|
115 -|(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %) |CL04 acceptable workload|
116 -|(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %) |CL04 acceptable workload|
117 -|(% style="width:376px" %)?|(% style="width:225px" %) |CL05 Trustworthy SA|
118 -|(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %) |CL06 Improved FR health|
119 -|(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %) |CL06 Improved FR health|
120 -|(% style="width:376px" %)Total mission completion time|(% style="width:225px" %) |CL07 Degraded mission efficiency|
121 -|(% style="width:376px" %)First responder time inside|(% style="width:225px" %) |CL07 Degraded mission efficiency|
149 +|(% style="width:376px" %)Safe path chosen|(% style="width:225px" %)No FR path data available.|CL01 Improve safety for responders |Unknown
150 +|(% style="width:376px" %)Decision quality|(% style="width:225px" %)Can we use decision making questions from questionnaire for this?  |CL01 Improve safety for responders |Unknown
151 +|(% style="width:376px" %)Near-incident and incident reports|(% style="width:225px" %)No incidents in baseline or run. 
152 +No near-incidents data available.|CL01 Improve safety for responders |Unknown
153 +|(% style="width:376px" %)Reported hazards, victims, layout correctness, based on robot intel before FR entry|(% style="width:225px" %)No ground truth data available.|CL02 Increased SA|Unknown
154 +|(% style="width:376px" %)How optimal was the FR path|(% style="width:225px" %)No FR path or ground truth data available.|CL02 Increased SA|Unknown
155 +|(% style="width:376px" %)Reported hazards, victims, layout correctness, after FR entry|(% style="width:225px" %)No FR path or ground truth data available.|CL02 Increased SA|Unknown
156 +|(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %)Questionnaire indicates increased workload of 25%. But not literature reviewed questionnaire and runs not clean. |CL03 improved mission effectiveness|Indication says strong no, but data unreliable.
157 +|(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %)Can we use trust questions from questionnaire for this? They did not target robots specifically.|CL04 acceptable workload|Unknown
158 +|(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %)Can we use decision making questions from questionnaire for this?  |CL04 acceptable workload|Unknown
159 +|(% style="width:376px" %)Trust in SA|(% style="width:225px" %)Questionnaire indicates decreased trust of -33%. But not literature reviewed questionnaire and runs not clean|CL05 Trustworthy SA|Indication says strong no, but data unreliable.
160 +|(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %)No health data available.|CL06 Improved FR health|Unknown
161 +|(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %)Do we have this?|CL06 Improved FR health|Unknown
162 +|(% style="width:376px" %)Total mission completion time|(% style="width:225px" %)Do we have this? Compared to baseline?|CL07 Degraded mission efficiency|Unknown
163 +|(% style="width:376px" %)First responder time inside|(% style="width:225px" %)Do we have this? Compared to baseline?|CL07 Degraded mission efficiency|Unknown
122 122  
165 +(% class="wikigeneratedid" %)
166 +(Reliable) data missing to evaluate all claims for the usecase and as such team design pattern.
167 +
168 +
123 123  === 5. Pros and cons for TDP ===
124 124  
125 125  [[TDP2 — Robots First, Then Humans Protocol>>2\. Specification.Requirements.TDP2 — Robots First, Then Humans Protocol.WebHome]]
1785164456543-963.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +146.4 KB
Content
1785164494950-855.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +162.9 KB
Content
1785164507435-933.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +108.5 KB
Content
1785167243750-606.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +149.0 KB
Content
1785167259751-744.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +178.2 KB
Content
1785167267780-382.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +115.0 KB
Content
1785167968429-388.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +161.2 KB
Content
1785167981571-844.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +211.8 KB
Content
1785167994552-876.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.TjallingHaije
Size
... ... @@ -1,0 +1,1 @@
1 +168.8 KB
Content