Project homeWiki source code of k. Design Patterns Evaluation SFT1
Version 3.4 by Tjalling Haije on 2026/07/27 14:33
Hide last authors
| author | version | line-number | content |
|---|---|---|---|
 |
2.2 | 1 | == Approach == |
| |
1.1 | 2 | |
| |
2.2 | 3 | The goal for the human-machine teaming framework is to (among others) develop human-machine teaming patterns that can be used to use technology |
| 4 | |||
| 5 | **Design patterns of interest: ** | ||
| 6 | |||
| |
1.1 | 7 | * [[TDP2 — Robots First, Then Humans Protocol>>2\. Specification.Requirements.TDP2 — Robots First, Then Humans Protocol.WebHome]] |
| 8 | * [[IDP1 — Role-Based, Context-Specific Information Distribution>>2\. Specification.Requirements.IDP1 — Role-Based, Context-Specific Information Distribution.WebHome]] | ||
| 9 | |||
| |
2.2 | 10 | **Approaches for evaluating DPs:** |
| |
1.1 | 11 | |
| 12 | 1. Make new usecases based on how the FR actually did the TDP and IDP with the robots with claims and check those with the raw data. | ||
| 13 | 1. Match the premade usecase [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] with the incident that fits best, and evaluate the claims for the TDP and IDP of interest using results from that incident. | ||
| 14 | |||
| 15 | Choosing approach 2 because it has been prepared better, and helps in improving the evaluation approach for SFT 2 by identifying what info we missed. | ||
| 16 | |||
| 17 | |||
| 18 | == Design Patterns Evaluation == | ||
| 19 | |||
| 20 | Approach: | ||
| 21 | |||
| |
1.5 | 22 | 1. Figure out in which incidents TDP2 and IDP1 were used |
| |
1.4 | 23 | 1. Check if the tech worked as required (or good enough to be of value) in those incidents |
| 24 | 1. Figure out which of those incidents correspond most with [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] | ||
| 25 | 1. Try to fill in [[j. SFT1 evaluation templates >>3\. Evaluation.SFT1.g\. Prototype .WebHome]]from the raw results, which | ||
| 26 | 1. Check for each of the claims the results for the connected measures > Evaluates UC | ||
| 27 | 1. Check for each of the pros/cons for the Design Pattern if they are validated > Evaluates Design Pattern | ||
| 28 | 1. Adjust the Design Patterns based on the results and/or their expected pros/cons. | ||
| 29 | 1. Update how the users should use the technology and/or the technology itself to achieve the objectives. | ||
| 30 | 1. Note missing data that should be included in the evaluation next time. | ||
| |
1.5 | 31 | |
| |
2.13 | 32 | === 1. Design Patterns per Incident === |
| |
1.5 | 33 | |
| |
2.6 | 34 | |(% style="width:98px" %)**Incident**|(% style="width:396px" %)**Incident Description**|(% style="width:150px" %)**Tech used**|(% style="width:254px" %)**Design Patterns** |
| |
2.3 | 35 | |(% style="width:98px" %)Incident 1|(% style="width:396px" %)((( |
| 36 | A collision caused by a major explosion has left casualties trapped in vehicles and created a hazardous chemical leak from a truck. The goal of the fire service is to rescue the casualties, identify and contain the chemical hazard, and safely manage the incident. | ||
| |
2.6 | 37 | )))|(% style="width:150px" %)?|(% style="width:254px" %)? |
| |
2.9 | 38 | |(% style="width:98px" %)**Incident 2**|(% style="width:396px" %)((( |
| |
2.3 | 39 | An explosion has caused the partial collapse of multiple industrial and office buildings, leaving an unknown number of people trapped or missing. The goal of the Urban Search and Rescue (USAR) teams and drone team is to assess the damage, locate and rescue victims, manage responder safety during a secondary collapse, and clear all affected buildings. |
| 40 | )))|(% style="width:150px" %)((( | ||
| |
2.6 | 41 | * [[ASTRAL RTK Drone>>Main.c\. Technology.ASTRAL RTK Drone.WebHome]] |
| 42 | * [[OWL Indoor Drone>>Main.c\. Technology.OWL Indoor Drone.WebHome]] | ||
| 43 | * [[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[SNAKE>>Main.c\. Technology.SNAKE.WebHome]] and with [[Robot Arm (for ANYmal)>>Main.c\. Technology.Robot Arm (for ANYmal).WebHome]] | ||
| 44 | * [[ASTRIAL C3I IMS>>Main.c\. Technology.ASTRIAL C3I IMS.WebHome]] | ||
| 45 | * [[WEARIN Wearables>>Main.c\. Technology.WEARIN Wearables.WebHome]] | ||
| 46 | * [[SYSNAV Personal Location Device>>Main.c\. Technology.SYSNAV Personal Location Device.WebHome]] | ||
| |
2.3 | 47 | )))|(% style="width:254px" %)((( |
| 48 | * [[TDP2 — Robots First, Then Humans Protocol>>2\. Specification.Requirements.TDP2 — Robots First, Then Humans Protocol.WebHome]] | ||
| 49 | * [[IDP1 — Role-Based, Context-Specific Information Distribution>>2\. Specification.Requirements.IDP1 — Role-Based, Context-Specific Information Distribution.WebHome]] | ||
| |
2.8 | 50 | * Others? |
| |
2.3 | 51 | ))) |
| 52 | |(% style="width:98px" %)Incident 3 |(% style="width:396px" %)((( | ||
| 53 | A blast has left victims stranded on the upper floors of a high-rise residential building. The goal of the fire crew and drone team is to assess the building, locate and safely rescue the trapped occupants using ladder access, and clear the building. | ||
| |
2.5 | 54 | )))|(% style="width:150px" %)((( |
| |
2.6 | 55 | ? |
| 56 | )))|(% style="width:254px" %)? | ||
| |
2.3 | 57 | |(% style="width:98px" %)Incident 4|(% style="width:396px" %)((( |
| 58 | A gas explosion and fire in a four-apartment residential building have left multiple occupants trapped and created a hazardous environment for firefighters conducting search, rescue, and firefighting operations. The goal of the fire crews is to extinguish the fire, rescue all occupants, manage firefighter safety during an emergency involving a downed rescuer, and clear the building. | ||
| |
2.4 | 59 | )))|(% style="width:150px" %)((( |
| |
2.6 | 60 | ? |
| 61 | )))|(% style="width:254px" %)? | ||
| |
1.5 | 62 | |
| |
2.9 | 63 | Based on the above list the next steps zoom in on incident 2. |
| 64 | |||
| |
2.13 | 65 | === 2. Incident 2 tech evaluation === |
| |
2.9 | 66 | |
| |
2.10 | 67 | |(% style="width:247px" %)**Tech**|(% style="width:319px" %)**How was it used**|(% style="width:329px" %)**Issues**|(% style="width:329px" %)**Did it work sufficiently?** |
| |
3.2 | 68 | |(% style="width:247px" %)[[ASTRAL RTK Drone>>Main.c\. Technology.ASTRAL RTK Drone.WebHome]] (outdoor)|(% style="width:319px" %)Baseline & Run 1-3: The drone team conducts an aerial assessment of the collapsed buildings, identifying structural risks, access points, and possible victim locations. |(% style="width:329px" %)((( |
| 69 | Run 1 USAR 1: Drone had connection issues which delayed its deployment. | ||
| 70 | |||
| 71 | {{warning}} | ||
| 72 | TODO: expand these issues with questionnaire answers | ||
| 73 | {{/warning}} | ||
| 74 | )))|(% style="width:329px" %)Yes | ||
| |
2.10 | 75 | |(% style="width:247px" %)[[OWL Indoor Drone>>Main.c\. Technology.OWL Indoor Drone.WebHome]] |(% style="width:319px" %)((( |
| 76 | Run 2 USAR 2: used to explore building and search for objects and victims | ||
| |
2.9 | 77 | |
| |
2.10 | 78 | |
| 79 | )))|(% style="width:329px" %)((( | ||
| 80 | Run 2 USAR 1: OWL missed victim and entire floor. Object detection initially disabled. | ||
| |
2.9 | 81 | |
| |
3.2 | 82 | {{warning}} |
| 83 | TODO: expand these issues with questionnaire answers | ||
| 84 | {{/warning}} | ||
| |
2.10 | 85 | )))|(% style="width:329px" %)Unreliable |
| 86 | |(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[Robot Arm (for ANYmal)>>Main.c\. Technology.Robot Arm (for ANYmal).WebHome]] |(% style="width:319px" %)((( | ||
| 87 | Run 1 USAR 1: remove beam blocking door | ||
| |
2.9 | 88 | |
| |
2.10 | 89 | Run 3 USAR 1: Used to close a valve |
| |
2.9 | 90 | |
| |
2.10 | 91 | |
| |
3.2 | 92 | )))|(% style="width:329px" %){{warning}}TODO: expand these issues with questionnaire answers{{/warning}}((( |
| 93 | |||
| 94 | )))|(% style="width:329px" %)Yes | ||
| 95 | |(% style="width:247px" %)[[ANYmal Robot>>Main.c\. Technology.Humanoid Robot.WebHome]] with [[SNAKE>>Main.c\. Technology.SNAKE.WebHome]] |(% style="width:319px" %)Run 3 USAR 1: Used to search for victims in a narrow space|(% style="width:329px" %){{warning}}TODO: expand these issues with questionnaire answers{{/warning}}|(% style="width:329px" %)? | ||
| |
2.9 | 96 | |
| |
2.13 | 97 | === 3. Incident 2 way of working === |
| |
2.9 | 98 | |
| |
2.13 | 99 | |(% style="width:166px" %)**Run & team**|(% style="width:442px" %)**How did the first responders use the tech**|(% style="width:238px" %)**Similar to what usecase** |
| 100 | |(% style="width:166px" %)Run 1, USAR team 1|(% style="width:442px" %)((( | ||
| |
2.11 | 101 | The outdoor drone provides the Team Leader with an initial assessment of the incident scene. The Team Leader uses this information to deploy USAR teams and robotic assets; ANYmal supports by investigating and clearing hazards, after which the USAR teams gain access, stabilize the structure, locate victims, and perform rescues while continuously reporting progress to the Team Leader. |
| 102 | )))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] | ||
| |
3.1 | 103 | Although the robot didn't explore inside. |
| |
2.13 | 104 | |(% style="width:166px" %)Run 2, USAR team 2|(% style="width:442px" %)((( |
| |
2.11 | 105 | The OWL indoor drone first explores the collapsed structure and streams its findings to the Team Leader. Based on this reconnaissance, the USAR team enters, searches for victims, responds to emergencies, and conducts rescues while the Team Leader maintains situational awareness through the digital support system. |
| 106 | )))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] | ||
| |
2.13 | 107 | |(% style="width:166px" %)Run 3, USAR team 1|(% style="width:442px" %)((( |
| |
2.11 | 108 | The outdoor drone assesses the scene, SNAKE inspects the interior, and ANYmal handles hazardous intervention tasks. Using the information gathered by these robotic systems, the USAR team enters, locates victims, and performs the rescue while the Team Leader maintains overall situational awareness and coordination. |
| |
3.1 | 109 | )))|(% style="width:238px" %)[[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] |
| 110 | Although the robot didn't explore inside. | ||
| |
2.9 | 111 | |
| |
2.13 | 112 | == 4. Raw results == |
| |
2.10 | 113 | |
| |
2.14 | 114 | **Results for Incident 2, Run 1, USAR team 1** |
| |
2.10 | 115 | |
| |
2.14 | 116 | Note: The measures are defined in [[UC04.3: Detailed indoor exploration with ANYMAL (USAR)>>2\. Specification.b\. Use Cases.UC04\.0\: Detailed indoor exploration.UC04\.3\: Detailed indoor exploration with ANYMAL (USAR).WebHome]] and [[x. Measures>>3\. Evaluation.Measures.WebHome]] |
| |
2.10 | 117 | |
| |
2.17 | 118 | |(% style="width:376px" %)**Measure**|(% style="width:225px" %)**Results**|**Connected Claim**|**Claim proven?** |
| 119 | |(% style="width:376px" %)Safe path chosen|(% style="width:225px" %) |CL01 Improve safety for responders | | ||
| 120 | |(% style="width:376px" %)Decision quality|(% style="width:225px" %) |CL01 Improve safety for responders | | ||
| 121 | |(% style="width:376px" %)Near-incident and incident reports|(% style="width:225px" %) |CL01 Improve safety for responders | | ||
| 122 | |(% style="width:376px" %)Reported hazards, victims, layout correctness, based on robot intel before FR entry|(% style="width:225px" %) |CL02 Increased SA| | ||
| 123 | |(% style="width:376px" %)How optimal was the FR path|(% style="width:225px" %) |CL02 Increased SA| | ||
| 124 | |(% style="width:376px" %)Reported hazards, victims, layout correctness, after FR entry|(% style="width:225px" %) |CL02 Increased SA| | ||
| 125 | |(% style="width:376px" %)Workload: e.g. NASA-TLX|(% style="width:225px" %) |CL03 improved mission effectiveness| | ||
| 126 | |(% style="width:376px" %)Appropriate trust in technology: e.g. trust survey|(% style="width:225px" %) |CL04 acceptable workload| | ||
| 127 | |(% style="width:376px" %)Decision confidence: e.g. Retrospective decision quality rating|(% style="width:225px" %) |CL04 acceptable workload| | ||
| 128 | |(% style="width:376px" %)?|(% style="width:225px" %) |CL05 Trustworthy SA| | ||
| 129 | |(% style="width:376px" %)Health indicators such as HR, temperature, etc.|(% style="width:225px" %) |CL06 Improved FR health| | ||
| 130 | |(% style="width:376px" %)Health issues tackled in acceptable time|(% style="width:225px" %) |CL06 Improved FR health| | ||
| 131 | |(% style="width:376px" %)Total mission completion time|(% style="width:225px" %) |CL07 Degraded mission efficiency| | ||
| 132 | |(% style="width:376px" %)First responder time inside|(% style="width:225px" %) |CL07 Degraded mission efficiency| | ||
| |
2.10 | 133 | |
| |
2.19 | 134 | === 5. Pros and cons for TDP === |
| 135 | |||
| 136 | [[TDP2 — Robots First, Then Humans Protocol>>2\. Specification.Requirements.TDP2 — Robots First, Then Humans Protocol.WebHome]] | ||
| 137 | |||
| 138 | |**Pro / Con**|**Results**|**Pro / Con proven?** | ||
| 139 | | | | | ||
| 140 | | | | | ||
| 141 | |||
| |
1.5 | 142 |