Virtual Fixture Generation for Task Planning With Complex Geometries

Increasingly, robotics are assigned to complete real-world tasks where treating task surfaces such as simple primitives (cylinders, squares, etc.) is unrealistic. An illustrative and motivating example is the dismantling of complex piping, as shown in Fig. 1, bottom [1].

Virtual fixtures (VFs) [2] are designed to simplify the operator’s job. VFs are virtual constraints imposed on a task’s workspace, much like jigs used in machining or computer-aided design (CAD) assembly constraints. VFs can provide motion guidance, Guidance VF, or an exclusion volume, Forbidden Region VF. Semi-autonomous behaviors are easily defined using VFs. Artificial end effector (EEF) motion guidance/constraints demonstrate decreased operator mental load and increased efficiency [3]. Examples include obstacle avoidance and collision detection in multiple domains such as mobile manipulation [4,5], surgical [6,7], and nuclear [8,9] research areas. Environments studied also include cluttered home environments [5] and human anatomy variability [7]. However, robotic systems using VFs lack the flexibility to assist operators with process adjustments for complex, unique geometries [8,10,11].

Volumetric primitive VFs, constructed from system and sensor data (Fig. 1, top left and right), were previously utilized to simplify inspection tasks by removing spatial transform management from the operator’s mental load [12,13]. Other work generated point cloud Guidance VFs based on a set of parametric surfaces and task parameters [14]. Volumetric primitive and parametric surface VFs (Fig. 2) are useful in some cases but cannot be adapted to complex geometries from common data sources (CAD, polygonal meshes, sensor data, etc.). Simple superposition of primitives is not viable as it created irreconcilable conflicts during VF generation and operator interface interpretability. This research expands VF generation to include complex geometries from all common data sources. It provides operators with more expressive, intuitive VFs to assist with any task.

In addition to generating VFs, it is necessary to not increase the burden on the robotic operator. With primitives (Figs. 2(a), (b), (d), and (h)), guidance is often simple (left, right, up, down, in, and out), but VFs generated from complex objects cannot be easily mapped to directions and the meaning of such terms can be ambiguous and change with perspective. Guidance tools must also be generated automatically and be easy to use. Robotic systems are operated by trained technicians with little or no intervention from robotics experts. To make the system usable, the interface must address the correspondence problem and context switching where differences between operator and manipulator kinematics complicate motion planning. For example, a user commanding a robot’s joints using a keyboard for a drum inspection (Fig. 1, top left) must internally manage every corresponding cause and effect (arm presses button, the button turns joint, joint motion changes gripper location relative to the wall, etc.). Complex VFs would unnecessarily burden an operator’s mental load. Therefore, a VF interaction method must be part and parcel with their generation.

Section 2 summarizes related research efforts and confirms the need for the developed tools in the robotics domain. Section 3 presents the VF construction pipeline, including algorithms for mesh interpolation, verification, point-normal calculations, and VF layer generation. We then outline a visualization and reachability analysis tool using VFs generated for general complex geometries. Section 5 discusses VF generation verification from a variety of data sources. Section 6 summarizes feedback and performance data from operator experiments. User evaluation is critical to ensure more complex VFs reduce the operator’s burden instead of increasing it. Section 7 presents conclusions and outlines future work.

Similar efforts to this work were identified in other domains including, computation geometry [15–17], sensor analysis [18,19], and 3D reconstruction fields [20–22]. Many of these solve portions of the problem. Some common techniques constitute aspects of the generation pipeline, which are documented below. However, the conversion of multiple layers of point cloud-based Guidance VFs into a graph structure and pairing with a Forbidden Region VF for use with a 3D robot control interface is a unique approach to reducing an operator’s mental load and is the focus of this article.

In the robotics domain, Bowyer et al. [23] provide a recent and comprehensive review on VFs and active constraints by reviewing 120+ publications where the majority pertain to teleoperated or hands-on master-slave systems with haptic feedback. There are various VF generation techniques presented (Fig. 2) with a high percentage of computer-assisted surgery research efforts among topic publications. In several cases, the variability of the human body led researchers to create polygonal mesh Forbidden Region VFs from sensor data [6,7]. This procedure constrains the surgical tool location and protects sensitive tissue near the surgery site. For patient protection and other high-value operations, automated system failure requires personnel to complete the task manually [8], repair the equipment [24], or both. Bowyer et al. draw several conclusions:

• Flexible structures (Fig. 2 meshes, point clouds) are more expressive and, therefore, of greater use.

• Effectively generating constraint geometries is the most significant current challenge.

• Future constraint research will proceed toward a more complete working environment representation.

There are numerous examples in the literature using primitive-based VFs. For instance, DeJong et al. [24] applied VFs to facility piping size reduction with a saw. A planar Forbidden Region VF perpendicular to the pipe-restricted EEF motion to reduce cutting blade stresses (Fig. 1, top left) and minimize the blade damage. Another robotics system is designed to clean spherical vessel interiors and utilized VFs to improve teleoperation and enable semi-autonomous behaviors to reduce operator burden [25]. Khan and Hilton [10,11] discuss the benefits of noncontact fiber optic laser systems for volume reduction decommissioning tasks (Fig. 1, top right). This system lacked the intelligence to flexibly adjust to complex objects or piping arrangements and depended on linear cutting motions to accomplish tasks. Hilton et al. [11] noted that future nuclear facility decommissioning will require noncontact remote cutting techniques, which can intelligently adapt to complex geometries. Kruusamäe and Pryor [26] used primitive Forbidden Region VFs to augment a hand gesture robotic teleoperation interface. Operator evaluation showed the interface-enabled high-precision task completion with similar performance to using robot operating system (ROS) based [27] interactive markers. Another approach by Bi and Lang [28] presents an analysis of a robotic-coating process. This approach has similar goals but relies on the coating cone concept describing the volume between a robot with a coating tool, a paint gun, and the task surface.

Researchers [12] generated more complex, multilayer point cloud Guidance VFs, from geometric primitives and task parameters to enable directional command steps (“Left,” “Right,” “Up,” “Down,” “In,” and “Out”). The system was demonstrated on a container inspection task. Multilayer point cloud Guidance VFs and the navigation interface were also combined with motion planning methods to demonstrate size-reduction tasks [29]. Additional control buttons (“Add Pt,” “Remove Pt,” “Plan,” “Execute,” and “Clear”) provided operators the ability to construct size-reduction paths, Fig. 1, top right for example. The controller reduced the operator’s mental burden, but noted geometric primitives have insufficient expressiveness for most real-world applications (Fig. 1, bottom). Other investigations [14] employed parametric surfaces instead of geometric primitives to generate multilayer point cloud Guidance VFs. Results demonstrated correlations between surface concavity/offset distance and the need to control layer point cloud growth through cloud density analysis.

Broad applicability for the tasks and domains identified requires VF generation from parametric surfaces, polygonal mesh models, CAD files, and RGB-depth sensor data. The computational geometry literature offers many tools to analyze and alter these file types. Also, the robotic literature demonstrates a clear need for general and flexible VF generation methods to provide broad and expressive environmental representations. Finally, since multilayer VFs generated from complex shapes have not been evaluated, it will be necessary to ensure their increased complexity does not itself unduly burden potential users. The usability study developed is based on the principles outlined by Nielsen and Landauer [30], which suggests a series for smaller user studies for newly developed tools or tools currently under development.

VF generation is conceptualized as a data pipeline traversing multiple algorithms (Fig. 3). Required data inputs are a task surface and task parameters. Guidance VF elements are calculated based on minimum and maximum distances (d_min, d_max) and intralayer and interlayer distances (d_intra, d_inter) task parameters (Fig. 4). The interlayer distance, d_inter, defines the distance between Guidance VF layers, while intralayer resolution, d_intra, determines the distance between points within a Guidance VF layer. The pipeline output is a graph structure containing layers of vectors, which are the inverse of the task or object normal at a surface location (Fig. 4, right).

The role of each element in the pipeline is as follows:

• Input: STL input and sources are detailed in Sec. 3.1.

• Interpolate: Input interpolation and conversion is discussed in Sec. 3.2.

• Extend and Interpolate: The creation of VF layers by extending and interpolating the data is described in Sec. 3.3. If sensor data or reconstructed point clouds are used, this is the where it enters the pipeline.

• Convert: VF layer conversion into a graph structure is detailed in Sec. 3.4.

The elements, input, output, and intermediary data structures are detailed here. While some actions utilized the referenced and established algorithms, several new components were developed and are discussed in more depth.

The STL triangle side lengths of a valid mesh T_surf are compared to the desired task intralayer distance, d_intra. Triangles with sides exceeding d_intra are subdivided iteratively until appropriately sized to have sufficient fidelity for task execution. The triangles are divided by defining points at each side’s midpoint, which divides the triangle into four geometrically similar triangles. This primal approximating dissection method was chosen to maintain original surface points, retain fine surface features, and increase point dispersion over simpler triangle bisection methods. More complex methods for mesh subdivision are available, including parameterization-based or surface-oriented remeshing techniques [15] and nonlinear subdivision [37,38], but were deemed unnecessary.

Point normals (Eq. (1)) are calculated using the normalized weighted average of the triangle’s $n^$ for triangles in which the point is a vertex. Each triangle normal is weighted by the ratio of the triangle’s area to the area of all triangles in which the point is co-located (Algorithm 1, Fig. 5). Thus, the Delaunay neighborhood is utilized similarly to Ref. [16], but the STL file provides the normal information. The integration of triangle area weighting better accommodates triangle size variations than previous methods as normals will be shifted toward the largest triangle in the neighborhood (Fig. 5). It also avoids recalculating surface information as with plane fitting approaches. Plane fitting algorithms function best with point distribution consistency [16], which may be lacking in the surface mesh and may require additional information such as a viewpoint [18]. If future algorithm development requires additional surface information, such as curvature, a plane fitting method [17,19] integration into the Task VF generation pipeline is an option.

During preliminary investigations, nonmanifold inputs were found to disrupt the Task VF generation pipeline. These nonmanifold inputs took the form of inverted $Tsurf(i)⋅n^$ regions from the Visualization Toolkit (VTK) [39] and duplicated polygons but with inverted surface normals from 3DWarehouse [35] and Sketchup [40]. Two straightforward checks were implemented to warn operators of defective meshes. The first check area weights and averages $Tsurf(i)⋅n^$ over the entire mesh, which should output approximately zero. However, if mesh regions are inconsistent, the average $n^$ will exceed the threshold, ɛ₁, and the operator is warned to examine the mesh. A secondary check compares each calculated $Psurf⋅n^$ to another threshold, ɛ₂, which is set to a small value (Algorithm 1). If $|Psurf⋅n^|$ is less than the threshold, the mesh is likely nonmanifold. If inconsistencies are detected, the operator is warned and the mesh can be reconstructed using common techniques [15], but reconstruction is outside the scope of this work.

Once the validation checks, interpolation, and conversion to a point cloud with normals (PCN) are completed, the task surface PCN is voxelized. Voxel filtering discretizes space into boxes of a provided size, and all points present within the same box are approximated a point located at their centroid [18]. The discretization box is a cube with side lengths based on a small percentage of the task d_intra to maintain high surface density without duplicate points.

The objective is to maintain the direction and distance to fine surface features and create a PCN of extended surface features. It is important to note that VF PCN layers are not required to form manifold surfaces, which is one of the advantages of this VF generation methodology. This distinction mainly applies to interior features’ extended points. Figure 6 illustrates possibilities where internal features composed of noncurved, discontinuous surfaces creates ambiguities in how layers should be generated. At close distances to the surface, they still look similar to the task surface (Fig. 6, layers 1 and 2). However, as the distance increases, the calculated PCN clusters in the center of the interior feature (Fig. 6, layer 3). Eventually, the VF PCN layer will approach the opposite side of the feature. In such cases, surface reconstruction methods would smooth away important information (Fig. 6, bottom black circle) or result in a nonmanifold geometry.

Complications can arise as d_layer increases and $Psurf(i)⋅n^$ are extended due to curvature-related convergence/divergence phenomena, discontinuities in convex external features, or the presence of internal features. Convex regions have lower $Tsurf(i)⋅n^$ densities for VF layers further from the surface and thus require interpolated points to maintain VF intralayer resolution. After extending $Tsurf(i)⋅n^$⁠, point neighborhood information is used for VF layer interpolation. To limit extraneous point creation—and thus extended computation times—a minimum interpolation radius, r_min, is defined. The r_radius is increased proportionally to d_layer, r_prop. Thus, a percentage of the VF layer surface area is maintained as distance increases while avoiding interpolating through thin surfaces. Distance proportional r_prop neighborhoods provide a more generic neighborhood selection process than hand-tuning k_nearest for each model. Once the local neighborhood is selected, the distance between the current point and each neighbor is calculated. If the Cartesian distance is between r_prop and r_min, a new P_vf is linearly interpolated and assigned the average of the two point’s $n^$⁠.

Concave and interior surface features will contain regions of converging $Tsurf⋅n^$⁠. For example, if the concave region is an interior slot, surface normals will intersect (Fig. 6). As with the task PCN, voxel filtering is one way to correct VF layer resolution. In this case, VF layer resolution needs to be limited, so the discretization box is a cube with side lengths proportional to d_intra. Interpolation and voxelization are executed only once per VF layer instead of iteratively to limit surface distortions in the resulting PCN.

Once constructed, the number of neighbors within a d_intra based r_radius volume of P_vf(i) is calculated to check VF layer resolution. The number of neighbors is averaged over an entire VF layer to ensure the VF layer resolution is maintained as d_layer increases. The VF PCN layer can be used as a Guidance VF or Forbidden Region VF. For a Guidance VF, normal vectors are determined by inverting each $Psurf(i)⋅n^$ to face toward the task surface (Eq. (3)). Utilizing Guidance VFs generated from PCN is a previously uninvestigated approach due to complexities related to data storage, visualization, and utilization. For a Forbidden Region VF, the PCN layer is converted into a polygonal mesh, or an exclusion volume is placed around each point similarly to previous approaches [41,42]. This process allows VF PCN layers to define a Forbidden Region VF offset from a complex geometry (Fig. 7, left). This implementation is particularly useful for noncontact tasks where d_min is small. Forbidden Region VFs also handle cases where d_max is greater than an interior space by removing regions altogether. As previously mentioned, VF PCN layer generation can result in nonmanifold surfaces, and Forbidden Region VFs conversion is only recommended when the ratio of task surface feature size to layer distance is high.

Locations in a global coordinate frame (or poses) in a Guidance VF layer must be traversable in the sense that a robot or other agent must be able to move from one to another. We note that a pose generally includes both a position and orientation. Simple commands for relative motion concepts (“up,” “right,” etc.) are insufficient to guide a robot or other agent. Thus, information is stored in a bidirectional graph structure, which also enables graph search tools such as Breadth First, Depth First, and Uniform Cost Search in addition to Dijkstra’s Shortest Paths and other path algorithms (Fig. 7). Task VFs include both the Forbidden Region VF task surface model and the Guidance VF bidirectional graph, which enables tasks for complex geometries (Fig. 1, top left).

The Task VF graph is constructed by converting each Guidance VF layer’s PCN into graph vertices. Each vertex contains its unique id, Guidance VF layer, and pose in the Task VF frame but could include other data such as curvature or local neighborhood information. Vertices are linked together using an adjacency list. Edge weights record Cartesian distance information but could employ a weighting of multiple metrics such as Cartesian and angular distances. Vertices are linked to all others in a Guidance VF layer but only the Cartesian closest vertex in neighboring Guidance VF layers (Fig. 7). Linking between Guidance VF layers can result in more than one interlayer edge. For example, in Fig. 6, converging $n^$ would cause multiple vertices in the blue layer to link to the same vertex in the green layer or vice versa. Using these linking methods simplifies visualization, operator intralayer and interlayer navigation, and future automated path planning.

Once the Task VF has been generated as stated earlier, the intention of conceptually directional actions (“Left,” “Up,” “Out,” etc.) are ambiguous. Even for a simple spherical model, commanded motions over the “top” (which is also ambiguous) yields different relative action concepts and final viewpoint than moving around the “side.” Therefore, to visualize and test operator interpretability of Task VF generation pipeline output, a new control interface was constructed. The design of the new control interface followed an analysis of 8 of the 15 DARPA Robotics Challenge Trial teams. It concluded integrating sensor/input windows, and more autonomy led to better performance [43]. Our interface uses ROS’s 3D visualization environment, RViz. Separate directional buttons [29] were replaced with interactive markers in the same RViz window presenting sensor and VF data. A single interface displays the task Forbidden Region VF surface and the Guidance VF layers and allows graph navigation. The interface maintains and displays the operator’s current Task VF vertex and neighboring vertices, including those in other Guidance VF layers.

The task surface and every Task VF vertex are represented as interactive markers. This allows a user to move the task frame (red: X, green: Z, blue: Z) around the virtual workspace (Fig. 8, bottom right) using the arrows. As the user moves the task surface around the workspace, they can also interact with specific Task VF vertices as shown by the colored ellipsoid markers in Fig. 8. Each Task VF interactive marker includes a list of menu accessible options:

• Make Current Pose: Make the selected Task VF vertex the current Task VF vertex and update the nearest neighbor graph and interlayer linking markers.

• Update Pose: Update the Task VF vertex information with the current interactive marker pose.

• Move to Pose: Plan and execute EEF motion to the current interactive marker pose.

• Add Pose to Path: Add the current interactive marker vertex to a graph for path motion.

• Remove Pose From Path: Remove the current interactive marker vertex from a graph for path motion.

• Test Path: Pass the path graph to Descartes planning and execution package [44].

• Clear Path: Clear the path graph of path.

The user’s current Task VF vertex is displayed as a white marker, which can be modified when the operator clicks Make current pose. All Task VF interactive markers have a fixed location in the task frame but can be rotated around their $n^$ (Fig. 8, white circle bottom left, bottom middle). A default orientation is updated by clicking Update pose in the menu. Neighboring vertices representing interlayer linking are displayed with blue markers, and moving to between layers updates the Guidance VF layer (Fig. 8) visualization.

The developed control and visualization interface was first considered for assistance with spatially discrete noncontact tasks such as visual inspection. Therefore, the first required control was EEF movement to the Task VF pose desired. When the operator commands a Move to pose, the motion will be successful if the pose has an IK solution and the manipulator is unobstructed by the planning scene. The tedium of manually testing poses led to the integration of reachability analysis into the Task VF visualization interface. Reachability analysis utilizes well-established IK analytical tools available as part of MoveIt! [45,46] to verify a collision-free motion plan from the current location to the Task VF pose. Task VF vertices are green (reachable) or red (not reachable) as shown in Fig. 8 for a given robot or other kinematically constrained device. The interface also aggregates reachability analysis information and displays the number and percentage of Task VF vertices along with the current task frame pose (Fig. 8, top). Each visualization is controlled through the RViz window (Fig. 8, top left side). These simple but necessary analytical tools are encapsulated in the manipulator to task transform tool (MTTT), allowing operators to evaluate possible task locations quickly. It is independent of the Task VF generation pipeline and, therefore, utilizes a bidirectional graph of poses from any source.

Task VF generation pipeline inputs include surface models from multiple sources including parametric surfaces, meshes, and point cloud data. Thus, a broad range of input data sets must be evaluated to assert the VF pipeline thst will be robust for general, real-world, complex geometries. Therefore, a large variety of each input data type were tested to identify incorrect surface $n^$⁠, and that VF layers are generated, resolutions are reasonable, and the Forbidden Region VF and Guidance VF layers are successfully converted into a Task VF graph.

The Task VF generation pipeline mesh $n^$ and point $n^$ and thresholds were set to ɛ₁ = 1e − 5, ɛ₂ = 1e − 8, respectively. At these thresholds, a bowl model [47], known to be nonmanifold, warned the operator 100% of points were above the ɛ₁ threshold (Algorithm 1). None of the superellipsoids generated operator warnings as expected. Seven of the eight non-manifold supertoroids generated operator warnings to evaluate the mesh. The eighth model was nonmanifold but symmetric, so it passed the test. Thus, the supertoroid surfaces should be reconstructed to contain correct surface information before being Task VF generation pipeline input but testing did validate the check for inverted $n^$⁠. The pipeline does not (and should not) correct ill-formed inputs, but it correctly informs users when they are present.

Previously, Task VF generation parameters were set d_min = 0.05 m, d_max = 1.05 m, d_intra = 0.5 m, and d_inter = 0.5 m to provide three Guidance VF layers. Repeating Task VF tests provides a comparison with previous pipeline algorithms. VF layer calculations reinforce the expectation that point the cloud growth is proportional to surface concavity (Fig. 11). Convex models, N_xy = N_z = 0 (Fig. 9, top left), show significant growth in the number of vertices with increasing distance, but the concave models N_xy = N_z = 4 (Fig. 9, bottom left) show less growth (Fig. 11). Supertoroids exhibit similar trends but are more erratic due to inverted $n^$ (Fig. 11).

Checking the selected superellipsoids and supertoroids VF layer resolutions provides several important observations (Fig. 12). Superellipsoid resolution varies between models but is maintained between VF layers. The erratic supertoroid results highlight the effect of inverted $n^$⁠. Superellipsoid and supertoroid VF layer sizes are smaller than previous results [14] but layer resolution is maintained.

Crowdsource-selected complex polygonal meshes from Thingiverse [34] provided additional Task VF generation pipeline input for evaluation. Eleven individuals each provided two to four models (32 total). Seven were eliminated as unsuitable as they contained multiple meshes or were impractically large. While Task VF generation completes for large STL files, the resulting graph structure computational strained MTTT’s RViz environment. The remaining 25 models (Table 1) represent a large variety of items, including cable holders (70549), desk figurines (906951), tools (1187995), and phone stands (2120591).

Point and mesh $n^$ thresholds were set to ɛ₁ = 1e − 5, ɛ₂ = 1e − 8, respectively, for mesh testing. Only one of the 25 models (3119803) was flagged by input data checks. Visual inspection found inverted $T(i)⋅n^$ on two holes (Fig. 15), demonstrating the effectiveness of this straightforward technique for finding mesh inconsistencies.

These parameters were tested on the parametric surfaces, and three variations (⁠$dintra=10*T¯(i)side,15*T¯(i)side,20*T¯(i)side$⁠) were applied to the polygonal mesh testing pool are shown in the red, green, and blue point clouds in Fig. 18. Average VF layer size grew at roughly an N = C * dist² rate, where N is the number of points and C is a constant (Fig. 16). This result was expected since spherical surface area, 4 * π * r², grows at a r² rate. VF layer resolution evaluation used the same parameters. The results show a correlation between superellipsoid and polygonal mesh VF layer growth and resolution, suggesting that estimating task parameters based on $T¯(i)side$ is an effective approach. Average VF layer resolution does slowly decrease for all input types (Fig. 17). Resolution decay at high d_layer to model size ratios could lead to cases where d_intra is unattainable. In such cases, authors recommend higher interpolation of the original task surface over multiple interpolation and voxelization iterations during the generation of a VF layer.

Four models were randomly selected for additional investigation (Fig. 18). Visual inspection of the first two Guidance VF layers displayed the PCNs oriented back toward the surface at regular intervals. The first Guidance VF layers are sparse, suggesting that a lower d_intra might be necessary for task completion. Parameters are easily adjusted to task-specific Task VF generation parameters from the generalized, model-based, parameters used in this evaluation.

3D LIDAR data were gathered in a mock setting constructed at UT Austin of a nuclear facility’s exhaust tunnel (Fig. 19, top), where a mobile manipulator must periodically perform visual and radiation surveys [48]. A section of the data around the large pipe was segmented for evaluation. Testing parameters varied from d_min = 0.05 m to d_max = 1.05 m and d_intra = 0.1 m, 0.5 m. Comparing results to superellipsoid and supertoroid calculations shows slower VF layer growth when d_intra = 0.5 m due to the extension of a partial surface instead of an enclosed 3D object (Fig. 20). Decreasing d_intra raises the layer size above superellipsoid and supertoroid levels.

Virtually inserting a Yaskawa SIA20 into MTTT RViz interface provides reachability information and the ability to move the manipulator tool among the generated Task VF (Fig. 19, bottom). This combination demonstrates the applicability of the Task VF generation pipeline to real-time sensor data for open-world scenarios.

The proposed Task VFs can be generated for virtually any surface but are inherently more complex than previous VFs. So even though Task VFs are automatically generated and more broadly applicable, the additional complexity may increase operator mental load instead of decreasing it. Thus, a preliminary experimental evaluation was completed to determine operator interpretability and MTTT usefulness. After some initial usability reviews, an experiment was setup following some of the principles outlined by Nielsen and Landauer [30]. Their work presents the idea that most usability problems can be found with a relatively small number of users, approximately 5, and the more users tested the less you learn from each. However, since the goal of the MTTT interface for Task VFs is to be usable by diversely experienced operators, none all the way to expert, a goal of ten users was identified.

The experiment was performed in a virtual workspace consisting of the ROS and RViz interfaces for a Yakasawa SIA20 manipulator with an IPG Photonics Compact Cutting Head [49] laser cutter (Fig. 21). This interface was used in the previous work along this research thread, which looked into pairing volumetric primitive VFs with laser cutting [29]. To keep the reachability analysis, calculations below 10 s required Guidance VF layers containing only ∼50 poses. Operator Task VF evaluation surfaces include a subset of the superellipsoids (Fig. 9) and a Thingiverse [34] model of a stone monolith (908062), as shown in Fig. 21.

Volunteer subjects² rated their experience with robotic manipulators on a scale of 1 (no robotics experience) to 10 (robotics expert). A (1–10) scale assured sufficient fidelity in the presence of floor/ceiling effects [50]. Subjects were shown the MTTT interface (Fig. 21) and read a script describing the interface, testing procedure, and recorded data. Their objective was to maximize the percentage of reachable Task VF points for several models in a virtual robot environment. Users explored the MTTT interface for 5 min to familiarize themselves with its functionality using the N_z = 4, N_xy = 4 superellipsoid model (Fig. 9, bottom right). Task VF pose reachability results were available during this training/exploration, and thus, the model was excluded from evaluated trials.

Users were divided into tracks T1 and T2 where each track included four models and four trials for each model. The first three superellipsoids were randomized, but the monolith model was always last. For the first and second trials, T1 users were provided with the task model and asked to place it in a location where they thought the robot would be able to reach the largest portion of the task surface. They were provided aggregated reachability data at the task surface location (Fig. 8, top white writing) as feedback without being able to examine individual vertex reachability (Fig. 21). Users were then asked to choose a location with higher reachability based on the provided feedback. Once the first two trials were completed, users were asked to complete a Likert scale questionnaire. For the 3th and 4th trials, users were allowed to examine individual Task VF vertex reachability in addition to the aggregated data (Fig. 21). After completing the third and fourth trials, users completed another Likert scale questionnaire.

In contrast, T2 users examined individual Task VF vertex reachability and aggregated data during all four trials with each of the four models (Fig. 8, top). Users filled out the same questionnaire as above after trials 2 and 4. After the 4th trial on the stone monolith, users in T1 and T2 could search for locations with higher reachability. For this test, Task VF pose reachability feedback was left active during object motion, thus providing rapid feedback to the user and increasing the search rate.

The user group consisted of 11 participants, one volunteer above goal of 10, which were all students and the University of Texas at Austin but at varying undergraduate and graduate educational levels. Users possessed sufficiently varied levels of robot experience to maintain test diversity when distributed between tracks T1, 6, and T2, 5, (Table 2). All but one user opted to try immediate feedback with the monolith resulting in 185 trials.

Each participant evaluated four models using the procedure detailed above. T1 and T2 results for four trials with each of the four models were averaged. The results show T1 users took longer to choose test locations even when Task VF vertex reachability was provided in trials 3 and 4 (Fig. 22, left scale). T2 users also achieved higher reachability even during the final trial with the monolith model (Fig. 22, right scale).

Likert scale questionnaire data was also aggregated (Fig. 23). The results show users thought the task was more difficult and frustrating without Task VF vertex reachability information for complex shapes. The usability between the T1 and T2 user interfaces was almost the same, suggesting the additional Task VF information was interpretable, which was one of the main reasons for this user study. Users also thought they were more likely to improve with practice and successfully complete the task in track T2 compared to T1. The last inquiry results demonstrate a greater placement certainty when Task VF vertex reachability information was provided from the beginning.

Additional insights were gleaned from the operator comments and observation. The use of EEF interactive markers to evaluate reachability at specific locations on the task surface varied but with a consistent pattern across all users (Fig. 21). The least and most experienced users were less likely to take advantage of the EEF interactive marker (Fig. 21). Inexperienced users were slow to recognize its value, and experienced users instead utilized a well-developed mental model of a manipulator’s dexterous workspace. Users with some, but limited, experience tested Task VF vertices more extensively. In addition, several users asked when Task VF vertex reachability information would be available again once they had completed a given set of trials. This is consistent with customer interview practices recognizing a latent need [50]. Other experienced users requested “live” feedback before it was provided in the optional trial. Many users were impatient with the lag caused by reachability calculations in a variety of trials. These observations held regardless of model order. Most importantly, Task VFs generated through the pipeline described in this article were observed to be interpretable and useful to users in these preliminary trials. Thus, concerns about more complex VFs increasing mental load were allayed.

This article presents the novel concept of a Task VF, which consists of layers of point cloud-based Guidance VFs converted into a bidirectional graph structure. The generation pipeline was tested for general, complex surfaces defined parametrically, stored as mesh files, or collected from sensor data. Calculations extend task surface normals to build layers of PCN VFs at task-defined offset distances from the task’s surface. These VF layers are interpolated and voxelized to maintain intralayer task resolution. The VF layers are converted into the novel PCN-based Guidance VFs or Forbidden Region VFs. Guidance VF layers are combined into a bidirectional graph structure and paired with the Forbidden Region VF surface model to form a Task VF. This pipeline creates more expressive VFs than volumetric primitives, thus addressing a well-documented VF generation shortcoming. Task VF generation was tested with superellipsoids, supertoroids, crowdsourced meshes, and LIDAR generated 3D PCN data. For each input type, the code generated a bidirectional Task VF graph.

The developed manipulator to task transform tool (MTTT) provides operator visualization and navigation of the Task VF graph. A user study was organized to test the usability of Task VFs and ensure their complexity did not add to the operator’s mental burden. A group of operators, with varied robot experience, were asked to maximize the reachable percentage of the task surface for several models in a virtual robot environment. Evaluation on a spatially discrete task demonstrated Task VFs generated from complex shapes reduced completion time and increased the score of the surface placement task. Likert questionnaire results support this data, suggesting Task VF pose feedback did result in a decreased operator burden. Therefore, it seems acceptable to conclude Task VFs are operator interpretable with applicability to multiple tasks including reachability analysis, trajectory generation, and constrained teleoperation around complex objects. Also, the MTTT can be used for training, task setup, or hardware design analysis.

Ongoing and future work includes upgrading the Task VF generation pipeline with curvature aware PCN interpolation advancements, possible integration of surface region clustering, and continued user evaluation. Clustering surface regions could reduce intralayer connections and provide subgraphs, which would increase the applicability of Dijkstra’s Shortest Paths and other algorithms to improve execution efficiency. Future evaluations will include spatially continuous tasks, such as laser cutting or painting, and integration with modern trajectory planners to generate smooth trajectories. Further user testing is planned with the MTTT on additional tasks and also outside of ROS with industrial manipulator interfaces.

This material is based upon work supported by Los Alamos National Laboratory and the University of Texas at Austin.

There are no conflicts of interest.

The datasets generated and supporting the findings of this article are obtainable from the corresponding author upon reasonable request. The authors attest that all data for this study are included in the paper.