Automated Procedure Reconfiguration Framework for Augmented Reality-Guided Maintenance Applications

Maintenance reduces the amount of downtime or machine malfunctions and is imperative in running operations and equipment effectively. Similar to annual vehicle inspections, maintenance procedures prolong the life of any mechanical system. Traditional maintenance typically involves the use of paper-based manuals that are invariant to the user’s experience level or environment. There are known inefficiencies with traditional maintenance, including the large amount of time technicians spend locating and comprehending procedures and the inability of traditional systems to adapt to each user’s level of experience. For instance, in the aircraft industry, one study by Ott found that 45% of an Aircraft Maintenance Technician’s shift is spent finding and reading procedures [1]. Even with substantial time spent finding and reading procedures, ambiguity can arise when the technician adapts the procedure for the current situation [2]. For maintenance on mechatronic systems, training a novice operator requires months or sometimes years, while even expert technicians must reference paper-based manuals for infrequent or complex procedures [3]. These issues increase not only labor cost but also production cost [4]. As systems become more complex over time, the cost of these challenges will only become more exorbitant [5].

The shortcomings of the traditional paper-based manual maintenance approach necessitate an alternative solution to continue performing maintenance effectively on modern mechanical systems. One such solution is the use of augmented reality (AR) for maintenance applications. Recently, AR-based maintenance applications have gained widespread traction [6,7]. AR is able to display the information required to perform maintenance in a more intuitive and interactive way than the traditional maintenance approach. AR-based maintenance systems increase the effectiveness of the maintenance operator and speed up the entire maintenance workflow [2]. AR also provides the opportunity for maintenance applications to be context-aware by adapting to the user’s characteristics and preferences and the maintenance environment [8]. Applying AR to maintenance has been shown to have various advantages including [3]

• – the ability to employ less-skilled operators,

• – have up-to-date maintenance data,

• – have time and cost savings,

• – reduce the error rate of operators,

• – reduce the amount of text needed,

• – retain information in the AR system and not in people, and

• – adapt the level of information to each user’s skills.

While the application of AR to maintenance has resulted in great gains with respect to user efficiency and comprehension as compared to traditional maintenance, one of the aspects of AR-guided maintenance applications that is not conducive to the adaptable nature of AR applications is the rigidity of the maintenance procedures within these systems [9]. If the user encounters an issue with one of the machine’s parts while performing maintenance, many of the current AR-guided maintenance solutions either cannot handle the change in working conditions, or they rely on user knowledge or expert intervention to move forward. The presented solution instead relies upon automated reconfiguration strategies to aid the user in performing maintenance. An example of an automated reconfiguration process is shown in Fig. 1. The target part of the procedure is the motor (green highlighted) within a desktop fan. The user is successfully able to perform steps 1 through 5 but encounters an issue with the fan blades in step 6. Thus, the system attempts to provide a new maintenance procedure based on the defective part. For the example given in Fig. 1, a new maintenance procedure cannot be found, so the user is instructed to replace the entire subassembly of the defective part.

While the ability to automatically reconfigure procedures was never a possibility with traditional paper-based maintenance manuals, it is a viable and attainable option with AR technology. Automated reconfiguration of AR-guided maintenance instructions is the primary focus of the outlined work.

A modification to the general framework used to create an AR-guided maintenance application is needed to allow for automated reconfiguring of procedures. The outlined framework in this paper is applicable to any AR-guided maintenance system provided that certain prerequisite steps are completed as described in Sec. 3. The two main contributions of this paper are as follows:

1. Present a novel framework to allow for automated reconfiguring of maintenance procedures.

2. Provide proof of concept for the framework by implementing it in a handheld AR-guided maintenance application.

The structure of this paper is as follows. In Sec. 2, a comprehensive review of solutions for procedure reconfiguration in AR-guided maintenance and (dis)assembly applications is provided. In Sec. 3, the novel framework for procedure creation that allows for reconfiguring is presented, and the methods to implement the framework are explained using a simple example. In Sec. 4, the implementation of the framework in a handheld AR application is presented. In Sec. 5, a discussion of the implementation, application, and implications of the framework is presented. Additionally, conclusive remarks and directions for future research are provided.

AR is a technology that supplements the real-world with virtual content in real-time [10]. The first conceptualization of AR and its counterpart virtual reality (VR) was theorized by Sutherland in his essay “The Ultimate Display” in 1965 [11]. More recently in 2001, Azuma et al. defined the properties of AR systems as combining real and virtual elements in the real-world, running in real time and interactively with the user, and registering real and virtual elements with respect to each other [10]. VR, while similar to AR, enables immersive user collaboration with virtual objects in a completely virtual environment.

Both AR and VR share similar industries and applications. AR is used as a supplement or support for applications in entertainment [12,13], commerce and advertising [14–17], navigation [18,19], education [20–24], design and product development [25,26], project management and construction [25,27–29], sightseeing [30–32], museums [33–35], medical use [36], technician training [37,38], and inspections [39–42]. VR is used in applications related to entertainment [43,44], tourism [45,46], fashion [47], training [48–50], education [23,51,52], and data interaction and visualization as in construction [53–55], manufacturing [56,57], design and prototyping [58,59], healthcare [60–65], and navigation [66]. For more information about AR, the reader is directed to Refs. [67–69]. Similarly, for more information about VR, the reader is directed to Ref. [70]. For information about mixed reality applications, the reader is referred to Ref. [71].

There are many different types of AR including head-mounted, handheld, and projector-based. The type of AR chosen depends on various constraints including budget, scene characteristics, the system’s required portability, and the tracking system’s required measurement precision and reliability [72].

For this work, only the procedure creation component of AR-guided maintenance and (dis)assembly applications is focused upon. Many AR-guided maintenance and (dis)assembly projects have been implemented since the 1990s. For a review of existing literature in AR-guided maintenance applications in the last decade of the century, the reader is directed to the related work section of Ref. [73]. More recent reviews of AR-guided maintenance projects are provided in Refs. [6,7].

The procedure reconfiguration methods within recent AR-guided maintenance and (dis)assembly systems can be classified into three main categories: off-site reconfiguring, on-site reconfiguring, and automated reconfiguring. Off-site reconfiguring refers to modification that must be completed away from the location where maintenance is being performed either by the maintenance technician or by an expert. On-site reconfiguring refers to modification that can be completed at the location where maintenance is being performed either by the user or by a remote expert. Automated reconfiguring refers to modification that is performed by the application itself and does not require user or expert intervention. The trigger to start reconfiguration is not taken into account when considering these categories.

There are various AR-guided maintenance and (dis)assembly solutions found in the literature that include off-site reconfiguring abilities, meaning that reconfiguring can be completed away from the maintenance site by experts or operators.

All of the off-site reconfiguring strategies include unique methods to create AR content for procedures. The AR-guided maintenance prototype system presented in Ref. [74] uses a language for the creation of AR content called AR Media Language. Procedures are created based on computer-aided design (CAD) models, text, or images by associating them with specific workpieces. The solution presented in Ref. [75] includes easy AR authoring through PowerSpace, an AR authoring system. PowerSpace utilizes Microsoft PowerPoint [76] to allow the user to create slides that become procedures or augmentations on the screen. The Authoring for Context-Aware AR (ACAAR) system presented in Ref. [77] allows the user to model AR content through arranging various media files such as texts, images, and 3D models spatially and specifying the logical relationships between the AR content and contexts. The AR-guided maintenance system presented by Ref. [78] initially derives procedures from paper manuals but allows experts to create AR content using CAD or reverse engineering models and shape template matching. The Authoring Wizard presented in Ref. [79] allows assembly experts to create and modify assembly procedures for the Mixed Reality Assembly Instructor program that technicians use to perform assembly tasks. Various researchers have developed tools to create AR instructions based on video sequences of expert technicians performing procedures [80–83]. These tools can be utilized for off-site reconfiguration by experts. AR instructions can also be automatically created from CAD model geometry [84–87] or from a combination of information such as design, factory, and sales information as in Refs. [86,88], technical documentation and CAD models as in Ref. [89], or product, resource, and process information as in Refs. [90,91].

While all of these solutions are improvements over AR-guided maintenance applications that do not allow for procedure modification, none of these solutions are ideal because the authoring methods presented are not directly available to the operator while they are performing maintenance on-site. In this scenario, if the user encounters an issue with a part in the machine while on-site, they must leave the site, reconfigure the procedure, then return to the site to continue maintaining the machine.

The systems proposed in the literature that include on-site reconfiguring methods are those that either allow for expert-driven procedure modification from a remote location over the Internet or allow for user-driven procedure modification at the location where maintenance is being performed.

The following systems only include remote expert-driven reconfiguration. The system within the ULTRA project presented in Ref. [92] includes a tele-consultation module that the operator can use to communicate with an expert. The expert is able to alter procedures using a template-based authoring tool. A remote reconfiguring solution is given in Ref. [9] where procedures are initially defined using a state machine, and tasks are defined for each part in the CAD model with elementary operations such as slide on/off, screw on/off, etc. Experts are able to remotely reconfigure or modify the procedures that are implemented by the technician. The AR-guided maintenance test system implemented in Ref. [93] includes a remote collaboration tool that allows the expert to modify the user’s 3D virtual annotations in real time. Reference [94] provides an AR-guided maintenance solution for industrial environments where experts within the support center of the industrial facility are able to add virtual information on the user’s display if needed. The predictive maintenance system in Ref. [95] allows experts to anchor annotations within the user’s field of view during a remote assistance call. Similarly, the AR training platform developed in Ref. [96] sends a video of the performed maintenance procedure to the expert trainer who is able to modify the procedure in real time or add “Virtual Post-Its” to the user’s screen for additional guidance. The Product Service System (PSS) AR solution in Ref. [84] includes a cloud-based remote maintenance solution that allows experts to exchange maintenance instructions with technicians. Likewise, the head-mounted cloud-based AR-guided maintenance solution presented in Ref. [97] allows operators to reach out to expert engineers when they encounter an issue while performing maintenance. The expert engineer then creates AR content remotely and projects the content onto the technician’s field of view.

There are systems in the literature that include both on-site user-driven reconfiguration and remote expert-driven reconfiguration. The Authorable Context-Aware AR System (ACARS) in Ref. [5] includes a bi-directional authoring tool where content can be created both off-site by experts and on-site by technicians. This expert-driven authoring tool provides a convenient method for users to implement significant changes to procedures, while the user-driven authoring tool is designed to allow operators to make smaller changes to procedures that are created by experts. The AR-assisted maintenance system (ARAMS) presented in Ref. [98] includes a bi-directional content creation tool and a remote collaboration mechanism that allows experts to modify procedures while the user is on-site. Finally, an AR-guided maintenance solution for the modern shop floor is given in Ref. [99]. The user has access to a database of predefined tasks that they can use if they encounter an issue while performing maintenance. If extensive modification is needed, the user has the ability to reach out to an expert engineer who can provide the user with new AR instructions.

While on-site reconfiguring methods provide operators with convenient access to solutions if they encounter an issue while performing maintenance, if experts are not available or if there is an unreliable Internet connection where the operator is performing maintenance, these methods fail. In the case of the bi-directional authoring tools as in Refs. [5,98,99], the operator is able to author content and modify procedures autonomously. This is ultimately an unsuitable solution because the quality of the produced procedures is dependent on the user’s level of knowledge and experience. This will then impact the success of reconfiguration in these systems. For these reasons, automated reconfiguration strategies are more reliable in the field.

There are few solutions present in the literature that include automated reconfiguring where the AR system itself can modify maintenance procedures on demand.

An interactive AR system was presented in Ref. [100] that incorporates context-awareness to detect a machine’s operating status after evaluation of the current condition of the machine. Procedures are automatically developed using condition information and graph-based algorithms. Reconfiguring is made possible by reevaluating the machine’s current state after every step in the procedure. Similarly, the AR assembly guidance system in Ref. [101] allows the user to choose the next task in the current procedure if multiple options are available based on the assembly structure of the machine. The visual assembly tree structure (VATS) used provides a kind of adaptive procedure creation strategy for automated reconfiguration. The AR-assisted product disassembly sequence planning (DSP) (ARDIS) system described in Ref. [102] utilizes a procedure regeneration strategy that automatically provides the user with a reconfigured procedure in the case of unforeseen obstacles. Procedure regeneration is only available via a web server.

Automated reconfiguring is the ideal option when compared to off- and on-site reconfiguring because it does not rely on any outside source or knowledge in order to produce a successful result. In cases where the operator is not experienced, does not have access to an expert, and encounters an issue with a part while performing maintenance that prevents them from moving forward with the procedure, automated reconfiguration is the only option. While the reconfiguring solutions presented in Refs. [100–102] are able to perform automated reconfiguring, the solution in Ref. [100] is restricted to systems that have flow such as piping systems, the assembly sequences in Ref. [101] are restricted to the hierarchical relationships defined in the VATS, and the solution in Ref. [102] requires an Internet connection. All of these solutions are not universally acceptable in AR-guided maintenance applications. The framework proposed in Sec. 3 for procedure creation is designed to allow for automated reconfiguring of procedures in any AR-guided maintenance application.

Automated reconfiguring is directly related to the work done in DSP. Thus, a brief review of DSP methods is given next.

DSP is a field of research that is traditionally concerned with finding the optimal disassembly sequence for End-Of-Life (EOL) recovery of materials or components within products at the end of their lifecycles [103]. There are different types of DSP methods including complete or selective and destructive or non-destructive. Complete disassembly means that the machine or product is disassembled to its base components or elements. Selective disassembly means that the goal of the disassembly plan is to obtain certain target parts. Destructive DSP methods change the physical structure of the machine, making reassembly impossible. Non-destructive methods preserve the original physical structure of the components of the machine or product, making reassembly possible. When considering maintenance solutions, the only applicable DSP methods are selective, non-destructive methods [104]. With selective disassembly, the goal can be to either obtain a single target part or multiple target parts. Further, parts can be disassembled in either a serial or parallel manner [105].

In order to create a disassembly sequence, relevant information about the machine or product that is being disassembled must be utilized in an organized manner. There have been many representations for product information proposed in the literature including graph-based and matrix-based methods. Some of the most prevalent graphical methods include adjacency [106], connection/liaison [107,108], interference [108], precedence graphs [109–111], AND/OR graphs [107,108,112], and Petri Nets (PNs) [113–115]. Some of the most prevalent matrix representations include matrix-based versions of the adjacency [106,116], connection/liaison [108,117,118], interference [108,119–123], and precedence matrices [111,119,124], contact matrices and translation matrices [102,122,123,125], transition matrices [122,126–129], and disassembly constraints matrices [130,131]. Search trees are an additional DSP representation that are popular in literature [132–134]. In cases of increasing product complexity, it is common to divide the product or machine into subassemblies either manually, as in Ref. [135], or automatically, as in Refs. [106–108,117,122].

When considering DSP methods for EOL products and machines, product condition can vary widely. This can be a challenging problem especially with non-destructive DSP methods, as is required in maintenance applications. In order to attempt to account for product condition when creating a disassembly sequence, researchers have investigated mathematical models including fuzzy logic [136,137] and probability [138].

An operation must be performed on the representation of product information in order to create a disassembly sequence. In many cases, this operation is an optimization. Some of the most prevalent optimization algorithms used for DSP are genetic algorithms [110,125,130,139] and swarm optimization algorithms such as ant colony optimization [116,140] and the enhanced discrete bees algorithm [121]. For selective disassembly, some popular operations for DSP include recursive- or rule-based methods [141–145] and the wave propagation approach, developed in Refs. [135,146–149] and expanded in Refs. [109,150] for general DSP applications [151]. Various heuristic methods including genetic algorithm [152–156], ant colony optimization [140,157,158], artificial bee colony algorithm [159], particle swarm optimization [160–162], tabu search [163], and scatter search [164–167] are also prevalent in the literature. Similarly, motion-planning or Rapidly-growing Random Tree-based algorithm (RRT) methods [168] and integer programming methods [112,169] are commonly used for selective disassembly operations. For a full review of selective disassembly sequence methods, the reader is referred to Refs. [141,170–172]. The main objective for these DSP methods is to minimize the disassembly cost, where the cost can encompass disassembly time, time for tool changes, monetary cost, safety of the disassembly plan, etc. Other algorithms that are used to produce disassembly sequences include the branch and bound method [112,132,134] and graph-based search algorithms such as the Dijkstra algorithm and A* algorithm [109,110,114,173].

The framework presented in this work requires the use of an automatic procedure creation method in order to allow for reconfiguration. Automatic procedure creation can be implemented via DSP methods, as in Refs. [100,102,174–176], or by other methods, such as from CAD model geometry in Refs. [84–87], from a combination of information such as design, factory, and sales information as in Refs. [86,88], from technical documentation and CAD models as in Ref. [89], or from product, resource, and process information as in Refs. [90,91].

The outlined framework in this paper builds upon complementary concepts developed by various researchers, including Belhadj et al. [117], Luo et al. [116], and Gungor and Gupta [177]. An overview of related pertinent concepts is outlined next.

Belhadj et al. presented a method in Ref. [117] to use part adjacency, fit, volume, and surface area information to automatically detect the subassemblies within a given assembly. Subassemblies are built upon base parts, which are parts within the assembly that have the highest number of connections, volume, and surface area based on a single-objective constrained optimization problem.

Luo et al. presented a multi-layer matrix representation that is used to store adjacency information for subassemblies of products [116]. This representation takes up less computational space when compared to one large adjacency matrix for the entire product. Additionally, the authors present two procedure creation methods, one being the Traversing searching algorithm (hereafter called the TSA) and the other being the Ant Colony Optimization method.

Lastly, Gungor and Gupta presented an algorithm for DSP with uncertainty [177]. Gungor and Gupta’s approach uses a product’s precedence matrix and information about defective parts within the product to analyze which parts are affected by the defective part and to disassemble the parts that are not directly affected by the defective part through DSP.

The overall framework, shown in Fig. 2, is an alternative to the typical procedure creation module within an AR-guided maintenance application. The various components in the framework are either performed offline or online. Offline components of the framework are created and loaded into the application before it is used, while online components of the framework are executed while the operator uses the application.

The input to the overall framework is the CAD model of the machine being maintained. Adjacency analysis is performed on the CAD model to find the adjacency information for each part of the machine. Then, subassembly detection uses the adjacency information to find subassemblies within the machine. The adjacency and subassembly information is then used to create a procedure.

The reconfiguration process then begins. If a feasible maintenance procedure is created, the user starts to perform the maintenance procedure. If a feasible maintenance procedure is not created, the subassembly of the current part in the procedure or the defective part, if applicable, is replaced. The user then proceeds to perform the modified maintenance procedure. If reconfiguration of the procedure is required while the user is performing maintenance, the user is able to trigger the Defective Part Analysis (DPA). This analysis determines which parts are affected by the user-inputted defective part. The affected parts information is then used in procedure creation to create a new maintenance procedure for the user.

A running example of a desktop fan (Fig. 3)² is used to elucidate the concepts presented in the following sub-sections. The following assumptions apply to the general framework:

• –

  The adjacency information for the parts in the machine being maintained does not change over time.

• –

  Only one machine is being maintained at any one time.

• –

  There is only one target part for maintenance when a procedure is initially created.

• –

  There is only one defective part when performing reconfiguration.

Procedure creation has two prerequisite tasks: adjacency analysis and subassembly detection. Both of these tasks are performed offline before the application is used by the maintenance technician.

Adjacency analysis refers to the process of determining the relationships between all of the parts in the machine being maintained. The implemented adjacency analysis method in the outlined framework is a combination of the ideas presented by Refs. [117] and [116]. The two major differences between the two approaches are 1) the choice between a global and local coordinate system and 2) the decision of what adjacency direction information to record for each part in the machine. Reference [117] uses a global coordinate system, whereas Ref. [116] uses a local coordinate system. A local coordinate system bases each part’s direction of motion on the previous part’s direction of motion. For an AR system where the user’s or device’s perspective changes with time, non-specific directions of motion can become ambiguous to the user as parts are moved around the scene. For this reason, a global coordinate system was chosen for this implementation so that definitive Cartesian directions could be provided to the user during the maintenance procedure.

Regarding adjacency directions, Ref. [117] considers adjacency in the Cartesian directions in general while Ref. [116] considers adjacency in both the positive and negative Cartesian directions. In order to use a modified version of the TSA presented by Ref. [116] for procedure creation, adjacency information in both the positive and negative Cartesian directions was extracted for each part in the machine.

The implemented adjacency analysis method takes as input the CAD model of the machine being maintained. The final outputs from the adjacency analysis process are the adjacency and contact matrices.

The following assumptions are imposed for the inputted CAD model:

• –

  All parts in the machine are available in the CAD model,

• –

  All parts in the CAD model adhere to the same length convention,

• –

  All parts in the CAD model adhere to the same global coordinate system, and

• –

  All parts in the CAD model are rigid in their composition and position and orientation (pose).

First, the parts list and bounding box dimensions are extracted from the CAD model. The parts list and bounding box dimensions for the desktop fan example are given in Table 1.

Subassembly detection refers to the process of automatically finding combinations of parts in the machine being maintained in order to introduce modularity into the procedure creation process and to permit reconfiguration. The implemented subassembly detection method is derived from Ref. [117], while the storage of the resulting subassembly matrices is derived from Ref. [116]. While Ref. [116] derives subassemblies of the machine from the CAD model’s Bill of Materials (BOM), the method presented by Ref. [117] is preferable because it is based solely on the geometry of the parts in the machine and not on the defined relationships within the CAD model, which can vary greatly depending upon the designing style of the CAD creator. The only major modification made to the subassembly detection method presented by Ref. [117] is that the relationships between subassemblies are not broken. Since procedures are being created with this adjacency information, the relationships between subassemblies must be preserved. The adjacency and contact matrices from the adjacency analysis are inputs to the subassembly detection process. The output from subassembly detection is a series of matrices that hold the adjacency information for the detected subassemblies.

The next step in the process of finding subassemblies includes finding the first-level base parts for the machine. As described above, base parts form the foundation of each subassembly. The first-level base parts are the initial base parts found for the machine. As a prerequisite to finding the first-level base parts, the number of connections, surface area, and volume of each part is calculated. The number of connections is calculated by considering the elements of each row of the adjacency matrix. Digits of 1 are considered as no connection, digits of 2 or 3 are considered as one connection, and digits of 4 are considered as two connections. The surface area and volume for each part are calculated based on the bounding boxes of each part.

After the identification of the initial base parts, an analysis is completed to identify parts in the machine that are connected to more than one base part. For each part that is connected to more than one base part, the weights of the connections between the current part and all possible base parts are calculated. This process is done by adding together the digits for the element of the combined matrix corresponding to the connection between the current part and each base part. A higher weight indicates that the connection has the tightest fit and the most amount of connections between the current and corresponding base part. Thus, the connection with the highest weight is kept. If the weight is the same between the current part and all possible base parts, the connection with the highest optimization function value is kept.

The adjacency information from the combined matrix for the first-level base parts is saved in a separate matrix according to Luo et al.’s multi-layer adjacency matrix representation [116]. The adjacency information for the base parts of each subassembly is recalculated based on the newly found subassemblies, which results in changes to the contact fit information and number of connections for each part. The process of finding base parts and subassemblies is then repeated for each base part to create a hierarchical subassembly structure.

The final step for base part determination and subassembly detection involves including the adjacency information for any parts that are not included in the base parts structure into the first-level subassembly. The resulting structure of base parts and subassemblies for the desktop fan example is given in Fig. 4.

The methods put forth by Refs. [116,117] only find one level of base parts and subassemblies. A hierarchy of base parts and subassemblies is found in this work in order to give increased flexibility to the reconfiguration process. The reconfiguration process works by finding a reconfigured procedure for the parts within the subassembly of the defective part. Having additional subassemblies and more layers allow for smaller units to be replaced if a reconfigured procedure cannot be found.

Procedure creation uses adjacency and subassembly information in order to create an ordered list of parts that make up the final maintenance procedure. The initial procedure is created based on a user-defined target part, which is the part of the machine that needs to be maintained. Procedures are created for each subassembly in segments depending upon how many subassemblies are between the first-level subassembly and the target part. The first-level subassembly represents the machine as a whole and contains both the initial subassemblies and parts that do not belong to any subassembly. The final procedure is a combination of each procedure segment.

The procedure creation method presented is based on DSP theory. The DSP method chosen for this implementation is a modification of the TSA given by Ref. [116]. The implemented method, hereafter called the modified traversing searching algorithm (MTSA), uses a suboptimal strategy to find a sufficient disassembly sequence. This sequence is then converted into the final overall procedure. The computational flow of the MTSA is illustrated in Fig. 5.

The inputs to the MTSA include the base parts and subassembly adjacency information, along with the user-defined target part. The MTSA is run for each hierarchical subassembly until the subassembly that contains the target part. When creating a new procedure, it is assumed that the procedure begins with the whole machine assembly. While it is possible to begin a procedure with a specified part, this stipulation is enforced in order to simplify the implementation.

In order to demonstrate the usage of the framework, a maintenance procedure is developed to replace the motor, which is part 8 or motor, highlighted in Fig. 3, in a desktop fan. Since the target part is a member of the second-level subassembly, the MTSA is run for both the first-level and second-level subassemblies in order to create the overall procedure.

A preliminary step to finding the initial procedure is adding the objective function values for the procedure into the diagonal elements of each subassembly adjacency matrix. The objective chosen for this implementation is the minimization of the total procedure time. The following assumptions are upheld for the objective function values in order to simplify this procedure creation implementation:

• –

  It is assumed that the disassembly time for each part is directly proportional to that part’s total number of connections.

• –

  It is assumed that the disassembly time for each part is the same as the assembly time for each part.

• –

  It is assumed that the disassembly time for each part is the same in all three Cartesian directions.

The theory behind the MTSA is that an attempt is made to find a disassembly direction with no 4s and either only 2s or only 3s for each part until retrieving the target part. If the row in the current Cartesian direction fits these qualifications, the part corresponding to the current row is added to the disassembly sequence in the direction opposite to the direction of its connections. The row and column that correspond to this part are then removed from all three Cartesian matrices. The MTSA then reiterates the process from the first or x-direction and first row in the matrices. Since the rows are sorted according to the objective function and the heuristic of the number of connections is used for the objective values, this method produces a suboptimal result. A suboptimal procedure is sufficient in this case for the purposes of showing the applicability and usage of the framework.

During the MTSA, each part in the subassembly is considered by its corresponding row in each Cartesian matrix. At the beginning of each iteration of the algorithm, the row corresponding to the target part is checked to see if it can be removed. If one of the directions in the row corresponding to the target part does not contain any 4s, contains only 2s or only 3s, or has no connections, the target part can be removed and is added to the new disassembly procedure in the direction opposite to the part’s other connections. If there are no connections with the target part in any direction, the default direction is set to the positive x-direction. Once the target part is retrieved, the algorithm ends. The algorithm iterates through the possible directions and rows, starting with the direction equal to zero, indicating that the current direction is x, and the row equal to zero. For the current row, the number of 4s is recorded. If the row contains no 4s and either only 2s or only 3s, the part is disassembled in the direction opposite to the direction it has connections to.

The resulting disassembly sequence from each subassembly until the subassembly containing the target part is combined to produce the final disassembly sequence. A reverse sequence of the overall disassembly sequence is appended onto the final sequence in order to guide the user in reassembling the machine after performing maintenance on the target part.

For the desktop fan example, the overall procedure is derived from the first-level and second-level subassembly matrices. The target part within the first-level subassembly matrix is part 1, back cage, while the target part within the second-level subassembly matrix is part 8, motor. The MTSA starts by checking if the target part can be removed easily in any direction, meaning that there are no four connections and only two or only three connections in the row corresponding to the target part in a certain direction. Part 1 or back cage in the sixth row of the first-level subassembly matrices cannot easily be removed because it contains four connections, so the algorithm starts with the x-direction and the first row and attempts to remove parts in order to free the target part. The first row, corresponding to part 3 or back plastic cover, cannot be removed because it contains four connections. The next row corresponding to part 2 or front cage, also cannot be removed due to four connections. Next, part 6 or stand rivet in row 3 can be removed in the positive x-direction because it only contains three connections. Part 6 is added to the disassembly plan in the positive x-direction and the adjacency information for part 6 is removed from all three Cartesian matrices. The MTSA then restarts by checking the target part for easy removal. Part 1 or back cage still cannot easily be removed in any direction, so the algorithm restarts from the first row in the x-direction. The next part that can be removed is part 7 or stand rivet1 in the negative x-direction. Part 7 is added to the disassembly plan and its adjacency information is removed from the Cartesian matrices. The algorithm then proceeds check if part 1 or back cage can be removed easily in any direction. Since part 1 or back cage is still not removable, the algorithm starts over from the first row in the x-direction. The algorithm proceeds and must remove part 5 or stand in the negative y-direction and part 3 or back plastic cover in the negative z-direction before being able to easily remove part 1 or back cage in the negative z-direction. From here, the algorithm moves onto the second-level subassembly matrix with part 8 or motor being the target part. The target part, part 8 or motor, cannot easily be removed in any direction, so the algorithm begins trying to remove parts with the first row in the x-direction. Part 1 or back cage represents the second-level subassembly and has already been disassembled from the machine, so it is skipped. Part 4 or blades can be removed in the positive z-direction, so it is added to the disassembly plan and part 4’s adjacency information is removed from the second-level Cartesian matrices. The target part, part 8 or motor, can then be removed in the positive z-direction. The resulting procedure is {6 in +x, 7 in −x, 5 in −y, 3 in −z, 1 in −z, 4 in +z, 8 in +z, 8 in −z, 4 in −z, 1 in +z, 3 in +z, 5 in +y, 7 in +x, 6 in −x}. A graphical depiction of the maintenance procedure is shown in Fig. 6.

Reconfiguration refers to the strategy implemented to allow for modification of maintenance procedures. The implemented reconfiguration method is performed online and is built upon the DSP method described in Sec. 3.3. For the implementation in the presented framework, an analysis about which parts are affected by the defective part is included, the DPA. The DPA is derived from the work of Ref. [177]. An overview of the DPA is given in Fig. 7.

Reconfiguration takes as input a user-defined defective part, which is the part that the operator is currently having an issue with, and outputs a list of parts that are affected by the defective part within the subassembly of the defective part. This information is used by the procedure creation algorithm to attempt to create a new procedure for the segment of the procedure that corresponds to the subassembly containing the defective part. This is done by executing the procedure creation process while keeping the adjacency information of the affected parts immovable within the subassembly of the defective part. If the procedure can be reconfigured within the subassembly, the overall procedure is updated, and the user can continue performing maintenance. If a feasible procedure cannot be found, the user is instructed to replace the subassembly that contains the defective part.

Since a feasible maintenance procedure was created from the procedure creation process for the desktop fan example, the user is able to begin performing the resulting maintenance procedure. The overall procedure found in Sec. 3.3 for the desktop fan example was {6 in +x, 7 in −x, 5 in −y, 3 in −z, 1 in −z, 4 in +z, 8 in +z, 8 in −z, 4 in −z, 1 in +z, 3 in +z, 5 in +y, 7 in +x, 6 in −x}. While the user is performing maintenance, if part 4, blades, is found to be defective, the user triggers the DPA.

The inputs to the DPA are the subassembly matrix of the defective part and the disassembly sequence corresponding to this subassembly. The subassembly adjacency matrix is saved to a local matrix, DP. For the desktop fan example, this is the second-level subassembly matrix. The current disassembly sequence for the subassembly of the defective part is saved to a vector, Φ. For the desktop fan example, this is [4, 8]. The DPA begins by initializing a vector, DV, that contains binary information about whether or not each part in the subassembly of the defective part is defective. For the desktop fan, DV is [0, 1, 0] corresponding to the parts in the second-level subassembly: 1, 4, and 8. The vectors Ψ and Ω are initialized and will contain the parts that cannot be disassembled with non-destructive methods and the set of parts that are affected by the defective part, respectively.

The DPA starts by analyzing each part in Φ. If the current part is not defective, this part is removed from Φ and the adjacency information for the part is removed from DP. If the current part is defective, this part and all of its adjacent parts are added to Ω. For the desktop fan, the algorithm starts with the first part in Φ, part 4 or blades. This part is defective, so all of the parts it is adjacent to are added to Ω so that Ω = [4, 8]. After analyzing each part, the parts in Ω are added to Ψ to produce Ψ = [4, 8] and removed from Φ, so Φ = [].

After the first analysis, a second check is conducted. For each part in Ψ, if the part is not defective and does not have any adjacency relationships in DP with any other parts, the part is removed from Ψ and DP. Otherwise, the part is kept in Ψ. For the desktop fan, the first part in Ψ, 4 or blades, is defective and thus stays in Ψ. The next part, part 8 or motor, is not defective but does have adjacency with other parts. Thus, part 8 is kept in Ψ. The final Ψ vector is given as [4, 8]. Since the final Ψ vector contains the target part, part 8 or motor, the subassembly containing this target part is replaced. The reconfigured procedure is given as {Replace back cage}. The user then continues performing maintenance by replacing this subassembly. A depiction of the reconfigured maintenance procedure is given in Fig. 8.

In order to demonstrate the efficacy of the proposed framework on a more complex machine, a MakerBot Replicator 2X 3D printer was chosen as an application problem (Fig. 9). One part that requires regular maintenance in a 3D printer is the leadscrew that provides upward and downward motion for the build plate. The framework was tested within an Android mobile application on Google Pixel 3 and Samsung Galaxy S6 smartphones.

When the operator first opens the app, the user must align the Guide View for the MakerBot (Fig. 10) with the real-life object in order to load the procedure reconfiguration interface of the application and the MakerBot 3D model AR augmentation. Once the interface of the application is loaded (Fig. 11(a)), the user is able to either create a new maintenance procedure, choose to perform a predefined procedure, or modify a predefined procedure manually. In order to begin the maintenance of the leadscrew, the operator chooses to create a new maintenance procedure. The user is then brought to a screen where they can choose the target part for the procedure (Fig. 11(b)). In this case, part 15 or Lead Screw is chosen. The maintenance procedure is automatically generated using the algorithms described in the preceding sections. The final procedure produced for the leadscrew is {20 in +x, 21 in −x, 2 in −x, 3 in −x, 4 in +x, 5 in +x, 6 in +x, 14 in −y, 17 in +y, 15 in +y, 15 in −y, 17 in −y, 14 in +y, 6 in −x, 5 in −x, 4 in −x, 3 in +x, 2 in +x, 21 in +x, 20 in −x}. For each step in the procedure, a Guide View similar to that shown in Fig. 10 corresponding to the relevant part or subassembly is displayed to the user. Once the user aligns the camera view with the Guide View, 3D model augmentations for the part or subassembly are loaded (Fig. 11(c)) in order to aid the user in identifying the relevant part or subassembly for that step of the procedure. While the user is performing maintenance, they encounter an issue with part 20 or side panel and cannot continue performing the procedure. They then select the “Reconfigure” button to start the reconfiguration process (Fig. 11(c)). In this case, part 20 or side panel is considered the defective part for the DPA. The resultant affected parts (Ψ) from the DPA include parts 20, 4, and 5. The procedure creation process (MTSA) is then rerun with parts 20, 4, and 5 marked as immovable. The new procedure is given as {21 in −x, 2 in −x, 3 in −x, 6 in −x, 14 in −y, 17 in +y, 15 in +y, 15 in −y, 17 in −y, 14 in +y, 6 in +x, 3 in +x, 2 in +x, 21 in +x}. The final maintenance procedure, which includes the original procedure and reconfigured procedure due to the defective part, is given in Fig. 12. The in-field computation times for the MTSA, procedure creation process, DPA, and reconfiguration process are given in Table 3.

A visual depiction of the application software architecture is provided in Fig. 13. The parts list, part bounding box dimensions, adjacency matrix, and contact matrix were computed using a macro in freecad [178] and saved to a text file that is inputted into the subassembly detection module. The developed script automatically moved each part in the CAD model with respect to every other part in the positive and negative Cartesian directions. For the adjacency matrix, the part was moved 10 mm in each direction. If there was a collision between the parts during or after this movement, the parts were said to be adjacent. For the contact matrix, the parts were moved 1 mm in each direction to test the fit of the parts. The subassembly detection method was implemented in matlab [179], taking as input the text file produced from adjacency analysis and outputting a text file containing the multi-level subassembly matrices and the base parts structure. Both the freecad macro and matlab code are run offline to be used by the online procedure creation steps.

The MTSA is implemented in Android Studio [180] online while the operator is using the application. When the user selects a target part to create a new maintenance procedure, as shown in Fig. 11(b), the algorithm searches for the target part in the base parts structure and runs the MTSA algorithm for all subassemblies up to and including that of the target part.

The reconfiguring process is also implemented in Android Studio [180] and is triggered by the press of a button by the user (Fig. 11(c)). The current part in the procedure is then deemed the defective part and the DPA is run.

Supplementary task information for each part is organized in an ontology that is stored in local storage on the AR device and utilized by the AR application. The ontology was developed offline using Protégé [181] and loaded into Android Studio [180] in the OWL format. The Jena reasoner [182] is used to validate the ontology and to infer relations between the instances. Task information is matched with each part or subassembly in the procedure as the current step is loaded within the application.

The AR portion of the application was created using the vuforia Software Development Kit (SDK) [183] in Android Studio [180]. For the whole object in addition to the individual parts and subassemblies within the machine, Model Targets were created based on Wavefront .obj files using Vuforia’s Model Target Generator [183]. The Guide Views for these Model Targets are presented to the user during both the initial activity of the application and for each step of the maintenance procedure in order to register the 3D augmentations. Once the Guide View is properly aligned with the real object in detection, CAD model augmentations are registered with respect to the real object and tracking of the real object begins. CAD model augmentations were created based on Wavefront .obj and .mtl files. A c++ parser, Tiny OBJ Loader [184], was used to load the .obj and .mtl files in Android Studio [180]. OpenGL for Embedded Systems (ES) [185] was used to display the .obj augmentations on the screen of the devices.

The framework outlined in this paper allows for the reconfiguration of procedures to be included in any AR-guided maintenance application regardless of the type of AR used, the application for the system, and the environment in which the system is being used. While researchers have implemented various reconfiguration strategies to permit users to continue performing maintenance when they encounter issues, these solutions only work locally for each implemented project and are limited to user knowledge and experience or expert availability. The presented framework has the ability to eliminate the need for expert availability during maintenance. This drastically reduces the time the user spends waiting for a solution and thus reduces maintenance costs overall.

The implementation of the presented framework has limitations. First, the adjacency analysis of the machine is restricted to six directions. In order to make the adjacency analysis and thus procedure more accurate, additional directions of motion should be considered, in linear and rotational directions. Second, the automatically determined subassemblies may not align with subassemblies within the real machine. While the autonomous nature of the subassembly detection method is convenient if the replacement of a subassembly is needed, replacing the detected subassembly may not be reasonable in real life. However, for a machine, the overall subassembly information is generally available a priori and can be provided directly as manual input to the framework, thus avoiding automated subassembly detection. Third, a major limitation of the implementation described is that if a feasible reconfigured procedure cannot be found, the entire subassembly containing the defective part is replaced.

The application of the framework requires improvements to increase computational speed and efficiency. As seen in Table 3, the computational times for both the MTSA in procedure creation and the DPA in reconfiguration are low (≤1 s). The original TSA in Ref. [116] uses an exhaustive search strategy to identify all possible disassembly sequences in order to obtain the target part. While the MTSA is able to reduce the computational time for the exhaustive search by stopping after obtaining the first feasible disassembly sequence, the MTSA produces a suboptimal disassembly sequence. In order to produce an optimized disassembly sequence for procedure creation, an efficient searching strategy should be used to obtain a disassembly plan such as genetic algorithms, swarm optimization algorithms, or graph searching algorithms as described in Sec. 2.4. Contrastingly, the actual wait times in order to obtain the original procedure in procedure creation and the modified procedure in reconfiguration are very high when compared to the computation times for the MTSA and DPA, respectively. The computational bottleneck within the application has been identified as the ontology. Ontology is an artificial intelligence (AI)-based dynamic data storage structure that has advantages over databases because of its ability to use ontological reasoners to validate the ontology and infer knowledge between instances. Databases, on the other hand, simply hold key-value pairs. For additional information about the use of ontology and other AI techniques within AR applications, the reader is referred to Ref. [186]. The specific issues encountered during implementation include the following: (1) the creation of ontology instances, (2) the overwriting of new information to ontology instances, (3) the validation of the ontology instances with the Jena reasoner, and (4) the retrieval of the properties for the instances from within the ontology. Based on the high wait times when utilizing and modifying this information, alternative ontology formats and reasoners should be investigated for use in this application or the use of simpler databases such as extensible markup language (xml) or relational database management systems (RDBMS) should be considered.

There are many AR SDKs available such as ARToolKit [187], Wikitude [188], Kudan [189], ARCore [190], ARKit [191], EasyAR [192], and DroidAR [193]. These SDKs include various AR tracking strategies including marker-based, feature-based, and model-based. These AR tracking strategies require varying inputs for training before use or reference while in use. Because CAD models are required as input to the presented framework, an SDK with model-based tracking, such as vuforia [183], was utilized in the application of the framework. Other tracking strategies may offer increased speed and efficiency for the application of the presented framework. Future implementations of this framework should explore the use of other AR SDKs and tracking strategies for increased computational speed and efficiency.

The AR SDK used in this application, vuforia [183], can be used within Unity [194], Android Studio [180], Xcode [195], and Visual Studio [196]. For the presented application, AR capabilities through vuforia [183] were integrated into a prototype maintenance application developed in Android Studio [180]. Game engine platforms, such as Unity [194], are commonly used to develop AR applications because they are optimized to handle 3D environments and complex user interactions, which are prevalent in AR systems. The utilization of game engines to increase the speed and efficiency of the application of this framework should be investigated in future research.

The presented AR application not only helps the user by producing an updated procedure on demand but also allows the user to make their own modifications to procedures. Additionally, users are able to add notes and safety warnings to procedures for additional assistance the next time that procedure is performed. In order to improve the usability of the developed application, a full test with more complex machinery and real maintenance workers in a realistic working environment would have to be conducted. These realistic tests would be insightful to assess the real-time usability of handheld AR within the maintenance environment.

Although there are innovations to be made with the presented application including reducing the wait time for procedures and testing the application in more realistic and complex maintenance environments, the developed framework and ideology remains invariant to these improvements. The framework is an effective tool that can be used to provide additional aid and autonomy to the user by performing automated AR-based reconfiguration of maintenance procedures. When implemented, the proposed framework allows any user of any experience level the ability to perform maintenance effectively.

The research outlined in the paper is based upon work supported by Naval Surface Warfare Center (NSWC) under the NEEC award No. N00174-19-0025.

There are no conflicts of interest.

The authors attest that all data for this study are included in the paper.