Abstract
Robotic technology can benefit disassembly operations by reducing human operators' workload and assisting them with handling hazardous materials. Safety consideration and prediction of the human movement are priorities in close collaboration between humans and robots. The point-by-point forecasting of human hand motion, which forecasts one point at each time, does not provide enough information on human movement due to errors between the actual movement and the predicted value. This study provides a range of possible hand movements to increase safety. It applies three machine learning techniques, including long short-term memory (LSTM), gated recurrent unit (GRU), and Bayesian neural network (BNN) combined with bagging and Monte Carlo dropout (MCD), namely, LSTM-bagging, GRU-bagging, and BNN-MCD to predict the possible movement range. The study uses an inertial measurement unit (IMU) dataset collected from the disassembly of desktop computers by several participants to show the application of the proposed method.
References
1.
Liao
, H. Y.
, Chen
, Y.
, Hu
, B.
, and Behdad
, S.
, 2023
, “Optimization-Based Disassembly Sequence Planning Under Uncertainty for Human-Robot Collaboration
,” ASME J. Mech. Des.
, 145
(2
), p. 022001
. 2.
Liau
, Y. Y.
, and Ryu
, K.
, 2021
, “Status Recognition Using Pre-Trained YOLOv5 for Sustainable Human-Robot Collaboration (HRC) System in Mold Assembly
,” Sustainability
, 13
(21
), p. 12044
. 3.
Liu
, Z.
, Liu
, Q.
, Xu
, W.
, Liu
, Z.
, Zhou
, Z.
, and Chen
, J.
, 2019
, “Deep Learning-Based Human Motion Prediction Considering Context Awareness for Human-Robot Collaboration in Manufacturing
,” Procedia CIRP
, 83
, pp. 272
–278
. 4.
Wang
, J.
, and Shen
, L.
, 2021
, “Semi-Adaptable Human Hand Motion Prediction Based on Neural Networks and Kalman Filter
,” J. Phys. Conf. Ser.
, 2029
(1
), p. 012091
. 5.
Kaipa
, K. N.
, Morato
, C. W.
, and Gupta
, S. K.
, 2018
, “Design of Hybrid Cells to Facilitate Safe and Efficient Human–Robot Collaboration During Assembly Operations
,” ASME J. Comput. Inf. Sci. Eng.
, 18
(3
), p. 031004
. 6.
Morato
, C.
, Kaipa
, K. N.
, Zhao
, B.
, and Gupta
, S. K.
, 2014
, “Toward Safe Human Robot Collaboration by Using Multiple Kinects Based Real-Time Human Tracking
,” ASME J. Comput. Inf. Sci. Eng.
, 14
(1
), p. 011006
. 7.
Ye
, Z.
, Wu
, H.
, and Jia
, J.
, 2021
, “Human Motion Modeling With Deep Learning: A Survey
,” AI Open
, 3
, pp. 35
–39
. 8.
Wang
, Y.
, Ye
, X.
, Yang
, Y.
, and Zhang
, W.
, 2017
, “Collision-Free Trajectory Planning in Human-Robot Interaction Through Hand Movement Prediction From Vision
,” 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids)
, Birgmingham, UK
, Nov. 15–17
, IEEE, pp. 305
–310
.9.
Li
, Q.
, Chalvatzaki
, G.
, Peters
, J.
, and Wang
, Y.
, 2021
, “Directed Acyclic Graph Neural Network for Human Motion Prediction
,” 2021 IEEE International Conference on Robotics and Automation (ICRA)
, Xi'an, China
, May 30–June 5
, IEEE, pp. 3197
–3204
.10.
Martinez
, J.
, Black
, M. J.
, and Romero
, J.
, 2017
, “On Human Motion Prediction Using Recurrent Neural Networks
,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, Honolulu, HI
, July 21–26
, pp. 2891
–2900
.11.
Pavllo
, D.
, Feichtenhofer
, C.
, Auli
, M.
, and Grangier
, D.
, 2020
, “Modeling Human Motion With Quaternion-Based Neural Networks
,” Int. J. Comput. Vision
, 128
(4
), pp. 855
–872
. 12.
Zheng
, P.
, Wieber
, P.-B.
, Baber
, J.
, and Aycard
, O.
, 2022
, “Human Arm Motion Prediction for Collision Avoidance in a Shared Workspace
,” Sensors
, 22
(18
), pp. 6951
. 13.
Wang
, Y.
, Yang
, Y.
, Zhao
, B.
, Qi
, X.
, Hu
, Y.
, Li
, B.
, Sun
, L.
, Zhang
, L.
, and Meng
, M. Q.-H.
, 2021
, “Variable Admittance Control Based on Trajectory Prediction of Human Hand Motion for Physical Human-Robot Interaction
,” Appl. Sci.
, 11
(12
), pp. 5651
. 14.
Zhang
, J.
, Liu
, H.
, Chang
, Q.
, Wang
, L.
, and Gao
, R. X.
, 2020
, “Recurrent Neural Network for Motion Trajectory Prediction in Human-Robot Collaborative Assembly
,” CIRP Ann.
, 69
(1
), pp. 9
–12
. 15.
Gril
, L.
, Wedenig
, P.
, Torkar
, C.
, and Kleb
, U.
, 2023
, “A Tensor-Based Regression Approach for Human Motion Prediction
,” Qual. Reliab. Eng. Int.
, 39
(2
), pp. 481
–499
. 16.
Liao
, H.
, Zheng
, M.
, Hu
, B.
, and Behdad
, S.
, 2022
, “Human Hand Motion Prediction in Disassembly Operations
,” Proceedings of the ASME 2022 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference
, St. Louis, MO
, Aug. 14–17
, p. V005T05A021, Paper No. DETC2022-89967
.17.
Burks
, L.
, Ray
, H. M.
, McGinley
, J.
, Vunnam
, S.
, and Ahmed
, N.
, 2023
, “HARPS: An Online POMDP Framework for Human-Assisted Robotic Planning and Sensing
,” IEEE Trans. Rob.
, 39
(4
), pp. 3024
–3042
. 18.
Sajedi
, S.
, Liu
, W.
, Eltouny
, K.
, Behdad
, S.
, Zheng
, M.
, and Liang
, X.
, 2022
, “Uncertainty-Assisted Image-Processing for Human-Robot Close Collaboration
,” IEEE Robot. Autom. Mag.
, 7
(2
), pp. 4236
–4243
. 19.
Furnari
, A.
, Battiato
, S.
, and Maria Farinella
, G.
, 2018
, “Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation
,” Proceedings of the European Conference on Computer Vision (ECCV) Workshops
, Munich, Germany
, Sept. 8–14
, pp 389
–405
.20.
Abu Farha
, Y.
, and Gall
, J.
, 2019
, “Uncertainty-Aware Anticipation of Activities
,” Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops
, Seoul, South Korea
, Oct. 27–28
, pp. 1197
–1204
.21.
Casalino
, A.
, Mazzocca
, E.
, Di Giorgio
, M. G.
, Zanchettin
, A. M.
, and Rocco
, P.
, 2019
, “Task Scheduling for Human-Robot Collaboration With Uncertain Duration of Tasks: A Fuzzy Approach
,” 2019 7th International Conference on Control, Mechatronics and Automation (ICCMA)
, TU Delft, Netherlands
, Nov. 6–8
, IEEE, pp. 90
–97
.22.
Liu
, W.
, Liang
, X.
, and Zheng
, M.
, 2023
, “Task-Constrained Motion Planning Considering Uncertainty-Informed Human Motion Prediction for Human–Robot Collaborative Disassembly
,” IEEE/ASME Trans. Mechatron.
, 28
(4
), pp. 2056
–2063
. 23.
Xu
, J.
, Chen
, X.
, Lan
, X.
, and Zheng
, N.
, 2021
, “Probabilistic Human Motion Prediction via a Bayesian Neural Network
,” 2021 IEEE International Conference on Robotics and Automation (ICRA)
, Xi'an, China
, May 30–June 5
, IEEE, pp. 3190
–3196
.24.
Yu
, W.
, Liu
, R.
, Zhou
, D.
, Zhang
, Q.
, and Wei
, X.
, 2021
, “An Improved GRU Network for Human Motion Prediction
,” 2021 IEEE 7th International Conference on Virtual Reality (ICVR)
, Foshan, China
, May 20–22
, IEEE, pp. 427
–433
.25.
Ren
, B.
, Zhang
, Z.
, Zhang
, C.
, and Chen
, S.
, 2022
, “Motion Trajectories Prediction of Lower Limb Exoskeleton Based on Long Short-Term Memory (LSTM) Networks
,” Actuators
, 11
(3
), p. 73
. 26.
Yu
, Y.
, Tian
, N.
, Hao
, X. Y.
, Ma
, T.
, and Yang
, C.
, 2022
, “Human Motion Prediction With Gated Recurrent Unit Model of Multi-Dimensional Input
,” Appl. Intell.
, 52
(6
), pp. 1
–13
. 27.
Forti
, V.
, Baldé
, C. P.
, Kuehr
, R.
, and Bel
, G.
, 2020
, “The Global E-Waste Monitor 2020
,” United Nations University (UNU), International Telecommunication Union (ITU) & International Solid Waste Association (ISWA)
, Bonn/Geneva/Rotterdam
, 120.28.
Zuidwijk
, R.
, and Krikke
, H.
, 2008
, “Strategic Response to EEE Returns: Product Eco-Design or New Recovery Processes?
,” Eur. J. Oper. Res.
, 191
(3
), pp. 1206
–1222
. 29.
Hochreiter
, S.
, and Schmidhuber
, J.
, 1997
, “Long Short-Term Memory
,” Neural Comput.
, 9
(8
), pp. 1735
–1780
. 30.
Paszke
, A.
, Gross
, S.
, Massa
, F.
, Lerer
, A.
, Bradbury
, J.
, Chanan
, G.
, Killeen
, T.
, Lin
, Z.
, Gimelshein
, N.
, and Antiga
, L.
, 2019
, “Pytorch: An Imperative Style, High-Performance Deep Learning Library
,” Adv. Neural Inf. Process. Syst.
, 32
(721
), pp. 8026
–8037
. https://dl.acm.org/doi/10.5555/3454287.345500831.
Breiman
, L.
, 1996
, “Bagging Predictors
,” Mach. Learn.
, 24
(2
), pp. 123
–140
. 32.
Husejinovic
, A.
, 2020
, “Credit Card Fraud Detection Using Naive Bayesian and C4. 5 Decision Tree Classifiers
,” Periodicals of Engineering and Natural Sciences
, 8
(1
), pp. 1
–5
. 33.
Hsiao
, H. Y.
, and Chiang
, K. N.
, 2021
, “AI-Assisted Reliability Life Prediction Model for Wafer-Level Packaging Using the Random Forest Method
,” J. Mech.
, 37
, pp. 28
–36
. 34.
Huang
, P.
, Li
, Y.
, Lv
, X.
, Chen
, W.
, and Liu
, S.
, 2020
, “Recognition of Common Non-Normal Walking Actions Based on Relief-F Feature Selection and Relief-Bagging-SVM
,” Sensors
, 20
(5
), pp. 1447
. 35.
Chen
, P.
, Wang
, X.
, Wang
, M.
, Yang
, X.
, Guo
, S.
, Jiang
, C.
, Cui
, G.
, and Kong
, L.
, 2021
, “Multi-View Real-Time Human Motion Recognition Based on Ensemble Learning
,” IEEE Sens. J.
, 21
(18
), pp. 20335
–20347
. 36.
Chen
, P.
, Guo
, S.
, Li
, H.
, Wang
, X.
, Cui
, G.
, Jiang
, C.
, and Kong
, L.
, 2021
, “Through-Wall Human Motion Recognition Based on Transfer Learning and Ensemble Learning
,” IEEE Geosci. Remote Sens. Lett.
, 19
, pp. 1
–5
. 37.
Cho
, K.
, Van Merriënboer
, B.
, Bahdanau
, D.
, and Bengio
, Y.
, 2014
, “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
,” arXiv
. https://arxiv.org/abs/1409.125938.
Graves
, A.
, and Schmidhuber
, J.
, 2005
, “Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures
,” Neural Netw.
, 18
(5–6
), pp. 602
–610
. 39.
Cheng
, W.
, Li
, J.
, Xiao
, H.-C.
, and Ji
, L.
, 2022
, “Combination Predicting Model of Traffic Congestion Index in Weekdays Based on LightGBM-GRU
,” Sci. Rep.
, 12
(1
), pp. 2912
. 40.
Ghalamzan-E
, A.
, 2021
, “Learning Needle Insertion From Sample Task Executions
,” arXiv
. https://arxiv.org/abs/2103.0793841.
Petneházi
, G.
, 2019
, “Recurrent Neural Networks for Time Series Forecasting
,” arXiv
. https://arxiv.org/abs/1901.0006942.
Jospin
, L. V.
, Laga
, H.
, Boussaid
, F.
, Buntine
, W.
, and Bennamoun
, M.
, 2022
, “Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users
,” IEEE Comput. Intell. Mag.
, 17
(2
), pp. 29
–48
. 43.
Lin
, X.
, Zhen
, H.-L.
, Li
, Z.
, Zhang
, Q.
, and Kwong
, S.
, 2018
, “A Batched Scalable Multi-Objective Bayesian Optimization Algorithm
.” arXiv preprint
. https://arxiv.org/abs/1811.0132344.
Maiti
, S.
, and Tiwari
, R. K.
, 2010
, “Neural Network Modeling and an Uncertainty Analysis in Bayesian Framework: A Case Study From the KTB Borehole Site
,” J. Geophys. Res. Solid Earth
, 115
(B10
), pp. B10208-1
–B10208-28
. 45.
Lee
, S.
, Kim
, H.
, and Lee
, J.
, 2022
, “Graddiv: Adversarial Robustness of Randomized Neural Networks via Gradient Diversity Regularization
,” IEEE Trans. Pattern Anal. Mach. Intell.
, 45
(2
), pp. 2645
–2651
. 46.
Gal
, Y.
, and Ghahramani
, Z.
, 2016
, “Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
,” International Conference on Machine Learning
, New York City
, June 19–24
, pp. 1050
–1059
.47.
Wang
, K.
, Du
, H.
, Jia
, R.
, and Jia
, H.
, 2022
, “Performance Comparison of Bayesian Deep Learning Model and Traditional Bayesian Neural Network in Short-Term PV Interval Prediction
,” Sustainability
, 14
(19
), p. 12683
. 48.
Wen
, Y.
, Rahman
, M. F.
, Xu
, H.
, and Tseng
, T.-L. B.
, 2022
, “Recent Advances and Trends of Predictive Maintenance From Data-Driven Machine Prognostics Perspective
,” Measurement
, 187
, pp. 110276
. 49.
Kendall
, A.
, and Gal
, Y.
, 2017
, “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
” Adv. Neural Inf. Process. Syst.
, 30
, pp. 5580
–5590
. https://dl.acm.org/doi/10.5555/3295222.329530950.
Kulshrestha
, S.
, and Patel
, S.
, 2021
, “An Efficient Host Overload Detection Algorithm for Cloud Data Center Based on Exponential Weighted Moving Average
,” Int. J. Commun. Syst.
, 34
(4
), pp. e4708
. 51.
Bishop
, C. M.
, 2006
, Pattern Recognition and Machine Learning
, Springer
, New York
.52.
Hu
, J.
, Zhuang
, Y.
, Yang
, J.
, Lei
, L.
, Huang
, M.
, Zhu
, R.
, and Dong
, S.
, 2018
, “PRNN: A Recurrent Neural Network Based Approach for Customer Churn Prediction in Telecommunication Sector
,” 2018 IEEE International Conference on Big Data (Big Data)
, Los Alamitos, CA
, July 2–7
, IEEE, pp. 4081
–4085
.Copyright © 2025 by ASME
You do not currently have access to this content.
