Abstract

Domain users (DUs) with a knowledge base in specialized fields are frequently excluded from authoring virtual reality (VR)-based applications in corresponding fields. This is largely due to the requirement of VR programming expertise needed to author these applications. To address this concern, we developed VRFromX, a system workflow design to make the virtual content creation process accessible to DUs irrespective of their programming skills and experience. VRFromX provides an in situ process of content creation in VR that (a) allows users to select regions of interest in scanned point clouds or sketch in mid-air using a brush tool to retrieve virtual models and (b) then attach behavioral properties to those objects. Using a welding use case, we performed a usability evaluation of VRFromX with 20 DUs from which 12 were novices in VR programming. Study results indicated positive user ratings for the system features with no significant differences across users with or without VR programming expertise. Based on the qualitative feedback, we also implemented two other use cases to demonstrate potential applications. We envision that the solution can facilitate the adoption of the immersive technology to create meaningful virtual environments.

Issue Section:
Special Issue: Extended Reality in Design and Manufacturing
Keywords:
human computer interfaces/interactions, virtual and augmented reality environments, virtual prototyping
Topics:
Design, Modeling, Virtual reality, Welding, Computer programming, Workflow

References

1.
Lehner
,
V. D.
, and
DeFanti
,
T. A.
,
1997
, “
Distributed Virtual Reality: Supporting Remote Collaboration in Vehicle Design
,”
IEEE Comput. Graph. Appl.
,
17
(
2
), pp.
13
–
17
.
2.
Kato
,
H.
, and
Billinghurst
,
M.
,
1999
, “
Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System
,”
Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99)
,
San Francisco, CA
,
Oct. 20–21
,
IEEE
, pp.
85
–
94
.
3.
Barsom
,
E. Z.
,
Graafland
,
M.
, and
Schijven
,
M. P.
,
2016
, “
Systematic Review on the Effectiveness of Augmented Reality Applications in Medical Training
,”
Surg. Endosc.
,
30
(
10
), pp.
4174
–
4183
.
4.
Zhang
,
Y.
,
Liu
,
H.
,
Kang
,
S.-C.
, and
Al-Hussein
,
M.
,
2020
, “
Virtual Reality Applications for the Built Environment: Research Trends and Opportunities
,”
Autom. Construct.
,
118
, p.
103311
.
5.
Ashtari
,
N.
,
Bunt
,
A.
,
McGrenere
,
J.
,
Nebeling
,
M.
, and
Chilana
,
P. K.
,
2020
, “
Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and Opportunities
,”
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
,
Honolulu, HI
,
Apr. 25–30
, pp.
1
–
13
.
6.
Gaspar
,
H.
,
Morgado
,
L.
,
Mamede
,
H.
,
Oliveira
,
T.
,
Manjón
,
B.
, and
Gütl
,
C.
,
2020
, “
Research Priorities in Immersive Learning Technology: The Perspectives of the ILRN Community
,”
Virtual Real.
,
24
(
2
), pp.
319
–
341
.
7.
Coelho
,
H.
,
Monteiro
,
P.
,
Gonçalves
,
G.
,
Melo
,
M.
, and
Bessa
,
M.
,
2022
, “
Authoring Tools for Virtual Reality Experiences: A Systematic Review
,”
Multimedia Tools Appl.
,
81
(
19
), pp.
28037
–
28060
.
8.
Ens
,
B.
,
Anderson
,
F.
,
Grossman
,
T.
,
Annett
,
M.
,
Irani
,
P.
, and
Fitzmaurice
,
G.
,
2017
, “
Ivy: Exploring Spatially Situated Visual Programming for Authoring and Understanding Intelligent Environments
,”
Proceedings of the 43rd Graphics Interface Conference
,
Edmonton, AB, Canada
,
May 16–19
, pp.
156
–
162
.
9.
Ng
,
G.
,
Shin
,
J. G.
,
Plopski
,
A.
,
Sandor
,
C.
, and
Saakes
,
D.
,
2018
, “
Situated Game Level Editing in Augmented Reality
,”
Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction
,
Stockholm, Sweden
,
Mar. 18–21
, pp.
409
–
418
.
10.
Lin
,
C.-E.
,
Cheng
,
T. Y.
, and
Ma
,
X.
,
2020
, “
Architect: Building Interactive Virtual Experiences From Physical Affordances by Bringing Human-in-the-Loop
,”
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
,
Honolulu, HI
,
Apr. 25–30
, pp.
1
–
13
.
11.
Dai
,
A.
,
Chang
,
A. X.
,
Savva
,
M.
,
Halber
,
M.
,
Funkhouser
,
T.
, and
Nießner
,
M.
,
2017
, “
Scannet: Richly-Annotated 3d Reconstructions of Indoor Scenes
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Honolulu, HI
,
July 21–26
, pp.
5828
–
5839
.
12.
Tchapmi
,
L.
,
Choy
,
C.
,
Armeni
,
I.
,
Gwak
,
J.
, and
Savarese
,
S.
,
2017
, “
Segcloud: Semantic Segmentation of 3d Point Clouds
,”
2017 International Conference on 3D Vision (3DV)
,
Qingdao, China
,
Oct. 10–12
,
IEEE
, p.
537
.
13.
Naseer
,
M.
,
Khan
,
S.
, and
Porikli
,
F.
,
2018
, “
Indoor Scene Understanding in 2.5/3d for Autonomous Agents: A Survey
,”
IEEE Access
,
7
, pp.
1859
–
1887
.
14.
Avetisyan
,
A.
,
Dahnert
,
M.
,
Dai
,
A.
,
Savva
,
M.
,
Chang
,
A. X.
, and
Nießner
,
M.
,
2019
, “
Scan2cad: Learning CAD Model Alignment in RGB-D Scans
,”
Proceedings of the Computer Vision and Pattern Recognition (CVPR)
,
Long Beach, CA
,
June 15–20
,
IEEE
, pp.
2614
–
2623
.
15.
Han
,
M.
,
Zhang
,
Z.
,
Jiao
,
Z.
,
Xie
,
X.
,
Zhu
,
Y.
,
Zhu
,
S.-C.
, and
Liu
,
H.
,
2021
, “
Reconstructing Interactive 3d Scenes by Panoptic Mapping and CAD Model Alignments
,”
2021 IEEE International Conference on Robotics and Automation (ICRA)
,
Xi'an, China
,
May 30–June 5
,
IEEE
, p.
12199
.
16.
Chen
,
K.
,
Lai
,
Y.-K.
, and
Hu
,
S.-M.
,
2015
, “
3d Indoor Scene Modeling From RGB-D Data: A Survey
,”
Comput. Visual Media
,
1
(
4
), pp.
267
–
278
.
17.
Cowan
,
B.
, and
Kapralos
,
B.
,
2014
, “
A Survey of Frameworks and Game Engines for Serious Game Development
,”
2014 IEEE 14th International Conference on Advanced Learning Technologies
,
Athens, Greece
,
July 7–10
,
IEEE
, pp.
662
–
664
.
18.
Kilteni
,
K.
,
Groten
,
R.
, and
Slater
,
M.
,
2012
, “
The Sense of Embodiment in Virtual Reality
,”
Presence: Teleoper. Virtual Environ.
,
21
(
4
), pp.
373
–
387
.
19.
Giunchi
,
D.
,
James
,
S.
,
Degraen
,
D.
, and
Steed
,
A.
,
2019
, “
Mixing Realities for Sketch Retrieval in Virtual Reality
,”
The 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry
,
Brisbane, QLD, Australia
,
Nov. 14–16
, pp.
1
–
2
.
20.
Schultheis
,
U.
,
Jerald
,
J.
,
Toledo
,
F.
,
Yoganandan
,
A.
, and
Mlyniec
,
P.
,
2012
, “
Comparison of a Two-Handed Interface to a Wand Interface and a Mouse Interface for Fundamental 3d Tasks
,”
2012 IEEE Symposium on 3D User Interfaces (3DUI)
,
Costa Mesa, CA
,
Mar. 4–5
,
IEEE
, pp.
117
–
124
.
21.
Seitz
,
S. M.
,
Curless
,
B.
,
Diebel
,
J.
,
Scharstein
,
D.
, and
Szeliski
,
R.
,
2006
, “
A Comparison and Evaluation of Multi-view Stereo Reconstruction Algorithms
,”
2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)
,
New York, NY
,
June 17–22
, Vol.
1
,
IEEE
, pp.
519
–
528
.
22.
Snavely
,
N.
,
Seitz
,
S. M.
, and
Szeliski
,
R.
,
2008
, “
Modeling the World From Internet Photo Collections
,”
Int. J. Comput. Vis.
,
80
(
2
), pp.
189
–
210
.
23.
Furukawa
,
Y.
,
Curless
,
B.
,
Seitz
,
S. M.
, and
Szeliski
,
R.
,
2010
, “
Towards Internet-Scale Multi-view Stereo
,”
2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
,
San Francisco, CA
,
June 13–18
,
IEEE
, pp.
1434
–
1441
.
24.
Lun
,
Z.
,
Gadelha
,
M.
,
Kalogerakis
,
E.
,
Maji
,
S.
, and
Wang
,
R.
,
2017
, “
3d Shape Reconstruction From Sketches Via Multi-view Convolutional Networks
,”
2017 International Conference on 3D Vision (3DV)
,
Qingdao, China
,
Oct. 10–12
,
IEEE
, pp.
67
–
77
.
25.
Ebner
,
T.
,
Feldmann
,
I.
,
Renault
,
S.
,
Schreer
,
O.
, and
Eisert
,
P.
,
2017
, “
Multi-view Reconstruction of Dynamic Real-World Objects and Their Integration in Augmented and Virtual Reality Applications
,”
J. Soc. Inform. Display
,
25
(
3
), pp.
151
–
157
.
26.
Shin
,
D.
,
Ren
,
Z.
,
Sudderth
,
E. B.
, and
Fowlkes
,
C. C.
,
2019
, “
Multi-layer Depth and Epipolar Feature Transformers for 3d Scene Reconstruction
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops
,
Long Beach, CA
,
June 16–17
, pp.
39
–
43
.
27.
Song
,
S.
,
Yu
,
F.
,
Zeng
,
A.
,
Chang
,
A. X.
,
Savva
,
M.
, and
Funkhouser
,
T.
,
2017
, “
Semantic Scene Completion From a Single Depth Image
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Honolulu, HI
,
July 21–26
, pp.
1746
–
1754
.
28.
Dourado
,
A.
,
de Campos
,
T. E.
,
Kim
,
H.
, and
Hilton
,
A.
,
2019
, “
Edgenet: Semantic Scene Completion From RGB-D Images
,” preprint arXiv:1908.02893.
29.
Mandikal
,
P.
, and
Radhakrishnan
,
V. B.
,
2019
, “
Dense 3d Point Cloud Reconstruction Using a Deep Pyramid Network
,”
2019 IEEE Winter Conference on Applications of Computer Vision (WACV)
,
Waikoloa Village, HI
,
Jan. 7–11
,
IEEE
, pp.
1052
–
1060
.
30.
Ochmann
,
S.
,
Vock
,
R.
, and
Klein
,
R.
,
2019
, “
Automatic Reconstruction of Fully Volumetric 3d Building Models From Oriented Point Clouds
,”
ISPRS J. Photogrammetry Remote Sensing
,
151
, pp.
251
–
262
.
31.
Yi
,
L.
,
Shao
,
L.
,
Savva
,
M.
,
Huang
,
H.
,
Zhou
,
Y.
,
Wang
,
Q.
,
Graham
,
B.
, et al.,
2017
, “
Large-Scale 3D Shape Reconstruction and Segmentation From Shapenet Core55
,” preprint arXiv:1710.06104.
32.
Izadi
,
S.
,
Kim
,
D.
,
Hilliges
,
O.
,
Molyneaux
,
D.
,
Newcombe
,
R.
,
Kohli
,
P.
,
Shotton
,
J.
,
Hodges
,
S.
,
Freeman
,
D.
,
Davison
,
A.
, and
Fitzgibbon
,
A.
,
2011
, “
Kinectfusion: Real-Time 3D Reconstruction and Interaction Using a Moving Depth Camera
,”
Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology
,
Santa Barbara, CA
,
Oct. 16–19
, pp.
559
–
568
.
33.
Kim
,
H.
,
Remaggi
,
L.
,
Dourado
,
A.
,
Campos
,
T. D.
,
Jackson
,
P. J.
, and
Hilton
,
A.
,
2021
, “
Immersive Audio-Visual Scene Reproduction Using Semantic Scene Reconstruction From 360 Cameras
,”
Virtual Real.
,
26
(
3
), pp.
823
–
838
.
34.
Jackson
,
B.
, and
Keefe
,
D. F.
,
2016
, “
Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR
,”
IEEE Trans. Visual. Comput. Graph.
,
22
(
4
), pp.
1442
–
1451
.
35.
Shapira
,
L.
, and
Freedman
,
D.
,
2016
, “
Reality Skins: Creating Immersive and Tactile Virtual Environments
,”
2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
,
Merida, Yucatan, Mexico
,
Sept. 19–23
,
IEEE
, pp.
115
–
124
.
36.
Sra
,
M.
,
Garrido-Jurado
,
S.
, and
Maes
,
P.
,
2017
, “
Oasis: Procedurally Generated Social Virtual Spaces From 3d Scanned Real Spaces
,”
IEEE Trans. Visual. Comput. Graph.
,
24
(
12
), pp.
3174
–
3187
.
37.
Hartmann
,
J.
,
Holz
,
C.
,
Ofek
,
E.
, and
Wilson
,
A. D.
,
2019
, “
Realitycheck: Blending Virtual Environments With Situated Physical Reality
,”
Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
,
Glasgow, Scotland, UK
,
May 4–9
, pp.
1
–
12
.
38.
Yang
,
J.
,
Holz
,
C.
,
Ofek
,
E.
, and
Wilson
,
A. D.
,
2019
, “
Dreamwalker: Substituting Real-World Walking Experiences With a Virtual Reality
,”
Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology
,
New Orleans, LA
,
Oct. 20–23
, pp.
1093
–
1107
.
39.
Cheng
,
L.-P.
,
Ofek
,
E.
,
Holz
,
C.
, and
Wilson
,
A. D.
,
2019
, “
Vroamer: Generating On-the-Fly VR Experiences While Walking Inside Large, Unknown Real-World Building Environments
,”
2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
,
Osaka, Japan
,
Mar. 23–27
,
IEEE
, pp.
359
–
366
.
40.
Lindlbauer
,
D.
, and
Wilson
,
A. D.
,
2018
, “
Remixed Reality: Manipulating Space and Time in Augmented Reality
,”
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
,
Montreal. QC, Canada
,
Apr. 21–26
, pp.
1
–
13
.
41.
Masnadi
,
S.
,
González
,
A. N. V.
,
Williamson
,
B.
, and
LaViola
,
J. J.
,
2020
, “
Affordit!: A Tool for Authoring Object Component Behavior in VR
,”
2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)
,
Atlanta, GA
,
Mar. 22–26
,
IEEE
, pp.
741
–
742
.
42.
Prouzeau
,
A.
,
Wang
,
Y.
,
Ens
,
B.
,
Willett
,
W.
, and
Dwyer
,
T.
,
2020
, “
Corsican Twin: Authoring in Situ Augmented Reality Visualisations in Virtual Reality
,”
Proceedings of the International Conference on Advanced Visual Interfaces
,
Salerno, Italy
,
Sept. 28–Oct. 2
, pp.
1
–
9
.
43.
Wang
,
Z.
,
Nguyen
,
C.
,
Asente
,
P.
, and
Dorsey
,
J.
,
2021
, “
Distanciar: Authoring Site-Specific Augmented Reality Experiences for Remote Environments
,”
Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
,
Yokohama, Japan
,
May 8–13
, pp.
1
–
12
.
44.
Qian
,
X.
,
He
,
F.
,
Hu
,
X.
,
Wang
,
T.
,
Ipsita
,
A.
, and
Ramani
,
K.
,
2022
, “
Scalar: Authoring Semantically Adaptive Augmented Reality Experiences in Virtual Reality
,”
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
,
New Orleans, LA
,
Apr. 29–May 5
, pp.
1
–
18
.
45.
Wang
,
Z.
,
Nguyen
,
C.
,
Asente
,
P.
, and
Dorsey
,
J.
,
2022
, “
Point Cloud Capture and Editing for AR Environmental Design
,”
Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
,
Bend, OR
,
Oct. 29–Nov. 2
, pp.
1
–
3
.
46.
Wang
,
Z.
,
Nguyen
,
C.
,
Asente
,
P.
, and
Dorsey
,
J.
,
2023
, “
Pointshopar: Supporting Environmental Design Prototyping Using Point Cloud in Augmented Reality
,”
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
,
Hamburg, Germany
,
Apr. 23–28
, pp.
1
–
15
.
47.
Kumar
,
H. G.
,
Khargonkar
,
N. A.
, and
Prabhakaran
,
B.
,
2018
, “
ScanToVR: An RGB-D to VR Reconstruction Framework
.”
48.
Zhang
,
L.
, and
Oney
,
S.
,
2020
, “
Flowmatic: An Immersive Authoring Tool for Creating Interactive Scenes in Virtual Reality
,”
Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology
,
Virtual Event
,
Oct. 20–23
, pp.
342
–
353
.
49.
Giunchi
,
D.
,
James
,
S.
, and
Steed
,
A.
,
2018
, “
3d Sketching for Interactive Model Retrieval in Virtual Reality
,”
Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-photorealistic Animation and Rendering
,
Victoria, BC, Canada
,
Aug. 17–19
, pp.
1
–
12
.
50.
Giunchi
,
D.
,
Sztrajman
,
A.
,
James
,
S.
, and
Steed
,
A.
,
2021
, “
Mixing Modalities of 3d Sketching and Speech for Interactive Model Retrieval in Virtual Reality
,”
ACM International Conference on Interactive Media Experiences
,
Virtual Event
,
June 21–23
, pp.
144
–
155
.
51.
Valentin
,
J.
,
Vineet
,
V.
,
Cheng
,
M.-M.
,
Kim
,
D.
,
Shotton
,
J.
,
Kohli
,
P.
,
Nießner
,
M.
,
Criminisi
,
A.
,
Izadi
,
S.
, and
Torr
,
P.
,
2015
, “
Semanticpaint: Interactive 3d Labeling and Learning at Your Fingertips
,”
ACM Trans. Graph.
,
34
(
5
), pp.
1
–
17
.
52.
Xu
,
K.
,
Chen
,
K.
,
Fu
,
H.
,
Sun
,
W.-L.
, and
Hu
,
S.-M.
,
2013
, “
Sketch2scene: Sketch-Based Co-retrieval and Co-placement of 3d Models
,”
ACM Trans. Graph.
,
32
(
4
), pp.
1
–
15
.
53.
Yu
,
L.-F.
,
Yeung
,
S.-K.
, and
Terzopoulos
,
D.
,
2015
, “
The Clutterpalette: An Interactive Tool for Detailing Indoor Scenes
,”
IEEE Trans. Visual. Comput. Graph.
,
22
(
2
), pp.
1138
–
1148
.
54.
Eitz
,
M.
,
Hays
,
J.
, and
Alexa
,
M.
,
2012
, “
How Do Humans Sketch Objects
?”
ACM Trans. Graph.
,
31
(
4
), pp.
1
–
10
.
55.
Henshilwood
,
C. S.
,
van Niekerk
,
K. L.
,
Dayet
,
L.
,
Queffelec
,
A.
, and
Pollarolo
,
L.
,
2018
, “
An Abstract Drawing From the 73,000-Year-Old Levels at Blombos Cave, South Africa
,”
Nature
,
562
(
7725
), pp.
115
–
118
.
56.
Yu
,
Q.
,
Liu
,
F.
,
Song
,
Y.-Z.
,
Xiang
,
T.
,
Hospedales
,
T. M.
, and
Loy
,
C.-C.
,
2016
, “
Sketch Me That Shoe
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Las Vegas, NV
,
June 27–30
, pp.
799
–
807
.
57.
Shao
,
T.
,
Xu
,
W.
,
Zhou
,
K.
,
Wang
,
J.
,
Li
,
D.
, and
Guo
,
B.
,
2012
, “
An Interactive Approach to Semantic Modeling of Indoor Scenes With an RGBD Camera
,”
ACM Trans. Graph.
,
31
(
6
), pp.
1
–
11
.
58.
Zhu
,
X.
,
Song
,
L.
,
You
,
L.
,
Zhu
,
M.
,
Wang
,
X.
, and
Jin
,
X.
,
2017
, “
Brush2model: Convolution Surface-Based Brushes for 3d Modelling in Head-Mounted Display-Based Virtual Environments
,”
Comput. Anim. Virtual Worlds
,
28
(
3–4
), p.
e1764
.
59.
Li
,
B.
,
Yuan
,
J.
,
Ye
,
Y.
,
Lu
,
Y.
,
Zhang
,
C.
, and
Tian
,
Q.
,
2021
, “
3d Sketching for 3d Object Retrieval
,”
Multimedia Tools Appl.
,
80
(
6
), pp.
9569
–
9595
.
60.
Luo
,
L.
,
Gryaditskaya
,
Y.
,
Yang
,
Y.
,
Xiang
,
T.
, and
Song
,
Y.-Z.
,
2020
, “
Towards 3d VR-Sketch to 3d Shape Retrieval
,”
2020 International Conference on 3D Vision (3DV)
,
Fukuoka, Japan
,
Nov. 25–28
,
IEEE
, pp.
81
–
90
.
61.
Liu
,
Z.
,
Zhang
,
F.
, and
Cheng
,
Z.
,
2021
, “
Buildingsketch: Freehand Mid-Air Sketching for Building Modeling
,”
2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
,
Bari, Italy
,
Oct. 4–8
,
IEEE
, pp.
329
–
338
.
62.
fxia22
,
2020
, “
A Pytorch Implementation for Pointnet
,” https://github.com/fxia22/pointnet.pytorch
63.
Ipsita
,
A.
,
Li
,
H.
,
Duan
,
R.
,
Cao
,
Y.
,
Chidambaram
,
S.
,
Liu
,
M.
, and
Ramani
,
K.
,
2021
, “
Vrfromx: From Scanned Reality to Interactive Virtual Experience With Human-in-the-Loop
,”
Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
,
Yokohama, Japan
,
May 8–13
, pp.
1
–
7
.
64.
Checa
,
D.
, and
Bustillo
,
A.
,
2020
, “
A Review of Immersive Virtual Reality Serious Games to Enhance Learning and Training
,”
Multimedia Tools Appl.
,
79
(
9
), pp.
5501
–
5527
.
65.
Bhattacharjee
,
S.
, and
Chaudhuri
,
P.
,
2020
, “
A Survey on Sketch Based Content Creation: From the Desktop to Virtual and Augmented Reality
,”
Comput. Graph. Forum
,
39
(
2
), pp.
757
–
780
.
66.
Annett
,
M. K.
, and
Bischof
,
W. F.
,
2010
, “
Investigating the Application of Virtual Reality Systems to Psychology and Cognitive Neuroscience Research
,”
Presence: Teleoper. Virtual Environ.
,
19
(
2
), pp.
131
–
141
.
67.
Garrido
,
D.
,
Rodrigues
,
R.
,
Augusto Sousa
,
A.
,
Jacob
,
J.
, and
Castro Silva
,
D.
,
2021
, “
Point Cloud Interaction and Manipulation in Virtual Reality
,”
2021 5th International Conference on Artificial Intelligence and Virtual Reality (AIVR)
,
Kumamoto, Japan
,
July 23–25
, pp.
15
–
20
.
68.
Yang
,
E. K.
, and
Lee
,
J. H.
,
2020
, “
Cognitive Impact of Virtual Reality Sketching on Designers’ Concept Generation
,”
Digital Creat.
,
31
(
2
), pp.
82
–
97
.
69.
Steinicke
,
F.
, and
Hinrichs
,
K.
,
2006
, “
Grab-and-Throw Metaphor: Adapting Desktop-Based Interaction Paradigms to Virtual Reality
,”
3D User Interfaces (3DUI’06)
,
Alexandria, VA
,
Mar. 25–26
,
IEEE
, p.
83
.
70.
Lee
,
G. A.
,
Kim
,
G. J.
, and
Billinghurst
,
M.
,
2005
, “
Immersive Authoring: What You Experience Is What You Get (wyxiwyg)
,”
Commun. ACM
,
48
(
7
), pp.
76
–
81
.
71.
Chaudhuri
,
S.
, and
Koltun
,
V.
,
2010
, “
Data-Driven Suggestions for Creativity Support in 3d Modeling
,”
ACM SIGGRAPH Asia 2010 Papers
,
Seoul, South Korea
,
Dec. 15–18
, pp.
1
–
10
.
72.
Oh
,
C.
,
Song
,
J.
,
Choi
,
J.
,
Kim
,
S.
,
Lee
,
S.
, and
Suh
,
B.
,
2018
, “
I Lead, You Help But Only With Enough Details: Understanding User Experience of Co-creation With Artificial Intelligence
,”
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
,
Montreal, QC, Canada
,
Apr. 21–26
, pp.
1
–
13
.
73.
Wang
,
T.
,
Qian
,
X.
,
He
,
F.
,
Hu
,
X.
,
Huo
,
K.
,
Cao
,
Y.
, and
Ramani
,
K.
,
2020
, “
Capturar: An Augmented Reality Tool for Authoring Human-Involved Context-Aware Applications
,”
Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology
,
Virtual Event
,
Oct. 20–23
, pp.
328
–
341
.
74.
Labs
,
L.
,
2021
, “
3D Scanner App™
,” https://apps.apple.com/us/app/id1419913995#?platform=ipad
75.
Qi
,
C. R.
,
Su
,
H.
,
Mo
,
K.
, and
Guibas
,
L. J.
,
2017
, “
Pointnet: Deep Learning on Point Sets for 3d Classification and Segmentation
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Honolulu, HI
,
July 21–26
, pp.
652
–
660
.
76.
Aoki
,
Y.
,
Goforth
,
H.
,
Srivatsan
,
R. A.
, and
Lucey
,
S.
,
2019
, “
Pointnetlk: Robust & Efficient Point Cloud Registration Using Pointnet
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Long Beach, CA
,
June 15–20
, pp.
7163
–
7172
.
77.
Cecil Piya
,
V.
,
2016
, “
Realfusion: An Interactive Workflow for Repurposing Real-World Objects Towards Early-Stage Creative Ideation
,”
Graphics Interface
,
Victoria, BC, Canada
,
June 1–3
, pp.
85
–
92
.
78.
Gibson
,
J. J.
,
1977
,
The Theory of Affordances
, Vol.
1
, No.
2
,
Lawrence Erlbaum Associates Inc.
,
Hilldale, NJ
, pp.
67
–
82
.
79.
Wang
,
Y.
,
Chen
,
Y.
,
Nan
,
Z.
, and
Hu
,
Y.
,
2006
, “
Study on Welder Training by Means of Haptic Guidance and Virtual Reality for Arc Welding
,”
2006 IEEE International Conference on Robotics and Biomimetics
,
Kunming, China
,
Dec. 17–20
,
IEEE
, pp.
954
–
958
.
80.
Wu
,
Z.
,
Song
,
S.
,
Khosla
,
A.
,
Yu
,
F.
,
Zhang
,
L.
,
Tang
,
X.
, and
Xiao
,
J.
,
2015
, “
3d Shapenets: A Deep Representation for Volumetric Shapes
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Boston, MA
,
June 7–12
, pp.
1912
–
1920
.
81.
Aoki
,
Y.
,
2020
, “
A Pytorch Implementation for Pointnetlk
,” https://github.com/hmgoforth/PointNetLK
82.
Yang
,
M. C.
, and
Cham
,
J. G.
,
2007
, “
An Analysis of Sketching Skill and Its Role in Early Stage Engineering Design
,”
ASME J. Mech. Des.
,
129
(
5
), pp.
476
–
482
.
83.
Conesa-Pastor
,
J.
, and
Contero
,
M.
,
2022
, “
EVM: An Educational Virtual Reality Modeling Tool; Evaluation Study With Freshman Engineering Students
,”
Appl. Sci.
,
12
(
1
), p.
390
.
84.
Machuca
,
M. D. B.
,
Asente
,
P.
,
Stuerzlinger
,
W.
,
Lu
,
J.
, and
Kim
,
B.
,
2018
, “
Multiplanes: Assisted Freehand VR Sketching
,”
Proceedings of the Symposium on Spatial User Interaction
,
Berlin, Germany
,
Oct. 13–14
, pp.
36
–
47
.
85.
Luo
,
H.
,
Wang
,
C.
,
Wen
,
C.
,
Chen
,
Z.
,
Zai
,
D.
,
Yu
,
Y.
, and
Li
,
J.
,
2018
, “
Semantic Labeling of Mobile Lidar Point Clouds Via Active Learning and Higher Order MRF
,”
IEEE Trans. Geosci. Remote Sens.
,
56
(
7
), pp.
3631
–
3644
.
86.
Li
,
L.
,
Sung
,
M.
,
Dubrovina
,
A.
,
Yi
,
L.
, and
Guibas
,
L. J.
,
2019
, “
Supervised Fitting of Geometric Primitives to 3D Point Clouds
,”
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
,
Long Beach, CA
,
June 15–20
, pp.
2652
–
2660
.
87.
Li
,
Y.
,
Bu
,
R.
,
Sun
,
M.
,
Wu
,
W.
,
Di
,
X.
, and
Chen
,
B.
,
2018
, “
Pointcnn: Convolution on X-Transformed Points
,”
Advances in Neural Information Processing Systems
,
Montréal, Canada
,
Dec. 3–8
, pp.
820
–
830
.
88.
Ipsita
,
A.
,
Erickson
,
L.
,
Dong
,
Y.
,
Huang
,
J.
,
Bushinski
,
A. K.
,
Saradhi
,
S.
,
Villanueva
,
A. M.
,
Peppler
,
K. A.
,
Redick
,
T. S.
, and
Ramani
,
K.
,
2022
, “
Towards Modeling of Virtual Reality Welding Simulators to Promote Accessible and Scalable Training
,”
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
,
New Orleans, LA
,
Apr. 29–May 5
, p.
1
.
89.
Bossard
,
C.
,
Kermarrec
,
G.
,
Buche
,
C.
, and
Tisseau
,
J.
,
2008
, “
Transfer of Learning in Virtual Environments: A New Challenge
?”
Virtual Real.
,
12
(
3
), pp.
151
–
161
.
90.
Blikstein
,
P.
,
2018
, “Maker Movement in Education: History and Prospects,”
Handbook of Technology Education
,
M. J. de Vries, ed.
,
Springer International Publishing, Cham
, Vol.
419
, p.
437
.
91.
Radu
,
I.
,
Joy
,
T.
, and
Schneider
,
B.
,
2021
, “
Virtual Makerspaces: Merging AR/VR/MR to Enable Remote Collaborations in Physical Maker Activities
,”
Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
,
Yokohama, Japan
,
May 8–13
, pp.
1
–
5
.
92.
Pellas
,
N.
,
Kazanidis
,
I.
, and
Palaigeorgiou
,
G.
,
2020
, “
A Systematic Literature Review of Mixed Reality Environments in K-12 Education
,”
Educ. Inform Technol.
,
25
(
4
), pp.
2481
–
2520
.
93.
Birt
,
J.
, and
Cowling
,
M.
,
2017
, “
Toward Future ‘Mixed Reality’ Learning Spaces for Steam Education
,”
Int. J. Innov. Sci. Math. Educ.
,
25
(
4
), pp.
1
–
16
.
94.
Kaminka
,
G. A.
,
2013
, “
Curing Robot Autism: A Challenge
,”
Proceedings of the 2013 International Conference on Autonomous Agents and Multi-Agent Systems
,
St. Paul, MN
,
May 6–10
,
Citeseer
, pp.
801
–
804
.
95.
Szafir
,
D.
,
Mutlu
,
B.
, and
Fong
,
T.
,
2014
, “
Communication of Intent in Assistive Free Flyers
,”
IEEE International Conference on Human–Robot Interaction (HRI)
,
Bielefeld, Germany
,
Mar. 3–6
,
ACM
, pp.
358
–
365
.
96.
Zhang
,
Y.
,
Sreedharan
,
S.
,
Kulkarni
,
A.
,
Chakraborti
,
T.
,
Zhuo
,
H. H.
, and
Kambhampati
,
S.
,
2017
, “
Plan Explicability and Predictability for Robot Task Planning
,”
2017 IEEE International Conference on Robotics and Automation (ICRA)
,
Marina Bay Sands, Singapore
,
May 29–June 3
,
IEEE
, p.
1313
.
97.
Williams
,
T.
,
Szafir
,
D.
,
Chakraborti
,
T.
, and
Ben Amor
,
H.
,
2018
, “
Virtual, Augmented, and Mixed Reality for Human–Robot Interaction
,”
Companion of the 2018 ACM/IEEE International Conference on Human–Robot Interaction
,
Chicago, IL
,
Mar. 5–8
, pp.
403
–
404
.
98.
Cao
,
Y.
,
Xu
,
Z.
,
Li
,
F.
,
Zhong
,
W.
,
Huo
,
K.
, and
Ramani
,
K.
,
2019
, “
V.RA: An In-Situ Visual Authoring System for Robot-IoT Task Planning With Augmented Reality
,”
Proceedings of the 2019 on Designing Interactive Systems Conference
,
San Diego, CA
,
June 23–28
, pp.
1059
–
1070
.
Copyright © 2023 by ASME
You do not currently have access to this content.