Citation review of Human-Machine Collaboration in Virtual Reality for Adaptive Production Engineering by de Giorgio et al.
Andrea de Giorgio
123∙ Mauro Onori
45Abstract
This article argues that despite a citation review is a rarely used research tool, this can be very useful to assess the impact of new research topics, both from the future research direction and the bibliometric perspectives. An explorative study is presented around the research area marked as Industry 4.0 with the conference paper mentioned in the title of this citation review. Even though the given reference paper is relatively recent, there are already twenty-seven citations listed among three different scholar databases. These are Google Scholar, ResearchGate and Semantic Scholar. In light of this, the article provides a bibliometric confirmation and analysis for the progression of the line of research adopted by de Giorgio et al. in the exploration of non-traditional methods using virtual reality technology and human-robot collaboration for adaptive applications in Industry 4.0. Furthermore, it represents a model for the authors’ self-development and an example of an unconventional approach to scientific work that may help improve related bibliometric research and scholar database strategies to index new articles and topics in the future.
Keywords
Virtual Reality, Simulation, Industry 4.0, Human-Robot Collaboration, Citation Review, Citation Metrics.
1. Background
The importance of citation review papers is of growing importance within the context of Industry 4.0, a notion that is slowly gaining ground. As the topic of study becomes ever more multi-disciplinary, the possibility for a research team to critically appraise all of its own developments tends to become incrementally difficult, if not impossible. The majority of research teams dealing with Industry 4.0 are either primarily production engineers or mainly computer & control experts. Ideally, however, one would prefer the developments to be carried out by multi-disciplinary teams in order to capture the essential details required for successful industrial applications. Due to this lack of diversity in the research teams, the citation review articles gain importance as critical appraisals of someone’s own work can be gathered from all disciplines. The work described by de Giorgio et al. (2017), and analyzed in terms of its citations, is a clear example of such multi- disciplinarity as it spans from pure computer science and virtual reality to production engineering and collaborative robots (cobots), with educational and control aspects included. Considering these inherent limitations, such as the common difficulty for mechanical engineers to abstract architectural requirements, or general engineers’ limitations in pedagogical approaches, the need to critically appraise the research becomes more evident.
1 Andrea de Giorgio andreadg@kth.se
https://andreadegiorgio.com
2KTH Royal Institute of Technology, Department of Production Engineering, SE-11428 Stockholm, Sweden
3ORCID 0000-0001-6064-5634
4KTH Royal Institute of Technology, Department of Production Engineering, SE-11428 Stockholm, Sweden
5ORCID 0000-0002-0006-283X
© 2020 by the author(s). Distributed under a Creative Commons CC BY license.
2. Introduction
Despite rare, but not unprecedented in its setup (Ho 2004), this paper is a detailed literature review of all the publications citing the conference paper “Human-machine Collaboration in Virtual Reality for Adaptive Production Engineering” authored by de Giorgio et al. (2017) and published on Elsevier’s journal Procedia Manufacturing. The reasoning being that reviewing an article’s citations may be used as a fundamental scientific process in the progression of a certain line of research and authors’ self-development.
Citation analysis, as a sub area of bibliometric and scientometrics, is not a young discipline, dating as far back as the 1950s (Baker 1990). It includes different forms of analyses, the simplest one being a citation count. This may be used to investigate the performance of scientists, institutions, nations and journals. Garfield (1979) points out that these uses of citation analysis generate the most controversy and heated debate among academicians and scientists. More complex methods include bibliographic coupling (Kessler 1963), document co-citation analysis (Eugene Garfield et al. 2005), author co-citation analysis (McCain 1986) and co-word analysis (Rip and Courtial 1984).
Bornmann and Daniel (2008) provide most of the literature review on citing behavior. Several studies have tried to define categories of reasons why scientific papers are cited. Among the studies, Chubin and Moitra (1975) show that, in 13% of full-length articles, the cited paper can be central to the reported research, i.e. the reported findings depend on the cited paper (basic essential citation). Otherwise, in 34% of the articles, the cited paper is not directly connected to the subject of the article, but still essential to the reported research. In 32% of the articles, the cited paper contains an independent supportive observation (idea or finding) with which the citer agrees (additional supplementary citation). In 17% of the articles, the paper is cited without any additional comments (perfunctory supplementary citation). Only in 4% of the cases, the citer suggests that the cited paper is erroneous in part and offers a correction (partial negational citation). There are practically no cases in which the citer refers to the cited paper as being completely wrong and offers an independent interpretation of solution (total negational citation).
Krampen and Montada (2002) define another category system. In 30% of the citations, there is a direct reference to an empirical finding in the cited document, e.g. a simple mention of the type “compare here also”,
“see also”, “see, for example”, etc. In 20% of the citations, there is a direct reference to a theory or concept in the cited document. In 9% of the citations, there is a direct reference to a method in the cited document. In 5% of the citations, there is an overview call to look in the cited document for more information. In 3% of the citations, there is a use of a data collection method, e.g. a test, taken from the cited document. In 3% of the citations appears a word-for-word quotation of text in the cited document. The remaining percentage is about substantial, theoretical, or methodological critique of the cited document or use of a table, a figure, or a list taken from the cited document.
The literature review shows that there are plenty of reasons to cite a paper (Bornmann and Daniel 2008);
therefore, this citation review article looks at the citing sources with an open mind, in order to spot patterns and citation behavior that might shed light on future trends, connected topics, and more.
This article is organized as follows. In section 3, the citation search methodology is explained and all the review source publications are listed. In section 4, all the results from the citation review are reported, including the related discussions. Finally, conclusions and future work are presented in section 5.
3. Citation search methodology
In order to retrieve the widest number of citations for this relatively young article, three large scholar databases are selected. Namely, Google Scholar (GS), ResearchGate (RG) and Semantic Scholar (SS).
3.1. Google Scholar
Google Scholar lists 23 citations for the paper. Five are journal articles (Anik et al. 2020; Malik et al. 2020;
Oliveira Valério et al. 2020; Peruzzini et al. 2020; Randon and Cecconello 2019), thirteen are conference papers (Arntz and Eimler 2020; H.-P. Balzerkiewitz and Stechert 2020; H. P. Balzerkiewitz and Stechert 2020; Chang et al. 2019; Etzi et al. 2019; Metzner, Utsch, et al. 2020; Mizuchi and Inamura 2018; Moon et al. 2019; Ojstersek et al. 2019; Ottogalli et al. 2018; Shu et al. 2018; Shu and Sziebig 2019; Yan et al. 2019), one is a preprint (Gao et al. 2019a) and four are theses (Gong 2019; Gustavsson 2020; Lloyd Reyson Robles 2020; Tolman 2018).
GS is missing four articles that are reported by RG. Among those, three are conference papers (Gao et al.
2019b; Janis and Alias 2018; Metzner, Krieg, et al. 2020), specifically, “Intuitive, VR- and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models” (Metzner, Krieg, et al. 2020), “VRKitchen: an
Interactive 3D Environment for Learning Real Life Cooking Tasks” (Gao et al. 2019b) from 2019 and “A Systematic Literature Review: Human Roles, Competencies And Skills In Industry 4.0” (Janis and Alias 2018);
the preprint is “Augmented Perception: Can virtual environments have an impact on well-being and work effectiveness?” (Marszalek and Volz 2018).
3.2. ResearchGate
ResearchGate lists 19 citations for the paper. However, there is one paper that has been uploaded twice using the format “code” and the number of real citations is 18. Of these, four are journal articles (Anik et al. 2020;
Malik et al. 2020; Peruzzini et al. 2020; Randon and Cecconello 2019), twelve are conference papers (H.-P.
Balzerkiewitz and Stechert 2020; Chang et al. 2019; Gao et al. 2019b; Janis and Alias 2018; Metzner, Krieg, et al.
2020; Mizuchi and Inamura 2018; Moon et al. 2019; Ojstersek et al. 2019; Ottogalli et al. 2018; Shu et al. 2018;
Shu and Sziebig 2019; Yan et al. 2019) and two are preprints (Gao et al. 2019a; Marszalek and Volz 2018). One of the journal articles is in Portuguese (Randon and Cecconello 2019).
Five citations are from 2018 (Janis and Alias 2018; Marszalek and Volz 2018; Mizuchi and Inamura 2018;
Ottogalli et al. 2018; Shu et al. 2018), eight citations are from 2019 (Chang et al. 2019; Gao et al. 2019b, 2019a;
Moon et al. 2019; Ojstersek et al. 2019; Randon and Cecconello 2019; Shu and Sziebig 2019; Yan et al. 2019) and five citations are from 2020 (Anik et al. 2020; H.-P. Balzerkiewitz and Stechert 2020; Malik et al. 2020;
Metzner, Utsch, et al. 2020; Peruzzini et al. 2020).
RG is missing nine articles that are reported by GS. Among those, one is the journal article “Strategic systematic for software development in industry 4.0” (Oliveira Valério et al. 2020). Four are conference papers (Arntz and Eimler 2020; H. P. Balzerkiewitz and Stechert 2020; Etzi et al. 2019; Metzner, Utsch, et al. 2020), specifically “Experiencing AI in VR: A Qualitative Study on Designing a Human-Machine Collaboration Scenario”
(Arntz and Eimler 2020), “Use of Virtual Reality in Product Development by Distributed Teams” (H. P.
Balzerkiewitz and Stechert 2020), “A system for human-in-the-loop simulation of industrial collaborative robot applications” (Metzner, Utsch, et al. 2020) and “Using virtual reality to test human-robot interaction during a collaborative task” (Etzi et al. 2019). Four are theses (Gong 2019; Gustavsson 2020; Lloyd Reyson Robles 2020;
Tolman 2018).
3.3. Semantic Scholar
Semantic Scholar lists 19 citations for the paper. Four are journal articles (Anik et al. 2020; Malik et al. 2020;
Oliveira Valério et al. 2020; Peruzzini et al. 2020), thirteen are conference papers (H.-P. Balzerkiewitz and Stechert 2020; Chang et al. 2019; Gao et al. 2019b; Janis and Alias 2018; Metzner, Krieg, et al. 2020; Metzner, Utsch, et al. 2020; Mizuchi and Inamura 2018; Moon et al. 2019; Ojstersek et al. 2019; Ottogalli et al. 2018; Shu et al. 2018; Shu and Sziebig 2019; Yan et al. 2019) and one is a preprint (Gao et al. 2019a). One citation is wrongly directed to the whole conference proceedings containing the conference paper “Intuitive, VR- and Gesture- based Physical Interaction with Virtual Commissioning Simulation Models” (Metzner, Krieg, et al. 2020).
SS is missing 9 citations from RG and GS. In particular, the Portuguese journal article “Simulação como Tecnologia Habilitadora da Indústria 4.0: Uma Revisão da Literatura” (Randon and Cecconello 2019); three conference papers (Arntz and Eimler 2020; H. P. Balzerkiewitz and Stechert 2020; Etzi et al. 2019) that are
“Using virtual reality to test human-robot interaction during a collaborative task” (Etzi et al. 2019),
“Experiencing AI in VR: A Qualitative Study on Designing a Human-Machine Collaboration Scenario” (Arntz and Eimler 2020) and “Use of Virtual Reality in Product Development by Distributed Teams” (H. P. Balzerkiewitz and Stechert 2020); one preprint (Marszalek and Volz 2018) and four theses (Gong 2019; Gustavsson 2020;
Lloyd Reyson Robles 2020; Tolman 2018).
4. Results and discussion
As result of the scholar search over three databases, there are 27 citations for the paper in exam. Five are journal articles (Anik et al. 2020; Malik et al. 2020; Oliveira Valério et al. 2020; Peruzzini et al. 2020; Randon and Cecconello 2019), sixteen are conference papers (Arntz and Eimler 2020; H.-P. Balzerkiewitz and Stechert 2020; H. P. Balzerkiewitz and Stechert 2020; Chang et al. 2019; Etzi et al. 2019; Gao et al. 2019b; Janis and Alias 2018; Metzner, Krieg, et al. 2020; Metzner, Utsch, et al. 2020; Mizuchi and Inamura 2018; Moon et al. 2019;
Ojstersek et al. 2019; Ottogalli et al. 2018; Shu et al. 2018; Shu and Sziebig 2019; Yan et al. 2019), two are preprints (Gao et al. 2019a; Marszalek and Volz 2018) and four are theses (Gong 2019; Gustavsson 2020; Lloyd Reyson Robles 2020; Tolman 2018). See Fig. 1 for a graphic illustration of the citation types. To be precise, the
total count goes to 28 if the necessary citation provided by this very paper (de Giorgio et al. 2017) is counted.
The citations are increasing year by year. In fact, in 2018 there are five citations, in 2019 eleven and in 2020 eleven (see Fig. 2).
A keyword analysis shows ninety-nine keywords associated with the citing work. The most common ones are Virtual Reality (twelve occurrences), Simulation (five occurrences) and Human-Robot-Collaboration (four occurrences). For a complete overview of the keyword count, see table 1. A treemap shows graphically the keyword distribution in figure 3.
Fig. 1 Type of publications Fig. 2 Citations per year
Fig. 3 Keyword treemap Fig. 4 Citations per country of the first author
Table 1 Most common keywords of the citing articles
Keyword Count
Virtual Reality 12
Simulation 5
Human-Robot Collaboration 4
Ergonomics 3
Human-Machine Collaboration 2
Industrial Robot 2
Product Development 2
Competencies and Skills 2
Engineering 2
Industry 4.0 2
Other 63
Total keywords 99
0 2 4 6 8 10 12
2018 2019 2020
The citing work comes from 13 countries with the distribution shown in Fig. 4. In particular, six citations come from USA, five from Germany, two from Brazil, Norway, Italy and Sweden. The rest of the countries give one citation each.
4.1. Summary of findings
The related work section of Peruzzini et al. (2020) lists de Giorgio et al. (2017) as those who “showed the potentiality of virtual reality (VR) in human-robot collaboration, using game engines as simulation software to control AR/VR hardware and industrial machines”. Further in the article, a table named “Comparison between different VM-based approaches” lists de Giorgio et al. (2017) as an application of collaborative robotics with a level of realism of score 3 out of 4, using Unity as VR software and HTC Vive with VR Controllers as hardware.
The table also explains that de Giorgio et al. (2017) performed a qualitative interaction evaluation over a VR application with level of interaction that is “360° visual” other than “tactile with feedback”. These results are of some importance as they may indicate in which area there may exist a need to improve or further elaborate.
In a section called “basics”, Balzerkiewitz and Stechert (2020) state that virtual reality is often characterized by five characteristics that are stated by several authors, including de Giorgio et al. (2017), these are a virtual world, immersion, feedback, interactivity and participants. This entails that the basic foundations of the cited article does find positive theoretical backing from other authors, which offers grounds for validation.
In the introduction, Ojstersek et al. (2019) use de Giorgio et al. to claim that “researchers relate the VR model to the real environment in cooperative workplaces, which represents a rapidly growing research field with broad applicative solutions”. Once again, this enhances the basic validity of the original work.
In the related work section, Metzner et al. (2020) use de Giorgio et al. (2017) as an example of research that implements robot models into a Unity®-based environment and uses it for VR visualization. The related findings, described in this article, also offer a notion of what may be improved in the future.
In the related work section, Gao et al. (2019b) present de Giorgio et al. (2017) as an example of research in human-robot interaction.
Similarly, Malik et al. (2020) cite de Giorgio et al. (2017) for “discussing an early-stage demonstrator to highlight the value of robot programming via VR using Unity game engine”. These two citations strengthen the validity of using such a novel approach.
Anik et al. (2020) use de Giorgio et al. to justify the opening sentence of the introduction section: “Recent breakthroughs in head-mounted display (HMD), positional tracking system, and software platform including game engines, etc., have been shifting the paradigm of virtual reality (VR) technology.”
In the related work section, Gao et al. (2019a) present de Giorgio et al. (2017) as an example of research in human-robot interaction. Note that this article is the preprint of “VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks” (Gao et al. 2019b).
Shu et al. (2018) cite de Giorgio et al. (2017) in their literature review section, for introducing a simulated robot controller using the Unity software built-in kinematics to achieve a realtime control of a specific type of ABB robot. They further point out that de Giorgio et al. (2017) use Robot Operating System (ROS) as a middleware to drive the robot in the VR world, and give feedback to the user using the HTC Vive hardware. This citation is of some importance as it indicates that there are other scientists that believe in the validity of the used software tools.
In the related work section, Ottogalli et al. (2018) cite de Giorgio et al. (2017) as a work in the field of Human- Robot Collaboration (HRC) in assembly. The work addresses the advantages, disadvantages and open questions regarding the use of VR for the simulation of HRC in production engineering, and explains the development of a virtual collaborative robot manipulator in Unity, controlled via HTC Vive hardware.
In a table named “Summary of the human role, technical competencies and skills required by Industry 4.0 – Manufacturing Sector”, Janis and Alias (Janis and Alias 2018) refer to de Giorgio et al. (2017) among a few works requiring the learning and use of an assistance system, in the customized end-user manufacturing sector, where humans are still required at the shop floor production as low skilled workers.
Mizuchi and Inamura (2018) present de Giorgio et al. (2017) in their introduction, as an example of an application of VR immersive systems for human-robot collaboration in production engineering.
Also in their introduction, Chang et al. (2019) cite de Giorgio et al. (2017) among others as industrial applications that simulate the process in order to prevent machine collisions.
Among the main findings of their literature review, Moon et al. (2019) present de Giorgio et al. (2017) as one of the papers about Virtual Learning Environments (VLE) for industrial robotics. In particular, as part of 35%
of the VLE applications in their review that are based on interactive scenarios, and 55% of those presenting the human-robot interaction as the main subject of study.
Shu et al. (2019) cite de Giorgio et al. (2017) for their experiment carried out with simulation tools and real- time connection to a physical robot. They complain about the limited use of only one type of robot in the solution presented.
Yan et al. (2019) affirm that de Giorgio et al. (2017) “outlined the main steps of an open adaptive human- machine collaborative simulation approach, explored the use of the latest software in the gaming industry for VR-based industrial production to overcome the limitations of industrial software. Furthermore, the relevance of the approach that considers both new software and hardware solutions is investigated in their paper”.
The article “Augmented Perception: Can virtual environments have an impact on well-being and work effectiveness?” announced in preprint by Marszalek and Volz (2018) is not retrievable.
Randon and Cecconello (2019) refer to de Giorgio et al. (2017) as one of 56 articles retrieved from Web of Science for their literature review of simulation as an enabling technology of Industry 4.0. However, the paper is not specifically mentioned for any particular results. They claim that Procedia Manufacturing, i.e. the publishing venue for de Giorgio et al. (2017), results to be the highest journal source with six papers.
In the background section of Tolman’s thesis (Tolman 2018), de Giorgio et al. (2017) is presented as related work done in finding VR simulation methods of human-robot collaboration (HRC) in production engineering.
Tolman notes that de Giorgio et al. (2017) provides a practical implementation in Unity as proof of concept.
In the introduction, Arntz and Eimler (2020) use de Giorgio et al. (2017) to claim that Virtual Reality (VR) delivers a tool to validate concepts for the production processes of the future, before that they can become reality.
In the introduction, Lloyd Reyson Robles (2020) uses de Giorgio et al. (2017) as an example of a research project that uses the game engine Unity to develop a VR simulation for an ABB IRB 120 robot manipulator. A figure is extracted from de Giorgio et al. (2017) and used with an explicit mention. Once again, these last few citations enforce the validity of the used software tools.
In the related work section, in the subsection related to the use of virtual reality in human-robot collaboration, Etzi et al. (2019) mention the work of de Giorgio et al. (2017) as a human-robot platform in which the user “wears a head-mounted display (HDM; HTC Vive™) and moves a robotic arm (ABB IRB 120) through a controller to perform an assembly task”.
In a section named “a brief overview about virtual reality”, Balzerkiewitz and Stechert (2020) mention de Giorgio et al. (2017) among other works to state that virtual reality is often characterized by five characteristics:
a virtual world, immersion, feedback, interactivity and participants. Later in the article, they further observe through de Giorgio et al. (2017), among a few works, that most CAD metadata, e.g. constraints, color, and material, is lost during the CAD exchanging process from the virtual reality world into classical CAD software.
In a section named “state of the art in human-robot collaboration system simulation”, Metzner et al. (2020) mention de Giorgio et al. as those who “integrate an industrial robot kinematic into the gaming engine Unity™
to enable VR visualization and virtual control of the robot arm”.
Oliveira Valério et al. (2020) cite de Giorgio et al. (2017) in a section named “software development and industry 4.0”: they name this work, among others, as those identifying the risks/challenges of software integration and new technologies. These are “the lack of systems compatibility, the labor replacement, the impact of consumer relations, the structure of big data, the security and robustness of information systems (legislation), the expansion of Internet services and communication, and the diffusion of the new technological scenario for students and researchers”. The article further uses de Giorgio et al. (2017) to justify the claim that
“possible risks/challenges should be discussed at the stage concerning the impact of extremely exclusive consumer and product relationships (Determine how customer dissatisfaction and possible discards should be regulated).”
In the related work of Gong’s thesis (Gong 2019), de Giorgio et al. (2017) is presented as an example of research in human-robot interaction with virtual reality. Note that this thesis contains the material used to produce the preprint “VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks” (Gao et al. 2019b) and the conference paper “VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning” (Gao et al. 2019a).
In a section on human-robot collaboration, subsection about virtual commissioning and virtual reality, Gustavsson (2020) cites de Giorgio et al. (2017) among “recent studies that have shown great potential in virtual training and testing using VR, which potentially could include the human in the commissioning process using a virtual plant”. Note that this PhD thesis links to a paper named “Evaluation of Human-Robot Interaction for
Assembly Manufacturing in Virtual Reality” authored by Patrik Gustavsson, Magnus Holm and Anna Syberfeldt, sent to Elsevier for publication. This paper cites de Giorgio et al. (2017) in the related work section, as a research concluding that “virtual reality for human-robot collaboration provides the possibilities for the operator to learn how to perform manufacturing processes.”
4.2. Citation clusters
Among the most citing authors are Ran Gong, Xiaofeng Gao and their colleagues with the VRKitchen series (Gao et al. 2019a, 2019b; Gong 2019). They start in 2019 with Ran Gong’s thesis “VRKitchen: A 3D Dynamic Interactive Environment for General Computer Vision Research” (Gong 2019), then publish in the same year a preprint “VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning” (Gao et al. 2019a) and a conference article “VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks” (Gao et al. 2019b).
Beibei Shu and colleagues in 2018 and 2019 publish two conference papers, namely “Human-robot collaboration: Task sharing through Virtual Reality” (Shu et al. 2018) and “Collaboration with high-payload industrial robots: Simulation for safety” (Shu and Sziebig 2019).
Hans Patrick Balzerkiewitz and colleagues in 2020 publish two conference papers, namely “The evolution of virtual reality towards the usage in early design phases” (H.-P. Balzerkiewitz and Stechert 2020) and “Use of Virtual Reality in Product Development by Distributed Teams” (H. P. Balzerkiewitz and Stechert 2020).
Maximilian Metzner and colleagues in 2020 publish two conference papers, namely “A system for human- in-the-loop simulation of industrial collaborative robot applications” (Metzner, Utsch, et al. 2020) and “Intuitive, VR- and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models” (Metzner, Krieg, et al. 2020).
4.4. Citation errors
While all the citations are correctly addressing the content of the cited paper, the author’s name used for the citation is often wrong. In fact, 45% of the citations are wrongly spelled. Half of them are simply using a capitalized “De” instead of an uncapitalized “de” for the last name, which appears as “De Giorgio et al.” Among the others, “Giorgio, A. D. et al.”, “Giorgio, A. et al.”, “Giorgio A de et al.” are quite wrong. Lastly, the wrong expression “Andrea De Giorgio et al.” seems to match the almost correct expression “Andrea de Giorgio et al.”, used if one wants to avoid separating name and last name by including both.
Despite the many errors, the scholar databases seem in most cases able to catch the reference to the paper.
However, there are a few missing documents in one search engine or another and the misspelling of the citation key could play a role in this problem. Sadly, in this citation review there are not enough missing samples to investigate such hypothesis. Future work should include such an investigation on a large source of misspelled citations.
4.5. Document type errors
Many documents are reported by the scholar database with the wrong document type attribution, i.e. journal article, conference paper, chapter, etc. In particular, on ResearchGate six conference papers are recorded as journal article (two occurrences), as chapter (three occurrences) or as data (one occurrence). One document on Semantic Scholar is named after the whole conference title in spite of the conference paper name (this has been removed from the citation review for obvious reasons); the conference paper was also reported in the same database as a separate document and included in this citation review. In one case, on RG the conference paper had the wrong publication year. Google Scholar and Semantic Scholar do not explicitly define the document type, thus it is not possible to perform the same document type analysis done for RG. However, the citing tool offered by the original publisher (in particular Springer) or the citation database extractor tend to wrongly mark conference paper types as a chapter types.
5. Conclusions
From the summary of findings, it is possible to count seven citations among the introductions. Of those, one is particularly interesting as Anik et al. (2020) open their paper citing de Giorgio et al. as an example of application that is possible due to recent breakthroughs in VR technology, placing de Giorgio et al. at the top of its scope. A higher number of citations comes from the related work sections (eleven occurrences) and literature review sections (one occurrence), which mainly attests the similarity of scope between de Giorgio et al. and the
citing articles. Equally great, with ten citations, is the number of mentions in the central sections of the citing articles, which provides a reasonable proof that de Giorgio et al. paves the way to new findings in the application of virtual reality in human-robot collaboration for adaptive production engineering.
From the work of Randon and Cecconello (2019) it results that the choice of publication venue, i.e. Procedia Manufacturing as part of the conference proceedings, was effective in delivering this kind of content to a broad audience. They point out that applications of simulation for Industry 4.0 are mostly published and cited from Procedia Manufacturing instead of other important journals in the field of manufacturing.
On the other hand, only 18% of the citations come from journal articles, while 59% of them come from other conference papers. Another 15% come from theses and 7% come from preprints that do not specify what type of article will be published.
Particularly appreciated is the reviewed definition of the five main characteristics of VR applications: a virtual world, immersion, feedback, interactivity and participants. Also the valid industrial scenario presented in the paper that allows to evaluate advantages and risks of applying VR in human-robot collaborative applications for Industry 4.0.
None of the citing papers tries to undermine or criticize the work presented by de Giorgio et al. (2017), which is expected as it relates to the low percentage of this kind of citations. Only one comment addresses a limitation regarding the test of the industrial scenario with only one robot model. Indeed, a perfect example of future work to develop. As named earlier, these citations, once put into their correct context, offer both validation data and potential areas of improvement. In other words, the citation review may act as an external tutorship in which both similar and adjacent disciplines can analyze the results within specific contexts.
One limitation of this citation review is the young age of the publication in exam and the relative low number of citations that can be collected from it. Despite that, having few citations allows to track them more precisely and it shows that scholar databases are constantly missing citations and each one of them indexes only a limited subset of all the existing scientific publications. A huge result is to count 27 citations over a maximum of 23 spotted by Google Scholar, and 19 different ones spotted by ResearchGate and Semantic Scholar.
A quick overview of the keyword treemap presented in Fig. 3 reveals what was mentioned earlier: there are very few keywords that stem from Industry 4.0-related work. This indicates that although this topic should be focusing on the IT and software enhancement of industry, the interest for virtual reality applications in industry is still coming from its conceptual developers. This should be highlighted since the integration of new technologies in a mature industrial environment is notoriously difficult.
Lastly, a reasonable speculation that can be done in conclusion of this work is that, eventually, the fact that this very article will cite non-indexed scientific work could lead to the scholar databases to use the citations hereby provided to track and index them. In fact, citations are the basis of automated document retrieval and indexing algorithms. Time will help to verify this hypothesis.
References
Anik, A. A., Xavier, B. A., Hansmann, J., Ansong, E., Chen, J., Zhao, L., & Michals, E. (2020). Accuracy and Reproducibility of Linear and Angular Measurements in Virtual Reality: a Validation Study. Journal of Digital Imaging, 33(1), 111–120.
https://doi.org/10.1007/s10278-019-00259-3
Arntz, A., & Eimler, S. C. (2020). Experiencing AI in VR: A Qualitative Study on Designing a Human-Machine Collaboration Scenario. In International Conference on Human-Computer Interaction (pp. 299–307). Springer, Cham.
https://doi.org/10.1007/978-3-030-60700-5_38
Baker, D. R. (1990). Citation analysis: A methodological review. Social Work Research and Abstracts, 26(3), 3–10.
https://doi.org/10.1093/swra/26.3.3
Balzerkiewitz, H.-P., & Stechert, C. (2020). The evolution of virtual reality towards the usage in early design phases. In Proceedings of the Design Society: DESIGN Conference (Vol. 1, pp. 91–100). Cambridge University Press (CUP).
https://doi.org/10.1017/dsd.2020.159
Balzerkiewitz, H. P., & Stechert, C. (2020). Use of Virtual Reality in Product Development by Distributed Teams. In Procedia CIRP (Vol. 91, pp. 577–582). Elsevier B.V. https://doi.org/10.1016/j.procir.2020.02.216
Bornmann, L., & Daniel, H. D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64(1), 45–80. https://doi.org/10.1108/00220410810844150
Chang, W. Y., Hsu, B. Y., & Hsu, J. W. (2019). Real-time collision avoidance for Five-Axis CNC machine tool based on cyber- physical system. In Proceedings of the 2018 IEEE International Conference on Advanced Manufacturing, ICAM 2018 (pp.
284–287). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/AMCON.2018.8615042
Chubin, D. E., & Moitra, S. D. (1975). Content analysis of references: Adjunct or alternative to citation counting? Social studies of science, 5(4), 423–441.
de Giorgio, A., Romero, M., Onori, M., & Wang, L. (2017). Human-machine Collaboration in Virtual Reality for Adaptive Production Engineering. Procedia Manufacturing, 11. https://doi.org/10.1016/j.promfg.2017.07.255
Etzi, R., Huang, S., Scurati, G. W., Lyu, S., Ferrise, F., Gallace, A., et al. (2019). Using virtual reality to test human-robot interaction during a collaborative task. In Proceedings of the ASME Design Engineering Technical Conference (Vol. 1).
American Society of Mechanical Engineers (ASME). https://doi.org/10.1115/DETC2019-97415
Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., & Zhu, S.-C. (2019a). VRKitchen: an Interactive 3D Virtual Environment for Task- oriented Learning. arXiv. arXiv. http://arxiv.org/abs/1903.05757. Accessed 14 November 2020
Gao, X., Gong, R., Shu, T., Xie, X., Wang, S., & Zhu, S.-C. (2019b). VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks. In ICML 2019 Workshop RL4RealLife. https://openreview.net/forum?id=rygmCx3vjE. Accessed 14 November 2020
Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scientometrics, 1(4), 359–375.
https://doi.org/10.1007/BF02019306
Garfield, Eugene, Pudovkin, A. I., & Istomin, V. S. (2005). Algorithmic citation-linked historiography-Mapping the literature of science. Proceedings of the American Society for Information Science and Technology, 39(1), 14–24.
https://doi.org/10.1002/meet.1450390102
Gong, R. (2019). VRKitchen: A 3D Dynamic Interactive Environment for General Computer Vision Research. Retrieved from https://escholarship.org/uc/item/20r9k1md
Gustavsson, P. (2020). Virtual Reality Platform for Design and Evaluation of the Interaction in Human-Robot Collaborative Tasks in Assembly Manufacturing. University of Skövde. Retrieved from http://urn.kb.se/resolve?urn=urn%3Anbn%3Ase%3Ahis%3Adiva-18919
Ho, Y. S. (2004). Citation review of Lagergren kinetic rate equation on adsorption reactions. Scientometrics, 59(1), 171–177.
https://doi.org/10.1023/B:SCIE.0000013305.99473.cf
Janis, I., & Alias, M. (2018). A Systematic Literature Review: Human Roles, Competencies And Skills In Industry 4.0. In AIMC 2017 Asia International Multidisciplinary Conference (pp. 1052–1072). Cognitive-Crcs.
https://doi.org/10.15405/epsbs.2018.05.84
Kessler, M. M. (1963). Bibliographic coupling between scientific papers. American Documentation, 14(1), 10–25.
https://doi.org/10.1002/asi.5090140103
Krampen, G., Burkard, P., & Montada, L. (2002). Wissenschaftsforschung in der Psychologie. Hogrefe, Verlag für Psychologie.
Lloyd Reyson Robles, E. (2020). Virtual Simulation of Robotic Operations Using Gaming Software. Retrieved from https://oaktrust.library.tamu.edu/handle/1969.1/188386
Malik, A. A., Masood, T., & Bilberg, A. (2020). Virtual reality in manufacturing: immersive and collaborative artificial-reality in design of human-robot workspace. International Journal of Computer Integrated Manufacturing, 33(1), 22–37.
https://doi.org/10.1080/0951192X.2019.1690685
Marszalek, Y., & Volz, D. (2018). Augmented Perception: Can virtual environments have an impact on well-being and work effectiveness? In Deutsche Gesellschaft für Designtheorie und -forschung. http://conference2018.dgtf.de/
McCain, K. W. (1986). Cocited author mapping as a valid representation of intellectual structure. Journal of the American Society for Information Science, 37(3), 111–122. https://doi.org/10.1002/(SICI)1097-4571(198605)37:3<111::AID- ASI2>3.0.CO;2-D
Metzner, M., Krieg, L., Krüger, D., Ködel, T., & Franke, J. (2020). Intuitive, VR- and Gesture-based Physical Interaction with Virtual Commissioning Simulation Models. In Annals of Scientific Society for Assembly, Handling and Industrial Robotics (pp. 11–20). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-662-61755-7_2
Metzner, M., Utsch, D., Walter, M., Hofstetter, C., Ramer, C., Blank, A., & Franke, J. (2020). A system for human-in-the-loop simulation of industrial collaborative robot applications. In IEEE International Conference on Automation Science and Engineering (Vol. 2020-Augus, pp. 1520–1525). IEEE Computer Society.
https://doi.org/10.1109/CASE48305.2020.9216766
Mizuchi, Y., & Inamura, T. (2018). Evaluation of Human Behavior Difference with Restricted Field of View in Real and VR Environments. In RO-MAN 2018 - 27th IEEE International Symposium on Robot and Human Interactive Communication (pp. 196–201). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/ROMAN.2018.8525840 Moon, S., Becerik-Gerber, B., & Soibelman, L. (2019). Virtual Learning for Workers in Robot Deployed Construction Sites. In
Advances in Informatics and Computing in Civil and Construction Engineering (pp. 889–895). Springer International Publishing. https://doi.org/10.1007/978-3-030-00220-6_107
Ojstersek, R., Palcic, I., & Buchmeister, B. (2019). Real-Time manufacturing optimization with a simulation model and virtual reality. In Procedia Manufacturing (Vol. 38, pp. 1103–1110). Elsevier B.V.
https://doi.org/10.1016/j.promfg.2020.01.198
Oliveira Valério, K. G., Silva, C. E. S., & Neves, S. M. (2020). Strategic systematic for software development in industry 4.0.
Strategic Change, 29(5), 517–529. https://doi.org/10.1002/jsc.2361
Ottogalli, K., Rosquete, D., Rojo, J., Amundarain, A., Rodríguez, J. M., & Borro, D. (2018). Framework for the simulation of an aircraft final assembly line. In MATEC Web of Conferences (Vol. 233, p. 00010). EDP Sciences.
https://doi.org/10.1051/matecconf/201823300010
Peruzzini, M., Grandi, F., Cavallaro, S., & Pellicciari, M. (2020). Using virtual manufacturing to design human-centric factories:
an industrial case. The International Journal of Advanced Manufacturing Technology, 1–15.
https://doi.org/10.1007/s00170-020-06229-2
Randon, G., & Cecconello, I. (2019). Simulação como Tecnologia Habilitadora da Indústria 4.0: Uma Revisão da Literatura.
Scientia cum Industria, 7(2), 117–125. https://doi.org/10.18226/23185279.v7iss2p117
Rip, A., & Courtial, J. P. (1984). Co-word maps of biotechnology: An example of cognitive scientometrics. Scientometrics, 6(6), 381–400. https://doi.org/10.1007/BF02025827
Shu, B., & Sziebig, G. (2019). Collaboration with high-payload industrial robots: Simulation for safety. In Lecture Notes in Electrical Engineering (Vol. 484, pp. 34–38). Springer Verlag. https://doi.org/10.1007/978-981-13-2375-1_5 Shu, B., Sziebig, G., & Pieskä, S. (2018). Human-robot collaboration: Task sharing through Virtual Reality. In Proceedings:
IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society (pp. 6040–6044). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/IECON.2018.8591102
Tolman, J. (2018). Evaluation of a Multi-User Virtual Reality System for Collaborative Layout Planning Processes. Retrieved from http://urn.kb.se/resolve?urn=urn%3Anbn%3Ase%3Akth%3Adiva-238186
Yan, J., Yan, S., Zhao, L., Wang, Z., & Liang, Y. (2019). Research on Human-Machine Task Collaboration Based on Action Recognition. In Proceedings - 2019 IEEE International Conference on Smart Manufacturing, Industrial and Logistics Engineering, SMILE 2019 (pp. 117–121). Institute of Electrical and Electronics Engineers Inc.
https://doi.org/10.1109/SMILE45626.2019.8965279
