Augmented Reality (AR) Experiences in Chemistry
The need for AR in the classroom
Chemistry is a difficult subject to master in part because the chemistry of the macroscopic phenomena we observe cannot be fully understood without looking at the molecular level and beyond. Ultimately, chemistry consists of three intimately related types of knowledge, macroscopic, sub-microscopic, and symbolic. To truly understand the chemistry of the world around us, we must seamlessly integrate all these knowledge types. Traditionally, chemists have used physical model kits to aid in integrating theses levels of knowledge, and to bring the abstract ideas of the sub-microscopic level of chemistry to life. The evolution of technology has lead to chemists turning to increasingly accurate and easy to use computerized models. Incorporating these computerized models, along with AR technology into smartphones provides students with a new and exciting way to interact with chemistry.
Current Work: Pairing AR and Machine Learning (ML) in an AR Molecule Viewing Mobile App
Unity 3D and ARCore were used to create an application that can provide an AR experience that will help students to verify and distinguish between different molecules. This is done through the use of Google AutoML which is a machine learning tool that allows the construction of an image recognition model that can differentiate between different chemistry molecules. Molecules such as water, benzene, cyclohexane, and aspirin are currently being integrated into the AutoML database. Over 1500 images for each molecule have been represented and organized into a repository on the Google Cloud Platform (GCP). These images consist of both 2D and 3D computer generated, and hand-drawn images. The image recognition model is able to use this dataset to recognize never-before-seen images of all labels from the trained dataset with an accuracy of 95% or greater. The process used to develop the model and integrate it into a mobile app is shown in the figure below.
A more detailed explanation of why we chose machine learning and Google AutoML can be found here.
Click here to watch a DEMO of our ML model.
The application allows students to take pictures of molecular structures, or select a particular molecular from a drop down menu, and convert them into an interactive AR based, 3D structure. The functionality of the application is split between the AR experience which can be seen on the blue section, and the image recognition portion that employs our ML model, which is shown in green.
Molecules and ML: Image Classification for an AR-Enabled Chemistry Classroom
In coordination with AR, artificial intelligence (AI) and ML has the potential of creating a new way of interacting with the world, particularly in the field of education. This initial proof of concept trained and utilized an AI model that will recognize molecular structures. Hand-drawn or computer-generated images are uploaded and ran through the model. They are then labeled and used train the model. Finally, the model is exposed to never before seen molecular representations and asked to classify them. With enough images of a particular molecule incorporated into the model, it can successfully differentiate between different molecules it has been trained on. A robust model trained on a larger number of molecules may be a viable image recognition solution for an AR based molecule viewing mobile app.
AR Mobile App Development
The AR and Image Recognition Project was conceived to help provide a 3D chemical representation of a 2D chemical structure. By aiming the camera of your device at the 2D chemical structure a 3D representation of that structure will appear on your device's display. The app is more portable than the commonly used chemistry model kits, and produces a 3D representation instantly rather than having to build the structure piece by piece. Our initial goal was the get the technology in the hands of the students as quickly as possible. We constructed the app using Aurasma (now HP Reveal) to generate our initial proof of concept AR molecule viewing app, which is depicted in the image to the right.
Click here to watch a DEMO of our Aurasma based AR molecule viewing app.
If you want to recreate the demo on your mobile device, follow the directions found here. Once the app is installed and setup, the image below with trigger the demo.
Pilot testing with students revealed an interest in using the Aurasma baed app for educational purposes, but students wanted the app to have better functionality. To accommodate these user requests, we needed to construct our own AR-Chem molecule viewing app. Version 1.0 of the app allows free movement of the 3D, AR-based molecules, and it can display multiple 3D, AR-based molecules at the same time. All of the 3D, AR-based structures are triggered from 2D representations of the molecules. This practice should allow students to build the representational competence necessary to understand concepts like chirality.
Click here to watch a DEMO of our AR-Chem App Version 1.0
Version 2.0 of the AR-Chem app incorporates a shared user experience. The shared experience allows an instructor to facilitate a group experience where all students are viewing the same AR-based molecular simulation on their own mobile device. Additionally, the instructor can give control to the students to explore individually at any point. The shared experience requires all users to be connect to the same wireless network.
MR Mol: Mixed Reality Molecule Viewing App
In addition to developing AR molecule viewing apps for mobile devices, we have developed a mixed reality (MR) app, known as MR. Mol, for Microsoft's HoloLens 1. MR Mol creates 3D chemical structures which the user may interact with through gestures and voice commands. MR Mol allows the user to freely move around the molecule in three dimensions offering the same molecule from different perspectives. Through use of MR Mol, students may development a better understanding of a molecule's molecular geometry, the effect single, double, and triple bonds have on bond distance, and how to differentiate stereo-isomers.
- Behmke, D.; Kerven, D.; Lutz, R.; Paredes, J.; Pennington, R.; Brannock, E.; Deiters, M.; Rose, J.; and Stevens, K. (2018) "Augmented Reality Chemistry: Transforming 2-D Molecular Representations into Interactive 3-D Structures," Proceedings of the Interdisciplinary STEM Teaching and Learning Conference: Vol. 2 , Article 3. DOI: 10.20429/stem.2018.020103
Recent Presentations and Awards
- Doghaimat, D.*; Tyner, N.*; Patel, S.*; Vazquez, L.*; Behmke, D.A.; Lutz, R.; Brannock, E. Molecules and Machine Learning Integration Utilizing AutoML. Poster Presentation, Georgia Gwinnett College CREATE Symposium, April 2019.
- Anderson, J.*; Smith, R.*; Shaik, A.*; Behmke, D.A.; Lutz, R.; Brannock, E. Shared Augmented Reality (AR) Experience of 3D Molecules for Chemistry Students. Poster Presentation, Georgia Gwinnett College CREATE Symposium, April 2019.
- Doghaimat, D.*; Tyner, N.*; Patel, S.*; Vazquez, L.*; Behmke, D.A.; Lutz, R.; Brannock, E. Molecules and Machine Learning Integration Utilizing AutoML. Poster Presentation, Georgia Gwinnett College School of Science and Technology STaRS, April 2019. (2nd Place Poster Contest)
- Anderson, J.*; Smith, R.*; Shaik, A.*; Behmke, D.A.; Lutz, R.; Brannock, E. Shared Augmented Reality (AR) Experience of 3D Molecules for Chemistry Students. Poster Presentation, Georgia Gwinnett College School of Science and Technology STaRS, April 2019. (1st Place Poster Contest)
- Doghaimat, D.*; Tyner, N.*; Patel, S.*; Vazquez, L.*; Behmke, D.A.; Lutz, R.; Brannock, E. Molecules and Machine Learning Integration Utilizing AutoML. Poster Presentation, Consortium for Computer Sciences in Colleges Central Plains Conference, April 2019. (2nd Place Research Contest)
- Anderson, J.*; Smith, R.*; Shaik, A.*; Behmke, D.A.; Lutz, R.; Brannock, E. Shared Augmented Reality (AR) Experience of 3D Molecules for Chemistry Students. Poster Presentation, Consortium for Computer Sciences in Colleges Central Plains Conference, April 2019. (2nd Place Research Contest)
- Paredes, J.E.B.; Behmke, D.A.; Pennington, R.; Kerven, D.; Lutz, R.; Brannock, E. Interdisciplinary Exploration of Augmented Reality to Enhance Student Learning in Organic Synthesis. Oral Presentation, American Chemical Society National Meeting, Orlando, FL, March 2019.
- Behmke, D.A.; Kerven, D.; Lutz, R.; Brannock, E.; Paredes, J.E.B.; Doghaimat, D.*; Tyner, N.*; Maldonado, C.*; Anderson, J.*; Moses, T.* Molecules and Machine Learning: Images Classified by ML Used as Input to an Augmented Reality App for Chemistry Education. Poster Presentation, American Chemical Society National Meeting, Orlando, FL, March 2019.
Click here for all presentations
For More Information
- Dr. Derek Behmke at email@example.com
- Dr. David Kerven at firstname.lastname@example.org
- Dr. Evelyn Brannock at email@example.com
- Dr. Robert Lutz at firstname.lastname@example.org