Augmented Reality Experiences in Chemistry
The need for AR in the classroom
Students have displayed varying areas of difficulty when learning the fundamentals of chemistry. One key problem is learning and understanding molecular structures. This could be attributed to a disconnect between written formulas and 2D examples, and being able to visualize a molecule's spatial structures from that information. Understanding and visualizing spatial structures is necessary for students to understand chemical reactions, molecule interactions, and even VSEPR theory. By use of Augmented Reality (AR), students bridge the gap between 2D or written representations, and 3D visualization. AR also engages the student in active learning to reinforce the concepts they are learning.
Unity 3D and ARCore were used to create an application that can provide an AR experience that will help students to verify and distinguish between different molecules. This is done through the use of Google AutoML which is a machine learning tool that allows the construction of an image recognition model that can differentiate between different chemistry molecules. Molecules such as water, benzene, cyclohexane, and aspirin are currently being integrated into the AutoML database. Over 1500 images for each molecule have been represented and organized into a repository on the Goole Cloud Platform (GCP). These images consist of both 2D and 3D computer generated, and hand-drawn images. The image recognition model is able to use this dataset to recognize never-before-seen images of all labels from the trained dataset with an accuracy of 95% or greater. The process used to develop the model and integrate it into a mobile app is shown in the figure below.
A more detailed explanation of why we chose machine learning and Google AutoML can be found here.
The application allows students to take pictures of molecular structures and convert them into a 3D structure. The functionality of the application is split between the AR experience which can be seen on the blue section, and the image recognition which is shown in green.
A more detailed Description of the Application Development
A more detailed Description of the AI and ML
Molecules and Machine Learning: Image Classification for an AR-Enabled Chemistry Classroom
In coordination with augmented reality (AR), artificial intelligence (AI) and machine learning (ML) has the potential of creating a new way of interacting with the world, particularly in the field of education. Our research team trained and utilized an AI model that will help chemistry students understand molecular structures. Hand-drawn or computer-generated images are uploaded and ran through the model. They are then labeled and used to generate an interactive 3D model that students can use to study.
Visualizing Molecules Using Augmented Reality and Image Recognition
The AR and Image Recognition Project was conceived to help provide a 3D chemical representation of a 2D chemical structure. By aiming the camera of your device at the 2D chemical structure a 3D representation of that structure will appear on your device's display. The app is more portable than the commonly used chemistry model kits, and produces a 3D representation instantly rather than having to build the structure piece by piece. The app allows free movement of the 3D object and can load multiple 3D objects of various 2D structures at the same time. Being able to load multiple structures will help students better understand structures like stereo-isomers.
MR Mol: Mixed Reality Molecule
MR Mol is a Mixed Reality application that creates 3D chemical structures in which the user may interact with through gestures and voice commands. MR Mol allows the user to freely move around the molecule in three dimensions offering the same molecule from different perspectives. Through use of MR Mol, students may development a better understanding of a molecule's molecular geometry, the effect single, double, and triple bonds have on bond distance, and how to differentiate stereo-isomers.
- PowerPoint Presentation
- MR Mol Project (5 mins)
- Mr. Mol Demo
- MR Mol Demo (41 secs)
MR Mol: Mixed Reality Molecule
MR Mol Project, Abridged (2 mins) Very Abridged Version (43 seconds) CCSC Conference Submission
Instructions for Demo Configuration
Instructions for Demo Configuration and application use are provided here.
Awards and Recognition
Click here for all presentations
Useful Links and References
For More Information
- Dr. Derek Behmke at email@example.com
- Dr. David Kerven at firstname.lastname@example.org
- Dr. Evelyn Brannock at email@example.com
- Dr. Robert Lutz at firstname.lastname@example.org