From GGCWiki
Jump to: navigation, search


Augmented Reality Experiences in Chemistry

The need for AR in the classroom

Students have displayed varying areas of difficulty when learning the fundamentals of chemistry. One key problem is learning and understanding molecular structures. This could be attributed to a disconnect between written formulas and 2D examples, and being able to visualize a molecule's spatial structures from that information. Understanding and visualizing spatial structures is necessary for students to understand chemical reactions, molecule interactions, and even VSEPR theory. By use of Augmented Reality (AR), students bridge the gap between 2D or written representations, and 3D visualization. AR also engages the student in active learning to reinforce the concepts they are learning.

Current Work

Unity 3D and ARCore were used to create an application that can provide an AR experience that will help students to verify and distinguish between different molecules. This is done through the use of Google AutoML which is a machine learning tool that allows the construction of an image recognition model that can differentiate between different chemistry molecules. Molecules such as water, benzene, cyclohexane, and aspirin are currently being integrated into the AutoML database. Over 1500 images for each molecule have been represented and organized into a repository on the Goole Cloud Platform (GCP). These images consist of both 2D and 3D computer generated, and hand-drawn images. The image recognition model is able to use this dataset to recognize never-before-seen images of all labels from the trained dataset with an accuracy of 95% or greater. The process used to develop the model and integrate it into a mobile app is shown in the figure below.


A more detailed explanation of why we chose machine learning and Google AutoML can be found here.

The application allows students to take pictures of molecular structures and convert them into a 3D structure. The functionality of the application is split between the AR experience which can be seen on the blue section, and the image recognition which is shown in green.


A more detailed Description of the Application Development

A more detailed Description of the AI and ML

Prior Projects

ChemReality Demo

Molecules and Machine Learning: Image Classification for an AR-Enabled Chemistry Classroom

In coordination with augmented reality (AR), artificial intelligence (AI) and machine learning (ML) has the potential of creating a new way of interacting with the world, particularly in the field of education. Our research team trained and utilized an AI model that will help chemistry students understand molecular structures. Hand-drawn or computer-generated images are uploaded and ran through the model. They are then labeled and used to generate an interactive 3D model that students can use to study.

 ARChem Demo Video

 CCSC Abstract Submission
 Image References

ChemReality Demo

Visualizing Molecules Using Augmented Reality and Image Recognition

The AR and Image Recognition Project was conceived to help provide a 3D chemical representation of a 2D chemical structure. By aiming the camera of your device at the 2D chemical structure a 3D representation of that structure will appear on your device's display. The app is more portable than the commonly used chemistry model kits, and produces a 3D representation instantly rather than having to build the structure piece by piece. The app allows free movement of the 3D object and can load multiple 3D objects of various 2D structures at the same time. Being able to load multiple structures will help students better understand structures like stereo-isomers.


 ARChem CCSC Poster 2018
 ARChem CCSC Abstract Paper 2018

MR Mol: Mixed Reality Molecule

MR Mol is a Mixed Reality application that creates 3D chemical structures in which the user may interact with through gestures and voice commands. MR Mol allows the user to freely move around the molecule in three dimensions offering the same molecule from different perspectives. Through use of MR Mol, students may development a better understanding of a molecule's molecular geometry, the effect single, double, and triple bonds have on bond distance, and how to differentiate stereo-isomers.

MR Mol: Mixed Reality Molecule

 MR Mol Project, Abridged (2 mins)
 Very Abridged Version (43 seconds)
 CCSC Conference Submission

Instructions for Demo Configuration

Instructions for Demo Configuration and application use are provided here.

Isomer Triggers

Awards and Recognition


Recent Presentations

Click here for all presentations

Useful Links and References

For More Information


  • Dr. Derek Behmke at
  • Dr. David Kerven at
  • Dr. Evelyn Brannock at
  • Dr. Robert Lutz at
Personal tools