Teaching Computers How to See the World

Let’s step into the future for a moment. Imagine you’re off to an urgent meeting, but stop when you realise you’ve lost your keys. You call out to your virtual personal assistant for help, and it says, “Don’t worry, you dropped them behind the sofa last night.” What a relief! You grab your keys, and you’re on your way without delay.

Bernard Ghanem, Professor of Electrical and Computer Engineering and Deputy Director of the AI Initiative at King Abdullah University of Science and Technology (KAUST) in Saudi Arabia, believes this type of scenario will become a reality in the future. He’s an expert in computer vision, the study of how computers understand the visual world. 

Recently, his lab was part of a long-term project called Ego4D involving 13 universities and labs from nine countries, as well as, Facebook Research. It asked 700 people from all over the world to use wearable cameras like GoPros and augmented reality (AR) glasses to record their daily lives. This gave the researchers more than 3,000 hours of egocentric perception video data, meaning that it has a first-person viewpoint. 

This type of video is more useful for training machines to interact with humans in a natural way than the third-person viewpoint you get from surveillance cameras. “We're building this dataset so that we can train intelligent systems to do interesting tasks with this kind of video, because we expect the metaverse and augmented reality to become a realistic part of our future,” he explains. “If everybody had a wearable device, your favorite smartphone assistant could answer much more personalized questions than they can now.”

Deep learning is another major theme of Bernard’s research. This is where machines learn from huge amounts of data. Deep learning technology is creeping into our lives, and it’s the driving force behind the artificial intelligence (AI) revolution our world is experiencing now. It’s used by virtual personal assistants like Apple’s Siri, at airports to screen faces, or in self-driving cars to help with navigation. The world-class computer facilities at KAUST enable Bernard to carry out his cutting edge deep learning research. “KAUST knows that AI is a big thing now, and so they’ve made a proportional investment in it because we know it’s going to be essential in the future,” he says. The university already has several hundred high-end GPUs and the newest version of KAUST’s supercomputer called Shaheen III is set to be operational by the end of 2023, which will enable even more advanced deep learning projects to take place.

Over the years, researchers have discovered that deep learning is not a very robust process as it’s prone to making errors that humans don’t. “Deep learning is very sensitive to noise. So let's say you train your deep learning system to give you 95% accuracy on classifying images. However, if I take a cat image that your network just labelled as a cat and add a very small amount of noise, to you and me, there's no difference,” he says. “But if I give that noisy image to the deep network, it may say that it’s an airliner, or it’s a dolphin, or something not even close.” This means that deep networks are very susceptible to hacking and manipulation. Bernard’s research is looking at where deep learning fails, why it fails, and how you could prevent that failure from happening. The goal is for deep learning to be certifiably safe so that it could be used in, for example, self-flying planes in the future.

Bernard has also been involved in a lot of interdisciplinary projects at KAUST where he’s used computer vision techniques to address research challenges in areas like plant science, clean combustion engineering, and archaeology. KAUST has many top-tier researchers working there dedicated to finding solutions for the world’s most pressing scientific and technological issues. The compact campus means that you can interact easily with colleagues from different departments, something that might be more difficult in a larger university. This has resulted in several fruitful collaborations for Bernard and publications in prestigious journals. 

He comments that KAUST is a unique institution with a rich multi-ethnic and multicultural environment where everyone is encouraged to carry out blue-sky research. The fact that KAUST is a young university, and not bound by 100-year-old bureaucracy means that change is possible and initiatives can be started very quickly. He says, “All of the resources you need are made available to you at KAUST so you can really focus on your own research. In my mind, it is a utopia for researchers.”