The incredible popularity of Pokemon Go has exposed millions to the promise of augmented reality.
Now the floodgates may open for apps that merge the real world with our phone screens. Facebook announced Thursday it is open sourcing its cutting-edge computer vision work.
“This will be huge,” said Maryland professor Amitabh Varshney, whose research includes computer vision. “It will greatly enable all these other applications.”
Computer vision allows machines to perceive and understand their surroundings. Apps with good computer vision can have more powerful augmented reality — where new information is overlaid on top of what our eyes see naturally. Facebook expects its vision recognition technology will be useful for many areas including commerce and health apps.
Now app creators who want to harness the potential of augmented reality can rely on Facebook’s open-sourced tools rather than having to create their own.
In a call with reporters, two of Facebook’s AI researchers talked about some possibilities. Imagine taking a picture of your breakfast, and being told how many calories were in your bagel. Or pointing your phone at a piece of furniture you love, and being told where you can buy it.
Athletes could potentially benefit as well. With better computer vision an app could diagnose your golf swing, yoga pose or basketball shot to provide feedback on what you could do better. Want to shoot like Steph Curry? Then move your hands to these positions.
Perhaps you need to do some work on your car, but are overwhelmed by what you see under the hood? An augmented reality app could tell you exactly where the dipstick is, or where you can find the container to refill your windshield washer fluid.
According to Varshney, better computer vision will hasten the arrival of robot assistants in our homes. A person could one day instruct a robot to get their medicine from the counter, and the machine would be able to pick up the right bottle.
Facebook’s latest work is a notable step in this direction as it can recognize the exact position of a bottle, rather than just realizing that a bottle is present.
Facebook said it’s sharing its findings with competitors to push the AI field forward.
“We care more about using AI than owning AI,” said Larry Zitnick, a research manager at Facebook’s AI lab in Menlo Park, California.
Advances in computer vision could help Facebook users search for images even if a photo isn’t clearly tagged. A person with vision loss could have their computer automatically tell them what’s happening in a photo, even if no caption was written.
Facebook is also interested in applying these techniques to video. If videos are classified in real-time, relevant Live videos on Facebook could be more easily shared.