Gemini Live is gaining a new accessibility tool called Guided Vision, which uses artificial intelligence to provide real-time audio descriptions of anything a user points their phone’s camera at. The feature is rolling out to compatible Android devices, allowing Google’s AI to assist with everyday visual tasks.
By sharing the camera within Gemini Live, users can call on the AI to read small text, describe their surroundings, locate or identify nearby objects, and explain the details of specific items. The functionality is designed primarily for people who are blind, have low vision, or want assistance in particular situations.
How Guided Vision works
Beyond the Gemini app, Guided Vision is also available through Google TalkBack. Users can configure an accessibility shortcut for it via the Settings app on devices running Android 9 and above. The feature was announced as part of a recent set of Pixel updates.
Guided Vision supports follow-up questions about whatever the camera is capturing. For example, users can ask Gemini to read out the expiration date on an item it has just helped them find. Google says the AI provides audio cues to help line up the camera when someone asks about an object that is not currently in the frame.
Limitations and intended use
Google advises users not to rely on Guided Vision for tasks such as navigation, safe-travel guidance, or obstacle detection. The company also states that the feature should not be used as a replacement for a cane or other mobility aid.
The approach is comparable to the VoiceOver Live Recognition feature that Apple has introduced on the iPhone and Vision Pro, which similarly uses on-device intelligence to describe visual surroundings for users who need assistance.
Guided Vision is available today on compatible Android devices, with access through the Gemini app, Google TalkBack, and an accessibility shortcut in Settings on devices running Android 9 and above.
Source
Image: theverge.com
