Visual-Intelligence-poster-photo

Make the Most of Visual Intelligence on the iPhone

ByMichael Tilleman

The “visual intelligence” aspect of Apple Intelligence leverages the artificial intelligence capabilities of your iPhone to make what you see through the iPhone’s camera or on its screen interactive and actionable in ways that weren’t previously possible. It’s one of the most useful aspects of Apple Intelligence.

Triggering Visual Intelligence

We offer numerous examples of visual intelligence’s superpowers later in this article, but first, let’s make sure you know how to activate its two modes: camera mode and screenshot mode. Use camera mode to learn more about the world around you; use screenshot mode for help with something on your iPhone’s screen. Here’s how to trigger each mode:

  • Camera mode: Press and hold the Camera Control button on all iPhone 16 models (except the iPhone 16e), all iPhone 17 models, and the iPhone Air. Press the Camera Control again or tap the shutter button to lock the image for visual intelligence. On the iPhone 15 Pro, iPhone 15 Pro Max, and iPhone 16e, which support Apple Intelligence but lack the Camera Control, use the Action button, a Lock Screen button, or a Control Center button.
  • Screenshot mode: Simultaneously press the side button and volume up button to display an interactive preview of what was on the screen.

Visual intelligence analyzes the content of your image and provides relevant action buttons based on what it detects. While the Ask and Search options are always available, other buttons appear contextually depending on the content.

  • Ask: Tapping the Ask button lets you pose a question about the image to ChatGPT. (But we’ve found that Apple won’t pass on health-related questions or queries with certain types of sensitive data.)
  • Search: Tapping the search button conducts a Google Image search for similar items. It may also display tabs for search results from other apps, such as Etsy or eBay.
  • Recognized objects: When visual intelligence identifies an object, such as a specific plant or animal, it displays a button that brings up more details.
  • Text: When it detects blocks of text, visual intelligence provides buttons to summarize the text, read it aloud, or translate it.
  • Dates: If it detects a date in text, visual intelligence displays an Add to Calendar button.
  • Contact info: When details like email addresses or phone numbers appear in the image, visual intelligence can help you call or message the number, or send email.
  • Addresses: When it identifies a physical address in text, visual intelligence displays a button that opens the address in Maps.
  • URLs: This works in the virtual world as well—a URL embedded in an image prompts visual intelligence to display a button that opens the website in Safari.
  • Businesses and locations: When you capture an image of a business or other location that’s known in Maps, visual intelligence can show hours, menus, reviews, and more.

Real-World Uses for Visual Intelligence

It can be challenging to think of uses for visual intelligence at first, simply because it’s a new way of engaging with the world around you and what you see on your iPhone. We’re used to taking pictures of event flyers we want to attend, doing Google searches for things we see, asking questions of chatbots, and using specialized apps to identify plants and animals—visual intelligence can do all that and more. Here are a few practical ways to use visual intelligence today:

  • Create calendar events: Create calendar events from posters, flyers, invitations, or Web pages. When you point the camera at a poster or take a screenshot of an event page, an Add to Calendar button allows you to create an event directly from the on-screen details.
  • Find business information: Point the camera at a business to retrieve details such as hours, menu/services, phone number, and website. It’s the same information you’ll find in Maps, but it’s easier to pull up using visual intelligence.
  • Search for products: Shopping for something? Once you find an example of what you like—such as this mid-century modern sofa—take a screenshot, circle the picture with your finger, and browse the search results for similar items.
  • Summarize and read text aloud: When you’re faced with a large amount of text, especially if the font size is difficult to read, visual intelligence can provide a summary or even read it aloud. The option to have text read aloud can be particularly helpful for those with low vision.
  • Translate text: iOS offers multiple ways to translate text in unfamiliar languages, including the Translate app, but visual intelligence is often the fastest way to get a quick translation of a sign or placard.
  • Quick object identification: We’ve all wondered what some plant or animal is—using visual intelligence, you can point your camera at it to find out quickly. Just tap the name that appears to get more details.
  • Research questions: Sometimes, you know what you’re looking at but have questions about it. Instead of starting a new search, you can use visual intelligence, tap the Ask button, and pose your question to ChatGPT. Tap the response to ask a follow-up question. If you connect Apple Intelligence to your ChatGPT account in Settings > Apple Intelligence & Siri > ChatGPT, your conversations will be saved in ChatGPT, where you can review and continue the discussion.
  • Manipulate real-world data: Anything that can be photographed or captured in a screenshot can be used as data for other manipulations. For example, you could take a picture of a bookshelf and ask for a list of all the titles, or take a screenshot of a recipe and request the calorie count per serving.

How does visual intelligence compare to apps like ChatGPT, Claude, Gemini, and others? It outdoes them in two ways but falls short in one. Thanks to its deep integration with iOS and the iPhone’s Camera Control, it’s easier to activate visual intelligence than any other app. It also transfers data more effectively to other apps, such as sending URLs to Safari, phone numbers to Phone or Messages, addresses to Maps, and more. However, chatbot apps—which can also analyze photos and screenshots—are more conversational, offer more detailed information, and are willing to discuss potentially sensitive topics that Apple won’t touch, such as health and politics. We use visual intelligence for straightforward tasks, but for more complex situations, we often turn to a chatbot app instead.

(Featured image by Apple)

Social Media: With its visual intelligence feature, Apple Intelligence brings superpowers to your iPhone’s camera and screen. Just point and hold to identify objects, translate text, create calendar events, and get answers about anything you see.

Similar Posts

  • New Urbanears Active Headphones

    So by now, it might be clear that I am a bit of a headphone junkie. I’ve come to the conclusion that I can collect and use headphones like I do bags and purses! In my world I can never have enough, so I jumped on the opportunity this week to test our the new “**Hellas**”:http://www.smalldog.com/wag900002177 and “**Reimers**”:http://www.smalldog.com/wag900002181 from Urbanears.

    The Hellas are an over the ear bluetooth headphone. It took a few minutes to get them paired with my computer. My only complaint about these headphones is that I found the touch-less buttons to be a bit clunky to use. The sleek design means you can’t physically see the buttons, so it take some practice and time to learn where the buttons are. However, there are a ton of other features with these headphones that make them extremely appealing. They are incredibly lightweight, comfortable to wear and they have good sound to them. I think my favorite feature about these is that the fabric on the headband and the earpieces are machine washable. You can actually wash the fabric and return your headphones to new condition. As I just started to use these the other day I have not actually tried to wash the fabric, but as more and more people are working out with over the ear headphones this is a huge selling point. Another hidden feature is that there is a built in microphone, so if you are using these with your iPhone, you can take calls without having to remove your headphones.

    Also part of the active line are the Reimers. They deliver on fit and comfort while providing some great sound! I am impressed with how far ear buds have come over the years. What’s interesting about these headphones is that while they are in the active category, they are actually a wired set of headphones. However, Urbanears added several great features to ensure these headphones can stand up to the intensity of any workout. They have two clips for securing them to your clothing and they feature reflective cabling to light up during early morning or late evening runs. They use a new earlock technology that ensures they stay in during your workout, so no more fears of them falling out during sit-ups, push-ups and the like. I admit I am just a runner, so some of the more intense workouts I’ve not tried these, but judging by how well they stayed in for runs I am confident in saying they will stay put! Last but not least, they feature standard iOS controls we have all grown to expect out of headphones.

  • _Hello Friends,_

    The big winter storm called Jonas is heading for the east coast and Vermonters are hoping that it tracks north of the current projections but it looks like it might miss northern New England and hit the coastal areas. The lack of snow in Vermont and the unusually high temperatures have hit the Vermont ski areas and local merchants pretty hard. Here it is the end of January and no significant snow in Vermont. Well for all the skiers and especially for the businesses I hope that those areas that can’t really handle snow get bypassed and Vermont gets the big dump because Vermonters are ready.

    Down here in Key West it has been sort of cool meaning I have to wear long pants and shoes, although if it was this warm in Vermont everyone would be in shorts. Down here when it gets below 65 you see people with down coats. NOAA reported this week that 2015 was the warmest year in recorded history breaking the record from 2014 by far.

    Apple honored Dr. Martin Luther King, Jr. on its home page on the national holiday on Monday with Dr. King’s quotation “Life’s most persistent and urgent questions is, What are you doing for others”. Tim Cook tweeted his respects, too. I had the honor of marching with Dr. King and share this respect, he was truly one of the great Americans of all time

    We always advertise our “Celebrate Diversity” slogan during this time of the year. We strongly feel that you gain strength through diversity whether it is race, religion, national origin or sexual orientation. I remember a long time ago we had a series of Celebrate Diversity web banners that created a lot of controversy with some sites refusing to run them. Diversity is much more mainstream now despite some candidates for President attempting to divide us.

    This week’s Kibbles & Bytes Exclusive is weather related with $20 off the “**Tempus Pro Bluetooth Weather Stations.**”:http://www.smalldog.com/wag900002127/special-save-20-on-tempus-pro-bluetooth-weather-station This was one of the most popular gifts this holiday season and is a great way to keep track of the weather in your microclimate. This complete weather station includes indoor and outdoor sensors that measure temperature, humidity, rainfall, wind and barometric pressure. Complete with mounting hardware it is simple to install and once you download the App you can monitor the weather on your iPhone or iPad. Read a Small Dogger review of the Tempus Pro “**here.**”:http://blog.smalldog.com/article/fody-weather-station/ Normally, $159.99 this week Kibbles & Bytes readers can get the Tempus Pro Weather Station for only **$139.99**