Visual-Intelligence-poster-photo

Make the Most of Visual Intelligence on the iPhone

ByMichael Tilleman

The “visual intelligence” aspect of Apple Intelligence leverages the artificial intelligence capabilities of your iPhone to make what you see through the iPhone’s camera or on its screen interactive and actionable in ways that weren’t previously possible. It’s one of the most useful aspects of Apple Intelligence.

Triggering Visual Intelligence

We offer numerous examples of visual intelligence’s superpowers later in this article, but first, let’s make sure you know how to activate its two modes: camera mode and screenshot mode. Use camera mode to learn more about the world around you; use screenshot mode for help with something on your iPhone’s screen. Here’s how to trigger each mode:

  • Camera mode: Press and hold the Camera Control button on all iPhone 16 models (except the iPhone 16e), all iPhone 17 models, and the iPhone Air. Press the Camera Control again or tap the shutter button to lock the image for visual intelligence. On the iPhone 15 Pro, iPhone 15 Pro Max, and iPhone 16e, which support Apple Intelligence but lack the Camera Control, use the Action button, a Lock Screen button, or a Control Center button.
  • Screenshot mode: Simultaneously press the side button and volume up button to display an interactive preview of what was on the screen.

Visual intelligence analyzes the content of your image and provides relevant action buttons based on what it detects. While the Ask and Search options are always available, other buttons appear contextually depending on the content.

  • Ask: Tapping the Ask button lets you pose a question about the image to ChatGPT. (But we’ve found that Apple won’t pass on health-related questions or queries with certain types of sensitive data.)
  • Search: Tapping the search button conducts a Google Image search for similar items. It may also display tabs for search results from other apps, such as Etsy or eBay.
  • Recognized objects: When visual intelligence identifies an object, such as a specific plant or animal, it displays a button that brings up more details.
  • Text: When it detects blocks of text, visual intelligence provides buttons to summarize the text, read it aloud, or translate it.
  • Dates: If it detects a date in text, visual intelligence displays an Add to Calendar button.
  • Contact info: When details like email addresses or phone numbers appear in the image, visual intelligence can help you call or message the number, or send email.
  • Addresses: When it identifies a physical address in text, visual intelligence displays a button that opens the address in Maps.
  • URLs: This works in the virtual world as well—a URL embedded in an image prompts visual intelligence to display a button that opens the website in Safari.
  • Businesses and locations: When you capture an image of a business or other location that’s known in Maps, visual intelligence can show hours, menus, reviews, and more.

Real-World Uses for Visual Intelligence

It can be challenging to think of uses for visual intelligence at first, simply because it’s a new way of engaging with the world around you and what you see on your iPhone. We’re used to taking pictures of event flyers we want to attend, doing Google searches for things we see, asking questions of chatbots, and using specialized apps to identify plants and animals—visual intelligence can do all that and more. Here are a few practical ways to use visual intelligence today:

  • Create calendar events: Create calendar events from posters, flyers, invitations, or Web pages. When you point the camera at a poster or take a screenshot of an event page, an Add to Calendar button allows you to create an event directly from the on-screen details.
  • Find business information: Point the camera at a business to retrieve details such as hours, menu/services, phone number, and website. It’s the same information you’ll find in Maps, but it’s easier to pull up using visual intelligence.
  • Search for products: Shopping for something? Once you find an example of what you like—such as this mid-century modern sofa—take a screenshot, circle the picture with your finger, and browse the search results for similar items.
  • Summarize and read text aloud: When you’re faced with a large amount of text, especially if the font size is difficult to read, visual intelligence can provide a summary or even read it aloud. The option to have text read aloud can be particularly helpful for those with low vision.
  • Translate text: iOS offers multiple ways to translate text in unfamiliar languages, including the Translate app, but visual intelligence is often the fastest way to get a quick translation of a sign or placard.
  • Quick object identification: We’ve all wondered what some plant or animal is—using visual intelligence, you can point your camera at it to find out quickly. Just tap the name that appears to get more details.
  • Research questions: Sometimes, you know what you’re looking at but have questions about it. Instead of starting a new search, you can use visual intelligence, tap the Ask button, and pose your question to ChatGPT. Tap the response to ask a follow-up question. If you connect Apple Intelligence to your ChatGPT account in Settings > Apple Intelligence & Siri > ChatGPT, your conversations will be saved in ChatGPT, where you can review and continue the discussion.
  • Manipulate real-world data: Anything that can be photographed or captured in a screenshot can be used as data for other manipulations. For example, you could take a picture of a bookshelf and ask for a list of all the titles, or take a screenshot of a recipe and request the calorie count per serving.

How does visual intelligence compare to apps like ChatGPT, Claude, Gemini, and others? It outdoes them in two ways but falls short in one. Thanks to its deep integration with iOS and the iPhone’s Camera Control, it’s easier to activate visual intelligence than any other app. It also transfers data more effectively to other apps, such as sending URLs to Safari, phone numbers to Phone or Messages, addresses to Maps, and more. However, chatbot apps—which can also analyze photos and screenshots—are more conversational, offer more detailed information, and are willing to discuss potentially sensitive topics that Apple won’t touch, such as health and politics. We use visual intelligence for straightforward tasks, but for more complex situations, we often turn to a chatbot app instead.

(Featured image by Apple)

Social Media: With its visual intelligence feature, Apple Intelligence brings superpowers to your iPhone’s camera and screen. Just point and hold to identify objects, translate text, create calendar events, and get answers about anything you see.

Similar Posts

  • BlueAnt Mini Wireless

    No, not some new invasive species, but rather some really cool wireless headphones! BlueAnt audio is one of our newest headphone lines that we have begun to carry here at Small Dog. The “Pump Mini”:http://www.smalldog.com/product/87884/blueant-pump-mini-wireless-hd-audio-sport-buds-purple Wireless HD Audio Sports Buds have become my latest must have! These are lightweight, hold a great charge and come in several colors.

    When I first saw these headphones I was a little skeptical. I actually hadn’t ventured into wireless earbuds before trying these headphones and had heard customer complaints in the stores about different varieties over the last year or so. The biggest issue that I had heard was comfort and how well they stayed in your ears, so I had no idea what I about to experience! I charged up the headphones and tested them out on some morning runs before work and was very pleased with them. I easily connected the headphones to my iPhone and enjoyed great sound out of them. They were comfortable and stayed in my ears easily. I was even able to take phone calls with them and those I was talking to could easily hear me. I also connected them to my Apple Watch and as you might guess, preferred this over carrying my iPhone with me.

    I’ve begun using these earbuds in the office as well with my MacBook Air. I unfortunately have to wear glasses, so I find that many over the ear headphones (and I’ve used countless ones!) inevitably give me a headache from the pressure the headphones put on my glasses behind my ears. Contacts aren’t an option for me so I have found these headphones have begun to serve a truly universal role!

  • Emily's Favorite Mac Accessories

    I am the kind of gal that admittedly likes gadgets and accessories just because they are cool, I don’t even need to have a use for them to make me want one! If there is a new iPad case, a new speaker, a new product announcement from Apple, I want it. This has resulted in a vast collection over the years of all kinds of gadgets, gizmos and some stuff that I just hang onto for no good reason other than I might need it one day. Some things have been a bit of a fad, others have stood the test of time. Over the years I have found that I few items in my collection of gadgets that have either become invaluable or the kind of item that really stands the test of time.

    p{text-align: center;}. !http://blog.smalldog.com/images/4651.jpg!

    My first must have and favorite gadgets are portable bluetooth speakers, and I have had several over the years. The battery powered speakers from Skullcandy I had were one of my most favorite pocket sized speakers! It took up little room in my bag and I could have my music wherever I wanted. You never realize how much you miss having music wherever you go until you start to use these kinds of speakers and then suddenly don’t have them! I eventually outgrew my little Skullcandy speakers and upgraded to a larger but still portable Altec system. This speaker system went everywhere with us. In fact, it still travels with us in our camper even though it features a 30 pin connection for our phones. What I loved most about this set up was that it had a rechargeable battery, so unlike my little Skullcandy speakers I didn’t need to worry about carrying around spare AA batteries. It also allows me to use a 3.5mm headphone cable and an FM radio. I recently started to use the “**Outdoortech Turtle Shell**”:http://www.smalldog.com/product/85338/outdoor-tech-turtle-shell-2-0-rugged-wireless-boombox-electric-blue speakers to replace the old Skullcandy speakers for immediate portable sound, and as a huge bonus this speaker is bluetooth, something none of my previous portable units had. I will admit, I still haven’t found a better replacement for my aging Altec portable speaker as far as sound quality goes, but I will also admit finding a newer solution would require my having to let go!

    My second must-have gadget is an airport express base station. You might be surprised that it is not the Airport Extreme. Why do I love the “**airport express**”:http://www.smalldog.com/product/80932/apple-airport-express-base-station-802-11n-wi-fi so much? Well part of it might be a bit nostaglic. It was my first introduction into wireless speakers. How cool was it to plug in my Harmon Kardon speakers, simply tap my iPhone while sitting on the couch, and my speakers came to life like magic? But the great features don’t stop there. It’s a really powerful and inexpensive wireless router. For many, we don’t have huge areas we need to cover with wireless. Maybe it’s just a small apartment, or perhaps your house just isn’t that big. When I first started to use the airport express the one draw back was that the only ethernet port had to be designed (in most cases) for your internet modem, but in recent years Apple has added a second ethernet port allowing for more flexibility and versatility with this little gem.

    p{text-align: center;}. !http://blog.smalldog.com/images/4650.jpg!

    iPad and iPhone cases are another huge weakness of mine. I probably have no less then five cases for my phone and for my iPad. You could say I collected cases like some girls collect shoes! But on a serious note, this is a very important accessory for your iPhone or your iPad. Some folks prefer form fitting and minimal, others prefer a more rugged case. I have come to prefer several cases. However, I think cases that combine both slim form factor and serious protection are the “**Lifeproof**”:http://www.smalldog.com/wag900002119 cases. I have one for my phone and one for my iPad. What I love most of them is that they come off easily! I don’t always want a heavy duty case on my devices, but have found that many cases that offer greater levels of protection end up being very difficult to take off your device.