Google’s Arts and Culture App Turns You Into a Work of Art
Editor’s Note: This article was originally published in an edition of NOVA’s email newsletter, NOVA Lens, and has now been repurposed for NOVA Next. Sign up for NOVA Lens here (select “NOVA Newsletters”).
If you’ve ever fantasized about discovering a lost-long identical twin, a newly updated app could help. Your lookalike, though, could end up being a painting from the 18th century. One of my friends got matched with Benjamin Franklin—another, with Australian painter and printmaker Margaret Preston. Google’s Arts and Culture app exploded in popularity over the past week due to this clever feature, which uses facial recognition to pair one of your “selfies” with a famous piece of artwork.
Facial recognition systems have been around for nearly a quarter of a century, though they’ve only recently grown sophisticated enough to tackle the challenge of matching living people with artistic masterworks. Anil K. Jain, a computer scientist at Michigan State University, says that these systems used to work only if the person was facing the camera directly, removed large accessories, had a relatively neutral expression.
The Google Arts and Culture app doesn’t work that way—even silly facial expressions or sideways glances at the camera will still spit out a result. “For this app, you can do anything,” Jain said. “You could even take a picture of your cat or dog.” The app then finds a numerical similarity (in the form of a percentage) between your face—or whatever picture you take—and however many millions of portraits are stored in Google’s database.
Google’s not the only company that’s been pushing the boundaries of inexpensive and accessible facial recognition technology—the future is looking increasingly biometric. Apple’s iPhone X lets you scan your face as a method of unlocking your device. Last month, Facebook introduced a facial recognition tool that sends you a message if you appear in someone else’s photo, even if the person who uploaded it didn’t tag you.
The algorithm in the Arts and Culture app is like most face recognition algorithms in that it assesses a visage using what’s called a deep neural network (learn more about those in this NOVA Next feature about algorithm bias). Earlier systems relied on landmarks on the face, such as the distance between someone’s eyes or the width of his or her mouth. The Google app is, by contrast, a black box. “It’s very difficult to interpret what this black box is doing,” Jain said. He pointed out that in the pair of matches below, the percentages Google reported don’t seem to make much sense. Clyfford Still’s match seems much more appropriate than BlakeJ98’s:
versus…
What’s going on here?
“I cannot explain to you why that happened,” Jain said. “Sometimes machine learning does unexpected things that are beyond common sense.” What we can infer, though, is that measuring similarity across individuals doesn’t work; the percentage is not a universal tool in that it only makes sense within the context of that particular photo comparison. In other words, comparing Clyfford Still’s 49% to BlakeJ98’s 59% is like comparing apples and oranges.
Most of the time, Google is going to suggest scores that are generally around 50 to 65%. “That means the system is not so sure about the similarity,” Jain said. The percentages also depend on how far you’re holding the camera away from your face. On top of that, the algorithm is also sensitive to illumination (what the lighting is like in the room) and the person’s facial expression. “These are perennial problems in face recognition,” Jain said—so common, that researchers have an acronym for it (pose, illumination, and expression, or PIE).
And because it’s a trained system, Google’s facial recognition tool might be biased toward Caucasian people; there simply might not be enough people of color in the historical portraits that Google has scanned to accurately capture a non-white person’s essence. That’s likely why one of my colleagues, who is of Mexican heritage, was dissatisfied with her results—and why one of Jain’s Chinese students was matched with someone of Western descent.
Still, “all of the underlying methodology for these systems is the same,” Jain said. “That same deep network can perform differently if you tune it differently. But otherwise, structurally, they are taking a different approach.”
I’ll leave you with this—my match! (Yep, that’s me!) Find out more about the limits of facial recognition here.
***
Want to influence the future of NOVA? We want to hear from you about how you interact and engage with us online. Take a few minutes to fill out this brief survey.
***
Flu season is upon us, and this year it’s looking especially bad. The CDC released a statement earlier this week saying that this strain is “moderately severe.” It’s already killed hundreds of Americans. Some areas, like San Diego, have reported record numbers of deaths.
The flu vaccine changes from year to year, and its efficacy varies. Here’s why.
WHAT’S ON YOUR MIND
We were so thrilled to see all of the positive comments in response to “Black Hole Apocalypse,” which premiered last week and was hosted by astrophysicist and author Janna Levin. Here’s one of our favorite comments, in part because it comes from a man who was excited to see so many women represented in the film. Thanks, Peter!
We hosted a Reddit AMA with Janna last Friday. If you missed it, check it out here.
WHAT’S ON OUR MIND
As David Foster Wallace put it, the Swiss are “considering the lobster.” We enjoyed this in-depth piece in The New York Times about the debate surrounding a new law passed in Switzerland banning the practice of boiling lobsters alive. “This topic comes up periodically,” said Bob Bayer, executive director of the Lobster Institute at the University of Maine. “It’s not a question that has an absolute answer.” It reminds us of this Radiolab episode about the “maddeningly subjective” nature of pain. It seems that pain is both a sensation and a psychological phenomenon, and that creatures with a primitive nervous system probably aren’t able to process pain like we do.
But not all scientists are sure, and one expert’s dissenting opinion is what drove the Swiss government to introduce the new law. We asked lobster biologist and marine scientist Rick Wahle of the University of Maine to lend us his thoughts on the topic. Here’s what he had to say:
It’s not surprising that lobsters are like other animals in being equipped to avoid “aversive stimuli” that might be associated with death or injury. How they perceive that stimulus is hard to say, but we should avoid anthropomorphizing.
But I’m not convinced the Swiss solution by stunning them with an electric shock is any better than dropping them head first in boiling water. Animal welfare policies in some institutions, such as the New England Aquarium, recommend injection with a solution of potassium chloride for research applications (though not for human consumption). And still other experts suggest chilling them on ice to slow their metabolism (and nervous system) may be more humane; it’s certainly the most practical and economical for the home kitchen.
One thing’s for sure. The statement in The New York Times article that “lobsters remain the only animals we still kill in our own kitchens” is plainly false. What about clams, mussels, and oysters? Maybe the Swiss would like to take that one on next.
—Rick Wahle, University of Maine
If you enjoyed this story, you won’t want to miss the NOVA episode “What Are Animals Saying?” airing this spring as part of our NOVA Wonders series. Stay tuned!
Have a question for NOVA? Want to see us cover something in the news? Let us know—tweet at @novapbs, use the hashtag #NOVALens, send us a direct message, or email us at nova_lens@wgbh.org. We might give you a shout-out in next week’s newsletter.
***
POSTSCRIPT
Facebook is shrinking our News Feed reach. Here’s how that affects us.
And here’s where YOU come in. Like nearly every other company, we’re proud to have built a strong community on Facebook that engages with and learns from our content there, including educational outreach programs, short form videos, photos, Facebook Live events, and more. That said, this change has reemphasized the importance for us of not relying on any one company to remain deeply connected to our audience. This recent letter from Senior Executive Producer Paula Apsell speaks to our interest in expanding—and listening to—our community. Readers of NOVA Next and NOVA Lens are a huge part of that community. Keep reading, keep writing to us, and keep thinking!
By the way, if you got the link to this newsletter elsewhere, subscribe here (select “NOVA Newsletters”) and check out last week’s edition.
See you next week,
Allison and the NOVA team
Comments
Post a Comment