Season 5 Episode 7 Sep 2, 2021

Apple’s #SpyPhone, an Apple App Store Settlement, and the Expansion of Government Facial Recognition Software


It's all a question of trust.


In this episode, we talk about an the expansion of government facial recognition software and an Apple App store settlement. Then we speak with Yafit Lev-Aretz, assistant professor of law at Baruch College and the Director of Tech Ethics program at the Zicklin Center for Corporate Integrity, about Apple scanning iCloud images and iMessages for child sexual abuse material. Then we speak with Anunay Kulshrestha, Princeton Computer Science doctoral candidate, whose team had built a similar child sexual abuse materal scanning system, about the potential privacy and cybersecurity risks that implementing such a system creates.


Saron Yitbarek

Disco - Founder

Saron Yitbarek is the founder of Disco, host of the CodeNewbie podcast, and co-host of the base.cs podcast.

Josh Puetz

Forem - Principal Engineer

Josh Puetz is Principal Software Engineer at Forem.


Yafit Lev-Aretz

Zicklin School of Business, Baruch College, CUNY - Assistant Professor of Law

Yafit Lev-Aretz is an assistant professor of law at the Zicklin School of Business (Baruch College, City University of New York), and the ​​Director of Tech Ethics program at the Zicklin Center for Corporate Integrity. Professor Lev-Aretz is a tech policy expert with over fifteen years of experience in studying and operationalizing the relationship between law, technology, and society.

Anunay Kulshrestha

Princeton University, Center for Information Technology Policy - Doctoral Candidate in Computer Science

Anunay Kulshrestha is a doctoral candidate in Computer Science at Princeton University, affiliated with the Center for Information Technology Policy. His research interests lie in applied cryptography and computer security, and their policy implications for privacy in today’s networked world. He is interested in understanding how cryptographic techniques can yield transparency- and accountability-enhancing privacy-preserving solutions to policy problems of trust.

Show Notes

Audio file size





[00:00:10] SY: Welcome to DevNews, the news show for developers by developers, where we cover the latest in the world of tech. I’m Saron Yitbarek, Founder of Disco.


[00:00:19] JP: And I’m Josh Puetz, Principal Engineer at Forem.


[00:00:22] SY: This week, we talk about an expansion of government facial recognition software and an Apple App Store settlement.


[00:00:29] JP: Then we’ll speak with Yafit Lev-Aretz, Assistant Professor of Law at Baruch College, and the Director of the Tech Ethics Program at the Zicklin Center for Corporate Integrity, about Apple scanning iCloud images and iMessages for child sexual abuse material.


[00:00:42] YL: My point is that we always have to be worried, but at the same time, we always have to understand that there is a balance to strike here.


[00:00:51] SY: And then we speak with Anunay Kulshrestha, Princeton Computer Science Doctoral Candidate, whose team had built a similar child sexual abuse material scanning system, about the potential privacy and cybersecurity risks that implementing such a system creates.


[00:01:06] AK: It really is a trust issue. Who do you trust with this sort of detection? Who do you trust will not repurpose this sort of detection system for censorship?


[00:01:16] SY: So we’re starting this episode off with a 90-page report published by the US Government Accountability Office on August 24th about how different parts of the government currently use facial recognition systems and how they plan on expanding the use of these systems in the future. The study looked at 24 federal agencies and found that 18 of them currently use facial recognition systems and 10 of the agencies plan to increase their usage. The list includes the Department of Defense, Homeland Security, Justice, Agriculture, Commerce, Health and Human Services, Interior State, and Veterans Affairs. Now some of the uses of this technology include aiding and searching surveillance video for investigations by the Office of the Inspector General, as well as the Department of Agriculture using it to monitor live surveillance videos for individuals on watch lists. One of the biggest concerning things about this planned expansion of facial recognition systems is that there’s currently no US federal regulation on law enforcement’s use of this technology. So Josh, what do you think about all this?


[00:02:21] JP: Well, I will admit, I did not read all 90 pages of the report.


[00:02:25] SY: Yeah. It’s very long.


[00:02:28] JP: I think we were talking about this before the show and some of the departments that said they were going to be using facial recognition were a little surprising at first, like the Department of Agriculture?


[00:02:37] SY: That felt very random.


[00:02:39] JP: I took a look at it. The best I could figure is I know there’s been a lot of concern with domestic terrorism using fertilizers and manures as bomb components. So maybe there’s something with that there.


[00:02:52] SY: Yeah. I think the other one that is a little surprising to me is the Veterans Affairs one, because is that the department that takes care of people after they come back from war?


[00:03:03] JP: Yeah, the VA hospitals.


[00:03:05] SY: Yeah, exactly. And so I’m kind of like, “What do they need? What are they recognizing?” Maybe fraught, like people who are pretending to be veterans who aren’t veterans? Is that a thing? I don’t know, but that one kind of confused me too. The other ones I think are not surprising, Defense, Homeland, Justice, State, those are predictable ones, I think. Not too surprising there. I’m reading this book called System Error, which talks about kind of big tech and how it’s affecting privacy and security and the government and kind of how it’s all coming together, a lot of topics that we talk about here. And one of the things he talks about is that we are so great at innovation that we are moving faster in terms of disruption and invention than we are able to understand what we have done. You know what I mean? So we get so excited and we’re moving forward and we’re breaking boundaries and doing all these things that the law has not caught up, not even the law, but we ourselves have not given ourselves enough time to really understand, “Here are the ramifications. Here are the pros. Here are the cons. Here’s what we’re affecting. Here are the laws that we should pass.” And it doesn’t help that a lot of our legislature, frankly, aren’t as familiar with technology in general. And so that doesn’t really help. But this kind of feels very similar to that where we are making so much progress in AI and machine learning and facial recognition that it feels like we have adopted it faster than we have been able to really understand it.


[00:04:32] JP: Also, it’s not as fun, right? I liken this to when you’re writing some interesting new features, you’re playing around with maybe a new framework, you’re writing something great for your software, and like, “You should go back and document it or write task.” But that’s the boring stuff.


[00:04:48] SY: That part is no fun. Yeah.


[00:04:50] JP: No. But making the laws and thinking through the ethical considerations, that’s not fun. But just seeing if you could recognize someone’s face and tie it up to a hundred databases, that’s exciting.


[00:04:58] SY: That’s right. Exactly.


[00:04:59] JP: I’m shocked there’s no federal regulation on the US’s use of facial recognition technology. That seems like a huge problem, especially when we have laws like the Freedom of Information Act that are supposed to let you, in theory, regular citizens or investigative journalists, understand how this data is being used. And without that legislation, it seems like there’s no recourse for citizens or the media to really ask and say, “How are you using this information? How long are you storing it? How do you get on one of these lists? How do you get off of these lists?” That seems like a real problem.


[00:05:35] SY: Yeah, I totally agree. On the one hand, given how apparently widespread facial recognition is, at least more widespread than I’d expected, it’s definitely nuts that we don’t have a law in place. However, again, given people who make laws, I am not surprised that we don’t have a law in place because having a law in place requires first understanding that law and really getting into the details and the nitty-gritty and here’s how this works and here’s the repercussion of that. I don’t know how great we’ve been in general with lawmakers in technology. That combination doesn’t seem to be a really great one. So in that respect, I’m not surprised, but it is definitely time to get on that for sure. We need to be working on that as soon as possible, if not already.


[00:06:19] JP: Agree. I absolutely agree. Well, one thing we’ve covered a lot on this show is the ongoing drama that is the Apple App Store’s 30% commission on in-app purchases of digital goods and services. With many app developers, large and small, including Epic Games, claiming that Apple wields too much power and its policies are unfair and hurt their businesses. This week, US app developers earned a small win against Apple when the company settled a class action lawsuit where it agreed to pay $100 million worth of payments to app makers. Apple will also allow app developers to promote alternative payment methods that can circumvent Apple’s commission, but don’t get excited. They still cannot do it in apps. Instead, Apple has pledged that they won’t ban app makers for emailing users about these ultimate payment methods that they could use outside of the app like on a webpage, for example. I thought this was going to be a big deal when I first read the story. And then as you get into the details, it seems less and less impactful. The idea that previously Apple could ban you for emailing a user outside of the app and just letting that user know, “Hey, I have a website where you could pay me outside of Apple’s app.” That communication doesn’t happen in the app. Just mentioning it to the user was evidently enough that Apple could ban your app.


[00:07:41] SY: That feels like such a ridiculous and very petty, is the word that comes to mind, requirements of like not only can’t you do this, but you can’t even talk about the fact that you can’t do this in your own private channel that has nothing to do with us and it’s not monitored or sent or done by us in any way. It just seems so ridiculous.


[00:08:03] JP: So from what I’ve read, Apple’s argument on this was a lot of times the email addresses that app makers were getting were coming through the app. They’re being generated by a user finding an app on the app store, downloading it, signing up for an account through the app, and that’s how the app maker got the email address. So Apple’s argument is, yes, even though you’re emailing them outside of the app, if you didn’t have us, you wouldn’t have had the email ever.


[00:08:31] SY: Oh, boy!


[00:08:31] JP: So I know. I don’t know how they policed it.


[00:08:34] SY: Yeah. It’s the policing. Even if we agree on that just conceptually, how would you know what I’m doing in an email?


[00:08:43] JP: Right.


[00:08:44] SY: But also, if you don’t email, how are your users supposed to know?


[00:08:48] JP: Well, like Apple, it would just like you do not talk about it, like just put up something in your app and give Apple a 30%. Right?


[00:08:54] SY: Right. Yeah.


[00:08:55] JP: When I originally read about this settlement, I thought, “Oh, hey, here we go! They’re going to have to actually open up the app store.” No, you can just talk about maybe buying something outside of an app. It’s not a big deal, but it’s like the absolute least Apple could have done. Notable things that this settlement does not include. It does not change the commissions that Apple is charging. So it’s still 30%, 50% for developers making less than 1 million a year. It does not require Apple to allow third-party app stores or sideloading of software. It’s all business as usual there. One little wrinkle that we’ll be watching on the horizon is that here in the US, there’s been no changes. But this week, South Korea’s Parliament just approved a bill that bans Apple and Google from forcing developers to use their respective payment systems. So this is widely expected to be approved by the Korean President. And if it goes through, Apple and Google both have a choice in South Korea, whether to completely leave the market or allow third-party app stores, allow third-party payment systems. That’s kind of a huge deal.


[00:10:02] SY: I’m really excited to see what they do. Yeah.


[00:10:04] JP: It’s going to be very interesting to see, especially if this is successful in Korea, every other country is going to do it. Right? The European Union is like, “Oh, I can’t believe they did it first.” So it’s not over. It’s still going on, really interesting stuff happening, I'm really interested to hear what mobile developers think about this.


[00:10:24] SY: Speaking of Apple, the company known for its staunch stance on user privacy and security and the slogan, “What happens on your phone stays on your phone,” has been scanning iCloud emails for child sexual abuse materials since 2019 and plans to implement new features, which will allow the company to scan iCloud photos and iMessages for this material as well. But this time, it’ll be on your phone. Now obviously, finding and fighting against child sexual abuse is very important and obviously a very good thing. However, this move has privacy and security experts alarmed at the potential dangers of implementing such a powerful and invasive surveillance system built into iOS 15. Even NSA whistleblower Edward Snowden weighed in on this move in a really great piece titled, “The All-Seeing “i”: Apple Just Declared War on Your Privacy,” which we’ll include in our show notes. In this piece, Snowden writes that it doesn’t matter what problem Apple is trying to solve, and that it doesn’t outweigh the consequences of Apple implementing their surveillance system saying, “Having read thousands upon thousands of remarks on this growing scandal, it has become clear to me that many understand it doesn’t matter. But few, if any, have been willing to actually say it. Speaking, candidly, if that’s still allowed, that’s the way it always goes when someone of institutional significance launches a campaign to defend an indispensable intrusion into our private spaces.” He also goes on to say that it doesn’t matter the way in which the system searches for and scans this data saying, “Apple’s new system, regardless of how anyone tries to justify it, will permanently redefine what belongs to you and what belongs to them.” But setting aside the privacy and security issues for a moment, which we’ll get to later in this episode, coming up next, we talk to a legal and tech ethics expert about their perspective on this debate after this.




[00:12:40] SY: Joining us is Yafit Lev-Aretz, Assistant Professor of Law at Baruch College, and the Director of the Tech Ethics Program at the Zicklin Center for Corporate Integrity. Thank you so much for being here.


[00:12:52] YL: Thanks for having me.


[00:12:53] SY: So Apple announced that they will include some new features in their new iOS and iPad updates to fight child sexual abuse. Can you talk very briefly about what those new features are and how they work?


[00:13:05] YL: So they announced three features. The first one was about enabling Siri voice assistant to offer links and resources to people it believes may be in serious situations, such as a child in danger. Advocates have been asking for this type of feature for a while now and Apple just provided it, finally. The second feature is a feature to the messages app to proactively protect children from explicit content, whether it’s in a green bubble conversation or blue bubble iMessage encrypted chat. And to implement it on notifications, Apple will be using on-device machine learning classifiers designated to detect what is considered to be sexually explicit images. And the third feature is designed to detect if people have child exploitation images or videos stored on their device. It will do this by converting the images into hashes based on what they detect and then the hashes are checked against a database of child exploitation content that’s managed by the National Center for Missing and Exploited Children. And if there is a certain number of matches found, Apple is alerted, then there’s going to be a human review and then further investigation with, of course, the potential of reporting that to the authorities. And this third feature is the one that got most of the backlash and the firestorm from civil liberties organizations and privacy advocates.


[00:14:44] JP: So at face value, searching out, finding child sexual abuse material, reporting into the authorities sounds like a great thing. But the downside, you mentioned, that there was been a lot of pushback about the idea of scanning through people’s photos on their devices. For some members of our audience that aren’t super familiar with the legal ramifications, I think we’re used to authorities needing a warrant to search our personal property. And I’m wondering if you could talk about how a company like Apple is able to do this, how it’s legal, and what are the issues around consent and letting Apple search our devices.


[00:15:23] YL: So in this context, there is no legal issue around consent and there is no need of a warrant because we all clicked agree to terms of service. We all do that many, many times, sometimes a day without reading what we agreed to. And even if we full read what we agreed to, we have very little power to negotiate or change the terms, right? These are what we call boilerplate standard term contracts that are applied to all of Apple’s users in the US. So it’s not like Yafit Lev-Aretz wakes up one morning and feels like, “No, I really don’t feel like signing this contract.” At that point, I can just not use the product, but I will not be able to change it. So in legal terms, it’s perfectly kosher. There is no issue with Apple doing that because they can launder a lot through their terms of service. And I mean, we all need to remember that privacy is the right, but we also have the right to waive it. And that’s what we constantly do when we agree to terms of service. I mean, it’s a trade-off. We agree to get some products, some services sometimes for free in return for information about us and access to our devices, to information that we exchange, et cetera.


[00:16:45] SY: So putting legality aside for a moment, I want to talk about some of the potential ramifications of this new technology, because it’s not really about scanning photos for CSAM. That’s not really the pushback. There are plenty of other companies. I think most big tech companies already do that to some capacity, have been doing that for many years. I think the real concern is this idea of creating what many people are claiming to be a backdoor to our devices where today they’re scanning for CSAM photos, which sounds wonderful and helpful and really great, but one day, with enough governmental pressure, with enough law enforcement pressure, there’s a potential that they may start scanning for other things on your device that weren’t even meant for iCloud. What are your thoughts on this?


[00:17:32] YL: The slippery slope argument is always there. It’s always there and we’re always going to see it in the context of technology. And I think we’re in a very unique time in which we have these questions kind of decided or these decisions kind of decided by tech companies and not by governments, but governments can later on use those technologies to assert power in ways that sometimes we do not agree with. But at the end of the day, there is a balance to be struck between privacy security and other values such as public and child safety, specifically for that feature of scanning photos, as they are uploaded to our cloud, all cloud providers have been scanning photos and Apple, I believe, has attempted to find a less invasive way to do that. So that’s Apple’s narrative. And I agree with it and believe it. I think the problem for backdoor may exist when we move to the second feature, not the third feature, which interestingly did not get a lot of media attention. So most of the media attention centered on the third feature on scanning those photos as they are uploaded to iCloud. But I think the real issue or the argument for a backdoor can be made in the context of the second feature, not the third feature. Because on the second feature, you have those photos that are sexually explicit. And then if there is a sexually explicit image, a notification will pop up either if the image is sent or received and then the notification will tell the child that their parent is going to be notified. The child can then choose to send or not to send. And if they choose to send or accept, the parent will be notified. And I think here, we can definitely make the argument about creating a backdoor. I’m not saying it’s a perfectly valid argument, but I think it’s an argument that we can discuss. So the claim made is that since the detection of the sexually explicit image will be using an on-device machine learning to scan the contents of messages, Apple cannot call its iMessage system end-to-end encrypted. Now Apple argues that scanning before or after the message is encrypted or decrypted keeps the end-to-end promise. Right? So we have two conflicting arguments here, but opponents say it is only a semantic maneuver to cover up basically what is a shift in Apple’s stance towards privacy and encryption. Now, in this case, it is important to know that Apple will never see the images sent or received by the user, which is a huge part of encryption, but Apple’s power does exist in the power to create the classifier that scans the image that would provide the notifications to the parent. So the concern is that it would now be possible for Apple to add new training data to the classifier to send to users, devices, and send notifications to a wider audience, basically with the option to censor and chill speech. But I think here, too, just like in the context of scanning iCloud photos, we must look at the broader context, right? We must protect privacy and security, absolutely. I’m a privacy scholar and I’m a great privacy advocate, but I don’t think that this is a tectonic shift in Apple’s stance towards privacy and security. There are broader concerns here and broader questions about choices of which technology we should develop, which technology we should adopt, to what extent we should avoid developing new technologies that can be used for good, but also for evil. How can we avoid or mitigate slippery slope concerns? Can we even do that? These are very, very challenging questions and academics, practitioners, policy makers, we all grapple with them every day. Right? I think the conflict here is real and there are valid arguments to be made on both sides, but to look at this development of this technology as kind of an isolated incident that signals this shift in Apple’s position towards privacy I think is oversimplifying the situation. There are no easy or definite answers. And there are also a lot of looming concerns that even if Apple or Google or other companies would refrain from developing the technologies of this sort, then we have other countries and they are also making headways in developing such technologies. And is it better that it’s going to come from China? Maybe it’s better to have it developed by Apple so we can still keep some measures of protections around privacy and security and gradually introducing this technology to the market with safety measures and protections.


[00:22:19] JP: I’m curious if Apple has or any cloud provider, Google, Microsoft, do they have a legal obligation to screen for CSAM in messages that are going through their servers?


[00:22:31] YL: No. That’s an excellent question. They do not have it. No. And this is actually, as I said, this discussion is a part of a broader discussion about tech platforms’ responsibility and liability for users’ content. The big elephant in the room here is Section 230 of the Communications Decency Act. Platforms are protected from legal liability for content created by their users and child abuse materials have been always around, but we see a dramatic rise in 2020. So this is why we see this conversation happening now. At this point, child abuse materials reside under Section 230 immunity. So no, they do not have the obligation to report. Back in March, the Department of Justice proposed the set of voluntary principles that took aim at tech giants in an effort to combat online sexual abuse. But at this point, we still don’t have these voluntary principles fully fleshed out. And also, as I mentioned, it’s voluntary. So absolutely not.


[00:23:38] JP: I'm really surprised by that because I guess I just assumed there was a legal obligation. So my follow-up question is why do you think Apple is waiting into this area? I’m curious about the two features and scopes of images they’re looking at. I think you can make a stronger case for looking at CSAM. There’s a lot of public support around it. We want to protect children. But why do you think Apple is expanding this to look at messages and iMessage and warn children about sexually explicit materials? Do you have any sense of why that might be something they want to tackle?


[00:24:13] YL: I believe the two are kind of grouped together in concerns about child safety. And there was at some point that Apple’s anti-fraud chief suggested that his company's commitments to privacy has led Apple to become what he called the greatest platform for distributing child porn. And again, 2020 with COVID and many people just basically shifting their whole offline life online, we’ve seen a huge increase in the number of reported child sexual abuse materials. So I believe that was kind of the driving force behind it.


[00:24:53] SY: So should we be worried about either the scanning feature, you mentioned the other two features that were announced? And should we be concerned about the slippery slope? And if not, when should we be worried?


[00:25:05] YL: We should always be worried. We should always be worried and we should always be very alert to any new technology that has been developed exactly for these reasons. My point is that we always have to be worried, but at the same time, we always have to understand that there is a balance to strike here. We do have privacy and security. They are very important values, but at the same time, we do want to protect children. At the same time, we do want to promote public safety. There are a lot of questions of power here. The fact that Apple holds this power is something that most of us feel very uncomfortable with. We don’t want a private company without any protections, without checks and balances to handle our information and have the power to scan it and to report. And I believe that all of these concerns are valid and should be part of a conversation about tech power and about how much power we give tech companies and watch checks and balances we should put in place in this context. But whether we should be worried, yes, we shouldn’t be worried. We always should be worried. The slippery slope is always a concern, but we’ll have to monitor as we go, because technology’s not going to stop from being developed. We have a lot of market incentives for startups and bigger companies to keep developing technologies. At the end of the day, this is humanity advancing. So we will have to monitor as we go. We will have to understand that we must move some of those decisions from individual hands. We cannot have companies put in their terms of service, many, many decisions for all of us to make decisions were unaware of when we click I agree, decisions that maybe we shouldn’t be making in an individual capacity because they have broader social consequences.


[00:27:03] JP: Is there anything else you’d like to add that we haven’t covered yet?


[00:27:07] YL: So other concerns that have been kind of cited by civil liberties organizations in this context of the second feature of scanning iMessages, photos were the creep factor, the fact that the notifications give this sense that Apple is watching over users. Right? And especially when we talk about kids. We’re talking about kids under 13 and between 13 and 17. It almost feels like Apple has given parents the ability to surveil them. Right? And sometimes, by the way, there is a real concern for abuse of the system by parents.


[00:27:42] JP: Oh! Yeah.


[00:27:42] YL: So the system will also give parents, including some who do not have the best interests of their children in mind, another way to monitor and control them and family sharing plans are organized by the parents and they could be organized by abusive parents. So this definitely become some sort of kind of disciplinary technology. And one more point that we haven’t discussed but is extensively being brought up in the context of big data and machine learning technologies is the choice of that technology in the second feature. So Apple has chosen the use of technology of machine learning classifiers, which is notoriously difficult to audit, right? Machine learning technologies without human oversight, and in this second feature, there is no human oversight before the reporting to the parent, they have the habit of wrongfully classifying content, and in this context, specifically, sexually explicit content. There have been multiple attempts in the past by other companies to use machine learning classifiers for that. These attempts have been really tremendous failures. So there are different concerns about biasing the data and the training data and the algorithms that are not huge concerns in this particular context, because the damage or the harm is not significant, but we should think about it because, as we said before, this technology could be further advancing, expanding, and use for other purposes.


[00:29:15] JP: Just one follow up. From my understanding, and I’m curious if you have interpreted it the same way, for what I’ve read, the way to opt out of what we’ve called the third feature, the scanning of photos in iCloud, is to turn off iCloud and not upload your photos to iCloud. But there doesn’t seem to be a way to opt out of the scanning of your photos in iMessage.


[00:29:38] YL: There is a way to opt out if you don’t have a family plan. Okay? So if the account is not designated as a minor account, if it’s not part of family plan, the feature will not be activated.


[00:29:51] JP: Oh, got you. Because this feature, the scanning of the messages is specifically looking for explicit material for under age iMessage users.


[00:29:59] YL: That is exchanged with an underage minor. Correct.


[00:30:03] JP: Got you.


[00:30:04] SY: Well, thank you so much Yafit for being here.


[00:30:06] YL: Thank you. I really enjoyed having this conversation with you.

[00:30:17] SY: Coming up next, we talk about the potential privacy and security dangers of these new Apple features from a Princeton Cybersecurity Researcher whose team had built a similar child sexual abuse materials scanning system after this.




[00:30:43] SY: Joining us is Anunay Kulshrestha, Princeton Computer Science Doctoral Candidate, and co-author of the research paper, Identifying Harmful Media in End-to-End Encrypted Communication: Efficient Private Membership Computation. Thank you so much for being here.


[00:30:58] AK: Thanks for having me.


[00:30:59] JP: Well, let’s get into your research paper. Can you talk about the paper? And I’m really interested in the system that your team created that’s similar to what Apple is doing.


[00:31:08] AK: So let me explain the key challenge here. The challenge here is that you would want innocent users that do not share CSAM to not have their privacy infringed upon. So you would want these innocent users to have complete privacy guaranteed, like complete privacy production. But the outcomes in the case of innocent users should be the same as what you have currently without the sort of scanning. So no innocent users should ever have their privacy compromised. That’s one of the key requirements that we call “client privacy” in the paper. The second important requirement, a subsistence, is server privacy. So you also want the hash set, in this case instead of perceptual hashes that you’re using for matching, you want that set to not be public because that creates all sorts of perverse incentives, can lead to a lot of evasion. Even having the possession of these perceptual hashes is illegal in many jurisdictions. So you don’t want the servers that to ever be revealed or any information about it to ever be revealed. This is the server privacy property in our paper. And so these are the two important privacy guarantees that we want. And the third guarantee that we want is that for honest users, no one learns anything about what they send, and for dishonest users, or, I mean, I guess like malicious users that sends CSAM, the server in our case, like in one instantiation of the system, the server realizes that CSAM is being sent or something malicious is being sent. It is also possible to do the sort of detection client side, as in like the end of the protocol, the result of the protocol can be learned by either the server or the client in our paper. So these have different use cases. So in case of CSAM, the recipient and the sender collude. So it’s not very useful for the recipient’s phone to learn whether CSAM was sent. In that case, you want the server to know and the server to catch them. But in case of disinformation and other sorts of things that are not illegal per se, you might want something that let’s say I’m a recipient and then I received some message, there’s own device detection that happens, client’s side, that tells me whether this thing is in some known misinformation database or not. So these are two different sorts of use cases of the same perceptual hash matching detection systems. And the challenge is that how do we achieve the client privacy property? How do we achieve the server privacy property? And how do we achieve this sort of correctness property all at the same time while making sure that we’re building is efficient enough to run on commodity hardware, like commodity smartphones? And so that’s the problem that we solve in the paper. The problem that we don’t solve in the paper is how do you reduce trust in the custodian of whoever is doing the matching? Right? So in the paper, we make it very clear that you really have to trust the server with our paradigm and with what Apple’s doing as well. You really have to trust that the block list isn’t compromised and hasn’t been repurposed or censorship or other nefarious purposes.


[00:34:00] SY: Is that if you are getting the block list from a server? Because Apple has said that the block lists are embedded in iOS system images and they’re not being downloaded over the air. So if you take that out of the equation, does that server trust issue still there?


[00:34:14] AK: Yeah, because this entire paradigm is fraught with multiple issues. One is that this doesn’t really do anything to reduce trust, right? Because it’s Apple embedding the block list, whether it’s downloaded every time or whether it’s downloaded once and shipped, it doesn’t matter. It does that. And secondly, there’s always the issue of shipping something to something that you don’t want to be revealed to millions of users, which is why you see all these collisions that are being found in neural hash and you see a bunch of image attacks and so on and so forth. It’s just very difficult to ship something to millions of users and not have someone reverse engineer the entire thing or in part. So I think that is the challenge here. If Apple indeed is shipping some form of this block list, in this case like they’re blinding the block list and then shipping it to people, I think it’s risky. The cryptographic blinding, although it does guarantee that people will not be able to just recover the block list, but this sort of paradigm where you ship some sort of secret information to people is not something we consider the best way to do this. And we talked about this in the paper as well, which is why we don’t actually try to protect the perceptual hash function at all. We use open source perceptual hash functions. So you know exactly what function was used to produce the perceptual hashes and we’re not trying to ship secret information somehow like encrypted and then believe that no one’s going to be able to get access to it. So that’s just not the paradigm that we follow in our paper.


[00:35:38] SY: So let’s walk through the steps technically on what actually happens. So I take a photo on my iPhone. It is synced to iCloud. So I expect that I want it to be uploaded to iCloud. What happens after I take that photo?


[00:35:55] AK: So iCloud backups anywhere on encrypted. So Apple, anyway, can scan all of that. What is interesting here is that Apple also wants to scan encrypted messages. So like your iMessages that contain images. The idea with Apple’s detection system is that they have a threshold of the number of images that they need to catch for user before any of them can be decrypted by Apple. So let’s say the threshold I think has been mentioned in media is around like 30. So unless 30 different matches occur for a single user, Apple cannot decrypt any of those 30. So let’s say you have 29 matches already and the 30th happened, as soon as that happens, Apple is able to decrypt all 30 of those using cryptographic techniques. This is how they use threshold secret sharing. So the idea is that whatever decryption key that Apple can come up with after 29 matches is just not the right decryption key. So they need 30 in order to decrypt the 30 things. And that’s the sort of privacy guarantee here. Just to reiterate, this does nothing about trust, right? You’re completely trusting Apple here on whatever the block list is.


[00:37:03] SY: Go through the steps before that 30, my understanding of it is as part of the, and they’ve been very adamant to be clear about this, that it’s not happening on all devices, it’s happening as part of the pipeline to upload to iCloud, right? It creates a hash of your image. It compares it to the image hashes of CSAM, and this is the on-device scanning that people are talking about, that determines is there a match. And from there, they create something that I think is called a safety voucher that kind of has the result of that match whether or not it is or is not. What’s interesting is that the safety voucher is locked. Your device can’t actually read it. So your device doesn’t actually know if there is a match or not until it’s uploaded to Apple. And as you said, you need 30 of those positive matches for Apple system to detect, “Okay, there’s a problem. There are now 30. Now let’s alert or let’s send it to our human reviewers first, actually. And then if it’s confirmed, then let’s send it over. Let’s make a report to NCMEC,” which is the organization that they’re working with for the CSAM scanning initiative. So that’s kind of my understanding of the technical steps along the way. Do I have that right?


[00:38:20] AK: Yeah, that’s correct. The only thing I’d like to add is after every match is Apple service do try to decrypt everything, right? They do try to come up with a decryption key, just that threshold secret sharing guarantees that whatever they come up with is gibberish and isn’t…


[00:38:34] SY: Okay.


[00:38:35] AK: Yeah. So after 30 matches or whatever the threshold is, that’s when they will be able to correctly decrypt everything and that’s when they’ll know. So they’ll know when they correctly decrypt things. It’s fairly simple to implement something like that. So they know when they correctly decrypt something and that’s when the process moves forward like said.


[00:38:52] JP: You mentioned in your interview with the Washington Post that you “sought to explore a possible middle ground, where online services could identify harmful content while otherwise preserving end-to-end encryption”. And you found that middle ground doesn’t exist. Why is that?


[00:39:08] AK: In our paper, it is evident that the technology for doing this sort of privacy preserving matching does exist. Right? So it’s not about the privacy preserving part. The issue is not with the privacy here. It really is a trust issue. Who do you trust with this sort of detection? Who do you trust will not repurpose this sort of detection system for censorship? That is the core issue that we’re trying to highlight in our paper and that’s the core issue with what Apple is trying to do also, right? They didn’t take any measures. They didn’t actually invest any resources in trying to figure out how to reduce trust in the custodian of the block list. The privacy preserving bits of their system are excellent. They are indeed privacy preserving, but they do have some good cryptography out there, which is novel, and there’s some neat tricks that they use to ensure that the system itself is privacy preserving. So that’s not the criticism here and that’s not the core issue with these sorts of detection systems. There were some skeptics that didn’t believe that privacy preserving detection was even possible. That’s no longer the case. People do think that privacy preserving detection is possible. It’s just the trust. And I mean, that’s like a much bigger issue than the privacy preserving bit, I think. Who do you trust will be the right custodian for this sort of block list and what all should even be added to that block list? Right? Like only CSAM should be added, I guess, because it’s basically the only thing that’s illegal in almost every jurisdiction. If you were to add disinformation or terrorist recruiting imagery or so on and so forth, these are all contextual things and may differ from jurisdiction to jurisdiction. So this is fraught with a lot of policy issues.


[00:40:44] SY: So in the past, Apple has been pretty adamant about not creating software that can enter someone’s phone and look at their data without their permission, even if it means not helping authorities solve major crimes. So I want to talk about some of the security risks that come along with such powerful features. What can happen by building something that’s so easy to scan through your data?


[00:41:05] AK: Well, I mean, the elephant in the room that everyone’s been talking about is a very simple scenario. Some government forces Apple into adding things like political information or something that the government wants censored or at least detected do this set and then the system is just repurposed for political censorship and persecution and so on. That’s, I guess, the biggest fear that people have. There’s obviously a bunch of other related issues whenever you expand, like you’re essentially expanding the attack surface. And we talk about this in our paper as well. So end-to-end encryption usually there’s a bunch of algorithms that are used for end-to-end encryption, the single algorithm is one of them and so on, the double ratchet and so on, right? So these algorithms already are quite complex, right? And there’s a risk of implementational error in them. And now you have this other privacy preserving detection system on top of it. Of course, that’s going to lead to more engineering work and can lead to more implementational errors that could then be exploited as vulnerabilities. We already know about the NSO Group trying to use these sort of zero-day vulnerabilities to get access to people’s iOS data. So more and more zero-day vulnerabilities could be created just because we’re now increasing the attack surface, like by deploying a much more complex system, which is just a much more complex in terms of implementation.


[00:42:23] JP: So originally, one of the concerns was that Apple would be able to scan user’s phones for images that might not be CSAM. So for example, the Chinese government could request scanning for images of Tiananmen Square or other banned images in China. So the news that came out after that, that explained that these sets of image hashes are going to be distributed with the OS and researchers will be able to look at them, I think, has tamped down the criticism a little bit. Do you think we should still be worried about governments using this system to scan for images? Or do you feel that Apple has tried to put enough guards in place to alleviate that concern?


[00:43:10] AK: I absolutely think we should be worried, especially because it’s not entirely clear what sort of like auditing Apple is even talking about here and it’s not entirely clear what sort of cryptographic guarantees that they would be able to provide on how this had originated. There are a bunch of ways they could go about this. There are some ways that we also discussed in our paper of how to reduce trust in the system. But of course, all of that, it requires some sort of policy framework, needs to be embedded in some sort of policy framework. I don’t think Apple has done enough. Apple has definitely not made people happy in the academic community with the way it is approached this entire question of trust. There’s really not much that they’ve come out with, like concretely apart from like some fuzzy talk about people would be allowed to audit it, but it’s not entirely clear how, and also how would you do this sort of public audit without revealing any information about the set? Right? That requires some novel cryptography and that hasn’t been discussed. So there’s a lot of open questions right now. Just putting out a press release saying that, “Oh, something will be done about it,” really doesn’t cut it. What we do need is some sort of concrete proposal, the same way Apple came out with a concrete proposal for CSAM detection, right? They should also come up with a concrete proposal for how it’s going to reduce trust in the system.


[00:44:21] SY: So the other big concern is this whole issue with on-device scanning and that’s caused a lot of alarm. And what’s interesting about the technical implementation of this type of scanning is that scanning is happening on the device, but you still need Apple servers to kind of finish the process. I think as we discuss walking through the technical steps of the scanning process, your photos get scanned and compared on your phone, a safety voucher is created, but from there, there’s nothing actually on your device to read that safety voucher or understand that safety voucher. And it has to be sent to the cloud to kind of make sense of it and actually discover whether or not there is an issue. And so given that it’s like half on device, half on cloud, is it as big of an issue in your mind as people are making out to be when they’re worried about on-device scanning? Does the server component make it any better?


[00:45:17] AK: No, it really doesn’t make it any better. I mean, this is also fraught with the same sort of issues, right? The only good thing I guess about this system is that the learning model just uses things that are already on your device. So it’s not really spreading your unencrypted data everywhere. But apart from that, this is also fraught with this and sort of trust issues, also fraught with the same sort of like implementational error issues and so on, like we spoke about. So it’s really no different from the other types of scanning that just happens in the cloud. And also in our paper, if I were to just quickly mention the first protocol that we talk about for exact matching, it’s pretty similar to what do you described where there’s like what Apple is doing also where there’s a safety voucher that’s created on the device, which is just a layer of encryption added on top of whatever was already encrypted, right? So there’s just a second layer of encryption. And then this double encrypted thing is sent to the server. And the very simple thing that’s happening here is, let’s say you have some value X, right? You just encrypt X and what the server is then doing is just checking whether your encryption of X is like the same as the encryption of some other Y in its set. So literally just like subtracting the encryption from the encryptions of everything else in the block list in our protocol. So the idea is just that you encrypt something and you subtract it from everything else, whatever decrypts to zero is a match. And so in order to do this sort of matching, you need both the client and server to participate. So the client generates the initial voucher, but then eventually the server must subtract it, and simplistically calling it subtraction, but performs some computation on that safety voucher or whatever you want to call it. Right? So the server must also compute, and that’s why this is a two-party computation system. Even if all of this was on the device, like no server was involved, you would still have that issue about the custodian of the block list, right? The server is always going to be involved in creation, curation, and just maintenance of that list, regardless of whether it’s involved in the actual computation or not.


[00:47:15] JP: There’s been really confusing reporting, whether Apple can access photos that are currently in iCloud, whether they can’t, whether they have been scanning or only had the capability to scan those photos, but haven’t been actively scanning it. I’m curious. Why do you think Apple is pursuing this on-device solution so hard instead of just scanning photos on iCloud servers?


[00:47:43] AK: So I think Apple hasn’t had a sort of content moderation team in the past. I’m not, not sure. I may be wrong with this. But I don’t think they have a sophisticated content moderation infrastructure, the same way Facebook does. So Apple, I don’t think is familiar with the challenges that come with this sort of content moderation. And I think this is the first foray that they’ve made into this sort of CSAM scanning and so on. And I figured they really upset law enforcement with a bunch of their positions in the past. Right? So this is probably some sort of compromise that Apple’s trying to reach, I guess. That’s my best guess, of course. I’m not currently doing [00:48:21] to any of those conversations so I wouldn’t know. But it is very surprising that people at Apple thought that this would be palatable. It’s just very surprising, I think. And there were also like, I guess, reports that Apple wants to perhaps encrypt iCloud backups, and that’s probably why they’re trying to do this. That could be it. I’m not sure, but they’re just all rumors, I think.


[00:48:42] SY: So a lot of critics have called this essentially just a backdoor and they painted some really dark images of government surveillance and targeting and suppression and potentially attacks. Given that Apple, it seems like they’re trying to alleviate these concerns and they’ve at least thought of a bunch of different precautions and safety measures and different ways that they are trying to make the public feel comfortable with these changes and I think different precautions that they believe will protect them from some of these bad actors. Given the way the technology works and given all the stuff that they have thought about and designed to make it safe, how much should we really worry at this point? How easy is it for a government to come in and just abuse the system?


[00:49:33] AK: I wouldn’t be surprised if governments are already trying to strong-arm Apple into repurposing this. I mean, I’m scared about countries like India, China, Russia, where there’s laws in the books that allow the government to do this sort of thing. There’s already this sort of scanning happening with WeChat in China. Apple technically is one of the only large firms, large tech companies that still does business in China. Right? So Google and Facebook aren’t a part of this debate, but there really is this concern that Apple will be strong-armed by the Chinese government into repurposing the system to do things that it’s not supposed to do. And Apple really hasn’t done anything to assuage those concerns. It is still not clear whether this sort of system would be deployed in other countries. It is not clear who would have the constitutional statutory responsibility, like the way NCMEC has in the US. I think there’s a carve out for NCMEC and NCMEC has allowed to have access to CSAM and can’t be prosecuted for that. That sort of like legal carve out probably doesn’t exist in most countries for any nonprofit to have access to these things. So I’m not sure how it would work in other jurisdictions. I wouldn’t be surprised if governments are already trying to get this sort of detection happening within their borders. It’s fraught with a lot of issues. What everyone’s been talking about how this is a backdoor, everyone’s just basically talking about the trust issue here, right? No one is saying that this is a privacy backdoor and the detection isn’t like privacy preserving and so on. So I do want to make that distinction, right? Even though that the detection is privacy preserving, It’s not transparent, right? There’s no mathematical proof or there’s no cryptographic proof of the fact that the things added to that hashes were indeed CSAM. And that is not just a technical problem. That’s also a law and policy problem. So it requires some sort of like syncretic solution that is both, aside if it's constitutional or whatever, is a legal solution, but it’s also a technical solution. So it could be that legislative bodies mandate some sort of detection system and mandate that only these sorts of hashes will be added and mandate that this body will be the custodian, this nonprofit will be the custodian or something like that. So that’s sort of like legal and policy framework is required in every country where Apple would like to do this. Without that, I think we should all be very scared.


[00:51:48] SY: So Apple recently came under fire for allowing their data centers to not only be located in specifically China, but also their data centers are now run and operated by China. Do you think that the combination of the data centers location as well as the creation of these new data scanning features is kind of like a perfect storm for governmental abuse? When we talk what we’re worried about will happen, is this the kind of thing that we are worried about?


[00:52:15] AK: Indeed. And the way Apple system currently works is that the entire block list essentially is multiplied by, obviously, you're talking about it in a very simplistic way, but essentially you take some secret number and you multiply everything with it. And that’s what Apple is trying to do. That’s what the blinding is essentially. How that lining works and how to reverse engineer it is entirely known to Apple, right? It’s not a one-way sort of blinding. You can re-compute what the thing was and so on. So whoever has access to that sort of data, whoever has access to whatever that secret key was to use to blind the entire hashset can actually do a lot with that, can figure it out what hashes were caught, what matched in some cases and so on. So the data sovereignty issue here is also, I think, pretty important. If Apple servers are located in China and China has some laws in the books that allow the government to get access to some of that data, I think that could be pretty harmful for the sort of detection system.


[00:53:14] SY: Well, thank you so much for your time.


[00:53:16] AK: Thanks so much for having me.


[00:53:28] SY: Thank you for listening to DevNews. This show is produced and mixed by Levi Sharpe. Editorial oversight is provided by Peter Frank, Ben Halpern, and Jess Lee. Our theme music is by Dan Powell. If you have any questions or comments, dial into our Google Voice at +1 (929) 500-1513 or email us at [email protected] Please rate and subscribe to this show wherever you get your podcasts.