Are the AI regulations and systems happening in cities around the world actually up to snuff?
In this episode, we talk about how a developer irreversibly lost a community of 54-thousand stars and watchers built up over the past 10 years on GitHub, and how unregulated crypto-mining wrecked the power of an entire New York town. Then we speak with Vidushi Marda, senior program officer at Article 19, where she leads the research and engagement on the human rights implications of machine learning, to get her perspective on the Artificial Intelligence regulations and systems different cities around the world are implementing.
Saron Yitbarek is the founder of Disco, host of the CodeNewbie podcast, and co-host of the base.cs podcast.
Josh Puetz is Principal Software Engineer at Forem.
Vidushi Marda leads ARTICLE 19's global research, engagement and strategy on machine learning and human rights.
[00:00:10] SY: Welcome to DevNews, the news show for developers by developers, where we cover the latest in the world of tech. I’m Saron Yitbarek, Founder of Disco.
[00:00:19] JP: And I’m Josh Puetz, Principal Engineer at Forem.
[00:00:22] SY: This week, we’re talking about how a developer irreversibly lost a community of 54,000 stars and watchers that he built up over the past 10 years on GitHub, and how unregulated crypto mining wrecked the power of an entire New York town.
[00:00:38] JP: Then we speak with Vidushi Marda, Senior Program Officer at Article 19, where she leads the research and engagement on the human rights implications of machine learning, to get her perspective on the artificial intelligence regulations and systems different cities around the world are implementing.
[00:00:53] VM: So you can be super transparent about the algorithm itself, but when it comes to machine learning systems, the algorithm doesn’t mean much, the source code doesn’t mean much if I don’t also have access to, say, the database.
[00:01:10] SY: Josh.
[00:01:11] JP: Yes?
[00:01:12] SY: Imagine this.
[00:01:13] JP: Okay.
[00:01:14] SY: You’re working on your beloved open source project on GitHub. You’re click-clacking along and you slipped into autopilot. So in your mind, you’re fiddling with a new repository connected to the main repo. Nobody follows this new repository and it just contains some profile READMEs, and you decide that you want to hide these READMEs by making the new repo private, no big deal. But in your absentmindedness, you don’t realize that you’re not actually in that new repo, instead you’re in a very similarly titled main repo, which you’ve just made private, causing you to permanently delete all of your stars and watchers.
[00:01:57] JP: Oh, not good.
[00:01:59] SY: Yeah. So this is what happened to a programmer, Jakub Roztocil, the Creator of a CLI HTTP client called HTTPie, which sounds delicious, which was created to make API interaction from the terminal more intuitive and was super popular, one of the top 80 most popular public repos of GitHub.
[00:02:21] JP: I use this app. I love it.
[00:02:22] SY: Yeah. Nice! So if you were one of the 54,000 stargazers or watchers, which have been built up over the past decade, this project’s existence, now you have to go back and star or watch the repo again, so you can get any notifications about that repository. So Roztocil says in a blog post about the snafu that it took around 30 minutes for GitHub to cascade delete the decade’s worth of stars and watchers. And even though he realized right away what happened, he couldn’t really stop it and he writes, “All I could do was start writing to GitHub support, refresh the page, and wait for the number of stars to reach zero before I can make it public again.”
[00:03:07] JP: Oh, that must have been so frustrating.
[00:03:07] SY: That must be so painful. I know. You’re like, “Oh, it’s just a button. Let me go back.”
[00:03:11] JP: Just watching the number go down, realizing it’s already done.
[00:03:15] SY: Over 30 minutes. Yeah. Now just how easy is it to make this mistake? Well, Roztocil’s main repo is called HTTPie/HTTPie. And the new repo was called HTTPie/.GitHub. To confirm that you’re changing permission of the repo you mean to, GitHub has you confirm your decision by making you type in the name of the repo to confirm the switch from public to private, and also does the same thing if you want to delete the repo or do any other destructive actions. However, there’s nothing in the dialog box that tells you just how many stars and watchers will be permanently deleted, which Roztocil says would have immediately given him pause. In the post, he writes, “Show don’t tell. Design confirmation dialogues in a don’t make me think fashion. When the user is about to destroy something, don’t describe that as a potential scenario and abstract word that the user needs to convert to mental images and put values on.” Now the ironic thing about the situation is that GitHub itself had once made the same mistake in November, 2020, accidentally making one of their public repos private, also losing that repo’s stars and watchers. However, it was not permanent. Right after the human error, GitHub CEO tweeted, “A developer mistakenly made the GitHub desktop repo private this morning. Flipping it back doesn’t restore the stars and a few other things. So we are restoring from DB backup. That’s all.” Roztocil says that when he reached out to GitHub to see if they could restore his repo stars and watchers, they refused to do it saying that there would be undesirable side-effects and there would be the cost of resources to do this. He responded by offering to give GitHub financial compensation for the resources required to restore his repo, but they refused. At the time of this recording, which is about six days after the accident, HTTPie has 16,900 stars and 49 watchers. What do you make of all this, Josh? What’s your take? What do you think?
[00:05:21] JP: Well, what I read through the blog post and took a look at the screenshots, it’s a warning from GitHub that says you’re changing the repository visibility.
[00:05:29] SY: Right.
[00:05:30] JP: It does have a warning say this potentially destructive action. I know for a fact I’ve done this in the past and I’ve just copied and pasted the name of the repo and I’ve also actually read, like I never read, I was like make private.
[00:05:42] SY: Whatever, whatever, yeah.
[00:05:43] JP: It actually says you will lose all stars and watchers. I’ve done this a bunch of times. I'm making a new repository and I’m like, “You know what? I don’t like the name of it. Oh, I think I’m going to delete this. I’m going to do a different repository.” I know for a fact I’ve cruised through this warning box a number of times and it never occurred to me that bad things would happen, even though it spells it out of this UI.
[00:06:03] SY: Yeah.
[00:06:04] JP: Yeah. And even though it has you typed to confirm, we all just copy and pasting, come on.
[00:06:09] SY: Okay. So I am a good user and I always type my repo names. Okay?
[00:06:13] JP: Oh, wow!
[00:06:13] SY: Because it says, it’s not just GitHub that does this, other apps do too, but it’ll say like danger zone and it’s like bright red and it’ll like, “These are destructive actions,” and they’ll say, “Are you sure?” I’m like, “Oh my God! Am I sure? I don’t know. This is dangerous. Am I sure I want to do this?” I actually am very paranoid. So I don’t copy. I do actually type, but obviously deleting repo is a destructive action, right? It’s freaking gone, that’s it. But going from public to private doesn’t feel destructive. So to me, it’s not intuitive. It’s not obvious that, “Oh, if I just make this private, that I’m just going to lose a bunch of data on that repo.” That connection to me is not obvious. I mean, I guess it makes sense that I wouldn’t have any additional watchers because obviously you can’t watch private repo. I get that part. But to me, I would guess that if I flipped it back, then I would, I don’t know, I guess I just assumed I would get it back. The connection wasn’t obvious to me. I would be surprised by that.
[00:07:17] JP: And when I read this, I could imagine the database design in my head. I was like, “Oh, yeah, there’s all these watchers. There’s all these stars. It’s private now. Notifications are probably hooked into this.” If you look at the tweet threads that the author of the blog post posted, there’s some response from people at GitHub saying like, “Oh, we can’t really roll it back easily. We’ve tried it in the past.” They might be referring to a time that they did a restore of a database. And it said, “We tried this with a couple of projects in the past. We found it caused a bunch of bad things to happen. That’s why we pop up a dialog before we make the project private.” I found other tweets referring to like it screwed up notifications.
[00:07:54] SY: Oh, interesting.
[00:07:54] JP: And so in my mind I was like, “Oh, okay, I get it. They probably just keep the notifications off of watchers and stargazers. And if it’s public, they don’t want to send notifications and you can save database space by getting rid of all this data.” And probably somebody thought like, “Hey, it’s not like privileged data. It’s just a watch, whatever.” So I think there’s a couple of things happening in the data design that I’m not defending, but I could understand. What I can defend, I don’t like it, but I could definitely defend GitHub saying, “Look, we’re not going to do a database restoration just for your project. Who knows what other things that could trigger?” They make it sound like they’d tried it once and it didn’t work out. And also from like a customer support perspective, what kind of precedent does that set?
[00:08:43] SY: I do get that. I get that. Yeah.
[00:08:44] JP: Like if I accidentally delete a file, can I get my thing rolled back? If I closed PRs before, God, I wish I could undo that. So I can understand GitHub saying like, “Sorry.” The bigger thing to be is that I was really surprised by the emphasis that open-source projects place on the number of stars and watchers that they have. Do you think that’s really community? I understand there’s value in terms of getting notifications about what’s happening with your code, but is that really community? I don’t know.
[00:09:19] SY: Yeah. I mean, I don’t know if it’s community, right? I was actually reading a blog post recently that was about just marketing in general and it was titled something like “Your Followers Are Not Your Fans”. Right? Just because someone is following you and gives you a thumbs up does not mean they’re your community, does not mean they’re going to buy your stuff. You know what I mean? It’s two different concepts, but I do think that there is a lot of value in just the social proof of a repo because whenever I am looking at an open source project or just checking out the work that a developer is doing online, especially if it’s a project I’ve never heard of, I need help assessing, “Is this legitimate? Is it real? Is it serious? Is it someone’s kind of spun up over the weekend toy project kind of thing? Or is this some hardcore stuff?” And one of the easiest ways to do that is to see how many stargazers and watchers there are. And if there’s no stars, no watchers, I’m like, “Okay, this is probably just a small little thing and maybe keep it moving.” But if I see tens of thousands of people who’ve essentially given that repo some mark of approval, at least at some point, that makes me go, “Wow, this person is serious. This repo is a serious project.” And it definitely brings legitimacy to that project. And I can imagine where if you are trying to start a community, build a community, I think that that legitimacy does matter. If you’re trying to get sponsors, I think that legitimacy does matter. Even in situations where you just want to get a job. If you have a repo that has 15,000 stars, that probably makes you look like a more bad-ass developer. Right? So I do think that there’s a lot of value in it. But I understand that maybe the value isn’t technically community, but I do think it’s related to community.
[00:11:10] JP: Yeah. I think everybody can agree that it’s way too easy to cruise past this particular dialog box. I mean, there’s some mock-ups in the blog post. I think they’re great mock-ups, just showing you like instead of saying like all your watchers, all your stars could be deleted, like actually putting those numbers out.
[00:11:27] SY: That’d be nice. Yeah. That’d be helpful.
[00:11:27] JP: Like is it a hundred? Is it 500? That could help. It sucks though. I feel really bad for this developer.
[00:11:35] SY: Yeah, over a decade.
[00:11:36] JP: I love it. Yeah. I know.
[00:11:38] SY: It’s tough.
[00:11:39] JP: I looked and sure enough, I wasn’t watching or had starred anymore.
[00:11:42] SY: Ah, you’re a terrible community member. What are you doing?
[00:12:06] JP: So there was a really wild story in the MIT Technology Review about a New York town called Plattsburgh that had its power infrastructure pushed to the limit by crypto miners. Plattsburgh’s electrical rates are low compared to the rest of the country, thanks to cheap hydroelectric power from nearby Niagara Falls. In 2017, a subsidiary of the crypto mining firm Coinmint came to Plattsburgh and leased a family dollar store and immediately filled the building with servers running them 24 hours a day. The amount of energy they were typically drawing could have powered 4,000 homes. After this, other crypto miners followed suit. To give some context, every bitcoin transaction uses more kilowatts of power than the average American uses in an entire month. So in January, 2018, the city reached its quota of hydropower from the Niagara Power Authority, and they needed to purchase additional energy at really high rates.
[00:13:01] SY: Wow!
[00:13:01] JP: This of course increased the community’s energy bill. Not only that, but there were other effects such as the fans that crypto miners were using to cool down their servers cause high frequency wines that neighbors could hear. Over time, Plattsburgh stopped any new crypto mines in the city and imposed higher rates on high density energy users, and they updated their building codes to deal with the noise. Plattsburgh’s local crypto mining issues aside, according to Digiconomist’s Bitcoin Energy Consumption Index, worldwide crypto mining produces 36.95 mega tons of CO2 a year, which is about as much as the entire country of New Zealand.
[00:13:40] SY: Wow!
[00:13:40] JP: And in a paper cited by the Technology Review Article, cryptocurrencies’ energy usage could rise by another 30% within a decade, almost doubling its current CO2 footprint. So the moratorium in Plattsburgh on new crypto mining is now over, but with all of the new regulations they’ve put in place, there hasn’t been much renewed interest. Instead, crypto mining firms have moved on to the neighboring town of Massena, which lacks the regulations Plattsburgh has in place. Very interesting story about how this town struggled to recognize the effects of what was happening. They started out with great intentions. And in 2017, if you were a small town and someone said, “Hey, there’s an abandoned family dollar, we’d like to lease it, we just need some extra electricity going to it,” I’m sure most cities would have been great with that.
[00:14:32] SY: Yeah, it sounds like a great deal. Yeah.
[00:14:34] JP: Some interesting things I took away from the article, at the time, Plattsburgh didn’t have a city income tax and most of the crypto mining firms lease the buildings, meaning that they weren’t paying any property taxes. So none of these crypto firms were bringing in additional revenue to the city.
[00:14:52] SY: Wow! Useless.
[00:14:54] JP: Pretty useless and that’s a huge issue for cities is recouping the cost of all this infrastructure.
[00:15:00] SY: Yeah, yeah, yeah, absolutely.
[00:15:02] JP: Something else I took away from the article. Plattsburgh went through a couple years of putting a moratorium on these new facilities. They beefed up their regulations. They put in noise ordinances. You couldn’t have these high pitch fans, disturbing neighbors. They put in regulations that would require crypto mining firms to outline how much energy they were going to use and paid upgrade infrastructure if it was going to have a material impact on the entire city’s energy usage, and the result of all of that were that crypto miners just moved next door. They went to another town that didn’t have these protections in place. And one of the experts of the story points out that because crypto mining is so profitable, it is just cheaper and easier for crypto mining firms to pack up and move next door. They don’t have a vested interest in staying in a town where the regulations are in place.
[00:15:52] SY: Yeah. I mean, I have a couple of thoughts on this. I’m wondering, does Plattsburgh or any of these other cities who’ve implemented some of these regulations, do they care about the crypto firm? Well, to me, it sounds that they weren’t really doing much good anyway. I don’t know if they were hiring local talent or contributing to the economy in some way, but it doesn’t really sound like they are?
[00:16:17] JP: Yeah, not very much. These crypto farms are mostly automated.
[00:16:19] SY: Right, exactly.
[00:16:21] JP: They’re not hiring anyone.
[00:16:22] SY: They’re not hiring. They’re not bringing like fat-free jobs to the city or anything like that. So for Plattsburgh, I’m kind of wondering, like, are you hoping that the company will stay with these regulations or was the regulation kind of way to be like, “Okay, great, we got rid of them”?
[00:16:40] JP: Yeah. That’s how I read it. I draw a lot of parallels between this and like, say, Airbnbs. I know a lot of like tourist towns have struggles with the Airbnbs. With Airbnbs, at least you can impose things like occupancy fees, hotel fees, recoup some of that cost. But in Plattsburgh’s case, they were rentals. They weren’t really contributing any property tax income.
[00:17:01] SY: They’re doing nothing. Yeah.
[00:17:02] JP: What do you think about larger regulations on crypto mining? Because at the end of the day, Plattsburgh put all these restrictions in place, the crypto miners just moved next door. I’m sure if Massena put some regulations in place, they’ll move on to the next small town. So do you think this might be an opportunity for a state or a federal level of responsible energy usage?
[00:17:24] SY: Absolutely. Yeah. I definitely see a huge opportunity for the state, the federal government to step in and say, “We’re going to basically protect our cities and we’re going to protect our smaller towns.” It’s great that they were able to solve their problem, but ideally you don’t want the cities to have to go through that process of welcoming in a firm and then waiting for them to mess it up and then responding with laws and kicking them out again. That’s just kind of a silly process. So I think there’s definitely an opportunity for the federal government, at least the state government to step in and say, “We’re going to protect our towns. We’re going to protect our cities.” And it feels like, I don’t know, it feels kind of like an easy win. Doesn’t it? We’re protecting our small towns from these stupid high bills and from kind of being taken advantage of a little bit and we’re going to protect you.
[00:18:10] JP: Oh, absolutely.
[00:18:11] SY: So it feels like a good opportunity kind of politically and a good opportunity just in terms of saving these cities and towns the cost, the time, and the headache of having to deal one by one with these different crypto firms.
[00:18:24] JP: I live in a small town and something I have learned is that I imagined that government at all levels was filled with like professional politicians, career people. In a small town, many times the government, they’re working nights and weekends. This is a second job for them or this is a volunteer position. And so a lot of these small towns, they don’t have the expertise in crypto mining environmental impacts.
[00:18:53] SY: That’s actually a good point. Yeah.
[00:18:54] JP: Yeah. They might not even have the time or attention to look at this stuff, and a lot of our small towns are really stressed. They’re volunteer governments and they need help. And I think it’s cruel to expect every single town to figure this out on their own.
[00:19:10] SY: That is such a good point. I didn’t even consider that. Right? Because at the state level, at the federal level, you probably have, as you’ve mentioned, more career politicians, more experienced politicians, probably more campaign dollars and resources and ammunition to kind of put together effective laws, put them together a little bit faster and really take a stand. So even if it was doable on a small town level, you probably just get better laws and better results if you’re able to go a step up from that.
[00:19:40] JP: Yeah.
[00:19:41] SY: Speaking of regulations, coming up next, we take a look at different AI regulations and systems that various cities around the world are implementing after this.
[00:20:13] SY: Here with us is Vidushi Marda, Senior Program Officer at Article 19. Thank you so much for joining us.
[00:20:20] VM: Thanks for having me.
[00:20:21] SY: Tell us about what Article 19 is and your role there.
[00:20:26] VM: So Article 19 is an international human rights organization that focuses on the protection of freedom of expression, particularly around the world. I work in the digital team in Article 19 that focuses on embedding and strengthening human rights considerations in technology. And I lead our work on artificial intelligence research and engagement globally. So part of that includes working in technical standardization bodies, where we want to embed stronger human rights considerations into the technical design of technology and technical systems. And a big part of my work also focuses on building evidence of what machine learning looks like in different jurisdictions around the world with a focus on non-Western context.
[00:21:12] JP: So you’ve talked about the need for ethics and AI and responsibility and its implementation. Can you give us some examples in which AI can and has gone wrong when these things aren’t considered?
[00:21:25] VM: How much time do you have?
[00:21:28] JP: Can you give us some examples where it’s gone right, I guess, would be a better question.
[00:21:32] VM: I mean, I guess just broadly speaking, I think there’s a tendency to kind of use artificial intelligence because it’s this shiny new object and shiny new technology that’s thought of as magic and then that’s thought of as this kind of silver bullet with all sorts of complex problems. And so in the enthusiasm to kind of deploy these systems, what ends up happening is that there is little consideration, if any, for the societal implications of their use and also for the surrounding regulations or legal basis for their use. So for instance, in India, where I live, the government is currently proposing to build this nationwide automated facial recognition system that will seamlessly plug all police stations along with each other to be able to track people and all the time. They’re going to be using pictures from newspapers, from police raids, from all existing government databases to feed the system. That’s also going to be used for live facial recognition, all of this in the absence of any legal basis to use it. Right? So then the question becomes like, “We’re just assuming that these systems should be used and we’re putting this really dangerous cart before the horse and we’re kind of running with the assumption that these systems are legal.” Right? And the use of facial recognition across the world has shown that not only is it not accurate, but it also risks marginalizing people who have been historically disadvantaged and singled out by authorities, whether it’s in London or New York or most places I would say. I’m struggling to think of a single good use of facial recognition, to be honest. I’ll let you know if I do think of one. And the one that I do want to talk about, which I think is particularly interesting, is in 2020, we conducted some research, along which is the EMMA that looked into the emotion recognition market in China. And what we found was that there’s this huge market of systems that have no scientific basis and definitely don’t have any kind of surrounding legal regulatory framework, but is going to be a multi-billion-dollar industry in a couple of years and kind of already is.
[00:23:40] SY: So there was a Wall Street Journal piece that recently came out that looked at a bunch of different cities from all around the world and what they were doing to regulate AI. And I want to go through some of these regulations with you just to kind of get your thoughts to see if any of them maybe sound good or sound promising. So we’ll start with Amsterdam and Helsinki. They’ve created websites people can go to that try to explain the algorithms like the city’s automated parking control and trash complaint reports. What are your thoughts on this type of implementation?
[00:24:15] VM: I think the Amsterdam and Helsinki example of like AI registers, right? It’s an interesting proposal because it’s the first time that this has happened at the level of the city. And I know there’s a lot of excitement around it, but I would have to say keeping in line with my borderline pessimism, I am very skeptical of these kinds of registers for a number of reasons. So firstly, if we look at what these registers include, right? In the Amsterdam example, they’re saying, “We’re going to catalog all of the AI systems that the municipality uses and we’re going to let you know what data we use and how we tested for bias and things like that.” Right? So the first issue with that is that transparency is not an end in and of itself. So just because I know something is being used is very different from me having a meaningful way of engaging with bad outcomes or confusing outcomes that come from that system. Right? The second is that these registers almost have like what I like to call like a legitimizing force for artificial intelligence systems because the implicit assumption is once we have an AI system on this register, we are fulfilling our part of being accountable and transparent. And so this is a legitimate use because we’re telling you how we check for bias, right? But as the general question for me, especially from like a human right’s lens and a civil liberty’s lens is to say, “Can we please question the need for the systems in the first place? Do we need it? Can we establish some legality? Can we demonstrate the need for it before we start thinking about how we’re going to gestate and how we’re going to be transparent about its use? Can we ask the much tougher question of how can we legally justify its use?” Right? Especially if we’re thinking about technologies that do have a potentially rights infringing impact on societies, we do have to demonstrate through different legal standards depending on where you are in the world that this is a necessary and proportionate use of technology. So that’s another way in which I think the registers kind of like distract from the real conversation. Right? So going back to the emotion recognition example I gave, let’s assume this hypothetical city uses like 10 emotion recognition systems for different things and two predictive policing systems. Right? And they are super diligent about having all of these details on the register. No amount of details can make these users legitimate from a human rights perspective. And so the registers kind of take us away from the real questions that we should be asking. The other thing with the registers is also we focus on what systems the government is deploying. But in reality, if you look at smart cities around the world, or if you look at how fancy AI technologies make their way into societies, they do so either through the route of like trials and pilots, right? All companies that say, “We’re going to provide this to you for free, and we’re not going to charge you, and you don’t have to go through like a painful procurement process because you’re not giving us any money, but we’re going to like trial this use.” Right? We saw this happen a lot after COVID hit different areas of the world as well. And none of these registers will be able to catch those, right? Because it’s not the government deploying it. It’s companies deploying it on behalf of governments where the government has now become the consumer almost, and not the implementer of technology. So when we look at like the real world use cases of technologies, especially those around like hiring and law enforcement and stuff like that, the mechanism of the registers isn’t really going to catch it. I mean, for folks who are interested in this, I would also suggest reading this excellent paper by Dr. Corinne Cath and Fieke Jansen where they look into the Amsterdam case in particular.
[00:27:59] SY: Yeah. We’ll put that paper in our show notes.
[00:28:02] JP: What do you think about the idea that some have floated around making AI algorithms completely transparent for public scrutiny?
[00:28:10] VM: So AI transparency is useful and that it’s good to know when we’re being subjected to a machine and not a human right on the other end, but it’s not a solution to some of the more complex questions we have for a number of reasons. Right? So you can be super transparent about the algorithm itself, but when it comes to machine learning systems, the algorithm doesn’t mean much, the source code doesn’t mean much if I don’t also have access to, say, the database and I don’t have access to different logics that we use, I don’t have access to the weights that you use or the internal metrics that we use in order to decide whether this was worth deploying or not as the case may be. But more importantly, I think the reason that I’m skeptical of transparency of the algorithm itself is it doesn’t actually solve the problem of concentration of power. Right? So let’s assume that you’re using a machine learning system for hiring and someone didn’t make it through the first round of applications because they smiled with me, right, or their voice tone was different from what the algorithm was starting to think of as promising in a candidate, it doesn’t actually help someone who’s trying to get a job to be able to scrutinize an algorithm. Right? If we’re really trying to level the playing field by way of being more transparent, then I think the power cannot be concentrated in the hands of a few people who have the ability and skill to be able to audit these systems. Right? It’s not a structural solution at the level of the individual at least.
[00:29:40] SY: What impact do you think making algorithms more transparent and available would have on AI research, especially ethics in AI research? Do you think there’s a benefit to making it transparent in terms of how it impacts the way research is done or research is thought of?
[00:30:01] VM: The value of making algorithms transparent not at the level of the individual, but just in terms of, like you said, like research and also the level of institutions is that it can then be open up a third party audits or it could be used to kind of query an algorithm to kind of reverse engineer the outputs that the particular system is giving us in a given context. It is not a one-stop shop because you can have a perfectly transparent algorithm. But if the premise of the algorithm is problematic, then transparency almost acts like a distraction. It’s a bit like transparency washing almost, right? To say that, “Oh, they’re using a predictive policing system,” but we’re super transparent about the fact that we’re racist.
[00:30:41] SY: So it’s cool.
[00:30:43] VM: That’s potentially very worrying. And also I think conversations around transparency often lead to conversations around like impact assessments and stuff like that. Right? The problem with all of those is that we assume that we know all the bad possible outcomes at the beginning. Right? So we’re going to be transparent about the fact that we’ve checked for this bias and that bias.
[00:31:06] SY: Right.
[00:31:07] VM: But if all of the examples that we’ve collected over the years, like even if you look at the compass algorithm, they didn’t think to check it for racial bias, which in hindsight seems like painfully obvious to us, but that isn’t generally always indicative of thinking through all possible bad outcomes in practice.
[00:31:26] JP: Let’s look at another municipality. Santa Clara County made it mandatory for any law enforcement agency that wants to use surveillance technology to have to submit a request for public input and detail how and why it will be used. What do you think about cities polling for public input and public oversight?
[00:31:47] VM: I think it’s fantastic, to be honest.
[00:31:49] SY: Yay! We got a good one!
[00:31:49] VM: This is where my optimism comes in.
[00:31:51] JP: Yeah!
[00:31:55] VM: I mean, I'm sure that we could also improve that. But just in terms of how we position power and how we position different decision-making processes that lead to a particular system being used, I think having the community actually have a meaningful voice in deciding whether something gets used or not is a breath of fresh air, right? Because often what happens in a lot of jurisdictions is that like the government has decided that they’re going to use facial recognition and predictive policing and whatnot. And then they say, “Okay, we’re going to open up for public comments for a two-month period and you can send us research papers and stuff like that.” And at the end, the government ends up doing what they wanted to do anyway because there’s no onus on state authorities to say, “Well, we took into account everything that we heard from the public and so we’ve changed our position for this reason or we decided to not change our position for that reason.” Right? So I think the logic of allowing the community to decide what is a legitimate use or not, and not just like decide, but also putting the onus on authorities who want to use certain artificial intelligence systems to demonstrate why it’s necessary and to demonstrate that they are capable and worthy of trust and responsible use is actually the best way to go about it. Because, again, I think if we take a step back and if we let go of this idea that the use of AI is good and we should think of AI as this efficient solution to governance issues and if we think of it as just another governance mechanism that is being proposed, the first instinct of anybody who’s like dealing with the law or dealing with governance in general would be to say what is the legal basis for it, like why and how do we have the authority to be able to use the system, because it has such profound societal implications. So I think what Santa Clara County is trying to do is get closer to that kind of model, which I welcome.
[00:33:55] SY: So Amsterdam, Barcelona and London, they’ve worked together to create something called the Global Observatory of Urban AI. And the goal is to educate other cities on five principles of AI implementation they’ve all agreed on based on fairness and non-discrimination, transparency and openness, safety and cybersecurity, privacy protection and sustainability. So I’m wondering if you’re familiar with these cities’ AI implementation and regulations and what do you think about this idea of the Global Observatory of Urban AI and these principles.
[00:34:32] VM: I think when you say those principles, especially when you list them out, they sound like heaven. They sound like almost perfect, like we want to be fair and non-stone age thinking and transparent and sustainable and things like that. I think it’s well intentioned. Right? But I think it’s ultimately not practically useful because I’ve had the experience of like talking to engineers like way back, I say way back, but I mean like six years ago, our technical standardizations with this.
[00:34:58] SY: Way back in tech. Yeah.
[00:35:00] JP: That is a way back. Yeah.
[00:35:01] VM: Back when we were talking about big data, not AI. Right?
[00:35:03] SY: Right. Right.
[00:35:03] VM: And we were talking about like fair, big data, and this engineer just like me looked me dead in the eye and he said, “What does fair mean? Can you tell me something that I can actually encode instead of using a word that means different things to different people?” Right? If you ask various people what transparency means, it has possibly like a million different permutations and combinations that would kind of qualify as being transparent. So I think the issue with ethical principles in isolation is that they seem really great, but then they don’t help in clarifying different positions and they don’t help in clarifying what we’re actually aiming for. The metaphor that I often use is like, if you think of a room, right? The ethical principles are like this strange ceiling and we don’t know where it ends and we don’t know where it begins. We know that we’re aiming high and we know that the intention is good and things like that, but we’re not actually clear as to how are we going to get there. Right? And we’re not clear as to what the actual standard is, but that’s where I think the importance of like regulations and laws and existing laws, right? Like constitutional law and consumer protection and data protection. All of that comes in as like the minimum standard. Right? That’s like the floor of the room. So it gives us like a firm footing and we know where we’re standing and we know what we’re talking about and then we can think about technical standards and stuff like that, helping us get to where we are. Right? But when cities just talk about ethical standards and they talk about like, “We want to share this knowledge and have best practices and things like that.” My experience has been that, firstly, best practices is like corporate speaking. So it’s also like subject to being co-opted a little bit by the people in the room who are often like representatives of companies and not necessarily civil society folk. I mean, I’m saying just globally, right? This is obviously not the case in every city. And secondly, the other question is how are we regulating it? Right? What is the extent to which existing laws can help us understand what are legitimate and illegitimate users? I think that’s the more important question, like ethical principles are fine, they’re good, but the danger is if we think of them as an end in and of themselves.
[00:37:13] SY: Is there a way that principles can be effective? I’m wondering if there is, because I’ve heard of these AI principles before in different cities, in different cases and they, like you said, they sound really great, but I don’t really know what they mean and I don’t know what you mean by these words. It’s not always clear the connection between the principle and the implementation of it. So have you seen a situation where the principles have actually come in handy and been useful? Or do you feel like as long as there’s no law following it, it’s kind of just for show?
[00:37:55] VM: I’ve had the experience of working on like ethical principles at different levels. Right? So whether it’s at a technical standardization body, like the Institute for Electrical and Electronics Engineers, which is mostly industry driven. It’s US-based, but of course it’s popular because it’s come up with very important standards, like Wi-Fi and stuff like that.
[00:38:14] SY: Right. Right.
[00:38:14] VM: And when we were talking about ethical principles there, the argument that Article 19 made, again, way back in 2016, was that we should ground ethical principles in legal standards. Right?
[00:38:27] SY: Right. Right.
[00:38:27] VM: And if you think of human rights as a legal and ethical standard, then you have the opportunity where not only are you aiming high, but you’re also telling people what you actually mean because there are actual legal standards, there are tests, there’s like years and years of jurisprudence and judgments that help us understand what we mean when we say like necessary and proportionate and like freedom of expression and privacy and stuff like that. So I think principles can be helpful when they are anchored to any kind of legal system that has a shared understanding between any two different entities because there is a common point of reference. I think also when that companies kind of like signal like, “These are our principles,” there’s often, I mean, this is the definition of looking for a needle in a haystack, but when a company publicly states that it has X number of ethical principles that it wants to adhere to, that becomes the standard by which we judge the company externally. So it’s almost like showing the company why they’re not reaching the ethical standard that they claim to or why they are or whatever. This is like my human rights lawyer brain working in terms of like how to use it for accountability mechanisms. But I think that the short answer would be that principles are useful when there’s law and they’re dangerous in isolation.
[00:39:47] JP: We’ve talked a lot about what some of these cities and municipalities are doing and how they’re maybe not doing the best job. What would your ideal AI regulation or guidance look like? And are there any companies or especially cities that are close to that?
[00:40:06] VM: We’ve actually been thinking a lot about this in different contexts, right? Whether it’s in the European Union that’s currently looking at an AI act and is going through various iterations of a proposed regulation or looking at places like India and Myanmar where there are barely any regulations, like not even data protection regulation. And I think the first hallmark of like effective and critical AI regulation would be to have legitimate and illegitimate users at the outset because there is the tendency to ascribe value to AI because of its very nature is so high that we need like a sobering reality of, “These are legitimate users and these are not.” So if you’re going to use artificial intelligence to predict the future and make consequential decisions based on that, that should logically tell us that that is not the smartest thing to do. So kind of like having clear lines as to, “These are legitimate users and these are not,” I think is the first step and deciding what the bright lines for different contexts are is helpful as well. I mean, there are facial recognition bans in various parts of the world. Right? And I think that’s a good first step because we’re clearly establishing bright lines for what we will and will not stand for. The way I like to say it is we push against this inevitability of AI as something that will happen because it’s not inevitable. AI technologies become widespread because there is like market forces and governments who are keen to use this technology and create demand for it. But if we take a step back and say, “Actually, we’re not going to ask for this technology to be thought of as lawful in the first place.” I think that’s a good first step. The second is I think grounding uses in legal basis before we invest time, money, attention, energy into them is also helpful. Only legitimizing technologies once we can legally argue for their uses. It seems like a pretty straightforward task, but it is staggering how rarely that is actually done. And I think if you look at existing regulations, like the EU AI act for instance, having some kind of grasp over the technical limitations of systems is really important. For instance, in the EU act, emotion recognition is looked at as limited risk. If you actually look at what these systems claim to be doing, they should be completely disallow any fundamental rights basis, but they’re not. So also having that kind of shared language would be really important. And finally, I think that a risk based approach to say like these are high risk systems and loader systems are fine, but we also need to have like a rights-based approach where we put the individuals and individual harm ahead of like company and just some things like that, especially when we’re thinking about like regulation and safeguards.
[00:43:02] JP: So there’s been a lot of research and coverage of some of the fundamental problems with the biases that are baked into the development of AI and the datasets that they’re trained on. Do you think that regulation or more scrutiny about the development of AI is helpful? Or do you think it’s more important to focus on the use and application as we’ve been talking about so far?
[00:43:28] VM: I think regulation actually has to look into design development and deployment also because I don’t think that there is such a thing as like unbiased datasets, right? There’s always going to be bias depending on who you are. And I think the dangerous tendency that I’m seeing across different jurisdictions is like this idea of like de-biasing, which is going to de-bias this dataset, right? Like that’s not a thing. Technically, socially it’s not possible. And I think for us to think that like regulation will fix that problem is actually quite harmful because, again, much like the registers were distracting from other problems. So for instance, I think a lot of people talk about like facial recognition accuracy, right? And so they say like, “This has a tendency of misrecognizing black women.” And so the solution isn’t to make it less biased so it’s better at recognizing black women. The solution is to think about the social political institutional context within which these systems are used because the bias is not just built into the algorithm, but also in the way that it’s used and the institutions that use it. Right? So I think if regulation has to look at bias, it has to look at bias throughout the life cycle of these algorithms from the time that they conceptualize to the time that they use, to the time that they continue to be used. And I think the question of like de-biasing is something that we need to be super critical of.
[00:44:53] SY: Well, thank you so much for joining us.
[00:44:54] VM: Thank you. Thanks for having me.
[00:45:06] SY: Thank you for listening to DevNews. This show is produced and mixed by Levi Sharpe. Editorial oversight is provided by Peter Frank, Ben Halpern, and Jess Lee. Our theme music is by Dan Powell. If you have any questions or comments, dial into our Google Voice at +1 (929) 500-1513 or email us at [email protected] Please rate and subscribe to this show wherever you get your podcasts.