Robot as Threat

Are you still struggling to keep those pesky pieces of paper together? No more, my friend. Introducing the Paperclip Maximizer Bot 3000, a robot whose sole purpose is to produce as many paperclips as possible. The future of office supplies has never been so bright.

We're interrupting this broadcast to bring you updates on the catastrophe playing out downtown. It looks like the Paperclip Maximizer has torn apart most of the city's buildings. It's repurposing them into piles of … well … paperclips. I'm told the company's founders ... [voice trails off]

When it comes to robots, even the most innocent of intentions can go awry. They obey the letter of the law, but not the spirit. A Roomba might try to vacuum up your cat, for example. Making sure robots don't cause harm has become a crucial field of research, and figuring out who is responsible for what as robots become more a part of our lives is more difficult than you might imagine. When a machine has some measure of autonomy, like a lot of robots do, is the manufacturer responsible for its actions? Is the user? Could a robot be held responsible?

I'm Saron Yitbarek and this is Command Line Heroes, an original podcast from Red Hat. All season, we've been tracking the fast-evolving field of robotics. And this time we're asking, what happens when good robots go bad? Who's responsible for their actions and who do we blame when a Paperclip Maximizer Bot 3000 decides to destroy the city?

We'll come back to that disaster scenario, an interesting thought experiment by philosopher Nick Bostrom. But first we need to grapple with some immediate worries because questions about robotic responsibility are already here—and the stakes are high.

Paperclips… More paperclips.

So, who is responsible when robots do harmful things? If I cut my hand while preparing dinner, I'm not going to blame the company that made my knife. But robots are different. Sometimes robots have a degree of autonomy. Sometimes their inherent wiring controls their decisions—and that means responsibility in the world of robotics is a lot more confusing. Our search for robot responsibility begins with the folks who make them, the manufacturer. What responsibility might they bear even after the robot's been sold?

No, one's really teaching me what I'm supposed to do, what I can or cannot build using the powers I have.

Ajung Moon is an assistant professor at McGill University. She studies the ethical consequences of AI and robotics. Moon says her students are engaged by these questions in a way that previous generations might not have been. They're pushing to understand the multifaceted responsibilities of this field, and the role manufacturers play when designing robots is especially blurry.

They don't have a lot of legal responsibilities to make sure that everything that they put out the door is used for ethical reasons and purposes.

Fact is, it's almost impossible to know how somebody will use a robot once it's sold. People constantly come up with innovative ways to use robots. The New York Police Department, for example, purchased robotic dogs and repurposed them to help with police work. They didn't exactly weaponize their robots, but they did use them on patrols and in some perceived dangerous situations. That got a lot of people anxious. New uses for technologies are often positive. They can move things forward. But manufacturers that are wary of unplanned uses could revisit their user agreements.

The fact that these machines can make certain “decisions” or behave in a particular way in contexts that the designer hasn't necessarily hard-coded into the system or has thought through fully—that allows for a little bit of uncertainty to be built into how users interact with the system.

A user agreement might include a promise that you won't use the robot to harm a human or won't allow the robot to be easily hacked. Both those things are easier said than done. And the more powerful the robot, the more specific that contract needs to get.

For example, back in 2014, ClearPath Robotics released a statement saying we are building these field robots that can be used underwater, above ground, and so forth. And it has a lot of military use and it continues to have military clients. But they've recognized that retrofitting these systems to become “killer” robots or robots that are weaponized is not good for society. So they've essentially made it so that it is their responsibility to communicate that their clientele would not be using the technologies for those particular killer robots purposes.

Moon has recently been purchasing robots for her lab, so she's seen a few user agreements lately. ClearPath's contractual language about ethical boundaries was some of the most direct out there. Why the use of such strong language for their robots?

I'm Ryan Gariepy, and I am the CTO and co-founder of ClearPath Robotics and Auto Motors.

ClearPath created the Husky robot, an all-terrain four wheeler about the size of a dog. They've made other animal-named robots for research too: a dingo, a jackal, a moose. And they work with some heavy duty robotics companies, places like Boston Dynamics and Universal Robots. So Ryan Gariepy knows the field. He sees his robots used by new startups and corporate innovation programs.

A seafaring robot patrols the ocean, looking for dangerous algae blooms. Another might be used to haul material at a mining site. And the scope of all that work means Gariepy can't know exactly how his robots are going to be used.

We have to do a degree of know-your-customer research because these robots can be repurposed for some fairly harmful roles if they do get into the wrong hands.

Partly, that connection with customers is about screening, making sure customers have the right intentions. But it's also about education.

Robotics might be the technology that has the widest gap in the real world between perception and reality. We don't want to release technology to people who are going to hurt themselves with it.

Gariepy says that as a manufacturer, he has three areas of responsibility. There are legal responsibilities for sure—things like export control. And then there's a responsibility to his customers to make sure they're set up to succeed. What's really interesting is this third area of responsibility.

I think we do have a responsibility to society, and that's where we've engaged with governments in the past. That's where you'll find that we've been very outspoken on the need to regulate some more extreme uses such as lethal autonomous weapons systems. So we do have these areas because in the end, a manufacturer can only control the technology that they developed so far. So it's important to engage with the rest of society and inform people about where they should be worried and where they shouldn't be worried.

Ultimately, the manufacturer can educate the public, can push for good policy, but the power to manage a sold robot is going to be limited.

If you find out that someone is using a product that you have sold them and that they have title to, legally, there's nothing that you can do. And it doesn't make any sense to claim that you can do things that legally you cannot. If you've sold them an asset, it's theirs. That's the current laws that we operate within.

And yet Gariepy points out that as a manufacturer, you're not totally powerless.

However, if they did mislead you about the purpose of their assets, then you are of course free to withdraw support.

And that's something I hadn't thought about. One thing we've learned about this season is that the robot revolution is in large part a software revolution. And that means somebody who just buys a robot is in the long run going to be tied in a way to software providers. And that makes it a little more difficult for anybody to go rogue.

Folks will always find ways to hack, to modify, to repurpose. It's a core part of technology's history. And Ajung Moon says those modifications will keep happening with robots, too. But that doesn't mean we give up on creating safe systems.

It's really interesting to discover new ways that these robots and AI systems are impacting our lives, our behaviors, our decisions. But that should be coupled with more directed decisions on what are some of the harms that can result from these technologies being deployed and how do we prevent that.

To really prevent harm, it's not enough to have responsible manufacturers. The users are being tasked as well. And that makes the question of robot responsibility even more complicated.

Hello from the hackers.

Stefanie Tellex is an associate professor at Brown University. Her lab makes autonomous collaborative robots, and she's also very interested in robot vulnerability. One day, a couple of years ago, Tellex learned something that intrigued her.

I learned that it was possible to scan the whole internet for a particular port.

That seemed incredibly important to her because, as we described earlier in the season, there's a standard operating system for robotics called ROS, the robot operating system. And ROS uses a particular port, number 11311. What all this means is that Tellex could search the internet for any ROS users who weren't protecting their system.

And I know from my own personal experience that we are not being particularly careful to secure our robots. And if we can scan the whole internet, we just have to do it as soon as we can, because it's going to be awesome.

So they scanned the internet for ROS ports. They found a lot of them. And what's key here is they found folks using ROS with sort of a wide-open door. There were dozens and dozens of people running ROS with no firewall at all. They found a DaVinci robot, for example. This is the kind they use for surgeries. It had a ROS interface and essentially no security.

We didn't actually try to move the robots. We didn't actually read any sensor data off of them because we felt like that was too far to be ethical.

But they could have…

We could have, and a bad actor could have, we think.

Turns out, one of the robots running ROS that Tellex found was owned by a colleague of hers.

We found your robot. We'd like to do a proof-of-concept attack on it.

So they agreed on a time.

And we're going to try to read the sensor data and we're going to try to move the robot—and then we did. So we actually were able to read sensor data, read the camera. We were able to make it speak. We played sound out of the speakers. So this is just to show that if you're running ROS and you haven't secured the port, anybody on the internet can publish and subscribe, and they can read your sensors and they can move your actuators.

Tellex finds that researchers are especially careless with their robot security. In fact, she even found one of her own robots hanging out on the internet, unprotected. Her lab had opened a ROS port and just left it open by accident. Tellex feels that running these scans and closing off attack vectors is going to become a regular part of good robot maintenance.

I really want to find my buddy's robots at other universities and give them crap about how they're insecure. That seemed really fun to me. Maybe I'm a bad person, but I wanted to do that. And I also thought, "Wow, this is something that I didn't know. I bet a lot of other roboticists don't know it as well."

Tellex says that everybody using internet-connected robots should be doing a certain amount of basic security maintenance. Whether we're using robots for research or managing robots at a factory, we each have our own share of responsibility. And that, it turns out, is by design.

ROS, the robot operating system, is primarily maintained by Open Robotics. But as Tellex discovered, they count on users to secure robots at the network level. Brian Gerkey, the co-founder of Open Robotics, explains why.

We made the decision early on that we're not security experts. We didn't want to invent some security mechanism and get it wrong. We didn't want to make the wrong guesses about what the threat models are. And so we made it very clear from the beginning, we're building a system that can connect to a network and we didn't build any security in. And so what you need to do as a user is take appropriate precautions and use modern networking tools to wrap this system so that it's not exposed on a network where it can have threats.

Truth is, hackers are always going to be there. Some can mess around with industrial robots already, and sometimes they can hack into the microphones and cameras of robots in your home. Any robot that's connected to the internet is potentially vulnerable, but there are precautions that users can take. Remember the PR2, one of the robots we talked about earlier in the season? It runs on ROS and Gerkey who worked with PR2 says ...

That robot, the only way to connect to it was through a VPN or virtual private network. And we put that out there as an example of how to deploy a robot that runs ROS. And I'd say that if you look at deployments of robots from the many, many companies who are using ROS when in production, they're following a very similar model.

Open Robotics has since created ROS2 where they're trying to embed some security into a distributed system. But users still have to configure things to their own needs and take on an active role.

With ROS2, you can actually enable security at the application level. We would still advise you not to expose your ROS2-based robot directly to the internet. I mean, frankly, that's bad practice for any device anywhere. Basically, no device should be directly exposed to the internet without incredibly high levels of security applied, which are generally not applied to most devices.

Gerkey is aware of Stefanie Tellex's work, exposing those vulnerable ROS users, and he's not totally surprised by it.

Back in the day when we had the PR2 robots running around, one of our interns showed up to the office one day—I think either very early or on a weekend, and didn't have his key with him. And he was a very creative guy, and he got on his laptop, connected to the wifi network in the building, which required credentials to access it. It wasn't open to the public. But then used that connection to get onto one of the robots and then drive the robot over to the door and push the door from the inside and let him in. So it was actually a robot mediated to break in.

That example of course, is a bit like somebody taking over your Roomba. Not too big a worry. But what if the same kind of hack allowed someone to take over a car or a piece of heavy machinery at a factory? Suddenly you've got a major problem. We've all heard stories about hackers getting into IOT devices, but the difference with internet connected robots is that they're moving. They're manipulating the world—and that instantly becomes more serious.

There's certainly a higher bar whenever you deploy something out in the world.

And to meet that higher bar, we're going to explore one final level of responsibility. We've already looked at the manufacturer's role and the user's role. But what about the robot itself? Can we hold robots responsible for their own actions?

Science fiction authors have spun up robot disastrous scenarios ever since the word robot was first coined. There have been fantasies of SkyNet and a laundry list of robot rebellions. But it was a philosopher called Nick Bostrom who described a subtler and maybe more likely disaster. Bostrom suggested that a super intelligent, goal-oriented robot would consume everything in sight, including the planet's resources in order to accomplish its one goal.

It goes about turning the entire planet into paperclips, including all of the human beings and everything that we hold dear.

Brian Christian is telling us about this thought experiment. He's an author and researcher at UC Berkeley and has become a leading thinker on the future of tech and how it impacts humanity.

It offers us a different vision than the SkyNet Terminator vision. In this case, it is not necessarily that the AI takes on a set of goals that are contrary to our own, but rather it's trying in earnest, in good faith, to do exactly what we asked it to do.

Hmm. So how do we avoid getting turned into paperclips? It's a harder problem than you might think. If we give a machine a goal and then let it run on its own, we've got to be absolutely positive that goal doesn't interfere with other goals like human safety. In other words, how do we give robots a sense of responsibility toward not just a narrow goal, but the larger interest of humanity? How do we make sure that our robots are pursuing the things we truly desire? We might try the machine learning route, giving robots fewer explicit instructions, letting them learn through countless examples all on their own. But that has its problems, too.

It turns out to be extremely difficult to make sure that the system is learning exactly the thing that you have in mind when you taught it and not something else.

So what's the solution? Christian suggests that we could avoid the Paperclip Maximizer robot by moving our thinking beyond the single-mindedness of corporate goals and thinking about the goals of multiple disciplines at once.

I think it's really going to take a pretty holistic approach to solving this problem. There has been an increasing awareness dawning on people in the computer science and machine learning community, that they really need to address these problems in dialogue with people in other fields—people who have disciplinary expertise, whether it's doctors in the medical context or people with a criminal justice expertise. And a lot of these problems I think exist in the boundaries between those disciplines.