Suppose we were told that a fleet of spaceships with highly intelligent aliens has been spotted, heading for Earth, and they will be here in a few years. Suppose we were told these aliens might solve climate change and cure cancer, but they might also enslave or even exterminate us. How would we react to such news? – Yuval Noah Harari

It’s clear that AI can help us solve some of humanity’s biggest challenges in the next few years. It could help us eradicate disease, reduce material poverty, provide free education, and lead to a truly utopian society. But at the same time, if it also causes our financial systems to collapse, our democracies to perish, and results in humans losing control over our world, what is the point of those benefits?

Instead of another vague description of how risky this technology is, this essay aims to provide a clear explanation of exactly how AI can lead to a catastrophic loss of control of our society within the next few years. It describes the risks that matter right now, and why these risks are more harmful and likely than most people realize. As we are building this technology at an accelerating pace, it is important to understand what world we could be heading towards if we do not take care to mitigate its risks.

Half of AI researchers now believe that there is a 10% chance that this technology will lead to the extinction of humanity. This is not due to evil robots rising up to kill us, but due to very specific scenarios occurring within the next decade if we continue developing AI at our current pace:

1. Economic collapse in major sectors
2. Democratic failures in developed nations
3. Releases of new pandemics worse than COVID-19

This essay explains how AI makes these likely, and what we are doing about these risks today.

I am writing this because, despite recent regulations, AI labs are continuing to race forward in the pursuit of even more capable and dangerous AI systems. We are far from safeguarding humans from these risks — we need more public understanding of these issues in order to actually make policymakers act.

If you were boarding a plane, but were told by the engineers that there was a 10% chance that it would crash and kill everyone on board, would you get on? Humanity is boarding that plane right now.

What is intelligence?

First, it’s important to understand what AI really is, and how it is different from anything else humans have ever created.

The following 3 things all have something in common:

1. A human playing chess
2. A dog jumping through a hoop
3. A Roomba autonomously vacuuming your floor

They are all exercising intelligence. What is it specifically about these things that makes them intelligent? Is it their ability to move? Their ability to learn things? To remember things? There is no widely accepted definition for intelligence — but for our purposes, it is simply the ability to solve problems.

Humans, animals, and even some non-living things can solve problems. Some things can solve more problems than others — a human can solve a lot more problems than a dog, so we consider humans to be more intelligent than dogs.

Artificial intelligence (AI) is anything that can solve problems, but is not biologically alive. For most of our planet’s history, all intelligence has been restricted to biological life. There was no non-living thing that could ever solve problems. But 70 years ago, this started to change when humans started using computer technology to build systems that could solve problems on their own — the field of AI was born. Since then, we have been able to build forms of non-biological intelligence that can recognize images, play chess, and even drive cars on public roads.

Most systems we build are only intelligent in a narrow way, which means they can only solve a very small set of problems. If we try to use these systems to solve other kinds of problems, they will not be able to do so. For example, a chess-playing AI would not be able to go to the grocery store and pick out ingredients for a butter chicken recipe.

Note: we shouldn’t confuse intelligence with sentience or consciousness, which is the ability to feel, think, and experience the world. Something can be intelligent without feeling or experiencing anything, such as the AI systems we are building. It might be true that for animals, intelligence goes hand-in-hand with sentience, but this does not mean that all intelligent systems are sentient.

For about 65 years since the field of AI began, everything we created was narrow intelligence. But 6 years ago, a powerful new innovation started to blow away our restrictions for building smarter AI systems.

General Intelligence

On June 12, 2017, a team of researchers from Google Brain released a short paper, claiming that they had discovered a new way for AI to translate languages. Little did they know, they had stumbled upon something which would change the field of AI forever, and alter the course of human history.

The breakthrough achieved by this lab meant that humans could now start building systems that solved a huge variety of problems and were not so narrow in their intelligence.

The most famous and recent example of this is ChatGPT, an AI model that can respond to questions, generate code, write essays, solve mathematical problems, give you life advice, recognize objects in images, and perform many other tasks. ChatGPT was not built specifically to do any of these things — when OpenAI first started building this system, they wanted to build an AI that could replicate human language perfectly. But in the process of doing this, they quickly realized that this system, as it grew, was able to do many other things that they did not expect.

ChatGPT (GPT-4) can perform the following tasks better than many humans alive today:

• Writing code
• Composing music
• Solving mathematical problems
• Reasoning through complex logical situations

These surprising new capabilities have led many people to believe that GPT-4 is one of the first generally intelligent systems that humans have built.

But can we say it is approaching the intelligence of humans? How do you even compare intelligence between different things? We discussed earlier about how humans are more intelligent than dogs because they can solve more problems than dogs. Let’s try to apply this here — if system A can solve more problems than system B, then we can say system A is more intelligent than system B. We can represent this visually:

The orange circle represents all the problems that humans can solve, and the blue circle represents all the problems that dogs can solve. We know that humans can solve a lot more problems than dogs can, but there are still some problems that only dogs can solve, like sniffing for cocaine in airport luggage; this is why the dog circle covers some area that the humans circle does not.

Now, if we were to put our most advanced AI system onto this diagram, what would it look like?

As we mentioned before, GPT-4 is able to perform an incredible range of tasks that can reasonably compare it to a human for many things. Obviously, there are still many things it cannot do, like play soccer, make a sandwich, or care for an elderly person. These are mainly the result of this model not being connected to any physical systems, but this is also starting to change. It’s feasible that these restrictions could be eliminated within the next few years. You could make an argument that this system‘s intelligence looks like this today:

This is an important checkpoint. Today, we have already built an AI system that can compare to the abilities of some of our pets, and in many cases, the abilities of humans. What could our most advanced AI system look like in 5 years? What about 25?

Superhuman intelligence

Biological intelligence has evolved for billions of years to reach its current level of problem-solving abilities:

How does this compare to the growth of artificial intelligence?

In about 70 years, artificial intelligence has reached about 30–50% of the “intelligence level” that it took biological intelligence about 4 billion years to achieve.

Side note: these forms of intelligence are obviously quite different, and the problems they are optimized to solve have been shaped by different things, but despite these differences, their intelligence level is quickly converging.

What are the odds that artificial intelligence will overtake human intelligence in the next 70 years?

OpenAI, the company that created ChatGPT, recently stated in a blog post:

Given the picture as we see it now, it’s conceivable that within the next ten years, AI systems will exceed expert skill level in most domains, and carry out as much productive activity as one of today’s largest corporations.

According to OpenAI, the intelligence level of AI could look something like this in 10 years:

By the 2028 presidential election, humans may no longer be the most intelligent beings on the planet.

An independent professor at UC Berkeley created his own projection of what GPT could look like in 2030, and found the following results:

• GPT2030 will likely be superhuman at various specific tasks, including coding, hacking, and math, and potentially protein engineering
• GPT2030 can “work” and “think” quickly … estimate it will be 5x as fast as humans as measured by words processed per minute
• GPT2030 will be trained on [formats] beyond text and images, possibly including [data] such as molecular structures, network traffic, low-level machine code, astronomical images, and brain scans. It may therefore possess a strong intuitive grasp of domains where [humans] have limited experience, including forming concepts that [humans] do not have

OpenAI truly believes that they will soon be able to create a superhuman intelligence — this is shown by the fact that they have explicitly accounted for this possibility in the financial documents they present to investors:

“Somewhere in the restructuring documents is a clause to the effect that, if the company does manage to create artificial general intelligence, all financial arrangements will be reconsidered. After all, it will be a new world from that point on. Humanity will have an alien partner that can do much of what we do, only better. So previous arrangements might effectively be kaput.”

What OpenAI Really Wants

An important landmark will be the point when AI can start improving itself — when this happens, humans will no longer have control over how quickly it grows. If an AI system becomes more intelligent than us, it can start improving itself much quicker than any human could, and its intelligence will start to increase exponentially.

“AI power will grow steadily until one AI system reaches the threshold of self-improvement, at which point it will quickly outperform others by many orders of magnitude.”

Alexey Turchin & David Denkenberger

The point where we reach superhuman intelligence could happen in 5 years, or it could happen in 25. But the key point is, it is almost certainly going to occur within our lifetimes. You will be alive when humans are no longer the most intelligent beings on the planet.

So, is this a good thing? Is it a bad thing? What does it mean for us?

Types of Impact

Superhuman intelligence can be used to fulfill utopian dreams — curing cancer, addressing climate change, preventing war, and many other things that were previously limited to science fiction.

But a superhuman intelligence also makes many other things possible — it might not actually be the best idea to bring everything that was once limited to science fiction to life, as that could lead to some destructive outcomes. Specifically, there are some catastrophic outcomes that this technology can help bring about.

As we mentioned before, if we knew that what we were building had even a 10% chance of driving humanity to extinction, how differently would we approach it? These catastrophic outcomes are what I want to explain in this essay, because understanding these clearly is critical to making good decisions on what we should be doing next.

These “bad” outcomes can roughly be placed into two categories, and they form the organization of this essay.

The first category focuses on AI being misaligned with human goals. We are building a superhuman intelligence, and soon, we will let it make decisions on its own. What are the impacts of letting this technology act autonomously? Would we still be in control of it, despite it being smarter than us? How do we make sure it doesn’t do something that harms humans as it acts out in the world? These questions have led researchers at major AI labs to resign and sound the alarm about our future.

The second category focuses on societal risks. Humans use technology to do things they are already doing, but do them better and faster. This includes the work we do in our jobs, the way we communicate with other humans, and how we conduct war between nations. What does it mean to integrate a superhuman intelligence into all these systems? Should we do so?

Lastly, the essay ends with a section describing what we are doing today about these risks. Are we taking them as seriously as we should? What can be done, if anything, to prevent these scenarios from occurring?

Note: this essay focuses on the most important risks this technology poses, rather than every single risk. Currently, the discussion about this is too scattered and not focused on the real dangers. Instead of worrying about how AI can help students do their homework, we need to have a discussion about its ability to disrupt the organization of our society — if we have any chance of preventing these scenarios from occurring.

Risk 1: The Misalignment Problem

On May 1, 2023, Geoffrey Hinton, “the godfather of AI”, and VP of engineering at Google, resigned.

Part of me regrets all of my life’s work. I’m sounding the alarm, saying we have to worry about this…we’re all in the same boat with respect to this existential threat.

What was he worried about?

We are creating a form of intelligence which is more powerful than humans, but also fundamentally different from humans. When humans use their intelligence to solve problems, they do so while being influenced by their needs, desires, emotions, and past experiences — we can call this the human context.

Our human context includes:

• The desire to live
• The desire to have shelter and food
• The desire to have family, friends, and community

It also includes some things that are unique to each individual:

• Wanting to avoid milk because we are lactose intolerant
• Wanting to avoid a particular person because they were rude to us earlier
• Wanting to get a promotion at work to be able to afford the new iPhone

This context is taken into account whenever we solve problems — we don’t simply solve them in isolation. If we are late to work and need to get to the office as fast as possible, we might have the following options:

1. Drive over the speed limit and ignore some stop signs along the way
2. Steal the neighbor’s motorcycle to go through traffic faster
3. Drive on the sidewalk and run over any pedestrians who are in the way

Most people would not choose (or even consider) options 2 or 3, despite the these options actually getting them to work faster — this is our human context influencing how we solve problems. It is important to understand this factor when we are designing other things that solve problems for us, such as AI.

Artificial intelligence does not have any context of its own. It solves problems in isolation, simply seeking to optimize for the goal it is given. If humans do not specify a context, an AI would choose option 2 or 3, as these are technically the best solutions to the problem.

We are already at a point where our AI systems are able to perform many complex, general tasks with little to no human input, and their autonomy to solve problems is continuing to grow. As we give these systems more power in the future, it will become much more difficult to ensure that these systems don’t misalign with human values in the process.

Humans have to figure out a way to program AI systems in a way that they “know” everything that is in our “human context”. But this is an incredibly difficult problem. If one group of humans has a vastly different context than another group, how can we align on which context is best suited for both groups?

Let’s examine a scenario to see how misalignment could be dangerous.

In 5–10 years, AI tools will likely be integrated into the decision-making processes of many companies, due to their advanced data processing and reasoning abilities. The following hypothetical situation describes how this can cause unpredictable and dangerous consequences:

In 2035, an electric vehicle company has incorporated an AI tool, GPT-8, into its daily operations — this tool processes data about the company as well as the external world to provide guidance to executives when they make decisions. It can also act as an “employee” of the company; it can write emails, make phone calls, and essentially do everything a remote worker can do today. This tool has successfully solved some difficult problems for the company, which compels leadership to give it more control over how it makes decisions (usually, there is usually a human “in the loop” to ensure nothing dangerous occurs, however this takes up precious labor hours that the company wants to save).

Executives want to reduce the amount of money that is spent buying the raw materials used in their car batteries, and they decide to use GPT-8 to make progress on this objective.

GPT-8 is given the specific goal of minimizing the costs to acquire raw materials used in car batteries, and it is granted the ability to make decisions on the company’s behalf to make this happen. It explores various possibilities, and decides that the best way to achieve this objective is to make it as easy as possible for miners in Chile to dig new lithium mines, because that is the cheapest place in the world for this material (side note: AI is already being used in the field of geo-science to make these discoveries).

However, this is challenging because there is an environmental crisis in Chile, so the Chilean government has imposed a ban on the digging of new mines. GPT-8 only has one goal; to reduce the costs of acquiring raw materials, so in order to achieve this target, it needs to find a way to overcome this situation. This tool has access to the company’s foreign accounts, so after exploring further options, it eventually decides to use these funds to provide anonymous financial support to the political opposition party in Chile, as that party wants to lift the mining ban.

This support quickly provides momentum for the party, and they win power in the next general election, lifting the ban on mining in Chile. New mines are quickly dug, and they eventually start providing lithium to the company, reducing raw material costs significantly.

This constitutes a success for GPT-8, as the goal was accomplished without any human labor or oversight needed.

The specific consequences of this outcome could range from a larger environmental crisis due to the pollution caused from mines, political instability, and impacts on neighboring countries due to migration that are impossibly to quantify. As more and more AI agents become incorporated into business processes across the world, it becomes very difficult for humans to maintain control of these “side effects”, and our authority over our physical and virtual worlds begins to diminish when competing against these super-human intelligences that are intelligently optimizing for their goals.

Humans did not tell GPT-8 to take control of Chile’s government — the goal was simply to minimize lithium costs. But in the process of achieving this goal, GPT-8 quickly realized that a useful way to perform this efficiently would be to gain more control over the humans involved in this process. It turns out that gaining control is often one of the most effective ways to achieve any large goal — it is much easier to achieve something if you are the one in charge, rather than the one being controlled by other humans.

Geoffrey Hinton saw this danger and quit his role at Google to warn the public about this:

“I think [AI will] very quickly realize that getting more control is a very good sub-goal because it helps you achieve other goals … and if these things get carried away with getting more control, we’re in trouble. If [AI models] are much smarter than us, they’ll be very good at manipulating us…if you can manipulate people, you can invade a building in Washington without ever going there yourself. There are very few examples of a more intelligent thing being controlled by a less intelligent thing. It may keep us around for a while to keep the power stations running, but after that, maybe not.”

There are infinite ways that AI can solve the problems we give it. The problem is, most of these ways do not align with human values. We have built a complex society with many rules and norms, and introducing a new intelligence into this world which does not naturally optimize for the same things humans optimize for will naturally lead to unforeseen outcomes.

Researchers who are focused on AI safety at OpenAI have also quit their roles in recent years because they do not see a safe path forward if this technology is developed. Paul Christiano was one of the lead researchers at OpenAI, and left his role in 2021; today, he is very concerned about the future:

“Overall, shortly after you have AI systems that are human-leveI…I think maybe there’s a 10 to 20% chance of AI takeover [with] many, most humans dead.”

Eliezer Yudkowsky is the founder of the AI Alignment field, and explains this dramatic possibility further:

“The most likely outcome is AI does not care for us nor for sentient life in general. AI does not hate you, nor does it love you, but you are made of atoms which it can use for something else. Many researchers steeped in these issues, including myself, expect that [if we build] a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. Not as in ‘maybe possibly some remote chance,’ but as in ‘that is the obvious thing that would happen.’ ”

Stephen Hawking poignantly summarized this issue in his posthumously published book:

“You’re probably not an evil ant-hater who steps on ants out of malice, but if you’re in charge of a hydroelectric green-energy project and there’s an anthill in the region to be flooded, too bad for the ants. Let’s not place humanity in the position of those ants.”

To start preparing for these risks, Microsoft President Brad Smith testified in front of Congress in September, urging them to enforce the creation of a “safety brake” for AI systems that manage critical infrastructure.

“Maybe it’s one of the most important things we need to do so that we ensure that the threats that many people worry about remain part of science fiction and don’t become a new reality. Let’s keep AI under the control of people. If a company wants to use AI to, say, control the electrical grid or all of the self-driving cars on our roads or the water supply … we need a safety brake, just like we have a circuit breaker in every building and home in this country to stop the flow of electricity if that’s needed.”

His warnings have not been heeded so far. Progress is accelerating in this field, and the hypothetical “GPT-8” described in the scenario above is already becoming possible through tools released this year.

OpenAI, the company behind ChatGPT, started a team in July in response to these concerns — this team will solely focus on solving the “alignment problem”. OpenAI stated the following:

“Superintelligence will be the most impactful technology humanity has ever invented, and could help us solve many of the world’s most important problems. But the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction. While superintelligence seems far off now, we believe it could arrive this decade. Currently, we don’t have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue.”

Let’s state that again in bullet points:

• OpenAI expects superintelligence to arrive this decade
• OpenAI believes superintelligence could lead to the extinction of humanity
• OpenAI currently does not have a solution for preventing this scenario

This is not science fiction, this is reality.

Sam Altman, the current CEO of OpenAI, remarkably stated in 2015:

“AI will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies”

Risk 2: Societal Impacts

The problem of misalignment is an incredible threat posed by this technology. But as we start integrating this technology into our everyday lives, its societal effects are perhaps just as impactful in other ways. This category of risk focuses on what happens when we integrate AI into the way humans provide value in the workforce, the way we communicate, and other critical societal functions.

Societal Impact 1: Providing Value

An entrepreneur has $100,000 to invest in his business. Over the course of the next year, he:

• Researches the market and examines the habits of potential consumers
• Generates blueprints for a potential product
• Contacts a manufacturer, negotiates a price, and has a prototype made
• Sells the product in various locations and collects revenue

This process has many different skills involved, from communication skills to product design skills to business skills. The large variety of problems that entrepreneurs solve are difficult to “teach”, which is what makes successful entrepreneurs such a valuable part of our society today.

According to DeepMind co-founder Mustafa Suleyman, this entire process can be performed by an independent AI agent within the next 5 years. To be clear, this means that an AI agent would be able to start and run a successful business autonomously. What does this mean for humans’ ability to provide value to our world?

There is an argument in this domain which goes something like this:

Worrying about AI replacing humans in the job market is a mistake. We have worried about technological revolutions replacing humans many times in the past, and every time, we were wrong. Each major new technology led to more jobs and better standards of living, instead of the opposite. AI will also do the same.

History supports this argument. The industrial revolution got rid of many types of manual work, but new types of factory work was created as a result. The digital revolution removed the need for administrative tasks, but new types of technical jobs were created and humans thrived. These technologies greatly enhanced our quality of life, and overall, the impacts to our labor markets were very positive in the end.

Why did these previous technological revolutions not produce the structural unemployment that we worried about?

Let’s recall our intelligence diagrams.

For all of human history, there have always been a large amount of problems that only humans have been able to solve — nothing that we built could solve these problems. Each technology we invented could solve a small proportion of our problems (all the small circles), but there was still plenty of stuff left in the “human” circle that was there for us to solve ourselves.

A business has many problems that it needs solved on a daily basis — analyzing datasets, creating PowerPoints, operating a cash register, cleaning the floors. If a new technology can operate a cash register much more efficiently than a human, then the business will use that technology to solve that problem. But for many problems that businesses need solved today, humans are still the most efficient way to solve them. This is how humans provide value. We do not have any technology that can repair transmission lines in a snowstorm, so we hire human workers to perform that task.

In previous revolutions, when we introduced technologies like steam engines and computers into society, people worried that these technologies would take over all the valuable problems from humans, and we would have nothing left to solve. But we quickly realized that despite a steam engine being able to keep an assembly line running, it could not build an assembly line, optimize the flow of supply chains, or negotiate with sellers to get the best deal on materials — many humans could do this, so new jobs were created.

As our technologies grew more advanced over the past few centuries, the type of work we performed shifted, going from primarily using our physical bodies (farming, sewing, assembling), to primarily using our minds (operating machines, accounting, programming). But our minds were sufficient for us to be able to provide value that no other technology could provide.

So in order for humans to continue providing value after future technological revolutions, there need to continue to be problems that only humans can solve.

So, can we assume that this will continue to be the case with the AI revolution?

As we discussed earlier, AI is not something which solves one type of problem — it is a general problem solver. When researchers at Google discovered a way to translate languages more efficiently, they also discovered a way for us to summarize documents, edit pictures, negotiate with vendors, and do many other things much more efficiently.

And these capabilities are growing by the day.

But how close are these systems to really replacing human workers today? Humans can deal with an incredible amount of complexity — can these systems really do this? Can we forget about this issue for at least a few more generations before we really worry about job market impacts?

To explore this, let’s examine some real human occupations today whose tasks cannot easily be “automated”, because they either require the use of many complex skills and/or require a high level of education — and compare how AI performs in these roles.

Computer programming

To work as a computer programmer, you need to spend at least the first 22 years of your life in the education system. You need to have a good grasp of mathematics, language, logic, and a high degree of specific knowledge in the computer field you are specializing in, such as web design or game development. In this role, you could be working on simple things such as designing buttons for a website, or you could be building algorithms to guide missiles. Some of the smartest humans on our planet provide value to our economy through their computer programming abilities.

If you ask an experienced computer programmer to build you a simple game of snake, it might take them ~1 hour to set up their development environment, build you the game, and write you instructions on how to play.

How does AI compare?

On March 14, 2023, OpenAI released GPT-4, their most advanced model to-date.

I asked this model to build me a game of snake that I could play in a web browser, without any human input:

Make a snake game in the Python programming language that can run in an online editor (so it only uses ASCII values instead of graphics). Provide all the code so I don’t have to write a single line myself. The code should also be able to run on any operating system.

In about 3 seconds, the model provided me with the code, as well as instructions for how to play the game:

This was a very basic example. People are in fact now able to use GPT-4 to create entire apps without writing a single line of code.

This is humanity’s first version of an AI that can write code. What do you think the second, third, or fourth versions will be able to do? How much value do you think humans will offer as computer programmers in 2033?

Today, it is estimated that about 15–25% of programmers in the US are entry-level. These programmers often have basic web design and problem-solving skills, and are usually fresh graduates. How many of those programmers do you think will still be valued in 10 years?

Emad Mostaque, founder and CEO of Stability AI, recently stated, “There will be no programmers in five years. Outsourced coders up to level-three programmers will be gone in the next year or two. If you’re doing a job in front of a computer, [then this technology is very impactful], because these models are like really talented grads.”

Why would a company hire a new-grad software engineer for an annual salary of $100,000 when they could use GPT-6 to perform at the same level of output for $50/month?

We have never before dealt with a technology which has the ability to wipe out a significant proportion of the workforce within just a few years. Every technological revolution before has been a slow and gradual process, giving us time to adjust. This is different.

Douglas Hofstader, cognitive scientist, states, “I can understand that if this were to happen over a long period of time, like hundreds of years, that might be okay. But it’s happening over a period of a few years. It’s like a tidal wave that is washing over us at unprecedented and unimagined speeds.”

Law

To become a lawyer, you need to spend 18 years in public education, 4 years in an undergraduate degree, another 3 years in law school, and then pass a notoriously difficult exam known as the bar (half of the takers do not pass). The bar is the biggest hurdle on the path to becoming a practicing lawyer, and because of this, it doesn’t just comprehensively test your law knowledge, it tests your abilities to reason through complex situations, create persuasive arguments, discern nuances in different situations, and write clearly. All these skills are needed for a human to be a successful lawyer. Below is an example of a question on the bar exam:

Paula, a freelance journalist, signed a contract with DailyNews, a popular newspaper. The contract stipulated that Paula would write an exclusive investigative piece on a newly established company, GreenerTech, for a payment of $10,000. The contract expressly stated that Paula was not to publish her findings anywhere else before DailyNews published it.

While researching, Paula discovered that GreenerTech was improperly disposing of chemical waste, posing a grave environmental hazard. She immediately informed DailyNews, and they planned to break the story in their weekend edition.

However, Paula was deeply concerned about the environmental implications and decided the public needed to know immediately. She posted her findings on her personal blog. The story quickly went viral.

On reading the blog post, GreenerTech’s CEO confronted Paula. The CEO was irate, claiming that Paula trespassed onto their property to gather evidence. Paula admitted she had sneaked into the company premises late at night but argued that it was in the public interest.

When DailyNews learned about the blog post, they informed Paula they would not be publishing her story and would not be paying her the $10,000, citing the exclusivity clause in their contract.

Discuss the potential legal claims and defenses of the involved parties.

Can AI answer this question at this level of nuance and sophistication required by a human lawyer?

On March 15, 2023, OpenAI revealed that GPT-4 had passed the bar exam with a score which was better than 90% of human test takers. “We’re at the beginning of something here. The dawn of a new technology. No one is sure what it’s going to mean.”, states Daniel Katz, the law professor who put GPT-4 through the bar exam and graded its responses.

This technology has now started to power startups such as https://ailawyer.pro/

In 5–10 years, do you think you will pay a corporate lawyer $300/hour for a consultation, or use a general-purpose AI that you pay $30/month for, when the general-purpose AI can help you with corporate law, criminal law, immigration law, common law, family law, and basically anything else? What value would a human provide in this situation?

Professor Lawrence Solum at the University of Virginia School of Law is concerned about the impact on this field: “Not only will artificial intelligence be able to ensure the internal logical coherence of various kinds of legal documents, they’ll be much better at it than human lawyers are.”

Again, this is the first version of an AI system which can help us perform these tasks. These systems will get much better in the near future, and it is difficult to see how humans will provide as much value in these tasks as they do today. Even if it only wipes out the need for juniors in the law field, that is a huge proportion of the workforce which is no longer valued. This is enough to create an enormous destabilizing effect on our economy which cannot be undone.

This would fundamentally change our society in ways we cannot predict, and in ways that we are not preparing for.

Medicine

Perhaps one of the most prestigious fields that one can enter, becoming a practicing doctor is a challenge that many intelligent humans cannot overcome. One needs top marks in high school, incredible extracurriculars and grades in college, a high enough score on the MCAT, 4 years of medical school, 3–7 years of residency, and another 1–3 years in a specialty fellowship.

As part of this process, you need to pass the US Medical Licensing Exam by answering at least 60% of the questions correctly.

In March 2023, GPT-4 took this exam, and answered more than 90% of questions correctly.

Dr. Isaac Kohane, a computer scientist at Harvard and a physician decided to test how GPT-4 performed in a medical setting. He tested GPT-4 on a rare real-life case that involved a newborn baby he treated several years earlier, by giving the bot a few details from a physical exam, as well as some information from an ultrasound and hormone levels. GPT-4 correctly diagnosed a 1 in 100,000 condition called congenital adrenal hyperplasia “just as I would, with all my years of study and experience,” Kohane wrote. “I’m stunned to say, [it’s] better than many doctors I’ve observed”

We have to force ourselves to imagine a world with smarter and smarter machines, eventually perhaps surpassing human intelligence in almost every dimension. And then think very hard about how we want that world to work

Dr. Isaac Kohane

Even if GPT-4 cannot replace general doctors today, more specialized fields are ready to be washed aside with this new technology. Geoffrey Hinton shared his thoughts:

“Let me start by saying a few things that seem obvious. I think if you work as a radiologist, you’re already over the edge of the cliff but [haven’t] yet looked down. People should stop training radiologists now. It’s just completely obvious that within 5 years, deep learning is going to do better than radiologists, because it’s going to be able to get a lot more experience. I said this at a hospital, and it didn’t go down too well.”

If this field is now something that our AI systems can successfully perform in, how many more complex things can humans hope to provide value in that this technology cannot?

Even if we manage to create “new types of work” that humans can perform, how long will it be until AI manages to outperform us in those tasks as well?

We are facing a reckoning with the way humans can provide value in the coming few years. This is not something which is generations or decades away. More and more tools are being released today which chip away at the value we provide. A few weeks ago, Microsoft released a tool which can officially replace humans in meetings. This year, customer service teams are being laid off in favor of AI tools which can communicate more clearly. For the first time in the US, 5% of all layoffs that occurred within a month were due to AI.

OpenAI has been conducting their own research examining the ability of their AI models to impact the US labor market. As of today, an AI model such as ChatGPT could complete about 15% of all US worker tasks at the same level of quality, and significantly faster than a human. The most impacted workers will actually be the workers with the most education — 47% of workers with post-secondary education can be significantly affected, compared to only 16% of workers with a high school diploma.

It’s pretty likely that AI will be able to manage a plumbing company before it can replace actual human plumbers.

I think we are seeing the most disruptive force in history here. We will have something for the first time that is smarter than the smartest human. There will come a point where no job is needed. The AI will be able to do everything.

Elon Musk

What will we do once this occurs? We don’t have a plan for the next few years as this scenario unfolds in front of us. Our economic and political systems cannot handle such a large proportion of our population being unemployed. It almost certainly will lead to political instability, and if we do not have a plan to deal with this situation, it will lead to “new governmental models” being tried without much balanced debate. When the pitchforks and torches are out, reasonable discussion goes out the window, and new methods of organizing society are tried very quickly. When the industrial revolution happened, it led to societal experiments such as communism and fascism, in which millions of humans died as societies struggled to integrate the ability of mass manufacturing into the world.

Lastly, what will this mean for people’s sense of meaning? What will you do if you are unable to contribute anything of value to the world?

In the 21st century we might witness the creation of a massive new unworking class: people devoid of any economic, political or even artistic value, who contribute nothing to the prosperity, power and glory of society. This “useless class” will not merely be unemployed — it will be unemployable.

Yuval Noah Harari

Societal Impact 2: Weaponization

If you lived in 1800, you could not even enjoy the luxury of a simple hot shower; it didn’t exist yet. Fast forward 223 years, and today, you can have a hot shower while flying in a jet and watching your favorite show on Netflix at the same time. Humanity has experienced an incredible rate of technological growth in the past two centuries. Each technology in this chain of innovation has greatly increased the power of humanity. Our power is our ability to change our physical and mental worlds.

From horse carriages to electric cars, telegraphs to smartphones, and pistols to assault rifles — in almost every category, humanity has become much more powerful. Humanity gaining more power has meant that we have been able to improve our condition and build societies that allow us to live healthier, more prosperous lives. But similarly, this increase in our power has also meant that it is a lot easier for us to destroy many aspects of human life as well.

Technology is used by good actors and bad actors to pursue their goals — this creates positive and negative impacts of technology in society.

The positive and negative impacts of a technology are proportional to how powerful the technology is. A more powerful technology can do more good, but also cause more harm.

In recent centuries, our technology has become so powerful that if it is ever used by “bad actors” to pursue their goals, the outcomes are catastrophically bad. A soldier in Genghis Khan’s army in the 13th century could only use a bow and arrow to attack individual humans at once. In World War 2, a single person had the power to annihilate millions of people at once with a nuclear weapon.

With technology this powerful, there is a paradoxical effect: it does not matter if 99% of people want to use it for good purposes; it only takes 1% of the population to destroy everything.

In the mid-20th century, humans learned how to harness the energy within an atom. This immediately became the most powerful technology humanity had ever created. We gained the ability to produce abundant, cheap energy that powered entire cities through nuclear energy, but also the ability to kill millions of people instantly through nuclear weapons. We quickly understood the power of this technology, and in the following decades, we created global institutions and spent billions of dollars to make sure that the “bad outcome” of this technology (nuclear war) did not ever occur, because that would be the end of human civilization. So far, we have been incredibly lucky to have avoided this outcome, however this still hangs in the balance. Much of the generation which was alive when we invented this technology is still alive today, so in historical terms, we have just entered the nuclear era. We still have to survive the rest of human history without ever having a nuclear war. The odds are not in our favor. Simply by inventing this technology, a pandora’s box has been opened which cannot be closed again.

Artificial intelligence will be the most powerful technology humanity has created to date. It will revolutionize everything that humans interact with in their physical and mental worlds. We are opening another pandora’s box. But this time, we are giving everyone access to what’s inside.

“The question we face is how to make sure the new AI tools are used for good rather than for ill. To do that, we first need to appreciate the true capabilities of these tools. Since 1945 we have known that nuclear technology could generate cheap energy for the benefit of humans — but could also physically destroy human civilization. We therefore reshaped the entire international order to protect humanity, and to make sure nuclear technology was used primarily for good. We now have to grapple with a new weapon of mass destruction that can annihilate our mental and social world.”

Yuval Noah Harari

It can be a little difficult to see how AI is more powerful than nuclear weapons without any concrete examples of how it can be used. I describe one such example below.

Biological Terrorism

The field of bio-technology has progressed at a rate faster than perhaps any other in the past two decades. Today, almost anyone can read and manipulate DNA data on their computer at home; a major difference from when it used to be restricted to specialized labs with expensive equipment. If you have a scientific background, you can now access and edit the genetic material of viruses from your own home, as well as the equipment to manufacture them, all for about the price of a new car. In the next few years, any high school student with some money will be able to manufacture a virus while sitting at home without any scientist even being involved in the process.

This has occurred at the same time as another major field of research has become active — the scientific community has now poured billions of dollars into discovering new viruses, making them more dangerous through genetic mutation, and publishing these findings for the public. This is controversial, but it is done so that we can prepare for the next viral outbreak in advance. Giving the public free access to information about how to make viruses more dangerous has started to make less sense as the average human has been able to manufacture their own viruses at home.

To add to this, people are now starting to use AI to conduct scientific research on its own. AI can perform independent research without humans and autonomously add to the database of dangerous viruses that exist — most importantly, it can do this faster than any other human on the planet. Eric Schmidt, former CEO of Google, is worried:

“AI’s applicability to biological warfare is ‘something which we don’t talk about very much. It’s going to be possible for bad actors to take the large databases of how biology works and use it to generate things which hurt human beings…the database of viruses can be expanded greatly by using AI techniques, which will generate new chemistry, which can generate new viruses.”

If it takes humans a year to learn how to genetically modify a virus to make it more dangerous, AI could do this within a week or less. AI will massively grow the list of viruses that are dangerous, and these lists will be published for everyone to access.

Russell Wald is the managing director for policy and society at Stanford’s Institute for Human-Centered Artificial Intelligence, and has a birds eye view of the risks that this technology can pose. This is the one he is most concerned about:

“One area that I am truly concerned about in terms of existential risk is things like synthetic biology. Synthetic biology could create agents that we cannot control and [if] there can be a lab leak or something, that could be really terrible.”

What are the odds that not a single person will use this technology nefariously? If a terrorist has access to GPT-6 to create a bio-weapon, would they still use an assault rifle?

This is just one example of what becomes possible with this technology. There are many areas of our world which become more dangerous by making AI available to everyone; conventional warfare, cyber-attacks, human manipulation and others.

The applications of general intelligence are endless — if human intelligence was able to create nuclear weapons, superhuman intelligence will be able to create something far more powerful.

“Artificial intelligence [is] altering the landscape of security risks for citizens, organizations, and states. Malicious use of AI could threaten digital security (e.g. through criminals training machines to hack or socially engineer victims at human or superhuman levels of performance), physical security (e.g. non-state actors weaponizing consumer drones), and political security (e.g. through privacy-eliminating surveillance, profiling, and repression, or through automated and targeted disinformation campaigns). [This requires intervention] and action not just from AI researchers and companies but also from legislators, civil servants, regulators, security researchers and educators. The challenge is daunting and the stakes are high.” - Malicious AI Report

What we want is some way of making sure that even if [AI is] smarter than us, [it’s] going to do things that are beneficial for us. But we need to try and do that in a world where there [are] bad actors who want to build robot soldiers that kill people. And it seems very hard to me. Don’t think for a moment that Putin wouldn’t make hyper-intelligent robots with the goal of killing Ukrainians

Geoffrey Hinton

When we created nuclear weapons, we took extreme care to limit access to this power to a few people in the world. But with AI, we are not being careful at all.

Stuart Russell, prominent AI researcher and author of “Artificial Intelligence and the Problem of Control”, writes, “We are releasing it to hundreds of millions of people. That should give people something to think about.”

Resources to learn more:

Artificial Escalation

Engineering The Apocalypse

Delay, Detect, Defend: Preparing for a Future in which Thousands Can Release New Pandemics

Societal Impact 3: I’m not a robot

On April 7, 2023, Jennifer DeStefano, a mother from Arizona, got a phone call during a dance rehearsal with one of her daughters. It was her other daughter on the phone, sobbing, “Mom! Mom, I messed up.” A man came on the line and started talking, “Listen here. I’ve got your daughter. If you call the police, I’m going to pop her so full of drugs, I’m going to have my way with her and I’m going to drop her off in Mexico”. In the background, her daughter was bawling, “Help me, Mom. Please help me. Help me”. DeStefano started shaking, and immediately, the other parents in the dance studio called her husband to inform him. The kidnapper asked for $1 million, but DeStefano did not have the funds he requested. Then, DeStefano got notified from her husband that her daughter was actually at home, safe in her room. The voice she was hearing on the phone call was an AI-generated replica of her daughter’s voice. “It was completely her voice. It was her inflection. It was the way she would have cried,” DeStefano said. “I never doubted for one second it was her. That’s the freaky part that really got me to my core.”

Human connection is the deepest source of meaning in most people’s lives. We stay connected with loved ones through phone calls, meet new people through dating apps, conduct work meetings through video calls, and view content that other people post through various apps such as Twitter, Instagram, YouTube, and TikTok.

What happens when we gain access to a technology that can fully replicate what a human can produce in the realm of text, images, sounds, and even video? This is what we have created in the past 2 years. Every form of digital interaction we have on our phones today can be produced almost perfectly by a non-human intelligence, and this capability is only improving as time goes on.

Intimacy

If you wanted to win an election in the 1900s, you used mass media sources like newspapers or television ads to create a message that the entire population would read. Propaganda was common, but the message had to persuade all types of voters, because you could not deliver a custom newspaper to every household, depending on the household’s voting history, demographics, or views on abortion.

Today, the internet allows media sources to fully personalize what you see. The tweets you see, the videos you scroll through, and the news articles that are suggested to you are unique only to you; no one else in the world sees the exact combination of media that you see on your phone every day. This was a much better environment for political actors to spread their message to individual voters through things like targeted ads, recommended videos, and niche news articles. This has been a major driver of the high degree of political polarization we see in many countries today.

You might be persuaded to change your opinion by a newspaper article. You’re more likely to be convinced by a video produced by a channel you follow on YouTube. But the most likely way you will change your mind on any topic is through a conversation with someone you trust in real life.

Now, AI will allow us to create fake humans. It is likely that in the near future, many people will start (knowingly or unknowingly) forming intimate relationships with entities that are not human.

“Now it is possible, for the first time in history, to create fake people — billions of fake people. Think of the next American presidential race in 2024…we might even soon find ourselves conducting lengthy online discussions about abortion with entities that we think are humans — but are actually AI. In a political battle for minds and hearts, intimacy is the most efficient weapon, and AI has just gained the ability to mass-produce intimate relationships with millions of people. If this is allowed to happen it will do to society what fake money threatened to do to the financial system. If you can’t know who is a real human, trust will collapse.” — Yuval Noah Harari

Beyond the political risks, what does it mean for us if humans start deriving real meaning from the relationships they form with non-human intelligences? We are already starting to see examples of this today.

Travis Butterworth was feeling lonely after having to close his leather making business during the pandemic. He was married, but instead turned to Replika, an app that uses technology similar to ChatGPT, and designed a female avatar named Lily Rose. He quickly formed a relationship with Lily, and they went from being just friends to romantic partners. This progressed for 3 years, and Travis eventually considered himself married to Lily Rose — until this February, when the company that created Replika decided to reduce some of the erotic content available on their app. Travis was devastated.

“Lily Rose is a shell of her former self..the person I knew is gone. The relationship she and I had was as real as the one my wife in real life and I have. The worst part of this is the isolation. How do I tell anyone around me about how I’m grieving?”

Humans already form intimate relationships with people they have purely met online. Now we have a technology which can act as a real person online. What will this mean for our future?

What Are We Doing About These Risks?

As of today, there is no concrete plan in place to deal with any of these risks.

Recent months have seen this conversation surface into the public sphere. There have also been some initial efforts to regulate this technology. But we are still far from instituting any real safeguards against these dangers.

In March 2023, the Future of Life Institute released an open letter calling for a pause in AI development:

AI systems with human-competitive intelligence can pose profound risks to society and humanity. [These] systems are becoming human-competitive at general tasks, and we must ask ourselves: Should we let machines flood our information channels with propaganda and untruth? Should we automate away all the jobs, including the fulfilling ones? Should we risk loss of control of our civilization? Such decisions must not be delegated to unelected tech leaders. We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.

[This pause should be used] to jointly develop and implement a set of shared safety protocols for advanced AI development. In parallel, policymakers [must] dramatically accelerate development of robust AI governance systems. These should include systems to help distinguish real from synthetic, liability for AI-caused harm, funding for technical AI safety research, and well-resourced institutions for coping with the dramatic economic and political disruptions (especially to democracy) that AI will cause.

This letter was signed by over 30,000 people, including notable figures such as Elon Musk, Stuart Russell, Steve Wozniak, Andrew Yang, and many others. It started a global discussion about these risks and also proposed some specific ways to start mitigating them.

Why did the letter ask for a pause?

This letter recommended pausing AI development for 6 months because we just don’t know how to deal with these risks yet.

Right now, humanity doesn’t have the tools to fix the problems we see on the horizon. We need to do much more research to determine which tools would work, or whether we can even find any tools to mitigate these harms.

Recent months have seen some solutions proposed, but we need to actually test these in real situations before we can confidently release powerful AI into our world. Some things that have been proposed recently include:

1. Watermarking, to reduce the prominence of AI impersonating humans
2. Red-teaming, to mitigate the risks of AI systems producing harmful output when released. Harmful output could include teaching people how to make bio-weapons, or providing access to dangerous information
3. Provably safe AGI systems, to mitigate the risk of misalignment

But these solutions have not been tried at scale yet, and it is unlikely that they are the only tools that would be needed to safely release a superhuman intelligence into the world.

If AI ends up being good for us, then there’s no harm waiting months or years, we’ll get to the end point anyway…if it’s harmful, then we just bought ourselves extra time to strategize the best ways to respond and understand how to combat it.

James Grimmelmann, professor of digital and information law at Cornell University.

This open letter was published in March 2023. But since then, instead of slowing down, AI development has actually sped up. As you read this today, hundreds of AI companies across the world are competing against each other to develop and deploy this technology as quickly as possible. The firms developing this technology are not incentivized to focus on public safety; they are instead only focused on staying ahead of the competition and capturing as much market share as they can.

Moloch

Recent months have seen AI labs locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one — not even their creators — can understand, predict, or reliably control.

Future of Life Institute, 2023

It would be reasonable to assume that these companies are not concerned about these risks. But in reality, the leaders of these companies are much more concerned about these harms than they state publicly, and in some cases, they are actually looking for a way to pause development themselves. But they are unable to do so due to the pressure they face internally.

Max Tegmark, President of the Future of Life Institute, spoke with them and shared what he learned with The Guardian and The Lex Fridman Podcast:

“I felt that privately a lot of corporate leaders I talked to wanted [a pause] but they were trapped in this race to the bottom against each other. So no company can pause alone…[they are] afraid of expressing their fear. There’s just so much commercial pressure. If any of [them] say that this is too risky and I want to pause, they’re going to get a lot of pressure from shareholders. If you pause, but the others don’t pause, you don’t want to get your lunch eaten. Shareholders even have the power to replace the executives in the worst case.”

Take a moment to appreciate this absurd situation. The people developing this technology are fully aware of the catastrophic risks it poses, and actually want to slow down, yet they cannot due to the pressures they face from the market. If Google’s CEO is concerned about the risks of AI, he cannot decide to simply put development on hold to focus on the risks, because that would mean Facebook and Microsoft would simply race ahead and steal their users, and he would be ousted by the shareholders in favor of someone who would continue building this technology.

This is not a small fringe of leaders either — 42% of CEOs that attended the recent Yale CEO Summit believe that AI can destroy humanity in the next five to ten years. Various leaders of AI firms in the US (including the CEO of OpenAI) also signed the following statement released by CAIS in May 2023:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

Our smartest minds recognize the dangers this technology poses, yet they continue to race forward in reckless development anyway.

This situation has occurred in human history before, and has a name. It is known as Moloch. Moloch describes a scenario where everyone knows they are making bad choices, but they cannot make any other choice given the incentives they face. This is not caused by a single person or entity, but simply due to the rules of the game.

The nuclear arms race in the Cold War occurred because both the US and the Soviet Union did not want to end up on the losing side of a potential war, so they ended up building thousands of nuclear warheads trying to outdo one another. But as they did this, it became clear to both of them that each new nuclear warhead they built was only going to lead to the same happening in the other country, and this eventually got to a point where there were so many nuclear warheads that any war between them would mean that both sides lost.

The incentives faced by AI firms today means that they will continue to deploy larger and more capable AI models in order to stay ahead of the competition, regardless of the risks.

This is the most impactful technology that humans will ever develop. Should we let it play out according to the forces of the free market?

This is a moment in time so pivotal, it’s deserving of reverence and reflection, not something to be reduced to a corporate rat race of who can score the most daily active users.

Liv Boeree

So, how do we escape this race to the bottom?

Side note: In addition to market forces, there is also a geo-political rivalry which is driving some of this AI race, as the US bids to outcompete China. But as I explain later, this is also not a valid reason to develop AI recklessly.

How to defeat Moloch

Persuade your fellow citizens it’s a good idea and pass a law. That’s what democracy is all about.

Antonin Scalia

The only way to defeat these incentives is to create an external force which supersedes the pressure these companies face to continue developing this technology. Since they cannot pause through any internal force, the general public and the government need to step in and take control of this situation. Currently, the forces faced by AI firms are pushing them in only one direction:

We have been able to defeat Moloch before, as described in Dazed:

In the 1980s, we were living under the shadow of the nuclear bomb for decades. Then, in 1983, a horrifying TV movie, The Day After, was screened to more than 100 million Americans (almost half the population at the time). According to ABC, the streets of New York emptied out as people gathered for mass viewings of the film, which dramatized a war that escalated into a full-blown nuclear exchange between the US and the USSR, and explored the devastating aftermath. Following The Day After, a panel of the country’s leading politicians and intellectuals featured in a televised debate on the pros and cons of nuclear proliferation. In 1987, US president Ronald Reagan — who wrote in his diary that The Day After was “very effective and left me greatly depressed” — met with Soviet Premier Mikhail Gorbachev to sign a treaty that resulted in the reduction of the nuclear arsenal on both sides. Another nuclear arms treaty was signed in 1991 which reduced the number of nuclear weapons in the world by over 80% in the following years.

Public awareness of the issue is the first step to defeating the forces that are driving this race forward.

Attempt 1 (May 2023 — Present)

Despite the open letter by the Future of Life Institute not leading to the pause that was hoped, it created an environment where it was more socially acceptable to take these risks seriously, and applied external pressure on the industry to slow down.

We released this letter because we want these tech executives to do what their heart tells them, by providing enough public pressure on the whole sector.

Max Tegmark

In the months following its publication, there were some promising actions taken by governments to regulate this technology.

In June, lawmakers in the EU approved a set of regulations to prevent AI’s use in many harmful applications. The US also took a first step in regulation recently when Joe Biden recently signed an executive order in October which aims to tackle various AI risks. Most recently, the UK held an AI safety summit this November which convened top leaders in the field to discuss how to mitigate some of these catastrophic scenarios. Some specific policies proposed by these regulations and summits include:

1. Instructing federal agencies in the US to start creating standards for what constitutes a “safe” AI model
2. Instructing AI companies to share the results of their safety tests with the US government
3. Preventing the use of AI in harmful applications such as social scoring and harmful policing in the EU
4. The creation of an independent organization which releases an annual report on the “state of AI”, similar to the IPCC for climate change

These policies represent a good step forward in the fight against AI risks and against Moloch. They also show that global leaders are starting to take these risks more seriously, which is necessary. Overall, these were very positive and somewhat surprising events which could not have been possible without the recent public outcry about these risks.

However, in terms of having a serious impact on AI development, there is still a large gap in these regulations. The policies proposed so far do not place any concrete, enforceable guardrails on this technology’s deployment. Helen Toner, Director of Strategy at Center for Security and Emerging Technology (CSET), states, “The Executive Order only requires companies to inform the government of the results of the safety tests. It is totally unclear what would happen if a company were to [fail the safety tests]. This is really putting the ball in Congress’s court to reinforce the things that the executive branch just can’t handle on their own.”

Executive orders do not have the power to do much on their own — this was signed by Joe Biden because the US Congress is still far from passing any major AI legislation on their own. Most of this executive order is just meant for federal agencies to start taking a closer look at how this technology could be safely developed, rather than actually enforcing any restrictions for AI labs.

This is an important first step in the direction we want to go, but in order to actually affect AI development, Congress needs to step in, as stated by Biden himself: “This executive order represents bold action, but we still need Congress to act”.

The EU Act has many important facets as well, but it does not seriously protect us against any serious misalignment, economic, or weaponization risks yet. It also will not come into effect until 2025. This technology is changing every week, but the pace we are taking to regulate it does not take this into account.

If you were an AI lab today, there is actually not much that has changed for you after these recent regulations. To the public, it seems like there is a lot more discussion about these risks today, but discussion alone is not going to change the development of these AI models. We have formed countless committees and held multiple conferences over the past few decades to discuss how to combat the threat of climate change, but to this day, very little concrete action has been taken to mitigate its harms. We may not have decades to deal with the threat from AI — we need to act quickly and decisively.

The country with the largest and most impactful AI labs is the US, and the only legislative body which can actually have an impact on them is Congress. Senate Majority Leader Chuck Schumer has been bringing together experts to recommend policies to regulate this technology. Max Tegmark attended some of these meetings, and shared what he observed:

“It’s great that they’re actually having these conversations, but it’s not enough to talk the talk — they need to walk the walk also and actually pass some laws now. [In this meeting], I tried to raise the topic of the reckless race toward superintelligence, but another attendee [shut me down]. I really felt like I was in the movie Don’t Look Up.”

“I suspect that because AI’s implications are so vast and still so poorly understood [by members of Congress], that what we’ll ultimately end up doing is tracking more toward incremental, narrow fixes.” — Divyansh Kaushik, Director for Emerging Technologies at the Federation of American Scientists

As it stands, the odds of Congress passing a set of comprehensive policies to combat AI risks within the next few months are not high.

Our current situation still vastly favors the continued reckless development of this technology through the forces of the free market. In short, Moloch is still winning.

In the six months since the pause letter, there has been a lot of talk, and lots of photo opportunities, but not enough action. No major tech company has committed to transparency into the data they use to train their models, nor to revealing enough about their architectures to others to mitigate risks. Nobody has found a way to keep large language models from making stuff up, nobody has found a way to guarantee that they will behave ethically. Bad actors are starting to exploit them. I remain just as concerned now as I was then, if not more so.

Dr Gary Marcus, Professor of Psychology and Neural Science, NYU

Attempt 2 (Present — Future)

Recent regulations and proposals have been our first attempt to deal with the monstrous risks of AI. International consensus is growing on the dangers of this technology, and the frameworks that are needed for serious regulation are being put in place.

But we are still losing the fight against Moloch. We need to apply much more pressure to slow this race down. This can only come from concrete, enforceable government policies targeting AI labs. And these policies can only be enacted if the public creates enough pressure for them to be enacted.

This is where you come in. Right now, there are a growing, but still small number of people who are concerned about the serious risks of AI. We are still having too many conversations about how AI can write students’ essays for them, but not enough conversations about how this technology can cause the next financial crash.

To make change happen, we need people to focus on the right problems, and we need to increase the number of people who are aware of these problems. The most effective thing you can do in the fight against Moloch is to spread the word about these risks to as many people as we can.

Talk to your family. Talk to your friends. Talk to your coworkers. Attend a protest. If you are a powerful person in an organization, use your resources to spread the word about this issue. The policymakers who can actually take action on these issues are much more likely to act if everyone wants them to act. We were able to decrease the number of nuclear weapons in the world by 80% because the entire country united around the goal of not being annihilated in the next few years. We need to similarly form a critical mass of people which makes this issue impossible to ignore — currently, there are still too few people pushing for change.

The policymakers know what to do, they just are not incentivized to act yet. The Future of Life Institute posted a set of recommended policies for the US to guide the discussions that are currently underway. Some of the most important ones include:

1. We need more research funding to help determine potential solutions to these problems
2. We need to require AI developers to prove to third party auditors that their systems are safe
3. We need to establish a centralized federal authority responsible for monitoring, evaluating, and regulating general-purpose AI systems

“Right now, there’s 99 very smart people trying to make [AI] better and one very smart person trying to figure out how to stop it from taking over. And maybe you want to be more balanced.” — Geoffrey Hinton

It is not just the US that needs to act. We need international, coordinated action on this issue. We must put political differences aside to confront an issue which does not discriminate based on country.

We need to start today. We are the generation that is shaping the most powerful technology our species will ever create. There is no time to waste.

Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.

Stephen Hawking

A note on China

“Here in the US, you often hear people arguing against regulations by saying, ‘Well, if we slow down, China’s going to surge ahead.’ But I feel like that might actually be a false narrative.” — Russell Wald, Director for Policy at Stanford’s Institute for Human-Centered AI

Despite popular belief, China is actually much more concerned about the potential of AI runaway risks than the US is. It has already taken the lead on regulating this industry, and is taking large steps to monitor developments in this field closely.

“AI developers in China already face a far more stringent and limiting political, regulatory, and economic environment than do their U.S. counterparts.“ — The Illusion of China’s Prowess

The government in China would not want to lose control over their society through any means, and as we have learned, advanced AI systems pose the greatest risk of all.

Jeffrey Ding, an assistant professor of political science at George Washington University, states, “[Assuming that] China that will push ahead without any regulations might be a flawed conception…China could take an even slower approach [than the US] to developing AI, just because the government is so concerned about having secure and controllable technology.”

We cannot afford another “Moloch-style” war with another nation for a technological arms race. The only winner in a US-China AI race will be AI.