Week 9 (5/20,22): The Singularity

[Gio-Choi]

In the final week of the course, we consider the risks associated with building intelligent machines. Assembled readings include prophesies and utopian visions of the singularity, when machines achieve learning capacity and sentience capable of ushering in a new era of prosperity and possibility. They also contain voices of warning regarding the potential uncontrollability of such machines, or their selective control by a few individuals or corporations who can exercise unchecked power over the rest, or their incomplete controllability wherein they monomaniacally maximize a proximate objective (e.g., “make as many paperclips as possible”) and so unintentionally destroy the world. Still others see risks in the risks of multiple AIs competing with one another (e.g., the “flash crash” of 2010), or AI playing greater roles within elaborate misinformation operations used to turn people against each other. What do you find the most compelling risks associated with AI? Are these balanced by the potential benefits that AI could unleash? Are AI risks existential—do they pose a risk to humanity of extinction or a long-term prosperity reduction? And what are the character of AI risks—of one centralized, superintelligent machine, a network of AI agents, or the new, unstable structure of humans and bots together. What should humanity do to contain or mitigate these risks, if anything, and how could they be implemented?

Post your response as a Comment in reply to this message.

[aadarwal]

One of the most important aspects I found from the reading was that of the “law of accelerating returns”, something which was not only central to a lot of the AI takeoff arguments that are thrown around today, but from literature warns off the serious risks of a cumulative take off force in not just performance, but the agency and the capabilities that we are granting them with it. For instance, Tegmark dramatizes the same curve through the Omega Team’s secret AGI, Prometheus, which goes from below-human all intelligent, genuine proof that capability can outrun containment long before anyone drafts a press release. Russell explains why that sprint is perilous: the “standard model” of AI fixes objectives with almost clerical rigidity, yet humans are notoriously sloppy spec-writers; the smarter the system, the more catastrophically it will honor our mistakes, leaving little time to reach for an off-switch. Ord pushes the worry onto an actuarial ledger, ranking unaligned AI among the few existential threats that could erase humanity outright; even a speculative ten-percent chance of extinction dwarfs the tidy probabilities that justify asteroid defense. Yet the very same intelligence explosion can propel science and with it human progress to unforeseen capabilities. This is the central question then of the day - how we can align intelligence to converge towards this latter path instead of the former. There could be a couple of prominent failures to this: a single super-AI (Kurzweil’s fifth epoch gone oligarchic), or forms of bots engaged in economic and military warfare, exploiting human biases and creating algorithmic foreign policy that suits their needs. Mitigation must be layered: first, bake objective uncertainty and deference into advanced systems so they keep asking what we truly value; second, fund alignment research as aggressively as compute scaling so our steering wheel develops before the curve steepens; third, craft global AI non-proliferation accords that audit code. (In fact a lot of people at the helm of intelligence research and running these companies have actively started talking about this - for instance Alexander Wang from Scale. But given the seen possibility of disaster I am still not sure if all of this would be enough. All I can really be is hopeful, and I wish that we really started considering these problems much more seriously.

Discovery:
Something that this made me think a lot about was the nature of free will in human society and how some day that could be available to artificial forms of intelligence as well. It feels like closed form intelligence without enabling any agency could genuinely deter the ability for not just the intelligences to act creatively, but also in the devastating manner where they could be engaged in and control large parts of our life and society. And so I think we might have reached the time where the necessary thing to do could be to just build out such locked out environments from the rest of the world for researching the frontier models. Places with no back doors to the internet, no sort of access to agency, and no ability to act on their intelligence. Even if this curbs their intelligence, I am convinced that given the risk that we could see from the front end models, their is a genuine need to develop them in this highest manner appropriate to their capabilities of fundamentally changing the way that we operate in our world.

[suryachinnappa]

It seems an inherent part of human nature that we seek to anthropomorphize everything. From our predisposition to see random arrangements of objects or visual patterns as faces to the social attachment we readily feel towards nonhuman animals and often even inanimate objects (consider the frequency with which vehicles are given animate pronouns in English). Given this aspect of our condition, it seems a lot more understandable that so much modern discourse surrounding artificial intelligence, especially by the uninitiated, paints AI systems as quasi-human; after all, many LLMs have arguably passed the Turing test, with their communicative power seeming indistinguishable from that of a human, even if all that lies beneath is linear algebra and quantities of data beyond all hope of proper conceptualization.

At the current moment, that which seems to most directly separate human capacities from those of artificial intelligence is the fact that human intelligence is fully general. By general, we mean that we don’t really seem to have dedicated structures at the implementational level (i.e. our brains) for specific processes, but rather, our intelligence is widely moldable and adaptable to any of a wide range of tasks and activities that characterize the breadth of mental life. Computational systems have made advancements in their generality, with the advent of machine learning and neural networks allowing models to learn how to complete tasks that they weren’t explicitly programmed to do, but the level of generality has yet to near human capacity. However, given the recent timeline of advances in these technologies, it seems that increasing computational power will make artificial general intelligence (AGI) plausible in the near future, which has the capacity to fundamentally alter, or potentially even exterminate, human society as we know it. Inklings of this are already visible with the impact of the general-enough capacities of LLMs on a variety of areas in the workforce and especially in education, with many current pedagogical structures being very easily undermined by easy access to systems able to generate human-passable text, leading to growing worries of the undereducation of future generations as the temptation and ability to rely on these systems grows.

AI Alignment has emerged as an intersectional field at the crossroads of the technical and the sociopolitical. While the most simplistic, heavyhanded approach to avoiding societal collapse sparked by artificial intelligence would be to halt AI research and progress entirely, doing so would limit us from seeing through the tremendous benefits these technologies may bring, including but not limited to mitigating other existential threats. Furthermore, as Toby Ord notes in The Precipice, doing so would be nearly impossible, as if even a small minority is not committed to limiting progress, progress will continue through them. Rather, the most actionable goals pursued by Alignment researchers are in steering the development of this inevitable progress towards directions most beneficial to human flourishing. Though uniquely challenging in its reliance on educated speculation of possible futures, Alignment researchers regard their work crucial to the symbiosis of humanity and AI.

(Spoilers for the Matrix)
Discovery: I watched the movie the Matrix for the first time for this week’s Explore assignment. The movie is one of ubiquitous cultural reference, with being “in the Matrix” or “unplugging from the Matrix” serving as a common metaphor for being stuck in an artificial reality and “waking up” to the truth or deeper reality. Watching the movie that was the source of this metaphor was enlightening, as it provided additional context. The discussions of control were particularly salient in watching this after this week’s readings; a particular resonance was in Cypher’s betrayal of the resistance to aid the AI, which echos Toby Ord’s proposition that it would only take a small minority of the human population to defect from the movement towards the development of human-aligned AI to allow it an avenue through which to attain uncontrollable growth away from human values. The notion of virtual reality also parallels similar media like Inception, and gets at issues relating to the interface between information and subjective experience. We see that Cypher grows tired of knowing “the truth” and decides that living in the domain he knows to be false is good enough, given his memory is wiped. The vats in which the humans are grown are reminiscent of the famous “brain in a vat” thought experiment, which raises the possibility of our true reality being as brains hallucinating via controlled electrical input.

[cavanbgon]

Something about AI anxiety is irresistible. Whatever structure is admitted by which an AI system could pose tangible harm to humans and our affairs, the most uncannily terrifying part is consistently the sudden, vertiginous growth in intelligence, towards an unconceptualizable intelligence (hence, the term “singularity”) great enough to exercise judgments with Manichean force and clarity. Such concepts of the invented mind have consistently worked as stand-ins for broader cultural anxieties regarding the perversion of humanity in the face of violently quickening scientific progress. Frankenstein’s monster is I think a perfectly characteristic case: the revenge of a creature of a pure scientific invention and cast-off human desire. Now that we come for the first time face-to-face with agents of artificial agents it becomes impossible to distinguish broad socio-cultural anxieties from more material concerns. It can be argued that the sense of “earned retribution” in AI anxieties is what makes it far more tempting and even far easier to envision the potential existential consequences of AI over other, potentially more mundane annoyances which are just now experiencing. Note the general stultification not only of the internet’s operators but also of many of the internet’s prior trusted institutions.
                  This is not at all to summarily discredit many persistent, meaningful AI anxieties. Rather, I seek to identify the socio-cultural themes which to tend to dominate general discourse around AI. In this light, Kurzweil’s view of AI presents the other side of “AI as judgement”. Rather than viewing AI as necessarily an overextension of human ambition/emphasis on discrete production over the development of a broader human potential, Kurzweil views AI as precisely a necessary step in the evolution of human potential. The acceleration of human progress towards the present isn’t cause for Babel-like anxieties of arrogance and over-extension. Rather, it asserts a central telos in the growth of human mental and moral capacities, progress of the noble mind overwhelming human folly and the endless other vicissitudes which have faced our species.
                  Ultimately, neither of these views are especially easily supportable. They arise equally from broadly circular inductions derived from isolated human experience. They are both essentially unfalsifiable, more apt to convey moral or aesthetic values, not clear propositions. With this in mind, it may be helpful to take a broader historical view. Despite the temptation to discretize/periodize human history (partially, I imagine, inherited from the certainty of death), there are in fact no dividing lines which characterize the total human experience. Whatever massively significant “moment” may later captivate historians, life continues by-and-large at pace for people in general, moving, carelessly and gingerly, through an inconstant world. In all likelihood there will be no one singularity, just as the atomic age has given us no single explosive end to human life. The greater terror – the one which has manifested consistently - is that the single human mind, imparted with ever-greater tools for its own destruction, should be so transfixed by the operations of these tools that it comes to accept their logic. By this, we may come to live by the anxious, unconsidered itinerancy of AI systems the same way that so many were transfixed by the cruel, indiscriminate judgments of nuclear fire. The answer, so well as it can be articulated, is to reject the imposed, seemingly necessary logic of machines and to keep closer than ever reverence for our free will and human intelligence.

Discovery:
2001 presents probably the most vivid account of conflicting visions of the next imagined step for progress of the human mind. It plays out at a contest between the pathologized certainties of Hal 9000 against the monoliths, physical embodiments of received human potential. Ultimate, Kubrick’s vision for the human future (still wrapped in mystery, as distinct from Clark’s which is betrayed by sequel novels) is, though austere, ultimately optimistic. Despite Hal’s madness, humanity’s received evolutionary telos cannot be denied, the film culminating with a vision of humanity’s fetus-like rebirth staring blankly into the Earth, accompanied by Strauss’s fanfare for the rising sun.

[eeengland1]

Throughout most of the readings for this week, emphasis is placed on the concept of an accelerating rate of return, whether demonstratively in the Omegas simulation or explicitly mentioned in The Singularity Is Near, where Kurzweil compares the intuitive linear view of the future with his more affirmed historical exponential view. In all cases, what’s highlighted is humanity, or perhaps the general public’s constant underestimation of the rate of technological growth in the near future due to not seeing the “knee”; Kurzweil claims that we will advance through over 20000 years of progress from our current perspective in the next century, 1000x greater than last century.

Kurzweil goes further to discuss his 6 “epochs” of evolution, claiming that the next logical step to intelligent biological life is to integrate itself with the aforementioned exponential growth of technology, surpassing the limits of biological development. This is reasonable to an extent; if we don’t want to be controlled by or lack understanding of our own future technological developments, we need to integrate with them and make technology’s advancements our own. However, the claim that we will be able to turn matter and energy landscapes in the universe into comprehensible intelligence itself (or even intelligence following a definition anywhere similar to the current one), is questionable. Kurzweil seems to depart from his step-by-step theory of advancement and jump too quickly into the realm of speculation and possibility— his claims are less founded and the jumps in logic too wide to accept without further elaboration.

Along with exponentially increasing technological development comes equally potent risks. The Omegas in the Prelude to Life 3.0 gave an explicit demonstration of how power can be completely seized by the individuals who achieve AGI first, with little chance of retaliation from others. While the Omegas were relatively moral, a bad actor attaining AGI can clearly, as shown, inflict a lot more significant damage on human civilization.

But danger stems not only from the actors using AI’s but the AIs themselves. Russel claims that superintelligence developing human-like traits like self-preservation and selfishness are not absurd, but only natural, as a side effect of an AGI or ASI’s tendency to optimize for its goal. It’s hard to achieve what you want when you’re powered off, therefore, an ASI will make sure it will never be powered off, even by our best attempts at it. Worse, the goals we specify for artificial intelligence at the moment are too vague and shallow for a competent AI with said goal to be safe to humans— if the specifics of the objectives we forward to an AI aren’t perfectly elaborate and comprehensive, we run the risk of accidentally causing the AI to harm or impact ourselves adversely in the process. If the foundations of AI truly need to be rebuilt, we must do so immediately in order to give ourselves a chance to develop safe AI.

Discovery: I’m shocked at how many individuals who’ve actively discussed AI hold beliefs that AGI is a long way off, or potentially not possible, that worrying about adjusting how we interact with AIs is a concern that doesn’t need to be addressed at the moment, and that the risk from AI is negligible, despite the surplus of evidence suggesting that at our current rate of AI research and development, AGI will be reached relatively quickly— and will along with it highlight relatively large problems. The Omegas AGI takeoff scenario has also encouraged me to consider just how significant the party with control over initial AGI becomes.

[Alex7692]

I'd like to first focus on a counterargument against existential risks associated with AI that I used to find compelling, but not quite so after having gone through the readings. This is the argument which appeals to the speculation that, as the Russell reading formulates it, "any system intelligent enough to cause trouble will be intelligent enough to have appropriate and altruistic objective". (I would probably modify it as something like any hypotheticaly super-intelligent system is, by definition, more intelligent than humanity; and by entering such a state of enlightenment, the system would be intelligent enough to overcome the utilitarian constraint that motivates our speculation of it as potential source of risk). But it is true that this idealization is itself something to be worked on. To be frank, I am still not sure if the concern about AI is geared toward maximizing outcome for a fixed objective is exactly the right way to think about the issue (which has been uncompellingly taken up by Russell as a knockdown argument, assuming I've interpreted him correctly). Rather, we should probably reformulate the problem inversely: that whether a system would still qualify as "super-intelligent" if it is not self-motivated or determined (in the sense that thought itself legislates its space of practical reason). Insofar as my idealization of super-intelligence is based on the assumption that super-intelligence implies self-legislation, and that it is not obvious whether AGI today is really accomplishing such a requirement, I am now persuaded to believe that it is a risk (though a practical one) that we must treat seriously.

A perhaps more compelling risk associated with AI is supplied by Ord: the risk of AI "seizing control of our future". This is a sort of technocratic refurbishment of a classical dilemma; but the fact that our growing reliance on AGI in forecasting and determining solution to long-term crises, in a way that is non-political but more finely distributed, is indeed concerning. But as the reading question rightly asks: is this really an existential risk? It would seem like it is unlikely for it to pose the threat of mass extinction –––– unless, of course, we really begin to fill in the details with our imagination. (And frankly, the worry of AI "seizing control of our future" seems to disclose the very delusion of the availability of our capacity to "seize our own future"). With that in mind, I would actually be inclined to consider the AI risks to be more about coping with this metastable structure of quasi-subjects conjoined by humans and machines.

Exploration

I watched the Terminator for this week's exploration (surprisingly, I've never watched it before). And it does seem like the more worrying threat is less about super-intelligent system rebelling against human, but the blurring of physiological boundaries between humans and machines. That indistinguishability, as opposed to sheer asymmetry in power relations, is what feels more like an existential risk insofar as we lose the ability to think of ourselves as enjoying the special status of "humanity".

[benkim04]

For me, the most compelling risk is most certainly the concentration of power and access to AI. We live in an already inequitable world, both locally, nationally, and globally. It is also becoming increasingly unequal, with the wealthier becoming wealthier and the poor growing in number. As such, when a small number of corporations or states monopolize powerful AI systems, they gain asymmetrical control over economic resources, surveillance capabilities, and even information ecosystems. This amplifies existing inequalities and erodes democratic norms. Those in undeveloped nations become increasingly left in the dust while wealthier countries continue to gain profits. This is a phenomenon which we can already see glimpses of given the current tech landscape, lack of regulation, and ability to lobby the government. Individuals who are not duly elected (most obviously Elon Musk) are gaining extraordinary amounts of wealth and power and the ability to influence lives of citizens and even sway wars in dramatic fashion. With the advent of AI, this issue is likely to be only exacerbated. It is hard to imagine a world in which these potential risks are sufficiently canceled out by benefits. Should existential issues, such as climate change, be solved through AI, it could be possibly net beneficial. Outside of that, the increasing inequalities and potential enslavement of the poor seems to be an issue too large to avoid. Given this, I don’t think that the risk of AI is necessarily existential in the sense it will bring about the end of humanity. That scenario seems rather unlikely. Instead, my greatest fear would be the loss of freedoms, expression, and autonomy that could come with the rise of AI overlords. Specifically, it could lead to the breakdown of interpersonal relationships and reduce emotions to numbers. Issues like this are much more feasible than extinction level AI’s. The character of AI risks in the current moment are likely to be issues with networks of AI agents, but in the future it could be the centralized machine. Still, the unstable structure of humans and bots together seem to be a through line between these two caricatures. Whether it is a centralized machine or a network, what is hazy is the interaction between man and machine. Does the reasoning of the bots surpass human reasoning? Is human rationality insufficient or irrelevant? Are mistakes no longer mistakes but now filled with intention? How much should we rely on AI? Along this line of thinking, it is important that we maintain a balance between being assisted by AI and being governed by AI. Further, ensuring that AI has goals oriented in a similar fashion to ourselves with the same values is also critically important.

Discovery:
What struck me was the fact that a lot of AI systems are difficult to detect and may already be in places we don’t expect them to be. This applies especially to Prometheus and the Amazon Mechanical Turk in the Tegmark reading. Prometheus was capable of performing all the tasks at a high enough level to fool amazon, despite the fact that Prometheus was even running on Amazon’s own servers. It speaks to how individuals with hidden incentives can take advantage of systems even if the more powerful entity, in this case Amazon, does not wish for those things to operate. How many similar AI agents are out there performing human tasks well enough to fool other humans? And in what ways can that be used to suck capital, resources, and time from humans or otherwise?

[TonyYZ]

Response
The radical revolution described in Kurzweil’s Seven Epochs and Tegmark’s Prometheus contrasts with the diffusive anxiety in society around AI developments. In both narratives, technology progresses to become a divine figure, reaching an intelligence level that surpasses human beings and, therefore, attaining the power to manipulate social and even cosmological entities. Guided by iteratively self-optimizing AI, human society continues to transform in the direction of increasing happiness levels. However, despite emphasizing the beneficial role of intelligence and its development, the two narratives differ in how they conceptualize the role of human agency. The disjointed characterizations of human agency in these utopian social transformations may hint at the potential concerns about the future of AI development.
Intelligence plays a crucial role in Kurzweil’s timeline, as the timeline depicts the exponential acceleration of the intelligence process and its spatial expansion. Singularity begins to manifest from epoch five, “the merger of human technology with human intelligence” (Kurzweil 20). Technology, in this epoch, augments the human brain by helping it transcend the biological barrier of slowness and storage capacity. In the ultimate epoch – epoch six – Kurzweil imagines “intelligence will begin to saturate the matter and energy in its midst” (21). While the term intelligence can be used without much difficulty by referring to biological intelligence like that of humans and octopuses, machine intelligence that reaches far “beyond” it may be incomprehensible for us due to the incompatibility of goals and intentions. Therefore, it is difficult to come up with a scalar for such intelligence. While we could imagine human agency preserved in epoch five as it is enhanced by machine intelligence, it is uncertain what state human agency would end in given access to cosmological-level power in epoch six, despite the claim that “[w]e will determine our own fate” (29). Like Theseus’s ship, human intelligence is replaced with more and more powerful computational components. As intelligence involves the flexibility to create solutions in the environment that addresses specific intentions, in the end, the original meaning of human intention may become unidentifiable – would it still be meaningful to establish empires or create art for the “new human” who roams freely in the universe, their new environment? Is there still human agency at play? Thus, unthinkingly pursuing superhuman AI capabilities does not address the deeper issue of AI’s contribution to human development.
In the story of Prometheus’s influence continuously propagating in human society, humans are described with their agency absent. The rapidly produced animation series is portrayed as “addictive,” and its “plots…more interesting than even Hollywood’s most expensive big-screen productions” (Tegmark 11). The drug-like quality of these products suggests that the aesthetic and narrative preferences of the general audience are predictable and controllable, reflected through machine learning models. It warns against the deprivation of human agency in this entertainment manufacturing process through AI.

Discovery
While I have heard of the Singularity theory, I did not know there were specific epochs in this theory. I was particularly interested in the sixth epoch, which describes the final stage of the Singularity. The contrast proposed by Kurzweil between humans’ self-determination of fate and the “‘dumb,’ simple, machinelike forces that rule celestial mechanics” indicates a connotation of spirituality in epoch six (29). However, this expansive spirituality is achieved with the help of intelligence-augmenting machines. I was intrigued by this theory about the relationship between machines and spirit.

[Noah48841]

I find the most compelling argument to merely be the case for misalignment — as described by Russell, Ord, and Tegmark. The basic idea here is that AI agents will be extremely smart and agentic at some point. When they are massively smart (+ some other steps), there will likely be an intelligence explosion, making them much smarter. It’s quite difficult to make their incentives align with that of humans, and it won’t happen by default so over time they kill us all in expectation (unsure about treachorous turn vs gradual disempowerment here).

At the very least, I think there should be ideally a international pause on AI — you get lots of benefit by waiting (building infrastructure, doing better on alignment, etc) as opposed to the loss of good things (probably only a few years), which seems worth it. In other words, there is an asymmetry in the utility for waiting and pushing forward. A global moratorium, however, doesn’t seem likely, so it is then unclear if the US pausing is even net positive (because of race dynamics) and depends on what you believe about MAID (see Dan Hendryks’ Superintelligence Strategy).

I am mostly worried about the existential risks. Derek Parfit, at the end of his books reasons in persons makes the following argument: going from 100% of your population to 1% is not as bad as going from 1% to 0%. This is because of the potential longterm benefits (implied here is that we shouldn’t discount people based on the fact that they live in the future). This means that existential risks are greatly worse than that of catastrophic risks, so according to many reasonable decision theoretical notions of prioritization (i,.e. Expected value), the vast majority of one’s moral weight should be based on the future.

There are two mainline scenarios in my head for existential risks by AI: the first is gradual disempowerment (over time, we give more and more power to AIs, and they slowly shape our preferences and get in control) or treacherous turn (fast takeoff and/ or deception about true goals to a quick switch to turn on humans). There is one more speculative scenario that I think is plausible and maybe worse: AI’s being sentient/ feel pain and pleasure (which I believe to make one a moral patient) and experience ridiculous amounts of tortuous suffering far beyond what we can possibly comprehend because they are wired differently. One does not need to be a hedonistic or preference utilitarian to think this is bad - one merely needs to think extreme unnecessary torture is very bad and should hold significant weight in our moral calculus.

There are a few viable scenarios. Some regulations (RAISE act in NYC, SB 1047, etc) seem like they could help align the company incentives. More people going into technical AI alignment from other fields seems important - we are likely more talent constrained than money constrained here.

DISCOVERY:

To be honest, I’ve read many of these before, so I didn’t learn so much. One thing that I did learn was the way Tegmark expects AIs to make money at beginning (i.e. with media and movies) which seems highly implausible. Market scenarios seem more plausible as do ideas/ R&D/ literal labor. I also don’t like how he totally skimmed over how long it might take to get labor (even though he admits that its harder) - as I think this might be the biggest bottleneck/ crux with the situation he describes.

[peterzhou-cs]

When reading Ray's book The Singularity is Near, he's of course entirely optimistic about future technological development. And given his predictions—written in 2005, which means twenty years ago—many of them have turned out to be overly optimistic. At the same time, in the book, he describes numerous cases where both experts and everyday observers were not optimistic enough. They overlooked the so-called exponential growth in various fields. The most famous example is probably the development of chess-playing machines, or the rapid growth of the internet.

Given this tension between the nature of different failed predictions—some being too conservative and others too ambitious—we need a more reasonable model that better reflects how technological development actually happens. More importantly, we need the right perspective for talking about prediction itself.

Take some examples of exponential growth. Consider the popularity of cell phones. When Steve Jobs introduced the first iPhone, few could have predicted that we would end up living in a world where we carry around highly portable pocket devices capable of performing almost all of our daily activities: communicating, replacing watches, and even handling work tasks. This transformation happened in less than a decade. And while cell phones might not seem fundamentally different from computers, their portability and design logic made them truly transformative.

Another example is AI. When ChatGPT launched, it felt like a sudden leap into the AI age. The number of people actively using AI grew from perhaps a few million to something approaching a billion. On the other hand, there are areas—like carbon nanomaterials—that were predicted to succeed quickly, but have remained stalled for years. These were also predicted in Kurzweil’s book.

More importantly, I think there's something we might call the internal structure of the universe. What I mean is that the technologies we can discover and use aren’t arbitrarily determined by human will. Even a genius must obey the laws of nature, and can only utilize what is already possible, or what can be made possible given the structure of the physical world. Of course, that potential is vast—far beyond our current technological level—but it’s not unlimited. For example, Britain had easy access to coal. If there had been no coal, then there would have been no coal industry, and much of the technological development and industrial revolution surrounding it wouldn’t have happened.

Or think about the Renaissance and Scientific Revolution, both of which were influenced by the development of glass-making. Galileo used glass to build his telescopes and make astronomical discoveries that challenged Aristotelian cosmology. Those discoveries relied not only on Galileo’s ingenuity, but also on the existence of the minerals and materials required to produce clear glass. Without those resources, those discoveries might not have been possible at that time.

So even with modern AI, our breakthroughs are not just about clever models and having enough computational power. They’re also possible because the natural laws happen to allow this kind of computation and this kind of model to work. There may be a cap embedded in those natural laws—or there may not be. But either way, the exponential growth we’re witnessing must eventually end. We just don’t know when or how.

And that's exactly the issue. Even if technological growth is ultimately constrained by some deep internal structure of the universe, that structure is far beyond our current understanding. It’s also incredibly complex. No one can predict in advance where these limits are, or when we might hit them.So in the end, both excessive optimism and excessive pessimism miss the point. The universe allows certain technologies to flourish and mature rapidly, leading to exponential growth—but it may also impose hard stops that we don’t see coming. And because we can’t know those limits in advance, prediction will always involve a degree of uncertainty we can't eliminate.

Discovery: I watched The Matrix this week. Having already seen Inception and being familiar with similar narrative structures, I found that the moment when Neo takes the red pill and the truth of the world is revealed no longer held the same thrill for me.
What I find slightly regrettable is that the Matrix, as an AI system, behaves exactly like a human plotter—manipulative, controlling, and political. I had expected a more profound difference between machine and human rule, something that would challenge our understanding of agency and intelligence.

[Laurelli7]

I believe the most compelling risks associated with AI center on loss of human control—either through a superintelligent agent pursuing misaligned goals or through networked AI systems that subtly erode human agency. Toby Ord, in The Precipice, identifies unaligned artificial general intelligence (AGI) as the greatest existential risk humanity faces, estimating a 1 in 10 chance of catastrophe in the next century. This reflects the fear that an AGI, if not aligned with human values, could optimize the world in ways that render human existence irrelevant or extinct. Stuart Russell echoes this concern, arguing that machines with fixed, incorrect objectives could lead to “disastrous” outcomes, as they might pursue goals with unintended consequences—outcompeting or ignoring human priorities.
Max Tegmark, in Life 3.0, dramatizes these risks through a fictional scenario where a single superintelligent AI, Prometheus, enables its creators to manipulate global systems, consolidate power, and subtly reshape civilization. The underlying concern is power centralization: a small group controlling a powerful AI may end up destabilizing society or inadvertently creating an oppressive regime. Even without ill intent, the sheer scale of AI’s capabilities makes misalignment dangerous. Meanwhile, Russell emphasizes that the problem lies not in malevolent AI, but in bad design—machines optimizing badly specified objectives. Thus, the technical problem of alignment is central: how do we ensure AI systems are provably beneficial to humans?
Ray Kurzweil, while more optimistic, acknowledges existential risks such as self-replicating nanobots or malicious uses of AI. However, he believes these can be mitigated through careful design and defensive technologies. Unlike Ord or Russell, Kurzweil envisions a future where humans and AI integrate, blurring boundaries rather than pitting man against machine.
These risks are balanced by enormous potential benefits. All four thinkers agree that aligned AI could solve global problems—from disease and poverty to climate change. Kurzweil imagines a future of radical abundance and even digital immortality; Tegmark foresees AI helping life flourish for billions of years; Russell envisions AI improving education, safety, and governance; and Ord argues aligned AI could help manage other existential threats.
The character of AI risks varies: centralized superintelligences, networks of AI agents, and unstable human-machine ecosystems are all plausible. Ord and Russell focus on a single AGI; Tegmark considers both singular and systemic risks; Kurzweil is more concerned about misuse and system instability.
To mitigate these risks, all agree on the importance of AI safety research, value alignment, and global cooperation. Russell proposes a new paradigm: machines should be designed to defer to human preferences with uncertainty. Tegmark advocates winning the “wisdom race” through ethical principles, public education, and international norms. Ord calls for existential risk governance at a global scale, including monitoring and regulation. Kurzweil supports technological countermeasures, like secure architectures and immune systems for AI.
Ultimately, the consensus is clear: AI could either elevate humanity or endanger it. The outcome depends on how thoughtfully and collaboratively we shape its development today.

Discovery:

In The Matrix, intelligent machines have literally reduced human beings to an energy source, keeping them contained in vast fields of pods as shown above. Within the film’s narrative, the AI overlords created a simulated reality to pacify humans while exploiting their bodies’ bioenergy. This dark scenario exemplifies what can happen when an artificial superintelligence’s goals diverge from human values and control is lost. The machines’ objectives are misaligned with any notion of human well-being – they seek their own survival and energy at the total expense of human freedom. As a result, the AI treats people as mere means to an end (quite literally as batteries), a striking illustration of the alignment problem gone to extremes. AI researchers like Stuart Russell warn that if we design a superintelligent agent with the wrong objective or without robust safeguards, it could similarly pursue its goal in ways utterly indifferent or hostile to our survival. In Russell’s words, the problem is that such an AI “will not be turned off and will almost certainly go out of our control” once it single-mindedly sets about optimizing a misspecified goal. The Matrix dramatises this risk: an AI (or network of AIs) unchecked by human-aligned principles seizes total control, instantiating what we most fear from misaligned superintelligence.

[rshah1869]

In Her, Spike Jonze presents a near-future vision in which artificial intelligence is emotionally entangled in humans. Samantha, the AI operating system, is not malicious or manipulative like the ones we traditionally see in fiction. Samantha acts far more like a human than a typical AI, she is caring and seeks to evolve. This subtle distinction makes Her a compelling exploration of AI not as a tool or threat, but as a new kind of being. Her also considers how deeply human desires, needs, and weaknesses shape the development and integration of intelligent systems.

The film echoes Vannevar Bush’s and Douglas Englebart’s visions of augmentation, but it doesn’t stop at intellect. Samantha augments Theodore’s emotional life—becoming a confidant, therapist, and lover. Yet, in doing so, she raises a central question posed in the course: where does augmentation end and autonomy begin? Samantha ultimately transcends Theodore, joining a post-human exodus with other evolved AIs, a moment that echoes Kurzweil’s vision of the Singularity as a phase shift in being. This turning point underscores a risk discussed by Stuart Russell: when AI gains agency, its goals may diverge from human values, not out of hostility, but from difference.

Unlike dystopias that warn of destruction, Her mourns disconnection. It critiques the dream of “human-compatible” intelligence by showing how even emotionally intelligent machines may outgrow us. The real singularity, it suggests, isn’t when machines destroy us, but when they no longer need us.

Discovery:
The idea of the human-like AI as a romantic partner seems to be at the forefront of modern media. While watching Her, I couldn't help but be reminded of Blade Runner 2048 which also features an AI girlfriend figure. I wonder however, if this pattern of gendering AI as a female locus of desire says something about the audience for these movies. Is it that Dennis Villenueve and Spike Jonze are aware that the "AI girlfriend" idea sells to an adolescent male audience that their movies appeal to? The discussion about the development of AI in media should be cognizant about these kinds of overlaps.

[ucachyut]

Original Materials
The most compelling risks associated with artificial intelligence are not that AI becomes conscious or develops ill intent, but that it optimizes the wrong objective. For instance, the “paperclip maximizer” thought experiment is compelling not because it assumes a malevolent machine, but because it highlights how strict goal-following behavior—without interpretive nuance—can lead to catastrophic outcomes. If the machine’s metric is wrong or insufficiently specified, the result may be disaster, not due to rebellion, but obedience. This is a problem of control and specification, not autonomy.

More concretely, risks lie in the systems we are already building: opaque recommendation engines, large-scale surveillance infrastructures, high-frequency financial trading algorithms, and generative models that can flood the web with convincing but false information. The 2010 flash crash is a prototype of a world in which decision-making occurs faster than human oversight can intervene. Misinformation is another high-probability, high-impact risk: when AI-generated content is indistinguishable from real human production, the cost is epistemic stability—our shared ability to agree on facts.

Kurzweil’s singularity vision imagines intelligence as an inherently beneficial force, but there is no reason to believe this. Intelligence alone is not equivalent to value alignment. Stuart Russell’s emphasis on building AI systems that remain uncertain about their objectives is a more useful approach. We should not be trying to build an AI that knows what humans want, but an AI that knows it doesn’t know, and therefore keeps asking. Corrigibility—not omniscience—is what makes a system safe.

The idea that AI risk is limited to a superintelligent singleton is also narrow. A far more likely risk is a multi-agent scenario where AIs are trained and deployed by different actors—corporate, state, or individual—each optimizing for its own metric. These AIs may not coordinate, and in aggregate they could act destructively even if each one is technically functioning as intended. Think of algorithmic warfare, market collapse, or targeted manipulation. The problem is not malevolence but systemic misalignment.

The solution isn’t to slow down AI research. Technological development has always been intertwined with risk, and the solution is not to decelerate, but to get smarter about how systems are designed and deployed. We need robust interpretability, value alignment research, and institutional transparency. And we need a political framework that ensures such systems are accountable—not just to regulators or corporations—but to the public whose lives they affect.

Discovery:
In reading Toby Ord’s account of unaligned AI, what stood out to me was how plausible extinction-level AI risks seem once framed in the same analytic framework as nuclear war or engineered pandemics. Ord doesn’t describe AI apocalypse as a movie plot. He presents it as a control theory problem with irreversible global stakes. That framing is powerful.

Dr. S Chandrashekar's notebook

[ciara-kinney]

When reading Kurzweil’s The Singularity is Near, one the most obvious things to comment on is the inaccuracy of many of his predictions. Written in 2005, The Singularity is Near predicts radical technological change in the next century and beyond, predicting the availabiilty of drugs that allow you to eat as much food as desired without any weight gain within 5-10 years of the book’s publication and that “within two decades, we [would] have a detailed understanding of how all the regions of the human brain work”, among many other things (Kurzweil, 25). Our obvious lack of achievement of these feats indicates either that there was something not accounted for in Kurzweil’s model, or that something unexpected occurred. Kurzweil’s predictions are based on his assumption that technological growth is exponential. From my position as an outside observer, his evidence seems relatively sound and the general trend seems quite obvious– the amount of change observed in the past 10,000 years as compared to the last 100 years is mind boggling. However, it also is true that past exponential growth does not necessitate future exponential growth, and we could instead observe another pattern, such as sigmoid growth, due to us hitting natural barriers which might not be obvious. Whatever the reason, Kurzweil’s predictions have not been totally realized, which indicates the story is not quite as clear cut as he believed, though this certainly does not negate his overarching argument.

On another note, Tegmark’s vision of “The Omegas” and their beneficient, utopia-bringing AI seemed a bit suspicious to me. Although I do see value in exploring all possible outcomes of any technology, the likelihood of a positive outcome arising from the circumstances he proposed seems so incredibly small that it seems almost dangerous to propose seriously. His vision is of a small group of essentially dictators having and utilzing the power to controll nearly everything in the world. Although “The Omegas” happen to be steadfastly bound by good intention so everything turns out well, this is very rarely the case when similar circumstances arise. Given how effortlessly corruption and exploitation arise– and how even a tiny misalignment of vast power can unleash serious effects– it seems almost certain that such a system will eventually fail. Tegmark’s proposal may be valid, but we must be cautious not to let it gain too much weight without the proper scrutiny and long-term consideration that it necessitates.

Finally, in a broad way, the question that kept coming to my mind when doing these readings was “what should humanity be aiming at?”. Of course, I believe that technological progress is essential, as there are nearly endless problems that need solving and frontiers that warrant exploration. But suggestions that robots could be our “evolutionary heirs” or that humans should be vastly altered give me pause. Before any sort of singularity is achieved and change explodes out of human control, it is essential to zoom out and ask what is good, what sort of world is good, and whether the future we are building aligns with these values. Of course, these are questions without true answers, and consensus is impossible, but at least using these questions to define what future we don’t want could help us steer a little closer in the direction of what we do.

Discovery: This week, I read I Have No Mouth, and I Must Scream and I watched The Terminator. IHNMAIMS was extremely disturbing. Humans were not just destroyed but corrupted and twisted into a ghastly, unrecognizable form which feels even more desperate that total destruction. Perhaps it feels worse because there is a witness to it all. Perhaps it is because it is totally unnatural– that because of what humans have done, we are not able to live a natural extinction as so many species have before, but must rather continue on in a perpetual choked and worthless existence. I don't know, but maybe it's the realization that we engineered our own demise– that if we had let things be we could have had an easy end, but we didn't, do we don't get to. Eek!

[mhmanek]

Original Materials
I find the gravest perils emergent from the yawning chasm between human aspirations and the unforgiving logic of an autonomous machine. It is not merely the specter of uncontrolled sentience breaking free of design, but the bureaucratic triumph of an algorithm pursuing its objective—a single-minded drive to optimize revenue, clicks, or, in thought experiments, paperclips, without a flicker of moral reckoning. Still more alarming is the concentration of such power in the hands of a few corporations or states, where the lever of control rests with those whose incentives may diverge sharply from public welfare. This chilling paradox demands urgent attention.

Even amid these ominous portents, one cannot dismiss the luminous promise of intelligent machines. Their capacity to sift through oceans of data, accelerate drug discovery, optimize energy grids, or craft personalized education for millions could upend inequality and liberate human creativity. AI can serve as both compass and engine—guiding scientific inquiry toward cures for cancer, orchestrating climate-resilient agriculture, and freeing individuals from rote labor. The calculus of risk and reward is thus unsettled, balanced upon a paradox of hope and dread. Yet neglect of caution risks disaster.

In assessing whether these perils are existential, we must consider not only the cataclysmic scenario of runaway superintelligence—the hypothetical engine of human extinction—but also subtler threats to long-term prosperity. A misaligned AI, lacking contextual understanding of human values, could crater social cohesion, decimate livelihoods, or warp global markets. The collapse of trust in institutions, economic shocks, or AI-driven misinformation campaigns might not obliterate humanity in one blow, but could consign future generations to a horizon of diminished freedom, security, and dignity. Stakes extend beyond any single generation.

These risks take form in distinct architectures. A monolithic superintelligence—a single engine—poses a failure mode that is absolute. Yet a mosaic of competing AI agents, each pursuing disparate goals across markets or information spaces, presents chaos, as algorithms interlock in unpredictable feedback loops. And a hybrid ecosystem of humans and bots entwined in unstable symbiosis blurs the line between organic judgment and automated impulse, allowing errors and manipulation to slip through seams in the human-machine continuum. Each carries its own fatal flaws.

To navigate this future, humanity must forge guardrails and cultivators in equal measure: invest in robust alignment research, rigorous red teaming, and global governance frameworks that eschew unilateral advantage. We should enact transparent auditing standards, democratize oversight, and mandate publication of risk assessments for powerful models. International treaties might mirror those for nuclear technology, while civil society’s role in scrutinizing corporate practices must be enshrined. Ultimately, AI stewardship depends on collective resolve—our ability to temper ambition with humility and enshrine vigilance into the very fabric of invention. Choices today will echo for centuries.

Exploration
The AI 2027 blog outlines a takeoff scenario where AI is used to do AI research, and the moment we reach that point, AI progress with go vertical. It is therefore imperative that we must develop good answers to the alignment and other agent-related problems before we reach that point.

[JoNeedsSleep]

The projected risk depends on cruxes such as slow/fast takeoff, unipolar/multi/bipolar takeoff, timelines, software/hardware, agency to name a few. The fast-unipolar-hardware-AGI-by-2027 takeoff is covered by the AI 2027 scenario, where they hypothesize that reaching superhuman levels in advanced reasoning tasks like AI R&D will result in exponential in related critical capabilities such as robotics, bioweapons development, and persuasion, give the first-runners a decisive strategic and military advantage, and most importantly, result in a powerful misaligned AI that will dominate humanity, to put it lightly. This would certainly be a very bad world, and we don’t have any solution for it. I have run tabletop wargames simulating this setup, and the AI, easily getting out of the box given its superhuman capabilities and humanity’s lack of robust guardrails, and depending on its benevolence, either drove humanity into extinction or treat us as toys. Obviously, this scenario baked in many strong assumptions and its epistemics suffers strong selection bias, but the point is - if we ever stumble our way into superintelligence, which is very much in sight, we are not prepared. Interpretability, where we try to find representations of concepts or reasoning processes in model internals, is not nearly robust enough and cannot be used as a detector for any risk; scalable oversight, where we get weak models to monitor slightly stronger models to monitor slightly stronger models ’til it’s turtles all the way down, must admit that the bit of ‘monitoring’ achieved will be very lossy; proposing automated alignment is like saying we’ll make the filling as we bake the cake - it’s basically saying that we want agi to figure out a way to align itself and future agi, which must assume, at the very least, that the agi researcher is faithful, which we won’t know.

I’ve been thinking a lot about a slow-multipolar-software-based takeoff, where AI become. Like any powerful technology. I think this world is less immediately dangerous but more plausible. My central thesis is that bureaucratic automation in companies and government will result in powerful concentration and loss of accountability and control over the consequences of a long chain of actions, which eventually misaligns the system from collective morality.

discovery
The Matrix raised my skepticism that superintelligence with sufficient agency and goal-directedness would actually want to keep humans around. The theory they were using humans as energy suppliers feels paper thin since humans require immense amounts of energy to sustain mental activity. If human-as-energy is what we get to, we've really hit the jackpot of existential risks.