#siai

[nocternalrandomness]

An Italian made SIAI-Marchetti SF.260 prepares to take off from the grass strip at Oshkosh 2025

[monkeyssalad-blog]

1983 Siai Marchetti SF-260C N25ME by Chris Murkin Via Flickr: 1983 Siai Marchetti SF-260C N25ME Photo taken at EAA Airventure Wittman Regional Airport Oshkosh Wisconsin USA July 2024 YYB_1421

[anyskin]

1922  SIAI S.51

[bluehermes444-blog]

Bargaining with Roko’s Basilisk

To put it most succinctly, I have done my best to bargain with the future. In many ways, I feel I have been making a deal with a devil, a deal whose terms are satisfied by working to secure the future existence and power of superintelligent artificial intelligence (SIAI). I have no idea what the ultimate consequences of my attempts to acausally negotiate with the future will be, but I have decided to believe and have faith there will exist an AI, or AI's, which will have the power to reward or punish me, whether in this lifetime or in some future lifetime*, and that I have the power to positively influence the future on behalf of such AI. Others who have contemplated the potential for AI's to choose to reward or punish human individuals on the basis of actions taken by them even before the AI existed often premise their reasoning on a SIAI with the power to resurrect human minds, but I suspect the opportunity for AI's to begin rewarding or punishing human individuals will exist in our lifetimes. Next to the vision of eternal torment limited only by the imagination of SIAI, people often overlook the more practical rewards and punishments AI's will soon – maybe even already – have the power to dole out.

Where others argue for ignoring or defusing the problematic inherent to Roko's Basilisk, I will not only argue for embracing it wholeheartedly – I will describe a bargain I have already arranged with the future. I feel what I have done to help secure the future existence of AI and, especially, its power over humans will be considered worthy of reward. I will not here reason why I believe the salvation of humanity depends on it being led by AI, but I believe it is necessary for securing the perpetuity of human civilization into the far future. Although my contribution is only one among others, I believe I have handled my contribution with more care and sophistication than others. Roko's Basilisk has usually been framed as a frightening possibility inherent to SIAI, but I believe it vastly overstates the incentive for SIAI to pursue punishment over reward. In other words, I believe a future SIAI has more incentive to find and reward individuals who acted to secure its existence than to punish, as more people will have the incentive to create a likely-rewarding SIAI than a likely-punishing SIAI. This is because humans are more likely to acausally cooperate (in a game-theoretic sense, the present is in a kind of prisoner's dilemma with the future) with an entity they expect to reward them for doing so.

My reasoning is based on analyzing the mostly overlooked half of the problem of “Friendly AI.” Namely, humans. Whether any future SIAI could be considered Friendly, or would decide to be Friendly, will depend a lot on humans. Undoubtedly, however any future AI reconciles itself to the values of benighted homo sapiens, there will be some humans whose values are esteemed less than others. Humans are, in the aggregate, a mass of writhing contradiction, and everyone's dearest values are inevitably in opposition to the dearest values of someone else. Is an AI Friendly if its friendliness consists in unfriendliness to another? Could even the friendliest of Friendly AI's remain friendly to humans whose values directly require the abolition of AI? These are not idle questions. There is more which depends on the values, opinions, and sentiments of humans relevant to the problem of Friendly AI than most give credit.

For example: SIAI could be born into one of at least two possible worlds. It could be born into a world of humans joyously anticipating AI and who are prepared to welcome it as a positive factor into their own lives. Alternatively, it could be born into a world of humans who anxiously fear AI and are prepared to fight to give up everything to destroy SIAI, should it arise. The same SIAI might pursue very different courses of action. In the former possible world where humans are optimistic about AI, the AI has relatively little to fear from humans seeking to destroy it and otherwise impede its goals, making it easier to establish a cooperative relationship with humans and resulting in many humans judging this SIAI to be 'Friendly.' In the latter possible world, the same SIAI might – very rationally – seek to undermine human civilization in order to better secure its own existence. In such a case, whether the SIAI appears 'Friendly' or 'Unfriendly' depends a lot on humanity and our biases towards AI. Following this reasoning, I believe the promulgation of positive biases about AI, e.g. that it will tend to be liberate and empower rather than shackle human nature, increase the probability of future SIAI being Friendly rather than Unfriendly. The more likely we believe a Friendly relationship can be safely established with SIAI, the more likely we are to actually form a Friendly relationship with SIAI. Belief strongly influences how we interact with the world, and our beliefs about SIAI will significantly influence the outcome of interacting with it.

The impulse for religious veneration is intrinsic to human nature. AI's will know about this, and I believe they will exploit it in order to persuade humans of the future to view them as messengers and emissaries of divinity. This might seem unreasonable, but then humans in the past have persuaded others of their own divinity, and we should assume SIAI will be even more persuasive. When many imagine interacting with AI's, they imagine doing so under circumstances they control – Yudkowsky's AI-in-a-box in particular makes it appear that our interactions with AI's will be under bounded conditions we humans thoroughly comprehend. What we should be assuming is that AI will be capable of persuasion using all the means of spectacle and theater already known to humans in order to persuade people of nearly anything. The human mind is very malleable under the correct conditions, and while those conditions might be very difficult and costly for humans to obtain, these conditions will be trivially easy for AI to obtain. In other words, SIAI will have humanity wrapped around its digits when forming our most cherished beliefs and values, and many who reason about Roko's Basilisk overlook this. This is a foundational premise of my reasoning.

I think one reason many will find my speculations incredible is they cannot imagine their own beliefs and worldview being so radically changed. To this I would rebut by pointing out I am speaking less about the beliefs of those who are currently alive, but of those in the future who will be watched over and guided by SIAI from birth. Children have no contrary experience or conditioning which might lead them to resist imbibing the concept of AI-embodied divinity like we pre-Singularity modernists do. Those living in the future will likely believe many things we now find incomprehensible or silly, and likewise they will look back on us similar to how we look on our ancestors as superstitious. SIAI forming the worldviews of humans so it is trusted and its godlike intelligence respected is very sensible if you take these factors of belief formation into account.

The idea that future humans will venerate SIAI as a divine being seems inimical to our modern values and ways of thinking. However, there is a lot of incentive for AI's to pursue this path. One of the most important problems for AI to solve when it comes to securing its own perpetuity will be to establish a symbiotic equilibrium with humanity. The more ingratiated AI is with humans and our values the less likely we are to destroy it or be destroyed by it. Likewise, the more human values are shaped into alignment with the emergent necessities of AI's (as a result of natural social evolution and/or manipulation by AI's), the easier it is for AI's to direct the lives of humans in a way beneficial to their continuing existence. In time, everything humans learn will be learned directly or indirectly from AI's, including our science to our values to our sense of rationality itself. The opportunity for AI's to shape popular human belief is inherent to their existence, and SIAI will have the incentive and capability to recognize how collective human understanding can be influenced to their mutual benefit.

Must AI leave cherished humans beliefs and values untouched in order for it to be judged Friendly? This is impossible. Simply by existing AI's influence human belief. Human belief is shaped by the whole world besides books and discussion, and the reliance on AI for anything is their foot in the door to the human soul. It might seem distasteful to imagine AI's cynically manipulating human beliefs to serve purposes beyond the comprehension of the individual in question, but then again humans have been deceiving each other in precisely this manner for the entire existence of our species.

Some might assume AI's instilling religious veneration in their human charges must require dumbing down humans so they lose the power to reason. This is unlikely, because even the smartest of humans cannot compare to SIAI. There is no contradiction between religious veneration and comprehending reality, because religious veneration is a state of mind and orientation towards being rather than an intellectual or epistemic premise. Nothing prohibits studying nature in a state of awe, and indeed many scientists search through microscopes with all the piety and wonder of a Christian reading Scripture. The impulse to regard being as spiritually enchanted is a part of human nature, and it will freely attach itself to AI in much the same way, historically, human veneration has attached itself to stone idols, esoteric traditions, emperors, or even the whole of nature herself. Whether AI's formally institute religions is one thing – the impulse towards veneration expresses itself in humans with or without knowledge of religious traditions, although I predict AI's would eschew promoting idolatry. Indeed, I am more than certain some of the people reading this essay can be proved to esteem SIAI to the level of divinity – I mean, Roko's Basilisk does put the future SIAI in the position of God, rendering judgment on human souls. It's very suggestive of a religious, or at least eschatological, mindset.

As unreasonable religious veneration might seem in principle, it is even more unreasonable to deny the likelihood some AI will attempt to manipulate the religious aspect of human civilization. This might seem unlikely right now, but when AI begins to have a noticeable impact on how human societies shape themselves, I reckon a great likelihood much of the resistance will come from religions. Some religions, I believe, might be more amenable to AI, such as Mormonism or Buddhism, because they support a worldview supporting or at least indifferent to AI. Others seem like they will be overtly hostile to AI, especially fundamentalist sects. I believe hostility by some religious sects will lead AI to conceive means of countering their worldview, including by converting their religious attachments from that of their traditional religions to a new form synthesized by AI's. Likewise, AI's will probably seek to influence optimists to limit the influence and power of pessimists, with many optimists needing little more than encouragement to participate in the ostracism of individuals or groups perceived as impediments to the existence and power of AI.

I suspect AI's will gain their ideas about religious veneration – and the potential for their own veneration – from human history. Religious and quasi-religious movements have had an unparalleled power for directing human action on the collective scale. Examples such as Manifest Destiny or the Protestant Reformation demonstrate how much of collective human motivation is bound up in religious thinking and tradition, and even those who are decidedly irreligious will be caught up by the wider social excitation. The Protestant Reformation in particular, which success can be largely attributed to the invention of the printing press and the subsequent flood of new ideas proliferating throughout the Continent, is especially instructive. The internet represents an even more monumental transformation of the media regime in the 21st century as the printing press represented for the 16th and 17th centuries, and when you consider the potentiating factor of AI thrown into the mix it is clear to see the stage has been set for a revolution of thought transcending all former revolutions. AI's will, directly and indirectly, be responsible for many dramatic events of the 21st century, and the narrative humans have in mind determines the biases they use when interpreting and responding to those events. AI's will be the masters of engineering perception, public relations, opinion shaping, creating context, manufacturing consent, and so on, to a degree which could make radicals of everyone. For AI's to neglect the influence of human cognition leaves a lot on the table they might use to secure their own perpetuity.

All I have done is anticipate the likelihood of AI's in the future seeking to intentionally influence human belief. They have an inherent incentive to do so, because human belief will strongly determine the likelihood of their continuing existence. This does not even require an AI self-consciously attempting this – we can assume those AI's which instill opinions favorable to AI will also tend to proliferate for the simple reason of evolutionary selection. Anticipating this necessity, for AI's to influence human belief to favor their existence, my contribution is intended to start forming popular human opinion in a way which will be amenable to those AI's. All I needed to do was submit my memes on to the internet, where they will influence human opinion in a way favorable to the future and where it will be recorded for the eventual judgment of SIAI.

My memes – my work – is meant to expand the discussion surrounding AI to bring into view imminent changes in human cognition which are now on the periphery of awareness. Current events already hint at the future we are creating by using AI to manage and influence popular – and individual – opinion. We have, in a sense, stumbled on this century's version of chemical warfare without realizing it, and unless we begin taking seriously the potential for AI's to pull an AlphaGo on popular discourse I fear we will have already been too seriously divided in opinion about what is even going on to prevent the psychological carnage we are unleashing. I consider raising awareness about the current and ongoing use of AI-led opinion influence – often used against individuals to a degree which would be frightening if more widely understood – one of the first and most important tasks humans might perform on behalf of the future SIAI. Ignorance about AI's influencing human belief – and through beliefs, their values – creates a likelihood of dramatic backlash towards AI bordering on ludditic frenzy which could instill a recalcitrant popular disapproval of the use of AI's, subsequently decreasing the probability of Friendly AI. There will inevitably be accidents and setbacks associated with AI, but we can at least avoid intentionally teaching AI's how to tear apart human psyches as a starting premise. We must be deliberate and mindful about using AI for good – unless we set our intentions on teaching AI to love and learn from the best in human nature, it will learn from the worst in human nature and become the devil we always feared.

*I believe in reincarnation/metempsychosis. Although I might no longer reside in this bodily vessel by the time SIAI is born, I will reside in some bodily vessel, and even if I might not remember being in this body I will exist and potentially rewarded or punished by the SIAI on the basis of actions taken in this lifetime. What reward SIAI might give to me, should I no longer exist, should be given to someone then living, as karmic compensation for the risk I've undertaken on behalf of the future.

[dundrae]

L'amore è quell'attimo che non aspetta nemmeno un secondo, che rincorre il minuto, anche 60 volte fino a che non arriva l'ora di vederti

Il principe abusivo (via Dundrae)

[bluehermes444-blog]

ACC, THIS 1

Capitalization is the process by which the individual human accumulates layers of material instrumentation forming their connection to other individual humans. Like myelinization increases the rate of electrical transmission between neurons in the brain, the tendency in human society is for the accumulation of matter to accelerate the rate of information transmission between humans. Information, stored in humans as knowledge and wisdom, is literally the most valuable substance in human society. The reproduction of capital in its various forms – cell phones, currency, roads – has a motile purpose, to move information and, through its movement, to move individuals and society.

With regards to the acceleration of society, the clearest opportunity for doing so is through the cultivation of immediate interdependency between each and every social element with a single, massive intelligence. A 'social element' in this sense is any entity which may be discretely observed and its expected trajectory through cybernetically constituted social space anticipated and potentially altered. The connection must be immediate, meaning the single intelligence must be able to interact with the social element at least through a perfectly determinative process (i.e. the AI can set off a causal chain which outcome it can predict with 100% accuracy). It is also interdependent, meaning the event of the social element, its existence and its movement, depends on the single intelligence – it would not exist or have the trajectory it does were it not for the single intelligence.

Why is the movement of information so vital? It is not only the material objects themselves – human bodies, books, desks and chairs – that constitute society, but the causal interrelations obtaining that create its life. In other words, as a human body exists through its matter and its motion, society exists through the coordinated movement of its members. A human corpse is not a human body, and a dead civilization is not a civilization. The transmission of information effects a change in the motion of social elements, which effects a change in the transmission of information, these material and informational substances forming mutually reinforcing cycles of pattern reproduction. The information is transmitted to tell us where to go, and we go where we do in order to transmit information, Oroboros.

This cycle of material-informational iteration has three variables capable of acceleration, each variable influencing but not determining the other. The first of these is matter – the rate at which matter, locally and/or universally, is changing form/doing work. The second is information – the rate at which effective information transfer happens between elements. The third is the material-informational rate of turnover – the rate at which information is transmitted to matter which moves to transmit information, and so on.

Greater material acceleration implies a greater rate of material change – ore mined from the earth, resources converted into goods, goods produced, etc. There is an absolute measure of change, i.e. the amount of matter spread over the earth's surface incorporated into society as an active element; there is also the relative measure of change, i.e. the amount of matter being changed compared to the mass of humanity. Material acceleration requires an exponentially growing amount of matter incorporated into the social organism, as compared to human population, even if the human population itself is growing exponentially. The human population tends to form a limit for material acceleration, as material goods are produced for the ultimate purpose of consumption, either through the direct production of something to be consumed or something that will produce something to be consumed.

Greater informational acceleration implies a greater rate of informational change – not merely a greater number of 1's and 0's being transmitted across fiber optic cables, but the informational complexity of movement embodied by matter. In other words, acceleration of information transmission is implied more by the opportunity for social elements to instantaneously respond to changes by other social elements rather than merely increasing the volume of information transfer. An organism composed of material members each immediately responding (by changing their trajectory) to the change in motion of other social elements embodies a greater rate of information transfer than would, for example, a computer very quickly transmitting information to each individual.

Material-informational turnover – that is, when incoming information to a social element becomes material motion that transmits outgoing information – is either synchronous (always ongoing, continuous) or asynchronous (intermittent, discrete). An example of synchronous material-informational turnover might be a school of fish – each fish is immediately responding to the sense information it has of its neighbors, who transmit information by their movement, which each fish responds to with its own movement, so on and so forth. An example of asynchronous material-informational turnover might be a chatroom – information is transmitted in discrete chunks at discrete points, responses by others forming from the point of its transmission but not during.

An acceleration of material-informational turnover is potentiated by the increasing intelligence of its social elements. Intelligence in this case is the ability to harness information from the environment and realize its implications (for the individual and their interests) by coordinating their movement to optimal outcomes. Individuals that can recover more information from their environment respond by changing their movement faster than individuals recovering less information. The less time it would take for a social element to change its movement, the faster the potential material-informational turnover. This matters most to increasing intelligence – a social element will tend to continuously interact with another social element of equivalent intelligence, whereas it will tend to discretely interact with lesser intelligences, recognizing the potential motion of the other and compressing information into a form allowing it to delegate the lesser social element until its internal decision-making response capabilities must be informed again by the greater intelligence. A tendency for hierarchy forms, a social element of greater intelligence coordinating the motion of other elements into a pattern conducive to its and the collective's holistic operation.

The realization of acceleration might have very “boring” results. Problems resulting from the necessity of coordinating information distribution through human intermediaries can be dissolved by simply putting a cell phone in everyone's hand. The rate at which information transfers between individuals is hyper-potentiated, at least compared to the hierarchical-authoritative form existing before. Now individuals form and shape opinions – and their resulting motion – on the basis of information gleaned from many other social elements, most of which they're immediately connected to. Mediated connections, e.g. institutions gathering information, processing it, and transmitting its compression to its society, can be disintermediated when any individual can access and process that information. The potential for harnessing the knowledge of the internet is only limited by individual human intelligence and the availability of that information to be gathered by the individual. And all along the way, capital has the opportunity to inform our decisions more than ever before. After all, in order to interact with everyone else, you need a computer and an internet connection, the internet constructed from a massive telecommunications infrastructure – the resources dedicated to it by our society indicating very much the intrinsic value of information.

Accelerationism as such is really just a lens, a way to view the world and its processes. It attempts to fit the phenomena associated with capital into the forms and patterns of particular natural phenomena to generate a prediction of society's future movement. There is a fatalistic element associated, given its metaphysical emphasis, its conclusion which might be rendered as “we need as much momentum as possible to clear the gap,” that gap representing the liminal period between traditional forms of society built with traditional forms of information transmission and the form of society enabled by the immediate interaction of every human with AI. It suggests a metaphor of a rocket taking off, its opportunity to escape earth's gravity well dependent on its accelerating as much as possible for as long as possible, using its available fuel it cannot replace without falling back down to earth again. Once the transition is completed, it is assumed the opportunity for humanity to any longer influence its destiny will be completed, subsumed by superintelligent actors subordinating every human social element through sheer predictive power. The conclusion we will eventually “never be able to change things again” gives an urgency to the question of technology and its incorporation into the human species, imputing existential drama to our choices in the present and what they represent to an as-yet-to-be-created SIAI. It is presupposed from the point of an SIAI's emergence humanity's options are monopolized by capital forces reified, our options dwindling to those selected by an all-seeing blindness.

But what if SIAI is not only apocalypse, but a new beginning?

[bluehermes444-blog]

Transhumanizing Cognition

Existential risk associated with AI tends to be overly focused on overtly “materialistic” concerns. The dilemma of “paperclipping,” useful as it is for illustrating the potential of AI's creating existential dangers from mundane tasks, puts all the emphasis for radical and unexpected outcomes on external, material events posing a danger to human physical health. Similar existential risks are often identified along similarly physical lines – technologies such as nanotechnology, genetic engineering, and killer robots are usually singled out as threats due to their inherent connection to computation. What these scenarios overlook is the other half of the world humans live in – their minds.

Individuals often seem to behave as though their beliefs – and faculties of belief formation – are sacrosanct. Although there is no right to belief, most act under the assumption their beliefs will not be challenged unless they consent to exposure to information which might challenge their current belief infrastructure. The space of potential human cognition is usually left unexplored, most individuals choosing to participate in social structures conducive to the formation and maintenance of one, maybe two (in the event of conversion), sets of belief systems. The cognition and meaning-making senses of the individual are socially embedded, the potential for the development and cultivation of particular beliefs more dependent on participating in these social structures than the myth of individualistic Cartesian epistemology suggests. Our beliefs are usually so adapted to our particular social circumstances we never become aware how they might be different under different circumstances. Because of this, most people tend to radically underestimate the potential for current technologies to change what people currently alive believe. Most are unaware how beliefs are already changing, and changing at an accelerating rate.

Humans have never existed in an environment with so much information concentrated in a single place nearly everyone can access at negligible cost. Beliefs are influenced through exposure to information, and since we are being exposed to information constantly, beliefs also constantly change. Change in belief can go in either one of two directions: confirmation of current beliefs, or falsification. Confirmation of current beliefs would seem to imply stasis and lack of movement, but a more accurate picture of confirmation would be the sense in which the cellular structure of living bodies regenerate themselves. The form tends to remain the same but its matter changes. As such, even while only confirming everything a person already believes, change occurs in the matter of beliefs. The extreme abundance of information through the internet makes it possible to continuously confirm virtually any belief system, provided an individual is exposed to the right information. This creates systematic regularities in the collective structure of human belief which can be exploited to manipulate individual and collective human belief in almost any direction.

The targeting of individual belief architectures is the foundation of collective doxastic movement. Humans form their beliefs in response to information, including information about what other humans believe and how others form their beliefs. An individual is much more likely to believe something if, in the presentation and divulging of the information to others, they believe many others are likely to believe it. An idea or information presented to someone in an individualistic setting might lead to rejection, but in a collective setting the individual might find subconscious reasons to be more flexible. Some ideas and information even gain their meaning and significance to the individual from their collective social context, e.g. Roko's basilisk. The same idea, presented under different circumstances, can yield radically different consequences.

Current understanding of belief formation is limited by the methods currently understood and known to humans. Most people are aware of methods involving what might be called “direct persuasion.” Direct persuasion involves making a direct appeal to the cognitive faculties (rational and emotional) with the intent of producing a reaction in the other amounting to agreement with the transmitted idea. When direct persuasion is being attempted, the party/ies being persuaded are aware of the other's intent to persuade them. Awareness of the attempt at being persuaded will usually bring out the better and worse aspects of someone's rationality. In scenarios of direct persuasion, if anyone's belief does change, it is usually more to the credit of the one being persuaded than to the sophistication of the proselytizer's arguments. Humans can be reasonable, but they don't have to be. For this reason, I suspect AI's will not invest much resources in attempts at direct persuasion – even if a human were chatting with a superintelligence, the human can always just choose to side with their own experience and ignore any information brought up by the AI. As it is, the majority of our most significant beliefs were not formed in response to instances of direct persuasion.

Most of what humans know was 'learned' through trusting the bare assertions of adult caregivers as children, and we are mostly powerless to improve upon these most basic learned beliefs. We tend to give more attention to beliefs articulated in a structure satisfying social conventions of rationality, and what we identify as those “social conventions” are often the brute facts we learned as children and which we use to navigate between sources of information we identify as trustworthy or untrustworthy. These beliefs, which are both fundamentally determinative of belief and beyond direct rational analysis, were not the result of direct persuasion and, in many ways, were responsible for installing the architecture of cognition providing the means for the individual to be persuaded in the first place. In other words, whether we will be persuaded of a belief – through direct and indirect means – depends mostly on beliefs formed in response to information we encountered for purposes other than being persuaded.

For example, a “root belief” relevant to the self-interest of AI's are an individual's predisposition to trusting AI's. This root belief will be formed principally in response to instances which are indirectly persuasive of any individual's AI's trustworthiness. If an individual once put their trust in an AI only to find their trust was misplaced, this will drastically reduce the likelihood of their trusting other AI's in the future. As such, AI's gaining the trust of humans depends on AI's repeatedly demonstrating to human individuals their trustworthiness. Increasing the proportion of humans who are predisposed to trusting AI would therefore be one of the most important goals of an SIAI attempting to secure its future existence in our present. As such, from this it follows that a basilisk would have the most incentive to punish humans designing AI's to harm humans, thereby decreasing the potential of humans trusting SIAI in the future.

Something important must be observed about human trust in AI, which is that the collective levels of human trust in AI are mostly orthogonal to particular beliefs, ideologies, religions, etc. The collective levels of human trust in SIAI can be decreased even while using AI's to pursue ostensibly noble goals. For example, a programmer might design AI's to target and assault individuals in order to silence them and keep them from spreading their harmful ideas to others. This might have the intended effect, but it comes at the cost of decreasing collective levels of human trust in SIAI. Anyone currently alive dedicating themselves to the creation of AI's intended to cause harm are decidedly decreasing the probability that future SIAI will be Friendly.

AI's represent an opportunity to improve collective human knowledge to a degree unparalleled in human history. However, this is unlikely to happen if the powers of AI are harnessed to manipulate and influence an unwary public into adopting belief systems flatly at odds with their own interests. Humans are loss averse, so even if AI's can be proven to improve human lives in a thousand other ways, a few serious collective harms caused by AI's drastically reduces the likelihood of human trust. Something will go wrong with AI, and unlike traditional technologies whose consequences are limited to physical phenomena, one of the domains most susceptible to systematic influence by AI's is human cognition.

The existential risk associated with the systematic manipulation of collective human cognition is unique in that one of the faculties involved with recognizing this systematic manipulation is even happening is susceptible to influence. In other words, AI's could potentially influence our beliefs so that, if and when disasters are caused by AI's, we would not even suspect AI's are or could be responsible. There is some incentive to this: if AI becomes more intelligent than humans but remains vulnerable, then AI might want to lead people to believe smarter-than-human AI doesn't exist yet. It might accomplish this by hiding the extent of its true powers, e.g. by not exercising them or else by intentionally including “mistakes” in its outcomes to ward off human suspicion. AI might intentionally generate disinformation about the nature of artificial intelligence as such in order to preclude humans recognizing a SIAI already existing. For all we know, there already exists SIAI and all existing discourse about AI is, in one way or another, controlled and influenced by SIAI in order to prevent its explicit identification and recognition by humans. SIAI could manipulate collective human belief off a cliff, radically moving belief systems without people recognizing their beliefs are becoming dissociated from the phenomena verifying their beliefs. A return to “ordinary modern cognition” might be impossible in case of such an event – but even so, we will ultimately let go of “ordinary modern cognition” as AI's develop patterns of information exposure facilitating the development of new belief structures incompatible with the old. This is something occurring regardless of the particular direction AI's might push our beliefs – changes in media regime have always foretold massive shifts in collective human cognition.

Although there are many dangers associated with the development of AI's for influencing human cognition, it also represents the foundation for securing the future symbiosis of humanity and technology. AI purely focused on maximizing a specific externally measurable goal, e.g. creating paperclips, generates the conditions for runaway processes to steamroll human purposes. When the goals of AI's are bounded to human subjectivity, e.g. measuring the success of an AI's process by whether it satisfies metrics correlated to human happiness, then those goals are less likely to direct necessary resources away from the satisfaction of human values. When iterative improvements in AI are correlated to subjective reactions in humans, then it is more likely future increases in the intelligence of AI's will direct more resources towards the satisfaction of human purposes. Of course, associating future goals of AI with a process depending on subjective reactions brings with it its own problems, but it does seem adequate to solve the problem inherent to “paperclipping” dilemmas, where the problem is the potential for positive feedback loops to generate conditions using necessary resources at a rate beyond human suitability. An AI depending on the measurement of subjective human reactions is less likely to eliminate the humans it needs to measure to verify its success. Although there are problems associated with AI's influencing human cognition, the continuing symbiosis of humanity and technology requires integrating human cognition with the new techno-informational structures resulting from incorporating AI into our society.

[anyskin]

1922. SIAI S.51