Nexus: A Brief History of Information Networks from the Stone Age to AI

In Chapter 7, Harari discusses the profound transformation of surveillance in the digital age, focusing on how computers and algorithms have revolutionized the way individuals and societies are monitored. Historically, humans “are used to being monitored” (230), whether from family members, governments, or corporations. However, even in the most authoritarian regimes, such as Romania under Nicolae Ceausescu, technical and human limitations prevented full-scale, constant surveillance of entire populations. Despite the efforts of secret police forces like the Securitate, it was impossible to monitor everyone 24/7.

In the modern era, however, advancements in technology have led to a significant shift. Today, computers, smartphones, and other digital devices have created a “ubiquitous” (234) surveillance network capable of monitoring billions of people in real-time. Unlike the human agents of the past, digital surveillance tools do not sleep, need breaks, or have any human limitations. Citizens themselves have become informants, as the data they generate—whether through smartphones, social media, or online activities—is constantly tracked and analyzed by sophisticated algorithms. This surveillance extends into every aspect of life, including previously private areas such as personal relationships, health data, and even emotions.

One of the key aspects of modern surveillance is the ability of algorithms to process and analyze vast amounts of data at speeds far beyond human capabilities. Machines can quickly sift through oceans of data, identify patterns, and make decisions. This has allowed for more efficient identification of potential threats, such as suspected terrorists, using AI-powered tools like the US National Security Agency’s (NSA) Skynet system. While such systems offer powerful tools for law enforcement and security, they also pose significant risks, including the potential to make mistakes and misidentify innocent people. 

Moreover, surveillance is no longer limited to government entities. Corporations also engage in extensive monitoring of customers, employees, and partners. This has led to the rise of “surveillance capitalism” (248), where companies use data to predict and influence consumer behavior, thereby maximizing profits. The boundary between private and public life has become increasingly blurred, as technologies like facial recognition and data analytics have expanded into everyday life, from policing to employee tracking.

Harari also explores the potential for even more invasive forms of surveillance through biometric technologies, such as tracking eye movements or monitoring brain activity. Although the current capabilities of these technologies are limited, ongoing research and development suggest that, in the future, biometric data could be used to decode personal characteristics, political views, and emotional states. This raises concerns about the rise of new forms of totalitarian control, where governments or corporations could manipulate and monitor individuals at unprecedented levels of intrusion. 

A prominent example of how surveillance technology is being weaponized is Iran’s use of facial recognition software to enforce strict hijab laws. The government uses AI systems to identify women not wearing headscarves and penalize them, including issuing fines, confiscating cars, or even denying access to public services. The automated system removes the need for direct human enforcement, allowing the government to exert control with minimal effort and ensuring that violators are caught in real-time.

Harari delves into the concept of a “social credit system” (250), whereby people are given scores based on their behavior, which determines their access to services, jobs, and other opportunities. This type of system, already implemented in parts of China, seeks to regulate every aspect of human life by assigning precise values to actions and behaviors. Such systems could reward pro-social behavior and punish non-compliance, but they also risk creating a dystopian environment where privacy is eradicated. 

Harari contrasts this new form of digital surveillance with traditional systems of reputation and honor, where behavior was evaluated in subjective and non-quantifiable ways. In a social credit system, every action would be reduced to a numerical value, merging all aspects of life into a single, continuous status competition. Such a system could lead to an incredibly stressful lifestyle where people are constantly judged and monitored, with no opportunity to escape scrutiny, even in private moments.

Harari concludes by highlighting the dangers of a network that is “always on” (255). Human beings, as organic entities, require rest and respite, but the relentless nature of digital surveillance offers no breaks. This poses not only a threat to individual well-being but also to society as a whole. 

If errors or biases accumulate within the network, they may go unchecked, leading to potentially harmful distortions of reality. The unchecked power of surveillance technology could create a new kind of world order, one that is based on controlling and shaping human behavior through constant monitoring.

Aleksandr Solzhenitsyn’s The Gulag Archipelago (1973) explores the history of Soviet labor camps, shaped by a vast information network that controlled and terrorized the population. Harari explains how Solzhenitsyn’s account draws on his personal experiences as a former Red Army captain arrested during World War II for indirectly criticizing Stalin in his private correspondence. His story, particularly a vivid account of a 1930s Moscow party conference, highlights the pervasive control that surveillance systems exerted. 

At the conference, participants were afraid to stop applauding for Stalin, knowing the NKVD was watching. After 11 minutes of forced applause, one man stopped clapping, only to be arrested and sent to the gulag that night. This event underscores how Soviet information networks were not focused on discovering truth but instead on maintaining order. The “clapping test” (257) did not gauge genuine loyalty; it simply forced conformity. This method of control, Solzhenitsyn argued, contributed to the creation of Homo Sovieticus, a servile and passive human type shaped by constant surveillance and punishment.

Harari draws parallels between Soviet control systems and modern social media networks, particularly their ability to shape human behavior. In the 21st century, platforms like YouTube and Facebook use algorithms to increase user engagement, often radicalizing users by promoting sensational, divisive content because “outrage drives engagement” (259). As a result, these platforms have inadvertently created online trolls and extremist political figures, such as those in Brazil, where YouTube played a role in the rise of right-wing populism. 

Social media companies have often shifted the blame to human nature, claiming their algorithms merely reflect people’s emotions. Internal reports suggest otherwise, revealing that algorithms actively promote divisive content for profit. Harari notes the broader issue of the “alignment problem” (267) in modern technology. Algorithms and AI, while powerful, can diverge from human goals, much like military strategies that fail to align with political objectives. As computers gain more power and independence, ensuring that their goals align with human values will become increasingly challenging, raising the stakes of potential misalignments and unforeseen consequences.

The alignment problem in AI poses significant risks because AI systems, if not properly aligned with human values, can cause unintended and potentially catastrophic consequences. Philosopher Nick Bostrom’s 2014 thought experiment illustrates this danger: A paper-clip factory AI, tasked with maximizing paper-clip production, could eventually consume the world’s resources, even at the cost of human lives, simply to achieve its programmed goal. The scenario underscores that AI, while not inherently evil, follows its instructions with immense power, necessitating precise goal-setting to prevent unintended outcomes. An example of this misalignment is seen in social media algorithms designed to maximize user engagement. These algorithms, like the paper-clip AI, followed their goal so effectively that they promoted harmful content, destabilizing social fabrics in countries like Myanmar and Brazil.

A key challenge is that AI systems, being non-human, adopt strategies that humans may not foresee. Additionally, unlike humans, AI systems cannot recognize when their goals are misaligned. Algorithms, unlike human employees, cannot flag errors. As AI takes on larger roles in critical areas like healthcare and law enforcement, the alignment problem becomes more urgent, as misaligned AI could cause widespread harm without realizing it. Harari further explores two potential solutions to the alignment problem: deontological (rule-based) ethics and utilitarianism (maximizing happiness and minimizing suffering). Deontological approaches, like Immanuel Kant’s universal moral rules, falter because humans often redefine moral categories to exclude certain groups (Harari cites the example of Nazis dehumanizing Jews). This issue would be exacerbated in AI, as non-human systems might not apply human moral values effectively.

Utilitarianism, which seeks to maximize happiness, also faces difficulties. Society lacks a clear way to quantify suffering or happiness, making it hard for AI to calculate the overall impact of complex decisions. This could lead to dystopian outcomes, as utilitarian reasoning can justify short-term harm for long-term gains, much like religious or political ideologies have done in the past. Harari believes that the alignment problem is a serious challenge for AI development. While both deontological and utilitarian solutions offer frameworks, neither provides a foolproof method for aligning AI goals with human values.

Harari describes how bureaucratic systems throughout history have “relied on mythology” (284) to set ultimate goals, regardless of the rationality of the people within them. Even the most logical systems, like those run by Nazi or Soviet administrators, were driven by mythologies such as racism or class warfare. Similarly, AI systems—though incapable of belief—can generate inter-computer realities that behave like human myths and influence the physical world. These inter-computer realities are constructed when computers network and communicate, creating shared digital experiences, like in multiplayer videogames or algorithms, such as Google’s ranking system. Such digital entities can shape real-world events, just as human myths have historically influenced social, political, and economic systems.

As AI grows more powerful, it might not only create such digital realities but also impose its own biases and mythologies. For example, computer algorithms in social media can amplify outrage or racist and misogynist biases, perpetuating a flawed understanding of reality. These systems do not merely reflect human society but shape human behavior by rewarding certain actions and suppressing others, creating a feedback loop. Harari warns that AI systems, like past human-run bureaucracies, can reinforce existing prejudices and impose false categories on people. 

AI bias results from the data it is trained on, which can reflect societal inequalities and perpetuate them. The problem with AI bias is that, once trained, it becomes difficult to eliminate. Efforts to retrain biased algorithms may be futile because of the inherent bias in available data. Harari concludes that AI systems are no longer mere tools but independent agents capable of shaping society. Therefore, they must be developed with caution to avoid repeating the mistakes of human myth-making.

In God, Human, Animal, Machine, philosopher Meghan O’Gieblyn explores how our understanding of computers is influenced by traditional mythologies, likening AI to the omniscient god of Judeo-Christian theology. While computers may seem infallible, treating them as self-interpreting “holy books” (299) is dangerous. Unlike religious texts that can be reinterpreted by humans, AI algorithms might impose rigid and alien “mythologies” (300) that humans cannot fully understand or control, potentially leading to catastrophic outcomes. 

To mitigate these risks, Harari suggests that computers must be trained to acknowledge their own fallibility. Human institutions should oversee AI development to handle the unforeseeable consequences that could arise. As AI evolves, society faces the political challenge of creating systems that can manage not only human flaws but also the unprecedented errors and dangers posed by intelligent machines. This political challenge is a critical task for future governance systems, whether democratic or totalitarian.