AI Doesn't Need to Understand the World, But We Need to Understand AI

Reprinted from Digital Society Development and Research, for academic sharing only. If there is any copyright infringement, please leave a message for removal.

By Hu Yong
Professor, School of Journalism and Communication, Peking University

In his book "The Edge of Knowledge," David Weinberger opened up new frontiers in our understanding of networked knowledge [1]. His insights into knowledge in the age of artificial intelligence can be summarized as follows:

Humans strive to gain an understanding of complex systems. However, predictions based on "human understanding" are not as accurate as those made by artificial intelligence, even though AI doesn't truly understand anything.
However, given that AI's predictions are more accurate than those based on human understanding, should we abandon the pursuit of understanding and focus instead on building AI that can make decisions for us?
By handing over the reins to predictive AI, we will usher in the next stage of human evolution [2].

Through large language models and the generative AI behind them, as well as the currently burgeoning agentic AI, we have actually stepped into what Weinberger calls "the next stage of human evolution."

Utilizing statistical models, regression analysis, and deep learning techniques, predictive AI aims to forecast future outcomes or trends with high precision. It can be applied across numerous industries—such as healthcare, finance, marketing, supply chain management, and law—to make data-driven decisions, optimize processes, and improve outcomes.

But as Weinberger observed, understanding remains a major issue. As AI systems become increasingly important in decision-making processes across various fields, the demand for Explainable AI (XAI) has emerged.

XAI is a field dedicated to exploring how to make AI systems more interpretable, helping users understand the reasoning behind AI predictions and decisions. This distinguishes it from traditional "black box" AI models, which may provide accurate predictions or classifications but offer no insight into how those conclusions were reached.

Undoubtedly, the key to the future of AI lies in whether we should abandon understanding or commit to building AI that can be understood.

Deep Learning: An Extremely Dark Black Box

With the advancement of technology, we may soon cross certain thresholds beyond which using AI will require a leap of faith. Of course, we humans cannot always truly explain our own thought processes either, but we have found ways to intuitively trust and measure human beings.

In other words, humans build trust not based on calculable transparency, but on an embodied, intuitive understanding: we judge goodwill through companionship, confirm existence through response, and measure responsibility through relationships. Is this possible for machines that think and decide in ways different from humans?

We have never before built machines that operate in ways their creators do not understand. Past tools—from hammers to steam engines, from spacecraft to supercomputers—though intricate and complex, still obeyed causality and followed logic; their modes of operation remained within human control and comprehension. We knew why they moved and why they stopped.

However, with the advent of deep learning, language models, and self-organizing algorithms, the situation has changed: for the first time in history, we possess creations that are harder for us to understand than ourselves.

These machines "think" in high-dimensional spaces, possessing representations and paths we cannot intuitively perceive. Their answers are effective, but explanations are absent; their judgments are precise, yet the reasons remain unknown. How well can we hope to communicate and coexist with these unpredictable and elusive intelligent machines? These questions will lead us to the forefront of AI research.

Artificial intelligence has not always been this way. From the very beginning, there have been two schools of thought regarding the comprehensibility, or explainability, of AI.

Many believed that building machines that reason based on rules and logic was the most meaningful approach, as this would make their internal workings transparent to anyone willing to examine the code.

Others argued that if machines were inspired by biology and learned through observation and experience, intelligence would emerge more easily. This meant handing over computer programming to the machines themselves. Instead of programmers writing commands to solve a problem, the program would generate its own algorithms based on example data and desired outputs. Machine learning technologies, which have evolved into today's most powerful AI systems, follow this latter path: machines essentially program themselves.

The inner workings of any machine learning technology are inherently more opaque than hand-coded systems, even to computer scientists. This is not to say that all future AI technologies will be equally unknowable. But by its very nature, deep learning is a particularly dark black box.

We feed it massive amounts of data, and large language models generate answers, judgments, preferences, and so on, but we do not know how they arrive at these conclusions. Their paths are untraceable; their "laws" are unprovable; their biases cannot be预警 (warned against) in advance. They act like silent oracles: we rely on them but cannot question them; we obey them but cannot hold them accountable.

Thus, technology is no longer just a tool; it has become an opaque form of power. It is an institutionalized actor exercising power in society. This power does not manifest through violence but permeates human life subtly through rules, standards, and algorithmic logic.

Once faced with such a black box, the issue of human trust in the system arises. A classic example cited by Weinberger is a medical learning monster called "Deep Patient." Researchers at a New York medical school fed it a full 700,000 medical records and let it loose to find whatever it could do. The result: its diagnoses and predictions far exceeded the capabilities of human doctors. Although this "black box" diagnostic system could not explain the predictions it gave, in some cases, it was indeed more accurate than human physicians.

This is deep learning, capable of bringing about discoveries humans have never considered or could even imagine. As Weinberger said, the lesson of "Deep Patient" is that deep learning systems do not need to simplify the world into something humans can understand.

The problem is that Weinberger did not delve deeply enough into the issue of human trust in AI. For instance, even if "Deep Patient's" diagnoses are more accurate than those of human doctors, would doctors and patients trust it if it cannot explain its own judgments?

For an AI doctor to be trustworthy, three conditions must be met:

First is Explainability: Patients and medical staff must be able to understand the decision logic, because trust is based not only on outcomes but also on the transparency and comprehensibility of the process.
Second is Accountability: Errors and risks must be clearly attributable, and responsibility must be traceable and enforceable.
Third is Ethical Embedding: Algorithm design must consider patient dignity, vulnerability, and rights.

If patients can only passively accept results, and the doctor's role gradually shifts from guide and guardian to merely an executor of algorithms, ethical responsibility issues arise. The patient's agency is stripped away, the doctor's scope of judgment is compressed, erroneous diagnoses are no longer easily traceable to a person, and algorithms cannot bear ethical responsibility; ultimately, it is the patient who gets hurt.

This illustrates that the more powerful technology becomes, the more we need to define its power and boundaries; otherwise, trust is merely an illusion.

Human Trust: Difficult to Withstand the Blow of Failure

Human trust is often based on our understanding of how others think and our empirical knowledge of the reliability of those thoughts. This helps create a sense of psychological safety.

For most people, however, AI remains quite novel and unfamiliar. It uses complex analytical systems to make decisions, identifying potential hidden patterns and faint signals from vast amounts of data.

Even if it can be technically explained, the AI decision-making process is usually incomprehensible to most people. Moreover, current AI development is accelerating in the direction of incomprehensibility. Interacting with things we don't understand causes anxiety and makes us feel like we are losing control.

The autonomous vehicles launched by chip manufacturer Nvidia look no different from other self-driving cars, but they are actually distinct from anything demonstrated by Google, Tesla, or General Motors, showcasing the rise of AI. Nvidia's car does not follow any single instruction provided by engineers or programmers. Instead, it relies entirely on an algorithm that learned to drive by observing human behavior.

Getting a car to drive this way is an impressive feat. But it is also somewhat unsettling because it is not entirely clear how the car's decisions are made. Information from the vehicle's sensors goes directly into a huge artificial neural network, which processes the data and then provides the necessary commands to operate the steering wheel, brakes, and other systems. The result seems consistent with the reactions one would expect from a human driver.

But what if one day it does something unexpected—like crashing into a tree or stopping motionless at a green light? Under current circumstances, it might be very difficult to figure out why it did so. The system is so complex that even the engineers who designed it struggle to isolate the cause of any single behavior. And you cannot ask it questions: there is no way to design a system that can always explain why it does what it does.

Unless we find ways to make technologies like deep learning more understandable to their creators and more accountable to users, it will be difficult to predict when failures might occur—and failure is inevitable.

Tommi Jaakkola, a professor at MIT who studies machine learning applications, said:

"This is an issue whose significance has already become apparent, and it will become even more meaningful in the future. Whether it's investment decisions, medical decisions, or possibly military decisions, you don't want to rely solely on 'black box' methods." [3]

Jaakkola offers important insights into the general limitations and common failures of AI, paying particular attention to problems in practical applications and the inability to account for all variables in complex scenarios. These include:

Misuse of AI – Jaakkola points out that AI doesn't fail because the technology itself is flawed, but because it is often applied to problems it is unsuited to solve. Incorrect application scenarios are one of the main reasons for AI failure.

Incomplete Data – In machine learning projects, dealing with incomplete data sources is a major challenge. This is a common cause of failure for many AI projects.

Model Limitations – He emphasizes the inherent limitations of current AI models. For example, even tiny changes in a neural network can cause it to fail to capture all necessary information, leading to unpredictable or erroneous behavior in complex real-world scenarios.

System Robustness – Jaakkola's work underscores the importance of building robust and trustworthy machine learning systems, which is crucial for preventing the erosion of public trust, especially in critical application areas like healthcare.

Jaakkola's insights are often cited when discussing why many corporate AI projects or pilot programs fail to yield expected results; some studies suggest failure rates can be as high as 95%. Therefore, to truly realize the value of AI, enterprises must not only optimize algorithms but also focus on trust-building and ethical design.

This leads to the issue of "Trustworthy AI." "Trustworthy AI" is a core concept in the field of AI ethics and governance, referring to AI systems that are trustworthy in technical, legal, ethical, and social dimensions. It pursues not only performance and efficiency but also emphasizes respect for human values, rights, and social responsibilities.

If AI systems are to be increasingly widely applied in various services and products in our daily lives, then we must view the importance of trusting (or not trusting) AI from the user's perspective. If AI-driven systems become agents or semi-agents in important aspects of human activity, we cannot help but worry about the extent to which these tools will influence human thinking, decision-making, and behavior.

Trust and distrust in AI, acting as a regulator, may significantly control the degree of AI technology diffusion. When users, institutions, or social groups maintain trust in AI systems, they are more willing to adopt, rely on, and invest resources in them, thereby accelerating the spread of technological applications. Conversely, distrust will suppress adoption rates, preventing potential technological advantages from translating into real-world benefits.

This regulatory effect is not only technical but also profoundly involves ethical and social-psychological dimensions. Trust is built on reliability and robustness, explainability and transparency, fairness and impartiality, privacy and security, traceability of responsibility, and ethical and legal compliance. Once these foundations are missing, high accuracy or efficiency in AI will hardly win user acceptance.

Thus, Trustworthy AI is not just a technical issue; it is a comprehensive practice involving technology, ethics, society, and law. Even if an AI system has advanced algorithms, if it is unreliable, opaque, heavily biased, or unable to bear responsibility, it cannot be called Trustworthy AI.

Destroying Trust Destroys Civilization

Therefore, whether to understand or not to understand is not a conclusion that can be easily reached, because the stakes we are betting are too high. Just as many aspects of human behavior cannot be explained in detail, perhaps it is impossible for AI to explain everything it does.

Perhaps this is a characteristic of the nature of intelligence: only a part of it is exposed to rational explanation. The rest is instinctive, subconscious, or elusive.

If so, then at some stage, we may simply have to trust AI's judgment (as Weinberger advocates) or simply not use AI. The choice between belief and non-use will have to be incorporated into social intelligence.

Just as society is built on contracts of expected behavior, we will need to design and use AI systems that respect and adapt to our social norms. If we are to create robot tanks and other killing machines, it is crucial that their decisions align with our moral judgments.

Philosopher Daniel Dennett holds a very cautious attitude towards explainability. He says:

"If we are going to use these machines and rely on them, then let us grasp as firmly as possible how and why they give us answers. But since there may be no perfect answers, we should be cautious about AI explanations, just as humans are with each other's explanations—no matter how smart the machine appears. And if it cannot explain what it is doing better than we can, then do not trust it." [4]

The reason for caution is that the future of humanity may not concern a takeover by some superintelligence, but rather that the widespread use of AI could pose an existential threat to civilization. This danger stems from human fragility—not knowing what we know, and not knowing whom to trust.

Rather than worrying about AI's impact on jobs, we should be more concerned about AI's impact on trust, because trust is one of the most important cornerstones of civilization.

We are spending more and more time in digital environments, and evolution has not prepared us well for this. Meanwhile, AI is likely to evolve to self-replicate, as evolution is not limited to biological organisms.

In this polarized development, while we approve of humans using expert systems in medicine or GPS in navigation to improve efficiency, we must also see a danger: as machines take on an increasing proportion of basic tasks like perception, memory, and algorithmic calculation, people may tend to personify these systems, endowing them with intellectual capabilities they do not possess.

As Dennett worries, people may misunderstand AI systems that are essentially "parasitic" rather than constructively using them to challenge and develop the user's own understanding [5]. The correct understanding of AI should not be "replacing humans" but "expanding human possibilities."

The rise of machine learning is one of the most significant transformations in human history. More and more machine learning models will become our knowledge base, much like libraries and human brains today.

However, there is no "knowledge" inside machine learning models; they are merely statistical fits to data patterns. Algorithms can identify associations and predict trends, but they do not understand causal relationships, contextual meanings, or ethical consequences. They do not reflect on their own judgments like humans, nor do they bear responsibility.

Therefore, even if a model's output is accurate, it cannot provide explainable reasons, nor can it respond to further questioning from users. The model's "intelligence" is merely superficial; true trust must still be built on understanding, response, and responsibility between humans.

Human cognition is not just pattern recognition; it also includes self-reflection, moral perception, and emotional response. When algorithms place us in situations of vulnerability, dependence, and helplessness, we need to rethink the nature and use of knowledge, and even rethink who we are as beings capable of understanding our own world.

*References:

[1] "The Edge of Knowledge" by David Weinberger, translated by Hu Yong and Gao Mei, Shanxi People's Publishing House, 2014.

[2] Weinberger, David (2019). Everyday Chaos: Technology, Complexity, and How We're Thriving in a New World of Possibility. Harvard Business Review Press.

[3] Knight, Will (Apr 11, 2017). "The Dark Secret at the Heart of AI." MIT Technology Review, https://www.technologyreview.com/2017/04/11/5113/the-dark-secret-at-the-heart-of-ai/.

[4] Knight, Will (Apr 11, 2017). "The Dark Secret at the Heart of AI." MIT Technology Review, https://www.technologyreview.com/2017/04/11/5113/the-dark-secret-at-the-heart-of-ai/.

[5] Dennett, Daniel C. (2017). From Bacteria to Bach and Back: The Evolution of Minds. Penguin, 402-3.

Selected Articles:

AI Doesn't Need to Understand the World, But We Need to Understand AI

Deep Learning: An Extremely Dark Black Box

Human Trust: Difficult to Withstand the Blow of Failure

Destroying Trust Destroys Civilization

分享網址