[Forums.xkcd.com] Random Matrix Theory

Moderators: Zamfir, Hawknc, Moderators General, Prelates

SUPPOSE we had a theory that could explain everything. Not just atoms and quarks but aspects of our everyday lives too. Sound impossible? Perhaps not.¹

假设我们有一个可以解释一切的理论。不仅是原子和夸克，而且是我们日常生活的方方面面。听起来不可能？也许不是。

Spoiler: It’s all part of the recent explosion of work in an area of physics known as random matrix theory. Originally developed more than 50 years ago to describe the energy levels of atomic nuclei, the theory is turning up in everything from inflation rates to the behaviour of solids. So much so that many researchers believe that it points to some kind of deep pattern in nature that we don’t yet understand. “It really does feel like the ideas of random matrix theory are somehow buried deep in the heart of nature,” says electrical engineer Raj Nadakuditi of the University of Michigan, Ann Arbor.

拆台：这是一个被称为随机矩阵论的物理学领域中最近的爆炸性工作的一部分。这一理论最初在50多年前被开发用于描述原子核的能级，现在它已经出现在从通胀率到固体行为等每一件事中。以至于许多研究者认为它指向某种我们尚未理解的深层模式。University of Michigan, Ann Arbor的电气工程师Raj Nadakuditi说：“随机矩阵论的思想感觉真的深埋在大自然的中心”。

All of this, oddly enough, emerged from an effort to turn physicists’ ignorance into an advantage. In 1956, when we knew very little about the internal workings of large, complex atomic nuclei, such as uranium, the German physicist Eugene Wigner suggested simply guessing.

奇怪的是，这一切都来源于把物理学家的无知转为某种优势的努力中。1956年，那时我们对铀等大的、复杂的原子核内部的运行机制知之甚少，而德国物理学家Eugene Wigner想到了一个简单的猜测。

Quantum theory tells us that atomic nuclei have many discrete energy levels, like unevenly spaced rungs on a ladder. To calculate the spacing between each of the rungs, you would need to know the myriad possible ways the nucleus can hop from one to another, and the probabilities for those events to happen. Wigner didn’t know, so instead he picked numbers at random for the probabilities and arranged them in a square array called a matrix.

量子论告诉我们，原子核有许多离散的能级，就像梯子上不均匀排列的横木。为了计算每层之间的间距，你需要知道原子核从一层迁跃到另一层无数可能的方式以及那些事件发生的概率。Wigner并不知道如何做，相反他随机挑选一些数字作为概率，并把它们排列成一个被称为矩阵的方形数组。

The matrix was a neat way to express the many connections between the different rungs. It also allowed Wigner to exploit the powerful mathematics of matrices in order to make predictions about the energy levels.

矩阵是表达不同层之间的许多连结的一种非常整洁的方式。它还使得Wigner能够利用矩阵的强大运算，来预测能级。

Bizarrely, he found this simple approach enabled him to work out the likelihood that any one level would have others nearby, in the absence of any real knowledge. Wigner’s results, worked out in a few lines of algebra, were far more useful than anyone could have expected, and experiments over the next few years showed a remarkably close fit to his predictions. Why they work, though, remains a mystery even today.

奇怪的是，他发现这个简单的方法使得他在没有任何实际知识的情况下，就能得出很可能任何一级附近就有其它的能级。Wigner的研究成果，用几行代数就能计算出来，但比任何人预期的都要有用得多，而且接下来几年的实验与他的预测都非常接近。尽管如此，它们为什么起作用至今仍是个谜。

What is most remarkable, though, is how Wigner’s idea has been used since then. It can be applied to a host of problems involving many interlinked variables whose connections can be represented as a random matrix.

但最值得关注的是，从那时起Wigner的想法是如何被使用的。它可以被应用于大量涉及许多互联变量的问题中，这些变量的连接可以表示为一个随机矩阵。

The first discovery of a link between Wigner’s idea and something completely unrelated to nuclear physics came about after a chance meeting in the early 1970s between British physicist Freeman Dyson and American mathematician Hugh Montgomery.

第一次发现Wigner的想法和一些跟核物理完全无关的问题之间的联系大概发生在上世纪70年代初英国物理学家Freeman Dyson和美国数学家Hugh Montgomery的一次偶然相遇。

Montgomery had been exploring one of the most famous functions in mathematics, the Riemann zeta function, which holds the key to finding prime numbers. These are numbers, like 2, 3, 5 and 7, that are only divisible by themselves and 1. They hold a special place in mathematics because every integer greater than 1 can be built from them.

Montgomery一直在探索数学中一个最著名的函数，即黎曼zeta函数，它是寻找素数的关键。素数就是那些如2、3、5、7只能被它们自身和1整除的数。它们在数学中具有特殊的地位，因为每个大于1的整数都可以由它们构建。

In 1859, a German mathematician called Bernhard Riemann had conjectured a simple rule about where the zeros of the zeta function should lie. The zeros are closely linked to the distribution of prime numbers.

1859年，一位名叫Bernhard Riemann的德国数学家推测了一个关于zeta函数零点位置的简单规则。这些零点与素数分布密切相关。

Mathematicians have never been able to prove Riemann’s hypothesis. Montgomery couldn’t either, but he had worked out a formula for the likelihood of finding a zero, if you already knew the location of another one nearby. When Montgomery told Dyson of this formula, the physicist immediately recognised it as the very same one that Wigner had devised for nuclear energy levels.

数学家们至今还未能证明黎曼假设。Montgomery也不能，但他已经发现了一个如果你已经知道附近的另一个零点的位置时找到一个零点的概率的公式。当Montgomery告诉Dyson这个公式时，这位物理学家立即就意识到这跟Wigner设计用于原子核能级的想法是完全一样的。

To this day, no one knows why prime numbers should have anything to do with Wigner’s random matrices, let alone the nuclear energy levels. But the link is unmistakable. Mathematician Andrew Odlyzko of the University of Minnesota in Minneapolis has computed the locations of as many as 1023 zeros of the Riemann zeta function and found a near-perfect agreement with random matrix theory.

直到今天，仍没人知道为什么素数与Wigner的随机矩阵有关，更不用说原子核能级了。但这一联系还是很明确的。University of Minnesota in Minneapolis的数学家Andrew Odlyzko计算了黎曼zeta函数多达1023个零点的位置后，发现这与随机矩阵论有着近乎完美的一致。

The strange descriptive power of random matrix theory doesn’t stop there. In the last decade, it has proved itself particularly good at describing a wide range of messy physical systems.

随机矩阵论的奇特描述能力并没有就此止步。在过去十年，它已经自证了它特别擅长描述一系列混乱的物理系统。

Universal law?

普适性？

Spoiler: Recently, for example, physicist Ferdinand Kuemmeth and colleagues at Harvard University used it to predict the energy levels of electrons in the gold nanoparticles they had constructed.

拆台：例如，最近哈佛大学的物理学家Ferdinand Kuemmeth和他的同事们用它来预测他们构造的金纳米粒子中电子的能级。

Traditional theories suggest that such energy levels should be influenced by a bewildering range of factors, including the precise shape and size of the nanoparticle and the relative position of the atoms, which is considered to be more or less random. Nevertheless, Kuemmeth’s team found that random matrix theory described the measured levels very accurately (arxiv.org/abs/0809.0670).

传统理论认为，这种能级应该会受到各种令人迷惑的因素影响，这些因素包括纳米粒子的精确形状和大小以及原子的相对位置，这些原子被认为或多或少是随机的。然而，Kuemmeth团队发现，随机矩阵论能够非常精确地描述被测能级（arxiv.org/abs/0809.0670）。

A team of physicists led by Jack Kuipers of the University of Regensburg in Germany found equally strong agreement in the peculiar behaviour of electrons bouncing around chaotically inside a quantum dot - essentially a tiny box able to trap and hold single quantum particles (Physical Review Letters, vol 104, p 027001).

德国University of Regensburg的Jack Kuipers领导的一个物理学家团队同样发现了在一个量子点（本质上是一个可以捕抓单量子粒子的小盒子）内来回无序跳跃的电子的奇特行为中的强一致性（Physical Review Letters, vol 104, p 027001）。

The list has grown to incredible proportions, ranging from quantum gravity and quantum chromodynamics to the elastic properties of crystals. “The laws emerging from random matrix theory lay claim to universal validity for almost all quantum systems. This is an amazing fact,” says physicist Thomas Guhr of the Lund Institute of Technology in Sweden.

这一列表已发展到令人难以置信的地步，从量子引力、量子色动力学（QCD）到晶体的弹性性质。瑞典Lund Institute of Technology物理学家Thomas Guhr说：“来自随机矩阵论的定律，对几乎所有的量子系统都具有普适的有效性。这是一个惊人的事实”。

Random matrix theory has got mathematicians like Percy Deift of New York University imagining that there might be more general patterns there too. “This kind of thinking isn’t common in mathematics,” he notes. “Mathematicians tend to think that each of their problems has its own special, distinguishing features. But in recent years we have begun to see that problems from diverse areas, often with no discernible connections, all behave in a very similar way.”

随机矩阵论吸引了很多数学家，如纽约大学Percy Deift，他料想那里可能也有更一般的模式。他提到，“这种思维在数学中并不常见。数学家们往往认为他们的每一个问题都有其独特的特点。但近年来，我们开始发现，来自不同领域的往往并没有明显联系的问题，都表现得非常相似。”

In a paper from 2006, for example, he showed how random matrix theory applies very naturally to the mathematics of certain games of solitaire, to the way buses clump together in cities, and the path traced by molecules bouncing around in a gas, among others.

例如，在2006年的一篇论文[Universality for Mathematical and Physical Systems]中，他展示了随机矩阵论如何很自然地应用到某些纸牌游戏的运算、城市中公交汽车聚集的方式以及空气中分子跳跃的轨迹等。

The most important question, perhaps, is whether there is some deep theory behind both physics and mathematics that explains why random matrices seem to capture essential truths about reality. “There must be some reason, but we don’t yet know what it is,” admits Nadakuditi. In the meantime, random matrix theory is already changing how we look at random systems and try to understand their behaviour. It may possibly offer a new tool, for example, in detecting small changes in global climate.

也许最重要的问题是，物理学和数学背后是否有一些深刻理论来解释为什么随机矩阵似乎捕捉到了关于现实的基本真理。Nadakuditi认为，“一定是有原因的，但我们还不知道它是什么”。同时，随机矩阵论正改变我们如何看待随机系统的方式，并试图了解它们的行为。例如，它或许提供一个新的工具来检测全球气候的微小变化。

Back in 1991, an international scientific collaboration conducted what came to be known as the Heard Island Feasibility Test. Spurred by the idea that the transmission of sound through the world’s oceans might provide a sensitive test of rising temperatures, they transmitted a loud humming sound near Heard Island in the Indian Ocean and used an array of sensors around the world to pick it up.

早在1991年，一项国际科学合作就开展了被称为“Heard Island Feasibility Test”的可行性试验。受穿越世界海洋的声音传播可能为气温上升提供了一个敏感测试这样的想法激励，他们在印度洋Heard Island附近发射了一个巨大的嗡嗡声，并使用分布于世界各地的传感器阵列来接收它。

Repeating the experiment 20 years later could yield valuable information on climate change. But concerns over the detrimental effects of loud sounds on local marine life mean that experiments today have to be carried out with signals that are too weak to be detected by ordinary means. That’s where random matrix theory comes in.

20年后重复这一实验可能会产生关于气候变化的宝贵信息。但是，关于噪声对当地海洋生物的有害影响的担忧表明今天的实验必须用微弱的信号来进行，而这些信号太过微弱以至于无法用普通手段来探测。这就是随机矩阵论的用武之地。

Over the past few years, Nadakuditi, working with Alan Edelman and others at the Massachusetts Institute of Technology, has developed a theory of signal detection based on random matrices. It is specifically attuned to the operation of a large array of sensors deployed globally. “We have found that you can in principle use extremely weak sounds and still hope to detect the signal,” says Nadakuditi.

在过去几年，Nadakuditi与Massachusetts Institute of Technology的Alan Edelman和其他人，合作开发了一套基于随机矩阵的信号检测理论。它特别适合于全球部署的大的传感器阵列操作。Nadakuditi说：“我们发现，原则上你可以使用极其微弱的声音，仍能探测到信号”。

Others are using random matrix theory to do surprising things, such as enabling light to pass through apparently impenetrable, opaque materials. Last year, physicist Allard Mosk of the University of Twente in the Netherlands and colleagues used it to describe the statistical connections between light that falls on an object and light that is scattered away. For an opaque object that scatters light very well, he notes, these connections can be described by a totally random matrix.

其他人正在用随机矩阵论来做令人惊奇的事，例如使光线能够穿过明显不可穿透的不透明材料。去年，荷兰University of Twente的物理学家Allard Mosk和他的同事们用它来描述落在物体上的光和散射光之间的统计联系。他指出，对于一个很好散射光线的不透明物体，这些联系可以用一个完全随机矩阵来描述。

What comes up are some strange possibilities not suggested by other analyses. The matrices revealed that there should be what Mosk calls “open channels” - specific kinds of waves that, instead of being reflected, would somehow pass right through the material. Indeed, when Mosk’s team shone light with a carefully constructed wavefront through a thick, opaque layer of zinc oxide paint, they saw a sharp increase in the transmission of light.

出现了一些其它分析没有给出的奇怪可能性。这个矩阵揭示了，应该存在Mosk所说的“开放通道”，也就是说特定类型的波会穿过材料，而不是被反射。事实上，Mosk团队照亮了一条道路，即精心构造的波前可以通过厚的不透明的氧化锌涂层，他们看到了光在传播过程中急剧增强。

Still, the most dramatic applications of random matrix theory may be yet to come. “Some of the main results have been around for decades,” says physicist Jean-Philippe Bouchaud of the École Polytechnique in Paris, France,” but they have suddenly become a lot more important with the handling of humungous data sets in so many areas of science.”

尽管如此，随机矩阵论最引人注目的应用可能还没有到来。法国巴黎École Polytechnique的物理学家Jean-Philippe Bouchaud说：“一些主要结果已经存在几十年了，但它们伴随着在许多科学领域内的大数据处理，突然变得越发重要。”

In everything from particle physics and astronomy to ecology and economics, collecting and processing enormous volumes of data has become commonplace. An economist may sift through hundreds of data sets looking for something to explain changes in inflation - perhaps oil futures, interest rates or industrial inventories. Businesses such as Amazon.com rely on similar techniques to spot patterns in buyer behaviour and help direct their advertising.

从粒子物理学、天文学到生态学、经济学等，收集、处理大量数据已经司空见惯。经济学家可能会从数百个数据集中筛选，以寻找能解释通货膨胀变化的东西——可能是石油期货、利率或工业库存等。像Amazon.com这样的企业依靠类似的技术来发现购买者的行为模式，并有助于定点广告投放。

While random matrix theory suggests that this is a promising approach, it also points to hidden dangers. As more and more complex data is collected, the number of variables being studied grows, and the number of apparent correlations between them grows even faster. With enough variables to test, it becomes almost certain that you will detect correlations that look significant, even if they aren’t.

虽然随机矩阵论表明这是一种很有前途的方法，但它也指出了隐藏的危险。随着越来越多的复杂数据被收集，所研究的变量数目也不断增长，它们之间的明显相关性的数目增长得更快。有了足够的变量可以测试，几乎可以肯定，你会发现看上去很重要的相关性，即使它们或许不是。

Curse of dimensionality

维度灾难

Spoiler: Suppose you have many years’ worth of figures on a large number of economic indices, including inflation, employment and stock market prices. You look for cause-and-effect relationships between them. Bouchaud and his colleagues have shown that even if these variables are all fluctuating randomly, the largest observed correlation will be large enough to seem significant.

拆台：假设你有多年的经济指数（包括通货膨胀、就业和股市价格）数据。你要寻找它们之间的因果关系。Bouchaud和他的同事们已经证明，即使这些变量都是随机波动的，最大的观测相关性也会大到看起来很重要。

This is known as the “curse of dimensionality”. It means that while a large amount of information makes it easy to study everything, it also makes it easy to find meaningless patterns. That’s where the random-matrix approach comes in, to separate what is meaningful from what is nonsense.

这就是所谓的“维数灾难”。这意味着，虽然大量的信息使得学习一切都很容易，但它也很容易找到无意义的模式。这就是随机矩阵方法，将有意义的和无意义的东西分开。

In the late 1960s, Ukrainian mathematicians Vladimir Marcenko and Leonid Pastur derived a fundamental mathematical result describing the key properties of very large, random matrices. Their result allows you to calculate how much correlation between data sets you should expect to find simply by chance. This makes it possible to distinguish truly special cases from chance accidents. The strengths of these correlations are the equivalent of the nuclear energy levels in Wigner’s original work.

在20世纪60年代末，乌克兰数学家Vladimir Marcenko和Leonid Pastur推导了描述非常大的随机矩阵的关键性质的一个基本数学结果。他们的结果使得你能够计算你希望偶然发现的数据集之间有多大相关性。这使得区分真正特殊的情况和偶然事故成为可能。这些关联的强度相当于Wigner最初工作中的原子核能级。

Bouchaud’s team has now shown how this idea throws doubt on the trustworthiness of many economic predictions, especially those claiming to look many months ahead. Such predictions are, of course, the bread and butter of economic institutions. But can we believe them?

Bouchaud团队现在已经表明，这一想法是如何对许多经济预测的可信度表示怀疑的，尤其是那些自称预测后面很多月的那种。当然，这样的预测是经济体制的基础。但是我们能相信它们吗？

To find out, Bouchaud and his colleagues looked at how well US inflation rates could be explained by a wide range of economic indicators, such as industrial production, retail sales, consumer and producer confidence, interest rates and oil prices.

为了弄明白，Bouchaud和他的同事们审视了如何通过大量的经济指数来很好地解释美国的通胀率，这些指数包括工业产量、零销售、消费者和生产者的信心、利率以及油价等。

Using figures from 1983 to 2005, they first calculated all the possible correlations among the data. They found what seem to be significant results - apparent patterns showing how changes in economic indicators at one moment lead to changes in inflation the next. To the unwary observer, this makes it look as if inflation can be predicted with confidence.

使用1983年到2005年的数字，他们首先计算了所有可能的数据相关性。他们发现了看起来很重要的结果，即一个明显的显示了某个时刻经济指数的变化如何引起下一时刻通货膨胀的变化的模式。在不警觉的观察者看来，这看起来好像可以有信心预测通货膨胀。

But when Bouchaud’s team applied Marcenko’s and Pastur’s mathematics, they got a surprise. They found that only a few of these apparent correlations can be considered real, in the sense that they really stood out from what would be expected by chance alone. Their results show that inflation is predictable only one month in advance. Look ahead two months and the mathematics shows no predictability at all. “Adding more data just doesn’t lead to more predictability as some economists would hope,” says Bouchaud.

但当Bouchaud团队运用Marcenko和Pastur的数学运算时，他们大吃一惊。他们发现这些明显的相关性中只有少数可以被认为是真实的，从某种意义上说，它们确实比偶然预期的要突出。他们的调查结果表明，通货膨胀只能提前一个月预测。预测未来两个月，计算显示没有任何可预测性。Bouchaud说：“仅仅添加更多数据并不会增加如一些经济学家希望的可预测性”。

In recent years, some economists have begun to express doubts over predictions made from huge volumes of data, but they are in the minority. Most embrace the idea that more measurements mean better predictive abilities. That might be an illusion, and random matrix theory could be the tool to separate what is real and what is not.

近年来，一些经济学家开始对从大量数据中所作的预测表示怀疑，但他们只是少数派。大多数人认为更多的测量意味着更好的预测能力。这可能是一种错觉，随机矩阵论或许是区分真实与不真实的工具。

Wigner might be surprised by how far his idea about nuclear energy levels has come, and the strange directions in which it is going, from universal patterns in physics and mathematics to practical tools in social science. It’s clearly not as simplistic as he initially thought.

Wigner可能会惊讶于他关于原子核能级的想法能走得这么远，惊讶于从物理、数学中的普适模式到社会科学中的实用工具这样奇怪的动向。这显然不像他最初想的那样简单。

Written on November 20th, 2017 by 李军