“A great war of algorithms is already under way” – scientist Neil Johnson

We saw a transition in the financial ecosystem from the Keynesian “animal spirits” of desk traders and investors to “microbe spirits” by computer algorithms simpler than animals or humans — but just much, much, much faster. To understand these new phenomena we need a complexity approach. Econophysicists and financial gurus in complexity are no more in the fringe.

Neil Johnson

A conversation with physicist Neil Johnson, professor at the University of Miami, about the HFT reality.

HIGLIGHTS

«In fact a global war between competing computer algorithms and near speed-of-light fast access to markets, is where things are heading. What our work suggests is that since this war is already under way, and because from our paper it seems to correlate very well with observable crashes in the markets over timescales such as months, then any government watchdog/regulator charged with controlling their financial system also ought to have its own ‘troops on the ground’.»

«So it is like having a big lake full of different types of piranhas. Not that this is wrong, but you can see that this level of competition would be very ferocious — which is what happens in the markets.»

FOREWORD

Science and Finance got breaking news this week. The magazine New Scientist featured Thursday the research of a group of American physicists about the dynamics of high frequency trading (HFT) and the complex ultra fast machine ecology behind these new financial agents and the most recent financialization wave. The results of the research by Neill Johnson, Guannan Zhao, Jung Meng, Amith Ravindar, Spencer Carran from the University of Miami, Eric Hunsader from Nanex and Brian Tivnan from the Complex Systems Center of the University of Vermont were published online on February 8th. The title of the paper: “Financial Black swans driven by ultra fast machine ecology“.

The group of physicists analyzed a set of 18,520 ultra fast black swan events “hidden” in stock-price wild oscillations between 2006 and 2011. They find that the emergent behaviour of these movements is part of a new all-machine system, the kingdom of algorithms that command systemic fluctuations that at a certain point of accumulation provokes big bangs, big abrupt changes. Systemic risk in the financial system went up. But the physicists say they can help providing tools to predict and mitigate that risk and the new context of this wave of dematerialization where we saw a decreasing ability of humans to influence price movements at smaller timescales.

HFT turned famous since the flash-crashes of 2010 – the most famous happened May 6th; another one probably on the 1st of September. HFT has taken place at least since 1999, after the SEC authorized electronic exchanges in the previous year. After the recent financial crisis, HFT even got fatter. In the US account for 73% of all equity orders volume; in Europe accounts for 40% and in Asia only 5%. By value, HFT makes 56% of equity trades in the US and 38% in Europe, based on estimations by consultancy Tabb Group.

INTERVIEW by Jorge Nascimento Rodrigues © 2012

Q: Why did you choose specifically the year 2006 as the starting point for your research? Would there be this kind of ‘flash’ fluctuations before 2006? Or just because data availability?
A: Up to a few years ago, the technology wasn’t fast enough for significant amounts of high frequency computer trading. Today, it is estimated that it makes up the majority of all trades in US, and a rapidly growing fraction in Europe. So 2006 seemed a good place to start. The data at the millisecond scale became more accurate in recent years — and also by starting in 2006, we managed to capture both the start of the global financial collapse in Sept 2008, and also the May 6th Flash Crash in 2010.

Q: Can we say that the Keynesian [human] “animal spirits” is nowadays embedded in these famous algorithms?
A: Yes, but with a twist: In the end, the computer trading algorithms have to be relatively simple since they need to execute fast and use little prior information — hence our ecology model, which reproduces the empirical transition we see in the data, would say that we are actually seeing something more like early Life on Earth with simple microbes being analogous to the computers. So the equivalent of “microbe spirits” might be closer to the truth since the computer algorithms are simpler than animals or humans — they are just much, much faster. The big question then is: How will Life evolve in this new (sub-second) world?

Q: The paper suggests that the cumulative number of wild oscillations of flash crashes and/or spikes behave as if following a logistic S-shaped curve. Could it be that a big crash results when the S-shaped curve reaches the turning point? Can we “import” for this field the process dynamics from materials fatigue after an accumulation of tiny fractures?
A: It could be. We are looking into these questions of exactly what combination of these fractures, and when, seems to arise at the start of the big system-wide collapse. My hunch is that we can import ideas related to the critical arrangement of fractures, etc. There is an awful lot known about fracturing — and while the market obviously isn’t a piece of metal, the concept of distance and material strength can also be mimicked (distance, for example, is how separated two stocks are in terms of market sectors).

Q: Because you mention that we assist to a kind of clusterization of those milliseconds (ms) wild oscillations, is it possible to get signals or patterns that will allert for a big bang event?
A: I think that the pattern in the fractures might well tell us something. At the very least, this should be explored in a rigorous scientific way, which is what we are trying to do. Where we want to head to is the analogy of a trained aircraft engineer, who can pretty much look at the microscopic arrangement of small fractures in an aircraft and judge whether it is safe to continue flying that plane or not. To have this for markets would be an incredibly important step toward understanding, and managing, risk — not just for the system of companies, but for everyone with an interest in the markets (which as we have seen, is all of us because markets end up affecting all of us).

Q: From the previous question, another angle: Do you think that it is possible to develop a software to instantaneously react (before any human reaction) to an impending collapse, putting in action one (or some) of the three interventions that you suggest in your paper? That could mean a “great war” of algorithms in the future?
A: Yes, absolutely. In fact, a global war between competing computer algorithms and near speed-of-light fast access to markets, is where things are heading. What our work suggests is that since this war is already under way, and because from our paper it seems to correlate very well with observable crashes in the markets over timescales such as months, then any government watchdog/regulator charged with controlling their financial system also ought to have its own ‘troops on the ground’. The predator-prey analogy is a good one I think — except that most computer trading algorithms are actually more like predator. So it is like having a big lake full of different types of piranhas. Not that this is wrong, but you can see that this level of competition would be very ferocious — which is what happens in the markets. I stress that I do not think it is necessarily bad — after all, traders are just trying to make profits for their companies and hence shareholders, or increase the size of our pensions — but it is important to recognize that this is the way it seems in terms of analogies.

Q: You refer that large groups rapidly detect imminent danger. Can you explain?
A: There is a long-term historic question in animal studies, which is what evolutionary advantage does a large group have. And one of these is the fact that there are more pairs of eyes looking at what is happening — so even if every member is selfish, they can go with the crowd in terms of responding to danger which they do not necessarily see themselves. In the markets, the fact that human traders are competitive has an indirect benefit in the same way that with so many humans looking at the changes, the chances are that someone will notice and react quickly and the other humans will simply follow. Hence, the onset of the transition for crashes starts at about 650ms, which is the ballpark value for the fastest that a trained human can detect a strategic situation (e.g. chess grandmaster). Below 650ms, it is just machines.

PROFILE

Neil Johnson
Department of Physics
University of Miami

His research looks at the physics of collective behavior and emergent properties in real-world systems which are ‘complex’: from the physical, biological, medical domains through to social and even financial domains. He heads an inter-disciplinary research group in Complexity in the University.

Website: http://www.physics.miami.edu/~njohnson/

8 Responses to ““A great war of algorithms is already under way” – scientist Neil Johnson”

  1. [...] “A great war of algorithms is already under way” [...]

  2. [...] Ecologists are now tracking financial algorithms and finding that their behavior mimics that of predator and prey animal species. One program may try to hide a large transaction, for example, by creating swarms of smaller transactions meant to fool a competing program. Sometimes the trick works, and sometimes the competing program detects the subterfuge and counterstrikes. [...]

  3. [...] “A great war of algorithms is already under way … We saw a transition in the financial ecosystem from the Keynesian “animal spirits” of desk traders and investors to “microbe spirits” by computer algorithms … [...]

  4. [...] depois das bolsas” | Jorge Nascimento Rodrigues English Edition by: JPO | LISwires Read also: “A great war of algorithms is already under way” – scientist Neil Johnson | JNR [...]

  5. [...] War of Algorithms This entry was posted in Uncategorized. Bookmark the permalink. ← Wow [...]

  6. [...] depois das bolsas” | Jorge Nascimento Rodrigues English Edition by: JPO | LISwires Read also: “A great war of algorithms is already under way” – scientist Neil Johnson | [...]

  7. I like the helpful info you provide in your articles. I will bookmark your weblog and check again here frequently.
    I’m quite certain I will learn plenty of new stuff right
    here! Good luck for the next!

  8. Asking questions are actually good thing if you are not understanding something completely, but
    this paragraph gives pleasant understanding yet.

Leave a Reply

You can use these XHTML tags: <a href="" title=""> <abbr title=""> <acronym title=""> <blockquote cite=""> <code> <em> <strong>