It felt like walking up a mountain during a temperature inversion. You struggle through fog so dense you can scarcely see where you’re going. Suddenly, you break through the top of the cloud, and the world is laid out before you. It was that rare and remarkable thing: a eureka moment.

For the past three years, I’d been struggling with a big and frustrating problem. In researching my book Regenesis, I’d been working closely with Iain Tolhurst (Tolly), a pioneering farmer who had pulled off something extraordinary. Almost everywhere, high-yield farming means major environmental harm, due to the amount of fertiliser, pesticides and (sometimes) irrigation water and deep ploughing required. Most farms with apparently small environmental impacts produce low yields. This, in reality, means high impacts, as more land is needed to produce a given amount of food. But Tolly has found the holy grail of agriculture: high and rising yields with minimal environmental harm.

He uses no fertiliser, no animal manure and no pesticides. His techniques, the result of decades of experiment and observation, appear to enrich the crucial relationships between crops and microbes in the soil, through which soil nutrients must pass. It seems that Tolly has, in effect, “trained” his soil bacteria to release nutrients when his crops require them (a process called mineralisation), and lock them up when his crops aren’t growing (immobilisation), ensuring they don’t leach from the soil.

So why the frustration? Well, Tolly has inspired many other growers to attempt the same techniques. Some have succeeded, with excellent results. Others have not. And no one can work out why. It’s likely to have something to do with soil properties. But what?

Not for the first time, I had stumbled into a knowledge gap so wide that humanity could fall through it. Soil is a fantastically complex biological structure, like a coral reef, built and sustained by the creatures that inhabit it. It supplies 99% of our calories. Yet we know less about it than any other identified ecosystem. It’s almost a black box.

Many brilliant scientists have devoted their lives to its study. But there are major barriers. Most soil properties cannot be seen without digging, and if you dig a hole, you damage the structures you’re trying to investigate. As a result, studying even basic properties is cumbersome, time-consuming and either very expensive or simply impossible at scale. To measure the volume of soil in a field, for example, you need to take hundreds of core samples. But as soil depths can vary greatly from one metre to the next, your figure........

© The Guardian