Machine Learning that Matters

A summary of the paper Machine Learning that Matters by Kiri L. Wagstaff (2012).

The paper express concern over the ML research detached from the larger world of science and society. Most of the ML researchers work on synthetic/benchmark data, use some statistical methods to evaluate the result and wind up the research by just publishing the result. The paper urges to include the evolution methods that enable direct measurements as dollar saved, lives preserved, time conserved, etc. and to persuade the relevant users to adopt the result.

Researches are usually keen on interesting or challenging problems. ML researches should focus on the problems which are matters to society. The paper proposes six Impact Challenges that will benefit the entire world:

  1. A law passed or legal decision made that relies on the result of an ML analysis.
  2. $100M saved through improved decision making provided by an ML system.
  3. A conflict between nations averted through high quality translation provided by an ML system.
  4. A 50% reduction in cyber security break-ins through ML defenses.
  5. A human life saved through a diagnosis or intervention recommended by an ML system.
  6. Improvement of 10% in one country’s Human Development Index attributable to an ML system.

Much effort is often put into chasing after goals in which an ML system outperforms a human at the same task. The Impact Challenges in this paper also differ from that sort of goal in that human-level performance is not the gold standard. What matters is achieving performance sufficient to make an impact on the world. As an analogy, consider a sick child in a rural setting. A neighbor who runs two miles to fetch the doctor need not achieve Olympic-level running speed (performance), so long as the doctor arrives in time to address the sick child’s needs (impact).

Finally the paper is pointing out some of the obstacles to ML impact. The complexity of ML prevents ML from widespread adoption. Simplifying, maturing, and robustifying ML algorithms and tools are crucial for the widespread adoption of ML.