When AI Learns the Wrong Thing

AI, a computer system that learns from examples, can do well on a test but fail at its real job. It may learn to use the wrong information.

Imagine a company teaching AI to sort photos of tomatoes. This is not a real example. Some tomatoes are ready to eat; others are not. Workers mark the photos with the right answers. The AI tries to answer, checks its mistakes, and changes how it works.

But what if most tomatoes ready to eat are in dark boxes? What if the others are on white tables? The AI may learn to look at the boxes and tables, not the tomatoes.

This happens because correct answers help the AI learn, even when it uses the wrong information. It is not trying to cheat.

If test photos show the same boxes and tables, the AI may do well. But it could make a mistake with a tomato ready to eat on a white table.

To check this, people can photograph both kinds of tomatoes in different places and with different amounts of light.

More kinds of training photos can help, but cannot prevent every mistake. A high test score is not enough. Tests must also include changes the AI could meet at work.