When AI Learns the Wrong Thing

AI, a computer that learns from examples, can do well in a test but make mistakes at work.

Think of this example. It is not a real story. A company wants AI to look at tomato photos. It must tell which tomatoes are ready to eat. People give it photos with the right answers to learn from.

But most tomatoes ready to eat are in dark boxes. The others are on white tables. The AI may learn to look at the boxes and tables, not the tomatoes. It learns from right answers, but it may use the wrong thing to find them.

Tests with photos like these may go well. But put a tomato ready to eat on a white table. Now the AI may be wrong.

People can test it with different boxes, tables and light. More kinds of photos can help it learn, but cannot stop every mistake. A good test must check for changes like those at work.