Information theory, a misleading name.

Today information theory has been widely used in artificial intelligence, machine learning, and being suggested that information theory is how our brain acquire information.

For example,  they define -log(P(x)) be the the amount of information in a pool of data, where P(x)
is the possibility event x occurs. The formula says when a event happens  frequently, there is no information; when a rare event happens, it contends a lot of information.

It appears to be reasonable at the first glance. However, it is totally irrelevant to the definition we know of information. What we think about information is something “meaningful”, not something “rare.”

Meaning has two meanings, the first one is inference of a word, concept, or action. The second one is significance, or importance of something.  What we mean information is something meaningful, something gives inference, not something important.

 We see sun rises from east everyday, but it is as informative as sun rise from west, which rarely, or never happens. They are equally informative because the former means the Earth spins from west to east, and the latter mean the opposite. Besides, not all new things are information, only those making senses are. You don't consider a misspelled word ‘demacrosy’ more informative than the word ‘democracy’ although the latter is more common to see.

Therefore, it will be more appropriate to call information theory “news theory” to avoid misleading
association.  Something happens rarely would be on the news, while something happens everyday is not a news because unexpected information are apparently more valuable a information than those expected to happen. So, it is safe to say information theory can be a measure of the surprise value of information instead of the amount of information itself.