Information theory

What this subject is

Information theory is the mathematics of communication under uncertainty. Its object is not what a message means but how much choice its source could have exercised, measured in bits. From that one move — treat a message source as a random process and measure its uncertainty by entropy — Shannon derived two limits in 1948 that bound every communication and storage system ever built. The source coding theorem says the entropy rate of a source is the exact floor on lossless compression: below it, no scheme works; above it, simple schemes do. The noisy-channel coding theorem says every channel has a capacity, computable as a maximised mutual information, such that arbitrarily reliable communication is possible at any rate below it and impossible at any rate above it — a result so counterintuitive that reliable transmission over an unreliable medium had been assumed to require either infinite redundancy or luck. The subject since divides into three strands: proving refinements and extensions of the theorems (rate- distortion, multiterminal, quantum, finite blocklength); constructing codes that actually reach the limits, which took from 1948 to the 1990s; and exporting the quantities — entropy, relative entropy, mutual information — into statistics, machine learning, physics and biology, an export the founder himself warned would outrun its evidence.

What the package holds

Curated scaffolding your agent loads before it researches, so it starts from vetted ground rather than a cold search.

• Tier 1: 41
• Tier 2: 5
• Tier 4: 2

The questions and claims below are quoted from the package files.

Where the field disagrees

Each one carries real proponents on more than one side, so your agent cannot quietly pick a winner.

• Can Shannon's measures be extended to semantic content, or is the exclusion of meaning permanent?

  4 named positions · settled inside engineering — the theory is semantics-free by construction; open in philosophy and unproven in the current semantic-communication revival

• Are the entropy of a message source and the entropy of a physical system the same quantity or the same formula?

  4 named positions · the mathematical relationship is exact and the physical bridge is experimentally supported under stated conditions; the interpretive claim that the two are one quantity remains philosophically contested

• Do information-theoretic quantities explain why deep networks generalise?

  4 named positions · the strong explanatory claim is not supported by the current evidence; the underlying objectives remain standard and useful, and the dispute is a live case study in exporting the theory

• Which notion of information is fundamental — the entropy of a source or the shortest description of an object?

  4 named positions · mathematically reconciled and pedagogically complementary; the claim that compression constitutes intelligence is a live and unresolved extrapolation

• Is channel capacity the right design target for real systems?

  4 named positions · capacity remains the correct asymptotic point-to-point limit and is routinely approached in practice; it is insufficient on its own for short-packet, nonlinear and network settings, where the refinements are the active research programme

Myths the package corrects

Widely held claims with the evidence that settles or bounds them.

• “Information in Shannon's sense measures the meaning or importance of a message.”

  debunked

  The exclusion is explicit and structural, not an oversight. In the second paragraph of the founding paper Shannon writes that "these semantic aspects of communication are irrelevant to the engineering problem" (shannon-1948), and the whole apparatus is built on that exclusion: entropy is a functional of a probability distribution over messages, so two messages with identical probabilities carry identical information whether one is a stock price and the other is noise. A random bit sequence has maximal entropy and no meaning at all — which is decisive, because if information measured meaning, the maximum would not sit on noise. Shannon returned to the point publicly in The Bandwagon (shannon-1956-bandwagon), and Elias satirised the empty cross-disciplinary applications two years later (elias-1958). Theories of semantic information exist and are respectable — Bar-Hillel and Carnap built one on logical rather than statistical probability (bar-hillel-carnap-1953), and Dretske and Floridi continued the line — but they are separate theories that borrow the word, not readings of Shannon's theorems.

• “Any file can be compressed if the algorithm is clever enough.”

  debunked

  A counting argument settles it without any appeal to cleverness. A lossless compressor must be injective — distinct inputs must produce distinct outputs, or decompression is impossible. There are two to the power n strings of length n, but only two to the power n minus one strings shorter than n in total. So no injective map can send every n-bit input to a shorter output, and any scheme that shortens some inputs must lengthen others by at least as much. Iterate the claim and it becomes absurd: a compressor that always shrinks its input could be applied to its own output repeatedly until every file in the world was one bit. The positive statement is Shannon's source coding theorem (shannon-1948): compression is possible exactly to the extent that the source is predictable, with the entropy rate as the floor, which is why English text compresses roughly threefold and already-compressed or encrypted data does not compress at all. Treatments with the argument spelled out: cover-thomas-1991 and mackay-2003.

• “Entropy in information theory is the same thing as entropy in thermodynamics.”

  contested-correction

  The relationship is real and load-bearing, and the honest correction is about scope rather than dismissal. Same functional form, different objects: Shannon entropy is dimensionless, defined for any probability distribution over anything, and depends on the modeller's choice of variables; thermodynamic entropy carries units of energy per temperature and is defined over the microstates of a physical system. The two coincide when the distribution in question is the distribution over those microstates — which is exactly Jaynes's maximum-entropy reconstruction of statistical mechanics (jaynes-1957), a genuine result and also a genuinely contested one, because it makes thermodynamic entropy partly a statement about an observer's knowledge. The physical bridge in the other direction is Landauer's principle (landauer-1961): erasing one bit dissipates at least k_B T ln 2, with Bennett locating the cost of Maxwell's demon in memory erasure (bennett-1982) and Bérut and colleagues measuring the bound in a colloidal system (berut-2012). What remains open is how much that bridge proves: Earman and Norton argue the principle is presupposed rather than derived, making the information-theoretic exorcism of the demon circular (earman-norton-1998). So: teach the analogy as deep and productive, teach the identification as conditional on a physical embedding, and teach the philosophical dispute as unresolved. What is not defensible is the loose inference — common in popular writing — that because a message source has entropy, transmitting messages is subject to the second law of thermodynamics.

• “More data always means more information.”

  debunked-as-stated

  Information is defined relative to uncertainty about something. Duplicating a dataset doubles its size and adds nothing: the joint entropy of a variable with an exact copy of itself equals the entropy of the variable alone. Highly redundant sources have entropy rates far below their raw size — Shannon's letter-guessing experiments put English at roughly one bit per character against about 4.7 bits for uniform letters (shannon-1951-english), which is why more English text buys proportionally less new information than its byte count suggests. More fundamentally, the data processing inequality says no transformation of the data can increase its mutual information about the source, so processing pipelines can only lose information, never manufacture it (cover-thomas-1991). And measured information can go the wrong way in practice: entropy and mutual information estimated from samples are biased downwards when the alphabet is large relative to the sample, so naive estimates on small data understate and on larger data shift for reasons that have nothing to do with the phenomenon (paninski-2003). The defensible statement is narrower: more independent samples from a relevant source reduce estimation error, with diminishing returns; more bytes of correlated, redundant or irrelevant data increase cost and can increase noise without increasing what is known.

• “The Shannon limit is just an engineering benchmark, and a clever enough scheme will beat it.”

  debunked

  The coding theorem has a converse, and the converse is what makes it a limit: for any rate above capacity the error probability is bounded away from zero no matter how the code is constructed (shannon-1948; proved in strong form in csiszar-korner-1981). Claims to beat it, on inspection, almost always change the channel rather than defeat the theorem — more bandwidth, more power, a better antenna, or multiple antennas, where the capacity of a multi-antenna link grows roughly with the number of antennas because it is a different channel being fed to the same theorem (telatar-1999). A second family of claims compares an uncoded or badly modelled baseline against a coded system and calls the difference a violation. The empirical situation is the opposite of loose: an irregular low-density parity-check code has been designed to operate 0.0045 decibels from the capacity of the binary-input Gaussian channel (chung-forney-richardson-urbanke-2001), and turbo, LDPC and polar codes are deployed within a fraction of a decibel in mobile standards — the gap has been closed from below and never crossed. The one legitimate refinement is not a violation but a correction in the other direction: at short blocklengths the achievable rate is strictly below capacity by a quantifiable dispersion penalty (polyanskiy-poor-verdu-2010).

Related topics

• Cryptography
• Systems thinking