info-theory

Episode 2: Information Theory

Art of the Problem

▶


▶


▶


What is Information Theory? (Information Entropy)
Art of the Problem
Transcript


imagine Alice has an idea and she wants
to share it there are so many ways to
share an idea she could draw a picture
make an engraving write a song
send a telegraph or an email but how are
these things different and more
importantly why are they the same this
story is about the fundamental particle
of all forms of communication it begins
with a special skill you likely take for
granted language all language allows you
to take a thought or mental object and
break it down into a series of
conceptual chunks these chunks are
externalized using a series of signals
or symbols humans express themselves
using a variation in sound and physical
action as do chirping birds and dancing
bees and man-made machines exchanging a
dancing stream of electrical vibrations
even our bodies are built according to
instructions stored inside microscopic
books known as DNA all are different
forms of one thing information in
simplest terms information is what
allows one mind to influence another
it's based on the idea of a
communication as selection information
no matter the form can be measured using
a fundamental unit in the same way we
can measure the mass of different
objects using a standard measure such as
kilograms or pounds this allows us to
precisely measure and compare the weight
of say rocks water or wheat using a
scale information to can be measured and
compared using a measurement called
entropy think of it as an information
scale we intuitively know that a single
page from some unknown book has less
information than the entire book we can
describe exactly how much using a unit
called the bit a measure of surprise
so no matter how Alice wants to
communicate a specific message
hieroglyphics music computer code each
would contain the same number of bits
though in different densities and a bit
is linked to a very simple idea the
answer to a yes or no question think of
it as the language of coins so how is
information actually measured does
information have a speed limit a maximum
density information theory holds the
exciting answer to these questions
it's an idea over 3,000 years in the
making but before we can understand this
we must step back and explore perhaps
the most powerful invention in human
history the alphabet and for this we
return to the cave
English (auto-generated)