Information Theory
Apply entropy, KL divergence, and mutual information to ML problems.
0%
Confidence · no data yet
After this skill you can…
- Calculate Shannon entropy and cross-entropy loss
- Explain KL divergence intuitively
- Use mutual information for feature selection
Prerequisites
Watch (6 videos)
Lecture 16: Data Compression and Shannon’s Noiseless Coding Theorem
→ Apply Shannon's Noiseless Coding Theorem→ Understand data compression principles
What is NOT Random?
→ Apply information theory to real-world problems→ Analyze entropy in different systems
Uses of Information Theory - Computerphile
→ Analyze data using Information Theory concepts→ Design efficient data compression algorithms
Stanford EE274: Data Compression I 2023 I Lecture 8 - Beyond IID distributions: Conditional entropy
→ Analyze data using information theory concepts→ Apply conditional entropy to real-world problems
"From Flat Earth to Fake Moon Landings: How Conspiracy Theories Took Over Pop Culture"
→ Understand the fundamental limits of information transmission in social networks→ Apply mathematical concepts to model the spread of conspiracy theories
The Ascension of Quantum Communication
→ Analyze quantum communication protocols for security→ Design quantum-inspired encryption algorithms
DeepCamp AI