The machine learning consultancy: https://truetheta.io
Join my email list to get educational and useful articles (and nothing else!): https://mailchi.mp/truetheta/truethe...
Want to work together? See here: https://truetheta.io/about/#wanttow...

What's the safest distribution to pick in the absence of information? What about in the case where you have some, though only partial, information? The Principle of Maximum Entropy answers these questions well and as a result, is a frequent guiding rule for selecting distributions in the wild.

SOCIAL MEDIA

LinkedIn :   / djrich90b91753  
Twitter :   / duanejrich  

Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon:   / mutualinformation  

Sources

Chapters 1112 of [2] were primary sources this is where I ironed out most of my intuition on this subject. Chapter 12 of [1] was helpful for understanding the relationship between the maximum entropy criteria and the form of the distribution that meets it. [3] was useful for a high level perspective and [4] was helpful for determining the list of maximum entropy distribution.

Also, thank you to Dr. Hanspeter Schmid of the University of Applied Sciences and Arts, Northwestern Switzerland. He helped me interpret some of the more technical details of [2] and prevented me from attaching an incorrect intuition to the continuous case much appreciated!

[1] T. M. Cover and J. A. Thomas. Elements of Information Theory. 2nd edition. John Wiley, 2006.

[2] E. T. Jaynes. Probability theory: the logic of science. Cambridge university press, 2003.

[3] Principle of Maximum Entropy, Wikipedia, https://en.wikipedia.org/wiki/Princip...

[4] Maximum Entropy Distribution, Wikipedia, https://en.wikipedia.org/wiki/Maximum...

Timestamps :
0:00 Intro
00:41 Guessing a Distribution and Maximum Entropy
04:16 Adding Information
06:40 An Example
08:00 The Continuous Case
10:26 The Shaky Continuous Foundation