The question of deriving the KL divergence of two Gaussians appears frequently. It arises especially often in the context of variational auto-encoders, because the essential paper on the topic by Kingma and Welling, "Auto-Encoding Variational Bayes", provided only the barest details about the result. Naturally, people will ask how to obtain that result.

So we have a few versions of this identical question asked.

There are, I assume, many more examples. This isn't great, because a goal of the site is to produce canonical answers to statistical questions. https://meta.stackoverflow.com/questions/291992/what-is-a-canonical-question-answer-and-what-is-their-purpose

  • Do we have a canonical Q&A that we can use as a one-stop duplicate target?
  • If not, what should a canonical Question address?
  • Should we merge any of these Questions?
  • Are there additional Questions that we should add to this list?

Note that marking as duplicate is a simple action & easily reversed. On the other hand, merging threads is permanent and irreversible, so we are hesitant to take that action unless we're absolutely certain that the questions are exactly the same — all of the answers to one are answers to the other and vice versa.

Update Sept. 16, 2023 -- Another question in this vein appeared today.

KL Divergence in VAE

  • $\begingroup$ "If not, what should a canonical Question address?" I personally like mathematical statistics oriented questions and a derivation of the divergence sounds interesting to me. What I am wondering about is the context. If this could be integrated into the question and answers, then this would be nice. .... $\endgroup$ Commented Sep 6, 2023 at 13:53
  • $\begingroup$ ... The question "Deriving the KL divergence loss for VAEs" seems already canonical to me. Are there arguments to make a new slightly duplicate question? $\endgroup$ Commented Sep 6, 2023 at 13:54
  • $\begingroup$ Reading further, the other questions are also good as well. So the problem is that there are many vague duplicates. Potentially a question could be formulated, about the use of divergence loss, that creates a question that links to all these type of posts. $\endgroup$ Commented Sep 6, 2023 at 13:57
  • $\begingroup$ @SextusEmpiricus "Deriving the KL divergence loss for VAEs" posts an expression for $\mathfrak{D}_\text{KL}[p_1 \| p_2]$ and then simplifies it, but doesn't show why it is true that $$\mathfrak{D}_\text{KL}[p_1 \| p_2] = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - n + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]$$ Regarding the "vague duplicates" -- I agree, we need a canonical thread. $\endgroup$
    – Sycorax Mod
    Commented Sep 6, 2023 at 13:59
  • $\begingroup$ A canonical thread, but that can be generated with a canonical answer to one of the pre-existing questions. $\endgroup$ Commented Sep 6, 2023 at 14:02
  • $\begingroup$ Which answer is canonical? What question does it address? Is the question that it addresses the best choice for a canonical thread? $\endgroup$
    – Sycorax Mod
    Commented Sep 6, 2023 at 14:14


You must log in to answer this question.

Browse other questions tagged .