References This post is based on several sources including; https://dirk.eddelbuettel.com/papers/useR2019_rcpp_tutorial.pdf (Intro to Rcpp and Rcpparmadillo) https://privefl.github.io/R-presentation/Rcpp.html (Intro to Rcpp) https://teuder.github.io/rcpp4everyone_en/ (for Rcpp object class, methods and functions) http://arma.sourceforge.net/docs.html (for RcppArmadillo class, methods and functions) http://dirk.eddelbuettel.com/code/rcpp/Rcpp-sugar.pdf (for Rcpp sugar functions, including prob dists) https://teuder.github.io/rcpp4everyone_en/310_Rmath.html (for Rmath in R:: namespace, mostly prob dists) https://www.w3schools.com/cpp/cpp_math.asp (for cmath in std:: namespace)

윈도우에서 리눅스 기반으로 만들어진 오픈소스 프로그램을 사용하면 필사적인 구글링을 해야하는 문제가 정말 주기적으로 발생하는 것 같습니다. Rmarkdown으로 Rcpp 함수를 쓸 때 knit가 안 되어, 해결하는데 꽤 고생했습니다. 혹시 이런 문제를 겪으실 분을 위해 간단한 가이드를 작성했습니다. 도움이 됐으면 좋겠습니다! Using opensource software such as R in windows is to face a constant barrage of quirky weird errors and to desperately search for help. Unexceptionally, Rcpp package kindly gave me another of such headache.

Suppose that the true density of a random variable $x$ is $p(x)$. Since this is unknown, we can try to come up with an approximation $q(x)$. Then KL divergences is a good measure of mismatch between $p$ and $q$ distribution. $$ \begin{align*} \text{KL divergence:}\quad KL(p||q) = \int p(x)\log \dfrac{p(x)}{q(x)}dx \end{align*} $$ From the formula we can see that KL divergence is a weighted average, with wighted $p(x)$, of an error induced by approximation ($\log p(x) - \log q(x)$).

How do we quantify an amount of information that some data $x$ contains? If the data is pretty much expected than it tells nothing new to us. But if it is so rare then it has some value. In this sense, we can think of an amount of information as a “degree of surprise”, and define $$ \text{information content of data $x$:}\quad h(x) = -\log p(x) $$ where the logarithm ensures $h(x,y)=h(x)+h(y) \Leftrightarrow p(x,y)=p(x)p(y)$, and the negative sign makes $h(x)\geq 0$.