User Tools

Site Tools


cm:block_entropy

Def: Let <latex> \mathcal{P} = … X_{-1} X_0 X_1 …</latex> be a discrete time stationary process over a finite alphabet of symbols <latex> \mathcal{A} </latex>. The (length L) block entropy of <latex> \mathcal{P} </latex> is defined as <latex> H[L] = H[\overrightarrow{X}^L] = - \sum_{\overrightarrow{x}^L} Pr(\overrightarrow{x}^L) \cdot \log_2[Pr(\overrightarrow{x}^L)] </latex>, where the sum is taken over all possible “sequence values” <latex> \overrightarrow{x}^L </latex> which the r.v. <latex> \overrightarrow{X}^L </latex> may take on.

The block entropy <latex> H[L] </latex> is a monotonically increasing function of L, which will normally scale linearly with the block length L as <latex> L \rightarrow \infty </latex>. The process's (per symbol) entropy_rate <latex> h_{\mu} </latex> is given by the asymptotic ratio <latex> h_{\mu} = \lim_{L \to \infty} H[L]/L </latex>.

cm/block_entropy.txt · Last modified: 2009/12/12 15:21 by ntravers