Chapter 10: Processes of Perception and Analysis

Section 9: Statistical Analysis

Entropy estimates [for sequences]

Fitting the number of distinct blocks of length b to the form k^(h b) for large b the quantity h gives the so-called topological entropy of the system. The so-called measure entropy is given as discussed on page 959 by the limit of -1/b Sum[pi Log[k, pi], {i, k^b}] where the pi are the probabilities for the blocks. Actually getting accurate estimates of such entropies is however often rather difficult, and typically upper bounds are ultimately all that can realistically be given. Note also that as discussed in the main text having maximal entropy does not by any means imply perfect randomness.

From Stephen Wolfram: A New Kind of Science [citation]