Variety generation—A reinterpretation of Shannon's mathematical theory of communication, and its implications for information science

Abstract The conventional interpretation of Shannon's mathematical theory of communication in relation to textual material is unduly restrictive and unhelpful. A reinterpretation which is based on the definition of new symbol sets, comprising approximately equally‐frequent strings of characters, is presented. It is shown to have wide applicability in computer‐processing of texts. Moreover, it provides a more general formalism for considering methods of representing, storing and retrieving the subject content of documents.