Observation Histories: A Compression Technique for Recording Discrete States

Felix Ritchie
Department of Economics, University of Stirling
F.J.Ritchie@stir.ac.uk

Abstract

This paper describes a way of recording information on discrete states by means of a technique called key transformation. This involves recognising that, where N discrete states of a variable are observed T groups or periods, any particular combination of states and periods can be uniquely expressed as an index number . The characteristics of these combinations and the individuals to whom the observations refer can then be stored with the index number. Appropriate software can then extract all the information contained within the patterns (and semi-aggregated information on the individuals with particular states at particular times) much more quickly and easily then searching through the dataset.

This is an extremely efficient way to store and access information on states and changes of state; the benefit is greatest where very large datasets or multiple states are involved. The resulting datasets can calculate information on numbers observed, state transitions, hasard rates, and so on. They can also be used as ``true" pseudo-panels or as grouped observation sets. A subsidiary consequence of this technique is that the semi-aggregated form of the data may satisfy confidentiality restrictions for sensitive datasets.

JEL-Classification: C81, C82, C88.