Statistics based on administrative data: towards a general framework Working Paper

The use of administrative data to produce statistics, and the questions arising from this approach, are nothing new to official statisticians. Nonetheless, they have not yet given rise to a methodological framework comparable to that available for survey data. This use of such data is sometimes assumed, incorrectly, to be straightforward, since the data already exist and statisticians do not need to conduct a separate data collection process to obtain them. However, this apparent simplicity is misleading: the fact that there is no data collection process also means that we lose control over the methods used to gather the data, their time frame, the definition of the variables, the classification systems used, and of course the opportunity to verify the data... Ultimately, these new data sources require us to rethink a whole swathe of the statistical production process, in order to address the new questions they raise.

By their very nature, administrative data are associated with a specific environment, with its own objectives, language, categories and dynamics. As by-products of administrative activities, they are not, strictly speaking, self-contained data suitable for statistical use. Indeed, they may turn out to be far removed from the environment in which data users usually operate. Using administrative data for statistical purposes raises specific quality issues, since quality as a concept is always dependent on intended use. The challenge is to successfully make the transition from the administrative realm to the statistical realm, detaching the data from one context and re-attaching it to another. Such an undertaking necessarily involves certain difficulties and tensions, encompassed in the notion of “data friction”.

In order to make this transition meticulously and effectively, we need an analytical framework. That framework must address five dimensions: objects (or statistical units), population and scope, variables, domains (or classification) and time frames.

In this paper we propose an approach based upon just such a framework, divided into three phases: acquisition, transformation and statistical processing.

For each of these phases, we highlight the importance of “feedback loops” based on the principle of data tracking: in order to guarantee the quality of the data, and thus of the resulting statistics, we sometimes need to trace problems back through the process. This is all the more difficult since, unlike when working with survey data, we do not have total control over the concepts. Numerous checks are thus required, at multiple levels: if an anomaly is detected, we follow its trail back into the earlier phases of the process, back to the source if necessary. The nature of these loops is very different from one phase to the next. Ultimately, because of the quality and explainability we expect from our results, these phases do not follow one another in a purely linear manner.