Statisticians are using administrative and transactional data more and more, either on their own or in addition to other sources (surveys, etc.) (Ouvrir dans un nouvel ongletHand, 2018). Therefore, assessing their quality is also a strategic requirement for statistics, whether they are public or not.

Ideally, a database should evolve along the interpretation of realities it allows to be understood. Normative realities are indeed shifting. Thus, after the attacks of 2015 and the following years, the police files on the issue, both in France (Chapuis, 2018) and in Belgium (Ouvrir dans un nouvel ongletAgence Belga, 2018), experienced numerous anomalies: potential duplicates, “false actives” despite a dismissal of the case, early deletions, interpretation difficulties, etc. These have accumulated as a result of the emergence of new threats categories, as well as the urgency these sensitive data had to be processed. Such developments are constantly at work in administrative databases, which are important sources for statisticians.

Every well-designed operational database is based on a hypothesis that of a “closed world”: definitions domains specify the set of values allowed within the database model or scheme (integrity constraints); “area rules” can also be expressed in the application code and thus contribute to the definition of data. Therefore, a value that is not included in the definition domain is considered false and must be rejected from the database.

However, at the level of millions of records and hundreds of fields and information flows, the phenomena emerging in the field are not immediately taken into account in databases. Information is build up gradually, through human interpretation and without any absolute reference frame.

Apart from the basic definition domain that meets the closed-world hypothesis, “normative reality” evolves continuously and unpredictably, ignoring any rule of deterministic causal explanation. Additionally, in the case of a database as an instrument of action on reality (in the administrative, medical, environmental, military sectors, etc.), these issues are fundamental and affect data quality.

However, we can better take these phenomena into account at an operational level, at the heart of the information system. This is especially made through the interpretation of anomalies and their processing.

Data quality research is concerned with the world of anomalies for operational purposes. Their analysis resulted in the implementation of a service called ATMS, Anomalies & Transactions Management System. This system allows the history of anomalies and their processing to be tracked over time based on indicators deemed strategic and variable according to the context of use. This is an innovative concept that has already been experienced and tested in practice and that is based on a method known as back tracking.

Many large-scale administrative information systems are affected by data quality, in France for example, the Nominative Social Declaration (Déclaration Sociale Nominative – DSN). Within this Declaration, any control deemed essential is blocking (Renne, 2018) but some controls are “non-blocking” in order to avoid slowing down the process of gathering or collating declarations. The latter require manual post-processing, which database managers try to streamline and raise a “cost-quality” trade-off (Renne, 2018). The example of the DSN also shows that database quality requires that special attention to manage regulatory changes (Humbert-Bottin, 2018).

In Belgium, the LATG database in the late 1990s, and its modernized heir, the DmfA database at the beginning of the 2000s, were “case studies” of data quality research (Boydens, 1999; Ouvrir dans un nouvel onglet2018) given their scale. The DmfA database currently allows for the annual collection and redistribution of 65 billion euros of social security contributions and benefits throughout Belgium. It is subject to quarterly legislative amendments and has been a large-scale integrated information system since 2001, including features that are functionally close to the DSN. It represents one of the platforms counting many statistics on employment and salaries in Belgium. Over the last 20 years, research has also focused on the quality of many other transactional databases that are the source of statistical output.

Data quality is also an issue in the context of cross-functional administrative registers, as those planned in Germany. These are indeed based on numerous interconnections that may raise important semantic issues.

The rate of adaptations made to an information system varies according to the aims: administrative or medical databases, for example, as tools that can have an impact on reality, on the one hand, or statistical information systems, on the other, as instruments to observe or support decision-making. For the former, the rate of change will ideally be fast. For the latter, the rate of structural change is much slower or even non-existent (e.g. in the case of survey results) so as to ensure long-term comparability, while the timeliness and quality of the sources that feed them will matter.

In this context, statisticians often wonder what is best moment to take “the picture” in order to extract data from an administrative information system, regarding which they experience a type of loss of control (Rivière, 2018). Thus, work achievements on data quality offer constructive possibilities.

The quality of a database refers to its relative suitability regarding the purposes it was designed for, under budgetary constraints. Research in this field has been ongoing since the 1980s (Ouvrir dans un nouvel ongletMadnick et alii,2009) alongside companies need to have adequate addresses and contact information in their customer files. This is how data quality tools appeared (Box 1), a field that developed very quickly internationally and which remains very active (Ouvrir dans un nouvel ongletHamiti, 2019).

As part of a “curative” approach (Figure 1), the purpose of data quality tools, based on thousands of algorithms that are regularly enriched, is to detect formally identifiable quality problems (suspected duplicates, etc.) that are already present in the databases and to solve them afterwards semi-automatically. These tools also allow for the automated management of problem cases “online”, for example when entering a website.

However, if we only act downstream, we do not structurally solve the cause of these problems which will constantly recur. These may actually be caused by design flaws, by the evolution of the reality represented, or by the flows and procedures that feed databases (for example, unnecessarily redundant processes that systematically generate duplicates). Therefore, without further action upstream, data quality tools are destined to be used ad infinitum. However, these remain essential, as users do not necessarily have access to the flows and procedures that produced the data they are using.

Therefore, in addition to “curative” interventions, “preventive” approaches are essential in order to identify and structurally resolve the causes of anomalies at the source (Figure 1).

The Free University of Brussels (Université libre de Bruxelles) has had a course dedicated to data quality since 2006, presenting both types of approach (Boydens, 2021). Some of the research is directed towards preventive methods (Boydens, 1999; 2010; Ouvrir dans un nouvel onglet2012; Ouvrir dans un nouvel onglet2018; Bade, 2011; Radio, 2014; Ouvrir dans un nouvel ongletDierickx, 2019). Current work in the field of data quality research also focuses on record linkage-type algorithms (Batini and Scannapieco, 2016). The latter are used by data quality tools in the context of “comparison and duplicate elimination” operations (Box 1).

Many studies take a deterministic view: they consider the deviation of databases from reality in terms of formal inaccuracy/accuracy (Srivastava et alii, 2019). However, there is no necessarily one-to-one projection between the empirical reality and its representation in a database.

Figure 1. Two Interdependent Approaches to Assess and Improve Data Quality

Aside from the obvious benefit for data quality, the study of anomalies is important because of the high percentage of anomalies that structurally affect information systems: up to 10% according to (Ouvrir dans un nouvel ongletBoydens, 2012) and other sources (Van Der Vlist, 2011). However, when the issues (social, financial, medical, etc.) so require, these anomalies must be examined semi-automatically, or even manually, which is often slow and tedious.

Where do anomalies come from, what is their typology and, accordingly, how can we manage them the best way? In order to answer these questions, we must first return to the notion of data as approached here and which was recently studied from the point of view of statisticians (Rivière, 2020).

In the world of databases (Hainaut, 2018), a piece of data is a triplet (t, d, v) composed of the following elements:

Thus, a distinction is made between deterministic data from empirical data (Boydens, 1999). The former are characterized by the fact that a theory is available at all times so that it is possible to decide whether a value v is correct or not. This is the case for a simple algebraic operation on an object deterministic in itself, such as the sum of values for a given numerical field in a database at a given instant i. Since the rules of algebra do not change over time, and neither does the object being evaluated, one can know at any time whether the result of such a sum is correct or not. Indeed, we have a stable reference frame to be used for that purpose.

By contrast, regarding empirical data, which are subject to the human experience, the rule changes over time alongside the interpretation of the values it allows to be understood. This is the case, for example, in the medical field (where theory evolves as observations are made on patients suffering from a pathology, as shown by current research on the coronavirus), as well as in the legal and administrative fields, where the interpretation of legal concepts changes with the continuous evolution of the reality being processed and with that of case law. How can we evaluate their validity without any absolute reference for this purpose?

Empirical data are like “moving” concepts at the heart of databases. In other words, the concepts meaning evolves with the interpretation of the values they allow to be understood, in the absence of an absolute and stable reference frame.

If we deepen this epistemological finding, which seems to have no operational outcome, it is nevertheless possible to build a widely applicable method for assessing and improving the quality of such information. By now, the question is not only “are the data correct?”, but above all “how are the data built up over time?”.

Thus, with globalization, new cases not initially foreseen in the reference tables and in the legislation may arise at national level. This can happen, for example, in the field of energy activity: geothermal energy production varies greatly around the world, and some foreign companies may therefore use a classification of their operating units (“renewable energy”) that is less precise than the one potentially required, after examination, by the legislation of the country of operation (“geothermal energy”), a category which, in our example, is not yet taken into account in the reference table, which will have to be adapted, as explained below (Figure 2).

In this case, it is not possible to check whether the database values are correct or not in a deterministic way. Indeed, when an inconsistency happens between such a value entered in the database and the reference tables used to test its validity, it may be essential, when the issues are strategic, to carry out a manual verification, by contacting the citizen or company concerned, for example. Additionally, such an intervention may often be used if the expected category in the reference file for a given employer does not correspond to the declared category, as the employer may have changed category since registration without this being recorded (because the employer did not report it, for example).

Among others, this is one of the areas where the interest of an ATMS lies, in recording and archiving anomalies and transactions, allowing continuous monitoring of these and possible subsequent modification of the definition domain to adapt it to a newly observed reality.

Let’s illustrate this mechanism (Figure 2) with another concrete example. In 2005, the catastrophe of Hurricane Katrina killed more than 1,800 people in the USA. The measuring instruments to alert citizens to leave the area were in place. However, in retrospect, we realized that the databases that fed them were not designed to take into account the evolution of certain phenomena which, although underestimated at the time, turned out to be decisive: these were the rise in water levels in the oceans as a result of global warming, as well as over-construction, which no longer allows for the rapid flow of water in the soil. Therefore, the evacuation of the population was far too late. These changes to reality are still at work today in the hydrological and climatic fields (Ouvrir dans un nouvel ongletBoydens, Hamiti and Van Eeckhout, 2020).

Figure 2. Violation of the “Closed World Assumption” in an Empirical Domain

A typology of anomalies then emerges, according to their potential cause and how to look at them:

The last two cases may indicate anomalies due to the evolution over time of the empirical domain represented and emerging new concepts not taken into account (Figure 2).

Depending on what is needed, it will be decided to consider these anomalies as:

The decision to identify “non-blocking” empirical anomalies is sensitive because it is based on predictive knowledge of the realities processed at a given time, which is itself an evolving element likely to be subjected to concerted adaptation within the information system. This brings us back to the epistemological issue of the “hermeneutic loop” (Boydens, 1999; Ouvrir dans un nouvel onglet2012).

How can “non-blocking anomalies” and their processing be taken into account without affecting either the performance or the integrity of data in production?

With the ATMS, or Anomalies and Transactions Management System, we switch from a “closed world assumption” to an “open world hypothesis” under automated controls.

The norm evolution, changes made within the databases, and the movement of observable “phenomena” in the field are interrelated but asynchronous. Depending on their nature, they operate within different time scales.

By applying the historian Fernand Braudel’s notion of “layered time scales” (Braudel, 1949) to the study of administrative information systems with the aim of improving their quality, we can differentiate:

Therefore, Fernand Braudel’s concept of “layered time scales”, is a construction allowing the identification of a hierarchy between several interactive sequences of change. This approach can be supplemented by the work of the German philosopher Norbert Elias and his notion of “evolutionary continuum” (Elias, 1986). Indeed, interactions between time scales are neither deterministic nor unidirectional, as Braudel’s model alone would suggest, in which the slowest sequences determine the fastest ones. Let’s illustrate these interactions with a concrete example.

In 1986, a team of British scientists, specializing in globe studies, reported falling ozone levels in the stratosphere. Based on this observation, NASA researchers re-examined their globally distributed stratospheric databases; they discovered that the phenomenon of declining ozone levels had remained hidden for a decade because the corresponding low values were systematically considered as measurement errors. Indeed, the scientific theory of the time, modelled in their databases, didn’t allow them to conceive that such values could be valid. Subsequently, the definition domain of the database was adapted in order to consider valid the low rates that were previously in an anomalous state (Boydens, 1999). Today, such phenomena continue to evolve.

From a dynamic point of view, an ideal database should therefore match the rate of its updates to the – unpredictable – distribution of changes to the reality it covers in “layered time scales”. On top of what looks to be a challenge is the need, again revealed afterwards, to integrate unforeseen observations, which are prohibited, in principle, by the closed world assumption and are revealed in particular through the anomalies mentioned above.

The ATMS or Anomalies and Transactions Management System helps to identify the emergence and increases in “validations” of anomalies deemed strategic during the manual processing phase. A validation operation means that, after examination, an officer considered that the anomaly actually corresponds to a relevant value. The operator can then “force” the system to accept the value without affecting the integrity of the main database; the system should especially include a version management system. Depending on their access rights, officers have access to both the main database and the ATMS: they can therefore have access to all data at any time, whether or not they are in potential anomaly status, regardless of their processing stage (correction, validation, etc.) (Figure 3).

If the rate of such anomaly validations is high and recurrent, or if the validated anomaly is strategic, it is possible that the database definition domain itself is no longer relevant. In principle, the approach focuses on systematic cases, but it can also cover less frequent cases involving types of anomalies that are sensitive for the area (the emergence of a rare pathology, for example).

An algorithm can then send a “signal” to the database managers so that they examine whether or not is required a structural change to its definition domain or even a revision of the corresponding standard (legislation, theory, etc.). The fluctuations in environmental or administrative data illustrated above are examples of where this mechanism is applied. In addition, their processing history must be kept (the same anomaly may be corrected or validated several times following field inspections or interpretation of regulations).

Without such intervention, the gap between the database and reality would widen. Indeed, if the scheme is not adapted, anomalies linked to these cases will continue to grow, requiring a potentially burdensome manual review that is likely to slow down the files processing and affects the quality of data with financial or social impacts.

Upstream, the ATMS helps to improve the quality of the “source” databases and provides various indicators on the status of the processing of anomalies to those involved in the management of the information system (project manager and project owner). For example, it allows to:

Some of these indicators could also be useful to better exploit administrative sources for statistical purposes. To support the back tracking, the ATMS is an even more powerful instrument for improving data quality, as part of a preventive approach (Figure 1).

The back tracking method is widely applicable and can be used for any application field. Thomas Redman initiated it, under the name data tracking.

Figure 3. The Data Flow between the Application, the Main Database and the ATMS

The data tracking proposed by Thomas Redman of AT&T Labs in the USA (Redman, 1996) aims to quantitatively assess the formal validity of the values entered into a database and to structurally improve its processing (Redman explicitly rules out questions of data interpretation, which he considers to be too complex).

According to Redman, a database is similar to a lake. Instead of occasionally cleaning the lake bottom that is continuously fed by external flows and currents (as advocated by data cleansing, an automatic correction method), Redman proposes, on the basis of a random sample of data taken at the input point (upstream) of the information system, methodically analysing the processes and flows that allow the data to be assembled. The end goal is to determine the causes of the identified formal errors in order to structurally fix them at source (e.g. bugs or programming errors).

The operation is based on the principle that a small number of flows, processes or practices are responsible for a large percentage of formal errors. The approach refers to the Pareto principle, also known as the “80/20 principle”. Therefore, it is based on the assumption that a significant proportion of problem cases (about 80%) are provoked by about 20% of the possible causes.

During the 2000s, the large-scale application of Redman’s method (Boydens, 2010) led to an original method called back tracking (Ouvrir dans un nouvel ongletBoydens, 2018). Still based on Redman’s Pareto principle, back tracking improves upon his method in relation to five important aspects:

Therefore, the back tracking operation is based on a prior monitoring of anomalies and transactions, which is set up after the specification of strategic quality indicators. It is then possible to identify, within the processes and data flows, in partnership with the information provider and the database manager, the elements at the source of the production of a large number of systematic anomalies or anomalies deemed to be strategic: inappropriate processing of certain data sources, emergence of new situations not yet taken into account in the database definition domain, inadequate interpretation of legislation, poorly documented concept, programming errors, etc. On this basis, a diagnosis and sustainable and structural corrective actions can be taken (correction of formal code in programmes, restructuring of processes, adaptation of the interpretation of a law, clarification of documentation, etc.).

In the field of social security in Belgium (in this case, based on the DmfA database mentioned above), the “full-scale” tests carried out in conjunction with developers, specialists in the application field as well as senders of the information were conclusive. The benefits of the method were repeatedly shown in the 2000s (structural reduction in the number of anomalies of 50% to 80%, time saving due to a reduction in the tedious intellectual work of verification, better interpretation of the law, faster financial collection and redistribution, etc.). Given its widely applicable nature, the method was enshrined in Belgian legislation in the form of a binding Royal Decree in 2017 (Ouvrir dans un nouvel ongletBoydens, 2018). This legislation falls within the framework of the “quality barometers” applied to social security. If the effects of the method are lasting and structural, it must be applied on a recurrent basis in order to take into account possible new phenomena, although requiring gradually decreasing effort over time. It is based on a rigorously documented organization, procedures and information system (Boydens, 2010).

The ATMS mechanism that supported the back tracking operations has so far been deployed on a large scale via a hierarchical Database Management System (DBMS) and specific outsourced code associated with a generic rules engine. Now, a new generic ATMS development is applicable to any relational DBMS and to the current technologies.

The Smals Data Quality Skills Centre (Centre de Compétence en Qualité de Données), which is based on a synergy between the Databases department and the Research department, undertook the implementation of a prototype applied to relational DBMSs and related standards. The prototype is based on the Crossroads Bank for Enterprises’s complete open data version, which is the Belgian equivalent of the SIRENE business register in France.

In this new model, the main database and the ATMS are separated, which requires agreement on the routing of data between both systems and the definition of principles for storing the anomalies and associated transactions.

One of the key traits of the specifications of the relational ATMS is the separation made between the main database on the one hand, and the database of anomalies and their processing on the other. Figure 4 illustrates this separation, which is based on the following two principles:

Beyond satisfying theoretical design hypotheses, the separation of the main database and the ATMS opens the door to a number of significant benefits:

Figure 4. Dynamic Separation between the Main Database and the ATMS

When data enter the information system, their compliance with the definition domain (see above) is verified. If non-blocking anomalies are detected, the corresponding data are sent to the ATMS (Figure 3). It is only after being subjected to verification and intellectual processing, automatically logged, that the data will be integrated into the main database. In a case of validation, as discussed above, the definition domain may need to be adapted before records can be accepted into the main database.

This automatic routing does not exclude by any means the possibility of an officer manually triggering an anomaly on data that conform to the definition domain and are already present in the main database. This could be justified, for example, by a field inspection which reveals that a record is fraudulent or obsolete.

The flow of data between the main database and the ATMS must be fluid and standardized; it is therefore implemented in automated procedures, the principle of which has already been illustrated in Figure 3. The bidirectional nature of this flow requires the ability to preserve or reconstruct the original state of the information from the ATMS. To that end, two constituent parts are used to store the definition domain and the associated anomalies and metadata respectively (Figure 4); we will not detail this function here but more references are available online (Ouvrir dans un nouvel ongletBoydens, Hamiti and Van Eeckhout, 2020).

The prototype fully implements the database of the ATMS. This was developed and tested repeatedly by exposing it early on to an information system simulated by a rudimentary application and a realistic main database, the open data of the Belgian Business Register (Répertoire des entreprises belges) (Box 2). In this context, four usage scenarios have been implemented by way of examples:

As the prototype was technically successful and aroused the interest of the contracting authority, a pilot project is underway. This initiative can later be applied in full scale to the databases of the Belgian administration. It is also widely applicable and can be used for any information system. There are several possible scenarios for the implementation of an ATMS. Ideally, this is taken into consideration from the design stage of the information system, together with the main database and data quality tools (Box 1), if available. A re-engineering project is also an appropriate time to integrate an ATMS into an existing system.

The recurrent contributions of back tracking mentioned above, a method that must be supported by an ATMS, applied to the DmfA database constitute a precedent that strongly encourages the widespread application of this prototype adapted to recent technologies: structural reduction in the number of anomalies by 50% to 80%, time saving thanks to a reduction in the tedious intellectual work of verification, a better interpretation of the norm, a faster financial collection and redistribution; more generally speaking, improvement in the quality of any empirical database, establishment of a partnership between the database managers and the suppliers of information, etc. As seen previously, the service can benefit both the managers of administrative databases, the quality of which is improved, and statisticians who want to use them for other purposes.

Ouvrir dans un nouvel ongletArrêté royal du 2 février 2017 modifiant le chapitre IV de l’arrêté royal du 28 novembre 1969 pris en exécution de la loi du 27 juin 1969 révisant l’arrêté-loi du 28 décembre 1944 concernant la sécurité sociale des travailleurs. In: Moniteur belge. [online]. [Accessed 31 May 2021].