Information High quality dimensions
Taking a shopper viewpoint of information high quality is undoubtedly a useful preliminary step. Nevertheless it won’t cowl the completeness of the check scope. Intensive literature critiques have addressed this difficulty for us, providing a range of data quality dimensions which are related to most use instances. It’s advisable to evaluation the record with knowledge customers and collectively decide which dimensions are relevant and create checks accordingly.
| Accuracy | Format | Comparability |
| Reliability | Interpretability | Conciseness |
| Timeliness | Content material | Freedom from bias |
| Relevance | Effectivity | Informativeness |
| Completeness | Significance | Degree of element |
| Foreign money | Sufficiency | Quantitativeness |
| Consistency | Usableness | Scope |
| Flexibility | Usefulness | Understandability |
| Precision | Readability | |
You would possibly discover this record too lengthy and surprise learn how to begin with it. Information merchandise or any info system could be noticed or analyzed from two views: exterior view and inner view.
Exterior view
The exterior view is about the usage of the information and its relation with the group. It’s typically thought of a “black field” with performance to characterize the real-world system. The size that fall into the exterior view are extremely business-driven. Typically, the analysis of these dimensions could be subjective, so it’s not at all times straightforward to create automated checks for them. However let’s take a look at just a few well-known dimensions:
- Relevancy: The extent to which knowledge are relevant and useful for the evaluation. Contemplating a market marketing campaign aimed toward selling a brand new product. All knowledge attributes ought to straight contribute to the success of the marketing campaign reminiscent of buyer demographic knowledge and buy knowledge. Information like metropolis climate or inventory market costs are irrelevant knowledge on this case. One other instance is the extent of element (granularity). If the enterprise needs the market knowledge to be on the day stage, however it’s delivered on the weekly stage, then it’s not related and helpful.
- Illustration: The extent to which knowledge is interpretable for knowledge customers and the information format is constant and descriptive. The significance of the illustration layer is usually missed when accessing knowledge high quality. It consists of the format of the information — being constant and user-friendly, and the which means of the information — being comprehensible. As an example, contemplate a situation the place knowledge is anticipated to be out there in a CSV file with descriptive column descriptions, and the values are anticipated to be in EUR foreign money slightly than in cents.
- Timeliness: The extent to which knowledge is recent for knowledge customers. For instance, the enterprise wants the gross sales transaction knowledge with a most delay of 1 hour from the purpose of sale. It signifies that the information pipeline needs to be refreshed often.
- Accuracy: The extent to which knowledge is compliant with enterprise guidelines. Information metrics are sometimes related to sophisticated enterprise guidelines reminiscent of knowledge mapping, rounding modes, and so forth. Automated checks on knowledge logic are extremely beneficial and the extra, the higher.
Out of the 4 dimensions, relating to creating knowledge checks, timeliness and accuracy are extra simple. Timeliness is achieved by evaluating the timestamp column with the present timestamp. Accuracy checks are possible by way of buyer queries.
Inner view
In distinction, the interior view is anxious with the operation that is still impartial of particular necessities. They’re important whatever the use instances at hand. Dimensions within the inner view are extra technical-driven versus business-driven dimensions within the exterior view. It additionally signifies that knowledge checks are much less depending on customers and could be automated more often than not. Listed below are just a few key views:
- High quality of information supply: The standard of the information supply considerably impacts the general high quality of the ultimate knowledge. The info contract is a superb initiative to make sure supply knowledge high quality. As knowledge customers of the supply, we are able to make use of an identical method to observe the supply knowledge as knowledge stakeholders do when evaluating the information merchandise.
- Completeness: The extent to which info is retained in its entirety. Because the complexity of the information pipeline will increase, there’s a increased probability of knowledge loss occurring throughout the intermediate levels. Let’s contemplate a monetary system that shops buyer transaction knowledge. The completeness check ensures that each one transactions efficiently traverse your complete lifecycle with out being omitted or ignored. For instance, the ultimate account steadiness ought to precisely mirror the real-world scenario, capturing each transaction with none omissions.
- Uniqueness: This dimension goes hand-in-hand with the completeness check. Whereas completeness ensures that nothing is misplaced, uniqueness ensures that no duplication happens throughout the knowledge.
- Consistency: The extent to which knowledge is constant throughout inner methods each day. The discrepancy is a standard knowledge difficulty that usually stems from knowledge silos or inconsistent metric calculation strategies. One other facet of the consistency difficulty happens between days when knowledge is anticipated to have a gentle progress sample. Any deviation ought to increase a flag for additional investigation.
It’s price noting that every dimension could be related to a number of knowledge checks. What’s essential is knowing the suitable utility of dimensions to particular tables or metrics. Solely then, the extra checks employed, the higher.
So far, we’ve mentioned the size of exterior views and inner views. In future knowledge check designs, it’s necessary to think about each the exterior and inner views. By asking the fitting inquiries to the fitting folks, we are able to improve effectivity and cut back miscommunication.
