Tejasvi Addagada

Head Data Management & Governance | Data Privacy Officer

September 28, 2021

The Value of a Data Catalog

Most organizations are moving their data onto the Cloud for improved versatility to support the needs of their business by being able to dynamically scale both horizontally and vertically. Once this data is moved to the Cloud, enterprises can also begin unlocking the value of the data by generating insights from AI and ML services. Data platform offerings from cloud vendors traditionally included traditional databases like SQL server or warehouses but recently, have also included data lakes, cloud warehouses, data engineering, ETL stacks, and analytical workbenches for storing this data at scale. However, such data modernization by moving to the Cloud also creates complexity to actively manage and govern data given its distribution across these hybrid and multi-cloud landscapes. Within this context, the value of a data catalog which can integrate metadata across different environments cannot be overstated.

 
For someone hearing about a catalog for the first time – It is a tool that can discover data assets and assist the business and data analysts to understand the structure of data elements which they would want to use in their projects.
 

At it’s core, a catalog scans any database ranging from relational databases to NoSQL or Graph and gives out useful information”. Some of this useful information includes –

 

    1. Table & Column Name

    2. Modeled data-type

    3. Inferred data types

    4. Patterns of data

    5. Data Length with minimum and largest threshold

    6. Minimal and maximum values

    7. Other profiling characteristics of data like frequency of values and their distribution

    8. End-End lineage across data sources along with Extract Transform Load characteristics

 

However, most on-prem catalog vendor solutions lack the functionality to scan different environments and specifically lack the capabilities to scan cloud object storage. Cloud vendor solutions on the other hand, have cloud connectors which provide deeper insight into the physical location of data assets and their semantic meaning, for distributed assets being managed across different environments. To contrast Cloud-based catalogs with traditional on-prem catalogs, let’s pick a few examples of the former and dig a bit deeper into their core capabilities.

 

Managing Metadata with a Catalog on the Cloud

 

To pick an example- AWS Glue catalog provides features that include scanning relevant storages, such as S3, RDS, Redshift etc. to build out the physical characteristics of the tables and files. Further, references to data that is used as sources and targets in Glue are stored as metadata in the catalog. This helps provide the lineage of the data as it moves through its lifecycle. Similarly, Azure has a catalog offering named Purview.

Let us look at how a scanner popularly called a crawler works in the Glue data catalog.

 

    1. The crawler connects to the data store of choice in AWS storage, such as S3 or Redshift. Connection properties would include configuring the data store properties, data paths etc.

    2. The type of crawling required – Full folder scans or incremental folder scans

    3. The inferred schema is then created from your data stores.

    4. The crawler writes metadata in the Data Catalog

    5. It creates definitions for databases and tables.

 
 

The Benefits of Metadata

Most data catalogs like Glue are not built to be glossaries but instead are used for the maintenance of schemas. The use of metadata has exploded over the last year, much of this attributed to technological advancements and public policy changes related to GDPR and CCPA regulations, to name a few. Some of the benefits of an Enterprise metadata function include simplification of the data landscape, democratizing search for data assets, and managing schema drifts.

 
The science of data discovery is associated with the doctrine of data-democratization and answers questions such as “Where does data exist physically?” in schemas as objects and instances as elements or searching for data in single or multiple applications systems-of-records, systems of reference like datalakes or cloud warehouses. Metadata Management also requires analysts to put information at a rapid pace into a catalog at the right stage of the change. Analysts will also require technical metadata from databases to be available instantly to be able to define them. An operating model can be defined by including the right stakeholders, process including toll-gates during change advisory board or a change committee. Data, as well, has a lifecycle, POSMAD (Plan, Obtain, Store/Share, Maintain, Apply, Decay), which when documented helps bring out the lineage. Having lineage will make it easier to perform impact analysis as well as to maintain the flows of data as part of data pipelines.
 
Even enriched agile management models like scrum, kanban, DAD, FDD can benefit from curating and using the definitions of data, in projects, for accelerated delivery of features. Data Governance can enable a balance of hosting and serving of metadata guaranteeing that metadata works for most use-cases. As data governance formalizes active management of metadata through a specific operating rhythm and processes, it becomes much easier to integrate it into project life cycles planning for data changes or usage.
 

To summarize, what are the basic benefits of managing Metadata in catalog?

 

1. Increased availability of intelligence about data that brings out better context to insights

 

2. The reduced turnaround time to find answers during the analysis

 

3. Increased efficiency of subject-matter-experts in turning out information for impact analysis

 

4. Removes ambiguity in relationships among data in the landscape

 

5. Simplifies the views of data through meaning, identified redundancy, and relationships

 
 What do you think? I’d love to hear your thoughts on this topic.