<div>
      <div>
      <div>
      <p>Data matters more than ever &ndash; we all know that. But at a time when being a data-driven business is so critical, how much can we trust data and what it tells us? That&rsquo;s the question behind data reliability, which focuses on having complete and accurate data that people can trust. This article will explore everything you know about data reliability and the important role of data observability along the way, including:</p>
      <ul>
      <li>What is data reliability?</li>
      <li>Why is it important?</li>
      <li>How do you measure it?</li>
      <li>Data quality vs. data reliability: What&rsquo;s the difference?</li>
      <li>What is a data quality framework?</li>
      <li>How can observability help improve data reliability?</li>
      <li>Top data reliability testing tools</li>
      </ul>
      </div>
      </div>
      <h3>What is data reliability?</h3>
      <div>
      <div>
      <p>Data reliability looks at the completeness and accuracy of data, as well as its consistency across time and sources. The consistency piece is particularly important, as data needs to be consistent to be truly reliable, that way it&rsquo;s always trustworthy.</p>
      <p>Data reliability is one element of data quality.</p>
      <p>Specifically, it helps build trust in data. It&rsquo;s what allows us to make data-driven decisions and take action confidently based on data. The value of that trust is why more and more companies are introducing Chief Data Officers &ndash; with the number doubling among the top publicly traded companies between 2019 and 2021,&nbsp;<a class="mds-text-primary" href="https://www.pwc.com/us/en/tech-effect/ai-analytics/building-trust-in-data.html" target="_blank" rel="noopener">according to PwC</a>.</p>
      </div>
      </div>
      <h3>Why is data reliability important?</h3>
      <div>
      <div>
      <p>Measuring data reliability requires looking at three core factors:</p>
      <ol>
      <li><strong>Is it valid?&nbsp;</strong>Validity of data looks at whether or not it&rsquo;s stored and formatted in the right way. This is largely a data quality check.</li>
      <li><strong>Is it complete?&nbsp;</strong>Completeness of data identifies if anything is missing from the information. While data can be valid, it might still be incomplete if critical fields are not present that could change someone&rsquo;s understanding of the information.</li>
      <li><strong>Is it unique?&nbsp;</strong>The uniqueness of data checks for any duplicates in the data set. This uniqueness is important to avoid over-representation, which would be inaccurate.</li>
      </ol>
      <p>To take it one step further, some teams also consider factors like:</p>
      <ul>
      <li>If and when the data source was modified</li>
      <li>What changes were made to data</li>
      <li>How often the data has been updated</li>
      <li>Where the data originally came from</li>
      <li>How many times the data has been used</li>
      </ul>
      <p>Overall, measuring data reliability is essential to not just help teams trust their data, but also to identify potential issues early on. Regular and effective data reliability assessments based on these measures can help teams quickly pinpoint issues to determine the source of the problem and take action to fix it. Doing so makes it easier to resolve issues before they become too big and ensures organizations don&rsquo;t use unreliable data for an extended period of time.</p>
      </div>
      </div>
      <h3>Data quality vs. data reliability: What&rsquo;s the difference?</h3>
      <div>
      <div>
      <p>All of this information begs the question: What&rsquo;s the difference between&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/what-is-good-data-quality-for-data-engineers/" target="_blank" rel="noreferrer noopener">data quality</a>&nbsp;vs. data reliability?</p>
      <p>Quite simply, data reliability is part of the bigger data quality picture. Data quality takes on a much bigger focus than reliability, looking at elements like completeness, consistency, conformity, accuracy, integrity, timeliness, continuity, availability, reproducibility, searchability, comparability, and &ndash; you guessed it &ndash; reliability.</p>
      <p>For&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/whats-the-difference-data-engineer-vs-data-scientist-vs-analytics-engineer/" target="_blank" rel="noreferrer noopener">data engineers</a>, there are typically four data quality dimensions that matter most:</p>
      <ul>
      <li><strong>Fitness:&nbsp;</strong>Is the data fit for its intended use, which considers accuracy and integrity throughout its lifecycle.</li>
      <li><strong>Lineage:&nbsp;</strong>Where and when did the data come from and where did it change, which looks at source and origin.</li>
      <li><strong>Governance:&nbsp;</strong>Can you control the data, which takes into account what should and shouldn&rsquo;t be controllable and by whom, as well as privacy, regulations, and security.</li>
      <li><strong>Stability:&nbsp;</strong>Is the data complete and available in the right frequency, which includes consistency, dependability, timeliness, and bias.</li>
      </ul>
      <p>Fitness, lineage, and stability all have elements of data reliability throughout them. Although taken as a whole, data quality clearly encompasses a much larger picture than data reliability.</p>
      </div>
      </div>
      <h3>What is a data quality framework?</h3>
      <div>
      <div>
      <p>A&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/data-quality-framework/" target="_blank" rel="noreferrer noopener">data quality framework</a>&nbsp;allows organizations to define relevant data quality attributes and provide guidance for processes to continuously ensure data quality meets expectations. For example, using a data quality framework can build trust in data by ensuring what team members view is always accurate, up to date, ready on time, and consistent.</p>
      <p>A good data quality framework is actually a cycle, which typically involves six steps largely led by data engineers:</p>
      <ol>
      <li><strong>Qualify:&nbsp;</strong>Understand a list of requirements based on what the end consumers of the data need.</li>
      <li><strong>Quantify:&nbsp;</strong>Establish quantifiable measures of data quality based on the list of requirements.</li>
      <li><strong>Plan:&nbsp;</strong>Build checks on those data quality measures that can run through a data observability platform.</li>
      <li><strong>Implement:&nbsp;</strong>Put the checks into practice and test that they work as expected.</li>
      <li><strong>Manage:&nbsp;</strong>Confirm the checks also work against historical pipeline data and, if so, put them into production.</li>
      <li><strong>Verify:&nbsp;</strong>Check with data engineers and data scientists that the work has improved performance and delivers the desired results, and check that the end consumers of the data are getting what they need.</li>
      </ol>
      </div>
      </div>
      <h3>How can observability help improve data reliability?</h3>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://databand.ai/data-observability/" target="_blank" rel="noreferrer noopener">Data observability</a>&nbsp;is about understanding the health and state of data in your system. It includes a variety of activities that go beyond just describing a problem. Data observability can help identify, troubleshoot, and resolve data issues in near real-time.</p>
      <p>Importantly, data observability is essential to getting ahead of&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/the-impact-of-bad-data-and-why-observability-is-now-imperative/" target="_blank" rel="noreferrer noopener">bad data</a>&nbsp;issues, which sit at the heart of data reliability. Looking deeper, data observability encompasses activities like monitoring, alerting, tracking, comparisons, analyses, logging, and SLA tracking, all of which work together to understand end-to-end data quality &ndash; including data reliability.</p>
      <p>When done well, data observability can help improve data reliability by making it possible to identify issues early on to respond faster, understand the extent of the impact, and restore reliability faster as a result of this insight.</p>
      </div>
      </div>
      <h3>Top data reliability testing tools</h3>
      <div>
      <div>
      <p>Understanding the importance of data reliability, how it sits within a broader data quality framework, and the importance of data observability is a critical first step. Next, taking action to invest in it requires the right technology.</p>
      <p>With that in mind, here&rsquo;s a look at the top data reliability testing tools available to data engineers. It&rsquo;s also important to note that some of these solutions are often referred to as data observability tools since better observability leads to better reliability.</p>
      </div>
      </div>
      <h4>1) Databand</h4>
      <div>
      <div>
      <p>Databand is a&nbsp;<a class="mds-text-primary" href="https://databand.ai/platform/" target="_blank" rel="noreferrer noopener">data observability platform</a>&nbsp;that helps teams monitor and control data quality by isolating and triaging issues at their source. With Databand, you can know what to expect from data by identifying trends, detecting anomalies, and visualizing data reads. This allows a team to easily alert the right people in real time about issues like missing data deliveries, unexpected data schemes, and irregular data volumes and sizes.</p>
      </div>
      </div>
      <h4>2) Datadog</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://www.datadoghq.com/" target="_blank" rel="noopener">Datadog&rsquo;s</a>&nbsp;observability platform provides visibility into the health and performance of each layer of your environment at a glance. It allows you to see across systems, apps, and services with customizable dashboards that support alerts, threat detection rules, and AI-powered anomaly detection.</p>
      </div>
      </div>
      <h4>3) Great Expectations</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://greatexpectations.io/" target="_blank" rel="noopener">Great Expectations</a>&nbsp;offers a shared, open standard for data quality. It makes data documentation clean and human-readable, all with the goal of helping data teams eliminate pipeline debt through data testing, documentation, and profiling.</p>
      </div>
      </div>
      <h4>4) New Relic</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://newrelic.com/platform/applied-intelligence" target="_blank" rel="noopener">New Relic&rsquo;s</a>&nbsp;data observability platform offers full-stack monitoring of network infrastructure, applications, machine learning models, end-user experiences, and more, with AI assistance throughout. They also have solutions specifically geared towards AIOps observability.</p>
      </div>
      </div>
      <h4>5) Bigeye</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://www.bigeye.com/" target="_blank" rel="noopener">Bigeye</a>&nbsp;offers a data observability platform that focuses on monitoring data, rather than data pipelines. Specifically, it monitors data freshness, volume, formats, categories, outliers, and distributions in a single dashboard. It also uses machine learning to set forecasting for alert thresholds.</p>
      </div>
      </div>
      <h4>6) Datafold</h4>
      <div>
      <div>
      <p><a class="mds-text-primary" href="https://www.datafold.com/data-observability" target="_blank" rel="noopener">Datafold</a>&nbsp;offers data reliability with features like regression testing, anomaly detection, and column-level lineage. They also have an&nbsp;<a class="mds-text-primary" href="https://github.com/datafold/data-diff/" target="_blank" rel="noopener">open-source</a>&nbsp;command-line tool and Python library to efficiently diff rows across two different databases.</p>
      <p>In addition to these five tools, others available include PagerDuty, Datafold, Monte Carlo, Cribl, Soda, and Unravel.</p>
      </div>
      </div>
      <h3>Make Data Reliability a Priority</h3>
      <div>
      <div>
      <p>The risks of bad data combined with the competitive advantages of quality data mean that data reliability must be a priority for every single business. To do so, it&rsquo;s important to understand what&rsquo;s involved in assessing and improving reliability (hint: it comes down in large part to data observability) and then to set clear responsibilities and goals for improvement.</p>
      </div>
      </div>
      </div>

What Is Data Reliability And How Observability Can Help ?

<div>
      <div>
      <div>
      <p>Data is the most valuable asset for most businesses today. Or at least it has the potential to be. But to realize the full value, organizations must manage their data correctly. This management covers everything from how it&rsquo;s collected to how it&rsquo;s maintained and analyzed. And a big component of that is data governance.</p>
      <p>Data governance refers to the policies, processes, roles, and technology that businesses use to ensure data availability, usability, integrity, and security. This article will explore everything you need to know about data governance, including:</p>
      <ul>
      <li>What is it?</li>
      <li>What&rsquo;s the difference between data governance vs. data management?</li>
      <li>Why is it important?</li>
      <li>What are the components of effective data governance?</li>
      <li>What are the key roles involved in it?</li>
      <li>What are its best practices?</li>
      </ul>
      </div>
      </div>
      <h3>What is Data Governance?</h3>
      <div>
      <div>
      <p>Data governance is a core component of any big data management strategy that organizations introduce to drive insights. Effective data governance ensures quality and consistency in the data used to power critical business decisions.</p>
      <p>At a high level, it can refer to data roles and responsibilities, data accessibility, data policies and processes, data creation procedures, data flows, and more. Digging deeper, it defines the architecture for decision-making and access rights around data, answering questions like:</p>
      <ul>
      <li>How do we define data?</li>
      <li>Where does data come from?</li>
      <li>How do we confirm the quality of data?</li>
      <li>How do we use data?</li>
      <li>Where do we store data?</li>
      <li>How do we protect data?</li>
      <li>How do we organize data?</li>
      <li>How do we connect data across systems?</li>
      <li>How do we maintain a current inventory of our data?</li>
      <li>How accurate does that data inventory need to be?</li>
      </ul>
      <p>Software for data governance can either be purpose-built or baked into applications that make up the<a class="mds-text-primary" href="https://www.kdnuggets.com/2021/11/modern-data-stack-leaving-behind.html" target="_blank" rel="noopener">&nbsp;modern data stack</a>.</p>
      </div>
      </div>
      <h3>What&rsquo;s the Difference Between Data Governance vs. Data Management?</h3>
      <div>
      <div>
      <p>Data governance and data management are often used interchangeably, however, the two terms refer to different practices.</p>
      <p>It sets the strategy by introducing policies and procedures throughout the data lifecycle. Meanwhile, data management is the practice of enforcing those policies and procedures so that the data is ready for use.</p>
      <p>In short, it is the cornerstone of all data management initiatives.</p>
      </div>
      </div>
      <h3>Why is Data Governance Important?</h3>
      <div>
      <div>
      <p>In today&rsquo;s data-driven world, organizations need effective data governance to be able to trust in the quality and consistency of their data.</p>
      <p>A strong approach to data governance benefits the entire organization by giving individuals a clear way to access data, shared terminology to discuss data, and a standard way to understand data and make it meaningful.</p>
      <p>Some of the key benefits of data governance include:</p>
      <ul>
      <li>Introducing a clear&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/data-quality-framework/" target="_blank" rel="noreferrer noopener">data quality framework</a>&nbsp;to bring together data and create a shared understanding for better insights and decisions</li>
      <li>Improving consistency of data across systems and processes, for efficient&nbsp;<a class="mds-text-primary" href="https://databand.ai/mad-data-podcast/modern-data-love-why-data-observability-data-integration-belong-together/" target="_blank" rel="noreferrer noopener">data integration</a></li>
      <li>Clearly defining policies and procedures around data-related activities to ensure standardization across the entire organization</li>
      <li>Outlining roles and responsibilities in terms of data management and data access for clarity among stakeholders</li>
      <li>Improving compliance by allowing for faster response and resolution to data incidents</li>
      </ul>
      <p>On the flip side, poor data governance can hamper regulatory compliance initiatives, which can create problems for companies when it comes to satisfying new data privacy and protection laws.</p>
      </div>
      </div>
      <h3>What are the Components of Effective Data Governance?</h3>
      <div>
      <div>
      <p>In order for it to be effective, it must encompass several key components that support the follow-on data management activities. These components include:</p>
      <h4>Data Standards</h4>
      <p>It should set explicit data standards for consistency across the entire organization. These standards should assess and verify data quality and should be transparent to everyone in the company. As a result, they should help teams better comprehend and use data.</p>
      <p>Data standards should also allow any third-party auditors to easily see how the organization handles sensitive data, how that data gets used, and why it gets used in that way. This transparency is essential for compliance, especially in the case of a data breach.</p>
      <h4>Data Integration</h4>
      <p><a class="mds-text-primary" href="https://databand.ai/blog/what-is-data-integration/" target="_blank" rel="noreferrer noopener">Data integration</a>&nbsp;brings together data from diverse sources to make data more readily available and power deeper insights. Good data governance requires a complete understanding of how data gets integrated across systems and processes. Specifically, the data governance program should define the tools, policies, and procedures used to pass data across systems and combine information.</p>
      <p>As a best practice, these data integration guidelines should be clear and easy to follow to ensure every new system adheres to them. Additionally, the team responsible for data governance should assist in reviewing these guidelines during any new technology implementations.</p>
      <h4>Data Security</h4>
      <p>Protecting the security of data is essential, as any unauthorized access to data or even loss of data can pose serious risks &ndash; from dangers to the subjects of data to financial loss to reputational damage. A data governance framework outlines a variety of elements related to data security, including where data is stored, how it&rsquo;s accessed, and what level of availability it has.</p>
      <p>Specifically, it should detail defenses like authentication tools and encryption algorithms that need to be implemented to protect the data network. Then, any teams working on data governance should partner closely with IT security to ensure adequate protection measures are in place based on those guidelines.</p>
      <h4>Data Lifecycle Management</h4>
      <p>Understanding the organization&rsquo;s data lifecycle means knowing where data resides at any given time as it moves through systems until it eventually gets discarded. Good data governance allows you to quickly discover and isolate data at any point in the lifecycle.</p>
      <p>This concept, also known as data lineage, allows analysts to trace data back to its source to confirm trustworthiness.</p>
      <h4>Data Observability</h4>
      <p><a class="mds-text-primary" href="https://databand.ai/data-observability/" target="_blank" rel="noreferrer noopener">Data observability</a>&nbsp;allows you to understand the health and state of data in your system to identify and resolve issues in near real-time. It includes a variety of activities that go beyond just describing the problem, providing context to also resolve the problem and work to prevent it from recurring.</p>
      <p>Data governance helps set the framework for data observability, setting guidelines for what to monitor and when and what thresholds should set off alerts when something isn&rsquo;t right. A good&nbsp;<a class="mds-text-primary" href="https://databand.ai/platform/" target="_blank" rel="noreferrer noopener">data observability platform</a>&nbsp;can handle these activities, making it important to choose a platform that can meet the requirements for identifying, troubleshooting, and resolving problems outlined in your strategy.</p>
      <h4>Metadata Management</h4>
      <p>Another critical component of data governance is metadata management, which focuses on maintaining consistent definitions of data across systems. This consistency is important to ensure data flows smoothly across integrated solutions and that everyone has a shared understanding of the data.</p>
      <p>The framework should include details on data definition, data security, data usage, and data lineage. In doing so, it should make it possible to clearly identify and classify all types of data in a standardized way across the organization.</p>
      <h4>Data Stewardship</h4>
      <p>Data stewardship is the practice that guarantees your organization&rsquo;s data is accessible, usable, secure, and trustworthy. While the data governance strategy determines your organization&rsquo;s goals, risk tolerance, security standards, and strategic data needs to set high-level policies, data stewardship focuses on making sure those policies get implemented.</p>
      <p>To achieve this follow-through, data stewardship assigns clear roles and responsibilities for various initiatives outlined in the strategy.</p>
      </div>
      </div>
      <h3>What are the Key Roles Involved in Data Governance?</h3>
      <div>
      <div>
      <p>Data governance programs can only succeed if they have clearly defined roles and responsibilities. As a result, it&rsquo;s important to identify the right people within your organization to take on this ownership and establish their roles in the program.</p>
      <p>Specifically, every data governance program requires people in three critical roles, each of which must be filled with qualified individuals who understand their specific responsibilities and how they contribute to the bigger picture. These roles include:</p>
      <h4>Chief Data Officer</h4>
      <p>The Chief Data Officer is the data governance leader. This person is responsible for overseeing the entire program, including enforcing and implementing all policies and procedures and leading the data committee and data stewards.</p>
      <h4>Data Committee</h4>
      <p>The data committee is a group of individuals that sets data governance policies and procedures, including rules for how data gets used and who can access it. They also resolve any disputes that arise regarding data usage or its role within the organization. The committee&rsquo;s purpose is to promote&nbsp;<a class="mds-text-primary" href="https://databand.ai/blog/what-is-good-data-quality-for-data-engineers/" target="_blank" rel="noreferrer noopener">data quality</a>&nbsp;and ensure that data owners and data stewards have what they need at every point in the data lifecycle to do their jobs effectively.</p>
      <h4>Data Stewards</h4>
      <p>The data stewards are responsible for carrying out the data governance policies set by the data committee. They oversee data, making sure everything adheres to policies throughout the entire data lifecycle from creation to archival. The data stewards also train new staff on policies.</p>
      <p>In some cases, data stewards might also be the data owners. In other cases, those might be two separate groups. Either way, the data owners are the people who manage the systems that create and house data.</p>
      </div>
      </div>
      <h3>What are Data Governance Best Practices?</h3>
      <div>
      <div>
      <p>When it comes to getting data governance off the ground (or improving what your organization already has in place) there are several best practices to consider:</p>
      <h4>Get Buy-In from the Top</h4>
      <p>As with any initiative, buy-in for data governance needs to start at the top. This top-down buy-in is important to make sure that everyone in the organization adheres to data governance policies and that those who are in a position to influence that acceptance understand the importance of your work.</p>
      <p>To achieve this buy-in, share with executives how your data governance plan can help them realize their strategic objectives. The more you can highlight the advantages of the program and how it relates to their work, the easier it will be.</p>
      <h4>Communicate Often</h4>
      <p>Communication beyond top-level executives is essential to effective data governance. To ensure everyone is aware of what your team is doing around data governance and why it matters, make a list of everyone in the organization who has a stake in or would be affected by that work.</p>
      <p>Then establish regular communications to share updates about program changes, roadblocks, and successes, that way everyone knows where to go for updates and can stay informed on a regular basis.</p>
      <h4>Combine Long-Term Goals with Short-Term Gains</h4>
      <p>When it comes to data governance, you won&rsquo;t be able to tackle everything at once. Instead, it should be a continuous effort to support data-driven decision-making and open up new opportunities for people throughout the organization.</p>
      <p>As a result, your long-term plan needs to include smaller, short-term initiatives that you can weave into the day-to-day operations of your company for immediate wins. This approach ensures that you see progress quickly and can help uncover any potential roadblocks faster. It also opens the door to new ideas that can even improve your long-term plan.</p>
      <h4>Assign Clear Responsibility &ndash; and Train People Accordingly</h4>
      <p>You can&rsquo;t simply assign someone the role of data steward and hope for the best. You need to make sure that anyone playing a role in your data governance program takes their part seriously, and that means you need to take their responsibilities just as seriously.</p>
      <p>This means you need to be clear about the responsibilities that data stewards and data committee members take on and offer training to support those people in their data governance roles. This training should cover everything from why it is so important to what&rsquo;s expected from people in different roles.</p>
      <h4>Audit Process Adoption</h4>
      <p>A big part of data governance involves developing processes for how the company will handle data, especially when it comes to sensitive information. Auditing how these processes are actually living in your organization and how well people are adopting them can be extremely informative as you continue to make program improvements.</p>
      <p>That&rsquo;s because even the best processes won&rsquo;t do your organization any good if no one adheres to them.</p>
      </div>
      </div>
      <h3>Regularly Measure Progress and Keep an Eye Toward Improvements</h3>
      <div>
      <div>
      <p>Finally, remember that data governance is not a one-and-done effort. It&rsquo;s a program that must continuously evolve based on factors like adoption and changing business needs.</p>
      <p>As a result, it&rsquo;s important to regularly check in on how policies are faring and the impact on data quality. The more you can measure that progress, the better you can manage the situation and identify what&rsquo;s working well and what needs to be improved.</p>
      </div>
      </div>
      </div>

What Is Data Governance And Where Observability Fits In ?

The Rise of DataOps

<div>
      <p>Everyone is talking about the modern data stack (MDS) nowadays. I am a data system person. I started building core database systems in the big data era, and have witnessed the birth and prosperity of cloud computing over the last decade. But the first time I came across the term &ldquo;modern data stack,&rdquo; I felt confused - is it just yet another buzzword that the cloud service vendors created to attract people&rsquo;s eyeballs? There are so many articles online, but most of them are quite markety and salesy. After running a startup building core systems in the modern data stack domain for a while, I would like to share my thoughts. In this article, I will explain &ldquo;modern data stack&rdquo; to you in simple terms, and discuss why modern data stack can really matter in companies.</p>
      <h3 tabindex="-1">What is a modern data stack?</h3>
      <p>The most general (and perhaps most unclear) explanation of the modern data stack is that it is a set of tools built around a data warehouse to simplify data integration. The fundamental starting point of a modern data stack is to save data engineers and data analysts time. Here, data integration essentially is to put data from different data sources together, and then analyze and operate. For example, in an e-commerce website, there must be data sources such as user access data, user order data, and commodity information data. By putting these data together in certain ways, we can get some knowledge, such as which kinds of users are interested in which products, which types of products are best to sell, and so on. The modern data stack, in general, is all about helping people turn data into knowledge.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/fivetran_mds_f4bb971fab.png" alt="" /></p>
      <center>An illustration of the modern data stack from Fivetran</center>
      <p>The image above is from the&nbsp;<a class="mds-text-primary" href="https://youtu.be/Zag5yeTPtPU?t=446">sharing</a>&nbsp;of&nbsp;<a class="mds-text-primary" href="https://www.fivetran.com/">Fivetran</a>&nbsp;CEO&nbsp;<a class="mds-text-primary" href="https://twitter.com/frasergeorgew">George Fraser</a>&nbsp;at Tableau Conference 2019. In the sharing George explains what a modern data stack looks like. The data warehouse takes input from various data sources, transform it internally inside the warehouse, and provides well-modeled data to support business intelligence,</p>
      <p>After reading the above definition of the modern data stack, you may ask: Why is this architecture "modern"? Does it sound like a buzzword for marketing gimmicks? It is true that the term "modern data stack" sounds so marketing-oriented that people might even dislike it. Before we dive deeper into the modern data stack, let's explore who invented the term first.</p>
      <h3 tabindex="-1">Who invented the term "modern data stack"?</h3>
      <p>Frankly speaking, it is very difficult to find a very clear answer regarding its roots due to existing ambiguities. However, it is certain that the term was popularized by&nbsp;<a class="mds-text-primary" href="https://www.fivetran.com/">Fivetran</a>,&nbsp;<a class="mds-text-primary" href="https://www.getdbt.com/">dbt</a>, and the venture capitals behind them.</p>
      <p>By searching on the internet, we can find that the first time this word first became popular after 2020. The most famous speech/article is from the second half of 2020 by dbt CEO&nbsp;<a class="mds-text-primary" href="https://twitter.com/jthandy">Tristan Handy</a>. In his article, &ldquo;<a class="mds-text-primary" href="https://www.getdbt.com/blog/future-of-the-modern-data-stack/">The Modern Data Stack: Past, Present, and Future</a>", the development of the modern data stack is into three eras: Cambrian Explosion I (2012-2016), Deployment (2016-2020), and Cambrian Explosion II (2020-2025). I suggest readers read the original article for the exact definitions of the three eras. In the same year, another report from&nbsp;<a class="mds-text-primary" href="https://a16z.com/">a16z</a>, a well-known venture capital, was published, titled "<a class="mds-text-primary" href="https://future.com/emerging-architectures-for-modern-data-infrastructure-2020/">Emerging Architectures for Modern Data Infrastructure: 2020</a>". This article explains the data infrastructure used by various tech companies in the United States in 2020 from VC&rsquo;s perspective.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/google_trend_MDS_0b1b6b4089.png" alt="" /></p>
      <center>The search popularity of &ldquo;modern data stack&rdquo; on Google Trends</center>
      <p>In fact, after a deeper search, you will find that&nbsp;<a class="mds-text-primary" href="https://twitter.com/frasergeorgew">George Fraser</a>, CEO of Fivetran, mentioned this concept in&nbsp;<a class="mds-text-primary" href="https://www.fivetran.com/blog/competitive-analytics-requires-modern-architecture">a speech in 2019</a>. On Twitter, Tristan also said that he proposed a similar idea of the &ldquo;modern BI stack&rdquo; back in 2016.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/twitter_yingjun_a3e172def3.png" alt="" /></p>
      <center>Reply from dbt CEO Tristan Handy on Twitter.</center>
      <h3 tabindex="-1">Why do we need a modern data stack?</h3>
      <p>Knowing the history of the modern data stack, we can go back to the previous question: why is this architecture "modern"? Why do we need a "modern" data stack? Was the previous data stack not perfect? To answer these questions, I think we should focus on "data", not "modern". It is data that lies at the heart of the modern data stack, not technology. The essence of this terminology is that it changes the way companies use technology: companies no longer build a stack for a certain technology product, but build a stack for their own data.</p>
      <p>Let&rsquo;s think back on how companies managed their data 20 years ago. Twenty years ago, corporate data was basically stored in database systems sold by Oracle, IBM, and Microsoft. Some typical characterizations in these use cases are the following.</p>
      <ul>
      <li>First, the database systems sold by tech giants such as Oracle, IBM, and Microsoft are expensive, and not all companies can afford those database systems.</li>
      <li>Second, database vendors provide consulting and support services, not operation and maintenance services. Usually, enterprises need to hire a dedicated DBA team to operate and maintain these databases and build applications on top of these databases.</li>
      <li>Third, neither the size of data nor the number of applications was large.</li>
      </ul>
      <p>Because of these reasons, when an enterprise tried to manage its own data, they were not really building a framework for the data. Instead, they were building a framework for the database.</p>
      <p>Now time is different. In the past 20 years, the scale of data and the number of applications of enterprises have grown exponentially. A number of database products have emerged from the needs of modern enterprises. The development and popularization of cloud computing have further stimulated more enterprises to use databases. On the other hand, the explosion of the market has also led to sustainable developments in the database field. Compared with 20 years ago:</p>
      <ul>
      <li>
      <p>The price and the usage bar of databases have been largely reduced.</p>
      </li>
      <li>
      <p>Database vendors do not only sell software, but services.</p>
      </li>
      <li>
      <p>Databases no longer compete for performance, but for ease of use.</p>
      </li>
      </ul>
      <p>When facing massive data and applications, if the price of data software is low enough and the performance is good enough, companies can save their time from the dirty work of dealing with the databases, and focus on how to make data management easier. That is exactly the original intention behind the modern data stack: making data management easier.</p>
      <h3 tabindex="-1">How to make data management easier?</h3>
      <p>As mentioned above, the advocates of the modern data stack include dbt, Fivetran, a16z, other data software companies, and many venture capitals. What they all want to change is the way how companies prepare data for analytics: from the traditional ETL (Extract, Transform, Load) to the ELT (Extract, Load, Transform). Below is an illustration also from George Fraser's&nbsp;<a class="mds-text-primary" href="https://youtu.be/Zag5yeTPtPU?t=488">sharing</a>&nbsp;at Tableau Conference 2019 explaining the difference between ETL and ELT. Here ETL means that to ingest the data source into the data warehouse, it must go through three steps: data extraction, transformation, and loading. On the contrary, what ELT means is that only data extraction and data loading is needed before loading data into the cloud data warehouses. Data transformation is carried out directly inside the data warehouse.</p>
      <p><img style="width: 100%" src="https://singularity-strapi-prod.s3.ap-southeast-1.amazonaws.com/fivetran_ELT_ELT_7c8954776f.png" alt="" /></p>
      <center>The difference between ETL and ELT</center>
      <p>Turning the traditional ETL into the new type of ELT allows the complex computation of data to be moved from outside the data warehouse to the inside. This movement can simplify the management of the entire data. This still sounds a bit confusing: how does ELT simplify our data management? Can you turn your company's "old tech stack" into a "modern data stack" in one simple step?</p>
      <p>Imagine what our data management would be like without ELT. Some employees of a company want to analyze raw data, they use ETL tools to build a pipeline to clean and process the data into table X in the data warehouse, and then analyze table X in the data warehouse. In the process of analysis, they find that they have not extracted a column of data, or the data processing method was wrong, or the unit of the data was wrong. At this time, what should she/he do? The only way is to re-fetch the data from the data source, rebuild the ETL pipeline, and re-analyze the data. However, the data in the data source is often temporary. Usually, the enterprise only saves the data source for seven days or 30 days. That is saying, when she/he wants to re-extract the data, build the pipeline, and perform analysis, the original data may be already lost.</p>
      <p>ELT solves this problem in a brute-force way: store all the original data in the data warehouse. When the original data is stored in the data warehouse, there is no so-called data loss problem, and all historical records can be found. When we want to process data, we do not build any pipelines for processing but directly write SQL and use the computing power of the data warehouse itself for processing.</p>
      <p>It sounds really simple, and doesn't seem like something that only appeared in the 2020s. Why was ELT not used 20 years ago? Why has ELT been promoted only in recent years? I think there are several reasons for this.</p>
      <ul>
      <li>Firstly, the amount of data 20 years ago was relatively small, the data format was relatively simple compared to today, and the processing requirements were not high. Even if it was troublesome for everyone to develop applications, they could all be solved by stack engineers.</li>
      <li>Secondly, database systems from 20 years ago were simply too expensive to buy or not want to spend a lot of money on storing large amounts of data.</li>
      <li>Thirdly, the performance of the database system at that time was still relatively weak. Today, the cost of engineers is high, the applications are complex and diverse, and it is no longer possible to simply rely on engineers to solve problems; the emergence of cloud computing has greatly reduced the cost of data warehouses, making it possible to store massive amounts of historical data cheaply; The performance has been greatly improved, and it is no longer difficult to process large amounts of data in the data warehouse.</li>
      </ul>
      <p>If we think about it carefully, we will find that the essence of ELT is actually to turn the data warehouse into the center of data management, and to use the capabilities of the data warehouse to deal with data problems as much as possible. Now back to the topic of this article, this way of building the stack centered on the data warehouse is the way advocated by the modern data stack: the extensive usage of data warehouse capabilities. ELT is only one part of the modern data stack. In fact, the modern data stack also includes other aspects, including data visualization, metadata information management, data discovery, data sharing, etc. We will not cover these topics in this article.</p>
      <h3 tabindex="-1">What does the modern data stack have to do with cloud computing?</h3>
      <p>Modern data stacks and cloud computing are strongly correlated but not necessarily causal. In fact, cloud computing has indeed catalyzed the development of the modern data stack. The essence of the modern data stack is to simplify data management, while cloud computing is to reduce the cost of data management; hence the starting point is different. Nevertheless, without a low-cost solution, there would be no way to make data management easier.</p>
      <h3 tabindex="-1">Will there be a one-size-fits-all system on the modern data stack?</h3>
      <p>The one-size-fits-all system here refers to a system that can do everything: it can support operational processing, analytical processing, stream processing, data visualization, data sharing, data governance, etc. All tasks are done with one system. I think it's an ideal vision, but practically speaking, we haven't reached the stage where one size can fit all. The key question here is not whether it is technically feasible, but whether the product can be widely accepted.</p>
      <p>I think this question can be approached from both a product competition and user needs perspective.</p>
      <ul>
      <li>From the product competition, modern data software competition is more than &ldquo;hard&rdquo; level competitions, e.g., performance and price. Vendors have expanded the competition to more points in the &ldquo;soft&rdquo; level, for example, security and usability. It is often difficult for a one-size-fits-all system to compete with a product specializing in a certain task. A good example is Zoom. Although many office software, including Slack, Microsoft Teams, etc., can do video conferencing, Zoom has become a worthy winner in this field through its professionalism and ease of use.</li>
      <li>In terms of user needs, users rarely need to use all the features in a one-size-fits-all system. Small and medium-sized enterprises tend to use only two or three functions, and naturally, they may prefer the system that does the best job of these two or three functions. In a large company, users do use a variety of functions, but when trying to convince such users to replace their existing systems, they are often deterred by the high cost of migration.</li>
      </ul>
      <p>Of course, various viewpoints justify the existence of a one-size-fits-all system. The most common argument is that enterprise-level users often only want to solve problems but do not want to spend time choosing solutions. The existence of a one-size-fits-all system can significantly save their time in selecting products. Furthermore, such a system can provide a unified user experience that is more suitable for enterprise users.</p>
      <p>I don't think there is a standard answer to this question, and I believe that the future trend (at least in the next 3-5 years) is coexistence. Nevertheless, the one-size-fits-all system will still be a minority. Whether it is a Swiss Army knife, a fruit knife, or a utility knife, they all have their own living space, so that the Swiss Army Knife will not occupy the market of other kinds of knives.</p>
      <h3 tabindex="-1">Summary</h3>
      <p>The modern data stack serves users' data, and it is completely changing the situation in which users are bound with a few tech giants. The goal of modern data stacks is to greatly simplify the difficulty of managing data for users, allowing users to care more about the data itself, rather than the software. Of course, the modern data stack is still quickly evolving. In the next article, I will explain the current landscape of modern data stack, and what's next in the near future.</p>
      </div>

Everything You Wanted to Know but were Afraid to Ask about Modern Data Stack (part I)

How to Create a Data Governance Team? 3 Essential Steps

Getting Started with Data Observability

Building a Modern Data Team: From Analytics to AI

Data Governance - A Thought Leader's Perspective

Decoding the Modern Data Stack

<p>&ldquo;There must be something wrong with Excel. I can't get these numbers to make sense.&rdquo; For anyone who has had a similar experience of staring at a spreadsheet for far too long, we have news for you: Excel isn&rsquo;t the problem; your data is.</p>
      <p>Growing businesses encounter this issue as they scale their offerings and onboard new technologies. The more information you gather and store, the more complex your data infrastructure becomes. One manual error &mdash; like a coding typo or selecting the wrong data type for a new CRM field &mdash; can cause entire data sets to be inaccurate or incomplete. </p>
      <p>Mistakes will inevitably happen. That&rsquo;s why it&rsquo;s important to have data reliability tools and procedures in place that prevent bad data from moving further through your tech stack. This enables your organization to be confident that the data being used for analysis, insights, and decision-making is accurate and up-to-date. </p>
      <p>In this guide, you&rsquo;ll learn why data reliability is important, how it affects business outcomes, and the tools companies can employ to strengthen reliability and ensure consistency.</p>
      <h3>Why is data reliability important?</h3>
      <p>&ldquo;<a class="mds-text-primary" href="https://www.mozartdata.com/post/how-to-be-a-data-driven-company" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>Data-driven decisions</u></a>&rdquo; has become quite the buzzword, but don&rsquo;t let that distract from the fact that making business decisions based on accurate data is vitally important. Over the last decade, businesses have gained unprecedented access to quantitative and qualitative data about their operations, customers, and prospects. With all this information at our fingertips, it&rsquo;s become best practice for every team, from marketing to customer success, to make data-driven decisions. Why trust your gut or talk about hypotheticals if you can use hard facts and figures to guide your next move?</p>
      <p>In some ways, access to this information has helped level the playing field among small, medium, and more mature businesses. The data is out there if you&rsquo;re able to tap into it and know how to use it. But this also presents a problem. </p>
      <p>You may be collecting and storing a sizable amount of information within your <a class="mds-text-primary" href="https://www.mozartdata.com/post/data-stack" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>data stack</u></a>, but are you certain that these data sets &mdash; individually and when combined for analysis &mdash; are always complete, accurate, and up-to-date? If your answer is not a confident &ldquo;yes,&rdquo; then your business can&rsquo;t reliably make data-driven decisions. </p>
      <p>Data reliability is the foundation for confident decision-making and a successful data team.</p>
      <h3>What tools can improve data reliability?</h3>
      <p>Proactive, automated alerts and transform tests are two crucial data reliability tools companies utilize to proactively catch and debug issues, ensuring their data is accurate and reliable as it flows through the data pipeline. </p>
      <h3>Alerts</h3>
      <p>If organizations don't monitor their data, they're setting themselves up to miss important outcomes. This could be flaws in the data, like duplicate fields, or important events, such as revenue goals. Proactive, automated alerts can be raised the moment that specific conditions &mdash; which are predefined by the business &mdash; are met. </p>
      <p>Data reliability should be an ongoing consideration, and data alerts can help maintain diligence. There are two notable points in time when data monitoring should lead to an alert, if necessary: as the data is loaded into a <a class="mds-text-primary" href="https://www.mozartdata.com/post/data-warehouse" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>data warehouse</u></a> and post-transform.</p>
      <p>When data flows from its source, such as a marketing automation platform, into a data warehouse, it&rsquo;s still considered to be raw data. At this point, it&rsquo;s likely the data will contain flaws like duplicates, missing values, and incorrect formatting. A <a class="mds-text-primary" href="https://www.mozartdata.com/post/data-transformation" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>transform</u></a> can clean the data, but many of these issues can and should be fixed before getting to that point. </p>
      <p>Alerts that identify issues as data enters the warehouse allow you to take corrective action before defective data is used downstream. Corrective action might involve editing the data in the source platform, or letting an engineering team know that data is not being stored or collected correctly in a database, so that more involved troubleshooting can be prioritized. These types of actions help build confidence that the transformed data teams are working with is as complete and accurate as possible. </p>
      <p>Additionally, the results of a transform should be monitored to help businesses track specified outcomes. Alerts raised post-transform might highlight potential discrepancies, like unusually low or high returned values, or monitor discrete metrics for irregularities, long-term trends, milestones, etc. </p>
      <h4>Tests</h4>
      <p>Related to alerts are tests. In the context of maintaining data reliability, this would often look like a transform test. A transform test runs before the actual transform and can be used to flag specific outcomes. After the test runs and identifies an issue, it can either prevent the transform from running or send out an alert. </p>
      <p>Transform tests can be an important tool to help prevent data errors from reaching others, proactively identify data collection issues or unexpected positive outcomes, and maintain everyone&rsquo;s trust in your data. </p>
      <h3>How to make your data more reliable with Mozart Data</h3>
      <p>Mozart Data&rsquo;s modern data platform provides data alerts, transform tests and many other tools to help you ensure your data is accurate and reliable. Maintaining trustworthy data gives business leaders confidence in their team&rsquo;s reports and analyses, and enables them to make strategic, data-driven decisions. </p>
      <p>Data reliability is only possible if you have a clear view of your entire data pipeline though. This is achieved through what&rsquo;s known as data observability. <a class="mds-text-primary" href="https://www.mozartdata.com/post/what-is-data-observability" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>Read this post</u></a> for a deep dive into this topic and how data observability is linked to business growth. </p>

What is Data Reliability?

<p>Do you know the current status &mdash; quality, reliability, and uptime &mdash; of your data and data systems? Not last month or last week, but where they stand at this moment. As businesses grow, being able to confidently answer this question becomes more important. That&rsquo;s because data needs to be clean, accurate, and up-to-date to be considered reliable for analysis and decision-making. This confidence comes through what&rsquo;s known as data observability.</p>
      <p>In this guide, we&rsquo;ll review what data observability is, why it&rsquo;s a key driver of business growth, and what tools are needed to support the process.</p>
      <p>Advanced data teams automate the movement of raw data from data sources (like a CRM, advertising platform, and marketing automation tool) to a modern data platform, where data sets are <a class="mds-text-primary"  href="https://www.mozartdata.com/post/what-is-the-best-way-to-clean-up-a-large-data-set" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>cleaned</u></a>, organized, and centralized for storage in a data warehouse. It&rsquo;s from this point that <a class="mds-text-primary"  href="https://www.mozartdata.com/post/data-transformation" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>transformed data</u></a> can be synchronized with business intelligence tools for reporting, creating visualizations, conducting analyses, and generating insights. The flow of data from disparate origin sources to the destination, whether it&rsquo;s a data warehouse or business intelligence tool, is known as the data pipeline.</p>
      <p><strong>Data observability is the ability to see what&rsquo;s happening in your entire data pipeline, optimize performance, and monitor the health of your data.</strong> Without data observability, businesses are only viewing data at one point in time and won&rsquo;t have a real-time, transparent view of the workflow, nor insight into previous versions of saved data sets. Data observability, therefore, is critical for rapidly detecting, troubleshooting, and resolving problems, as well as ensuring that the data flowing into your <a class="mds-text-primary"  href="https://www.mozartdata.com/post/is-a-business-intelligence-tool-enough-for-your-business" target="_blank" rel="noopener noreferrer" data-hook="linkViewer"><u>business intelligence tool</u></a> is reliable for analysis, detecting patterns, and developing insights. </p>
      <h3>Why is data observability important? </h3>
      <p>Data observability is a key driver of business growth, as it ensures data quality and enables you to be agile and make sound, data-driven decisions in real time. </p>
      <p>Imagine, for example, a business called Intrepidly runs a B2B subscription-based productivity tool that&rsquo;s accessible to users on its desktop website, Android app, and iOS app. Intrepidly has already launched a new feature on the web, and next it&rsquo;ll be released in the iOS app. The updated version has been deployed and appears to be running fine. New subscriber profiles are being created in the CRM via the iOS point of sale. But Intrepidly&rsquo;s revenue operations team notices that key profile fields, such as subscription type and discount level, are being populated with NULL values. </p>
      <p>Where along the data pipeline is the issue occurring? Is it before or after the transform? Are these key pieces of missing information retrievable? Depending on what went wrong and where, Intrepidly could experience significant effects on revenue and the customer experience. </p>
      <p>Errors and omissions in the data pipeline can lead to downtime, which is when data is inaccurate or unavailable. In businesses that lack data observability, it could take multiple developers hours to answer these questions and rectify the issue. Troubleshooting data issues is not where you want your high-value employees spending their time. </p>
      <p>Data observability could also mean the difference between catching an error right after the version release versus moments before the start of an all-hands presentation when the data in the deck looks &ldquo;off." In other scenarios, data issues can delay product launches, internal technology implementation, reporting obligations, and decision-making. </p>
      <p>Maintaining data observability helps prevent &ldquo;drop everything to find the cause&rdquo; situations because it takes a proactive approach to monitoring the health of a business&rsquo;s data pipelines.</p>
      <h3>Data observability tools</h3>
      <p>There are specific tools, including data lineage and run history, that provide businesses with a transparent view of their data pipeline. Although these capabilities may not prevent human errors that result in situations like that mentioned above, businesses that have complete visibility into their data pipelines will be able to detect issues early and significantly accelerate the resolution process. </p>
      <p>Data lineage is one tool that&rsquo;s used to monitor the pipeline so businesses can see what information their transforms and reports are relying on and quickly trace errors back to the source. Mozart Data&rsquo;s modern data platform infers lineage so businesses don&rsquo;t have to set it up themselves.</p>
      <p>Run history makes troubleshooting failed transforms easier by showing whether they were run manually or automatically. To get to the root of broken transforms, you can use version history to see when changes were made, what those changes were, and who made them.</p>
      <p>Snapshots record historical information by capturing transform or source table information at a specific point in time and at a set frequency &mdash; typically daily. These are helpful for performing automated checks. For instance, when you need to track down the date that a particular field goes missing or automatically compare relative changes (e.g., is the amount we're billing this month close to what we billed last month).</p>
      <h3>How Mozart Data increases visibility into your data</h3>
      <p>Mozart Data&rsquo;s modern data platform provides data lineage, run history, version history, snapshots, and other data observability tools to give you visibility into your data systems and processes. This enables teams to streamline workflows, optimize their pipelines, and realize operational efficiency gains. It&rsquo;s also key to maintaining trustworthy data that org leaders feel confident using for analyses, insights, and business decisions. </p>

What is Data Observability?

The Modern Data Stack Ecosystem: Spring 2022 Edition

What is Data Onboarding? And 3 Ways It's Overwhelming Your Teams

What's the Difference Between Data Wrangling vs Data Cleansing vs Data Transformations

What is Reverse ETL? The Definitive Guide

Change Data Capture: What It Is and How to Use It

How to Build and Operate an Open-Source Data Stack on Kubernetes

<div>
        <p data-selectable-paragraph="">As data became more and more available to companies, data integration became one of
            the most crucial challenges for organizations. In the past decades, ETL (extract, transform, load) and later ELT
            (extract, load, transform) emerged as data integration methods that transfer data from a source to a data
            warehouse. Despite their similarities, the two differ in fundamental ways.</p>
        <p data-selectable-paragraph="">This blog aims to give an overview of ETL and ELT processes, describe the difference
            between them and outline which tools are now available for organizations. We do so by covering the following
            points:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>A brief history of data integration</strong></li>
            <li data-selectable-paragraph=""><strong>What is ETL?</strong></li>
            <li data-selectable-paragraph=""><strong>Why do we need ETL tools?</strong></li>
            <li data-selectable-paragraph=""><strong>ETL vs. ELT &mdash; What&rsquo;s the difference?</strong></li>
            <li data-selectable-paragraph=""><strong>When should you use ELT instead of ETL?</strong></li>
            <li data-selectable-paragraph=""><strong>What characteristics should your ETL/ELT tool have?</strong></li>
            <li data-selectable-paragraph=""><strong>What are the best ETL/ELT tools available right now?</strong></li>
        </ul>
    </div>
    <div>
        <h3><strong>A brief history of data integration</strong></h3>
        <p data-selectable-paragraph="">Today, businesses require data integration to centralize, access, and activate their
            data. As businesses start leveraging data from hundreds of different sources across continents and business
            departments and teams to make data-driven decisions, data integration has never been more critical. Although
            this heavily fractured landscape might seem new, it is not. In fact, the first challenges related to data
            integration date back to the 1960s when IBM and other companies pioneered the first Database Management Systems
            (DBMS), whose later advancements led to the sharing of data between computers. The process of integrating data
            with external machines soon became a challenge. This was challenging because organizations integrate with
            systems that they cannot control, and this often fails. So, it becomes really hard to create systems that
            don&rsquo;t break when you do data integration.</p>
        <p data-selectable-paragraph="">Data integration can be defined as the practice of consolidating data from separated
            source systems to transfer it to a data warehouse. Integration starts at the ingestion phase and includes steps
            like cleansing, ETL mapping, and transformation. ETL plays a central role in transforming raw data into usable
            data &mdash; enabling data teams to trust their data and make informed business decisions. Without data
            integration, accurate analytics is impossible to achieve as making decisions with incomplete information most
            likely leads to undesirable outcomes. And if these decisions involve hundreds of data sources, terabytes of
            data, and millions of dollars, integration becomes especially important.</p>
        <p data-selectable-paragraph="">ETL was the first standardized approach to dealing with data integration which
            appeared in the 1970s. It first emerged as a response to the adoption of relational databases and the increasing
            need to move information between them efficiently. Then, in the 1980s, data integration became even more
            critical with the rise of data warehouses. Data warehouses could integrate data from different sources. However,
            they typically required custom ETLs for each data source. This led to the emergence of a wide variety of ETL
            tools. And by the end of the 1990s, these solutions were affordable for both enterprises and mid-market
            businesses.</p>
        <p data-selectable-paragraph="">Finally, with the emergence of cloud computing in the 2000s, cloud data warehouses
            and data lakes brought forward a new development: ELT, which, as we will see later in this blog, allows
            businesses to load unlimited raw data into a cloud data warehouse.</p>
    </div>
    <div>
        <h3><strong>What is ETL?</strong></h3>
        <p data-selectable-paragraph="">As organizations collect data from multiple sources &mdash; such as CRM systems,
            Cloud applications, and more &mdash; the ETL process aims to gather data from separate sources and create a
            centralized database. Practically, ETL pipelines have the role of extracting raw data from its source,
            transforming it, and finally loading it in the warehouse &mdash; which is a centralized database.</p>
        <p data-selectable-paragraph="">Let&rsquo;s dive into each one of these steps:</p>
        <p data-selectable-paragraph=""><strong>Extract</strong></p>
        <p data-selectable-paragraph="">The first step is the process of data extraction. Here, data pipelines extract data
            from the different sources and gather it in a staging area. The staging area is vital because it allows the
            extraction of data from various sources at different times, and it prevents the simultaneous performance of
            extractions and transformations from happening &mdash; therefore averting the risk of overwhelming data sources.
        </p>
        <p data-selectable-paragraph=""><strong>Transform</strong></p>
        <p data-selectable-paragraph="">The second step is the data transformation process &mdash; in other words, making
            the raw data fit for analysis. This consists of&nbsp;<strong>improving the quality</strong>&nbsp;of the data by
            performing actions like sorting, joining, reformatting, filtering, merging, aggregation, and so on,
            and&nbsp;<strong>restructuring and reformatting</strong>&nbsp;the data to adapt it to the specific business need
            it will be used for.</p>
        <p data-selectable-paragraph=""><strong>Load</strong></p>
        <p data-selectable-paragraph="">The last step is loading the transformed data into the warehouse. The loading can be
            full &mdash; so the data is fully loaded into the warehouse at once &mdash; or incremental &mdash; where the
            data is loaded at different scheduled intervals using batch or stream loading. Batch loading consists of the
            scheduled extraction of different batches of data from a source system. In contrast, stream loading consists of
            data ingestion by pipelines as soon as they are made available by the source system.</p>
    </div>
    <div>
        <h3><strong>Why do we need ETL tools?</strong></h3>
        <p data-selectable-paragraph="">The ETL process of migrating transformed data into a warehouse enables organizations
            to break down silos and makes it easier for data users to turn the available information into business
            intelligence.</p>
        <p data-selectable-paragraph="">ETL is beneficial for organizations because it combines diverse data into a unified
            view. This enables data users to find and access information more efficiently, consequently improving
            productivity.</p>
        <p data-selectable-paragraph=""><strong>The advantages of ETL</strong></p>
        <p data-selectable-paragraph="">Implementing ETL into your data stack can bring many benefits, including:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>Speed:&nbsp;</strong>There is no waiting involved when using ETL tools,
                as they instantly load data into your data warehouse, where they are ready for transformation.</li>
            <li data-selectable-paragraph=""><strong>Flexibility:&nbsp;</strong>As transformations do not need to be defined
                at the beginning, it is pretty straightforward to integrate additional data sources into the ELT process.
            </li>
            <li data-selectable-paragraph=""><strong>Low initial cost:&nbsp;</strong>ETL automates the process of onboarding
                data, and as you don&rsquo;t need to define transformations, the initial cost is relatively low.</li>
            <li data-selectable-paragraph=""><strong>Low maintenance requirements:&nbsp;</strong>As the process is more
                straightforward and automated, it will require less maintenance. It is also easier to fix bugs since
                transformation only happens in the last step.</li>
            <li data-selectable-paragraph=""><strong>Scalability:&nbsp;</strong>As the amount of data you use increases, you
                can expand your storage in the cloud.</li>
        </ul>
    </div>
    <div>
        <h3><strong>Challenges to ETL and the rise of ELT</strong></h3>
        <p data-selectable-paragraph="">The main characteristic of ETL is the order in which the process is performed
            &mdash; specifically, the data goes through transformation before being loaded into the warehouse. This specific
            order is the reason behind some of the difficulties organizations face when using ETL. These are the following:
        </p>
        <ul>
            <li data-selectable-paragraph=""><strong>Dealing with heavy data loads:&nbsp;</strong>Volume and demand for data
                have been dramatically growing in the past few years. Unfortunately, many ETL processes are failing to scale
                to match these heavy loads appropriately. When dealing with large amounts of data, some common mistakes are
                loading irrelevant data, bottlenecks due to insufficient CPU and serial data processing instead of parallel
                data processing.</li>
            <li data-selectable-paragraph=""><strong>Maintenance of the pipelines:&nbsp;</strong>In ETL, pipelines extract
                and transform data all at once, breaking as soon as schema changes in the data source. For instance, when a
                field is removed. This compromises the code needed to transform the data, and it has to be re-written by
                data engineers.</li>
            <li data-selectable-paragraph=""><strong>Data security risks:&nbsp;</strong>When loading large amounts of raw
                data into your storage, data security can become a concern. To deal with this challenge, managing user
                access to the raw data stored in the data warehouse is crucial.</li>
            <li data-selectable-paragraph=""><strong>Low compliance with regulations:&nbsp;</strong>Since data is stored
                with minimal processing, there may be some additional steps to ensure that you comply with the main security
                protocols, including GDPR and HIPAA.</li>
        </ul>
    </div>
    <div>
        <h3><strong>ETL vs. ELT &mdash; What&rsquo;s the difference?</strong></h3>
        <p data-selectable-paragraph="">Traditional ETL is characterized by the performance of the transformation process
            before the loading into the warehouse. The reason for this is that, back in the day when ETL was created,
            storage, computation, and bandwidth were costly. Hence the need to reduce the volume of data before it gets to
            the data warehouse.</p>
        <p data-selectable-paragraph="">As storage and computation cost dramatically decreased, cloud storage grew
            exponentially. This led to the entrance into the market of data warehouses like Amazon Redshift and Google
            BigQuery. Cloud data warehouses are infinitely more scalable compared to traditional warehouses, allowing
            organizations to fit enormous amounts of data.</p>
        <p data-selectable-paragraph="">The emergence of cloud data warehouses called for a radical change in ETL processes
            too. If ETL processes were fine for traditional warehouses, their performance did not improve in a cloud
            environment. The reason for this is that the scalability issues were inherent to the process of moving data into
            the warehouse. At the same time, due to the arrival of cloud storage, organizations started loading large
            volumes of untransformed data into the warehouse without having to worry about its cost. From this moment on, it
            has been possible to extract data from the source systems and load it into the cloud warehouse without needing
            to go through the transformation process. The Extract-Load-Transform (ELT) process replaced traditional ETL.</p>
        <p data-selectable-paragraph="">Let&rsquo;s take a side-by-side look at the differences between ETL and ELT, their
            pros and cons, and how they can work together to provide a holistic data integration solution for businesses.
            Here are the ten critical differences between ETL and ELT.</p>
        <figure>
            <div tabindex="0" role="button">
                <div><img role="presentation" src="https://miro.medium.com/max/1400/1*nY2lYvuPTC_ywQOs0nny7Q.png" alt="" width="700" height="628"></div>
            </div>
            <figcaption data-selectable-paragraph="">Source: Benedetta Cittadin</figcaption>
        </figure>
    </div>
    <div>
        <h3><strong>What characteristics should your ETL/ELT tools have?</strong></h3>
        <p data-selectable-paragraph="">If you&rsquo;ve made it so far, it probably means you are interested in integrating
            ETL or ELT in your data stack. But what characteristics should these tools have to meet your standards?
            We&rsquo;ve gathered some general, critical requirements that any of these tools should have:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>Built-in integrations</strong>: You will need your ETL/ELT tools to
                interact with many different programs and APIs to use your data.</li>
            <li data-selectable-paragraph=""><strong>Ease of use:&nbsp;</strong>The tool should have an intuitive interface
                that assists you in simplifying as many functions as possible as well as making it possible to train
                non-developers to use it.</li>
            <li data-selectable-paragraph=""><strong>Automation:&nbsp;</strong>The more tasks you can automate, the faster
                and easier it will be to get your work done. And if you adopt a tool, you don&rsquo;t want to have to
                program SQL workarounds every time you need to get something done. This is why your ETL/ELT tool should have
                automation in place.</li>
        </ul>
        <h3><strong>What are the best ETL tools available right now?</strong></h3>
        <p data-selectable-paragraph="">With companies collecting increasingly high amounts of data daily, there is a
            growing need to distribute data from the source to the data warehouse. This is why we have recently seen a
            blowing up of ETL/ELT tools in the market. In this section, I&rsquo;ll briefly outline the different ETL tools
            that are currently being used by organizations.</p>
        <p data-selectable-paragraph="">ETL can be grouped into different categories based on their infrastructure.
            I&rsquo;ll briefly go through three of these groups.</p>
        <p data-selectable-paragraph=""><strong>Enterprise software ETL</strong></p>
        <p data-selectable-paragraph="">Developed by commercial organizations, these solutions are often the most mature
            ones in the marketplace. These tools include graphical user interfaces (GUIs) for architecting ETL pipelines,
            support for relational and non-relational databases, and extensive documentation and user groups.</p>
        <p data-selectable-paragraph="">Although they offer more functionality, enterprise ETL tools are usually more
            expensive and require more employee training because of their complexity.</p>
        <p data-selectable-paragraph="">Think of tools like IBM DataStage and Oracle Data Integrator.</p>
        <p data-selectable-paragraph=""><strong>Open-source</strong></p>
        <p data-selectable-paragraph="">Many free ETL tools offer GUIs for data-sharing processes and monitoring the flow of
            information. An advantage of open-source tools is that organizations can access the source code and extend its
            capabilities.</p>
        <p data-selectable-paragraph="">Think of tools like Airbyte and Talend.</p>
        <p data-selectable-paragraph=""><strong>Custom ETL tools</strong></p>
        <p data-selectable-paragraph="">Cloud service providers (CSPs) now offer ETL tools built on their infrastructure. An
            advantage of this type of solution is that if the organization also stores its data using the same CSP, the
            pipeline is optimized because all processes occur within a single infrastructure. A disadvantage, however, is
            that cloud-based ETL tools only work within the CSP environment and don&rsquo;t support data stored in other
            locations.</p>
        <p data-selectable-paragraph="">Think of tools like Azure Data Factory and Google Cloud Dataflow.</p>
        <h3><strong>Conclusion</strong></h3>
        <p data-selectable-paragraph="">ETL and ELT can serve your data integration purpose in different ways. So, to choose
            the solution that&rsquo;s right for you, you need to consider factors like</p>
        <ul>
            <li data-selectable-paragraph="">The data you have</li>
            <li data-selectable-paragraph="">The type of storage you use</li>
            <li data-selectable-paragraph="">The (long-term) needs of your business</li>
        </ul>
    </div>

Survival of the Fittest: ETL vs. ELT



        <h3>&zwj;<strong>What is data product management</strong>?</h3>
        <p>In our experience at Secoda working with many data teams, we've seen most data teams do not have the tools they need
            to succeed. For growing organizations, the data function is usually an afterthought. The first data hire is brought
            on before raising a Series A and is expected to manage the workload that comes afterward with little to no support.
            This means that this first data hire usually spends all of their time answering other people&rsquo;s questions.
            These teams tend to be overworked and are likely to remain a reactive function for the foreseeable future.</p>
        <p>This is because most data teams are understaffed and misunderstood. Instead of reacting to employee data requests,
            working with data requires the data team to treat employees as their customers and their data as the product.</p>
        <p>An organization's data product is its collection of all of its data knowledge, along with the resources used for
            accessing, creating, and analyzing that data. Making the data product more self comprehensive, accessible,
            trustworthy, accurate, and secure should be the goal of the data team and the data team.&nbsp;</p>
        <p>Additionally, the data PM should help direct resources towards the highest leverage areas of the business. When
            products are built with everyone in mind, they usually don&rsquo;t accomplish what they need to for any specific
            subset of users. Data PMs should think about their data products the same way. Building data products for specific
            levels of technical capabilities and specific&nbsp;<em>persona&rsquo;s&nbsp;</em>of data consumers should be the
            goal of the data PM.&nbsp;</p>
        <h3>What makes a good data product manager</h3>
        <p>A good data product manager understands the company ecosystem of people and data. It is crucial to understand the
            different segments of the organization and document the requirements and knowledge from both data producers and
            consumers to generate and manage clean, reliable, and meaningful data. A good data product manager will apply
            product management practices to guide the roadmap for the data team by displaying some of the competencies below:
        </p>
        <ul role="list">
            <li>Conducting employees interviews to understand how they want to use data</li>
            <li>Running data-design sprints (How do we get a minimum version of this product to the employees in the shortest
                amount of time)&nbsp;</li>
            <li>Data product prioritization and road map planning.</li>
            <li>Communicating with the product team and updating the data team on changes that may affect their work.</li>
            <li>Performing vendor assessments.</li>
            <li>Translating business-to-data requirements, and vice versa.</li>
            <li>Defining and tracking key metrics for each stakeholder.</li>
            <li>Sharing knowledge and tracking data product usage metrics.</li>
        </ul>
        <p>We think about data PMs as the bridge between the data team and the rest of the organization. It takes time, tooling,
            and measurement to make sure that all stakeholders can access the right data, but when it&rsquo;s achieved,
            organizations can increase enterprise value by up to 5%.</p>
        <h3>Role requirements</h3>
        <ul role="list">
            <li><strong>Create Data Requirements</strong>&nbsp;&mdash; Data PMs should create clear data product requirements
                that satisfy the preferences of the end-users and data developers they work with. Even though data product
                requirements are still being defined, we like the development, release, and measurement model for building
                useful data products and determining how they are used.&nbsp;</li>
        </ul>
        <ul role="list">
            <li><strong>Employee feedback</strong>&nbsp;&mdash; Data PMs should bring together basic employee insights through
                interviews, understanding department OKR&rsquo;s, taking inbound questions, etc. Based on an explicit set of
                requests from the employees for whom the data product is being developed, they meet their baseline expectations
                (e.g. the employee says a part of the data is broken).</li>
        </ul>
        <ul role="list">
            <li><strong>Data insights</strong>- Data PMs should have a data proficiency where they can look at the data related
                to their data product (very meta) and use that data to make decisions about which products are performing well
                and which should be depreciated. They should have a working knowledge of data languages like SQL and know how to
                use popular tools like dbt, Looker, Metabase, Snowflake, Fivetran, and so on.</li>
        </ul>
        <ul role="list">
            <li><strong>Run Meetings</strong>&nbsp;&mdash; Data PMs should also run effective meetings with the data team and
                advocate for the team's resources to leadership. That includes setting the agenda and tracking action items as
                well as making sure the data team is well staffed and able to contribute to the data product the way that
                employees require.&nbsp;</li>
        </ul>
        <ul role="list">
            <li><strong>Vendor evaluation</strong>&nbsp;&mdash; The data PM should work closely with the engineering, sales, and
                operations teams to understand which products can give their team the best visibility and transparency into the
                data. By evaluating the right tools with other department leads, the data PM can help accelerate their team's
                data literacy and data maturity.&nbsp;</li>
        </ul>
        <h3>Data analytics</h3>
        <p>Think about how difficult it would be for a product manager to manage features without critical information about how
            people are using the product. Today, many data teams are operating in this same way. It&rsquo;s very unclear which
            tables, dashboards, metrics, and charts are used frequently and which should be depreciated. PMs can empower
            themselves and their teams with the information they need to make decisions to improve data products by adopting a
            tool that enables them to measure the most used data knowledge.&nbsp;</p>
        <p>It should be possible for data PMs to see how often teams use data, whether they are using the right data, and
            whether they aren't using any data. By understanding how people use analytics', data PMs can focus on the most
            important tasks that drive data literacy across their company.&nbsp;</p>
        <h3>The future of the Data PM</h3>
        <p>Over the next three to five years, we see the product-management role continuing to evolve toward a deeper focus on
            data as a product. Data product managers of the future will work with the data team to produce high-quality data
            products that can help drive self-service across the organization.&nbsp;</p>
        <p>To communicate data with the rest of the organization, data PM's might choose to implement tools like dbt, Looker,
            and Secoda. As we continue to help data PMs for the organization share their knowledge, we hope to be able to offer
            a greater depth of assistance and support to all employees trying to use data to make decisions. Data interactions
            are becoming increasingly complex, and Secoda hopes to simplify the interaction with data for every employee.</p>

Data Product Manager: The role and best practices for beginners

<p>From personal experience, I have always found it interesting to learn how to create an organized catalog of data.
        However, this interest was transformed into a passion when I began to realize the amount of time and effort it could
        save me within my job responsibilities. Creating a data catalog can greatly help you with organizing the data they
        collect, therefore making it easier to find what you need when you need it.</p>
    <p>By the end of this post, we should&rsquo;ve shed some light on the following questions:</p>
    <p><strong>Why all data teams need data discovery tools?</strong></p>
    <p><strong>What kind of data catalogue should teams look for?</strong></p>
    <p><strong>How to get buy in for a data catalogue?</strong></p>
    <p><strong>Which tools are here to stay?</strong></p>
    <h3>Why do you need a data catalog?</h3>
    <p>Simple data cataloging starts with a great organization. A data catalog is a collection of metadata and documentation
        that helps make sense of the data sprawl that exists in most growing companies. Getting together and starting to use
        a data catalog is a simple process, but starting to get adoption and having the&nbsp;<a
            href="https://www.secoda.co/blog/create-a-data-dictionary" style="color: #f66640;">catalog</a>&nbsp;exist as
        part of your workflow is a
        little bit more difficult.&nbsp;</p>
    <p>Even though it may seem like an easy task, getting different stakeholders to change their routines and start using a
        new tool can be very challenging. An example of the data catalog problems shared by one of the delivery companies we
        spoke with. At this company, it was difficult to get aligned on which tables were commonly used, joined, how they
        were used together and what columns meant. Similarly, it&rsquo;s difficult to monitor the number of data assets that
        exist across different departments, especially when the number of resources grows at a faster rate than people. Why
        is this the case?&nbsp;</p>
    <p>Data is becoming more decentralized through concepts like the data mesh. As more teams outside of the data function
        start to use data in their day-to-day, different tables, dashboards and definitions are being created at an almost
        exponential rate. Data catalogs are important because they help you organize your data whether you are working with
        structured or unstructured data. They help you identify what kind of data you have, how it is related to each other
        and what the best means to store it is so that you can quickly find it when needed.</p>
    <p>It happens to all data-driven companies as they grow both their data and their people. While it sounds like an easy
        problem, which might require a meeting, converging the business and data to eliminate confusion can be a very
        challenging task. That's why a data catalog can be one of the most valuable tools that a data team can use to
        start&nbsp;<a href="https://www.secoda.co/blog/net-promoter-score-for-data-teams" style="color: #f66640;">measuring
            their impact.&nbsp;</a>
    </p>
    <p>Below are the steps that teams need to take when creating a data catalog:</p>
    <h3>1. Gather sources from across the organization&nbsp;</h3>
    <p>The first step data teams need to take is to collect the different resources that are scattered across different
        tools in the organization. This may require multiple meetings and stakeholders to come together and figure out which
        resources need to be in the catalog. This collection could be done in a spreadsheet with an ongoing list of all
        resources and how they connect.</p>
    <p>It is common for teams to work with a tool that connects to multiple sources and collects the metadata from those
        sources into a central location. Using&nbsp;<a
            href="https://www.secoda.co/blog/keep-all-your-data-knowledge-in-one-place" style="color: #f66640;">data
            catalogs</a>&nbsp;makes this
        extremely easy and is a vital component of any data-driven organization. With the volume of data collected by
        businesses increasing, properly organizing it is becoming increasingly important. Priority should always be given to
        ensuring that the data is useful, but we should also take into account gains to be reaped through improved
        efficiency and collaboration.</p>
    <h3>2. Give each resource an owner</h3>
    <p>After data teams have identified all the resources from across the company that they would like to include in their
        data catalog, we recommend assigning ownership to each resource. Teams that we&rsquo;ve worked within the past have
        assigned ownership based on the source, schema or even domain. Teams that start assigning ownership should look for
        people who are familiar with the data knowledge they are responsible for managing and are willing to help others who
        want to learn how to use it.&nbsp;</p>
    <p>Ownership doesn&rsquo;t have to reside in the data team. Many products, operations and growth leaders will likely
        want to own certain pieces of&nbsp;<a
            href="https://www.secoda.co/blog/the-ultimate-guide-to-increasing-your-teams-data-literacy"
            style="color: #f66640;">company data
            knowledge</a>. By empowering these leaders as data stewards, the data team can start automating the repetitive
        questions they get from employees about how to use company data.&nbsp;</p>
    <h3>3. Get support and sign off</h3>
    <p>Once these meetings conclude and owners are on the same page, have the owners sign off on their responsibilities. The
        owners should be in alignment with the documentation and feel like the data team worked&nbsp;<a
            href="https://www.secoda.co/blog/how-to-champion-the-shift-towards-a-collaborative-data-culture"
            style="color: #f66640;">collaboratively</a>&nbsp;with
        them to come to this ownership structure. One effective strategy is to involve the leadership team in the exercise
        early to make sure that their team leads are signing off on the owners of data. This way, leadership can see how
        widespread the understanding of data is across the company. If the team leadership team sees&nbsp;<a
            href="https://www.secoda.co/blog/rethinking-the-data-catalog" style="color: #f66640;">the value of a data
            catalog</a>, this can move at
        a much faster pace.</p>
    <h3>4. Integrate the catalog base into your workflow</h3>
    <p>After data teams have received support for their data documentation process, they should look for ways to integrate
        this tool into their workflow. This step is critical for maintenance and upkeep. Without a tool that allows
        teammates to receive notifications on Slack, it will likely be forgotten. By creating a process around the data
        catalog, teams can ensure that it is not left behind as the team grows</p>
    <p>One additional piece of context that teams should look for is the usage and adoption of the data catalog across the
        company. If a tool can display the way that employees are searching and using the knowledge base, it can be a great
        source of information for the data team as they continue to iterate on the information in this central repository.
        By ensuring that there are triggers and notifications that prompt users to enter the application, teams can make
        sure that they are taking advantage of the&nbsp;<a
            href="https://www.secoda.co/blog/data-product-manager-the-role-and-best-practices-for-beginners"
            style="color: #f66640;">data catalog
            and improving it like a product.&nbsp;</a></p>
    <h3>5. Upkeep the data catalog</h3>
    <p>Although the documentation should be stable,&nbsp;<a
            href="https://www.secoda.co/blog/analytics-teams-need-more-than-traditional-documentation"
            style="color: #f66640;">it may need to change
            over time</a>. One instance that might require documentation to change is when a new revenue stream is
        introduced or when the pricing of an existing revenue line changes. These changes traditionally come from the
        business team and might require the data team to implement the changes into the data catalog.</p>
    <p>Documenting your data should be essential to any data-driven business process. It allows data practitioners to see
        how their work is used, by whom, and for what purpose. Just the act of documenting the data will help you understand
        what you have, where it is, and how it can be used. It may seem like an intimidating task at first, but I encourage
        you to start small, with a few datasets here and there, until the process becomes second nature. And once it does,
        you'll never look back!</p>
    <h3>Final thoughts</h3>
    <p>To sum it up, data catalogs are valuable for any organization, especially those growing quickly. No matter the size,
        data documentation is important in the pursuit of data operations. Having a tool that can track, document, store and
        provide metadata insights for systems beyond just your current data stack will help you identify your blind spots in
        your current architecture and integration points moving forward.&nbsp;</p>
    <p>Teams that invest the time to get alignment using a data catalog can see major benefits in the long term as they make
        faster decisions as a team. Creating a data catalog is not a small undertaking,&nbsp;<a
            href="https://www.secoda.co/blog/create-a-data-dictionary" style="color: #f66640;">that's why we make it easier
            with Secoda</a>. The
        process requires patience and alignment with leadership.&nbsp;</p>

How to Create A Data Catalog, A Step-by-Step Guide

Things I Wish I Knew When I Was Building a Data Team

<p>Let&rsquo;s not mince words. Product led growth (PLG) isn&rsquo;t something that happens overnight. It has to infuse company culture and involves commitment from every team - not just the go-to-market teams on the front lines.&nbsp;</p>
      <p>PLG introduces a lot of new data about how customers are using the product that GTM teams can leverage, but this data presents new challenges to data teams who are tasked with figuring out how to model data in a scalable and reliable way. It doesn&rsquo;t help that data leaders are being constantly bombarded with one-off analytics requests.&nbsp;</p>
      
      <p>Although the situation sounds dire, enabling PLG data will pay dividends. Revenue teams at product led companies need to leverage product data in order to run the most effective, scalable go-to-market (GTM) motions.&nbsp;</p>
      <ul role="list">
      <li>Sales teams can use product data to identify the best customers to connect with so that they are spending time on the customers most likely to buy.&nbsp;</li>
      <li>Marketing teams can use product data to refine nurture campaigns to provide a more personalized, relevant brand interaction with users.&nbsp;</li>
      <li>Customer Success teams can use product data to identify superstar customers or proactively reach out to customers who are at risk to churn.&nbsp;</li>
      </ul>
      
      <p>We&rsquo;ve spoken to hundreds of companies implementing PLG or looking to transition to PLG, and throughout all these conversations, we&rsquo;ve identified 5 common obstacles data teams need to address when it comes to enabling GTM teams with product data. In this blog post, we&rsquo;ll elaborate on these 5 challenges and provide suggestions on how to address them.</p>
      
      <h3 id="toc-0">The 5 common challenges for PLG data teams</h3>
      <ul role="list">
      <li>Data is generated by multiple teams across your company</li>
      <li>Data isn't owned by the stakeholders who use it (i.e. your revenue team)</li>
      <li>Data is surfaced in platforms that teams don&rsquo;t use on a daily basis</li>
      <li>Data that is surfaced in downstream platforms is overwhelming in scope</li>
      <li>Data that is converted into PQLs or Scores become black boxes that are rarely actionable</li>
      </ul>
      
      <h3 id="toc-1">Data is generated by multiple teams across your company</h3>
      <p>Every team across your company is generating data about customers on a daily basis.&nbsp;</p>
      <ul role="list">
      <li>Sales teams are manually inputting data about opportunities in their CRMs, tracking activity, or automatically enriching accounts and contacts with tools like Clearbit and ZoomInfo.&nbsp;</li>
      <li>Marketing teams are automatically adding leads into CRMs and tracking marketing activity.&nbsp;</li>
      <li>Product teams are tracking product usage data in data warehouses.&nbsp;</li>
      <li>Customer Success teams are working with customers through tools like Intercom.&nbsp;</li>
      </ul>
      <p>The data inputs are endless and are not uniformly collected, which means that everything lives in different data silos. It&rsquo;s difficult to stitch all this data together into a single view for all teams to consume.</p>
      <p>Further, when different teams are generating data, they are generating it for their own use cases rather than thinking about how they can scale their data sets across the organization. We often observe this when sales teams get access to Segment data, which is often generated by Marketing or Product. Sales teams don&rsquo;t understand what various button clicks mean, making the barrier to adopting Segment data in sales processes that much higher. This is a challenge we navigated over at Correlated as well! For example, we track the various actions that users automate in our product and build alerts around &ldquo;exceptions&rdquo;. However, &ldquo;exceptions&rdquo; are labeled differently for different actions, so we have to catch all the labels correctly in order to get a complete alert for all &ldquo;exceptions&rdquo;. Obviously, if we threw this raw data at our sales team, they would have absolutely no idea how to use it.</p>
      <h4>Suggestion: Leverage a data warehouse as your source of truth and invest in enabling technologies in your data stack</h4>
      <p>Ultimately, the customers who are most successful in implementing a PLG GTM motion with us are the ones who have a data warehouse. This is because a data warehouse can act as your source of truth for all your customer and user data. It is certainly a challenge to sync all of your data sources with your data warehouse, but it is not an insurmountable one. ETL tools like&nbsp;<a class="mds-text-primary"  href="https://www.fivetran.com/">Fivetran</a>,&nbsp;<a class="mds-text-primary"  href="https://www.stitchdata.com/" target="_blank" rel="noopener">Stitch</a>, and&nbsp;<a class="mds-text-primary"  href="https://airbyte.com/">Airbyte</a>&nbsp;make it possible to do this using a single platform.&nbsp;</p>
      <p>Once all the data is loaded into your data warehouse, you can leverage all the features and functionalities of data warehouses to shape and model your data. We highly recommend&nbsp;<a class="mds-text-primary"  href="https://www.getdbt.com/">dbt</a>&nbsp;as a great option for those looking to scale their data modeling across their organization.</p>
      <p>Leveraging a data warehouse is not without its limitations. It requires deep institutional knowledge and expertise from data teams in order to unblock key business challenges, and of course this becomes a cost center. Investing in tools that allow teams to self-serve their way into pre-built models is a great option, and there are many tools in the market today that can help. For example, Reverse ETL solutions like&nbsp;<a class="mds-text-primary"  href="https://www.getcensus.com/">Census</a>&nbsp;and&nbsp;<a class="mds-text-primary"  href="https://hightouch.io/">Hightouch</a>&nbsp;connect into the models your data teams have built and make it easy for downstream applications to consume that data. This makes it possible for your data teams to scale their efforts to support multiple use cases. BI solutions like&nbsp;<a class="mds-text-primary"  href="https://www.tableau.com/">Tableau</a>,&nbsp;<a class="mds-text-primary"  href="https://www.looker.com/">Looker</a>, and more recently,&nbsp;<a class="mds-text-primary"  href="https://hex.tech/">Hex</a>, provide business users with access to pre-modeled data. Tools like&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/">Correlated</a>&nbsp;sit on top of your data warehouse to enable business users to build automated playbooks.&nbsp;</p>
      
      <h3 id="toc-2">Data isn't owned by the stakeholders who use it</h3>
      <p>One of the biggest challenges with implementing a data warehouse as described above is that you have to figure out how to enable the various personas who need that data. If all you do is dump data into a data warehouse, none of the teams who rely on that data to do their jobs will be able to access it. If left to fester, data teams and GTM teams can ultimately reach a stalemate where data teams have implemented processes that prevent GTM teams from getting what they want. Alternatively, GTM teams could be requesting data team resources all day every day, completely inundating data teams with requests. We&rsquo;ve seen both situations at companies, and it is ugly.</p>
      <h4>Suggestion: Focus on driving alignment between data teams and business goals. Make cross-team collaboration part of every day processes.</h4>
      <p>Companies should view their data teams as a pooled resource that supports the entire company. As a result, data team resources should be allocated and scheduled similarly to how engineering teams schedule and allocate time to work on product roadmap. It&rsquo;s important, however, to not treat the data team as a separate silo where requests are sent (and often added to the backlog). Instead, scheduling and allocation of resources should happen hand in hand with the business units who will benefit from data team resources. This is the model that we&rsquo;ve seen work best in companies with successful PLG GTM motions that leverage data.</p>
      <p>One of the reasons why it&rsquo;s important to intertwine business use cases with data team projects is that we&rsquo;ve rarely seen companies successfully implement a product-led data strategy without doing so. If you send the data team off to &ldquo;solve our usage tracking problems,&rdquo; they&rsquo;ll end up coming up with a solution that might not work with the downstream tools your business teams use, or they&rsquo;ll try to find a solution that works for everything, essentially boiling the ocean on all possibilities. This ends up taking upwards of a year to implement, leaving GTM teams flying blind in the meantime.&nbsp;</p>
      <p>Instead, focus on use cases that you as a business want to accomplish and build out the infrastructure, models, and views that can support those use cases. Leveraging data is a core strategic competency for SaaS businesses, so it&rsquo;s not realistic to expect to never have to touch your existing data models again. This is why the tools you choose to use are so important - although you will have to adjust your data models from time to time, it doesn&rsquo;t have to be painful every single time.</p>
      
      <h3 id="toc-3">Data is surfaced in platforms that revenue teams don&rsquo;t use on a daily basis</h3>
      <p>Typically when companies start scratching the surface on leveraging product data to run a product led growth motion, they&rsquo;ll take an initial approach of driving &ldquo;visibility&rdquo;. This is essentially Looker, Mode, Tableau, or some other business intelligence (BI) dashboard that shows things like how an account is using the product or a list of &ldquo;product qualified leads&rdquo;. This is a worthwhile exercise because it allows your data team to build out the data models that your GTM teams can benefit from.&nbsp;</p>
      <p>However, time and time again, we&rsquo;ve heard feedback that GTM teams simply do not look at dashboards. In the rare cases that they do, they find that dashboards are rarely actionable. For example, let&rsquo;s say that you have a dashboard that lists out PQLs. An AE can&rsquo;t just take an email off that dashboard and email them. That&rsquo;s because (1) they have to figure out if they&rsquo;ve already emailed the user before (check Salesforce/Hubspot), (2) they have to figure out what to say to the user (check another dashboard on product usage), (3) they aren&rsquo;t sure why the user is even a PQL in the first place (Slack data team to ask how a PQL is triggered).</p>
      <h4>Solution: Pipe data into downstream applications.</h4>
      <p>Rather than expecting GTM teams to look at dashboards and analyze the data, send data directly to the downstream applications that they live in on a daily basis. We&rsquo;ll talk about some of the downsides to this approach in the next &ldquo;problem,&rdquo; but some approaches we&rsquo;ve seen companies take is to build very specific views in Salesforce that contain all the information an AE (account executive) or CSM (customer support manager) needs to reach out in an intelligent way. For example, on the Account view, they might include things like: seats sold, seats utilized, number of active users. This allows the revenue team to stay in the tools they use, but still see the data they need to reach out to customers appropriately.</p>
      
      <h3 id="toc-4">Data that is surfaced in revenue team platforms is overwhelming in scope</h3>
      <p>This problem is linked in many ways to the prior problem - when companies discover that their teams are not using dashboards, their initial response is to pipe&nbsp;<em>data</em>&nbsp;into downstream applications. Now, I highlight&nbsp;<em>data</em>&nbsp;here because data is messy, there is a lot of it, and it requires analysis. We&rsquo;ve seen companies pipe so much data into Salesforce that they literally run out of custom fields. As an AE or CSM looking at Salesforce, those views simply become information overload. You end up giving so much information to your GTM teams that they literally cannot do anything with it.</p>
      <h4>Solution: Strive to pipe actionable insights over data.</h4>
      <p>A better approach is to think about the end goal you want to achieve by sending data to downstream applications. Are you trying to surface a lead? Are you trying to suggest a next-best-action? That should determine what data you send and how you send it.&nbsp;</p>
      <p>For example, let&rsquo;s dig into the &ldquo;seats sold&rdquo; and &ldquo;seats utilized&rdquo; example. You could pipe that information into Salesforce and surface that as a custom field on the Account object. However, wouldn&rsquo;t it be better if you created a lead in Salesforce, assigned it to an AE, and added a note that said - this customer is hitting 80% license utilization, add them to the &ldquo;Upsell&rdquo; Outreach sequence? You are sending insights that are immediately actionable.&nbsp;</p>
      <p>Some patterns we&rsquo;ve seen that we really like are: auto-creating Salesforce Leads, auto-creating Salesforce Opportunities, adding users to Outreach or Salesloft Sequences / Cadences, adding users to lists in Hubspot. The opportunities are endless. By thinking about insights and expected action over just data, you can ensure that all your efforts to build out a product-led data strategy are actually adopted by the GTM teams who have to put them into practice.</p>
      
      <h3 id="toc-5">Data that is converted into PQL Scores becomes a black box</h3>
      <p>Final topic: PQL Scoring. People really love PQL Scores recently, but it&rsquo;s something that you should definitely approach with care. In some ways, PQL Scores are the ultimate effort to convert data into insights. E.g. I crunched all the data for you, and this customer is great to reach out to. The problem is that GTM teams need to know how to reach out, which means they have to understand the &ldquo;<em>why&rdquo;</em>&nbsp;behind PQL Scores. If they don&rsquo;t understand the why, they are not going to adopt the score. You&rsquo;ll spend 6-12 months building out a score that ultimately no one uses.&nbsp;</p>
      <p>This is not to say scores are not useful - they certainly are, particularly when you&rsquo;re trying to get an overall directional feel of if someone is worth reaching out to. However, they should only be used as one piece of the puzzle and should not be viewed as a solution to PLG.</p>
      <h4>Solution: Think about the playbooks you want to enable after receiving an insight</h4>
      <p>There are a lot of vendors out there claiming to surface leads &ldquo;magically&rdquo; and with &ldquo;machine learning&rdquo;. We ourselves run machine learning models to help our customers identify which Signals might be interesting to go after, so it&rsquo;s a viable strategy and can surface some interesting insights. However, again, insights are more valuable if they drive a downstream action.&nbsp;</p>
      <p>If you&rsquo;re thinking of using a score, we&rsquo;d suggest tying it to a playbook you want to execute. For example, create an &ldquo;Upsell&rdquo; score, and align it with an &ldquo;Upsell&rdquo; playbook. Create an &ldquo;Onboarding Self-Serve&rdquo; score, and align it with an appropriate playbook. If you simply use one score, your results won&rsquo;t be as good as focusing on personalized, actionable playbooks that drive real results.&nbsp;</p>
      
      <h3 id="toc-6">Time to dive in!</h3>
      <p>We&rsquo;ve explored a lot of surface area in this blog post, from data warehouses to delivering that data in actionable ways to GTM teams. This may sound like a lot, but we&rsquo;ve seen a lot of companies get to success by taking things one step at a time. What we&rsquo;ve seen work best is actually a more vertical approach. Rather than trying to solve problem 1, then moving onto problem 2, etc etc, pick one important business challenge that aligns with a top-line goal and solve all the problems vertically.</p>
      <p>For example, let&rsquo;s say that your business wants to drive self-serve conversion. Next, identify the &ldquo;actions&rdquo; you want your GTM teams to take, the &ldquo;data&rdquo; you need to make that possible, and the &ldquo;tools&rdquo; that you&rsquo;ll need to tie everything together. Finally, go forth and do it! By aligning data teams with actionable business outcomes, you&rsquo;ll be able to get the backing and support you need across teams to get things done.&nbsp;</p>
      <p>If you'd like to learn how Correlated can assist your product-led journey,&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/get-started">get started</a>&nbsp;for free or&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/demo-request">schedule a call</a>&nbsp;with our team.</p>

5 challenges Data Teams face supporting PLG revenue teams and how to address them

<div>&nbsp;</div>
   <p>Active metadata is like hot gossip. Here&rsquo;s why.</p>
   <p id="4e49">
     Just like data mesh or the metrics layer,&nbsp;<em class="mj"
       >active metadata</em
     >
     is the latest hot topic in the data world. As with every other new concept
     that gains popularity in the data stack, there&rsquo;s been a sudden explosion
     of vendors rebranding to &ldquo;active metadata&rdquo;, ads following you
     everywhere, and&hellip; confusion.
   </p>
   <p id="22a1">
     With everyone talking about active metadata, it must be pretty easy to
     understand, right?
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go mk">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*oj53dlEWig_Hcxmtj4GKJA.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go ml">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*bF3m6G6XM4D4UrYgMD8NsQ.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <p id="5481">
     Apparently not! I&rsquo;ve been talking about active metadata for over a year
     now, but I still see questions like these all the time.
   </p>
   <p id="c0f1">
     Active metadata can sound a bit scary, but it doesn&rsquo;t have to be. It is
     a must-have tool in the modern data toolbox, so if you&rsquo;re still
     wondering what it means, this article is for you.
   </p>
   <p id="76b6">
     I&rsquo;ve broken down the ideas behind active metadata with as little jargon
     as possible. Keep reading to learn what active metadata is, what it looks
     like, how you can actually use it, how it fits into the modern data stack, and
     why it even matters.
   </p>
   <h3 id="6b2d">What is active metadata?</h3>
   <p id="569c">
     I could start dropping some jargon here, but then both you and I will be
     asleep in seconds. So let&rsquo;s jump into an analogy instead.
   </p>
   <p id="4dd5">
     Imagine that you got your hands on the juiciest piece of tech gossip &mdash;
     Apple is expanding into recreational marijuana to literally help people
     &ldquo;think different&rdquo;.
   </p>
   <p id="aee1">
     There&rsquo;s no way you&rsquo;re going to keep something this exciting a
     secret. The world has to know. So you post it on your blog,&nbsp;<em
       class="mj"
       >blogspot.applefansunite.com</em
     >. All done, right?
   </p>
   <p id="b968">
     Just like&nbsp;<a
       class="mds-text-primary"
       href="https://www.vice.com/en/article/n7n74d/traffic-jams-are-possible-in-elon-musks-tunnels-apparently"
       target="_blank"
       rel="noopener ugc nofollow"
       >a car in the Hyperloop tunnel</a
     >, we all know that&rsquo;s not going anywhere. You can&rsquo;t just put the
     story somewhere and hope people will find it. You have to actually deliver it
     into people&rsquo;s hands.
   </p>
   <p id="47ef">
     You sharpen your PR chops, blast the news to tech reporters and news sites,
     and lo and behold it&rsquo;s&nbsp;<em class="mj">everywhere</em> in no time.
     It&rsquo;s already been mummified, and your grandfather just asked why apple
     farmers are talking about this Molly girl on your group chat.
   </p>
   <blockquote class="nj">
     <p id="988d" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt">Metadata is like this information. If it sits&nbsp;</em
       >passively<em class="nt"
         >&nbsp;in its own little world, with no one seeing or sharing it, does it
         even matter? But if it&nbsp;</em
       >actively<em class="nt"
         >&nbsp;moves to the places where people already are, it becomes part of
         and adds context to a larger conversation.</em
       >
     </p>
   </blockquote>
   <h3 id="36ed">What is the difference between active and passive metadata?</h3>
   <p id="e3db">
     <strong>Passive metadata</strong>&nbsp;is the standard way of aggregating and
     storing metadata into a static data catalog. This usually covers basic
     technical metadata &mdash; schemas, data types, models, etc.
   </p>
   <p id="4780">
     Think of passive metadata as putting out information on a personal blog. Every
     so often, it&rsquo;ll get picked up and go viral on Hacker News. But most of
     the time it&rsquo;s just going to sit unseen and unused, even when people
     actually need to know it.
   </p>
   <p id="c214">
     <strong>Active metadata</strong>&nbsp;makes it possible for metadata to flow
     effortlessly and quickly across the entire data stack, embedding enriched
     context and information in every tool in the data stack. It is usually more
     complex than passive metadata, covering operational, business, and social
     metadata along with basic technical information.
   </p>
   <p id="16a6">
     Think of active metadata as a viral story. It shows up everywhere you already
     live in what seems like seconds. It&rsquo;s immediately cross-checked against
     and combined with other information, bringing together a network of related
     context into a larger trend or story. And it sparks conversations, making
     everyone more knowledgable and informed in the end.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go nx">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*h71nNqp0b03WgO_XI2o8fw.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt">The four main types of metadata. (Image by author.)</em>
     </figcaption>
   </figure>
   <h3 id="7541">Why does active metadata matter?</h3>
   <blockquote class="nj">
     <p id="302a" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt"
         >To put it simply, no one wants to go to another website to &ldquo;browse
         the metadata&rdquo;.</em
       >
     </p>
   </blockquote>
   <p id="ca3d">
     As we embraced the internet and data exploded in the early aughts, companies
     realized they needed to manage all their new data.
   </p>
   <p id="091a">
     We entered a golden age of metadata management. New companies like&nbsp;<a
       class="mds-text-primary"
       href="https://www.informatica.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Informatica</a
     >,&nbsp;<a
       class="mds-text-primary"
       href="https://www.collibra.com/us/en"
       target="_blank"
       rel="noopener ugc nofollow"
       >Collibra</a
     >, and&nbsp;<a
       class="mds-text-primary"
       href="https://www.alation.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Alation</a
     >&nbsp;were created, and they hyped the importance of data catalogs. People
     needed a way to sort through all their options, so we got reports like
     Gartner&rsquo;s&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/3993025"
       target="_blank"
       rel="noopener ugc nofollow"
       >Magic Quadrant for Metadata Management</a
     >. Billion-dollar companies emerged, and companies spent hundreds of millions
     of dollars on metadata management.
   </p>
   <p id="41e1">
     Yet just last year, Gartner released their&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/4006759"
       target="_blank"
       rel="noopener ugc nofollow"
       >Market Guide for Active Metadata</a
     >&nbsp;and declared that &ldquo;Traditional metadata practices are
     insufficient&hellip;&rdquo;
   </p>
   <p id="5d60">
     That&rsquo;s because passive data catalogs solve the &ldquo;too many
     tools&rdquo; problem by adding&hellip; another tool. They aggregate metadata
     from different parts of the data stack, and it stagnates there. User adoption
     suffers, and&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-future-of-data-catalogs-868ffbe7354d"
       target="_blank"
       rel="noopener"
       >these exciting tools turn into expensive shelfware</a
     >.
   </p>
   <p id="2761">
     Active metadata sends metadata back into every tool in the data stack, giving
     the humans of data context wherever and whenever they need it &mdash; inside
     the BI tool as they wonder what a metric actually means, inside Slack when
     someone sends the link to a data asset, inside the query editor as try to find
     the right column, and inside Jira as they create tickets for data engineers or
     analysts.
   </p>
   <h3 id="050b">How does active metadata fit into the modern data stack?</h3>
   <p id="68c9">
     Active metadata functions as a layer on top of the modern data stack.
   </p>
   <p id="bdd2">
     It leverages open APIs to connect all the tools in your data stack and ferry
     metadata back and forth in a two-way stream. This is what allows active
     metadata to bring context, say, from Snowflake into Looker, Looker into Slack,
     Slack into Jira, and Jira back into Snowflake.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go od">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*53oiNCxKcmY9bLFNCKNnfg.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Active metadata in the modern data stack. (Image by author.)</em
       >
     </figcaption>
   </figure>
   <h3 id="dbd9">4 characteristics of active metadata</h3>
   <p id="e3b1">
     According to&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-gartner-magic-quadrant-for-metadata-management-was-just-scrapped-d84b2543f989"
       target="_blank"
       rel="noopener"
       >Gartner&rsquo;s new Market Guide for Active Metadata</a
     >, active metadata is an always-on, intelligence-driven, action-oriented,
     API-driven system, the opposite of its passive, static predecessor.
   </p>
   <p id="0837">
     This can be broken down into the four key characteristics of active metadata.
   </p>
   <ul>
     <li id="4667">
       <strong>Always on</strong>: Active metadata is always on. Rather than
       waiting for people to manually enter or parse metadata, this means
       continually collecting metadata at every stage of the modern data stack
       &mdash; logs, query history, usage statistics, and more.
     </li>
     <li id="fd88">
       <strong>Intelligent</strong>: Active metadata isn&rsquo;t just about
       collecting metadata. It&rsquo;s about constantly processing metadata to
       connect the dots and create intelligence from it. This means that with
       active metadata, the system will only get smarter over time as people use it
       more and it observes more metadata.
     </li>
     <li id="c801">
       <strong>Action-oriented</strong>: Active metadata doesn&rsquo;t just stop at
       intelligence. It should drive action by curating recommendations, generating
       alerts, and making it easier for people to make decisions &mdash; or even
       automatically making decisions without human intervention, like stopping
       downstream pipelines when data quality issues are detected.
     </li>
     <li id="3a52">
       <strong>Open by default</strong>: Active metadata platforms use APIs to hook
       into every piece of the modern data stack. This makes magical user
       experiences possible by saving data practitioners from the endless tool- and
       context-switching. This is called embedded collaboration, which is when work
       happens where you are with the least amount of effort.
     </li>
   </ul>
   <h3 id="a46d">5 use cases of active metadata</h3>
   <p id="b89a">
     There are dozens, if not hundreds, of use cases of active metadata. (Enough
     for several articles of their own &mdash; coming soon!) Let&rsquo;s go through
     a few of my favorites.
   </p>
   <ul>
     <li id="e13d">
       <strong>Purge stale or unused assets</strong>: Use active metadata to
       periodically calculate when each asset (e.g. a data table, dashboard, etc)
       was last used and/or how many people used it. If it was used within the last
       30 days, great! If an asset hasn&rsquo;t been used in the last 60 days,
       automatically archive it. If no one has touched it in the last 90 or 120
       days, purge it entirely.
     </li>
     <li id="a52a">
       <strong>Allocate compute resources dynamically</strong>: Imagine that 90% of
       users log in to a BI tool during the last week of a financial quarter.
       Active metadata can be used to automatically scale up compute resources just
       before that week and scale them down again afterward.
     </li>
     <li id="34bd">
       <strong>Enrich user experience in BI tools</strong>: Instead of switching
       between a BI tool and data catalog, use active metadata to bring context
       into dashboards. Relevant metadata (like business terms, descriptions,
       owners, and lineage) can be pushed into the BI tool. Then when someone is
       looking at each table, they can understand who owns it, where the data came
       from, etc. This information could even be used as labels in auto-generated
       reports.
     </li>
   </ul>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go os">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*aLuvXmyV2tyIr1YKbIbNZA.gif"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Enriching BI tools with active metadata. (Image from author.)</em
       >
     </figcaption>
   </figure>
   <ul>
     <li id="2ef1">
       <strong>Identify popular assets</strong>: Use active metadata to create a
       custom relevance score for each asset. This can be based on usage
       information from places like query logs, lineage, and BI dashboards. Then
       the most popular, relevant assets should be surfaced more frequently in
       search and checked more frequently for data quality issues.
     </li>
     <li id="d9f9">
       <strong>Notify downstream consumers</strong>: It&rsquo;s awful if the CEO
       ends up seeing a broken dashboard before the data team. Active metadata can
       be used to check for issues when a data store changes and notify downstream
       data users about potential issues. For example, when a data store is
       crawled, the new metadata would be compared against previous metadata. If
       there are any potential breaking changes (e.g. the addition or removal of a
       column), lineage could be used to find who owns this data store and notify
       them in Slack, Jira, email, etc.
     </li>
   </ul>
   <h3 id="aac4">The future of active metadata</h3>
   <p>
     As metadata becomes big data and big data becomes a behemoth, active metadata
     isn&rsquo;t just a wonderful dream. It&rsquo;s a necessity &mdash; the only
     way to understand today&rsquo;s data.
   </p>
   <p id="cb3d">
     Managing, processing, and analyzing metadata is the new normal for modern data
     teams. Doing this passively and manually, though, isn&rsquo;t possible.
     That&rsquo;s why it&rsquo;s been so exciting to see active metadata take shape
     in the last year and become the de facto standard for what people expect out
     of modern metadata.
   </p>
   <p id="f4b4">
     All of these use cases &mdash; like auto-tuned pipelines, automated data
     quality alerts, and continuously validated calculations &mdash; would have
     sounded wildly impossible just a few years ago. Today, they&rsquo;re actually
     in reach. I couldn&rsquo;t be more excited to see the intelligent data dream
     become a reality as active metadata continues to evolve in the coming years.
   </p>    
   

What Is Active Metadata, and Why Does It Matter?

<p><em>The modern data stack is on the rise. Many companies use raw data from their SaaS analytics tools as input for their data warehouse, but this introduces problems downstream. Are there better ways?</em></p>
      <h3 id="popularization-of-the-all-in-one-saas-analytics-tool" class="anchor anchorWithStickyNavbar_mojV">Popularization of the all-in-one SaaS analytics tool<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#popularization-of-the-all-in-one-saas-analytics-tool">​</a></h3>
      <p>When Google Analytics launched nearly 20 years ago, it took the market by storm. Its success created the mold for SaaS-based analytics offerings that included everything in one box: data collection, storage, analysis &amp; visualization.</p>
      <p>Ease-of-use and low barrier of entry were two of its main drivers. Instrumentation typically required little in terms of engineering resources and the included analyses, although somewhat limited, were ready to be used by a broad audience.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/urchin.jpeg" alt="Urchin Analytics, to become Google Analytics after acquisition." />
      <p class="imageCaption___5l">Urchin Analytics, to become Google Analytics after acquisition.</p>
      </div>
      <h3 id="the-big-shift-towards-data-warehousing" class="anchor anchorWithStickyNavbar_mojV">The big shift towards data warehousing<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-big-shift-towards-data-warehousing">​</a></h3>
      <p>While SaaS analytics tools are still very popular and widely used, their wild growth has left the data analysis landscape highly fragmented. Most companies have numerous tools collecting data, and each of them have a different way of analyzing &amp; consuming it. As a result, it&rsquo;s very hard to get a holistic view of what all that combined data is telling you.</p>
      <p>We&rsquo;re seeing a major shift where companies export data from their SaaS analytics tools to use as input for their data warehouse. Not only to create a single source of truth for all data consumers, but also to unlock the potential of raw data. By combining full raw data sets, you can go way beyond the scope of your SaaS analytics tools. You&rsquo;re technically only limited by the quality and completeness of the data.</p>
      <h3 id="the-rise-of-the-modern-data-stack" class="anchor anchorWithStickyNavbar_mojV">The rise of the modern data stack<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-rise-of-the-modern-data-stack">​</a></h3>
      <p>The big shift towards data warehousing is represented by the upsurge of companies that operate in this space and how fast they&rsquo;re growing.&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://snowplowanalytics.com/" target="_blank" rel="noopener noreferrer">Over 10.000 companies now use Snowplow</a>&nbsp;to process events at scale and send them straight into the DWH. dbt now has over&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://www.getdbt.com/blog/next-layer-of-the-modern-data-stack/" target="_blank" rel="noopener noreferrer">9.000 weekly active data transformation projects</a>.&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://www.rudderstack.com/" target="_blank" rel="noopener noreferrer">13.000+ sites and apps are running RudderStack</a>&nbsp;to pull data from SaaS tools and send it into data warehouses. The list goes on.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/dbt-cloud.png" alt="dbt Cloud in action" />
      <p style="text-align: center;" class="imageCaption___5l">dbt Cloud in action</p>
      </div>
      <h3 id="-but-saas-analytics-tools-arent-great-input-for-your-dwh" class="anchor anchorWithStickyNavbar_mojV">... but SaaS analytics tools aren&rsquo;t great input for your DWH<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#-but-saas-analytics-tools-arent-great-input-for-your-dwh">​</a></h3>
      <p>Most data warehouses are fed with data that&rsquo;s collected by all-in-one-box SaaS analytics tools. While historically explainable, this often causes problems downstream.</p>
      <h3 id="the-data-they-collect-is-not-designed-for-advanced-modeling" class="anchor anchorWithStickyNavbar_mojV">The data they collect is not designed for advanced modeling<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-data-they-collect-is-not-designed-for-advanced-modeling">​</a></h3>
      <p>There is a big difference between what data teams want their raw data to look like and what it actually looks like when it comes from SaaS analytics tools. The data sets they collect were never designed for advanced modeling on the raw data. It is often incomplete or overcomplete, unstructured and ambiguous. Significant grunt work is typically required before it can be used for modeling.</p>
      <h3 id="data-duplication-frustrates-adoption-of-a-single-source-of-truth" class="anchor anchorWithStickyNavbar_mojV">Data duplication frustrates adoption of a single source of truth<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#data-duplication-frustrates-adoption-of-a-single-source-of-truth">​</a></h3>
      <p>By extracting data from SaaS tools and loading it into a data warehouse, you&rsquo;re effectively duplicating a source.</p>
      <p>SaaS analytics tools all offer their own way to consume the collected data, and chances are many of your company&rsquo;s data consumers still rely on the dashboards of these tools to inform their daily decisions. Either out of habit, or simply because it offers an easy &amp; fast way to get data.</p>
      <p>In practice, this often leads to discussions about why numbers of different teams don&rsquo;t seem to be adding up, as the data team and data consumers both use different versions of the same source of data as input.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/before-saas.png" alt="" />
      <p class="imageCaption___5l">&nbsp;</p>
      </div>
      <h3 id="they-operate-as-black-boxes" class="anchor anchorWithStickyNavbar_mojV">They operate as black boxes<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#they-operate-as-black-boxes">​</a></h3>
      <p>SaaS analytics tools typically don&rsquo;t disclose the actual models behind an analysis. As a result, rebuilding an analysis to work with the data set in your data warehouse requires significant reverse engineering effort, and troubleshooting &amp; debugging can be painful when anything is off.</p>
      <hr />
      <h3 id="unbundling-saas-analytics-tools" class="anchor anchorWithStickyNavbar_mojV">Unbundling SaaS analytics tools<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#unbundling-saas-analytics-tools">​</a></h3>
      <p>The way data is used has evolved, and the all-in-one-box model that SaaS analytics tools use no longer fits. In order to effectively combine and analyze data from multiple sources and truly serve all data consumers from a single source of truth, we need to unbundle the black box and separate data collection, storage, modeling and visualization.</p>
      <h3 id="moving-all-data-consumption-downstream" class="anchor anchorWithStickyNavbar_mojV">Moving all data consumption downstream<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#moving-all-data-consumption-downstream">​</a></h3>
      <p>By separating data collection, all collected data can be sent straight into the data warehouse without stopovers. This eliminates source duplication and moves data consumption downstream. It enables&nbsp;<em>all</em>&nbsp;analysis &amp; visualization to occur on a single source of truth: the data warehouse.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/after-saas.png" alt="" />
      <p class="imageCaption___5l">&nbsp;</p>
      </div>
      <h3 id="shifting-from-vendor-locked-tools-to-raw-data-and-code" class="anchor anchorWithStickyNavbar_mojV">Shifting from vendor-locked tools to raw data and code<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#shifting-from-vendor-locked-tools-to-raw-data-and-code">​</a></h3>
      <p>Unbundling the black boxes also provides a big opportunity for the adoption of open standards. Shifting from vendor-locked tools to raw data and code enables data teams to adopt generic ways of structuring data to share and reuse each others&rsquo; tools, models and analyses.</p>
      <p>We think the future of analytics is open and data warehouse-native, and there are clear signals we&rsquo;re already on the way. It&rsquo;s time to enable meaningful collaboration and to unbundle SaaS analytics.</p>

The Unbundling of SaaS Analytics

<p>Breaking down some of the problems I’ve seen in data collaboration and offering advice on how to make better, faster decisions with collaborative analytics.</p>
<p>We all know how analytics can help make businesses make better decisions. Being data-driven just simply isn&rsquo;t an option anymore. Startups are collecting more data than ever before and are increasingly using that data to&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://a16z.com/2011/08/20/why-software-is-eating-the-world/">eat the world</a>. We&rsquo;ve seen the rise of&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://www.whatmatters.com/articles/ted-talk">OKR goal-setting</a>, reporting tabs in nearly every SaaS tool, and metrics dashboards lining the walls in office hallways.<br /><br />Yet for all of this progress, the ways we share insights and make decisions together haven&rsquo;t evolved over the last decade. Too many companies spend too long collecting, preparing, and reporting on data to improve. In this post, I&rsquo;ll break down some of the problems I&rsquo;ve seen first-hand in data collaboration and offer advice on how you can create a better environment for creating, sharing, and measuring data together.</p>
<h2 id="the-problems">The problems</h2>
<h3 id="finding-a-source-of-truth"><br />Finding a Source of Truth</h3>
<p>Startups now have a plethora of dashboards in our tools to find data. But it&rsquo;s still too challenging to find the data you need. Legacy BI tools suffer from&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://venturebeat.com/2022/04/10/7-easy-ways-to-fight-the-data-sprawl/">dashboard sprawl</a>&nbsp;and poor search experiences. It&rsquo;s challenging to agree upon and standardize metrics to get common questions answered in a scalable way.</p>
<p>To fix this, we&rsquo;ve tried creating data catalogs so operators can more easily answer FAQs and find the data they&rsquo;re looking for. We&rsquo;ve tried to provide definitions for data to help operators understand what tables entail, how metrics are calculated, and what a particular column means. These efforts result in incremental progress towards becoming data-driven, but more is needed.</p>
<h3 id="data-breadlines">Data Breadlines</h3>
<p>Too few employees have access to real-time data to make everyday decisions. When operators need to query the database or ask a question involving data from multiple sources, they&rsquo;re&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/why-were-building-canvas-a-data-tool-for-non-technical-teams/">forced to ask for help</a>&nbsp;in a shared analytics channel or submit a ticket to their data or engineering teams.</p>
<p>This process results in a painful waiting period until the technical team can prioritize the request. Even when the data or engineering team can prioritize a request, there&rsquo;s typically a frustrating back-and-forth of questions and answers across Slack, Email, and Zoom.</p>
<p>Operators shouldn&rsquo;t have to wait days or even weeks for the answers they need, while technical teams should be able to focus on strategic work without constant pings for help on&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/enabling-ad-hoc-analysis/">tactical, ad-hoc questions.</a></p>
<h3 id="siloed-broken-workflows">Siloed, Broken Workflows</h3>
<p>Stop me if you&rsquo;ve gone through this process before. You need to prepare for a presentation. You go to your apps, pull CSVs, build some metrics, screenshot some charts, and throw them into a deck. You present the information and get some live questions about the data. Unfortunately, because the chart is a static screenshot, you can&rsquo;t dive into the data live, so you promise to follow up afterward.</p>
<p>The hours-long workflow we endure to prepare presentations across a suite of apps is fundamentally broken. There&rsquo;s no connective tissue between the data sources, the analysis tools, the presentation, and the feedback on the data.</p>
<p>Because of these silos, we may get fewer eyeballs from peers before a presentation. We might forget to include a killer insight or make a potential error in our analyses. Because it's too difficult to answer questions together live during a presentation, we can get caught flat-footed and look unprepared. And because it's so difficult to measure and discuss progress on metrics together, we fail to monitor updates closely and act quickly and decisively as teams.</p>
<h2 id="how-collaborative-analytics-helps">How Collaborative Analytics Helps</h2>
<p>To help teams move quickly to get the answers they need, you should consider investing in collaborative processes and tools. See below for some key components you should consider.</p>
<h3 id="the-warehouse-as-a-source-of-truth">The Warehouse as a Source of Truth</h3>
<p>Having a single source of truth that&rsquo;s accurate and up-to-date ensures your teams can all work off the correct data independently and confidently. Best-in-class companies typically use a data warehouse and ETL tools to centralize data from multiple sources in one place. If you don&rsquo;t know where to start, check out my guide on navigating the tools in the modern data stack.</p>
<p>Suppose you don&rsquo;t have the bandwidth or technical experts to implement a data stack correctly. In that case (time for a quick plug),&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/no-warehouse-no-problem/">Canvas provides a managed data stack</a>&nbsp;for centralizing your apps and modeling your data in one place.</p>
<h3 id="reusable-datasets-for-fast-discovery">Reusable Datasets for Fast Discovery</h3>
<p>With your data in one place, you&rsquo;ll need to organize it, provide context, and make it easy to find. Key metrics and common areas should be searchable and social proof should help teams quickly identify the right data to answer their questions.</p>
<p>You should have clear owners for building and maintaining dashboards on the company and team levels. They should define and document data clearly and ensure a clear process for getting help on questions. You should be aware of which dashboards are frequently used and invest accordingly, while continually improving or pruning dashboards that are ghost towns.</p>
<h3 id="collaborative-workspaces-for-exploration">Collaborative Workspaces for Exploration</h3>
<p>Shared, collaborative workspaces are critical in breaking down silos and helping teams share and discuss insights with others.</p>
<p>Yes, having dashboards are a must. But as I pointed out, your teams will still have everyday questions that dashboards don&rsquo;t answer. Instead of a ticketing system, you should have a way for your operators to answer these questions without knowing SQL.</p>
<p>No-code and spreadsheet-like interfaces help your operators gain self-serve access without going days or weeks for technical teams to answer strategic and tactical questions. And when they have a question, operators and technical teams can collaborate and comment right where the data lives. This way, you can ditch the screenshots, snippets, and Slack channels and start making decisions together from anywhere.</p>
<h2 id="canvas-the-collaborative-analytics-platform">Canvas, the Collaborative Analytics Platform</h2>
<p>At&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/">Canvas</a>, we&rsquo;re thrilled to be working on the future of data collaboration. Unlike legacy BI tools, Canvas has:</p>
<ul>
<li>Integrations to over a hundred apps so you can centralize and explore your data without a warehouse</li>
<li>A spreadsheet-like interface for your teams to pivot, write formulas, join, and create charts without SQL</li>
<li>Figma-like collaboration lets your teams communicate where the data lives</li>
<li>Out-of-the-box usage metrics inform data teams of who is using specific models so they can invest in popular areas or fix problems affecting less helpful models</li>
<li>Role-based access control so you can make sure the right people have access to the right data</li>
<li>A native dbt integration so your data models and documentation about the tables and columns are automatically synced and surfaced directly to operators</li>
</ul>

How to Make Better Decisions Together with Collaborative Analytics

<p class="p2">
    <span class="s1"
      >The term &ldquo;observability&rdquo; means many things to many people. A
      lot of energy has been spent&mdash;particularly among vendors offering an
      observability solution&mdash;in trying to define what the term means in one
      context or another.</span
    >
  </p>
  <p class="p2">
    <span class="s1"
      >But instead of getting bogged down in the &ldquo;what&rdquo; of
      observability, I think it&rsquo;s more valuable to address the
      &ldquo;why.&rdquo; What are we trying to accomplish with observability? What
      is the end goal?&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >At&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/why-unravel/"
        target="_blank"
        rel="noopener"
        >Unravel</a
      >, I&rsquo;m not only the co-founder and CTO but also the head of our
      Customer Success team, so when thinking about modern data stack
      observability, I look at the issue through the lens of customer problems and
      what will alleviate their pain and solve their challenges.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >I start by considering the ultimate end goal or Holy Grail: autonomous
      systems. Data teams just want things to work; they want everything to be
      taken care of, automatically. They want the system itself to be watching
      over issues, resolving problems without any human intervention at all.
      It&rsquo;s the true spirit of AI: all they have to do is tell the system
      what they want to achieve. It&rsquo;s going from reactive to
      proactive.&nbsp;</span
    ><span class="s3"
      >No one ever has to troubleshoot the problem, because it was
      &ldquo;fixed&rdquo; before it ever saw the light of day. The system
      recognizes that there will be a problem and auto-corrects the issue
      invisibly.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >As an industry, we&rsquo;re not completely at that point yet for the modern
      data stack. But we&rsquo;re getting closer.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >We are on a continuous and incremental spectrum of progressively less toil
      and more automation: going from the old ways of manually stitching together
      logs, metrics, traces, and events from disparate systems; to accurate
      extraction and correlation of all that data in context; to automatic
      insights and identification of significant patterns; to AI-driven actionable
      recommendations; to autonomous governance.</span
    >
  </p>
  <img
    style="width: 100%"
    src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-observability-spectrum-750x299.png"
    loading="lazy"
    alt=""
  />
  <p></p>
  <h3 class="p5">Beyond observability: to optimization and on to governance</h3>
  <p class="p2">
    <span class="s1"
      >Let&rsquo;s say you have a data pipeline from Marketing Analytics called
      ML_Model_Update_and_Scoring that&rsquo;s missing its SLA. The SLA specifies
      that the pipeline must run in less than 20 minutes, but it&rsquo;s taking 25
      minutes. What happened? Why is the data pipeline now too slow? And how can
      you tune things to meet the SLA? This particular application is pretty
      complex, with multiple jobs processing in parallel and several components
      (orchestration, compute, auto-scaling clusters, streaming platforms,
      dashboarding), so the problem could be anywhere along the line.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >It&rsquo;s virtually impossible to manually pore through t</span
    ><span class="s3"
      >he thousands of logs that are generated at each stage of the pipeline from
      the various tools</span
    ><span class="s2"
      >&mdash;Spark and Kafka and Airflow logs, Databricks cluster logs,
      etc.&mdash;to &ldquo;untangle the wires&rdquo; and figure out where the
      slowdown could be. But where should you even start? Even if you have an idea
      of what you&rsquo;re looking for, it can take hours&mdash;even days or weeks
      for highly complex workflows&mdash;to stitch together all the raw
      metrics/events/logs/traces data to figure out what&rsquo;s meaningful to why
      your data pipeline is missing its SLA. Just a single app can run 10,000
      containers on 10,000 nodes, with 10,000 logs.&nbsp;</span
    >
  </p>
  <img style="width: 100%" src="https://miro.medium.com/max/1200/1*XYQOw2kNJKZGKzBOrR0E7A.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s2">That&rsquo;s where observability comes in.</span>
  </p>
  <h3>
    <span id="Observability_tells_you_what"
      ><span class="s1">OBSERVABILITY TELLS YOU &ldquo;WHAT&rdquo;</span></span
    >
  </h3>
  <p class="p2">
    <span class="s1"
      >Instead of having to sift through reams of logs and cobble together
      everything manually,&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/dataops/"
        target="_blank"
        rel="noopener"
        >full-stack observability</a
      >&nbsp;extracts all the relevant raw data from all the various components of
      your data stack and correlates it to paint a picture of what&rsquo;s going
      on. All the information captured by telemetry data (logs, metrics, events,
      traces) is pulled together in context and in the &ldquo;language
      you&rsquo;re speaking&rdquo;&mdash;the language of&nbsp;</span
    ><span class="s4">data applications and pipelines</span
    ><span class="s1">.</span>
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-observability-before-2-750x264.png" loading="lazy" alt="" />
  <p></p>
  <p id="attachment_9257" class="wp-caption alignnone">
    In this case, observability shows you that while the application completed
    successfully, it took almost 23 minutes (22m 57s)&mdash;violating the SLA of
    20 minutes. Here, Unravel takes you to the exact spot in the complex pipeline
    shown earlier and, on the left, has pulled together granular details about the
    various jobs processing in parallel. You can toggle on a Gantt chart view to
    get a better view of the degree of parallelism:
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-gantt-750x544.png" loading="lazy" alt="" />
  <p></p>
  
  <p>
    So now you know what caused the pipeline to miss the SLA and where it happened
    (jobs #0 and #3), but you don’t know why. You’ve saved a lot of investigation
    time—you get the relevant data in minutes, not hours—and can be confident that
    you haven’t missed anything, but you still need to do some detective work to
    analyze the information and figure out what went wrong.
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-chart-1-750x171.png" loading="lazy" alt="" />
  <h3 class="p7">
    <span id="Optimization_tells_you_whyand_what_to_do_about_it"
      ><span class="s1"
        >OPTIMIZATION TELLS YOU &ldquo;WHY&rdquo;&mdash;AND WHAT TO DO ABOUT
        IT</span
      ></span
    >
  </h3>
  <p class="p2">
    <span class="s4"
      >The better&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/dataops/"
        target="_blank"
        rel="noopener"
        >observability tools</a
      >&nbsp;also point out, based on patterns, things you should pay attention
      to&mdash;or areas you don&rsquo;t need to investigate.&nbsp;</span
    ><span class="s1"
      >By applying machine learning and statistical algorithms, it essentially
      throws some math at the correlated data to identify significant
      patterns</span
    ><span class="s4"
      >&mdash;what&rsquo;s changed, what hasn&rsquo;t, what&rsquo;s different.
      This pipeline runs regularly; why was it slow this time?&nbsp;</span
    ><span class="s1"
      >It&rsquo;s the same kind of analysis a human expert would do, only done
      automatically with the help of ML and statistical algorithms.</span
    >
  </p>
  <p class="p2">
    <span class="s1"
      >While it would certainly be helpful to get some generalized insight into
      why the pipeline slowed</span
    ><span class="s4"
      >&mdash;it&rsquo;s a memory issue&mdash;what you really need to know is what
      to do about it.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >AI-enabled observability goes beyond telling you what happened and why, to
      pinpointing the root cause and providing recommendations on exactly what you
      need to do to fix it.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-ai-reco-750x468.png" loading="lazy" alt="" />
  <p class="p4">
    <span class="s2"
      ><span class="s3"
        >AI-driven recommendations provide specific configuration parameters that
        need to be applied in order for the pipeline to run faster</span
      >&nbsp;and meet the 20-minute SLA. After the AI recommendations are
      implemented, we can see that the pipeline now runs in under 11
      minutes&mdash;a far cry from the almost 23 minutes before.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-after-750x103.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s3"
      >Too often getting to the point of actually fixing the problem is another
      time-consuming effort of trial and error. Actionable AI recommendations
      won&rsquo;t fix things automatically for you</span
    ><span class="s2"
      >&mdash;taking the action still requires human intervention&mdash;but all
      the Sherlock Holmes work is done for you. The fundamental questions of what
      went wrong, why, and what to do about it are answered automatically.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >Beyond optimizing performance, AI recommendations can also identify
      where&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/cluster-resource-optimization/"
        target="_blank"
        rel="noopener"
        >cost could be improved</a
      >. Say your pipeline is completing within your SLA commitments, but
      you&rsquo;re spending much more on cloud resources than you need to. An AI
      engine can determine how many or what size containers you actually need to
      run each individual component of the job&mdash;vs. what you currently have
      configured. Most enterprises soon realize that they&rsquo;re overspending by
      as much as 50%.&nbsp;</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-cost-1-1024x476.png" loading="lazy" alt="" />
  <p></p>
  <h3 class="p7">
    <span id="Governance_helps_avoid_problems_in_the_first_place"
      ><span class="s1"
        >GOVERNANCE HELPS AVOID PROBLEMS IN THE FIRST PLACE</span
      ></span
    >
  </h3>
  <p class="p4">
    <span class="s2"
      >These capabilities save a ton of time and money, but they&rsquo;re still
      relatively reactive. What every enterprise wants is a more proactive
      approach to making their data applications and pipelines run better, faster,
      cheaper. Spend less time firefighting because there was no fire to put out.
      The system itself would understand that the memory configurations were
      insufficient and automatically take action so that pipeline performance
      would meet the SLA.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >For a whole class of data application problems, this is already happening.
      AI-powered recommendations and insights lay the groundwork for putting in
      place some automated governance policies that take action on your
      behalf.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-chart-2-750x141.png" loading="lazy" alt="" />
  <p></p>
  <p>
    Governance is really all about converting the AI recommendations and insights
    into impact. In other words, have the system run automatic responses that
    implement fixes and remediations for you. No human intervention is needed.
    Instead of reviewing the AI recommendation and then pulling the trigger, have
    the system apply the recommendation automatically.&nbsp;
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/new-alert.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s2"
      >Policy-based governance rules could be as benign as sending an alert to the
      user if the data table exceeds a certain size threshold or as aggressive as
      automatically requesting a configuration change for a container with more
      memory.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >This is true&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/aiops-for-big-data/"
        target="_blank"
        rel="noopener"
        >AIOps</a
      >. You don&rsquo;t have to wait until after the fact to address an issue or
      perpetually scan the horizon for emerging problems. The system applies AI/ML
      to all the telemetry data extracted and correlated from everywhere in your
      modern data stack to not only tell you what went wrong, why, and what to do
      about it, but it predicts and prevents problems altogether without any human
      having to touch a thing.</span
    >
  </p>
  

Beyond Observability for the Modern Data Stack

<p>
      A
      <a
        style="color: #f66640"
        href="https://venturebeat.com/2021/08/04/97-of-execs-say-data-science-is-crucial-to-maintaining-profitability/"
        target="_blank"
        >majority</a
      >
      of business leaders believe data insights are key to the success of their<br />business
      in a digital environment. However, many companies
      <a
        style="color: #f66640"
        href="https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard"
        target="_blank"
        >struggle</a
      >
      to build a<br />data-driven culture, with a key reason being the lack of a
      sound data democratization<br />strategy.
    </p>
    <p>
      A survey by Google Cloud and Harvard Business Review shows
      <a
        style="color: #f66640"
        href="https://cloud.google.com/resources/hbr-turn-data-into-business-value-report"
        target="_blank"
        >97%</a
      >
      of companies<br />recognize that democratizing data is critical to their
      success, but only
      <a
        style="color: #f66640"
        href="https://cloud.google.com/resources/hbr-turn-data-into-business-value-report"
        >60%</a
      >
      are<br />"effectively" moving toward that goal.
    </p>
    <p>But what's data democratization?</p>
    <p>
      Democratizing data in an organization is much more than just giving employees
      access<br />to data. It's a process that enables everyone in an organization,
      regardless of their<br />technical skills, to be comfortable working with
      data, feel confident talking about it, and<br />make data-driven decisions
      that can positively impact various aspects of the business.<br />It's an
      ongoing process because it goes hand in hand with data literacy, which is also
      a<br />continuous process.
    </p>
    <h3>The rise of all-in-one data solutions</h3>
    <p>
      Until recently, data was only used by a few technical specialists in
      organizations who<br />would set up and operate the systems needed to make
      data available and accessible to<br />the rest of the team. But as the data
      democratization trend grows, new tools are being<br />developed for
      non-technical professionals to provide organization-wide access to data<br />and
      analytics.
    </p>
    <p>
      Of the data software available on the market, all-in-one tools that offer a
      comprehensive<br />range of solutions through a single platform are the most
      popular options for many<br />organizations looking to extract more value from
      their data.
    </p>
    <p>
      There are two main challenges that all-in-one data tools are designed to
      address,<br />namely the gap between data and commercial teams, and the
      expensive and<br />time-consuming nature of the process of data
      maturity&mdash;a measurement of how capable<br />an organization is at making
      the most of its data.
    </p>
    <h3>The gap between data and commercial teams</h3>
    <p>
      Data and commercial teams are often siloed functions within a company, and
      it's not<br />uncommon for them to fail to communicate with each other and
      even get in each other's<br />way. Different training backgrounds, different
      goals, or lack of context are just some of<br />the reasons.
    </p>
    <p>
      This inevitably leads to inefficiencies as each department takes a different
      direction on a<br />given project. In fact,
      <a
        style="color: #f66640"
        href="https://blog.bit.ai/collaboration-statistics/"
        target="_blank"
        >97%</a
      >
      of employees and executives believe lack of communication<br />and alignment
      within an organization impacts the outcome of a task or project.
    </p>
    <p>
      Better collaboration and cross-functional coordination between different
      departments is<br />therefore a key success factor, especially in today's
      world of remote work where a<br />company's growth and survival depends more
      than ever on effective teamwork.
    </p>
    <h3>Data maturity being an expensive and long process</h3>
    <p>
      Data maturity can be a costly and lengthy process for many organizations. A
      <a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        >Gartner</a
      ><br /><a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        >survey</a
      >
      reveals that 87.5% of companies have low levels of data and analytics
      maturity.<br />This means that individual business units are pursuing their
      own data and analytics<br />initiatives as stand-alone projects, and there's
      no common structure that would enable<br />the enterprise to efficiently
      leverage advanced analytics.
    </p>
    <p>
      According to
      <a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        target="_blank"
        >Gartner</a
      >, low maturity can be the result of a lack of vision and skills, a lack<br />of
      experience in strategic planning and execution, limited budgets, and primitive
      or<br />outdated infrastructure. As a result, organizations suffer from
      incomplete or inconsistent<br />data sets, poor data quality, duplicate data
      processes, data security issues, and poor<br />coordination across the
      enterprise.
    </p>
    <p>
      For data democratization to reach its full potential, organizations must first
      improve their<br />maturity, but they may not have enough time and financial,
      technological, and human<br />resources to do so. It can also be difficult to
      bring on board skilled data experts&mdash;who<br />are in high demand
      today&mdash;or to get buy-in from leadership and senior management to<br />advance
      data initiatives.
    </p>
    <p>
      This is where all-in-one data tools can shine. They eliminate the need to
      invest in<br />multiple external tools or in-house solutions, consolidate and
      centralize all data, and<br />save teams the time and effort of searching
      multiple data sources.
    </p>
    <h3>Benefits of all-in-one data tools</h3>
    <p>
      Let's now take a closer look at how commercial teams, and companies in
      general, can<br />access and benefit from all-in-one data tools.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >1. Improved marketing performance</strong
      >
    </p>
    <p>
      According to a study,
      <a
        style="color: #f66640"
        href="https://get.treasuredata.com/rs/714-XIJ-402/images/the-state-of-the-customer-journey-arm-treasure-data.pdf"
        target="_blank"
        >47%</a
      >
      of marketers say that data silos are their biggest challenge<br />when it
      comes to gaining insights from data.
    </p>
    <p>
      A single data platform allows companies to create a journey map for customers
      that<br />shows how they interact with the brand over time. This journey map
      helps marketers<br />identify key touchpoints that can be improved or better
      leveraged. By being able to<br />segment users by their behavior, location, or
      device, companies can better target them.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >2. Improved customer experience</strong
      >
    </p>
    <p>
      A Forrester report shows that 72% of companies find it challenging to manage
      multiple<br />customer relationship management (CRM) tools across geographies
      and technology<br />silos.
    </p>
    <p>
      An all-in-one data platform enables the collection of customer feedback from
      multiple<br />channels, providing a holistic view of customer sentiment and
      satisfaction. This<br />feedback helps commercial teams identify what
      customers want and find ways to<br />improve the overall experience for their
      customers, while data teams get to reclaim their<br />time and energy to
      in-depth, impactful work.
    </p>
    <p>
      This is especially important because
      <a
        style="color: #f66640"
        href="https://go.emplifi.io/us-uk-customer-expectations-report.html"
        target="_blank"
        >61%</a
      >
      of people are willing to pay at least 5% more<br />if they know they'll have a
      good customer experience, and
      <a style="color: #f66640;" href="https://go.emplifi.io/us-uk-customer-expectations-report.html"
        >56%</a
      >
      say that the quality of<br />customer service has a greater impact on how
      positively they view a brand than any<br />other factor.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >3. Better customer lifetime value and retention</strong
      >
    </p>
    <p>
      The ability to identify key metrics in real time using a single data platform
      enables rapid<br />decision making and helps increase customer loyalty and
      engagement. This also<br />improves customer retention as companies can tailor
      their products or services to meet<br />customers' needs.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >4. Reduced costs and time allocated to data operations</strong
      >
    </p>
    <p>
      As all-in-one data tools help companies identify their customers' needs in
      real time, they<br />can help reduce costs, while increasing profits, and time
      spent on data operations.<br />That's because they can become data-driven
      without a full team of specialists, and if<br />and when they do hire a data
      expert or team, they&rsquo;ll be well equipped to dive right into<br />the
      work. Additionally, companies can identify when the cost of customer
      acquisition<br />outweighs the benefits, allowing for rapid growth or
      discontinuation of programs that<br />aren't working.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >5. Higher efficiency and revenue</strong
      >
    </p>
    <p>
      When a company has access to information about customer needs through a
      single<br />data platform, it can streamline its operations and achieve
      greater efficiency by<br />delivering products or services in the shortest
      possible time. This, in turn, enables faster<br />revenue growth.
    </p>
    <h3>Future-proofing your company through Data</h3>
    <p>
      All-in-one data solutions can facilitate the seismic shift to a data-driven
      culture. They<br />can help organizations launch their data operations in a
      matter of hours instead of<br />months or even years. They enable businesses
      to become data-driven without a full<br />team of specialists, and even if
      they do hire a data expert or team, they're well-equipped<br />to get started
      right away.
    </p>
    <p>
      Meanwhile, teams in each department can focus on important tasks instead of
      sifting<br />through and pulling data from multiple data sources. Easy access
      to data-driven insights<br />would lead to greater efficiency, productivity,
      and profits as well as higher data security.<br />Customers will also have a
      much better experience. So it's a win-win situation.
    </p>
    <p>
      In today's borderless digital world where data is the lingua franca, data
      democratization<br />is no longer a nice-to-have, but a necessity for
      organizations to achieve their goals and<br />gain a competitive advantage.
    </p>
    <p>
      To succeed, executives should invest in all-in-one data platforms and take a
      coordinated<br />and holistic approach to their efforts toward data
      democratization, which has triggered a<br />seismic shift in the business
      world.
    </p>
    

How All-in-One Tools Are Accelerating Data Democratization

<div>
      <p class="lead-paragraph blog-post"></p>
      <div class="cms-rich-text blog-post w-richtext">
          <p>The Modern Data Stack has been the hot topic of the data world for quite some time now. In the 12 years since
              Redshift was initially released, we’ve seen a mass explosion of capital and innovation that has enabled
              organizations to transform how they think about and leverage data.</p>
          <p>For this article, I’d like to discuss one of the major advantages of the MDS — the shift from ETL to ELT —
              and the primary users who are served and underserved in this paradigm. Today, I believe that the MDS
              provides a massive unlock in value for data engineers and analytic engineers but lacks the easy-to-use and
              intuitive toolkits for the data analyst, citizen data scientist, and knowledge worker that sits closest to
              the decision.</p>
          <p>The purpose of this blog is not to discuss the past, present, or future of the Modern Data Stack. There are
              many articles that provide a fantastic history lesson on the transition of on-prem legacy systems to the
              truly cloud native, modern data stack most of us know and love today. If you’re new to the MDS and
              interested in learning more, dbt’s CEO, Tristan Handy, wrote a great post, <a
              class="mds-text-primary"
                  href="https://www.getdbt.com/blog/future-of-the-modern-data-stack/" target="_blank">The Modern Data
                  Stack: Past, Present, and Future</a>, in December 2020, that perfectly articulates the transformation at
              hand.</p>
          <h3><strong>What is ELT and why is it important?</strong></h3>
          <p>You’ve likely heard about ELT — Extract Load and Transform… the Modern Data Stack’s evolution on ETL. This is
              a game changer by nature in that it enables organizations to ingest raw data into the data warehouse and
              transform it later. ELT gives end-users access to the entirety of the datasets they need by circumventing
              downstream issues of missing data that could prevent a specific business question from being answered.</p>
          <p>ELT isn’t a new topic. I remember the early days of my career in the Big Data/Hadoop space. Originally
              introduced by Google in 2005, the promise of Hadoop was, more or less, ELT. Don’t transform data on its way
              into the data platform and risk losing critical data that you need down the road. Instead, ship the entirety
              of the raw data into Hadoop, and you’ll have it…. for whenever you need it.</p>
          <p>The promise was great! Truly an amazing sales pitch, but unfortunately, it just didn’t work out. Data
              processing was too slow. It frustrated end-users, was not usable by the knowledge workers, and didn’t enable
              agility in analytically driven decision making. Spark made huge progress in reducing the speed of
              processing, but a better solution has emerged.</p>
          <p>This is where the Modern Data Stack, and the Cloud Data Warehouse (the core foundation of the MDS), differ.
              By separating storage from compute and enabling elastic computation in the cloud, tools such as Snowflake,
              BigQuery, Delta Lake, and Redshift have changed this paradigm. Data teams can now extract and load raw data
              directly into cloud storage for a super low cost per terabyte, and the users who need prepared data for
              analysis can get lightning-fast speeds throughout their data analysis workflow.</p>
          <h3><strong>What are the components, and who are the major players powering ELT?</strong></h3>
          <p>To make things simple, I’ve outlined a few of the major technologies being leveraged today. Again, for a
              complete overview of the MDS, check out Tristan Handy’s blog that I posted above.</p>
          <ul role="list">
              <li>Cloud Data Warehouse — Snowflake, BigQuery, Delta Lake, Redshift, and Synapse</li>
              <li>Data Integration — Fivetran, Airbyte, and Segment</li>
              <li>Data Transformation — dbt (the clear winner in the data transformation game)</li>
              <li>Reverse ETL — Census and Hightouch</li>
          </ul>
          <figure  >
              <div><img
  style="width:100%" src="https://uploads-ssl.webflow.com/6276a6c8de031667550a3847/62866cc38480f6788747d7c9_mds.png"
                      loading="lazy" alt=""></div>
              <figcaption>The modern data stack with leading vendors in each Silo</figcaption>
          </figure>
          <h3><strong>Who are the main users being served? Who are the underserved users?</strong></h3>
          <p>To answer this question, it’s first important to level set on the key users we see in the end to end data
              stack. I like to think of this by breaking down users into <em>data producers</em> and <em>data
                  consumers</em>. Data producers are the individuals who are integrating and modeling data on behalf of
              the organization. Data consumers consume the cleaned and modeled data to answer critical business questions
              from that data.</p>
          <figure  >
              <div><img
  style="width:100%"
                      src="https://uploads-ssl.webflow.com/6276a6c8de031667550a3847/62867609a729465066769724_personas.png"
                      loading="lazy" alt=""></div>
              <figcaption>Professionals who use the modern data stack and descriptions</figcaption>
          </figure>
          <p>In working in this space for years, I’ve seen firsthand that these tools (mentioned above) enable the data
              engineer and the (newly minted job title) analytic engineer super well. They provide modern, easy-to-use,
              and cloud native code-first tools for the ELT workflow and provide the organizations that employ these users
              with better speed and governance of their data assets.</p>
          <p>Said differently, the Modern Data Stack is awesome in that it enables centralization, speed, agility, and
              performance. But, today, it is a walled garden for users who are highly proficient in SQL development.</p>
          <p>So who is left out in this world? In Tristan’s article on the past, present, and future of the Modern Data
              Stack (posted above), he argues that one of the biggest gaps still to be closed in the MDS is the bridge to
              the data consumer.</p>
          <p>Tristan goes as far as saying, “Data consumers were actually more self-serve <em>prior</em> to the advent of
              the modern data stack: Excel skills are widely dispersed through the population of knowledge workers. There
              has not yet been an analogous interface where all knowledge workers can seamlessly interact with data in the
              modern data stack in a horizontal way.”</p>
          <p>I couldn’t agree more! In my viewpoint, the data consumer, in the analytics workflow, is not only underserved
              but arguably the most essential user who needs a solution. After all, this user is the one who sits closest
              to the business decision that can drive differentiated advantage in the marketplace.</p>
          <p>This knowledge user is different. They typically understand SQL conceptually, and many are proficient at
              writing SQL code, but the true power of this user is that they act as both the data SME and the business SME
              that truly understands the problem at hand and the business context of what they are trying to solve. Said
              differently, the critical value of the knowledge user is domain expertise, not programming expertise.</p>
          <p>This user needs to answer questions from data. They need insights from data…. fast. They are typically on the
              hook for preparing critical data insights that lead to executive and senior management decisions on the
              direction of the company. Important stuff!</p>
          <h3><strong>So… what does ELT mean for this knowledge user? Envisioning a new world.</strong></h3>
          <p>In my opinion, ELT represents a transformational advantage for this user. Traditional BI tools such as
              Tableau do an amazing job at enabling visualization of data to answer questions but rely on the data
              engineer or analytic engineer to first prepare raw data into an analytical table that Tableau can query.</p>
          <p>This means that the visual insights from raw data workflow is, today, inherently not self-service. An analyst
              needs to partner with the analytics engineering team to get the data they need in the format they need. This
              slows down decision making to a halt and typically annoys the analytic engineering team with ad hoc requests
              that fall outside the scope of their prioritized sprint.</p>
          <p>Analytic engineers own production-level data pipelines that the business relies on. These pipelines are
              static, hardened, and scalable. This is a fundamentally different requirement than ad hoc data insights
              which are agile, iterative, and experimental. A new solution paradigm is needed.</p>
          <p>This requirement is not new. Alteryx built a massive business around no-code data preparation, but it was
              built for the traditional, on-premise world. It does not give users the advantages of ELT within the Modern
              Data Stack. It is, instead, ETL, forcing users to extract data from the central warehouse, impacting
              performance, speed, and auditability of the transformation pipeline.</p>
          <p>In another blog from Tristan, <a class="mds-text-primary" href="https://roundup.getdbt.com/p/code-and-no-code-having-it-both-ways?s=r"
                  target="_blank">“Code and No-Code, Having it Both Ways”</a>, he describes a major gap in the marketplace
              requiring no-code tools that, behind the scenes, write Modern Data Stack native SQL for the end user that is
              automatically pushed down to data in the Cloud Data Warehouse.</p>
          <p>A solution like this would enable data consumers to capture the value of ELT in the Modern Data Stack without
              forcing them to become an expert-level SQL developer. It would enable a rapid up-leveling of skill sets
              within a user community overnight.</p>
          <h3><strong>No-Code on the Modern Data Stack — A new paradigm to enable the knowledge worker</strong></h3>
          <p>At Rasgo, this is a problem we’ve thought a lot about. We’ve worked with hundreds of data analysts, business
              users, and citizen data scientists who have heard a lot about the Modern Data Stack and the value their data
              engineering teams have gleaned, but haven’t yet been able to fully capture the value of the cloud data
              ecosystem themselves.</p>
          <p>From my viewpoint, a new breed of tools purpose built for these end users needs to emerge to truly bring data
              consumers into the MDS ecosystem. These tools need to be UI first, enable a more iterative workflow, and be
              fully native to the underlying dbt transformation engine. They need to enable business users to answer
              ad-hoc questions from data as they come up, whether or not these questions rely on metrics that have already
              been created by the Analytic Engineering team. Without this, data consumers will always be at the mercy of
              their engineering counterparts, and the engineering teams will always be responding to ad hoc requests from
              the business, instead of thinking about modeling data for the holistic needs of the enterprise.</p>
          <p>Lately, I’ve been describing this to folks within the industry as “The UI for the Modern Data Stack.” It’s
              something we at Rasgo are heavily committed to perfecting, and we believe that, without this, true data
              driven decision making will never fully be realized.</p>
          <p>Really appreciate you reading!! Super exciting times to be in the data world and we look forward to what’s
              ahead :)</p>
      </div>
  </div>

ELT for the Data Consumer


      <div data-qlinks="content">
      <p>I&rsquo;ve come across multiple conversations and opinions about data contracts lately and have been thinking about how they would apply to the current data team&rsquo;s realities.</p>
      <p>The idea is that data contracts can help us prevent data quality issues by formalizing interactions and handovers between different systems (and teams) handling data.</p>
      <p>&zwj;</p>
      <p>Tristan&nbsp;<a class="mds-text-primary" href="https://roundup.getdbt.com/p/interfaces-and-breaking-stuff">suggested</a>&nbsp;a powerful analogy with the software world:</p>
      <blockquote><em>&ldquo;This is how our world works today in the MDS: for example, Fivetran loads data into a table, dbt reads data from a table. If Fivetran changes the schema of that table, it can easily break the dbt code reading from that table. No contracts, no interfaces, no guarantees. [...]<br /><br />This works very differently in a well-architected software system. The interfaces between systems are well-defined, and when you change something that breaks something else you can actually see that error at compile-time rather than at runtime. [...]<br /><br />The importance of catching errors at compile-time cannot be understated&mdash;knowing that something is broken when you are writing the code is exactly when you need that information! Finding out that something is broken once it&rsquo;s been pushed to production causes havoc.&rdquo;</em></blockquote>
      <h3>Why data contracts and why now?</h3>
      <p>There are three powerful trends that force us look for solutions for breaking data:</p>
      <p>&zwj;</p>
      <p><strong>Analytical data becomes increasingly complex</strong></p>
      <p>An average Datafold customer (mid-size modern data stack user) has over 15K tables and ~500K columns in their warehouse. At the top end, we see data warehouses with 30M+ tables and over 1B+ columns. At this scale,&nbsp;<em>metadata becomes big data</em>.</p>
      <p>Data undergoes multiple steps of sophisticated transformations with domain-specific business logic. For example, Gitlab&rsquo;s&nbsp;<a class="mds-text-primary" href="https://gitlab.com/gitlab-data/analytics">internal analytics repo</a>&nbsp;has over 132K lines of SQL code.</p>
      <p>This means when something breaks, figuring out what the origin is becomes a huge task.</p>
      <p>&zwj;</p>
      <h4><strong>Analytical data becomes software</strong></h4>
      <p>It is increasingly used for business automation by feeding into software that runs the business:</p>
      <ul role="list">
      <li>Online machine learning for search, recommendations, fraud detection, etc.</li>
      <li>User-facing analytics</li>
      <li>Data activation in business tools (e.g. Salesforce)</li>
      </ul>
      <p>This means errors don't just impact internal dashboards, they can have real financial consequences. We should not let&nbsp;<a class="mds-text-primary" href="https://www.datafold.com/blog/the-day-you-stopped-breaking-your-data">data break in production</a>&nbsp;anymore.</p>
      <p>&zwj;</p>
      <h4><strong>Analytical data becomes more realtime</strong></h4>
      <p>As a byproduct of becoming an input to software that runs businesses (and not just human decisions as in BI), analytical data demands lower latency &ndash; in other words, data needs become more and more realtime. Reducing the time from data collection to acting based on data leaves little time to identify and react to bugs and makes data quality issues far more costly.</p>
      <p>With such trends at play, we can&rsquo;t rely on humans to ensure data quality, nor can we afford to find out about errors in production, even if those are detected fast. Data bugs are mostly&nbsp;<a class="mds-text-primary" href="https://www.datafold.com/blog/the-day-you-stopped-breaking-your-data?exp_id=1#s1">software bugs introduced</a>&nbsp;by people, and data contracts, although designed to be consumed by machines, help prevent human mistakes.</p>
      <h3>What is a data contract?</h3>
      <p>Data contract formally describes a programmatic interface to data. Wait, Gleb, isn&rsquo;t sending SQL to a warehouse a programmatic access? It is, but this kind of query lacks the formal interface and any guarantees. If I run a SELECT * FROM MYTABLE today and tomorrow, how can I be sure I get the same structure of the result back and that no one has modified the schema of the table or the definitions of its columns?</p>
      <p>To provide another analogy, data contracts are what API is for the web services. Say we want to get data from Twitter. One way is to scrape it by downloading and parsing the HTML of Twitter&rsquo;s webpage. This may work, but our scraper will likely break occasionally, if Twitter, for instance, changes a name of a CSS class or HTML structure. There is no contract between Twitter&rsquo;s web page and our scraper. However, if we access the same data via&nbsp;<a class="mds-text-primary" href="https://developer.twitter.com/en/docs/twitter-api">Twitter&rsquo;s API</a>, we know exactly the structure of the response we&rsquo;re going to get. An API has required inputs, predictable outputs, error codes, SLAs (service level agreements &ndash; e.g. uptime), and terms of use, and other important properties. Importantly, API is also versioned which helps ensure that changes to the API won&rsquo;t break end user&rsquo;s applications, and to take advantage of those changes users would graciously migrate to the new version.</p>
      <p>&zwj;</p>
      <p>So what an actual data contract may look like? This is a toy example of a data contract for a table containing temperature readings in a data warehouse:</p>
      <p>&zwj;</p>
      <figure>
      <div><img style="width: 100%" src="https://assets.website-files.com/60f894ce49eed78a623e58c8/6316a724c9476ca7ee260ea2_KlIlzsWItYOGRV-DQALpy3PmA85ok_hrNf4_ymXfL_sm9JzOyPJXgj8sjVsBRoC6JLAOExqFZLncS4COlevoszs8tlcWCb2PpeWauKHU_jQRkKPMMP-9Hl8yzesBJbDjSCQ2-M_U6WFayFekA5vFNYk.png" alt="" /></div>
      </figure>
      <p>&zwj;</p>
      <p>This data contract is essentially a structured description of a table that can be published somewhere and consumed programmatically. What can we do with it?</p>
      <h3>Data contracts use cases:</h3>
      <p>&zwj;</p>
      <p><strong>Validate data in production</strong></p>
      <p>In production, we can use contracts to validate data on the fly at the source: if the table contract says that the temperature is always in the [-100;60] range, we can have this checked as soon as the table is refreshed. A BI dashboard application reading this table doesn&rsquo;t need to worry about messed up temperature since the contract sets the expectation.</p>
      <p>Same for update time: if SLAs are defined in the contract, data consumers know when can they expect the data, and data producers will be notified if the table is late so they can take action.</p>
      <p>&zwj;</p>
      <p><strong>Prevent breaking changes in development</strong></p>
      <p>In development, data contracts help us prevent breaking changes by validating the new versions of data against the contract. For example, we can have a CI (e.g. Github Actions) check the schema of the table against the contract and raise an error if someone attempts to modify the schema without updating the contract.</p>
      <p>Wait, you may think, but what if someone updates the table schema by modifying the contract, how would that help from breaking downstream data uses? Just like a compiler for a statically typed programming language (or a sophisticated IDE for any language) would check the dependencies between function and variable uses and their definitions, the system that orchestrates our data pipelines would check required inputs of downstream tasks against the interfaces of their upstream dependencies. E.g. Dagster, a competitor to Airflow, allows to&nbsp;<a class="mds-text-primary" href="https://docs.dagster.io/concepts/types">define inputs and outputs of any task</a>&nbsp;so that they can be verified before those actually run data &ndash; addressing one of Airflow&rsquo;s chronic pain points.</p>
      <p>&zwj;</p>
      <p><strong>Improve discoverability and data understanding</strong></p>
      <p>Interestingly, data catalogs (they like to call themselves metadata platforms these days), attempt to aggregate and present similar types of information about data assets as data contracts without formalizing them. The primary difference between having data contracts in place vs. a catalog is that catalog is descriptive whereas contracts are both&nbsp;<em>descriptive</em>&nbsp;and&nbsp;<em>prescriptive&nbsp;</em>i.e. they define not just how the data looks, but how it must look.&nbsp;</p>
      <p>Catalogs are made for humans, and contracts are made primarily for machines (so that they can do the hard work of validation for humans). However, the metadata from data contracts is an excellent source of information for a data catalog to help people discover and understand the data better.</p>
      <h2>Data contracts in modern data stack</h2>
      <p>We have seen elements of that in particular layers of the data stack:&nbsp;<a class="mds-text-primary" href="https://iterative.ly/">Iteratively</a>&nbsp;and&nbsp;<a class="mds-text-primary" href="https://www.avo.app/">Avo</a>&nbsp;make it easy to&nbsp;<a class="mds-text-primary" href="https://www.avo.app/docs/">define, evolve, and validate event schemas</a>&nbsp;(collaborating on event schemas in spreadsheets felt too much even for Excel lovers like me). Dagster brings defined inputs and outputs to data orchestration.&nbsp;</p>
      <p>However, these solutions each work in particular layers of the data stack which means they offer only partial solutions to the problem of end-to-end data reliability. The weak link in the data chain is almost always the interface between different technologies or products.&nbsp;</p>
      <p>Let&rsquo;s imagine what an end-to-end data contract may look like if defined through the entire stack: if our event collection tool was aware that the &ldquo;promo_code&rdquo; field in the &ldquo;user_signup&rdquo; event is used in dbt model A which gets pulled to dbt models B and C and eventually into a Hightouch sync to a Mailchimp campaign (while also being renamed to &ldquo;discount_id&rdquo; in somewhere in the middle) &ndash; then it could show a warning to a software engineer working on the Identity microservice that fires the &ldquo;user_signup&rdquo; event right in IDE.</p>
      <h3>Contracts across the stack?</h3>
      <p>At the moment it seems unlikely that modern data stack vendors, who mostly speak to each other in SQL, API calls or pointers to database tables, would adopt a unified data contract interface. Unless emerging platform frameworks such as dbt will establish and&nbsp;<a class="mds-text-primary" href="https://benn.substack.com/p/the-data-os">force such upon everyone</a>.</p>
      <p>Widespread, vertical implementations of data contracts seem to only exist in FAANG-level fully integrated data platforms such as&nbsp;<a class="mds-text-primary" href="https://medium.com/airbnb-engineering/data-quality-at-airbnb-870d03080469">Airbnb</a>, which employ internal data tools teams larger than most modern data stack vendors do, as well as in some unicorns such as&nbsp;<a class="mds-text-primary" href="https://www.datafold.com/blog/data-quality-meetup-6#s2">Convoy, who built Chassis</a>&nbsp;(<a class="mds-text-primary" href="https://github.com/ananthdurai/schemata">Schemata</a>&nbsp;is a philosophically similar nascent open source project). It&rsquo;s unlikely that an average data team would be able to build something like that internally.</p>
      <p>The idea of data contracts throughout the stack is extremely powerful but remains largely aspirational for most data teams: Modern Data Stack has no signs of converging into a unified cross-tool contract interface, and outside of MDS everything is possible but most of us are not Uber.</p>
      <p>I believe over the next year, data contracts will become increasingly wide spread. Most promising frameworks will be open-source built by teams at/from big data platforms. The adoption of data contracts will start with most business-critical and time-sensitive data, and most likely close to the source (events). And if you are curious how this may unfold, Chris Riccomini&nbsp;<a class="mds-text-primary" href="https://www.dataengineeringpodcast.com/automation-for-data-engineers-with-chris-riccomini-episode-305/">draws interesting parallels</a>&nbsp;from the distributed software engineering world.</p>
      <p>&zwj;</p>
      <figure>
      <div><img style="width: 100%" src="https://assets.website-files.com/60f894ce49eed78a623e58c8/6317b798a759159c30afc5ef_Data%20Contracts%202.png" alt="" /></div>
      </figure>
      <h3>Handshakes before contracts</h3>
      <p>If having data contracts everywhere is not attainable anytime soon, what can we do to prevent data from breaking?</p>
      <p>We can use data handshakes. In business, handshake agreements are used to agree on key terms in a deal before formalizing it in a scrupulous legal detail. We can apply the same concept to data.</p>
      <p>If Silicon Valley&nbsp;<a class="mds-text-primary" href="https://www.ycombinator.com/handshake#:~:text=Silicon%20Valley%20runs%20on%20handshake,signed%20and%20money%20changes%20hands.">runs on handshake deals</a>, data teams can too.&nbsp;</p>
      <h3>So what&rsquo;s a handshake for data?</h3>
      <p>It&rsquo;s a pull (or merge, okay, okay, Gitlab fans) request.</p>
      <p>Pull request is a critical step in software development process where someone who wants to introduce a change effectively follows a handshake protocol:</p>
      <ol role="list">
      <li>Here is my change (description)</li>
      <li>Here&rsquo;s is the code it modifies (git diff)</li>
      <li>It doesn&rsquo;t break anything (tests pass)</li>
      <li>I would like you, owner/contributor/stakeholder, to review it</li>
      <li>Once you give +1 it, I will merge it and it will go into production</li>
      </ol>
      <p>Within the pull request process, once can validate most of things that data contracts would do:</p>
      <ol role="list">
      <li>Will the schema change and if so, how? (schema diff)</li>
      <li>Will the data change and if so, how? (data diff)</li>
      <li>What impact will this change have on downstream data applications? (data lineage)</li>
      </ol>
      <p>&zwj;</p>
      <p>If every part of the data processing pipeline is version-controlled</p>
      <p>If one can easily know the impact of every change (<a class="mds-text-primary" href="https://www.datafold.com/blog/the-day-you-stopped-breaking-your-data?exp_id=1#s2">Know What You Ship principle</a>)</p>
      <p>If every change is staged and reviewed as a pull request before getting into production</p>
      <p>&gt;&gt; Then you can achieve data reliability without implementing data contracts throughout the entire data platform!</p>
      </div>

The Best Data Contract is the Pull Request

<p>This is a long due post, predominantly because there is a lot of confusion around data lineage , data observability and the interdependencies.</p>
      <p>As many of us know, Data Lineage is one of the most discussed topics today and so is data observability. This article covers the applicability of data lineage in Data Observability.</p>
      <p>PS: If you are specifically looking to understand lineage better, these are few of my favorites on this topic.</p>
      <p>Building Data Lineage: <a class="mds-text-primary" href="https://medium.com/netflix-techblog/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977">netflix</a> &amp; Leveraging Apache spark for Data Lineage : <a class="mds-text-primary" href="https://engineeringblog.yelp.com/2022/08/spark-data-lineage.html">Yelp</a></p>
      <h3><strong>What is Data Lineage?</strong></h3>
      <p>Data lineage tracks the changes and transformations impacting any data record. Data lineage tells us where data is coming from, where it is going, and what happens as it flows from data sources and pipeline workflows to downstream data marts and dashboards.</p>
      <p>In short, it enables better data governance by giving you a more complete, top-down picture of your data and analytics ecosystem.</p>
      <h3><strong>Why is it so important for data quality?</strong></h3>
      <p>Data lineage is essential for understanding the health of the data because it ensures transparency around the source of data, ownership, access, and its transformation. These are excellent indicators of the reliability and trust of the data. However unlike other data quality indicators like <strong>Completeness, Validity, Accuracy, Consistency, Uniqueness, and Timeliness</strong>.(Read <a class="mds-text-primary" href="https://docs.telm.ai/academy/">here</a>) Data lineage can not be metricized.</p>
      <p>The insights provided by Lineage enable data users to solve all kinds of problems encountered within mass quantities of interconnected data like Data troubleshooting, Impact analysis, Discovery and trust, Data privacy regulation (GDPR and PII mapping), and Data asset cleanup/technology migration, and Data valuation.</p>
      <h3><strong>What is Data Observability?</strong></h3>
      <p>Data Observability is a set of measures that can help predict, identify, and resolve the data issues. This is often done by leveraging statistical analysis and machine learning.</p>
      <p>Data Observability aims to reduce the mean time to detect (MTTD) and mean time to resolve (MTTR) data issues.</p>
      <h3><strong>Why is Lineage important for data Observability?</strong></h3>
      <p>I MG SRC : Telm.ai</p>
      <p>Data lineage can be very effective in getting good Observability outcomes.The insights provided by Lineage enable data users to quickly find the root cause of issues , conduct impact analysis and also invoke a remediation.</p>
      <p>So your data observability should leverage lineage information.</p>
      <h3><strong>How is data lineage used in Data Observability?</strong></h3>
      <p>Today, most data observability companies leverage Data Lineage for mean time to resolve (MTTR) issues by doing two things,</p>
      <p><strong>Root cause analysis (RCA)</strong></p>
      <p>Once a monitoring system detects an issue, Lineage can help investigate the previous steps in the pipeline and changes. Specifically, if you are using Datawarehouse monitoring tools like Montecarlo data, Metaplane etc you can find table/view lineage that can help shrink the root cause analysis time.</p>
      <p><strong>Downstream Impact Analysis</strong></p>
      <p>Often Data Lineage can help find downstream impact and invoke remediation flow. Remediation by definition is an reactive approach and can become a nightmare.</p>
      <p>If you have a datamesh architecture, and have multiple consuming products using the data, leveraging lineage helps you identify which products are impacted and who own those products/systems. Now you can systematically notify those users. Another use case is using Lineage where you can identify the impacted dashboards.</p>
      <p>The above methods are <strong>definitely useful</strong> but still suboptimal for data observability.</p>
      <p>Data Lineage for RCA and Impact analysis</p>
      <p>&zwj;</p>
      <h3><strong>What is the correct usage of Lineage in Observability?</strong></h3>
      <p>Data Observability, by definition, should be <strong>proactive</strong>. Data engineers should use <strong>Lineage to check downstream impact</strong> before adding any changes. For Example, am I making schema changes? Let me automatically check the downstream implications.</p>
      <p>For data in motion a full pipeline monitoring approach will not only help with (resolve) MTTR but also MTTD(detect) issues. Design a metric monitoring system that monitors every step in the data pipeline. So users get alerted when there is a drift or outlier in a specific step of the pipeline, and multiple data points will help detect issues even before they have downstream impact.</p>
      <p>Example : Get alerted that the CRM system got partial data at 2pm PST.Such an alert can automatically be sent to the CRM system owner, so data engineers don't have to find out about these issues inside Snowflake or BigQuery and reverse track them back to the CRM system. Often this approach will also help orchestrate the date pipeline flow refer: <a class="mds-text-primary" href="https://www.telm.ai/blog/who-has-control-of-your-data-pipeline">Circuit breaker pattern</a>.&nbsp;</p>
      <p>A good analogy would be monitoring vs. tracing and logs. If you have a lot of monitoring, you may not need tracing and log review because tracking at every level will highlight the same problems but sooner. This is because all methods highlight the same issues just from different angles. So I would say that Lineage is a good tool for root cause analysis and impact analysis (MTTR) but your observability tools should focus higher on for mean time to detect (MTTD) issues i.e be more proactive.</p>
      <h3><strong>Summary</strong></h3>
      <p>Data Lineage is a crucial aspect of Data reliability. However, to effectively reduce both the MTTD and MTTR data issues :</p>
      <p>1: Leverage lineage to understand the downstream impact <strong>before</strong> making changes.</p>
      <p>2: Monitor important data metrics every step of the pipeline and leverage Lineage to identify exactly where(step of the pipeline) the which metric has drifted.</p>
      <p>This approach enables users to reduce both time to detect and time to resolve data issues.</p>
      <p>&zwj;</p>
      

Modern Data Stack - Sitemap