Things I Wish I Knew When I Was Building a Data Team

<p>Let&rsquo;s not mince words. Product led growth (PLG) isn&rsquo;t something that happens overnight. It has to infuse company culture and involves commitment from every team - not just the go-to-market teams on the front lines.&nbsp;</p>
      <p>PLG introduces a lot of new data about how customers are using the product that GTM teams can leverage, but this data presents new challenges to data teams who are tasked with figuring out how to model data in a scalable and reliable way. It doesn&rsquo;t help that data leaders are being constantly bombarded with one-off analytics requests.&nbsp;</p>
      
      <p>Although the situation sounds dire, enabling PLG data will pay dividends. Revenue teams at product led companies need to leverage product data in order to run the most effective, scalable go-to-market (GTM) motions.&nbsp;</p>
      <ul role="list">
      <li>Sales teams can use product data to identify the best customers to connect with so that they are spending time on the customers most likely to buy.&nbsp;</li>
      <li>Marketing teams can use product data to refine nurture campaigns to provide a more personalized, relevant brand interaction with users.&nbsp;</li>
      <li>Customer Success teams can use product data to identify superstar customers or proactively reach out to customers who are at risk to churn.&nbsp;</li>
      </ul>
      
      <p>We&rsquo;ve spoken to hundreds of companies implementing PLG or looking to transition to PLG, and throughout all these conversations, we&rsquo;ve identified 5 common obstacles data teams need to address when it comes to enabling GTM teams with product data. In this blog post, we&rsquo;ll elaborate on these 5 challenges and provide suggestions on how to address them.</p>
      
      <h3 id="toc-0">The 5 common challenges for PLG data teams</h3>
      <ul role="list">
      <li>Data is generated by multiple teams across your company</li>
      <li>Data isn't owned by the stakeholders who use it (i.e. your revenue team)</li>
      <li>Data is surfaced in platforms that teams don&rsquo;t use on a daily basis</li>
      <li>Data that is surfaced in downstream platforms is overwhelming in scope</li>
      <li>Data that is converted into PQLs or Scores become black boxes that are rarely actionable</li>
      </ul>
      
      <h3 id="toc-1">Data is generated by multiple teams across your company</h3>
      <p>Every team across your company is generating data about customers on a daily basis.&nbsp;</p>
      <ul role="list">
      <li>Sales teams are manually inputting data about opportunities in their CRMs, tracking activity, or automatically enriching accounts and contacts with tools like Clearbit and ZoomInfo.&nbsp;</li>
      <li>Marketing teams are automatically adding leads into CRMs and tracking marketing activity.&nbsp;</li>
      <li>Product teams are tracking product usage data in data warehouses.&nbsp;</li>
      <li>Customer Success teams are working with customers through tools like Intercom.&nbsp;</li>
      </ul>
      <p>The data inputs are endless and are not uniformly collected, which means that everything lives in different data silos. It&rsquo;s difficult to stitch all this data together into a single view for all teams to consume.</p>
      <p>Further, when different teams are generating data, they are generating it for their own use cases rather than thinking about how they can scale their data sets across the organization. We often observe this when sales teams get access to Segment data, which is often generated by Marketing or Product. Sales teams don&rsquo;t understand what various button clicks mean, making the barrier to adopting Segment data in sales processes that much higher. This is a challenge we navigated over at Correlated as well! For example, we track the various actions that users automate in our product and build alerts around &ldquo;exceptions&rdquo;. However, &ldquo;exceptions&rdquo; are labeled differently for different actions, so we have to catch all the labels correctly in order to get a complete alert for all &ldquo;exceptions&rdquo;. Obviously, if we threw this raw data at our sales team, they would have absolutely no idea how to use it.</p>
      <h4>Suggestion: Leverage a data warehouse as your source of truth and invest in enabling technologies in your data stack</h4>
      <p>Ultimately, the customers who are most successful in implementing a PLG GTM motion with us are the ones who have a data warehouse. This is because a data warehouse can act as your source of truth for all your customer and user data. It is certainly a challenge to sync all of your data sources with your data warehouse, but it is not an insurmountable one. ETL tools like&nbsp;<a class="mds-text-primary"  href="https://www.fivetran.com/">Fivetran</a>,&nbsp;<a class="mds-text-primary"  href="https://www.stitchdata.com/" target="_blank" rel="noopener">Stitch</a>, and&nbsp;<a class="mds-text-primary"  href="https://airbyte.com/">Airbyte</a>&nbsp;make it possible to do this using a single platform.&nbsp;</p>
      <p>Once all the data is loaded into your data warehouse, you can leverage all the features and functionalities of data warehouses to shape and model your data. We highly recommend&nbsp;<a class="mds-text-primary"  href="https://www.getdbt.com/">dbt</a>&nbsp;as a great option for those looking to scale their data modeling across their organization.</p>
      <p>Leveraging a data warehouse is not without its limitations. It requires deep institutional knowledge and expertise from data teams in order to unblock key business challenges, and of course this becomes a cost center. Investing in tools that allow teams to self-serve their way into pre-built models is a great option, and there are many tools in the market today that can help. For example, Reverse ETL solutions like&nbsp;<a class="mds-text-primary"  href="https://www.getcensus.com/">Census</a>&nbsp;and&nbsp;<a class="mds-text-primary"  href="https://hightouch.io/">Hightouch</a>&nbsp;connect into the models your data teams have built and make it easy for downstream applications to consume that data. This makes it possible for your data teams to scale their efforts to support multiple use cases. BI solutions like&nbsp;<a class="mds-text-primary"  href="https://www.tableau.com/">Tableau</a>,&nbsp;<a class="mds-text-primary"  href="https://www.looker.com/">Looker</a>, and more recently,&nbsp;<a class="mds-text-primary"  href="https://hex.tech/">Hex</a>, provide business users with access to pre-modeled data. Tools like&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/">Correlated</a>&nbsp;sit on top of your data warehouse to enable business users to build automated playbooks.&nbsp;</p>
      
      <h3 id="toc-2">Data isn't owned by the stakeholders who use it</h3>
      <p>One of the biggest challenges with implementing a data warehouse as described above is that you have to figure out how to enable the various personas who need that data. If all you do is dump data into a data warehouse, none of the teams who rely on that data to do their jobs will be able to access it. If left to fester, data teams and GTM teams can ultimately reach a stalemate where data teams have implemented processes that prevent GTM teams from getting what they want. Alternatively, GTM teams could be requesting data team resources all day every day, completely inundating data teams with requests. We&rsquo;ve seen both situations at companies, and it is ugly.</p>
      <h4>Suggestion: Focus on driving alignment between data teams and business goals. Make cross-team collaboration part of every day processes.</h4>
      <p>Companies should view their data teams as a pooled resource that supports the entire company. As a result, data team resources should be allocated and scheduled similarly to how engineering teams schedule and allocate time to work on product roadmap. It&rsquo;s important, however, to not treat the data team as a separate silo where requests are sent (and often added to the backlog). Instead, scheduling and allocation of resources should happen hand in hand with the business units who will benefit from data team resources. This is the model that we&rsquo;ve seen work best in companies with successful PLG GTM motions that leverage data.</p>
      <p>One of the reasons why it&rsquo;s important to intertwine business use cases with data team projects is that we&rsquo;ve rarely seen companies successfully implement a product-led data strategy without doing so. If you send the data team off to &ldquo;solve our usage tracking problems,&rdquo; they&rsquo;ll end up coming up with a solution that might not work with the downstream tools your business teams use, or they&rsquo;ll try to find a solution that works for everything, essentially boiling the ocean on all possibilities. This ends up taking upwards of a year to implement, leaving GTM teams flying blind in the meantime.&nbsp;</p>
      <p>Instead, focus on use cases that you as a business want to accomplish and build out the infrastructure, models, and views that can support those use cases. Leveraging data is a core strategic competency for SaaS businesses, so it&rsquo;s not realistic to expect to never have to touch your existing data models again. This is why the tools you choose to use are so important - although you will have to adjust your data models from time to time, it doesn&rsquo;t have to be painful every single time.</p>
      
      <h3 id="toc-3">Data is surfaced in platforms that revenue teams don&rsquo;t use on a daily basis</h3>
      <p>Typically when companies start scratching the surface on leveraging product data to run a product led growth motion, they&rsquo;ll take an initial approach of driving &ldquo;visibility&rdquo;. This is essentially Looker, Mode, Tableau, or some other business intelligence (BI) dashboard that shows things like how an account is using the product or a list of &ldquo;product qualified leads&rdquo;. This is a worthwhile exercise because it allows your data team to build out the data models that your GTM teams can benefit from.&nbsp;</p>
      <p>However, time and time again, we&rsquo;ve heard feedback that GTM teams simply do not look at dashboards. In the rare cases that they do, they find that dashboards are rarely actionable. For example, let&rsquo;s say that you have a dashboard that lists out PQLs. An AE can&rsquo;t just take an email off that dashboard and email them. That&rsquo;s because (1) they have to figure out if they&rsquo;ve already emailed the user before (check Salesforce/Hubspot), (2) they have to figure out what to say to the user (check another dashboard on product usage), (3) they aren&rsquo;t sure why the user is even a PQL in the first place (Slack data team to ask how a PQL is triggered).</p>
      <h4>Solution: Pipe data into downstream applications.</h4>
      <p>Rather than expecting GTM teams to look at dashboards and analyze the data, send data directly to the downstream applications that they live in on a daily basis. We&rsquo;ll talk about some of the downsides to this approach in the next &ldquo;problem,&rdquo; but some approaches we&rsquo;ve seen companies take is to build very specific views in Salesforce that contain all the information an AE (account executive) or CSM (customer support manager) needs to reach out in an intelligent way. For example, on the Account view, they might include things like: seats sold, seats utilized, number of active users. This allows the revenue team to stay in the tools they use, but still see the data they need to reach out to customers appropriately.</p>
      
      <h3 id="toc-4">Data that is surfaced in revenue team platforms is overwhelming in scope</h3>
      <p>This problem is linked in many ways to the prior problem - when companies discover that their teams are not using dashboards, their initial response is to pipe&nbsp;<em>data</em>&nbsp;into downstream applications. Now, I highlight&nbsp;<em>data</em>&nbsp;here because data is messy, there is a lot of it, and it requires analysis. We&rsquo;ve seen companies pipe so much data into Salesforce that they literally run out of custom fields. As an AE or CSM looking at Salesforce, those views simply become information overload. You end up giving so much information to your GTM teams that they literally cannot do anything with it.</p>
      <h4>Solution: Strive to pipe actionable insights over data.</h4>
      <p>A better approach is to think about the end goal you want to achieve by sending data to downstream applications. Are you trying to surface a lead? Are you trying to suggest a next-best-action? That should determine what data you send and how you send it.&nbsp;</p>
      <p>For example, let&rsquo;s dig into the &ldquo;seats sold&rdquo; and &ldquo;seats utilized&rdquo; example. You could pipe that information into Salesforce and surface that as a custom field on the Account object. However, wouldn&rsquo;t it be better if you created a lead in Salesforce, assigned it to an AE, and added a note that said - this customer is hitting 80% license utilization, add them to the &ldquo;Upsell&rdquo; Outreach sequence? You are sending insights that are immediately actionable.&nbsp;</p>
      <p>Some patterns we&rsquo;ve seen that we really like are: auto-creating Salesforce Leads, auto-creating Salesforce Opportunities, adding users to Outreach or Salesloft Sequences / Cadences, adding users to lists in Hubspot. The opportunities are endless. By thinking about insights and expected action over just data, you can ensure that all your efforts to build out a product-led data strategy are actually adopted by the GTM teams who have to put them into practice.</p>
      
      <h3 id="toc-5">Data that is converted into PQL Scores becomes a black box</h3>
      <p>Final topic: PQL Scoring. People really love PQL Scores recently, but it&rsquo;s something that you should definitely approach with care. In some ways, PQL Scores are the ultimate effort to convert data into insights. E.g. I crunched all the data for you, and this customer is great to reach out to. The problem is that GTM teams need to know how to reach out, which means they have to understand the &ldquo;<em>why&rdquo;</em>&nbsp;behind PQL Scores. If they don&rsquo;t understand the why, they are not going to adopt the score. You&rsquo;ll spend 6-12 months building out a score that ultimately no one uses.&nbsp;</p>
      <p>This is not to say scores are not useful - they certainly are, particularly when you&rsquo;re trying to get an overall directional feel of if someone is worth reaching out to. However, they should only be used as one piece of the puzzle and should not be viewed as a solution to PLG.</p>
      <h4>Solution: Think about the playbooks you want to enable after receiving an insight</h4>
      <p>There are a lot of vendors out there claiming to surface leads &ldquo;magically&rdquo; and with &ldquo;machine learning&rdquo;. We ourselves run machine learning models to help our customers identify which Signals might be interesting to go after, so it&rsquo;s a viable strategy and can surface some interesting insights. However, again, insights are more valuable if they drive a downstream action.&nbsp;</p>
      <p>If you&rsquo;re thinking of using a score, we&rsquo;d suggest tying it to a playbook you want to execute. For example, create an &ldquo;Upsell&rdquo; score, and align it with an &ldquo;Upsell&rdquo; playbook. Create an &ldquo;Onboarding Self-Serve&rdquo; score, and align it with an appropriate playbook. If you simply use one score, your results won&rsquo;t be as good as focusing on personalized, actionable playbooks that drive real results.&nbsp;</p>
      
      <h3 id="toc-6">Time to dive in!</h3>
      <p>We&rsquo;ve explored a lot of surface area in this blog post, from data warehouses to delivering that data in actionable ways to GTM teams. This may sound like a lot, but we&rsquo;ve seen a lot of companies get to success by taking things one step at a time. What we&rsquo;ve seen work best is actually a more vertical approach. Rather than trying to solve problem 1, then moving onto problem 2, etc etc, pick one important business challenge that aligns with a top-line goal and solve all the problems vertically.</p>
      <p>For example, let&rsquo;s say that your business wants to drive self-serve conversion. Next, identify the &ldquo;actions&rdquo; you want your GTM teams to take, the &ldquo;data&rdquo; you need to make that possible, and the &ldquo;tools&rdquo; that you&rsquo;ll need to tie everything together. Finally, go forth and do it! By aligning data teams with actionable business outcomes, you&rsquo;ll be able to get the backing and support you need across teams to get things done.&nbsp;</p>
      <p>If you'd like to learn how Correlated can assist your product-led journey,&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/get-started">get started</a>&nbsp;for free or&nbsp;<a class="mds-text-primary"  href="https://www.getcorrelated.com/demo-request">schedule a call</a>&nbsp;with our team.</p>

5 challenges Data Teams face supporting PLG revenue teams and how to address them

<div>&nbsp;</div>
   <p>Active metadata is like hot gossip. Here&rsquo;s why.</p>
   <p id="4e49">
     Just like data mesh or the metrics layer,&nbsp;<em class="mj"
       >active metadata</em
     >
     is the latest hot topic in the data world. As with every other new concept
     that gains popularity in the data stack, there&rsquo;s been a sudden explosion
     of vendors rebranding to &ldquo;active metadata&rdquo;, ads following you
     everywhere, and&hellip; confusion.
   </p>
   <p id="22a1">
     With everyone talking about active metadata, it must be pretty easy to
     understand, right?
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go mk">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*oj53dlEWig_Hcxmtj4GKJA.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go ml">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*bF3m6G6XM4D4UrYgMD8NsQ.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption></figcaption>
   </figure>
   <p id="5481">
     Apparently not! I&rsquo;ve been talking about active metadata for over a year
     now, but I still see questions like these all the time.
   </p>
   <p id="c0f1">
     Active metadata can sound a bit scary, but it doesn&rsquo;t have to be. It is
     a must-have tool in the modern data toolbox, so if you&rsquo;re still
     wondering what it means, this article is for you.
   </p>
   <p id="76b6">
     I&rsquo;ve broken down the ideas behind active metadata with as little jargon
     as possible. Keep reading to learn what active metadata is, what it looks
     like, how you can actually use it, how it fits into the modern data stack, and
     why it even matters.
   </p>
   <h3 id="6b2d">What is active metadata?</h3>
   <p id="569c">
     I could start dropping some jargon here, but then both you and I will be
     asleep in seconds. So let&rsquo;s jump into an analogy instead.
   </p>
   <p id="4dd5">
     Imagine that you got your hands on the juiciest piece of tech gossip &mdash;
     Apple is expanding into recreational marijuana to literally help people
     &ldquo;think different&rdquo;.
   </p>
   <p id="aee1">
     There&rsquo;s no way you&rsquo;re going to keep something this exciting a
     secret. The world has to know. So you post it on your blog,&nbsp;<em
       class="mj"
       >blogspot.applefansunite.com</em
     >. All done, right?
   </p>
   <p id="b968">
     Just like&nbsp;<a
       class="mds-text-primary"
       href="https://www.vice.com/en/article/n7n74d/traffic-jams-are-possible-in-elon-musks-tunnels-apparently"
       target="_blank"
       rel="noopener ugc nofollow"
       >a car in the Hyperloop tunnel</a
     >, we all know that&rsquo;s not going anywhere. You can&rsquo;t just put the
     story somewhere and hope people will find it. You have to actually deliver it
     into people&rsquo;s hands.
   </p>
   <p id="47ef">
     You sharpen your PR chops, blast the news to tech reporters and news sites,
     and lo and behold it&rsquo;s&nbsp;<em class="mj">everywhere</em> in no time.
     It&rsquo;s already been mummified, and your grandfather just asked why apple
     farmers are talking about this Molly girl on your group chat.
   </p>
   <blockquote class="nj">
     <p id="988d" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt">Metadata is like this information. If it sits&nbsp;</em
       >passively<em class="nt"
         >&nbsp;in its own little world, with no one seeing or sharing it, does it
         even matter? But if it&nbsp;</em
       >actively<em class="nt"
         >&nbsp;moves to the places where people already are, it becomes part of
         and adds context to a larger conversation.</em
       >
     </p>
   </blockquote>
   <h3 id="36ed">What is the difference between active and passive metadata?</h3>
   <p id="e3db">
     <strong>Passive metadata</strong>&nbsp;is the standard way of aggregating and
     storing metadata into a static data catalog. This usually covers basic
     technical metadata &mdash; schemas, data types, models, etc.
   </p>
   <p id="4780">
     Think of passive metadata as putting out information on a personal blog. Every
     so often, it&rsquo;ll get picked up and go viral on Hacker News. But most of
     the time it&rsquo;s just going to sit unseen and unused, even when people
     actually need to know it.
   </p>
   <p id="c214">
     <strong>Active metadata</strong>&nbsp;makes it possible for metadata to flow
     effortlessly and quickly across the entire data stack, embedding enriched
     context and information in every tool in the data stack. It is usually more
     complex than passive metadata, covering operational, business, and social
     metadata along with basic technical information.
   </p>
   <p id="16a6">
     Think of active metadata as a viral story. It shows up everywhere you already
     live in what seems like seconds. It&rsquo;s immediately cross-checked against
     and combined with other information, bringing together a network of related
     context into a larger trend or story. And it sparks conversations, making
     everyone more knowledgable and informed in the end.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go nx">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*h71nNqp0b03WgO_XI2o8fw.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt">The four main types of metadata. (Image by author.)</em>
     </figcaption>
   </figure>
   <h3 id="7541">Why does active metadata matter?</h3>
   <blockquote class="nj">
     <p id="302a" class="nk nl ji bn nm nn no np nq nr ns mi co">
       <em class="nt"
         >To put it simply, no one wants to go to another website to &ldquo;browse
         the metadata&rdquo;.</em
       >
     </p>
   </blockquote>
   <p id="ca3d">
     As we embraced the internet and data exploded in the early aughts, companies
     realized they needed to manage all their new data.
   </p>
   <p id="091a">
     We entered a golden age of metadata management. New companies like&nbsp;<a
       class="mds-text-primary"
       href="https://www.informatica.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Informatica</a
     >,&nbsp;<a
       class="mds-text-primary"
       href="https://www.collibra.com/us/en"
       target="_blank"
       rel="noopener ugc nofollow"
       >Collibra</a
     >, and&nbsp;<a
       class="mds-text-primary"
       href="https://www.alation.com/"
       target="_blank"
       rel="noopener ugc nofollow"
       >Alation</a
     >&nbsp;were created, and they hyped the importance of data catalogs. People
     needed a way to sort through all their options, so we got reports like
     Gartner&rsquo;s&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/3993025"
       target="_blank"
       rel="noopener ugc nofollow"
       >Magic Quadrant for Metadata Management</a
     >. Billion-dollar companies emerged, and companies spent hundreds of millions
     of dollars on metadata management.
   </p>
   <p id="41e1">
     Yet just last year, Gartner released their&nbsp;<a
       class="mds-text-primary"
       href="https://www.gartner.com/en/documents/4006759"
       target="_blank"
       rel="noopener ugc nofollow"
       >Market Guide for Active Metadata</a
     >&nbsp;and declared that &ldquo;Traditional metadata practices are
     insufficient&hellip;&rdquo;
   </p>
   <p id="5d60">
     That&rsquo;s because passive data catalogs solve the &ldquo;too many
     tools&rdquo; problem by adding&hellip; another tool. They aggregate metadata
     from different parts of the data stack, and it stagnates there. User adoption
     suffers, and&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-future-of-data-catalogs-868ffbe7354d"
       target="_blank"
       rel="noopener"
       >these exciting tools turn into expensive shelfware</a
     >.
   </p>
   <p id="2761">
     Active metadata sends metadata back into every tool in the data stack, giving
     the humans of data context wherever and whenever they need it &mdash; inside
     the BI tool as they wonder what a metric actually means, inside Slack when
     someone sends the link to a data asset, inside the query editor as try to find
     the right column, and inside Jira as they create tickets for data engineers or
     analysts.
   </p>
   <h3 id="050b">How does active metadata fit into the modern data stack?</h3>
   <p id="68c9">
     Active metadata functions as a layer on top of the modern data stack.
   </p>
   <p id="bdd2">
     It leverages open APIs to connect all the tools in your data stack and ferry
     metadata back and forth in a two-way stream. This is what allows active
     metadata to bring context, say, from Snowflake into Looker, Looker into Slack,
     Slack into Jira, and Jira back into Snowflake.
   </p>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go od">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*53oiNCxKcmY9bLFNCKNnfg.png"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Active metadata in the modern data stack. (Image by author.)</em
       >
     </figcaption>
   </figure>
   <h3 id="dbd9">4 characteristics of active metadata</h3>
   <p id="e3b1">
     According to&nbsp;<a
       class="mds-text-primary"
       href="https://towardsdatascience.com/the-gartner-magic-quadrant-for-metadata-management-was-just-scrapped-d84b2543f989"
       target="_blank"
       rel="noopener"
       >Gartner&rsquo;s new Market Guide for Active Metadata</a
     >, active metadata is an always-on, intelligence-driven, action-oriented,
     API-driven system, the opposite of its passive, static predecessor.
   </p>
   <p id="0837">
     This can be broken down into the four key characteristics of active metadata.
   </p>
   <ul>
     <li id="4667">
       <strong>Always on</strong>: Active metadata is always on. Rather than
       waiting for people to manually enter or parse metadata, this means
       continually collecting metadata at every stage of the modern data stack
       &mdash; logs, query history, usage statistics, and more.
     </li>
     <li id="fd88">
       <strong>Intelligent</strong>: Active metadata isn&rsquo;t just about
       collecting metadata. It&rsquo;s about constantly processing metadata to
       connect the dots and create intelligence from it. This means that with
       active metadata, the system will only get smarter over time as people use it
       more and it observes more metadata.
     </li>
     <li id="c801">
       <strong>Action-oriented</strong>: Active metadata doesn&rsquo;t just stop at
       intelligence. It should drive action by curating recommendations, generating
       alerts, and making it easier for people to make decisions &mdash; or even
       automatically making decisions without human intervention, like stopping
       downstream pipelines when data quality issues are detected.
     </li>
     <li id="3a52">
       <strong>Open by default</strong>: Active metadata platforms use APIs to hook
       into every piece of the modern data stack. This makes magical user
       experiences possible by saving data practitioners from the endless tool- and
       context-switching. This is called embedded collaboration, which is when work
       happens where you are with the least amount of effort.
     </li>
   </ul>
   <h3 id="a46d">5 use cases of active metadata</h3>
   <p id="b89a">
     There are dozens, if not hundreds, of use cases of active metadata. (Enough
     for several articles of their own &mdash; coming soon!) Let&rsquo;s go through
     a few of my favorites.
   </p>
   <ul>
     <li id="e13d">
       <strong>Purge stale or unused assets</strong>: Use active metadata to
       periodically calculate when each asset (e.g. a data table, dashboard, etc)
       was last used and/or how many people used it. If it was used within the last
       30 days, great! If an asset hasn&rsquo;t been used in the last 60 days,
       automatically archive it. If no one has touched it in the last 90 or 120
       days, purge it entirely.
     </li>
     <li id="a52a">
       <strong>Allocate compute resources dynamically</strong>: Imagine that 90% of
       users log in to a BI tool during the last week of a financial quarter.
       Active metadata can be used to automatically scale up compute resources just
       before that week and scale them down again afterward.
     </li>
     <li id="34bd">
       <strong>Enrich user experience in BI tools</strong>: Instead of switching
       between a BI tool and data catalog, use active metadata to bring context
       into dashboards. Relevant metadata (like business terms, descriptions,
       owners, and lineage) can be pushed into the BI tool. Then when someone is
       looking at each table, they can understand who owns it, where the data came
       from, etc. This information could even be used as labels in auto-generated
       reports.
     </li>
   </ul>
   <figure>
     <div tabindex="0" role="button">
       <div class="gn go os">
         <img
           role="presentation"
           src="https://miro.medium.com/max/1400/1*aLuvXmyV2tyIr1YKbIbNZA.gif"
           alt=""
           style="width: 100%"
         />
       </div>
     </div>
     <figcaption>
       <em class="nt"
         >Enriching BI tools with active metadata. (Image from author.)</em
       >
     </figcaption>
   </figure>
   <ul>
     <li id="2ef1">
       <strong>Identify popular assets</strong>: Use active metadata to create a
       custom relevance score for each asset. This can be based on usage
       information from places like query logs, lineage, and BI dashboards. Then
       the most popular, relevant assets should be surfaced more frequently in
       search and checked more frequently for data quality issues.
     </li>
     <li id="d9f9">
       <strong>Notify downstream consumers</strong>: It&rsquo;s awful if the CEO
       ends up seeing a broken dashboard before the data team. Active metadata can
       be used to check for issues when a data store changes and notify downstream
       data users about potential issues. For example, when a data store is
       crawled, the new metadata would be compared against previous metadata. If
       there are any potential breaking changes (e.g. the addition or removal of a
       column), lineage could be used to find who owns this data store and notify
       them in Slack, Jira, email, etc.
     </li>
   </ul>
   <h3 id="aac4">The future of active metadata</h3>
   <p>
     As metadata becomes big data and big data becomes a behemoth, active metadata
     isn&rsquo;t just a wonderful dream. It&rsquo;s a necessity &mdash; the only
     way to understand today&rsquo;s data.
   </p>
   <p id="cb3d">
     Managing, processing, and analyzing metadata is the new normal for modern data
     teams. Doing this passively and manually, though, isn&rsquo;t possible.
     That&rsquo;s why it&rsquo;s been so exciting to see active metadata take shape
     in the last year and become the de facto standard for what people expect out
     of modern metadata.
   </p>
   <p id="f4b4">
     All of these use cases &mdash; like auto-tuned pipelines, automated data
     quality alerts, and continuously validated calculations &mdash; would have
     sounded wildly impossible just a few years ago. Today, they&rsquo;re actually
     in reach. I couldn&rsquo;t be more excited to see the intelligent data dream
     become a reality as active metadata continues to evolve in the coming years.
   </p>    
   

What Is Active Metadata, and Why Does It Matter?

<p><em>The modern data stack is on the rise. Many companies use raw data from their SaaS analytics tools as input for their data warehouse, but this introduces problems downstream. Are there better ways?</em></p>
      <h3 id="popularization-of-the-all-in-one-saas-analytics-tool" class="anchor anchorWithStickyNavbar_mojV">Popularization of the all-in-one SaaS analytics tool<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#popularization-of-the-all-in-one-saas-analytics-tool">​</a></h3>
      <p>When Google Analytics launched nearly 20 years ago, it took the market by storm. Its success created the mold for SaaS-based analytics offerings that included everything in one box: data collection, storage, analysis &amp; visualization.</p>
      <p>Ease-of-use and low barrier of entry were two of its main drivers. Instrumentation typically required little in terms of engineering resources and the included analyses, although somewhat limited, were ready to be used by a broad audience.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/urchin.jpeg" alt="Urchin Analytics, to become Google Analytics after acquisition." />
      <p class="imageCaption___5l">Urchin Analytics, to become Google Analytics after acquisition.</p>
      </div>
      <h3 id="the-big-shift-towards-data-warehousing" class="anchor anchorWithStickyNavbar_mojV">The big shift towards data warehousing<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-big-shift-towards-data-warehousing">​</a></h3>
      <p>While SaaS analytics tools are still very popular and widely used, their wild growth has left the data analysis landscape highly fragmented. Most companies have numerous tools collecting data, and each of them have a different way of analyzing &amp; consuming it. As a result, it&rsquo;s very hard to get a holistic view of what all that combined data is telling you.</p>
      <p>We&rsquo;re seeing a major shift where companies export data from their SaaS analytics tools to use as input for their data warehouse. Not only to create a single source of truth for all data consumers, but also to unlock the potential of raw data. By combining full raw data sets, you can go way beyond the scope of your SaaS analytics tools. You&rsquo;re technically only limited by the quality and completeness of the data.</p>
      <h3 id="the-rise-of-the-modern-data-stack" class="anchor anchorWithStickyNavbar_mojV">The rise of the modern data stack<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-rise-of-the-modern-data-stack">​</a></h3>
      <p>The big shift towards data warehousing is represented by the upsurge of companies that operate in this space and how fast they&rsquo;re growing.&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://snowplowanalytics.com/" target="_blank" rel="noopener noreferrer">Over 10.000 companies now use Snowplow</a>&nbsp;to process events at scale and send them straight into the DWH. dbt now has over&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://www.getdbt.com/blog/next-layer-of-the-modern-data-stack/" target="_blank" rel="noopener noreferrer">9.000 weekly active data transformation projects</a>.&nbsp;<a target="_blank" rel="noopener" class="mds-text-primary"  href="https://www.rudderstack.com/" target="_blank" rel="noopener noreferrer">13.000+ sites and apps are running RudderStack</a>&nbsp;to pull data from SaaS tools and send it into data warehouses. The list goes on.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/dbt-cloud.png" alt="dbt Cloud in action" />
      <p style="text-align: center;" class="imageCaption___5l">dbt Cloud in action</p>
      </div>
      <h3 id="-but-saas-analytics-tools-arent-great-input-for-your-dwh" class="anchor anchorWithStickyNavbar_mojV">... but SaaS analytics tools aren&rsquo;t great input for your DWH<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#-but-saas-analytics-tools-arent-great-input-for-your-dwh">​</a></h3>
      <p>Most data warehouses are fed with data that&rsquo;s collected by all-in-one-box SaaS analytics tools. While historically explainable, this often causes problems downstream.</p>
      <h3 id="the-data-they-collect-is-not-designed-for-advanced-modeling" class="anchor anchorWithStickyNavbar_mojV">The data they collect is not designed for advanced modeling<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#the-data-they-collect-is-not-designed-for-advanced-modeling">​</a></h3>
      <p>There is a big difference between what data teams want their raw data to look like and what it actually looks like when it comes from SaaS analytics tools. The data sets they collect were never designed for advanced modeling on the raw data. It is often incomplete or overcomplete, unstructured and ambiguous. Significant grunt work is typically required before it can be used for modeling.</p>
      <h3 id="data-duplication-frustrates-adoption-of-a-single-source-of-truth" class="anchor anchorWithStickyNavbar_mojV">Data duplication frustrates adoption of a single source of truth<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#data-duplication-frustrates-adoption-of-a-single-source-of-truth">​</a></h3>
      <p>By extracting data from SaaS tools and loading it into a data warehouse, you&rsquo;re effectively duplicating a source.</p>
      <p>SaaS analytics tools all offer their own way to consume the collected data, and chances are many of your company&rsquo;s data consumers still rely on the dashboards of these tools to inform their daily decisions. Either out of habit, or simply because it offers an easy &amp; fast way to get data.</p>
      <p>In practice, this often leads to discussions about why numbers of different teams don&rsquo;t seem to be adding up, as the data team and data consumers both use different versions of the same source of data as input.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/before-saas.png" alt="" />
      <p class="imageCaption___5l">&nbsp;</p>
      </div>
      <h3 id="they-operate-as-black-boxes" class="anchor anchorWithStickyNavbar_mojV">They operate as black boxes<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#they-operate-as-black-boxes">​</a></h3>
      <p>SaaS analytics tools typically don&rsquo;t disclose the actual models behind an analysis. As a result, rebuilding an analysis to work with the data set in your data warehouse requires significant reverse engineering effort, and troubleshooting &amp; debugging can be painful when anything is off.</p>
      <hr />
      <h3 id="unbundling-saas-analytics-tools" class="anchor anchorWithStickyNavbar_mojV">Unbundling SaaS analytics tools<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#unbundling-saas-analytics-tools">​</a></h3>
      <p>The way data is used has evolved, and the all-in-one-box model that SaaS analytics tools use no longer fits. In order to effectively combine and analyze data from multiple sources and truly serve all data consumers from a single source of truth, we need to unbundle the black box and separate data collection, storage, modeling and visualization.</p>
      <h3 id="moving-all-data-consumption-downstream" class="anchor anchorWithStickyNavbar_mojV">Moving all data consumption downstream<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#moving-all-data-consumption-downstream">​</a></h3>
      <p>By separating data collection, all collected data can be sent straight into the data warehouse without stopovers. This eliminates source duplication and moves data consumption downstream. It enables&nbsp;<em>all</em>&nbsp;analysis &amp; visualization to occur on a single source of truth: the data warehouse.</p>
      <div><img style="width: 100%"  class="blogImage_Nvn_ blogImageLarge_RuN5" src="https://objectiv.io/img/blog/after-saas.png" alt="" />
      <p class="imageCaption___5l">&nbsp;</p>
      </div>
      <h3 id="shifting-from-vendor-locked-tools-to-raw-data-and-code" class="anchor anchorWithStickyNavbar_mojV">Shifting from vendor-locked tools to raw data and code<a target="_blank" rel="noopener" class="mds-text-primary"  class="hash-link" title="Direct link to heading" href="https://objectiv.io/blog/unbundling-saas-analytics/#shifting-from-vendor-locked-tools-to-raw-data-and-code">​</a></h3>
      <p>Unbundling the black boxes also provides a big opportunity for the adoption of open standards. Shifting from vendor-locked tools to raw data and code enables data teams to adopt generic ways of structuring data to share and reuse each others&rsquo; tools, models and analyses.</p>
      <p>We think the future of analytics is open and data warehouse-native, and there are clear signals we&rsquo;re already on the way. It&rsquo;s time to enable meaningful collaboration and to unbundle SaaS analytics.</p>

The Unbundling of SaaS Analytics

<p>Breaking down some of the problems I’ve seen in data collaboration and offering advice on how to make better, faster decisions with collaborative analytics.</p>
<p>We all know how analytics can help make businesses make better decisions. Being data-driven just simply isn&rsquo;t an option anymore. Startups are collecting more data than ever before and are increasingly using that data to&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://a16z.com/2011/08/20/why-software-is-eating-the-world/">eat the world</a>. We&rsquo;ve seen the rise of&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://www.whatmatters.com/articles/ted-talk">OKR goal-setting</a>, reporting tabs in nearly every SaaS tool, and metrics dashboards lining the walls in office hallways.<br /><br />Yet for all of this progress, the ways we share insights and make decisions together haven&rsquo;t evolved over the last decade. Too many companies spend too long collecting, preparing, and reporting on data to improve. In this post, I&rsquo;ll break down some of the problems I&rsquo;ve seen first-hand in data collaboration and offer advice on how you can create a better environment for creating, sharing, and measuring data together.</p>
<h2 id="the-problems">The problems</h2>
<h3 id="finding-a-source-of-truth"><br />Finding a Source of Truth</h3>
<p>Startups now have a plethora of dashboards in our tools to find data. But it&rsquo;s still too challenging to find the data you need. Legacy BI tools suffer from&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://venturebeat.com/2022/04/10/7-easy-ways-to-fight-the-data-sprawl/">dashboard sprawl</a>&nbsp;and poor search experiences. It&rsquo;s challenging to agree upon and standardize metrics to get common questions answered in a scalable way.</p>
<p>To fix this, we&rsquo;ve tried creating data catalogs so operators can more easily answer FAQs and find the data they&rsquo;re looking for. We&rsquo;ve tried to provide definitions for data to help operators understand what tables entail, how metrics are calculated, and what a particular column means. These efforts result in incremental progress towards becoming data-driven, but more is needed.</p>
<h3 id="data-breadlines">Data Breadlines</h3>
<p>Too few employees have access to real-time data to make everyday decisions. When operators need to query the database or ask a question involving data from multiple sources, they&rsquo;re&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/why-were-building-canvas-a-data-tool-for-non-technical-teams/">forced to ask for help</a>&nbsp;in a shared analytics channel or submit a ticket to their data or engineering teams.</p>
<p>This process results in a painful waiting period until the technical team can prioritize the request. Even when the data or engineering team can prioritize a request, there&rsquo;s typically a frustrating back-and-forth of questions and answers across Slack, Email, and Zoom.</p>
<p>Operators shouldn&rsquo;t have to wait days or even weeks for the answers they need, while technical teams should be able to focus on strategic work without constant pings for help on&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/enabling-ad-hoc-analysis/">tactical, ad-hoc questions.</a></p>
<h3 id="siloed-broken-workflows">Siloed, Broken Workflows</h3>
<p>Stop me if you&rsquo;ve gone through this process before. You need to prepare for a presentation. You go to your apps, pull CSVs, build some metrics, screenshot some charts, and throw them into a deck. You present the information and get some live questions about the data. Unfortunately, because the chart is a static screenshot, you can&rsquo;t dive into the data live, so you promise to follow up afterward.</p>
<p>The hours-long workflow we endure to prepare presentations across a suite of apps is fundamentally broken. There&rsquo;s no connective tissue between the data sources, the analysis tools, the presentation, and the feedback on the data.</p>
<p>Because of these silos, we may get fewer eyeballs from peers before a presentation. We might forget to include a killer insight or make a potential error in our analyses. Because it's too difficult to answer questions together live during a presentation, we can get caught flat-footed and look unprepared. And because it's so difficult to measure and discuss progress on metrics together, we fail to monitor updates closely and act quickly and decisively as teams.</p>
<h2 id="how-collaborative-analytics-helps">How Collaborative Analytics Helps</h2>
<p>To help teams move quickly to get the answers they need, you should consider investing in collaborative processes and tools. See below for some key components you should consider.</p>
<h3 id="the-warehouse-as-a-source-of-truth">The Warehouse as a Source of Truth</h3>
<p>Having a single source of truth that&rsquo;s accurate and up-to-date ensures your teams can all work off the correct data independently and confidently. Best-in-class companies typically use a data warehouse and ETL tools to centralize data from multiple sources in one place. If you don&rsquo;t know where to start, check out my guide on navigating the tools in the modern data stack.</p>
<p>Suppose you don&rsquo;t have the bandwidth or technical experts to implement a data stack correctly. In that case (time for a quick plug),&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/blog/no-warehouse-no-problem/">Canvas provides a managed data stack</a>&nbsp;for centralizing your apps and modeling your data in one place.</p>
<h3 id="reusable-datasets-for-fast-discovery">Reusable Datasets for Fast Discovery</h3>
<p>With your data in one place, you&rsquo;ll need to organize it, provide context, and make it easy to find. Key metrics and common areas should be searchable and social proof should help teams quickly identify the right data to answer their questions.</p>
<p>You should have clear owners for building and maintaining dashboards on the company and team levels. They should define and document data clearly and ensure a clear process for getting help on questions. You should be aware of which dashboards are frequently used and invest accordingly, while continually improving or pruning dashboards that are ghost towns.</p>
<h3 id="collaborative-workspaces-for-exploration">Collaborative Workspaces for Exploration</h3>
<p>Shared, collaborative workspaces are critical in breaking down silos and helping teams share and discuss insights with others.</p>
<p>Yes, having dashboards are a must. But as I pointed out, your teams will still have everyday questions that dashboards don&rsquo;t answer. Instead of a ticketing system, you should have a way for your operators to answer these questions without knowing SQL.</p>
<p>No-code and spreadsheet-like interfaces help your operators gain self-serve access without going days or weeks for technical teams to answer strategic and tactical questions. And when they have a question, operators and technical teams can collaborate and comment right where the data lives. This way, you can ditch the screenshots, snippets, and Slack channels and start making decisions together from anywhere.</p>
<h2 id="canvas-the-collaborative-analytics-platform">Canvas, the Collaborative Analytics Platform</h2>
<p>At&nbsp;<a class="mds-text-primary" rel="noopener" target="_blank"  href="https://canvasapp.com/">Canvas</a>, we&rsquo;re thrilled to be working on the future of data collaboration. Unlike legacy BI tools, Canvas has:</p>
<ul>
<li>Integrations to over a hundred apps so you can centralize and explore your data without a warehouse</li>
<li>A spreadsheet-like interface for your teams to pivot, write formulas, join, and create charts without SQL</li>
<li>Figma-like collaboration lets your teams communicate where the data lives</li>
<li>Out-of-the-box usage metrics inform data teams of who is using specific models so they can invest in popular areas or fix problems affecting less helpful models</li>
<li>Role-based access control so you can make sure the right people have access to the right data</li>
<li>A native dbt integration so your data models and documentation about the tables and columns are automatically synced and surfaced directly to operators</li>
</ul>

How to Make Better Decisions Together with Collaborative Analytics

<p class="p2">
    <span class="s1"
      >The term &ldquo;observability&rdquo; means many things to many people. A
      lot of energy has been spent&mdash;particularly among vendors offering an
      observability solution&mdash;in trying to define what the term means in one
      context or another.</span
    >
  </p>
  <p class="p2">
    <span class="s1"
      >But instead of getting bogged down in the &ldquo;what&rdquo; of
      observability, I think it&rsquo;s more valuable to address the
      &ldquo;why.&rdquo; What are we trying to accomplish with observability? What
      is the end goal?&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >At&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/why-unravel/"
        target="_blank"
        rel="noopener"
        >Unravel</a
      >, I&rsquo;m not only the co-founder and CTO but also the head of our
      Customer Success team, so when thinking about modern data stack
      observability, I look at the issue through the lens of customer problems and
      what will alleviate their pain and solve their challenges.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >I start by considering the ultimate end goal or Holy Grail: autonomous
      systems. Data teams just want things to work; they want everything to be
      taken care of, automatically. They want the system itself to be watching
      over issues, resolving problems without any human intervention at all.
      It&rsquo;s the true spirit of AI: all they have to do is tell the system
      what they want to achieve. It&rsquo;s going from reactive to
      proactive.&nbsp;</span
    ><span class="s3"
      >No one ever has to troubleshoot the problem, because it was
      &ldquo;fixed&rdquo; before it ever saw the light of day. The system
      recognizes that there will be a problem and auto-corrects the issue
      invisibly.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >As an industry, we&rsquo;re not completely at that point yet for the modern
      data stack. But we&rsquo;re getting closer.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >We are on a continuous and incremental spectrum of progressively less toil
      and more automation: going from the old ways of manually stitching together
      logs, metrics, traces, and events from disparate systems; to accurate
      extraction and correlation of all that data in context; to automatic
      insights and identification of significant patterns; to AI-driven actionable
      recommendations; to autonomous governance.</span
    >
  </p>
  <img
    style="width: 100%"
    src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-observability-spectrum-750x299.png"
    loading="lazy"
    alt=""
  />
  <p></p>
  <h3 class="p5">Beyond observability: to optimization and on to governance</h3>
  <p class="p2">
    <span class="s1"
      >Let&rsquo;s say you have a data pipeline from Marketing Analytics called
      ML_Model_Update_and_Scoring that&rsquo;s missing its SLA. The SLA specifies
      that the pipeline must run in less than 20 minutes, but it&rsquo;s taking 25
      minutes. What happened? Why is the data pipeline now too slow? And how can
      you tune things to meet the SLA? This particular application is pretty
      complex, with multiple jobs processing in parallel and several components
      (orchestration, compute, auto-scaling clusters, streaming platforms,
      dashboarding), so the problem could be anywhere along the line.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >It&rsquo;s virtually impossible to manually pore through t</span
    ><span class="s3"
      >he thousands of logs that are generated at each stage of the pipeline from
      the various tools</span
    ><span class="s2"
      >&mdash;Spark and Kafka and Airflow logs, Databricks cluster logs,
      etc.&mdash;to &ldquo;untangle the wires&rdquo; and figure out where the
      slowdown could be. But where should you even start? Even if you have an idea
      of what you&rsquo;re looking for, it can take hours&mdash;even days or weeks
      for highly complex workflows&mdash;to stitch together all the raw
      metrics/events/logs/traces data to figure out what&rsquo;s meaningful to why
      your data pipeline is missing its SLA. Just a single app can run 10,000
      containers on 10,000 nodes, with 10,000 logs.&nbsp;</span
    >
  </p>
  <img style="width: 100%" src="https://miro.medium.com/max/1200/1*XYQOw2kNJKZGKzBOrR0E7A.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s2">That&rsquo;s where observability comes in.</span>
  </p>
  <h3>
    <span id="Observability_tells_you_what"
      ><span class="s1">OBSERVABILITY TELLS YOU &ldquo;WHAT&rdquo;</span></span
    >
  </h3>
  <p class="p2">
    <span class="s1"
      >Instead of having to sift through reams of logs and cobble together
      everything manually,&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/dataops/"
        target="_blank"
        rel="noopener"
        >full-stack observability</a
      >&nbsp;extracts all the relevant raw data from all the various components of
      your data stack and correlates it to paint a picture of what&rsquo;s going
      on. All the information captured by telemetry data (logs, metrics, events,
      traces) is pulled together in context and in the &ldquo;language
      you&rsquo;re speaking&rdquo;&mdash;the language of&nbsp;</span
    ><span class="s4">data applications and pipelines</span
    ><span class="s1">.</span>
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-observability-before-2-750x264.png" loading="lazy" alt="" />
  <p></p>
  <p id="attachment_9257" class="wp-caption alignnone">
    In this case, observability shows you that while the application completed
    successfully, it took almost 23 minutes (22m 57s)&mdash;violating the SLA of
    20 minutes. Here, Unravel takes you to the exact spot in the complex pipeline
    shown earlier and, on the left, has pulled together granular details about the
    various jobs processing in parallel. You can toggle on a Gantt chart view to
    get a better view of the degree of parallelism:
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-gantt-750x544.png" loading="lazy" alt="" />
  <p></p>
  
  <p>
    So now you know what caused the pipeline to miss the SLA and where it happened
    (jobs #0 and #3), but you don’t know why. You’ve saved a lot of investigation
    time—you get the relevant data in minutes, not hours—and can be confident that
    you haven’t missed anything, but you still need to do some detective work to
    analyze the information and figure out what went wrong.
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-chart-1-750x171.png" loading="lazy" alt="" />
  <h3 class="p7">
    <span id="Optimization_tells_you_whyand_what_to_do_about_it"
      ><span class="s1"
        >OPTIMIZATION TELLS YOU &ldquo;WHY&rdquo;&mdash;AND WHAT TO DO ABOUT
        IT</span
      ></span
    >
  </h3>
  <p class="p2">
    <span class="s4"
      >The better&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/dataops/"
        target="_blank"
        rel="noopener"
        >observability tools</a
      >&nbsp;also point out, based on patterns, things you should pay attention
      to&mdash;or areas you don&rsquo;t need to investigate.&nbsp;</span
    ><span class="s1"
      >By applying machine learning and statistical algorithms, it essentially
      throws some math at the correlated data to identify significant
      patterns</span
    ><span class="s4"
      >&mdash;what&rsquo;s changed, what hasn&rsquo;t, what&rsquo;s different.
      This pipeline runs regularly; why was it slow this time?&nbsp;</span
    ><span class="s1"
      >It&rsquo;s the same kind of analysis a human expert would do, only done
      automatically with the help of ML and statistical algorithms.</span
    >
  </p>
  <p class="p2">
    <span class="s1"
      >While it would certainly be helpful to get some generalized insight into
      why the pipeline slowed</span
    ><span class="s4"
      >&mdash;it&rsquo;s a memory issue&mdash;what you really need to know is what
      to do about it.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >AI-enabled observability goes beyond telling you what happened and why, to
      pinpointing the root cause and providing recommendations on exactly what you
      need to do to fix it.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-ai-reco-750x468.png" loading="lazy" alt="" />
  <p class="p4">
    <span class="s2"
      ><span class="s3"
        >AI-driven recommendations provide specific configuration parameters that
        need to be applied in order for the pipeline to run faster</span
      >&nbsp;and meet the 20-minute SLA. After the AI recommendations are
      implemented, we can see that the pipeline now runs in under 11
      minutes&mdash;a far cry from the almost 23 minutes before.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-after-750x103.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s3"
      >Too often getting to the point of actually fixing the problem is another
      time-consuming effort of trial and error. Actionable AI recommendations
      won&rsquo;t fix things automatically for you</span
    ><span class="s2"
      >&mdash;taking the action still requires human intervention&mdash;but all
      the Sherlock Holmes work is done for you. The fundamental questions of what
      went wrong, why, and what to do about it are answered automatically.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >Beyond optimizing performance, AI recommendations can also identify
      where&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/cluster-resource-optimization/"
        target="_blank"
        rel="noopener"
        >cost could be improved</a
      >. Say your pipeline is completing within your SLA commitments, but
      you&rsquo;re spending much more on cloud resources than you need to. An AI
      engine can determine how many or what size containers you actually need to
      run each individual component of the job&mdash;vs. what you currently have
      configured. Most enterprises soon realize that they&rsquo;re overspending by
      as much as 50%.&nbsp;</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-cost-1-1024x476.png" loading="lazy" alt="" />
  <p></p>
  <h3 class="p7">
    <span id="Governance_helps_avoid_problems_in_the_first_place"
      ><span class="s1"
        >GOVERNANCE HELPS AVOID PROBLEMS IN THE FIRST PLACE</span
      ></span
    >
  </h3>
  <p class="p4">
    <span class="s2"
      >These capabilities save a ton of time and money, but they&rsquo;re still
      relatively reactive. What every enterprise wants is a more proactive
      approach to making their data applications and pipelines run better, faster,
      cheaper. Spend less time firefighting because there was no fire to put out.
      The system itself would understand that the memory configurations were
      insufficient and automatically take action so that pipeline performance
      would meet the SLA.</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >For a whole class of data application problems, this is already happening.
      AI-powered recommendations and insights lay the groundwork for putting in
      place some automated governance policies that take action on your
      behalf.</span
    >
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/beyond-obs-chart-2-750x141.png" loading="lazy" alt="" />
  <p></p>
  <p>
    Governance is really all about converting the AI recommendations and insights
    into impact. In other words, have the system run automatic responses that
    implement fixes and remediations for you. No human intervention is needed.
    Instead of reviewing the AI recommendation and then pulling the trigger, have
    the system apply the recommendation automatically.&nbsp;
  </p>
  <img style="width: 100%" src="https://www.unraveldata.com/wp-content/uploads/2022/04/new-alert.png" loading="lazy" alt="" />
  <p></p>
  <p class="p4">
    <span class="s2"
      >Policy-based governance rules could be as benign as sending an alert to the
      user if the data table exceeds a certain size threshold or as aggressive as
      automatically requesting a configuration change for a container with more
      memory.&nbsp;</span
    >
  </p>
  <p class="p4">
    <span class="s2"
      >This is true&nbsp;<a
      style="color: #f66640"
        href="https://www.unraveldata.com/aiops-for-big-data/"
        target="_blank"
        rel="noopener"
        >AIOps</a
      >. You don&rsquo;t have to wait until after the fact to address an issue or
      perpetually scan the horizon for emerging problems. The system applies AI/ML
      to all the telemetry data extracted and correlated from everywhere in your
      modern data stack to not only tell you what went wrong, why, and what to do
      about it, but it predicts and prevents problems altogether without any human
      having to touch a thing.</span
    >
  </p>
  

Beyond Observability for the Modern Data Stack

<p>
      A
      <a
        style="color: #f66640"
        href="https://venturebeat.com/2021/08/04/97-of-execs-say-data-science-is-crucial-to-maintaining-profitability/"
        target="_blank"
        >majority</a
      >
      of business leaders believe data insights are key to the success of their<br />business
      in a digital environment. However, many companies
      <a
        style="color: #f66640"
        href="https://hbr.org/2022/02/why-becoming-a-data-driven-organization-is-so-hard"
        target="_blank"
        >struggle</a
      >
      to build a<br />data-driven culture, with a key reason being the lack of a
      sound data democratization<br />strategy.
    </p>
    <p>
      A survey by Google Cloud and Harvard Business Review shows
      <a
        style="color: #f66640"
        href="https://cloud.google.com/resources/hbr-turn-data-into-business-value-report"
        target="_blank"
        >97%</a
      >
      of companies<br />recognize that democratizing data is critical to their
      success, but only
      <a
        style="color: #f66640"
        href="https://cloud.google.com/resources/hbr-turn-data-into-business-value-report"
        >60%</a
      >
      are<br />"effectively" moving toward that goal.
    </p>
    <p>But what's data democratization?</p>
    <p>
      Democratizing data in an organization is much more than just giving employees
      access<br />to data. It's a process that enables everyone in an organization,
      regardless of their<br />technical skills, to be comfortable working with
      data, feel confident talking about it, and<br />make data-driven decisions
      that can positively impact various aspects of the business.<br />It's an
      ongoing process because it goes hand in hand with data literacy, which is also
      a<br />continuous process.
    </p>
    <h3>The rise of all-in-one data solutions</h3>
    <p>
      Until recently, data was only used by a few technical specialists in
      organizations who<br />would set up and operate the systems needed to make
      data available and accessible to<br />the rest of the team. But as the data
      democratization trend grows, new tools are being<br />developed for
      non-technical professionals to provide organization-wide access to data<br />and
      analytics.
    </p>
    <p>
      Of the data software available on the market, all-in-one tools that offer a
      comprehensive<br />range of solutions through a single platform are the most
      popular options for many<br />organizations looking to extract more value from
      their data.
    </p>
    <p>
      There are two main challenges that all-in-one data tools are designed to
      address,<br />namely the gap between data and commercial teams, and the
      expensive and<br />time-consuming nature of the process of data
      maturity&mdash;a measurement of how capable<br />an organization is at making
      the most of its data.
    </p>
    <h3>The gap between data and commercial teams</h3>
    <p>
      Data and commercial teams are often siloed functions within a company, and
      it's not<br />uncommon for them to fail to communicate with each other and
      even get in each other's<br />way. Different training backgrounds, different
      goals, or lack of context are just some of<br />the reasons.
    </p>
    <p>
      This inevitably leads to inefficiencies as each department takes a different
      direction on a<br />given project. In fact,
      <a
        style="color: #f66640"
        href="https://blog.bit.ai/collaboration-statistics/"
        target="_blank"
        >97%</a
      >
      of employees and executives believe lack of communication<br />and alignment
      within an organization impacts the outcome of a task or project.
    </p>
    <p>
      Better collaboration and cross-functional coordination between different
      departments is<br />therefore a key success factor, especially in today's
      world of remote work where a<br />company's growth and survival depends more
      than ever on effective teamwork.
    </p>
    <h3>Data maturity being an expensive and long process</h3>
    <p>
      Data maturity can be a costly and lengthy process for many organizations. A
      <a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        >Gartner</a
      ><br /><a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        >survey</a
      >
      reveals that 87.5% of companies have low levels of data and analytics
      maturity.<br />This means that individual business units are pursuing their
      own data and analytics<br />initiatives as stand-alone projects, and there's
      no common structure that would enable<br />the enterprise to efficiently
      leverage advanced analytics.
    </p>
    <p>
      According to
      <a
        style="color: #f66640"
        href="https://www.gartner.com/smarterwithgartner/take-your-analytics-maturity-to-the-next-level#:~:text=Low%20maturity%20can%20be%20the,ability%20to%20exploit%20advanced%20analytics."
        target="_blank"
        >Gartner</a
      >, low maturity can be the result of a lack of vision and skills, a lack<br />of
      experience in strategic planning and execution, limited budgets, and primitive
      or<br />outdated infrastructure. As a result, organizations suffer from
      incomplete or inconsistent<br />data sets, poor data quality, duplicate data
      processes, data security issues, and poor<br />coordination across the
      enterprise.
    </p>
    <p>
      For data democratization to reach its full potential, organizations must first
      improve their<br />maturity, but they may not have enough time and financial,
      technological, and human<br />resources to do so. It can also be difficult to
      bring on board skilled data experts&mdash;who<br />are in high demand
      today&mdash;or to get buy-in from leadership and senior management to<br />advance
      data initiatives.
    </p>
    <p>
      This is where all-in-one data tools can shine. They eliminate the need to
      invest in<br />multiple external tools or in-house solutions, consolidate and
      centralize all data, and<br />save teams the time and effort of searching
      multiple data sources.
    </p>
    <h3>Benefits of all-in-one data tools</h3>
    <p>
      Let's now take a closer look at how commercial teams, and companies in
      general, can<br />access and benefit from all-in-one data tools.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >1. Improved marketing performance</strong
      >
    </p>
    <p>
      According to a study,
      <a
        style="color: #f66640"
        href="https://get.treasuredata.com/rs/714-XIJ-402/images/the-state-of-the-customer-journey-arm-treasure-data.pdf"
        target="_blank"
        >47%</a
      >
      of marketers say that data silos are their biggest challenge<br />when it
      comes to gaining insights from data.
    </p>
    <p>
      A single data platform allows companies to create a journey map for customers
      that<br />shows how they interact with the brand over time. This journey map
      helps marketers<br />identify key touchpoints that can be improved or better
      leveraged. By being able to<br />segment users by their behavior, location, or
      device, companies can better target them.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >2. Improved customer experience</strong
      >
    </p>
    <p>
      A Forrester report shows that 72% of companies find it challenging to manage
      multiple<br />customer relationship management (CRM) tools across geographies
      and technology<br />silos.
    </p>
    <p>
      An all-in-one data platform enables the collection of customer feedback from
      multiple<br />channels, providing a holistic view of customer sentiment and
      satisfaction. This<br />feedback helps commercial teams identify what
      customers want and find ways to<br />improve the overall experience for their
      customers, while data teams get to reclaim their<br />time and energy to
      in-depth, impactful work.
    </p>
    <p>
      This is especially important because
      <a
        style="color: #f66640"
        href="https://go.emplifi.io/us-uk-customer-expectations-report.html"
        target="_blank"
        >61%</a
      >
      of people are willing to pay at least 5% more<br />if they know they'll have a
      good customer experience, and
      <a style="color: #f66640;" href="https://go.emplifi.io/us-uk-customer-expectations-report.html"
        >56%</a
      >
      say that the quality of<br />customer service has a greater impact on how
      positively they view a brand than any<br />other factor.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >3. Better customer lifetime value and retention</strong
      >
    </p>
    <p>
      The ability to identify key metrics in real time using a single data platform
      enables rapid<br />decision making and helps increase customer loyalty and
      engagement. This also<br />improves customer retention as companies can tailor
      their products or services to meet<br />customers' needs.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >4. Reduced costs and time allocated to data operations</strong
      >
    </p>
    <p>
      As all-in-one data tools help companies identify their customers' needs in
      real time, they<br />can help reduce costs, while increasing profits, and time
      spent on data operations.<br />That's because they can become data-driven
      without a full team of specialists, and if<br />and when they do hire a data
      expert or team, they&rsquo;ll be well equipped to dive right into<br />the
      work. Additionally, companies can identify when the cost of customer
      acquisition<br />outweighs the benefits, allowing for rapid growth or
      discontinuation of programs that<br />aren't working.
    </p>
    <p>
      <strong style="font-weight: 600; font-size: 18px; line-height: 27px"
        >5. Higher efficiency and revenue</strong
      >
    </p>
    <p>
      When a company has access to information about customer needs through a
      single<br />data platform, it can streamline its operations and achieve
      greater efficiency by<br />delivering products or services in the shortest
      possible time. This, in turn, enables faster<br />revenue growth.
    </p>
    <h3>Future-proofing your company through Data</h3>
    <p>
      All-in-one data solutions can facilitate the seismic shift to a data-driven
      culture. They<br />can help organizations launch their data operations in a
      matter of hours instead of<br />months or even years. They enable businesses
      to become data-driven without a full<br />team of specialists, and even if
      they do hire a data expert or team, they're well-equipped<br />to get started
      right away.
    </p>
    <p>
      Meanwhile, teams in each department can focus on important tasks instead of
      sifting<br />through and pulling data from multiple data sources. Easy access
      to data-driven insights<br />would lead to greater efficiency, productivity,
      and profits as well as higher data security.<br />Customers will also have a
      much better experience. So it's a win-win situation.
    </p>
    <p>
      In today's borderless digital world where data is the lingua franca, data
      democratization<br />is no longer a nice-to-have, but a necessity for
      organizations to achieve their goals and<br />gain a competitive advantage.
    </p>
    <p>
      To succeed, executives should invest in all-in-one data platforms and take a
      coordinated<br />and holistic approach to their efforts toward data
      democratization, which has triggered a<br />seismic shift in the business
      world.
    </p>
    

How All-in-One Tools Are Accelerating Data Democratization

<div>
      <p class="lead-paragraph blog-post"></p>
      <div class="cms-rich-text blog-post w-richtext">
          <p>The Modern Data Stack has been the hot topic of the data world for quite some time now. In the 12 years since
              Redshift was initially released, we’ve seen a mass explosion of capital and innovation that has enabled
              organizations to transform how they think about and leverage data.</p>
          <p>For this article, I’d like to discuss one of the major advantages of the MDS — the shift from ETL to ELT —
              and the primary users who are served and underserved in this paradigm. Today, I believe that the MDS
              provides a massive unlock in value for data engineers and analytic engineers but lacks the easy-to-use and
              intuitive toolkits for the data analyst, citizen data scientist, and knowledge worker that sits closest to
              the decision.</p>
          <p>The purpose of this blog is not to discuss the past, present, or future of the Modern Data Stack. There are
              many articles that provide a fantastic history lesson on the transition of on-prem legacy systems to the
              truly cloud native, modern data stack most of us know and love today. If you’re new to the MDS and
              interested in learning more, dbt’s CEO, Tristan Handy, wrote a great post, <a
              class="mds-text-primary"
                  href="https://www.getdbt.com/blog/future-of-the-modern-data-stack/" target="_blank">The Modern Data
                  Stack: Past, Present, and Future</a>, in December 2020, that perfectly articulates the transformation at
              hand.</p>
          <h3><strong>What is ELT and why is it important?</strong></h3>
          <p>You’ve likely heard about ELT — Extract Load and Transform… the Modern Data Stack’s evolution on ETL. This is
              a game changer by nature in that it enables organizations to ingest raw data into the data warehouse and
              transform it later. ELT gives end-users access to the entirety of the datasets they need by circumventing
              downstream issues of missing data that could prevent a specific business question from being answered.</p>
          <p>ELT isn’t a new topic. I remember the early days of my career in the Big Data/Hadoop space. Originally
              introduced by Google in 2005, the promise of Hadoop was, more or less, ELT. Don’t transform data on its way
              into the data platform and risk losing critical data that you need down the road. Instead, ship the entirety
              of the raw data into Hadoop, and you’ll have it…. for whenever you need it.</p>
          <p>The promise was great! Truly an amazing sales pitch, but unfortunately, it just didn’t work out. Data
              processing was too slow. It frustrated end-users, was not usable by the knowledge workers, and didn’t enable
              agility in analytically driven decision making. Spark made huge progress in reducing the speed of
              processing, but a better solution has emerged.</p>
          <p>This is where the Modern Data Stack, and the Cloud Data Warehouse (the core foundation of the MDS), differ.
              By separating storage from compute and enabling elastic computation in the cloud, tools such as Snowflake,
              BigQuery, Delta Lake, and Redshift have changed this paradigm. Data teams can now extract and load raw data
              directly into cloud storage for a super low cost per terabyte, and the users who need prepared data for
              analysis can get lightning-fast speeds throughout their data analysis workflow.</p>
          <h3><strong>What are the components, and who are the major players powering ELT?</strong></h3>
          <p>To make things simple, I’ve outlined a few of the major technologies being leveraged today. Again, for a
              complete overview of the MDS, check out Tristan Handy’s blog that I posted above.</p>
          <ul role="list">
              <li>Cloud Data Warehouse — Snowflake, BigQuery, Delta Lake, Redshift, and Synapse</li>
              <li>Data Integration — Fivetran, Airbyte, and Segment</li>
              <li>Data Transformation — dbt (the clear winner in the data transformation game)</li>
              <li>Reverse ETL — Census and Hightouch</li>
          </ul>
          <figure  >
              <div><img
  style="width:100%" src="https://uploads-ssl.webflow.com/6276a6c8de031667550a3847/62866cc38480f6788747d7c9_mds.png"
                      loading="lazy" alt=""></div>
              <figcaption>The modern data stack with leading vendors in each Silo</figcaption>
          </figure>
          <h3><strong>Who are the main users being served? Who are the underserved users?</strong></h3>
          <p>To answer this question, it’s first important to level set on the key users we see in the end to end data
              stack. I like to think of this by breaking down users into <em>data producers</em> and <em>data
                  consumers</em>. Data producers are the individuals who are integrating and modeling data on behalf of
              the organization. Data consumers consume the cleaned and modeled data to answer critical business questions
              from that data.</p>
          <figure  >
              <div><img
  style="width:100%"
                      src="https://uploads-ssl.webflow.com/6276a6c8de031667550a3847/62867609a729465066769724_personas.png"
                      loading="lazy" alt=""></div>
              <figcaption>Professionals who use the modern data stack and descriptions</figcaption>
          </figure>
          <p>In working in this space for years, I’ve seen firsthand that these tools (mentioned above) enable the data
              engineer and the (newly minted job title) analytic engineer super well. They provide modern, easy-to-use,
              and cloud native code-first tools for the ELT workflow and provide the organizations that employ these users
              with better speed and governance of their data assets.</p>
          <p>Said differently, the Modern Data Stack is awesome in that it enables centralization, speed, agility, and
              performance. But, today, it is a walled garden for users who are highly proficient in SQL development.</p>
          <p>So who is left out in this world? In Tristan’s article on the past, present, and future of the Modern Data
              Stack (posted above), he argues that one of the biggest gaps still to be closed in the MDS is the bridge to
              the data consumer.</p>
          <p>Tristan goes as far as saying, “Data consumers were actually more self-serve <em>prior</em> to the advent of
              the modern data stack: Excel skills are widely dispersed through the population of knowledge workers. There
              has not yet been an analogous interface where all knowledge workers can seamlessly interact with data in the
              modern data stack in a horizontal way.”</p>
          <p>I couldn’t agree more! In my viewpoint, the data consumer, in the analytics workflow, is not only underserved
              but arguably the most essential user who needs a solution. After all, this user is the one who sits closest
              to the business decision that can drive differentiated advantage in the marketplace.</p>
          <p>This knowledge user is different. They typically understand SQL conceptually, and many are proficient at
              writing SQL code, but the true power of this user is that they act as both the data SME and the business SME
              that truly understands the problem at hand and the business context of what they are trying to solve. Said
              differently, the critical value of the knowledge user is domain expertise, not programming expertise.</p>
          <p>This user needs to answer questions from data. They need insights from data…. fast. They are typically on the
              hook for preparing critical data insights that lead to executive and senior management decisions on the
              direction of the company. Important stuff!</p>
          <h3><strong>So… what does ELT mean for this knowledge user? Envisioning a new world.</strong></h3>
          <p>In my opinion, ELT represents a transformational advantage for this user. Traditional BI tools such as
              Tableau do an amazing job at enabling visualization of data to answer questions but rely on the data
              engineer or analytic engineer to first prepare raw data into an analytical table that Tableau can query.</p>
          <p>This means that the visual insights from raw data workflow is, today, inherently not self-service. An analyst
              needs to partner with the analytics engineering team to get the data they need in the format they need. This
              slows down decision making to a halt and typically annoys the analytic engineering team with ad hoc requests
              that fall outside the scope of their prioritized sprint.</p>
          <p>Analytic engineers own production-level data pipelines that the business relies on. These pipelines are
              static, hardened, and scalable. This is a fundamentally different requirement than ad hoc data insights
              which are agile, iterative, and experimental. A new solution paradigm is needed.</p>
          <p>This requirement is not new. Alteryx built a massive business around no-code data preparation, but it was
              built for the traditional, on-premise world. It does not give users the advantages of ELT within the Modern
              Data Stack. It is, instead, ETL, forcing users to extract data from the central warehouse, impacting
              performance, speed, and auditability of the transformation pipeline.</p>
          <p>In another blog from Tristan, <a class="mds-text-primary" href="https://roundup.getdbt.com/p/code-and-no-code-having-it-both-ways?s=r"
                  target="_blank">“Code and No-Code, Having it Both Ways”</a>, he describes a major gap in the marketplace
              requiring no-code tools that, behind the scenes, write Modern Data Stack native SQL for the end user that is
              automatically pushed down to data in the Cloud Data Warehouse.</p>
          <p>A solution like this would enable data consumers to capture the value of ELT in the Modern Data Stack without
              forcing them to become an expert-level SQL developer. It would enable a rapid up-leveling of skill sets
              within a user community overnight.</p>
          <h3><strong>No-Code on the Modern Data Stack — A new paradigm to enable the knowledge worker</strong></h3>
          <p>At Rasgo, this is a problem we’ve thought a lot about. We’ve worked with hundreds of data analysts, business
              users, and citizen data scientists who have heard a lot about the Modern Data Stack and the value their data
              engineering teams have gleaned, but haven’t yet been able to fully capture the value of the cloud data
              ecosystem themselves.</p>
          <p>From my viewpoint, a new breed of tools purpose built for these end users needs to emerge to truly bring data
              consumers into the MDS ecosystem. These tools need to be UI first, enable a more iterative workflow, and be
              fully native to the underlying dbt transformation engine. They need to enable business users to answer
              ad-hoc questions from data as they come up, whether or not these questions rely on metrics that have already
              been created by the Analytic Engineering team. Without this, data consumers will always be at the mercy of
              their engineering counterparts, and the engineering teams will always be responding to ad hoc requests from
              the business, instead of thinking about modeling data for the holistic needs of the enterprise.</p>
          <p>Lately, I’ve been describing this to folks within the industry as “The UI for the Modern Data Stack.” It’s
              something we at Rasgo are heavily committed to perfecting, and we believe that, without this, true data
              driven decision making will never fully be realized.</p>
          <p>Really appreciate you reading!! Super exciting times to be in the data world and we look forward to what’s
              ahead :)</p>
      </div>
  </div>

ELT for the Data Consumer

<div>
        <p data-selectable-paragraph="">As data became more and more available to companies, data integration became one of
            the most crucial challenges for organizations. In the past decades, ETL (extract, transform, load) and later ELT
            (extract, load, transform) emerged as data integration methods that transfer data from a source to a data
            warehouse. Despite their similarities, the two differ in fundamental ways.</p>
        <p data-selectable-paragraph="">This blog aims to give an overview of ETL and ELT processes, describe the difference
            between them and outline which tools are now available for organizations. We do so by covering the following
            points:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>A brief history of data integration</strong></li>
            <li data-selectable-paragraph=""><strong>What is ETL?</strong></li>
            <li data-selectable-paragraph=""><strong>Why do we need ETL tools?</strong></li>
            <li data-selectable-paragraph=""><strong>ETL vs. ELT &mdash; What&rsquo;s the difference?</strong></li>
            <li data-selectable-paragraph=""><strong>When should you use ELT instead of ETL?</strong></li>
            <li data-selectable-paragraph=""><strong>What characteristics should your ETL/ELT tool have?</strong></li>
            <li data-selectable-paragraph=""><strong>What are the best ETL/ELT tools available right now?</strong></li>
        </ul>
    </div>
    <div>
        <h3><strong>A brief history of data integration</strong></h3>
        <p data-selectable-paragraph="">Today, businesses require data integration to centralize, access, and activate their
            data. As businesses start leveraging data from hundreds of different sources across continents and business
            departments and teams to make data-driven decisions, data integration has never been more critical. Although
            this heavily fractured landscape might seem new, it is not. In fact, the first challenges related to data
            integration date back to the 1960s when IBM and other companies pioneered the first Database Management Systems
            (DBMS), whose later advancements led to the sharing of data between computers. The process of integrating data
            with external machines soon became a challenge. This was challenging because organizations integrate with
            systems that they cannot control, and this often fails. So, it becomes really hard to create systems that
            don&rsquo;t break when you do data integration.</p>
        <p data-selectable-paragraph="">Data integration can be defined as the practice of consolidating data from separated
            source systems to transfer it to a data warehouse. Integration starts at the ingestion phase and includes steps
            like cleansing, ETL mapping, and transformation. ETL plays a central role in transforming raw data into usable
            data &mdash; enabling data teams to trust their data and make informed business decisions. Without data
            integration, accurate analytics is impossible to achieve as making decisions with incomplete information most
            likely leads to undesirable outcomes. And if these decisions involve hundreds of data sources, terabytes of
            data, and millions of dollars, integration becomes especially important.</p>
        <p data-selectable-paragraph="">ETL was the first standardized approach to dealing with data integration which
            appeared in the 1970s. It first emerged as a response to the adoption of relational databases and the increasing
            need to move information between them efficiently. Then, in the 1980s, data integration became even more
            critical with the rise of data warehouses. Data warehouses could integrate data from different sources. However,
            they typically required custom ETLs for each data source. This led to the emergence of a wide variety of ETL
            tools. And by the end of the 1990s, these solutions were affordable for both enterprises and mid-market
            businesses.</p>
        <p data-selectable-paragraph="">Finally, with the emergence of cloud computing in the 2000s, cloud data warehouses
            and data lakes brought forward a new development: ELT, which, as we will see later in this blog, allows
            businesses to load unlimited raw data into a cloud data warehouse.</p>
    </div>
    <div>
        <h3><strong>What is ETL?</strong></h3>
        <p data-selectable-paragraph="">As organizations collect data from multiple sources &mdash; such as CRM systems,
            Cloud applications, and more &mdash; the ETL process aims to gather data from separate sources and create a
            centralized database. Practically, ETL pipelines have the role of extracting raw data from its source,
            transforming it, and finally loading it in the warehouse &mdash; which is a centralized database.</p>
        <p data-selectable-paragraph="">Let&rsquo;s dive into each one of these steps:</p>
        <p data-selectable-paragraph=""><strong>Extract</strong></p>
        <p data-selectable-paragraph="">The first step is the process of data extraction. Here, data pipelines extract data
            from the different sources and gather it in a staging area. The staging area is vital because it allows the
            extraction of data from various sources at different times, and it prevents the simultaneous performance of
            extractions and transformations from happening &mdash; therefore averting the risk of overwhelming data sources.
        </p>
        <p data-selectable-paragraph=""><strong>Transform</strong></p>
        <p data-selectable-paragraph="">The second step is the data transformation process &mdash; in other words, making
            the raw data fit for analysis. This consists of&nbsp;<strong>improving the quality</strong>&nbsp;of the data by
            performing actions like sorting, joining, reformatting, filtering, merging, aggregation, and so on,
            and&nbsp;<strong>restructuring and reformatting</strong>&nbsp;the data to adapt it to the specific business need
            it will be used for.</p>
        <p data-selectable-paragraph=""><strong>Load</strong></p>
        <p data-selectable-paragraph="">The last step is loading the transformed data into the warehouse. The loading can be
            full &mdash; so the data is fully loaded into the warehouse at once &mdash; or incremental &mdash; where the
            data is loaded at different scheduled intervals using batch or stream loading. Batch loading consists of the
            scheduled extraction of different batches of data from a source system. In contrast, stream loading consists of
            data ingestion by pipelines as soon as they are made available by the source system.</p>
    </div>
    <div>
        <h3><strong>Why do we need ETL tools?</strong></h3>
        <p data-selectable-paragraph="">The ETL process of migrating transformed data into a warehouse enables organizations
            to break down silos and makes it easier for data users to turn the available information into business
            intelligence.</p>
        <p data-selectable-paragraph="">ETL is beneficial for organizations because it combines diverse data into a unified
            view. This enables data users to find and access information more efficiently, consequently improving
            productivity.</p>
        <p data-selectable-paragraph=""><strong>The advantages of ETL</strong></p>
        <p data-selectable-paragraph="">Implementing ETL into your data stack can bring many benefits, including:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>Speed:&nbsp;</strong>There is no waiting involved when using ETL tools,
                as they instantly load data into your data warehouse, where they are ready for transformation.</li>
            <li data-selectable-paragraph=""><strong>Flexibility:&nbsp;</strong>As transformations do not need to be defined
                at the beginning, it is pretty straightforward to integrate additional data sources into the ELT process.
            </li>
            <li data-selectable-paragraph=""><strong>Low initial cost:&nbsp;</strong>ETL automates the process of onboarding
                data, and as you don&rsquo;t need to define transformations, the initial cost is relatively low.</li>
            <li data-selectable-paragraph=""><strong>Low maintenance requirements:&nbsp;</strong>As the process is more
                straightforward and automated, it will require less maintenance. It is also easier to fix bugs since
                transformation only happens in the last step.</li>
            <li data-selectable-paragraph=""><strong>Scalability:&nbsp;</strong>As the amount of data you use increases, you
                can expand your storage in the cloud.</li>
        </ul>
    </div>
    <div>
        <h3><strong>Challenges to ETL and the rise of ELT</strong></h3>
        <p data-selectable-paragraph="">The main characteristic of ETL is the order in which the process is performed
            &mdash; specifically, the data goes through transformation before being loaded into the warehouse. This specific
            order is the reason behind some of the difficulties organizations face when using ETL. These are the following:
        </p>
        <ul>
            <li data-selectable-paragraph=""><strong>Dealing with heavy data loads:&nbsp;</strong>Volume and demand for data
                have been dramatically growing in the past few years. Unfortunately, many ETL processes are failing to scale
                to match these heavy loads appropriately. When dealing with large amounts of data, some common mistakes are
                loading irrelevant data, bottlenecks due to insufficient CPU and serial data processing instead of parallel
                data processing.</li>
            <li data-selectable-paragraph=""><strong>Maintenance of the pipelines:&nbsp;</strong>In ETL, pipelines extract
                and transform data all at once, breaking as soon as schema changes in the data source. For instance, when a
                field is removed. This compromises the code needed to transform the data, and it has to be re-written by
                data engineers.</li>
            <li data-selectable-paragraph=""><strong>Data security risks:&nbsp;</strong>When loading large amounts of raw
                data into your storage, data security can become a concern. To deal with this challenge, managing user
                access to the raw data stored in the data warehouse is crucial.</li>
            <li data-selectable-paragraph=""><strong>Low compliance with regulations:&nbsp;</strong>Since data is stored
                with minimal processing, there may be some additional steps to ensure that you comply with the main security
                protocols, including GDPR and HIPAA.</li>
        </ul>
    </div>
    <div>
        <h3><strong>ETL vs. ELT &mdash; What&rsquo;s the difference?</strong></h3>
        <p data-selectable-paragraph="">Traditional ETL is characterized by the performance of the transformation process
            before the loading into the warehouse. The reason for this is that, back in the day when ETL was created,
            storage, computation, and bandwidth were costly. Hence the need to reduce the volume of data before it gets to
            the data warehouse.</p>
        <p data-selectable-paragraph="">As storage and computation cost dramatically decreased, cloud storage grew
            exponentially. This led to the entrance into the market of data warehouses like Amazon Redshift and Google
            BigQuery. Cloud data warehouses are infinitely more scalable compared to traditional warehouses, allowing
            organizations to fit enormous amounts of data.</p>
        <p data-selectable-paragraph="">The emergence of cloud data warehouses called for a radical change in ETL processes
            too. If ETL processes were fine for traditional warehouses, their performance did not improve in a cloud
            environment. The reason for this is that the scalability issues were inherent to the process of moving data into
            the warehouse. At the same time, due to the arrival of cloud storage, organizations started loading large
            volumes of untransformed data into the warehouse without having to worry about its cost. From this moment on, it
            has been possible to extract data from the source systems and load it into the cloud warehouse without needing
            to go through the transformation process. The Extract-Load-Transform (ELT) process replaced traditional ETL.</p>
        <p data-selectable-paragraph="">Let&rsquo;s take a side-by-side look at the differences between ETL and ELT, their
            pros and cons, and how they can work together to provide a holistic data integration solution for businesses.
            Here are the ten critical differences between ETL and ELT.</p>
        <figure>
            <div tabindex="0" role="button">
                <div><img role="presentation" src="https://miro.medium.com/max/1400/1*nY2lYvuPTC_ywQOs0nny7Q.png" alt="" width="700" height="628"></div>
            </div>
            <figcaption data-selectable-paragraph="">Source: Benedetta Cittadin</figcaption>
        </figure>
    </div>
    <div>
        <h3><strong>What characteristics should your ETL/ELT tools have?</strong></h3>
        <p data-selectable-paragraph="">If you&rsquo;ve made it so far, it probably means you are interested in integrating
            ETL or ELT in your data stack. But what characteristics should these tools have to meet your standards?
            We&rsquo;ve gathered some general, critical requirements that any of these tools should have:</p>
        <ul>
            <li data-selectable-paragraph=""><strong>Built-in integrations</strong>: You will need your ETL/ELT tools to
                interact with many different programs and APIs to use your data.</li>
            <li data-selectable-paragraph=""><strong>Ease of use:&nbsp;</strong>The tool should have an intuitive interface
                that assists you in simplifying as many functions as possible as well as making it possible to train
                non-developers to use it.</li>
            <li data-selectable-paragraph=""><strong>Automation:&nbsp;</strong>The more tasks you can automate, the faster
                and easier it will be to get your work done. And if you adopt a tool, you don&rsquo;t want to have to
                program SQL workarounds every time you need to get something done. This is why your ETL/ELT tool should have
                automation in place.</li>
        </ul>
        <h3><strong>What are the best ETL tools available right now?</strong></h3>
        <p data-selectable-paragraph="">With companies collecting increasingly high amounts of data daily, there is a
            growing need to distribute data from the source to the data warehouse. This is why we have recently seen a
            blowing up of ETL/ELT tools in the market. In this section, I&rsquo;ll briefly outline the different ETL tools
            that are currently being used by organizations.</p>
        <p data-selectable-paragraph="">ETL can be grouped into different categories based on their infrastructure.
            I&rsquo;ll briefly go through three of these groups.</p>
        <p data-selectable-paragraph=""><strong>Enterprise software ETL</strong></p>
        <p data-selectable-paragraph="">Developed by commercial organizations, these solutions are often the most mature
            ones in the marketplace. These tools include graphical user interfaces (GUIs) for architecting ETL pipelines,
            support for relational and non-relational databases, and extensive documentation and user groups.</p>
        <p data-selectable-paragraph="">Although they offer more functionality, enterprise ETL tools are usually more
            expensive and require more employee training because of their complexity.</p>
        <p data-selectable-paragraph="">Think of tools like IBM DataStage and Oracle Data Integrator.</p>
        <p data-selectable-paragraph=""><strong>Open-source</strong></p>
        <p data-selectable-paragraph="">Many free ETL tools offer GUIs for data-sharing processes and monitoring the flow of
            information. An advantage of open-source tools is that organizations can access the source code and extend its
            capabilities.</p>
        <p data-selectable-paragraph="">Think of tools like Airbyte and Talend.</p>
        <p data-selectable-paragraph=""><strong>Custom ETL tools</strong></p>
        <p data-selectable-paragraph="">Cloud service providers (CSPs) now offer ETL tools built on their infrastructure. An
            advantage of this type of solution is that if the organization also stores its data using the same CSP, the
            pipeline is optimized because all processes occur within a single infrastructure. A disadvantage, however, is
            that cloud-based ETL tools only work within the CSP environment and don&rsquo;t support data stored in other
            locations.</p>
        <p data-selectable-paragraph="">Think of tools like Azure Data Factory and Google Cloud Dataflow.</p>
        <h3><strong>Conclusion</strong></h3>
        <p data-selectable-paragraph="">ETL and ELT can serve your data integration purpose in different ways. So, to choose
            the solution that&rsquo;s right for you, you need to consider factors like</p>
        <ul>
            <li data-selectable-paragraph="">The data you have</li>
            <li data-selectable-paragraph="">The type of storage you use</li>
            <li data-selectable-paragraph="">The (long-term) needs of your business</li>
        </ul>
    </div>

Survival of the Fittest: ETL vs. ELT



        <h3>&zwj;<strong>What is data product management</strong>?</h3>
        <p>In our experience at Secoda working with many data teams, we've seen most data teams do not have the tools they need
            to succeed. For growing organizations, the data function is usually an afterthought. The first data hire is brought
            on before raising a Series A and is expected to manage the workload that comes afterward with little to no support.
            This means that this first data hire usually spends all of their time answering other people&rsquo;s questions.
            These teams tend to be overworked and are likely to remain a reactive function for the foreseeable future.</p>
        <p>This is because most data teams are understaffed and misunderstood. Instead of reacting to employee data requests,
            working with data requires the data team to treat employees as their customers and their data as the product.</p>
        <p>An organization's data product is its collection of all of its data knowledge, along with the resources used for
            accessing, creating, and analyzing that data. Making the data product more self comprehensive, accessible,
            trustworthy, accurate, and secure should be the goal of the data team and the data team.&nbsp;</p>
        <p>Additionally, the data PM should help direct resources towards the highest leverage areas of the business. When
            products are built with everyone in mind, they usually don&rsquo;t accomplish what they need to for any specific
            subset of users. Data PMs should think about their data products the same way. Building data products for specific
            levels of technical capabilities and specific&nbsp;<em>persona&rsquo;s&nbsp;</em>of data consumers should be the
            goal of the data PM.&nbsp;</p>
        <h3>What makes a good data product manager</h3>
        <p>A good data product manager understands the company ecosystem of people and data. It is crucial to understand the
            different segments of the organization and document the requirements and knowledge from both data producers and
            consumers to generate and manage clean, reliable, and meaningful data. A good data product manager will apply
            product management practices to guide the roadmap for the data team by displaying some of the competencies below:
        </p>
        <ul role="list">
            <li>Conducting employees interviews to understand how they want to use data</li>
            <li>Running data-design sprints (How do we get a minimum version of this product to the employees in the shortest
                amount of time)&nbsp;</li>
            <li>Data product prioritization and road map planning.</li>
            <li>Communicating with the product team and updating the data team on changes that may affect their work.</li>
            <li>Performing vendor assessments.</li>
            <li>Translating business-to-data requirements, and vice versa.</li>
            <li>Defining and tracking key metrics for each stakeholder.</li>
            <li>Sharing knowledge and tracking data product usage metrics.</li>
        </ul>
        <p>We think about data PMs as the bridge between the data team and the rest of the organization. It takes time, tooling,
            and measurement to make sure that all stakeholders can access the right data, but when it&rsquo;s achieved,
            organizations can increase enterprise value by up to 5%.</p>
        <h3>Role requirements</h3>
        <ul role="list">
            <li><strong>Create Data Requirements</strong>&nbsp;&mdash; Data PMs should create clear data product requirements
                that satisfy the preferences of the end-users and data developers they work with. Even though data product
                requirements are still being defined, we like the development, release, and measurement model for building
                useful data products and determining how they are used.&nbsp;</li>
        </ul>
        <ul role="list">
            <li><strong>Employee feedback</strong>&nbsp;&mdash; Data PMs should bring together basic employee insights through
                interviews, understanding department OKR&rsquo;s, taking inbound questions, etc. Based on an explicit set of
                requests from the employees for whom the data product is being developed, they meet their baseline expectations
                (e.g. the employee says a part of the data is broken).</li>
        </ul>
        <ul role="list">
            <li><strong>Data insights</strong>- Data PMs should have a data proficiency where they can look at the data related
                to their data product (very meta) and use that data to make decisions about which products are performing well
                and which should be depreciated. They should have a working knowledge of data languages like SQL and know how to
                use popular tools like dbt, Looker, Metabase, Snowflake, Fivetran, and so on.</li>
        </ul>
        <ul role="list">
            <li><strong>Run Meetings</strong>&nbsp;&mdash; Data PMs should also run effective meetings with the data team and
                advocate for the team's resources to leadership. That includes setting the agenda and tracking action items as
                well as making sure the data team is well staffed and able to contribute to the data product the way that
                employees require.&nbsp;</li>
        </ul>
        <ul role="list">
            <li><strong>Vendor evaluation</strong>&nbsp;&mdash; The data PM should work closely with the engineering, sales, and
                operations teams to understand which products can give their team the best visibility and transparency into the
                data. By evaluating the right tools with other department leads, the data PM can help accelerate their team's
                data literacy and data maturity.&nbsp;</li>
        </ul>
        <h3>Data analytics</h3>
        <p>Think about how difficult it would be for a product manager to manage features without critical information about how
            people are using the product. Today, many data teams are operating in this same way. It&rsquo;s very unclear which
            tables, dashboards, metrics, and charts are used frequently and which should be depreciated. PMs can empower
            themselves and their teams with the information they need to make decisions to improve data products by adopting a
            tool that enables them to measure the most used data knowledge.&nbsp;</p>
        <p>It should be possible for data PMs to see how often teams use data, whether they are using the right data, and
            whether they aren't using any data. By understanding how people use analytics', data PMs can focus on the most
            important tasks that drive data literacy across their company.&nbsp;</p>
        <h3>The future of the Data PM</h3>
        <p>Over the next three to five years, we see the product-management role continuing to evolve toward a deeper focus on
            data as a product. Data product managers of the future will work with the data team to produce high-quality data
            products that can help drive self-service across the organization.&nbsp;</p>
        <p>To communicate data with the rest of the organization, data PM's might choose to implement tools like dbt, Looker,
            and Secoda. As we continue to help data PMs for the organization share their knowledge, we hope to be able to offer
            a greater depth of assistance and support to all employees trying to use data to make decisions. Data interactions
            are becoming increasingly complex, and Secoda hopes to simplify the interaction with data for every employee.</p>

Data Product Manager: The role and best practices for beginners

<p>From personal experience, I have always found it interesting to learn how to create an organized catalog of data.
        However, this interest was transformed into a passion when I began to realize the amount of time and effort it could
        save me within my job responsibilities. Creating a data catalog can greatly help you with organizing the data they
        collect, therefore making it easier to find what you need when you need it.</p>
    <p>By the end of this post, we should&rsquo;ve shed some light on the following questions:</p>
    <p><strong>Why all data teams need data discovery tools?</strong></p>
    <p><strong>What kind of data catalogue should teams look for?</strong></p>
    <p><strong>How to get buy in for a data catalogue?</strong></p>
    <p><strong>Which tools are here to stay?</strong></p>
    <h3>Why do you need a data catalog?</h3>
    <p>Simple data cataloging starts with a great organization. A data catalog is a collection of metadata and documentation
        that helps make sense of the data sprawl that exists in most growing companies. Getting together and starting to use
        a data catalog is a simple process, but starting to get adoption and having the&nbsp;<a
            href="https://www.secoda.co/blog/create-a-data-dictionary" style="color: #f66640;">catalog</a>&nbsp;exist as
        part of your workflow is a
        little bit more difficult.&nbsp;</p>
    <p>Even though it may seem like an easy task, getting different stakeholders to change their routines and start using a
        new tool can be very challenging. An example of the data catalog problems shared by one of the delivery companies we
        spoke with. At this company, it was difficult to get aligned on which tables were commonly used, joined, how they
        were used together and what columns meant. Similarly, it&rsquo;s difficult to monitor the number of data assets that
        exist across different departments, especially when the number of resources grows at a faster rate than people. Why
        is this the case?&nbsp;</p>
    <p>Data is becoming more decentralized through concepts like the data mesh. As more teams outside of the data function
        start to use data in their day-to-day, different tables, dashboards and definitions are being created at an almost
        exponential rate. Data catalogs are important because they help you organize your data whether you are working with
        structured or unstructured data. They help you identify what kind of data you have, how it is related to each other
        and what the best means to store it is so that you can quickly find it when needed.</p>
    <p>It happens to all data-driven companies as they grow both their data and their people. While it sounds like an easy
        problem, which might require a meeting, converging the business and data to eliminate confusion can be a very
        challenging task. That's why a data catalog can be one of the most valuable tools that a data team can use to
        start&nbsp;<a href="https://www.secoda.co/blog/net-promoter-score-for-data-teams" style="color: #f66640;">measuring
            their impact.&nbsp;</a>
    </p>
    <p>Below are the steps that teams need to take when creating a data catalog:</p>
    <h3>1. Gather sources from across the organization&nbsp;</h3>
    <p>The first step data teams need to take is to collect the different resources that are scattered across different
        tools in the organization. This may require multiple meetings and stakeholders to come together and figure out which
        resources need to be in the catalog. This collection could be done in a spreadsheet with an ongoing list of all
        resources and how they connect.</p>
    <p>It is common for teams to work with a tool that connects to multiple sources and collects the metadata from those
        sources into a central location. Using&nbsp;<a
            href="https://www.secoda.co/blog/keep-all-your-data-knowledge-in-one-place" style="color: #f66640;">data
            catalogs</a>&nbsp;makes this
        extremely easy and is a vital component of any data-driven organization. With the volume of data collected by
        businesses increasing, properly organizing it is becoming increasingly important. Priority should always be given to
        ensuring that the data is useful, but we should also take into account gains to be reaped through improved
        efficiency and collaboration.</p>
    <h3>2. Give each resource an owner</h3>
    <p>After data teams have identified all the resources from across the company that they would like to include in their
        data catalog, we recommend assigning ownership to each resource. Teams that we&rsquo;ve worked within the past have
        assigned ownership based on the source, schema or even domain. Teams that start assigning ownership should look for
        people who are familiar with the data knowledge they are responsible for managing and are willing to help others who
        want to learn how to use it.&nbsp;</p>
    <p>Ownership doesn&rsquo;t have to reside in the data team. Many products, operations and growth leaders will likely
        want to own certain pieces of&nbsp;<a
            href="https://www.secoda.co/blog/the-ultimate-guide-to-increasing-your-teams-data-literacy"
            style="color: #f66640;">company data
            knowledge</a>. By empowering these leaders as data stewards, the data team can start automating the repetitive
        questions they get from employees about how to use company data.&nbsp;</p>
    <h3>3. Get support and sign off</h3>
    <p>Once these meetings conclude and owners are on the same page, have the owners sign off on their responsibilities. The
        owners should be in alignment with the documentation and feel like the data team worked&nbsp;<a
            href="https://www.secoda.co/blog/how-to-champion-the-shift-towards-a-collaborative-data-culture"
            style="color: #f66640;">collaboratively</a>&nbsp;with
        them to come to this ownership structure. One effective strategy is to involve the leadership team in the exercise
        early to make sure that their team leads are signing off on the owners of data. This way, leadership can see how
        widespread the understanding of data is across the company. If the team leadership team sees&nbsp;<a
            href="https://www.secoda.co/blog/rethinking-the-data-catalog" style="color: #f66640;">the value of a data
            catalog</a>, this can move at
        a much faster pace.</p>
    <h3>4. Integrate the catalog base into your workflow</h3>
    <p>After data teams have received support for their data documentation process, they should look for ways to integrate
        this tool into their workflow. This step is critical for maintenance and upkeep. Without a tool that allows
        teammates to receive notifications on Slack, it will likely be forgotten. By creating a process around the data
        catalog, teams can ensure that it is not left behind as the team grows</p>
    <p>One additional piece of context that teams should look for is the usage and adoption of the data catalog across the
        company. If a tool can display the way that employees are searching and using the knowledge base, it can be a great
        source of information for the data team as they continue to iterate on the information in this central repository.
        By ensuring that there are triggers and notifications that prompt users to enter the application, teams can make
        sure that they are taking advantage of the&nbsp;<a
            href="https://www.secoda.co/blog/data-product-manager-the-role-and-best-practices-for-beginners"
            style="color: #f66640;">data catalog
            and improving it like a product.&nbsp;</a></p>
    <h3>5. Upkeep the data catalog</h3>
    <p>Although the documentation should be stable,&nbsp;<a
            href="https://www.secoda.co/blog/analytics-teams-need-more-than-traditional-documentation"
            style="color: #f66640;">it may need to change
            over time</a>. One instance that might require documentation to change is when a new revenue stream is
        introduced or when the pricing of an existing revenue line changes. These changes traditionally come from the
        business team and might require the data team to implement the changes into the data catalog.</p>
    <p>Documenting your data should be essential to any data-driven business process. It allows data practitioners to see
        how their work is used, by whom, and for what purpose. Just the act of documenting the data will help you understand
        what you have, where it is, and how it can be used. It may seem like an intimidating task at first, but I encourage
        you to start small, with a few datasets here and there, until the process becomes second nature. And once it does,
        you'll never look back!</p>
    <h3>Final thoughts</h3>
    <p>To sum it up, data catalogs are valuable for any organization, especially those growing quickly. No matter the size,
        data documentation is important in the pursuit of data operations. Having a tool that can track, document, store and
        provide metadata insights for systems beyond just your current data stack will help you identify your blind spots in
        your current architecture and integration points moving forward.&nbsp;</p>
    <p>Teams that invest the time to get alignment using a data catalog can see major benefits in the long term as they make
        faster decisions as a team. Creating a data catalog is not a small undertaking,&nbsp;<a
            href="https://www.secoda.co/blog/create-a-data-dictionary" style="color: #f66640;">that's why we make it easier
            with Secoda</a>. The
        process requires patience and alignment with leadership.&nbsp;</p>

Modern Data Stack - Sitemap