Where there is data, there is the risk of bad data. And bad data is expensive. So it’s no surprise that tools for monitoring and improving data quality are as old as databases themselves.

State of Data Quality Monitoring in 2021

How to make Business Intelligence future-proof by applying software engineering principles

Here is What Happens If You Decouple Your BI Stack

Part-I: Introducing Minerva — Airbnb’s Metric Platform

How Airbnb Achieved Metric Consistency at Scale

Before starting Falkon, I interviewed over 46 data and analytics teams to understand their day to day workflows. One of the most important…

Why metrics should be a first class citizen in a modern data stack

Data quality testing: we all recognize its importance, but sometimes have trouble figuring out exactly how it manifests itself in data engineering workflows. Ingraining data quality within CI/CD and ongoing data modeling will ensure notification upon data quality drift.

Data Quality In Data Engineering Workflows

We at Wrike are creating a roadmap with best practices to help data engineers achieve good data quality in their teams and projects.

Data Quality Roadmap. Part I

“Who’s the data owner?” Learn how to define data ownership and how a data catalog can help you document this information.

Defining Data Ownership

Data has been part of our identity as humans since the time of the ancient Romans. Our growing interconnectivity caused us to bump into…

Data & Data Engineering — the past, present, and future

Gaining trust in data with extensive data quality, accuracy and anomaly checks

How Airbnb Built “Wall” to prevent data bugs

Damn. You just discovered the latest sales report doesn’t include the new category of product and you know it’s going to take forever for…

The Case Against Self-Service Data and Analytics

Have a glimpse on what we do at Telkomsel’s Data Engineering Team and how to build a great data engineering pipeline.

Insights From Telkomsel’s Data Engineering Team — Framework, Key Principles and Tools

Looking beyond the diagram unlocks use cases for data lineage technology that could change how companies govern their data in the future.

The Future of Data Lineage — Beyond a Diagram

This book aims to give you a high-level framework and understanding of a proper modern analytics setup, and how each component interacts with each other.

Introduction | The Analytics Setup Guidebook

📚 Restructure your knowledge of the complex data analytics landscape, and learn how to build scalable analytics & BI stacks

The Analytics Setup Guidebook: Build Scalable Data Analytics and BI Stacks in the Modern Cloud Era

Now teams are adopting yet another new approach, called “reverse ETL,” the process of moving data from a data warehouse into 3rd-party systems to make data operational. The emergence of reverse ETL solutions is a useful component of the stack to get better leverage out of data.

What is Reverse ETL?

Over the past 9 months, I’ve been working on a book to be published by O’Reilly Media. This past week, Data Pipelines Pocket Reference was officially published in print, e-book, and on …

My New Book: Data Pipelines Pocket Reference

A new generation of data tooling and infrastructure companies are enabling companies to create value with their data.

The Data Opportunity: the Emergence of a New Layer in Modern Enterprise

What your data team is using: the analytics stack

Technically: the analytics stack

If you're looking to build the ideal modern data stack for analytics using only open-source options, this is the blog for you. Find all the best open-source alternatives to your favorite paid tools.

The Modern Data Stack: Open-source Edition

Fivetran co-founder and CEO George Fraser shares the importance of the modern data stack and five developments he’s eagerly following.

Five Reasons To Consider A Modern Data Stack | Blog | Fivetran

Data pipelining is the act of moving data from one location to another and is one of the most critical operations in data infrastructure. Data pipelines are a superset of Extract, Transform, Load…

Data Pipelining — A Primer

Learn how to get the value out of your organizations data. The analytics stack is the link between raw data and business intelligence, a platform for business growth and innovation.

Building an Analytics Stack: A Guide

The following is based on a talk we gave to the Y Combinator W21 batch on how to get started with data, analytics, and business…

No Nonsense Intro to Data, Analytics, & Data Stacks

A simple example and checking its capabilities and features. Is it worth the hype?

Airbyte — Worth the hype?

Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape

Headless BI is a new way of looking at business intelligence powered by cloud-based data warehouses.

Headless Business Intelligence

Dashboards and reports are the lingua franca in the world of business. Simple as they may seem, behind each KPI dashboard are data analysts who are responsible for keeping dashboards working…

Data Discovery for Business Intelligence

The feature store is the central place to store curated features for machine learning pipelines, FSML aims to create content for information and knowledge in the ever evolving feature store's world and surrounding data and AI environment.

Feature Store for ML

Learnings from dozens of companies on how to make your data catalog successful

How to Make Your Data Catalog Successful

Evolution of data warehouses and technical analysis of why Snowflake has a major technical advantage over others

The #1 Reason Snowflake is Different

Forbes has rightly coined a term for this era which is “Age of (too much) data” and as such it is becoming imperative for organizations and…

Stop using the terms Data Lake and Data Warehouse interchangeably!

What could work for true data democratization at scale?

The Missing Piece of Data Discovery and Observability Platforms: Open Standard for Metadata

Apple’s Data Platform for ML

A look at the entire ecosystem of data engineering tools and technologies.

The State of Data Engineering in 2021

What are the Key Elements that are used to meet the Challenges?

The Secret to Modernizing Data Analytics Platforms

Is it analytics? Is it engineering? What do these people do?

Analytics Engineer: The Newest Data Career Role

This post follows up on The Rise of the Data Engineer, a recent post that was an attempt at defining data engineering and described how…

The Downfall of the Data Engineer

As more startups collect data at an earlier stage, many companies are thinking about their analytics stack earlier in their life cycle. How to set up your data stack is a common question for early-stage companies. This is understandable, as most early-stage companies rely on analysis to gather insights to help them grow and these insights depend on clean and accessible analytics.

How Can Startups Adopt a Modern Data Stack?

Key areas which will transform data stacks

How the modern data stack will transform in 2021 and beyond

Instead of using dashboards or reports to understand your data, operational analytics drives action by automatically delivering real-time data to the exact place it’ll be most useful.

What Is Operational Analytics?

How founders and startups can take advantage of the next generation in data infrastructure

What Is Driving Data 3.0?

Despite benefits of high-quality data available, most companies are still in the process of developing their data governance systems

Key Challenges For Data Governance

What does a world class analytics stack look like in 2019?

How to set up a modern analytics stack

It’s time for a modern metadata solution, one that is just as fast, flexible, and scalable as the rest of the modern data stack.

Data Catalog 3.0: Modern Metadata for the Modern Data Stack

Data pipeline orchestration is traditionally engineering-heavy, but a modern data stack can free your engineers for higher-value projects.

Data Orchestration Explained – and Why You Shouldn't DIY | Blog | Fivetran

When we invest at Series A startups are often at the point where Business Intelligence is still primarily run with a combination of Excel…

A Modern Business Intelligence Stack for Startups

The data infrastructure is changing, and data warehouses are at the center of it all. We dig into why, plus where the industry is headed.

The Ecosystem (and Future) of the Modern Data Infrastructure – Indicative

Is BI dead?

Today, most organisations build their data platforms following this ELT approach where they create a centralised data asset in their cloud data warehouse or lake, combining all their source data, that acts as the source of truth for all their business systems and appreciates in value over time. 

The modern data stack in 2021

Data teams love calculating and tracking everything with metrics. We already have the infrastructure in place to do so… yet often fail to apply the same strategy for our own work. Let’s fix that!

How To Measure a Data Engineering Team

A lot of organisations still struggling to become data-driven. There are a lot of issues due to data silos, scalability problems with data infrastructure, and difficulty to use analytics. In his blog, Francois explained how these issues are being solved by the partnership of Google & Tableau.

Self-Service Analytics for Google Cloud, now with Looker and Tableau

Inefficient data operations within an organization or an inefficient DataOps team can have an impact on both data inputs and the outputs that are generated. So it becomes very  important to have a data ops team that is efficient & well organized. In his article, Eitan has discussed how you can achieve this at your organization.

The ideal DataOps org structure

Though organizations have a large amount of data at their disposal to build and train their ML models, there are certain challenges that also arise due to this large amount of data - data access approaches, the need to migrate and/or reformat data before analysis can begin, and the adoption of different operating models to name a few. 

How to Simplify Your Approach to Data Analytics

A journey through my learnings of the world of Data — this article describes how data is treated and consumed in a product organization, and how various professional roles interact with data at various stages of its lifecycle

An experience of a “Data Ecosystem”

The tech giants have build unique architectures to manage datasets in large scale machine learning solutions.

How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine…

At Rill, we believe the need for operational intelligence will dramatically expand in the coming years. In this post we lay out why operational intelligence matters now, its salient differences with traditional business intelligence, and why it demands new technology architectures.

Operational intelligence and the new frontier of data

Note: This article was written by Oz Katz and was published on the lakeFS blog on 4/12/21.

Hudi, Iceberg and Delta Lake: Data Lake Table Formats Compared

Uber delivers efficient and reliable transportation across the global marketplace, which is powered by hundreds of services, machine learning models, and tens of thousands of datasets. While growing rapidly, we’re also committed to maintaining data quality, as it can greatly impact business operations and decisions. Without data quality guarantees, downstream service computation or machine learning model performance quickly degrade, which requires a lot of laborious manual efforts to investigate and backfill poor data. In the worst cases, degradations could go unnoticed, silently resulting in inconsistent behaviors.

How Uber Achieves Operational Excellence in the Data Quality Experience

For many years, Snowflake has been the byword for the entire cloud data warehouse market. The company’s growth stands testament to how easy…

Why we invested in Firebolt: Snowflake catapulted the data warehouse into the cloud.

A look back at a handful of data projects delivered by the data science, data engineering and analytics team here at Gousto in 2020.

Gousto Data Team — Best of 2020

The People Who Reinvented the Notebook at Netflix are Back To Reinvent the Notebook

Noteable: The Interactive Notebook Document for Modern Data Teams

Yet Another Blog on Data

My thoughts on where our space has been and where it might be going.

The Modern Data Stack: Past, Present, and Future

Data plays a critical role in business decisions, AI/ML, product evolution and much more. Timeliness, accuracy, and reliability are the…

Superglue — Journey of Lineage, Data Observability & Data Pipelines

Would you hire a Python/Airflow expert OR a Database/SQL expert for a Data Engineer position?

Data Engineers are no longer “Data” folks

The cloud analytics stack allows businesses of any size to quickly maximize the value of their data with its cost-effective, elastic architecture and near-infinite scale.

Building A Modern Cloud Analytics Stack: A Playbook for Data-driven Companies | Sigma Computing

As volumes of collected data increase exponentially, methods to ensure data quality for big data are critical to accurate and effective business decisions.

Data quality for big data: Why it's a must and how to improve it

In Part 2 of the Data Quality for Everyone, Everywhere series, learn about the 5 main reasons why data quality can be so inconsistent throughout the enterprise.

Why Data Quality Does Not Extend to Everyone, Everywhere Today | Informatica | Informatica

225 votes and 19 comments so far on Reddit

r/dataengineering - 1 year of must-read articles

Metriql doesn't have a user interface that lets you analyze the data because it's Headless BI. However, it comes with a dashboard that enables you to use the integrations that we have. Here is how it looks like:

How we built the Metriql public demo

On Redshifts, Data Catalogs, Query Engines like Presto, and the troubles of machine learning engineers to get their data.

There’s More Than One Kind of Data Mesh — Three Types of Data Meshes

What is necessary and sufficient to understand the state of your data?

The Four Pillars of Data Observability

Exploring the areas within sports that are the most receptive to data science solutions

October Edition: Data Science Meets Sports

Modern Data Stack is paving the way for a data-driven future. The modern data stack consists of a lot of elements that are rapidly growing each year and its ever-increasing popularity in the data-driven business world has made an impact. The team at Continual has done a great job with this article that shares what is in the  "Modern Data Stack" ecosystem in 2021.

The Modern Data Stack Ecosystem - Fall 2021 Edition

 In this edition of dbt roundup, Anna has discussed some crucial issues related to data warehouse. It's a great read with a spooky Halloween touch. Three data horror stories that are guaranteed to keep you up tonight: Zombie data, Knowledge spiderwebs, Data discovery witchcraft.

What's haunting your data warehouse?

The terms “data fabric” and “data mesh” are often used conversely as both provide an architecture to access data across various platforms. The only difference is that a data fabric deals with technology, while a data mesh deals with organizational changes.

Data Fabric vs. Data Mesh

Our cool new house needs one more plank in its foundation.

The missing piece of the modern data stack

A few thoughts on Airbnb’s metrics layer.

Is Minerva the answer?

In this tutorial we will build a Slackbot using JavaScript/Node.js that reaches out to 2 public APIs and tells random jokes within Slack. We will use the Sla...

Build A Slackbot

5 Trends In Data & What Buyers Are Looking For In New Tools

Unpacking The Data Hype

Why Vectorized’s focus on developer experience will unlock real-time streaming for the great majority of developers

Why Developers Love Redpanda

Nowadays most companies sit on lots of very valuable data, and as most refer to, data is the new gold. Data pipelines have evolved…

Reverse ETL — an approach to Operational Data Insights

In Part-1, I explained what is Data Quality in Big data ecosystem and what are the challenges in ensuring data quality with the data…

Data Quality: The holy grail of a Data Fluent organisation | Part-2

Data has evolved to be the core foundation of most organisations. Data Quality ensures that this core foundation is giving us correct…

Data Quality: The holy grail of a Data Fluent organisation | Part-1

For the last 6 months we have been working tirelessly to solve the problem of data collaboration in the enterprise and now we have our MVP…

A brief history of Paradime

Our approach for reducing data issues by a factor of 5, and our lead times for dashboards & reports by a factor of 10.

How Mercateo is Rolling Out a Modern Data Platform

This is what the professionals want you to stop doing.

The Bad Habits To Break Now: Data & Analytics Leaders

In 2021 we began to form an Analytics Engineering team to support and complement self service. This post covers how we spin up this new…

Spinning up an Analytics Engineering Team

How a revolution comes together. Or doesn’t.

The Modern Data Experience

A radically new approach to data integration saves engineering time, allowing engineers and analysts to pursue higher-value activities.

What Is the Modern Data Stack? | Blog | Fivetran

There is a massive opportunity in startup land. Startups are leveraging Cloud DataWarehouses to build the modern Data Stack.

Why Snowflake could be the next Great Platform

How to migrate from a single tool to many tools while keeping your data clean and accurate to drive your business.

Graduating to the Modern Data Stack for Startups | Census

A Comprehensive Glossary For Data Engineers

150+ Concepts Heard in Data Engineering

What if … You had to build a data stack from scratch?

Welcome to The Data Source, your monthly newsletter covering the top innovation in data engineering, analytics and developer-first tooling. Subscribe now and never miss an issue 👇🏼 Looking back to the past 5 or so years in data infrastructure, there’s been significant innovation in tooling and engineering best practices that have contributed to what is known today as the Modern Data Stack.

Resources

Articles

Videos

Data Observability - Barr Moses

An Approach to Data Quality for Netflix Personalization Systems

Data Mesh in Practice: How Europe's Leading Online Platform for Fashion Goes Beyond the Data Lake

Reverse ETL on Airflow

Why Develop Data Products on Top of Headless BI

Looker and BigQuery: Enterprise digital transformation at Sunrun

Taylor Brown and George Fraser, Co-Founders of Fivetran, keynote the Modern Data Stack Conference

Introduction to headless BI - Webinar

The Modern Data Analytics Stack for AI-Driven Decision Intelligence

PLG is for Sales

Did we miss anything?