Where there is data, there is the risk of bad data. And bad data is expensive. So it’s no surprise that tools for monitoring and improving data quality are as old as databases themselves.

State of Data Quality Monitoring in 2021

How to make Business Intelligence future-proof by applying software engineering principles

Here is What Happens If You Decouple Your BI Stack

The tech giants have build unique architectures to manage datasets in large scale machine learning solutions.

How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine…

Our cool new house needs one more plank in its foundation.

The missing piece of the modern data stack

At Rill, we believe the need for operational intelligence will dramatically expand in the coming years. In this post we lay out why operational intelligence matters now, its salient differences with traditional business intelligence, and why it demands new technology architectures.

Operational intelligence and the new frontier of data

Part-I: Introducing Minerva — Airbnb’s Metric Platform

How Airbnb Achieved Metric Consistency at Scale

Note: This article was written by Oz Katz and was published on the lakeFS blog on 4/12/21.

Hudi, Iceberg and Delta Lake: Data Lake Table Formats Compared

Headless BI is a new way of looking at business intelligence powered by cloud-based data warehouses.

Headless Business Intelligence

A few thoughts on Airbnb’s metrics layer.

Is Minerva the answer?

Dashboards and reports are the lingua franca in the world of business. Simple as they may seem, behind each KPI dashboard are data analysts who are responsible for keeping dashboards working…

Data Discovery for Business Intelligence

Before starting Falkon, I interviewed over 46 data and analytics teams to understand their day to day workflows. One of the most important…

Why metrics should be a first class citizen in a modern data stack

The feature store is the central place to store curated features for machine learning pipelines, FSML aims to create content for information and knowledge in the ever evolving feature store's world and surrounding data and AI environment.

Feature Store for ML

Data plays a critical role in business decisions, AI/ML, product evolution and much more. Timeliness, accuracy, and reliability are the…

Superglue — Journey of Lineage, Data Observability & Data Pipelines

In this tutorial we will build a Slackbot using JavaScript/Node.js that reaches out to 2 public APIs and tells random jokes within Slack. We will use the Sla...

Build A Slackbot

Data quality testing: we all recognize its importance, but sometimes have trouble figuring out exactly how it manifests itself in data engineering workflows. Ingraining data quality within CI/CD and ongoing data modeling will ensure notification upon data quality drift.

Data Quality In Data Engineering Workflows

5 Trends In Data & What Buyers Are Looking For In New Tools

Unpacking The Data Hype

Learnings from dozens of companies on how to make your data catalog successful

How to Make Your Data Catalog Successful

Why Vectorized’s focus on developer experience will unlock real-time streaming for the great majority of developers

Why Developers Love Redpanda

We at Wrike are creating a roadmap with best practices to help data engineers achieve good data quality in their teams and projects.

Data Quality Roadmap. Part I

Uber delivers efficient and reliable transportation across the global marketplace, which is powered by hundreds of services, machine learning models, and tens of thousands of datasets. While growing rapidly, we’re also committed to maintaining data quality, as it can greatly impact business operations and decisions. Without data quality guarantees, downstream service computation or machine learning model performance quickly degrade, which requires a lot of laborious manual efforts to investigate and backfill poor data. In the worst cases, degradations could go unnoticed, silently resulting in inconsistent behaviors.

How Uber Achieves Operational Excellence in the Data Quality Experience

Nowadays most companies sit on lots of very valuable data, and as most refer to, data is the new gold. Data pipelines have evolved…

Reverse ETL — an approach to Operational Data Insights

In Part-1, I explained what is Data Quality in Big data ecosystem and what are the challenges in ensuring data quality with the data…

Data Quality: The holy grail of a Data Fluent organisation | Part-2

“Who’s the data owner?” Learn how to define data ownership and how a data catalog can help you document this information.

Defining Data Ownership

For many years, Snowflake has been the byword for the entire cloud data warehouse market. The company’s growth stands testament to how easy…

Why we invested in Firebolt: Snowflake catapulted the data warehouse into the cloud.

Data has evolved to be the core foundation of most organisations. Data Quality ensures that this core foundation is giving us correct…

Data Quality: The holy grail of a Data Fluent organisation | Part-1

Evolution of data warehouses and technical analysis of why Snowflake has a major technical advantage over others

The #1 Reason Snowflake is Different

Forbes has rightly coined a term for this era which is “Age of (too much) data” and as such it is becoming imperative for organizations and…

Stop using the terms Data Lake and Data Warehouse interchangeably!

What could work for true data democratization at scale?

The Missing Piece of Data Discovery and Observability Platforms: Open Standard for Metadata

For the last 6 months we have been working tirelessly to solve the problem of data collaboration in the enterprise and now we have our MVP…

A brief history of Paradime

A look back at a handful of data projects delivered by the data science, data engineering and analytics team here at Gousto in 2020.

Gousto Data Team — Best of 2020

Apple’s Data Platform for ML

Our approach for reducing data issues by a factor of 5, and our lead times for dashboards & reports by a factor of 10.

How Mercateo is Rolling Out a Modern Data Platform

A look at the entire ecosystem of data engineering tools and technologies.

The State of Data Engineering in 2021

What are the Key Elements that are used to meet the Challenges?

The Secret to Modernizing Data Analytics Platforms

This is what the professionals want you to stop doing.

The Bad Habits To Break Now: Data & Analytics Leaders

Data has been part of our identity as humans since the time of the ancient Romans. Our growing interconnectivity caused us to bump into…

Data & Data Engineering — the past, present, and future

The People Who Reinvented the Notebook at Netflix are Back To Reinvent the Notebook

Noteable: The Interactive Notebook Document for Modern Data Teams

Gaining trust in data with extensive data quality, accuracy and anomaly checks

How Airbnb Built “Wall” to prevent data bugs

Is it analytics? Is it engineering? What do these people do?

Analytics Engineer: The Newest Data Career Role

Damn. You just discovered the latest sales report doesn’t include the new category of product and you know it’s going to take forever for…

The Case Against Self-Service Data and Analytics

This post follows up on The Rise of the Data Engineer, a recent post that was an attempt at defining data engineering and described how…

The Downfall of the Data Engineer

In 2021 we began to form an Analytics Engineering team to support and complement self service. This post covers how we spin up this new…

Spinning up an Analytics Engineering Team

Have a glimpse on what we do at Telkomsel’s Data Engineering Team and how to build a great data engineering pipeline.

Insights From Telkomsel’s Data Engineering Team — Framework, Key Principles and Tools

Looking beyond the diagram unlocks use cases for data lineage technology that could change how companies govern their data in the future.

The Future of Data Lineage — Beyond a Diagram

How a revolution comes together. Or doesn’t.

The Modern Data Experience

As more startups collect data at an earlier stage, many companies are thinking about their analytics stack earlier in their life cycle. How to set up your data stack is a common question for early-stage companies. This is understandable, as most early-stage companies rely on analysis to gather insights to help them grow and these insights depend on clean and accessible analytics.

How Can Startups Adopt a Modern Data Stack?

This book aims to give you a high-level framework and understanding of a proper modern analytics setup, and how each component interacts with each other.

Introduction | The Analytics Setup Guidebook

Key areas which will transform data stacks

How the modern data stack will transform in 2021 and beyond

Yet Another Blog on Data

My thoughts on where our space has been and where it might be going.

The Modern Data Stack: Past, Present, and Future

A radically new approach to data integration saves engineering time, allowing engineers and analysts to pursue higher-value activities.

What Is the Modern Data Stack? | Blog | Fivetran

📚 Restructure your knowledge of the complex data analytics landscape, and learn how to build scalable analytics & BI stacks

The Analytics Setup Guidebook: Build Scalable Data Analytics and BI Stacks in the Modern Cloud Era

Now teams are adopting yet another new approach, called “reverse ETL,” the process of moving data from a data warehouse into 3rd-party systems to make data operational. The emergence of reverse ETL solutions is a useful component of the stack to get better leverage out of data.

What is Reverse ETL?

Instead of using dashboards or reports to understand your data, operational analytics drives action by automatically delivering real-time data to the exact place it’ll be most useful.

What Is Operational Analytics?

Would you hire a Python/Airflow expert OR a Database/SQL expert for a Data Engineer position?

Data Engineers are no longer “Data” folks

How founders and startups can take advantage of the next generation in data infrastructure

What Is Driving Data 3.0?

There is a massive opportunity in startup land. Startups are leveraging Cloud DataWarehouses to build the modern Data Stack.

Why Snowflake could be the next Great Platform

Despite benefits of high-quality data available, most companies are still in the process of developing their data governance systems

Key Challenges For Data Governance

How to migrate from a single tool to many tools while keeping your data clean and accurate to drive your business.

Graduating to the Modern Data Stack for Startups | Census

Over the past 9 months, I’ve been working on a book to be published by O’Reilly Media. This past week, Data Pipelines Pocket Reference was officially published in print, e-book, and on …

My New Book: Data Pipelines Pocket Reference

What does a world class analytics stack look like in 2019?

How to set up a modern analytics stack

A new generation of data tooling and infrastructure companies are enabling companies to create value with their data.

The Data Opportunity: the Emergence of a New Layer in Modern Enterprise

What your data team is using: the analytics stack

Technically: the analytics stack

If you're looking to build the ideal modern data stack for analytics using only open-source options, this is the blog for you. Find all the best open-source alternatives to your favorite paid tools.

The Modern Data Stack: Open-source Edition

The data infrastructure is changing, and data warehouses are at the center of it all. We dig into why, plus where the industry is headed.

The Ecosystem (and Future) of the Modern Data Infrastructure – Indicative

Fivetran co-founder and CEO George Fraser shares the importance of the modern data stack and five developments he’s eagerly following.

Five Reasons To Consider A Modern Data Stack | Blog | Fivetran

It’s time for a modern metadata solution, one that is just as fast, flexible, and scalable as the rest of the modern data stack.

Data Catalog 3.0: Modern Metadata for the Modern Data Stack

Data pipelining is the act of moving data from one location to another and is one of the most critical operations in data infrastructure. Data pipelines are a superset of Extract, Transform, Load…

Data Pipelining — A Primer

A Comprehensive Glossary For Data Engineers

150+ Concepts Heard in Data Engineering

The cloud analytics stack allows businesses of any size to quickly maximize the value of their data with its cost-effective, elastic architecture and near-infinite scale.

Building A Modern Cloud Analytics Stack: A Playbook for Data-driven Companies | Sigma Computing

Data pipeline orchestration is traditionally engineering-heavy, but a modern data stack can free your engineers for higher-value projects.

Data Orchestration Explained – and Why You Shouldn't DIY | Blog | Fivetran

What if … You had to build a data stack from scratch?

As volumes of collected data increase exponentially, methods to ensure data quality for big data are critical to accurate and effective business decisions.

Data quality for big data: Why it's a must and how to improve it

When we invest at Series A startups are often at the point where Business Intelligence is still primarily run with a combination of Excel…

A Modern Business Intelligence Stack for Startups

Welcome to The Data Source, your monthly newsletter covering the top innovation in data engineering, analytics and developer-first tooling. Subscribe now and never miss an issue 👇🏼 Looking back to the past 5 or so years in data infrastructure, there’s been significant innovation in tooling and engineering best practices that have contributed to what is known today as the Modern Data Stack.

The Data Source #3 |  💥 Two Trends Shaping the Modern Data Stack 

Learn how to get the value out of your organizations data. The analytics stack is the link between raw data and business intelligence, a platform for business growth and innovation.

Building an Analytics Stack: A Guide

The following is based on a talk we gave to the Y Combinator W21 batch on how to get started with data, analytics, and business…

No Nonsense Intro to Data, Analytics, & Data Stacks

In Part 2 of the Data Quality for Everyone, Everywhere series, learn about the 5 main reasons why data quality can be so inconsistent throughout the enterprise.

Why Data Quality Does Not Extend to Everyone, Everywhere Today | Informatica | Informatica

Data governance can enhance various areas of your business, from your enhancing digital marketing strategies to positively impacting your financials.

7 Ways Data Governance Can Benefit Your Business – Around the Web

Explore tools to organize and govern data to ensure quality and searchability

Implement data governance to manage and secure clients' data

225 votes and 19 comments so far on Reddit

r/dataengineering - 1 year of must-read articles

Is BI dead?

Metriql doesn't have a user interface that lets you analyze the data because it's Headless BI. However, it comes with a dashboard that enables you to use the integrations that we have. Here is how it looks like:

How we built the Metriql public demo

A simple example and checking its capabilities and features. Is it worth the hype?

Airbyte — Worth the hype?

Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape

DISCOVER THE DIFFERENCE BETWEEN A DATA LAKE AND A DATA WAREHOUSE.

The 9 Differences Between a Data Lake and a Data Warehouse

Today, most organisations build their data platforms following this ELT approach where they create a centralised data asset in their cloud data warehouse or lake, combining all their source data, that acts as the source of truth for all their business systems and appreciates in value over time. 

The modern data stack in 2021

How a changing technical environment has shifted power away from decision-makers

Are Decision-Makers Disempowered by the Modern Data Stack?

Data teams love calculating and tracking everything with metrics. We already have the infrastructure in place to do so… yet often fail to apply the same strategy for our own work. Let’s fix that!

How To Measure a Data Engineering Team

 In her article Operationalization: the art and science of making metrics, Cassie talks about the essential psychological elements important for all data professionals to make metrics that can be defined and named!

Operationalization: the art and science of making metrics

A lot of organisations still struggling to become data-driven. There are a lot of issues due to data silos, scalability problems with data infrastructure, and difficulty to use analytics. In his blog, Francois explained how these issues are being solved by the partnership of Google & Tableau.

Self-Service Analytics for Google Cloud, now with Looker and Tableau

This article is about how HomeToGo uses dbt as its main tool for all data transformations in its ELT process and creating a scalable framework for this process

DBT at HomeToGo

On Redshifts, Data Catalogs, Query Engines like Presto, and the troubles of machine learning engineers to get their data.

There’s More Than One Kind of Data Mesh — Three Types of Data Meshes

A better way to make daily decisions is already here—just not at work.

The future of operational analytics

What is necessary and sufficient to understand the state of your data?

Resources

Articles

Videos

Looker and BigQuery: Enterprise digital transformation at Sunrun

An Approach to Data Quality for Netflix Personalization Systems

Augmented Analytics Explained

Augmented Analytics and the Future of Data Analytics

The Changing Landscape of Data Collaboration

Taylor Brown and George Fraser, Co-Founders of Fivetran, keynote the Modern Data Stack Conference

Introduction to headless BI - Webinar

The Modern Data Analytics Stack for AI-Driven Decision Intelligence

Data Mesh in Practice: How Europe's Leading Online Platform for Fashion Goes Beyond the Data Lake

Data Cataloging with Knowledge Graphs

Did we miss anything?