Why Vectorized’s focus on developer experience will unlock real-time streaming for the great majority of developers

Why Developers Love Redpanda

In Part-1, I explained what is Data Quality in Big data ecosystem and what are the challenges in ensuring data quality with the data…

Data Quality: The holy grail of a Data Fluent organisation | Part-2

For the last 6 months we have been working tirelessly to solve the problem of data collaboration in the enterprise and now we have our MVP…

A brief history of Paradime

A look back at a handful of data projects delivered by the data science, data engineering and analytics team here at Gousto in 2020.

Gousto Data Team — Best of 2020

Looking beyond the diagram unlocks use cases for data lineage technology that could change how companies govern their data in the future.

The Future of Data Lineage — Beyond a Diagram

How a changing technical environment has shifted power away from decision-makers

Are Decision-Makers Disempowered by the Modern Data Stack?

On Redshifts, Data Catalogs, Query Engines like Presto, and the troubles of machine learning engineers to get their data.

There’s More Than One Kind of Data Mesh — Three Types of Data Meshes

How a revolution comes together. Or doesn’t.

The Modern Data Experience

A Comprehensive Glossary For Data Engineers

150+ Concepts Heard in Data Engineering

 In her article Operationalization: the art and science of making metrics, Cassie talks about the essential psychological elements important for all data professionals to make metrics that can be defined and named!

Operationalization: the art and science of making metrics

A journey through my learnings of the world of Data — this article describes how data is treated and consumed in a product organization, and how various professional roles interact with data at various stages of its lifecycle

An experience of a “Data Ecosystem”

Headless BI is a new way of looking at business intelligence powered by cloud-based data warehouses.

Headless Business Intelligence

Dashboards and reports are the lingua franca in the world of business. Simple as they may seem, behind each KPI dashboard are data analysts who are responsible for keeping dashboards working…

Data Discovery for Business Intelligence

As more startups collect data at an earlier stage, many companies are thinking about their analytics stack earlier in their life cycle. How to set up your data stack is a common question for early-stage companies. This is understandable, as most early-stage companies rely on analysis to gather insights to help them grow and these insights depend on clean and accessible analytics.

How Can Startups Adopt a Modern Data Stack?

Key areas which will transform data stacks

How the modern data stack will transform in 2021 and beyond

Despite benefits of high-quality data available, most companies are still in the process of developing their data governance systems

Key Challenges For Data Governance

The data infrastructure is changing, and data warehouses are at the center of it all. We dig into why, plus where the industry is headed.

The Ecosystem (and Future) of the Modern Data Infrastructure – Indicative

Uber delivers efficient and reliable transportation across the global marketplace, which is powered by hundreds of services, machine learning models, and tens of thousands of datasets. While growing rapidly, we’re also committed to maintaining data quality, as it can greatly impact business operations and decisions. Without data quality guarantees, downstream service computation or machine learning model performance quickly degrade, which requires a lot of laborious manual efforts to investigate and backfill poor data. In the worst cases, degradations could go unnoticed, silently resulting in inconsistent behaviors.

How Uber Achieves Operational Excellence in the Data Quality Experience

For many years, Snowflake has been the byword for the entire cloud data warehouse market. The company’s growth stands testament to how easy…

Why we invested in Firebolt: Snowflake catapulted the data warehouse into the cloud.

What are the Key Elements that are used to meet the Challenges?

The Secret to Modernizing Data Analytics Platforms

This post follows up on The Rise of the Data Engineer, a recent post that was an attempt at defining data engineering and described how…

The Downfall of the Data Engineer

When we invest at Series A startups are often at the point where Business Intelligence is still primarily run with a combination of Excel…

A Modern Business Intelligence Stack for Startups

Modern Data Stack is paving the way for a data-driven future. The modern data stack consists of a lot of elements that are rapidly growing each year and its ever-increasing popularity in the data-driven business world has made an impact. The team at Continual has done a great job with this article that shares what is in the  "Modern Data Stack" ecosystem in 2021.

The Modern Data Stack Ecosystem - Fall 2021 Edition

The terms “data fabric” and “data mesh” are often used conversely as both provide an architecture to access data across various platforms. The only difference is that a data fabric deals with technology, while a data mesh deals with organizational changes.

Data Fabric vs. Data Mesh

Have a glimpse on what we do at Telkomsel’s Data Engineering Team and how to build a great data engineering pipeline.

Insights From Telkomsel’s Data Engineering Team — Framework, Key Principles and Tools

Fivetran co-founder and CEO George Fraser shares the importance of the modern data stack and five developments he’s eagerly following.

Five Reasons To Consider A Modern Data Stack | Blog | Fivetran

Our cool new house needs one more plank in its foundation.

The missing piece of the modern data stack

We at Wrike are creating a roadmap with best practices to help data engineers achieve good data quality in their teams and projects.

Data Quality Roadmap. Part I

Our approach for reducing data issues by a factor of 5, and our lead times for dashboards & reports by a factor of 10.

How Mercateo is Rolling Out a Modern Data Platform

This is what the professionals want you to stop doing.

The Bad Habits To Break Now: Data & Analytics Leaders

Data has been part of our identity as humans since the time of the ancient Romans. Our growing interconnectivity caused us to bump into…

Data & Data Engineering — the past, present, and future

In 2021 we began to form an Analytics Engineering team to support and complement self service. This post covers how we spin up this new…

Spinning up an Analytics Engineering Team

What your data team is using: the analytics stack

Technically: the analytics stack

Where there is data, there is the risk of bad data. And bad data is expensive. So it’s no surprise that tools for monitoring and improving data quality are as old as databases themselves.

State of Data Quality Monitoring in 2021

Data governance can enhance various areas of your business, from your enhancing digital marketing strategies to positively impacting your financials.

7 Ways Data Governance Can Benefit Your Business – Around the Web

A better way to make daily decisions is already here—just not at work.

The future of operational analytics

n her blog, she discusses the common mistakes that new data recruits make. A detailed explanation about how these mistakes can be avoided and what can be done instead. She has also given a few suggestions that will be beneficial for a beginner's data journey. 

Tough Love for Naïve Data Newbies

Note: This article was written by Oz Katz and was published on the lakeFS blog on 4/12/21.

Hudi, Iceberg and Delta Lake: Data Lake Table Formats Compared

A few thoughts on Airbnb’s metrics layer.

Is Minerva the answer?

Data plays a critical role in business decisions, AI/ML, product evolution and much more. Timeliness, accuracy, and reliability are the…

Superglue — Journey of Lineage, Data Observability & Data Pipelines

Nowadays most companies sit on lots of very valuable data, and as most refer to, data is the new gold. Data pipelines have evolved…

Reverse ETL — an approach to Operational Data Insights

There is a massive opportunity in startup land. Startups are leveraging Cloud DataWarehouses to build the modern Data Stack.

Why Snowflake could be the next Great Platform

Welcome to The Data Source, your monthly newsletter covering the top innovation in data engineering, analytics and developer-first tooling. Subscribe now and never miss an issue 👇🏼 Looking back to the past 5 or so years in data infrastructure, there’s been significant innovation in tooling and engineering best practices that have contributed to what is known today as the Modern Data Stack.

The Data Source #3 |  💥 Two Trends Shaping the Modern Data Stack 

Explore tools to organize and govern data to ensure quality and searchability

Implement data governance to manage and secure clients' data

Metriql doesn't have a user interface that lets you analyze the data because it's Headless BI. However, it comes with a dashboard that enables you to use the integrations that we have. Here is how it looks like:

How we built the Metriql public demo

DISCOVER THE DIFFERENCE BETWEEN A DATA LAKE AND A DATA WAREHOUSE.

The 9 Differences Between a Data Lake and a Data Warehouse

This article is about how HomeToGo uses dbt as its main tool for all data transformations in its ELT process and creating a scalable framework for this process

DBT at HomeToGo

In this article Nicholas has talked about how he redesigned over 100 ETL into ELT data pipelines due reasons like cost & scalability. He has discussed the "How" & "Why" of this entire process in detail. 

How I Redesigned over 100 ETL into ELT Data Pipelines

At Rill, we believe the need for operational intelligence will dramatically expand in the coming years. In this post we lay out why operational intelligence matters now, its salient differences with traditional business intelligence, and why it demands new technology architectures.

Operational intelligence and the new frontier of data

Gaining trust in data with extensive data quality, accuracy and anomaly checks

How Airbnb Built “Wall” to prevent data bugs

As volumes of collected data increase exponentially, methods to ensure data quality for big data are critical to accurate and effective business decisions.

Data quality for big data: Why it's a must and how to improve it

 In this edition of dbt roundup, Anna has discussed some crucial issues related to data warehouse. It's a great read with a spooky Halloween touch. Three data horror stories that are guaranteed to keep you up tonight: Zombie data, Knowledge spiderwebs, Data discovery witchcraft.

What's haunting your data warehouse?

How to make Business Intelligence future-proof by applying software engineering principles

Here is What Happens If You Decouple Your BI Stack

Before starting Falkon, I interviewed over 46 data and analytics teams to understand their day to day workflows. One of the most important…

Why metrics should be a first class citizen in a modern data stack

This book aims to give you a high-level framework and understanding of a proper modern analytics setup, and how each component interacts with each other.

Introduction | The Analytics Setup Guidebook

Over the past 9 months, I’ve been working on a book to be published by O’Reilly Media. This past week, Data Pipelines Pocket Reference was officially published in print, e-book, and on …

My New Book: Data Pipelines Pocket Reference

A new generation of data tooling and infrastructure companies are enabling companies to create value with their data.

The Data Opportunity: the Emergence of a New Layer in Modern Enterprise

If you're looking to build the ideal modern data stack for analytics using only open-source options, this is the blog for you. Find all the best open-source alternatives to your favorite paid tools.

The Modern Data Stack: Open-source Edition

5 Trends In Data & What Buyers Are Looking For In New Tools

Unpacking The Data Hype

A radically new approach to data integration saves engineering time, allowing engineers and analysts to pursue higher-value activities.

What Is the Modern Data Stack? | Blog | Fivetran

How to migrate from a single tool to many tools while keeping your data clean and accurate to drive your business.

Graduating to the Modern Data Stack for Startups | Census

What if … You had to build a data stack from scratch?

A lot has improved in the data world in recent years, still the core problems remain a pain point for organizations. And the solution to these problems lies in the way organizations handle their data. The next big challenge for data is an organizational one i.e, how scaled companies organize their data teams.

The Next Big Challenge for Data is Organizational

In this tutorial we will build a Slackbot using JavaScript/Node.js that reaches out to 2 public APIs and tells random jokes within Slack. We will use the Sla...

Build A Slackbot

Data has evolved to be the core foundation of most organisations. Data Quality ensures that this core foundation is giving us correct…

Data Quality: The holy grail of a Data Fluent organisation | Part-1

Evolution of data warehouses and technical analysis of why Snowflake has a major technical advantage over others

The #1 Reason Snowflake is Different

Damn. You just discovered the latest sales report doesn’t include the new category of product and you know it’s going to take forever for…

The Case Against Self-Service Data and Analytics

It’s time for a modern metadata solution, one that is just as fast, flexible, and scalable as the rest of the modern data stack.

Data Catalog 3.0: Modern Metadata for the Modern Data Stack

Data pipeline orchestration is traditionally engineering-heavy, but a modern data stack can free your engineers for higher-value projects.

Data Orchestration Explained – and Why You Shouldn't DIY | Blog | Fivetran

Today, most organisations build their data platforms following this ELT approach where they create a centralised data asset in their cloud data warehouse or lake, combining all their source data, that acts as the source of truth for all their business systems and appreciates in value over time. 

The modern data stack in 2021

Part-I: Introducing Minerva — Airbnb’s Metric Platform

How Airbnb Achieved Metric Consistency at Scale

“Who’s the data owner?” Learn how to define data ownership and how a data catalog can help you document this information.

Defining Data Ownership

Apple’s Data Platform for ML

Learn how to get the value out of your organizations data. The analytics stack is the link between raw data and business intelligence, a platform for business growth and innovation.

Building an Analytics Stack: A Guide

The following is based on a talk we gave to the Y Combinator W21 batch on how to get started with data, analytics, and business…

No Nonsense Intro to Data, Analytics, & Data Stacks

A simple example and checking its capabilities and features. Is it worth the hype?

Airbyte — Worth the hype?

Yet Another Blog on Data

📚 Restructure your knowledge of the complex data analytics landscape, and learn how to build scalable analytics & BI stacks

The Analytics Setup Guidebook: Build Scalable Data Analytics and BI Stacks in the Modern Cloud Era

Now teams are adopting yet another new approach, called “reverse ETL,” the process of moving data from a data warehouse into 3rd-party systems to make data operational. The emergence of reverse ETL solutions is a useful component of the stack to get better leverage out of data.

What is Reverse ETL?

Data pipelining is the act of moving data from one location to another and is one of the most critical operations in data infrastructure. Data pipelines are a superset of Extract, Transform, Load…

Data Pipelining — A Primer

In Part 2 of the Data Quality for Everyone, Everywhere series, learn about the 5 main reasons why data quality can be so inconsistent throughout the enterprise.

Why Data Quality Does Not Extend to Everyone, Everywhere Today | Informatica | Informatica

The feature store is the central place to store curated features for machine learning pipelines, FSML aims to create content for information and knowledge in the ever evolving feature store's world and surrounding data and AI environment.

Feature Store for ML

Data quality testing: we all recognize its importance, but sometimes have trouble figuring out exactly how it manifests itself in data engineering workflows. Ingraining data quality within CI/CD and ongoing data modeling will ensure notification upon data quality drift.

Data Quality In Data Engineering Workflows

Learnings from dozens of companies on how to make your data catalog successful

How to Make Your Data Catalog Successful

What could work for true data democratization at scale?

The Missing Piece of Data Discovery and Observability Platforms: Open Standard for Metadata

Instead of using dashboards or reports to understand your data, operational analytics drives action by automatically delivering real-time data to the exact place it’ll be most useful.

What Is Operational Analytics?

How founders and startups can take advantage of the next generation in data infrastructure

What Is Driving Data 3.0?

What does a world class analytics stack look like in 2019?

How to set up a modern analytics stack

Is BI dead?

Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape

A lot of organisations still struggling to become data-driven. There are a lot of issues due to data silos, scalability problems with data infrastructure, and difficulty to use analytics. In his blog, Francois explained how these issues are being solved by the partnership of Google & Tableau.

Self-Service Analytics for Google Cloud, now with Looker and Tableau

Forbes has rightly coined a term for this era which is “Age of (too much) data” and as such it is becoming imperative for organizations and…

Stop using the terms Data Lake and Data Warehouse interchangeably!

A look at the entire ecosystem of data engineering tools and technologies.

The State of Data Engineering in 2021

Is it analytics? Is it engineering? What do these people do?

Analytics Engineer: The Newest Data Career Role

My thoughts on where our space has been and where it might be going.

Resources

Articles

Videos

Introduction to headless BI - Webinar

Data Cataloging with Knowledge Graphs

PLG is for Sales

The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analytics Engineer

Looker and BigQuery: Enterprise digital transformation at Sunrun

The Modern Data Analytics Stack for AI-Driven Decision Intelligence

Taylor Brown and George Fraser, Co-Founders of Fivetran, keynote the Modern Data Stack Conference

Augmented Analytics Explained

The Changing Landscape of Data Collaboration

Introduction to Data Mesh - Zhamak Dehghani

Did we miss anything?