How to make Business Intelligence future-proof by applying software engineering principles

Here is What Happens If You Decouple Your BI Stack

Before starting Falkon, I interviewed over 46 data and analytics teams to understand their day to day workflows. One of the most important…

Why metrics should be a first class citizen in a modern data stack

Why Vectorized’s focus on developer experience will unlock real-time streaming for the great majority of developers

Why Developers Love Redpanda

In Part-1, I explained what is Data Quality in Big data ecosystem and what are the challenges in ensuring data quality with the data…

Data Quality: The holy grail of a Data Fluent organisation | Part-2

For the last 6 months we have been working tirelessly to solve the problem of data collaboration in the enterprise and now we have our MVP…

A brief history of Paradime

A look back at a handful of data projects delivered by the data science, data engineering and analytics team here at Gousto in 2020.

Gousto Data Team — Best of 2020

Have a glimpse on what we do at Telkomsel’s Data Engineering Team and how to build a great data engineering pipeline.

Insights From Telkomsel’s Data Engineering Team — Framework, Key Principles and Tools

Looking beyond the diagram unlocks use cases for data lineage technology that could change how companies govern their data in the future.

The Future of Data Lineage — Beyond a Diagram

This book aims to give you a high-level framework and understanding of a proper modern analytics setup, and how each component interacts with each other.

Introduction | The Analytics Setup Guidebook

Yet Another Blog on Data

📚 Restructure your knowledge of the complex data analytics landscape, and learn how to build scalable analytics & BI stacks

The Analytics Setup Guidebook: Build Scalable Data Analytics and BI Stacks in the Modern Cloud Era

Now teams are adopting yet another new approach, called “reverse ETL,” the process of moving data from a data warehouse into 3rd-party systems to make data operational. The emergence of reverse ETL solutions is a useful component of the stack to get better leverage out of data.

What is Reverse ETL?

Over the past 9 months, I’ve been working on a book to be published by O’Reilly Media. This past week, Data Pipelines Pocket Reference was officially published in print, e-book, and on …

My New Book: Data Pipelines Pocket Reference

A new generation of data tooling and infrastructure companies are enabling companies to create value with their data.

The Data Opportunity: the Emergence of a New Layer in Modern Enterprise

If you're looking to build the ideal modern data stack for analytics using only open-source options, this is the blog for you. Find all the best open-source alternatives to your favorite paid tools.

The Modern Data Stack: Open-source Edition

Fivetran co-founder and CEO George Fraser shares the importance of the modern data stack and five developments he’s eagerly following.

Five Reasons To Consider A Modern Data Stack | Blog | Fivetran

Data pipelining is the act of moving data from one location to another and is one of the most critical operations in data infrastructure. Data pipelines are a superset of Extract, Transform, Load…

Data Pipelining — A Primer

In Part 2 of the Data Quality for Everyone, Everywhere series, learn about the 5 main reasons why data quality can be so inconsistent throughout the enterprise.

Why Data Quality Does Not Extend to Everyone, Everywhere Today | Informatica | Informatica

How a changing technical environment has shifted power away from decision-makers

Are Decision-Makers Disempowered by the Modern Data Stack?

On Redshifts, Data Catalogs, Query Engines like Presto, and the troubles of machine learning engineers to get their data.

There’s More Than One Kind of Data Mesh — Three Types of Data Meshes

Where there is data, there is the risk of bad data. And bad data is expensive. So it’s no surprise that tools for monitoring and improving data quality are as old as databases themselves.

State of Data Quality Monitoring in 2021

Our cool new house needs one more plank in its foundation.

The missing piece of the modern data stack

The feature store is the central place to store curated features for machine learning pipelines, FSML aims to create content for information and knowledge in the ever evolving feature store's world and surrounding data and AI environment.

Feature Store for ML

Data quality testing: we all recognize its importance, but sometimes have trouble figuring out exactly how it manifests itself in data engineering workflows. Ingraining data quality within CI/CD and ongoing data modeling will ensure notification upon data quality drift.

Data Quality In Data Engineering Workflows

5 Trends In Data & What Buyers Are Looking For In New Tools

Unpacking The Data Hype

Learnings from dozens of companies on how to make your data catalog successful

How to Make Your Data Catalog Successful

We at Wrike are creating a roadmap with best practices to help data engineers achieve good data quality in their teams and projects.

Data Quality Roadmap. Part I

What could work for true data democratization at scale?

The Missing Piece of Data Discovery and Observability Platforms: Open Standard for Metadata

Our approach for reducing data issues by a factor of 5, and our lead times for dashboards & reports by a factor of 10.

How Mercateo is Rolling Out a Modern Data Platform

This is what the professionals want you to stop doing.

The Bad Habits To Break Now: Data & Analytics Leaders

Data has been part of our identity as humans since the time of the ancient Romans. Our growing interconnectivity caused us to bump into…

Data & Data Engineering — the past, present, and future

Damn. You just discovered the latest sales report doesn’t include the new category of product and you know it’s going to take forever for…

The Case Against Self-Service Data and Analytics

In 2021 we began to form an Analytics Engineering team to support and complement self service. This post covers how we spin up this new…

Spinning up an Analytics Engineering Team

How a revolution comes together. Or doesn’t.

The Modern Data Experience

A radically new approach to data integration saves engineering time, allowing engineers and analysts to pursue higher-value activities.

What Is the Modern Data Stack? | Blog | Fivetran

Instead of using dashboards or reports to understand your data, operational analytics drives action by automatically delivering real-time data to the exact place it’ll be most useful.

What Is Operational Analytics?

How founders and startups can take advantage of the next generation in data infrastructure

What Is Driving Data 3.0?

How to migrate from a single tool to many tools while keeping your data clean and accurate to drive your business.

Graduating to the Modern Data Stack for Startups | Census

What does a world class analytics stack look like in 2019?

How to set up a modern analytics stack

What your data team is using: the analytics stack

Technically: the analytics stack

A Comprehensive Glossary For Data Engineers

150+ Concepts Heard in Data Engineering

What if … You had to build a data stack from scratch?

Data governance can enhance various areas of your business, from your enhancing digital marketing strategies to positively impacting your financials.

7 Ways Data Governance Can Benefit Your Business – Around the Web

Is BI dead?

Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape

 In her article Operationalization: the art and science of making metrics, Cassie talks about the essential psychological elements important for all data professionals to make metrics that can be defined and named!

Operationalization: the art and science of making metrics

A lot of organisations still struggling to become data-driven. There are a lot of issues due to data silos, scalability problems with data infrastructure, and difficulty to use analytics. In his blog, Francois explained how these issues are being solved by the partnership of Google & Tableau.

Self-Service Analytics for Google Cloud, now with Looker and Tableau

A better way to make daily decisions is already here—just not at work.

The future of operational analytics

A lot has improved in the data world in recent years, still the core problems remain a pain point for organizations. And the solution to these problems lies in the way organizations handle their data. The next big challenge for data is an organizational one i.e, how scaled companies organize their data teams.

The Next Big Challenge for Data is Organizational

n her blog, she discusses the common mistakes that new data recruits make. A detailed explanation about how these mistakes can be avoided and what can be done instead. She has also given a few suggestions that will be beneficial for a beginner's data journey. 

Tough Love for Naïve Data Newbies

A journey through my learnings of the world of Data — this article describes how data is treated and consumed in a product organization, and how various professional roles interact with data at various stages of its lifecycle

An experience of a “Data Ecosystem”

Note: This article was written by Oz Katz and was published on the lakeFS blog on 4/12/21.

Hudi, Iceberg and Delta Lake: Data Lake Table Formats Compared

Headless BI is a new way of looking at business intelligence powered by cloud-based data warehouses.

Headless Business Intelligence

A few thoughts on Airbnb’s metrics layer.

Is Minerva the answer?

Dashboards and reports are the lingua franca in the world of business. Simple as they may seem, behind each KPI dashboard are data analysts who are responsible for keeping dashboards working…

Data Discovery for Business Intelligence

Data plays a critical role in business decisions, AI/ML, product evolution and much more. Timeliness, accuracy, and reliability are the…

Superglue — Journey of Lineage, Data Observability & Data Pipelines

In this tutorial we will build a Slackbot using JavaScript/Node.js that reaches out to 2 public APIs and tells random jokes within Slack. We will use the Sla...

Build A Slackbot

Nowadays most companies sit on lots of very valuable data, and as most refer to, data is the new gold. Data pipelines have evolved…

Reverse ETL — an approach to Operational Data Insights

Data has evolved to be the core foundation of most organisations. Data Quality ensures that this core foundation is giving us correct…

Data Quality: The holy grail of a Data Fluent organisation | Part-1

Evolution of data warehouses and technical analysis of why Snowflake has a major technical advantage over others

The #1 Reason Snowflake is Different

Forbes has rightly coined a term for this era which is “Age of (too much) data” and as such it is becoming imperative for organizations and…

Stop using the terms Data Lake and Data Warehouse interchangeably!

A look at the entire ecosystem of data engineering tools and technologies.

The State of Data Engineering in 2021

What are the Key Elements that are used to meet the Challenges?

The Secret to Modernizing Data Analytics Platforms

Is it analytics? Is it engineering? What do these people do?

Analytics Engineer: The Newest Data Career Role

As more startups collect data at an earlier stage, many companies are thinking about their analytics stack earlier in their life cycle. How to set up your data stack is a common question for early-stage companies. This is understandable, as most early-stage companies rely on analysis to gather insights to help them grow and these insights depend on clean and accessible analytics.

How Can Startups Adopt a Modern Data Stack?

Key areas which will transform data stacks

How the modern data stack will transform in 2021 and beyond

My thoughts on where our space has been and where it might be going.

The Modern Data Stack: Past, Present, and Future

There is a massive opportunity in startup land. Startups are leveraging Cloud DataWarehouses to build the modern Data Stack.

Why Snowflake could be the next Great Platform

Despite benefits of high-quality data available, most companies are still in the process of developing their data governance systems

Key Challenges For Data Governance

The data infrastructure is changing, and data warehouses are at the center of it all. We dig into why, plus where the industry is headed.

The Ecosystem (and Future) of the Modern Data Infrastructure – Indicative

It’s time for a modern metadata solution, one that is just as fast, flexible, and scalable as the rest of the modern data stack.

Data Catalog 3.0: Modern Metadata for the Modern Data Stack

The cloud analytics stack allows businesses of any size to quickly maximize the value of their data with its cost-effective, elastic architecture and near-infinite scale.

Building A Modern Cloud Analytics Stack: A Playbook for Data-driven Companies | Sigma Computing

Data pipeline orchestration is traditionally engineering-heavy, but a modern data stack can free your engineers for higher-value projects.

Data Orchestration Explained – and Why You Shouldn't DIY | Blog | Fivetran

Welcome to The Data Source, your monthly newsletter covering the top innovation in data engineering, analytics and developer-first tooling. Subscribe now and never miss an issue 👇🏼 Looking back to the past 5 or so years in data infrastructure, there’s been significant innovation in tooling and engineering best practices that have contributed to what is known today as the Modern Data Stack.

The Data Source #3 |  💥 Two Trends Shaping the Modern Data Stack 

Explore tools to organize and govern data to ensure quality and searchability

Implement data governance to manage and secure clients' data

Metriql doesn't have a user interface that lets you analyze the data because it's Headless BI. However, it comes with a dashboard that enables you to use the integrations that we have. Here is how it looks like:

How we built the Metriql public demo

DISCOVER THE DIFFERENCE BETWEEN A DATA LAKE AND A DATA WAREHOUSE.

The 9 Differences Between a Data Lake and a Data Warehouse

Today, most organisations build their data platforms following this ELT approach where they create a centralised data asset in their cloud data warehouse or lake, combining all their source data, that acts as the source of truth for all their business systems and appreciates in value over time. 

The modern data stack in 2021

Data teams love calculating and tracking everything with metrics. We already have the infrastructure in place to do so… yet often fail to apply the same strategy for our own work. Let’s fix that!

How To Measure a Data Engineering Team

This article is about how HomeToGo uses dbt as its main tool for all data transformations in its ELT process and creating a scalable framework for this process

DBT at HomeToGo

Inefficient data operations within an organization or an inefficient DataOps team can have an impact on both data inputs and the outputs that are generated. So it becomes very  important to have a data ops team that is efficient & well organized. In his article, Eitan has discussed how you can achieve this at your organization.

The ideal DataOps org structure

Though organizations have a large amount of data at their disposal to build and train their ML models, there are certain challenges that also arise due to this large amount of data - data access approaches, the need to migrate and/or reformat data before analysis can begin, and the adoption of different operating models to name a few. 

How to Simplify Your Approach to Data Analytics

In this article Nicholas has talked about how he redesigned over 100 ETL into ELT data pipelines due reasons like cost & scalability. He has discussed the "How" & "Why" of this entire process in detail. 

How I Redesigned over 100 ETL into ELT Data Pipelines

The tech giants have build unique architectures to manage datasets in large scale machine learning solutions.

How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine…

At Rill, we believe the need for operational intelligence will dramatically expand in the coming years. In this post we lay out why operational intelligence matters now, its salient differences with traditional business intelligence, and why it demands new technology architectures.

Operational intelligence and the new frontier of data

Part-I: Introducing Minerva — Airbnb’s Metric Platform

How Airbnb Achieved Metric Consistency at Scale

Uber delivers efficient and reliable transportation across the global marketplace, which is powered by hundreds of services, machine learning models, and tens of thousands of datasets. While growing rapidly, we’re also committed to maintaining data quality, as it can greatly impact business operations and decisions. Without data quality guarantees, downstream service computation or machine learning model performance quickly degrade, which requires a lot of laborious manual efforts to investigate and backfill poor data. In the worst cases, degradations could go unnoticed, silently resulting in inconsistent behaviors.

How Uber Achieves Operational Excellence in the Data Quality Experience

“Who’s the data owner?” Learn how to define data ownership and how a data catalog can help you document this information.

Defining Data Ownership

For many years, Snowflake has been the byword for the entire cloud data warehouse market. The company’s growth stands testament to how easy…

Why we invested in Firebolt: Snowflake catapulted the data warehouse into the cloud.

Apple’s Data Platform for ML

The People Who Reinvented the Notebook at Netflix are Back To Reinvent the Notebook

Noteable: The Interactive Notebook Document for Modern Data Teams

Gaining trust in data with extensive data quality, accuracy and anomaly checks

How Airbnb Built “Wall” to prevent data bugs

This post follows up on The Rise of the Data Engineer, a recent post that was an attempt at defining data engineering and described how…

The Downfall of the Data Engineer

Would you hire a Python/Airflow expert OR a Database/SQL expert for a Data Engineer position?

Data Engineers are no longer “Data” folks

As volumes of collected data increase exponentially, methods to ensure data quality for big data are critical to accurate and effective business decisions.

Resources

Articles

Videos

Introduction to headless BI - Webinar

Data Cataloging with Knowledge Graphs

PLG is for Sales

The Modern Data Team for the Modern Data Stack: dbt and the Role of the Analytics Engineer

Data Observability - Barr Moses

Augmented Analytics Explained

The Changing Landscape of Data Collaboration

Introduction to Data Mesh - Zhamak Dehghani

Reverse ETL on Airflow

An Approach to Data Quality for Netflix Personalization Systems

Did we miss anything?