Prefect offers an open-source data flow automation platform. The product offerings include Prefect Core, to run data applications, built on Python and composed of a task library that enables looping, adding parameters, mapping dynamic tasks, and more. Another offering is Prefect Cloud, a command center for user workflows enabled by Cloud UI, state locking, hybrid execution, GraphQL for ETL, and more. The clients of the company include CISCO, SymphonyRM, trademe, etc.

Prefect is the new standard in dataflow automation , trusted to build, run, and monitor millions of data workflows and pipelines.

Flyte provides a cloud-based machine learning and data processing platform. The open source structured programming and distributed solution enables concurrent, scalable and maintainable workflows for machine learning and data processing. Uses protocol buffers as the specification language to specify these workflows and tasks.

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Apache Airflow is a workflow automation and scheduling system that can be used to author and manage data pipelines. Airflow uses workflows made of directed acyclic graphs (DAGs) of tasks.

Airflow is a platform to programmatically author, schedule and monitor workflows. 

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.



Infinitely scalable open source orchestration and scheduling platform.


Dagster is an orchestrator that's designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.You declare functions that you want to run and the data assets that those functions produce or update. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Dagster is a cloud-native orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

Rivery offers a cloud based data pipeline preparation tool. The platform can be used to collect data from various enterprise data sources such as Salesforce, Google Adwords, AppNexus, MySQL, OpenX, MongoDB, Oracle, Google Analytics. The platform can also collect streaming data, process the data, and persist into databases such as HDFS or Bigtable.

Rivery is a serverless, SaaS DataOps platform that empowers companies of all sizes around the world to consolidate, orchestrate, and manage internal and external data sources with ease and efficiency. 

Orchest is an open-source, cloud-native, development environment built for data science. Orchest helps you build data pipelines, the easy way. No frameworks. No YAML. Just write Python and R code in notebooks.

Orchest is an open-source machine learning platform built for data scientists.

Prophecy democratizes the development and deployment of high-quality data pipelines, uniquely combining visual development with agile software engineering best practices. The developed code is open source and is targeted at Apache Spark & Apache Airflow.

Prophecy.io is the only low-code data engineering platform.

Kubeflow provides an open-source machine learning toolkit for Kubernetes. Enables users to train, deploy and manage a machine learning stack on Kubernetes. Is compatible with PyTorch, Apache MXNet, MPI, XGBoost, Chainer & more and can be integrated with Istio and Ambassador.

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable.

Astronomer is an open-source data integration platform that helps data engineers to collaboratively contribute to the code which can be used by data scientists/engineers for integrating web applications. It is a hosted data pipeline service that can be used to connect various data source like oracle, mysql, salesforce etc to destinations like cloud platform amazon redshift, bi tools, google analytics and website analytics tools like kissmetrics. The company also provides implementation services to the companies with no data engineer.

Astronomer helps organizations adopt Apache Airflow®, the leading open-source data workflow orchestration platform that helps organizations get their data in motion.

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).

Open source Kubernetes native workflows, events, CI and CD.

Cloud Composer is a fully managed workflow orchestration service, enabling you to create workflows that span across clouds and on-premises data centers. Built on the popular Apache Airflow open source project and operated using the Python programming language, Cloud Composer is free from lock-in and easy to use.

Cloud Composer is a fully managed data workflow orchestration service that empowers you to author, schedule, and monitor pipelines.

Cloud based analytics processing software solution provider. It provides a cloud-first platform that automates and monitors the data analysis infrastructure. It also provides data journey analysis and a dashboard for data pipelines monitoring for getting specific details.

By automating your data operations STOIX significantly decreases time spent on infrastructure, allowing your data team to focus solely on providing business value.

Ascend provides big data management and analytics platform. It allows businesses to manage collect and classify data, source data from various sources, clean and perform the analysis. It enables users to design workflows, update real-time data, manage data infrastructure, and integrate data from existing systems. It provides solutions for businesses to detect trends, build predictive models, and extract business actionable insights. The clients of the company include Google, Facebook, LinkedIn, Twitter, and Square.

Ascend.io provides the only unified platform for self-service, autonomous data pipelines freeing data teams to spend more time bringing innovations to life.

Shipyard is a modern orchestration platform for data engineers to build out solid data operations from day one. The platform makes it easy to define data workflows by connecting existing data tools and scripts together so you can see the big picture of your end-to-end data touchpoints. Shipyard offers low-code templates that are configured using a visual or YAML-based interface, replacing the need to write code to build data workflows while enabling data engineers to get their work into production faster. If a solution can’t be built with existing templates, engineers can always automate scripts in the language of their choice to bring any internal or external process into their workflows. The Shipyard team has built data products for some of the largest brands in business and deeply understands the problems that come with scale. Observability, retries, and alerting are built into the Shipyard platform, ensuring that breakages are identified and addressed before being discovered downstream by business teams. The platform has automatic scaling, high levels of concurrency, and end-to-end encryption built-in, so data teams can focus on solving problems with data without relying on other teams or worrying about infrastructure challenges. This results in quicker, resilient data deliverables that business teams can trust.

Shipyard gives Data Engineers the tools to launch, monitor, and share error-free data workflows like a 10x engineer.

Proven at global web scale in production for modern data services, Alluxio is the developer of open source data orchestration software for the cloud. Alluxio moves data closer to big data and machine learning compute frameworks in any cloud across clusters, regions, clouds and countries, providing memory-speed data access to files and objects.  Intelligent data tiering and data management deliver consistent high performance to customers in financial services, high tech, retail and telecommunications. Alluxio is in production use today at seven out of the top ten internet companies. Venture-backed by Andreessen Horowitz and Seven Seas Partners, Alluxio was founded at UC Berkeley’s AMPLab by the creators of the Tachyon open source project.

Open Source Data Orchestration for Analytics and Machine Learning in the Cloud

Luigi is a Python package that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.

The Cadence solution is a fault-oblivious stateful programming model that obscures most of the complexities of building scalable distributed applications. In essence, Cadence provides a durable virtual memory that is not linked to a specific process, and preserves the full application state, including function stacks, with local variables across all sorts of host and software failures. This allows you to write code using the full power of a programming language while Cadence takes care of durability, availability, and scalability of the application.

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic.

Apache Nifi is an easy-to-use, powerful, and reliable system to process and distribute data.

Conductor was built to help Netflix orchestrate microservices-based process flows with the following features:


Conductor is a Workflow Orchestration engine that runs in the cloud.

Orkes enables you to reliably run your Microservices and Workflow Orchestration at scale. Orkes Conductor is a cloud hosted, fully managed version of Netflix Conductor OSS offering full compatibility with the open-source version.


Cloud native microservices and workflow orchestration powered by Netflix Conductor

FloSQL, makes scheduling and dependency management for SQL pipelines and data models easy to do and accessible to anyone with basic SQL skills saving 100s of hours of development time and costs. Its unique features make it easier to build out SQL models and even automatically define dependencies.

FloSQL builds and schedules data pipelines on top of your favorite data warehouse

Union AI, the team behind Flyte, has created a managed version of the workflow orchestrator, freeing data and ml teams from infrastructure constraints and setup. Union is designed to complement any data and ML stack. It enable software users with the tools to take their ideas from conception to high quality production.

Union AI provides unified orchestration for ML worflows, data pipelines and ml-driven products.

Companies

Prefect

Flyte

Airflow

Kestra

Dagster

Rivery.io

Orchest

Prophecy

Kubeflow

Astronomer

Argo

Google Cloud Composer

Stoix

Ascend

Shipyard

Alluxio

Apache Luigi

Cadence

Apache Nifi

Conductor

Orkes

FloSQL

Union.ai

Did we miss any company?