Familiar with graphs? How about DAGs? This is not a paradigm shift, but think of DAG as a cool way for tiny team in Austin/Dallas Texas to build an Extract Transform and Load software!
The DAG engine gives this small team the ability to create an ETL software with rules and futuristic features.
We are using the same principles other Directed Acyclic Graph tools we know and love today. Like Apache Spark, Apache Airflow. Apache Beam, Kubeflow Pipelines, MLflow, TensorFlow, Dagster, Prefect, Argo Workflows, Google Cloud Composer, Azure Data Factory, and many other cool tools.
We created our own custom DAG engine using JavaScript, and this enables us to flow data downstream in a web application, also it streams. Data steaming in a no-code ETL software, without a setup or install, feels like a big win for any ETL software.
What is a graph?
From a data perspective, a graph is a non-linear data structure used to model and store information where the relationships between individual data points are as important as the data itself. Natively a graph engine would treat data as a first class citizen, enabling real-time data processing and the ability to only compute what needs to be computed.
Unlike tables in a relational database, which store data in a fixed, row-and-column format, a graph is a flexible, interconnected network of entities and their relationships. With ET1, we fix together this graph engine so that it looks and feels like a regular ETL software, enabling a lot of cool functionality and features that regular ETL software is unable to offer.
We don’t mean to appear as we are reinventing the wheel but rather adding a different style to the typical nodes or tools you have come to learn and love.
No looping… Acyclic
ET1 is by law, acyclic, meaning not forming part of a cycle. In a data world that means no looping is possible. This application is unable to loop back on itself. This is a beneficial rule for an ETL software.
In this diagram, 2 goes to 5 then back to 1. This isn’t possible in ET1.
Imagine an ETL application that could easily loop on itself, that would be a negative because it would allow people to break their system. DAG is predictable and a great engine to use for flowing data downstream.
Core Concept: No Play Button, Data Flows, DAG Guides, Ready?
The DAG (Directed Acyclic Graph) system is like a digital assembly line for your data, where each node is a workstation that processes data and passes it along. This changes how data is computed.
Instead of maxing out a few nodes because you’re querying all the data at once before starting a new node, each piece of your data is treated like a first class citizen in ET1.
Here’s how it works:
Is this data ready?
Yes or no?
When you go climbing, you are always talking to your partner, are they “Ready or not.” Is the person keeping you safe ready for you to fall? Are you ready? The person keeping you safe should always be ready. ET1 is always ready, so data is always flowing.
Being “always ready” is the key, DAG the bumpers to fall within, and our guide. It creates things like streaming, processing only what’s necessary, and branching off big ideas is simplistic.
Key Components
- Nodes – Individual processing units (like filters, joins, calculations)
- Edges – Connections showing data flow between nodes
- Data Streams – The actual data flowing through the system
How It Works
Automatic Updates
- Change a node? The system only recalculates what’s needed downstream
- No manual refreshing – updates happen in real-time
Smart Processing
- Only processes changed data paths
- Alteryx and Knime users tired of data processing unnecessarily will be excited about this feature
- Avoids redundant calculations
- The DAG engine lets you only calculate what changes, decreasing your compute and time spent creating solutions
Visual Flow
- See your data transform step by step
- Easy to spot where changes are needed
- Intuitive drag-and-drop interface
Why ET1 is Better
- No More Waiting: Only recalculates what’s necessary
- Never get stuck waiting on data to re-run because you made a change, only calculate what matters, graph enables the ability to calculate one thing at a time
- Most products have to re-calculate the entire table before it’s ready to move forward
- Mistake-Proof: Can’t create circular references, very helpful
- Users are unable to make big mistakes like spamming their API in an infinite loop
- No one will be able to increase their cloud costs because they made an easy mistake
- Exploration has no penalties, crafting a sense of trust in non-technical users
- Decrease stress and network strains by avoiding infinite loops
- Visual Debugging: See exactly where data changes happen, a visual teacher
- Created to help people visually understand their data processes
- Highlight to quickly see and understand the data automation
- Scalable: Handles simple to complex workflows with ease
Think of it like a factory conveyor belt system – each station (node) does its job and passes the product (data) to the next station, with the system automatically managing the flow and only processing what’s needed.
Competitive analysis
Instead of constantly recycling the same rows over and over, ET1 enables anyone the ability to only compute rows that need to be updated VS re-running each table unnecessarily.
This is the tools for problem solving like KNIME, Alteryx, Tableau, Power BI, and most BI Platforms.
In most software; If your pipeline changes, you have to run 100% of the records.
ET1 defeats this with this engine.
The DAG engine introduces what we feel is great foundation for a powerful ETL tools that can scale in the future.
We believe only the data that matters should flow down stream, DAG natively supports that by design. So using this DAG engine, we are able to only flow what matters, and make problem solving feel modern.
Future outlooks
We are not married to this engine but believe it’s very beneficial thus far. Our goal is not become fixated on the engine but rather what features it can offer.
Graph means it’s easy for us to scale up to cloud or server off loading situations in the future and that’s the easy piece.
Knowing that DAG systems are the backbone of many major big data appliances, know we are thinking bigger, big picture, and next steps too.
If you have a use case that isn’t possible on your current machine, let us know.
Return to ET1 Overview to learn more.
