If a web-cam is available ET1’s code will try to automatically sync and if you can wave and point, this radial menu is a fun way to augment your ETL development.
Using computer vision (explained in depth below) we map a skeleton over your hands and able to create a radial menu we are calling Navigator, which is overlayed on your hand.
The skeleton helps us build a menu on your hand, like an augmented environment to solve problems, and change the audience’s perception on the way problems are solved.
Notice the pointer finger has a yellow dot? This dot is used to click buttons.
We are utilizing Google AI Edge MediaPipe, more tech spec and links listed at the bottom of the article. No installs or license is required to use Google AI Edge API. In the Screenshot above we are demonstrating the skeleton we are using to build a menu around your hand.
What is the radial menu in ET1?
The radial menu is a user-friendly way to interact with ET1. If a webcam is available the end user can use their left or right hand to access multiple menus which have sub menus and interact together.
How does the radial menu work?
The video needs to see a hand aiming up right. Fingers above the wrist is the best posture. Like you’re waving hello to a friend, and aim your pointer finger to press a button.
The menu tracks your palm and depending on the distance from your hand and the web cam, you’re able to start clicking on buttons.
Using your pointer finger, point at a button. Either the left hand or right hand pointer finger is the finger that pushes buttons.
The Node Creation Menu
The node creation menu is comprised of 5 different buttons in the top ring. Each of these buttons requires a ‘charging’ effect to avoid accidental usage.
Extract Nodes
Clean Nodes
Filter Nodes
Join Nodes
Load Nodes
Inside of each click is another sub menu that has various nodes to build your data pipelines.
In the palm, notice “ETL” button and swap arrow. A subtle tap on your palm will jump between Canvas Controls and Node Creation menu, which are the primary two menus.
Pro tip: hand menu can be clicked by a mouse, which means you can more than one person working on the computer at a time, making this a two player solving software.
The Canvas Controls Menu
The Canvas Controls offers 5 different options.
Select Node
Create Node
Collapse Node
Settings Node
Delete Node
Inside Select Node button is a sub menu allowing you to swap between different nodes without having to use a mouse!
Inside Create Node button is the Node Creation Menu and connects nodes if you have a node selected.
Notice this is a low light capture. In this screenshot above, the palm button allows you to swap back to the node creation menu.
While Creating a New Node, The Hand Menu Works
Without a hand menu, ET1 works perfectly fine with the mouse and keyboard, however if you’re presenting and creating new nodes, the hand menu will sync up with your mouse interactions.
In the process of creating a new node the hand menu will sync up and allow you to create your node in the hand menu OR the mouse/keyboard will work in this quick menu feature. This opens the door for a different way of explaining the solution as you’re creating it for your audience.
Selecting a node
If you Select a node, you can collapse/open to reduce space, open/close settings, create new connected nodes down stream, delete the node,
Clicking a node selects a node. Selected nodes have a white box around the node. Canvas control menu allows you to select nodes. Canvas control menu lets you swap between selected nodes.
Selected nodes come to the front of the page, allowing you to quickly see what was behind another node.
De-selecting a node is important if you need the menu to create an ad-hoc node without it connecting. To de-select, click in the blank space in your workflow canvas!
Charging to click buttons in the radial menu.
Some buttons require a charge. Some buttons charge faster than others due to how often you’d want to use those buttons. In some circumstances the speed of the button is related to the likely hood that an end user would need to click that button more than once.
The charge up timers are important because some menus you might not want to accidentally click, and some menus you may be in a “thinking mode” so the radial menu allows you to think before immediately selecting most buttons.
In the Select Node sub menu you will notice buttons charge faster due to how often you may want to move faster in this menu.
Will the radial menu work at night?
Yes, the light from your monitor is enough to initiate the menu.
Low light settings seem to work better.
You will need to be closer to the computer if it’s completely dark in your room. The light from the computer may help.
ET1 uses Google AI Edge Implementation
We are not creating this solution from scratch and instead building on Google’s tooling.
What is Google’s AI Edge?
Google created tasks-vision (among other great tools), in their Google AI Edge product line. Google AI Edge? Allows people to deploy AI across mobile, web, and embedded applications.
Performantly run JAX, Keras, PyTorch, and TensorFlow models on Android, iOS, web, and embedded devices, optimized for traditional ML and generative AI.
This is a ‘low-code’ solution for common AI related tasks.
How are we using Google AI Edge?
In our instance, we are using tasks-vision because it allows us to create radial hand menus.
Using Google AI Edge API we enhance the user experience using MediaPipe.
From Google; MediaPipe Solutions provides a suite of libraries and tools for you to quickly apply artificial intelligence (AI) and machine learning (ML) techniques in your applications. You can plug these solutions into your applications immediately, customize them to your needs, and use them across multiple development platforms. MediaPipe Solutions is part of the MediaPipe open source project, so you can further customize the solutions code to meet your application needs.
Dealing with duplicate columns? This particular node is designed to remove similarly named column headers.
If “State”=”State” then we remove the last column and only keep the first column.
“Remove duplicate columns based on similar header names”
Currently there are no settings for this tool because today Duplicate Columns Node solves problems without requiring a settings configuration.
The Duplicate Columns Node is built to automatically remove any duplicate headers.
The hardest part is connecting the dots. This is a node that automatically works, without any effort!
Here’s the various ways you will see The Duplicate Columns node in ET1.
A Use case Duplicate Columns Node
Crazy scenario; someone is sending you tables of data with duplicate columns, and now it’s your job to clean the table. This is a great tool, which simply removes Duplicate Columns.
Familiar with graphs? How about DAGs? This is not a paradigm shift, but think of DAG as a cool way for tiny team in Austin/Dallas Texas to build an Extract Transform and Load software!
Like a guitar pedal, there’s an input and output. Sometimes it’s just an output. Then you have your input only tools. Very much like our ETL software ET1.
The DAG engine gives this small team the ability to create an ETL software with rules and futuristic features.
We leverage the same principles employed by other well-regarded Directed Acyclic Graph (DAG) tools, such as Apache Spark, Apache Airflow, Apache Beam, Kubeflow Pipelines, MLflow, TensorFlow, Dagster, Prefect, Argo Workflows, Google Cloud Composer, and Azure Data Factory, among others.
We created our own custom DAG engine using JavaScript, and this enables us to flow data downstream in a web app. Data steaming in a no-code ETL software, without a setup or install, feels like a big win for any ETL software.
In simple terms, Acyclic means not looping, this diagram/graph shows no loops.
What is a graph?
From a data perspective, a graph is a non-linear data structure used to model and store information where the relationships between individual data points are as important as the data itself. Natively a graph engine would treat data as a first class citizen, enabling real-time data processing and the ability to only compute what needs to be computed.
Unlike tables in a relational database, which store data in a fixed, row-and-column format, a graph is a flexible, interconnected network of entities and their relationships. With ET1, we fix together this graph engine so that it looks and feels like a regular ETL software, enabling a lot of cool functionality and features that regular ETL software is unable to offer.
We don’t mean to appear as we are reinventing the wheel but rather adding a different style to the typical nodes or tools you have come to learn and love.
No looping… Acyclic. Stop recycling the same rows…
Focusing solely on the detrimental effects of loops is insufficient. While infinite loops can undoubtedly disrupt systems and lead to financial losses, a more significant concern is the unnecessary recycling of data, a practice prevalent in many software applications. Why is it necessary to repeatedly query data when it is not needed? Many tools, including Tableau, Power BI, Alteryx, and Knime, inherently recycle data rows. This necessitates re-querying 100% of each table during both development and production phases, resulting in frequent downtime, managing downtime due to backfills, the constant need for backfilling, increased system strain, and a continual escalating cost. Where has the concept of incremental data loading gone?
We unblock the ability to stream data incrementally to avoid unnecessary backfilling by using this DAG system!
This is one of many reasons we feel ET1 is powerful data engineering solution.
ET1 is by law is a acyclic
Meaning not forming part of a cycle. However many ETL style tools, both in visualization and ETL, still remain with the same engine as 20 years ago. Many things have changed in 20 years. Like the ability to avoid recycling data natively.
In the data world, acyclic means no looping is possible, and from a row perspective, this is powerful because you’re always incrementally loading downstream.
This application is unable to loop back on itself. A safe behavior to avoid novice mistakes that can instantly cost a lot of money in the wrong hands.
The DAG engine, consider it a beneficial rule for an ETL software that cares about rows, most ETL software cares about columns + tables, rows become second class citizens because in database land – columns and tables are king.
These classic ETL tools constantly recycle, most allow looping, and naturally this will push more work on your systems and increase costs.
This is one of many reasons we feel the DAG engine is important, and in this diagram, 2 goes to 5 then back to 1. This isn’t possible in ET1 in the UX, and also not possible per row. This enables incremental row level refreshing. Saving time when engineering solutions, and making tweaks doesn’t cause considerable downtime ever again!
This diagram is not possible due to the rule base in the engine and not a DAG, due to the loop.
Looping still confusing as a negative? Imagine an application that could easily loop on itself, like a delay pedal that can feedback on itself, this would infinitely get louder and could destroy your ears or speakers… And from a data perspective, looping on yourself could spell a disaster for your computer, other computers, your network, your API bills, and much more… Loops would be a negative because it would allow people to break their computer and attached machines..
DAG is predictable and a great engine to use for flowing data downstream with rules, better feature sets, and enables easier visual feedback to teach end users…
Core Concept: No Play Button, Data Flows, DAG Guides, Ready?
The DAG (Directed Acyclic Graph) system is like a digital assembly line for your data, where each node is a workstation that processes data and passes it along. This changes how data is computed.
Instead of maxing out a few nodes because you’re querying all the data at once before starting a new node, each piece of your data is treated like a first class citizen in ET1.
Here’s how it works:
Is this data ready?
Yes or no?
When you go climbing, you are always talking to your partner, are they “Ready or not.” Is the person keeping you safe ready for you to fall? Are you ready? The person keeping you safe should always be ready. ET1 is always ready, so data is always flowing.
Being “always ready” is the key, DAG the bumpers to fall within, and our guide. It creates things like streaming, processing only what’s necessary, and branching off big ideas is simplistic.
Key Components
Nodes – Individual processing units (like filters, joins, calculations)
Edges – Connections showing data flow between nodes
Data Streams – The actual data flowing through the system
How It Works
Automatic Updates
Change a node? The system only recalculates what’s needed downstream
No manual refreshing – updates happen in real-time
Smart Processing
Only processes changed data paths
Alteryx and Knime users tired of data processing unnecessarily will be excited about this feature
Avoids redundant calculations
The DAG engine lets you only calculate what changes, decreasing your compute and time spent creating solutions
Visual Flow
See your data transform step by step
Easy to spot where changes are needed
Intuitive drag-and-drop interface
Why ET1 is Better
No More Waiting: Only recalculates what’s necessary
Never get stuck waiting on data to re-run because you made a change, only calculate what matters, graph enables the ability to calculate one thing at a time
Most products have to re-calculate the entire table before it’s ready to move forward
Mistake-Proof: Can’t create circular references, very helpful
Users are unable to make big mistakes like spamming their API in an infinite loop
No one will be able to increase their cloud costs because they made an easy mistake
Exploration has no penalties, crafting a sense of trust in non-technical users
Decrease stress and network strains by avoiding infinite loops
Visual Debugging: See exactly where data changes happen, a visual teacher
Created to help people visually understand their data processes
Highlight to quickly see and understand the data automation
Scalable: Handles simple to complex workflows with ease
Think of it like a factory conveyor belt system – each station (node) does its job and passes the product (data) to the next station, with the system automatically managing the flow and only processing what’s needed.
Competitive analysis
Instead of constantly recycling the same rows over and over, ET1 enables anyone the ability to only compute rows that need to be updated VS re-running each table unnecessarily.
This is the tools for problem solving like KNIME, Alteryx, Tableau, Power BI, and most BI Platforms.
In most software; If your pipeline changes, you have to run 100% of the records.
ET1 defeats this with this engine.
The DAG engine introduces what we feel is great foundation for a powerful ETL tools that can scale in the future.
We believe only the data that matters should flow down stream, DAG natively supports that by design. So using this DAG engine, we are able to only flow what matters, and make problem solving feel modern.
Future outlooks
We are not married to this engine but believe it’s very beneficial thus far. Our goal is not become fixated on the engine but rather what features it can offer.
Graph means it’s easy for us to scale up to cloud or server off loading situations in the future and that’s the easy piece.
Knowing that DAG systems are the backbone of many major big data appliances, know we are thinking bigger, big picture, and next steps too.
If you have a use case that isn’t possible on your current machine, let us know.
Aggregation, what a classic. Aggregating your data is a landmark trait for any data steward, data wrangler, or data analyst. In ET1, you can easily aggregate your data.
The Power of Grouping (Group By in ET1) with the Aggregate Node
Aggregations turn a sea of numbers into meaningful insights. Group by in ET1 is nested in the aggregate node.
Concat merges everything, and it doesn’t care about data types.
What it does: Merges text from different columns
Add a custom string between what you’re merging.
Perfect for:
Creating full names from first/last
Building addresses from components
Generating unique IDs or labels
Bringing together: State with City in 1 column.
Real-World Examples
Join:
Match customer emails with their support tickets
Combine product IDs with inventory details
Union:
Merge Q1, Q2, Q3 sales into one report
Combine survey responses from different regions
Concat:
Create “Last, First” name formats
Build URLs from domain + path components
Pro Tips
Joins work best with unique identifiers
Union requires matching column structures
Concat can add custom separators (spaces, dashes, etc.)
Remove duplicate records
No more copy-pasting between spreadsheets or writing complex formulas – just connect the dots and let the data flow! No strange joining tools in Excel, no learning the difference between joins, and just get your data wrangled already!
