If a web-cam is available ET1 will automatically sync and help you navigate the software. Using computer vision we map a skeleton over your hands and able to create a radial menu we are calling Navigator.
The skeleton helps us build a menu on your hand, like an augmented environment to solve problems, and change the audience’s perception on the way problems are solved.
Notice the pointer finger has a yellow dot? This dot is used to click buttons.
We are utilizing Google AI Edge MediaPipe, more tech spec and links listed at the bottom of the article. No installs or license is required to use Google AI Edge API. In the Screenshot above we are demonstrating the skeleton we are using to build a menu around your hand.
What is the radial menu in ET1?
The radial menu is a user-friendly way to interact with ET1. If a webcam is available the end user can use their left or right hand to access multiple menus which have sub menus and interact together.
How does the radial menu work?
The video needs to see a hand aiming up right. Fingers above the wrist is the best posture. Like you’re waving hello to a friend, and aim your pointer finger to press a button.
The menu tracks your palm and depending on the distance from your hand and the web cam, you’re able to start clicking on buttons.
Using your pointer finger, point at a button. Either the left hand or right hand pointer finger is the finger that pushes buttons.
The Node Creation Menu
The node creation menu is comprised of 5 different buttons in the top ring. Each of these buttons requires a ‘charging’ effect to avoid accidental usage.
Extract Nodes
Clean Nodes
Filter Nodes
Join Nodes
Load Nodes
Inside of each click is another sub menu that has various nodes to build your data pipelines.
In the palm, notice “ETL” button and swap arrow. A subtle tap on your palm will jump between Canvas Controls and Node Creation menu, which are the primary two menus.
Pro tip: hand menu can be clicked by a mouse, which means you can more than one person working on the computer at a time, making this a two player solving software.
The Canvas Controls Menu
The Canvas Controls offers 5 different options.
Select Node
Create Node
Collapse Node
Settings Node
Delete Node
Inside Select Node button is a sub menu allowing you to swap between different nodes without having to use a mouse!
Inside Create Node button is the Node Creation Menu and connects nodes if you have a node selected.
Notice this is a low light capture. In this screenshot above, the palm button allows you to swap back to the node creation menu.
While Creating a New Node, The Hand Menu Works
Without a hand menu, ET1 works perfectly fine with the mouse and keyboard, however if you’re presenting and creating new nodes, the hand menu will sync up with your mouse interactions.
In the process of creating a new node the hand menu will sync up and allow you to create your node in the hand menu OR the mouse/keyboard will work in this quick menu feature. This opens the door for a different way of explaining the solution as you’re creating it for your audience.
Selecting a node
If you Select a node, you can collapse/open to reduce space, open/close settings, create new connected nodes down stream, delete the node,
Clicking a node selects a node. Selected nodes have a white box around the node. Canvas control menu allows you to select nodes. Canvas control menu lets you swap between selected nodes.
Selected nodes come to the front of the page, allowing you to quickly see what was behind another node.
De-selecting a node is important if you need the menu to create an ad-hoc node without it connecting. To de-select, click in the blank space in your workflow canvas!
Charging to click buttons in the radial menu.
Some buttons require a charge. Some buttons charge faster than others due to how often you’d want to use those buttons. In some circumstances the speed of the button is related to the likely hood that an end user would need to click that button more than once.
The charge up timers are important because some menus you might not want to accidentally click, and some menus you may be in a “thinking mode” so the radial menu allows you to think before immediately selecting most buttons.
In the Select Node sub menu you will notice buttons charge faster due to how often you may want to move faster in this menu.
Will the radial menu work at night?
Yes, the light from your monitor is enough to initiate the menu.
Low light settings seem to work better.
You will need to be closer to the computer if it’s completely dark in your room. The light from the computer may help.
ET1 uses Google AI Edge Implementation
We are not creating this solution from scratch and instead building on Google’s tooling.
What is Google’s AI Edge?
Google created tasks-vision (among other great tools), in their Google AI Edge product line. Google AI Edge? Allows people to deploy AI across mobile, web, and embedded applications.
Performantly run JAX, Keras, PyTorch, and TensorFlow models on Android, iOS, web, and embedded devices, optimized for traditional ML and generative AI.
This is a ‘low-code’ solution for common AI related tasks.
How are we using Google AI Edge?
In our instance, we are using tasks-vision because it allows us to create radial hand menus.
Using Google AI Edge API we enhance the user experience using MediaPipe.
From Google; MediaPipe Solutions provides a suite of libraries and tools for you to quickly apply artificial intelligence (AI) and machine learning (ML) techniques in your applications. You can plug these solutions into your applications immediately, customize them to your needs, and use them across multiple development platforms. MediaPipe Solutions is part of the MediaPipe open source project, so you can further customize the solutions code to meet your application needs.
The Unique Filter Node or Unique Tool finds unique values per row in your data pipelines, or allows people to quickly review duplicates only.
Plus, you can select what column(s) to find unique values within. This enables people to easily understand what is inside of a column.
Duplicate rows happen, The Unique Filter node manages these rows for you automatically.
Whether you’re eager to only look at unique rows or drilling into the duplicates, ET1’s Unique Filter Node is the data engineering tool for your unique or duplicated needs.
Unique Filter Node: Filter mode explained
The Unique Tool or Unique Filter Node
Show unique only – this setting means you will stream only the unique values through the pipeline
You may want to run this across all pipelines as a way to verify
This is an easy way to create look up tables
Build a tool to understand what is inside of a column
Show duplicate only – will stream duplicates only and remove the unique values found
Drill into duplicates only, great for deep dives and researchers
Helpful for auditing pipelines, does your pipeline have duplicates?
Using the Unique Filter Node in ET1
Drag and drop your data pipeline arrow connection to the input of the Unique Filter to begin immediately reporting on unique rows only.
Open the settings for more granular options.
ET1’s Unique Filter Node automatically removes duplicate rows based on selected columns, however we automatically infer you are eager to use all columns and start there. Opening the settings for more options will offer a cool way to group data.
Creating look up tables with Unique Filter Node
Auditing your column? How about the values inside of each column? This is a great tool for understanding what is possible in your data pipeline.
In this example, a dataset comprising 6 million transactions is provided through an email you never wish you looked at, and the objective is to identify the originating cities. While an aggregation tool could be utilized (where you are able to use group by), however the Unique Tool offers a more effective approach for grouping data in these circumstances.
The Unique Tool facilitates a comprehensive understanding of individual column content. A common strategy involves removing unnecessary columns and employing the Unique Filter Node to extract the distinct values within the remaining table, thereby enabling the surfacing of valuable insights.
ET1 is designed to facilitate straightforward data filtering and transformation processes. It is helpful to consider data analysis as a communicative exchange with the dataset.
Technical specs on the Unique Tool’s Data Processing
JavaScript that filters data rows for uniqueness or duplication based on specified columns.
It processes tabular data in a browser-based ETL pipeline, determining which rows are unique or duplicate by constructing composite keys from selected column values. The behavior depends on the filterMode configuration: when set to 'unique', it retains only the first occurrence of each key; when set to 'duplicates', it excludes first occurrences and keeps only subsequent repeats.
Composite keys use a rare delimiter ('␟'): The character U+241F (Symbol for Unit Separator) is used to join column values into a single key string. This prevents collisions that could occur with common delimiters like commas or pipes, especially when column values themselves contain such characters.
Robust handling of missing or invalid configurations: If node.columns is not an array or contains invalid column names, the function defaults to using all available headers, ensuring that filtering still occurs meaningfully instead of failing silently or throwing errors.
Two-pass algorithm ensures correctness: The first pass counts all key occurrences, which could be used for analytics (though currently unused); the second pass performs the actual filtering. This structure allows future enhancements, such as filtering by occurrence count thresholds.
Seeking to append fields like the Alteryx Desktop software? The Joiner Node and Aggregate node is here to help. You will also need a constant to join on across both data streams. We will show you how in this example below where we are sent 6million records in a CSV and asked to append the row count at a row level per row.
ET1 is able to append fields using the Joiner node!
Before the join node, we simply utilize an aggregate column to count rows. You could also swap this to aggregate any measure in the data pipeline.
Joiner automatically infers the join using similar column headers and matches the default header column ‘const’ applied in the constant node.
Creating the append field solution in ET1
Using the join to merge the data across rows, and then the constant to create a joinable column. Here’s the exact settings.
Append field is all about knowing how to join to a table.
Joiner has the ability to inner join, left join, and assume the left/right side of the join based on what you plug into the node first. Joins can be used as a filter, a labeler, a way to append fields, merge tables, and join at a row level with primary key to foreign key.
With ET1 we make this entire process of appending data at a row level easy to do, so that you’re able to complete the same solution as Alteryx’s Append Tool.
Dealing with duplicate columns? This particular node is designed to remove similarly named column headers.
If “State”=”State” then we remove the last column and only keep the first column.
“Remove duplicate columns based on similar header names”
Currently there are no settings for this tool because today Duplicate Columns Node solves problems without requiring a settings configuration.
The Duplicate Columns Node is built to automatically remove any duplicate headers.
The hardest part is connecting the dots. This is a node that automatically works, without any effort!
Here’s the various ways you will see The Duplicate Columns node in ET1.
A Use case Duplicate Columns Node
Crazy scenario; someone is sending you tables of data with duplicate columns, and now it’s your job to clean the table. This is a great tool, which simply removes Duplicate Columns.
Familiar with graphs? How about DAGs? This is not a paradigm shift, but think of DAG as a cool way for tiny team in Austin/Dallas Texas to build an Extract Transform and Load software!
Like a guitar pedal, there’s an input and output. Sometimes it’s just an output. Then you have your input only tools. Very much like our ETL software ET1.
The DAG engine gives this small team the ability to create an ETL software with rules and futuristic features.
We leverage the same principles employed by other well-regarded Directed Acyclic Graph (DAG) tools, such as Apache Spark, Apache Airflow, Apache Beam, Kubeflow Pipelines, MLflow, TensorFlow, Dagster, Prefect, Argo Workflows, Google Cloud Composer, and Azure Data Factory, among others.
We created our own custom DAG engine using JavaScript, and this enables us to flow data downstream in a web app. Data steaming in a no-code ETL software, without a setup or install, feels like a big win for any ETL software.
In simple terms, Acyclic means not looping, this diagram/graph shows no loops.
What is a graph?
From a data perspective, a graph is a non-linear data structure used to model and store information where the relationships between individual data points are as important as the data itself. Natively a graph engine would treat data as a first class citizen, enabling real-time data processing and the ability to only compute what needs to be computed.
Unlike tables in a relational database, which store data in a fixed, row-and-column format, a graph is a flexible, interconnected network of entities and their relationships. With ET1, we fix together this graph engine so that it looks and feels like a regular ETL software, enabling a lot of cool functionality and features that regular ETL software is unable to offer.
We don’t mean to appear as we are reinventing the wheel but rather adding a different style to the typical nodes or tools you have come to learn and love.
No looping… Acyclic. Stop recycling the same rows…
Focusing solely on the detrimental effects of loops is insufficient. While infinite loops can undoubtedly disrupt systems and lead to financial losses, a more significant concern is the unnecessary recycling of data, a practice prevalent in many software applications. Why is it necessary to repeatedly query data when it is not needed? Many tools, including Tableau, Power BI, Alteryx, and Knime, inherently recycle data rows. This necessitates re-querying 100% of each table during both development and production phases, resulting in frequent downtime, managing downtime due to backfills, the constant need for backfilling, increased system strain, and a continual escalating cost. Where has the concept of incremental data loading gone?
We unblock the ability to stream data incrementally to avoid unnecessary backfilling by using this DAG system!
This is one of many reasons we feel ET1 is powerful data engineering solution.
ET1 is by law is a acyclic
Meaning not forming part of a cycle. However many ETL style tools, both in visualization and ETL, still remain with the same engine as 20 years ago. Many things have changed in 20 years. Like the ability to avoid recycling data natively.
In the data world, acyclic means no looping is possible, and from a row perspective, this is powerful because you’re always incrementally loading downstream.
This application is unable to loop back on itself. A safe behavior to avoid novice mistakes that can instantly cost a lot of money in the wrong hands.
The DAG engine, consider it a beneficial rule for an ETL software that cares about rows, most ETL software cares about columns + tables, rows become second class citizens because in database land – columns and tables are king.
These classic ETL tools constantly recycle, most allow looping, and naturally this will push more work on your systems and increase costs.
This is one of many reasons we feel the DAG engine is important, and in this diagram, 2 goes to 5 then back to 1. This isn’t possible in ET1 in the UX, and also not possible per row. This enables incremental row level refreshing. Saving time when engineering solutions, and making tweaks doesn’t cause considerable downtime ever again!
This diagram is not possible due to the rule base in the engine and not a DAG, due to the loop.
Looping still confusing as a negative? Imagine an application that could easily loop on itself, like a delay pedal that can feedback on itself, this would infinitely get louder and could destroy your ears or speakers… And from a data perspective, looping on yourself could spell a disaster for your computer, other computers, your network, your API bills, and much more… Loops would be a negative because it would allow people to break their computer and attached machines..
DAG is predictable and a great engine to use for flowing data downstream with rules, better feature sets, and enables easier visual feedback to teach end users…
Core Concept: No Play Button, Data Flows, DAG Guides, Ready?
The DAG (Directed Acyclic Graph) system is like a digital assembly line for your data, where each node is a workstation that processes data and passes it along. This changes how data is computed.
Instead of maxing out a few nodes because you’re querying all the data at once before starting a new node, each piece of your data is treated like a first class citizen in ET1.
Here’s how it works:
Is this data ready?
Yes or no?
When you go climbing, you are always talking to your partner, are they “Ready or not.” Is the person keeping you safe ready for you to fall? Are you ready? The person keeping you safe should always be ready. ET1 is always ready, so data is always flowing.
Being “always ready” is the key, DAG the bumpers to fall within, and our guide. It creates things like streaming, processing only what’s necessary, and branching off big ideas is simplistic.
Key Components
Nodes – Individual processing units (like filters, joins, calculations)
Edges – Connections showing data flow between nodes
Data Streams – The actual data flowing through the system
How It Works
Automatic Updates
Change a node? The system only recalculates what’s needed downstream
No manual refreshing – updates happen in real-time
Smart Processing
Only processes changed data paths
Alteryx and Knime users tired of data processing unnecessarily will be excited about this feature
Avoids redundant calculations
The DAG engine lets you only calculate what changes, decreasing your compute and time spent creating solutions
Visual Flow
See your data transform step by step
Easy to spot where changes are needed
Intuitive drag-and-drop interface
Why ET1 is Better
No More Waiting: Only recalculates what’s necessary
Never get stuck waiting on data to re-run because you made a change, only calculate what matters, graph enables the ability to calculate one thing at a time
Most products have to re-calculate the entire table before it’s ready to move forward
Mistake-Proof: Can’t create circular references, very helpful
Users are unable to make big mistakes like spamming their API in an infinite loop
No one will be able to increase their cloud costs because they made an easy mistake
Exploration has no penalties, crafting a sense of trust in non-technical users
Decrease stress and network strains by avoiding infinite loops
Visual Debugging: See exactly where data changes happen, a visual teacher
Created to help people visually understand their data processes
Highlight to quickly see and understand the data automation
Scalable: Handles simple to complex workflows with ease
Think of it like a factory conveyor belt system – each station (node) does its job and passes the product (data) to the next station, with the system automatically managing the flow and only processing what’s needed.
Competitive analysis
Instead of constantly recycling the same rows over and over, ET1 enables anyone the ability to only compute rows that need to be updated VS re-running each table unnecessarily.
This is the tools for problem solving like KNIME, Alteryx, Tableau, Power BI, and most BI Platforms.
In most software; If your pipeline changes, you have to run 100% of the records.
ET1 defeats this with this engine.
The DAG engine introduces what we feel is great foundation for a powerful ETL tools that can scale in the future.
We believe only the data that matters should flow down stream, DAG natively supports that by design. So using this DAG engine, we are able to only flow what matters, and make problem solving feel modern.
Future outlooks
We are not married to this engine but believe it’s very beneficial thus far. Our goal is not become fixated on the engine but rather what features it can offer.
Graph means it’s easy for us to scale up to cloud or server off loading situations in the future and that’s the easy piece.
Knowing that DAG systems are the backbone of many major big data appliances, know we are thinking bigger, big picture, and next steps too.
If you have a use case that isn’t possible on your current machine, let us know.
