Yes. DEV3LOPCOM, LLC built an asynchronous ETL software, here in Austin, Texas.
Other ETL software is not asynchronous.
Question: Is ET1 an Asynchronous ETL software?
Answer: Yes
Here’s why:
Incremental Processing:
The DAG streaming engine allows for incremental processing.
- ET1 uses a dirty queue system with markDirty() and drainDirtyQueue() functions to process nodes asynchronously
- The scheduleRecompute() function is used to queue up computations
Event-Driven Architecture:
- The UI is built with event listeners (e.g., button clicks, drag events)
- Data flows through the pipeline reactively when inputs change
Non-Blocking Operations:
- File I/O operations (like CSV/JSON imports) are handled asynchronously
- GitHub data fetching is done asynchronously (see fetchGitHubData function)
Data Flow:
- Data propagates through the graph asynchronously using a pull-based model
- Each node processes its data independently when its inputs are ready
Visual Feedback:
- The UI updates incrementally as data flows through the pipeline
- Preview panes refresh asynchronously when data changes
The application is designed to handle potentially large datasets without blocking the main thread, making it suitable for interactive data transformation workflows.
In conclusion, ET1 is an asynchronous ETL (Extract, Transform, Load) application.
Competitive analysis and the future…
Alteryx and KNIME are two very competitive ETL solutions that I have grown to enjoy. Below we will compare the two.
Future thoughts, we know most major cloud-based and real-time ETL platforms are built on an asynchronous architecture, and feel ET1 is a competitive fit in the desktop ETL market.
We know asynchronous and minimal tooling is an approach which enables software to handle high-volume data streams and ensure reliability, without turning your computer into a toaster.
Is Alteryx Desktop asynchronous?
No, Alteryx Desktop is not asynchronous; it processes workflows synchronously, meaning operations complete in sequence. The Desktop application executes tasks one after the other as they appear in the workflow, rather than in a parallel, non-blocking manner.
ET1 allows each row to incrementally flow to each node, and instead of waiting for a node to finish, rows are already processing to the end VS being stuck in one node. ET1’s streaming DAG engine enables any computer to become an Asynchronous ETL workhourse.
Is KNIME asynchronous?
KNIME supports asynchronous operations in specific components, but generally operates synchronously by default, requiring specific node configurations or server-side settings to achieve asynchronous execution.
ET1 is different because natively it’s asynchronous natively! Without a setup or install.
Questions? Please contact us.
Return to ET1 Overview to learn more.