If a web-cam is available ET1 will automatically sync and help you navigate the software. Using computer vision we map a skeleton over your hands and able to create a radial menu we are calling Navigator.
The skeleton helps us build a menu on your hand, like an augmented environment to solve problems, and change the audience’s perception on the way problems are solved.
Notice the pointer finger has a yellow dot? This dot is used to click buttons.
We are utilizing Google AI Edge MediaPipe, more tech spec and links listed at the bottom of the article. No installs or license is required to use Google AI Edge API. In the Screenshot above we are demonstrating the skeleton we are using to build a menu around your hand.
What is the radial menu in ET1?
The radial menu is a user-friendly way to interact with ET1. If a webcam is available the end user can use their left or right hand to access multiple menus which have sub menus and interact together.
How does the radial menu work?
The video needs to see a hand aiming up right. Fingers above the wrist is the best posture. Like you’re waving hello to a friend, and aim your pointer finger to press a button.
The menu tracks your palm and depending on the distance from your hand and the web cam, you’re able to start clicking on buttons.
Using your pointer finger, point at a button. Either the left hand or right hand pointer finger is the finger that pushes buttons.
The Node Creation Menu
The node creation menu is comprised of 5 different buttons in the top ring. Each of these buttons requires a ‘charging’ effect to avoid accidental usage.
- Extract Nodes
- Clean Nodes
- Filter Nodes
- Join Nodes
- Load Nodes
Inside of each click is another sub menu that has various nodes to build your data pipelines.
In the palm, notice “ETL” button and swap arrow. A subtle tap on your palm will jump between Canvas Controls and Node Creation menu, which are the primary two menus.
Pro tip: hand menu can be clicked by a mouse, which means you can more than one person working on the computer at a time, making this a two player solving software.
The Canvas Controls Menu
The Canvas Controls offers 5 different options.
- Select Node
- Create Node
- Collapse Node
- Settings Node
- Delete Node
Inside Select Node button is a sub menu allowing you to swap between different nodes without having to use a mouse!
Inside Create Node button is the Node Creation Menu and connects nodes if you have a node selected.
Notice this is a low light capture. In this screenshot above, the palm button allows you to swap back to the node creation menu.
While Creating a New Node, The Hand Menu Works
Without a hand menu, ET1 works perfectly fine with the mouse and keyboard, however if you’re presenting and creating new nodes, the hand menu will sync up with your mouse interactions.
In the process of creating a new node the hand menu will sync up and allow you to create your node in the hand menu OR the mouse/keyboard will work in this quick menu feature. This opens the door for a different way of explaining the solution as you’re creating it for your audience.
Selecting a node
If you Select a node, you can collapse/open to reduce space, open/close settings, create new connected nodes down stream, delete the node,
Clicking a node selects a node. Selected nodes have a white box around the node. Canvas control menu allows you to select nodes. Canvas control menu lets you swap between selected nodes.
Selected nodes come to the front of the page, allowing you to quickly see what was behind another node.
De-selecting a node is important if you need the menu to create an ad-hoc node without it connecting. To de-select, click in the blank space in your workflow canvas!
Charging to click buttons in the radial menu.
Some buttons require a charge. Some buttons charge faster than others due to how often you’d want to use those buttons. In some circumstances the speed of the button is related to the likely hood that an end user would need to click that button more than once.
The charge up timers are important because some menus you might not want to accidentally click, and some menus you may be in a “thinking mode” so the radial menu allows you to think before immediately selecting most buttons.
In the Select Node sub menu you will notice buttons charge faster due to how often you may want to move faster in this menu.
Will the radial menu work at night?
Yes, the light from your monitor is enough to initiate the menu.
Low light settings seem to work better.
You will need to be closer to the computer if it’s completely dark in your room. The light from the computer may help.
ET1 uses Google AI Edge Implementation
We are not creating this solution from scratch and instead building on Google’s tooling.
What is Google’s AI Edge?
Google created tasks-vision (among other great tools), in their Google AI Edge product line. Google AI Edge? Allows people to deploy AI across mobile, web, and embedded applications.
Performantly run JAX, Keras, PyTorch, and TensorFlow models on Android, iOS, web, and embedded devices, optimized for traditional ML and generative AI.
This is a ‘low-code’ solution for common AI related tasks.
How are we using Google AI Edge?
In our instance, we are using tasks-vision because it allows us to create radial hand menus.
Tech specs
Learn more about tasks-vision documentation.
Specifically, we are using Tensowflow; Console log: INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
We grab the bundle here from a javascript perspective: https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.13/vision_bundle.mjs
MediaPipe Solutions enhances ET1
Using Google AI Edge API we enhance the user experience using MediaPipe.
From Google; MediaPipe Solutions provides a suite of libraries and tools for you to quickly apply artificial intelligence (AI) and machine learning (ML) techniques in your applications. You can plug these solutions into your applications immediately, customize them to your needs, and use them across multiple development platforms. MediaPipe Solutions is part of the MediaPipe open source project, so you can further customize the solutions code to meet your application needs.
Questions? Please contact us.
Return to ET1 Overview to learn more.
