Catwalk Pose Estimation
Software replacement for Kinect3d pose esimator using Grasshopper3D
TAGS:
Grasshopper3D, Javascript, Pose Estimation,
Analysis, SImulation, Robots
This plugin allows users to estimate poses in realtime using only a webcam and Grasshopper3D. It acts as the starting point for the development of intricate interactive solutions for a wide variety of use cases.
Year
2020-2021
Team
Praneet Mathur
Languages
(p5.js, ml5.js), C#
Platform
Grasshopper3D
What is it about?
In 2020, for any type of real-time pose estimation in an environment like Grasshopper3D, an Xbox Kinect was the most efficient method but unfortunately not as accessible to creators to be a viable or affordable option for everyone.
This plugin employs Machine Learning and lets you use just your Webcam to estimate human figures similar to the XBox Kinect.
The Challenge
The project was started as a software replacement for the Kinect sensor in the Grasshopper3D environment. To achieve this, certain challenges had to be overcome:
1. We first had to determine whether p5.js/ml5.js could run inside the Grasshopper3D environment
2. The objective was to achieve some functions similar to a Kinect sensor or alternate hardware that allow real time processing of physical movements by just using p5.js/ml5.js implemented inside of Grasshopper 3D.
The Solution
‘Catwalk’ was a 1 day project. It is a PoseNet JavaScript Implementation within Grasshopper for Pose Estimation. Using ResNet Model and API through ml5.js. This plugin employs Machine Learning and lets the user estimate human figures similar to the XBox Kinect using just their webcam.
It essentially runs an instance of the Google Chrome browser (Chromium, to be precise) within Grasshopper by simply referencing an open source library — CefSharp.The webpage runs a JS script, but instead of displaying the pose in a p5.js canvas , it simply exposes the raw pose data through a JS function (CefSharp allows calling functions through it’s C# API).
Then it's simply calling the function from a Grasshopper component and it would return all the data in JSON format which could then be de-serialized to an object. Exporting the properties of this object in the component’s output parameter gives a simple way to run JavaScript functions inside Grasshopper, in real-time.
Use cases
So what can you use applications like catwalk for? The possibility of use cases is unlimited but here we provide a few use cases to stimulate your imagination!
01
For Sports coaches, athletes → Using motion sensers to analyze the motion of the players and use this for futher optimisation.
02
For Software developers -> utilising intricate gesture control to add an advanced layer of interactiviy to the software.
03
For corporate training, Factory managers -> Analyse the motion of the employees to ensure that proper postures and procedures are followed.
And many many many more!
Project Catwalk enables users to conduct pose estimation using only a webcam in conjugation with Grasshopper3D. It acts as the starting point for the development of intricate solutions for a wide variety of use cases.
