University required us to create some interactive software using OpenCV. While many other groups opted for a control system based on colors present on the screen or screen-based location input, I somehow managed to convince my group that I could probably create our own Kinect like system easily. 👋🏻📸

I mean, if Microsoft needs 4 years then surely I can do it in 3 days! 😎

What did we create?

While other groups opted for games, we wanted to create something practical that would look good on our CVs. Hence, why we decided to create a proof of concept for a training software for traffic controllers.

To simplify this we decided to basically make the user direct the approaching vehicle towards the direction specified on screen.

These are some notable requirements (simplified for reading purposes):

• Poses for 'go straight forward', 'turn left', 'turn right' & 'stop' implemented. 🕺🏻
• A score system where each correct pose would be rewarded with (a) point(s). ⭐
• All webcams should be usable, so that anyone could run it locally. 📷
• The software was optimized enough to be runnable on CPU only, no GPU. ~~Surely this was a good idea~~
• Sound! 🔊

What did we build this with?

First of all, it's noteworthy that we are using C++ 20, and as an IDE we are using Visual Studio 2019.

Pose Estimation

Well, so, I just said I could get a Kinect clone working in a couple of days...

And so I did! 😎 After doing some extensive research, I came across a beautiful article explaining how this could be achieved using a DNN model made by an American university. However, to my horror, the actual file download was no longer available, and I could not find an official download link. So what now? 😭

Well, it will not surprise you that people are not that carefully with the data they upload to their repositories using Git. On GitHub especially it is extremely easy to find API keys people did not exclude from being tracked. It is the same with files!

So all I did is look for this model using the GitHub searchbar, and I found a ton of people that commited it! Thank you very much! That is mine now!

Then all that was required for me to do was optimize this so we could calculate poses often enough for the gameplay experience to be considered smooth, and to actually calculate poses. For this what I did is create an enum with all directions and then return the direction of each lower-arm. Then, if you take that + the height of your wrist compared to your shoulder, we have our poses! 🧠

Debugging & Testing

Obviously we wanted our software to always work. What if a user has no webcam connected? What if the webcam is slanted? What if the DNN model file was missing? So many things to consider while developing! 🤯

To test such circumstances, we can either:

• Test all scenarios manually and not have time to see our families and children.
• Or write unit tests!

Obviously we chose to write unit tests! We chose for utest.h, a very underrated project! It is a single header unit testing framework for C & C++ and works magnificently. 😍 A simple test looks like this:

#include "utest.h"

UTEST(foo, bar) {
  ASSERT_TRUE(1);
}

In fact, we can even test the pose estimation code we wrote by creating some test pictures and using that as input for some tests. Brilliant!

Furthermore, it is obviously very important to print information we need to figure out if everything is working as it should! Introducing: Easylogging++. This helped us a ton! For example, this is what one of my colleagues saw after forgetting to download the DNN model:

GUI

How would one draw the instructions and the scores on the screen? 🤔

Time for a GUI! 😁

For our project we decided to use Dear ImGui, a bloat-free graphical user interface for C++ with minimal dependencies. The reason for this is that it is very popular, and thus we were able to find a lot of resources on how to use this beautiful library. 🤩

This is an example of how this looks:

3D Graphics

For the 3D Graphics we had to code everything ourselves using OpenGL. To get us started with creating a window etc. we used GLFW.

Then we wrote our own model loader. Together with ImGui it looks like this right now!

Furthermore, we are programmers, not 3D artists! Thus, we are using the following assets:

• Kenney's Car Kit
• Kenney's City Kit (Roads)

The Result

In this video you can see the result!

The teachers were very impressed by this project, as it was the only project anyone could take control right away with any instructions regarding controls. It is also worth mentioning they were a big fan of this idea since this could actually be used for training, and that it was able to run so smoothly. Hard work pays off! 🤪

Conclusion

I am very proud of this project! I really liked working on such an enormous project in CPP from scratch and would love to do it more often!

That was it for this read. Thank you very much for your time and 'till next time! 😁👋🏻