In this article, we want to share a work developed by one of our readers that shows how to use the ESP32/ESP32-CAM streaming servers in robotic systems using OpenCV or OpenCV.js.
This project/tutorial was created by Andrew R. Sass (one of our readers/followers).
This post is just an introduction to his work (Andrew Sass). The complete work is compiled in a PDF paper with full descriptions and explanations of the code. You can also have access to all the codes used throughout the paper. You can access and download all his work on the GitHub page:
- PDF: Introduction to ESP32 Robotics Systems
- Access all code on Github:
Andrew Sass has already written a tutorial about OpenCV.js color detection and tracking on an ESP32-CAM video streaming server: ESP32-CAM Web Server with OpenCV.js: Color Detection and Tracking.
Introduction
The ESP32-CAM can act as a video streaming server for different clients. In his work, he covers the ESP32-CAM acting as a video streaming server for a browser client and for a Python client window. Then, he shows how to process the video using OpenCV.js or OpenCV to get output for robotics autonomous and manual operation. Additionally, he also created a user interface/control using an XBOX controller.
ESP32-CAM Video Streaming Server – Browser Client
When the ESP32-CAM acts as a video streaming server for a browser client, we can use OpenCV.js on the browser to process the image, and send the results back to the server.
ESP32-CAM Video Streaming Server – Python Client
On the other hand, when the ESP32-CAM acts as a video streaming server for a Python client, we have access to a more extensive library of OpenCV software than is currently available in OpenCV.js, because OpenCV is resident in the users’ laptop as opposed to the ESP32.
However, unlike the Browser Streaming System, data communication back to the server is not inherently available in the Python Streaming System. Since communication back to the server is essential for any complete robotic system, a novel method for this is presented in this work. It is, admittedly more complex, yet hopefully balanced by the advantages of the above-mentioned Python OpenCV library for those users not bound to a Browser application.
General Description of the Systems
Following is a brief introductory pictorial description of the Browser Streaming System operation.
Browser Streaming System
The Browser Server publishes its IP address and is accessed by using a web browser. After pressing the Color Track button on the Chrome screen, the Client video screen is seen below.
The red screen patch above shows sample sensor data transmitted by the ESP32 Server to the Client.
The XBOX Controller transmits Joystick, Button, Trigger, and Hat data to the ESP32. The ESP32 is the front end of the Robot and sends/receives data to the Robot’s main processor. This mode, in which a human operator controls the robot, is called the Teleop Mode.
When the list box selector below the left side of the video is set to NO, the view in the following picture is seen.
This is the Autonomous Mode in which the robot is under program control. In this mode, OpenCV (computer vision) is used to guide the robot. The top left view is the author wearing a red hat. The bottom right view shows a white dot approximately in the lower center screen. This dot is moveable and probes the position of the screen where the program can measure the RGB content of the picture. In this example, the white dot probe was initially moved to the red hat where the measurement was made and then moved back to its initial position.
The RGB sliders were set to the red hat RGB measurement. The top right view shows the binary image corresponding to the color range set by the sliders. The hat is the largest portion of the image corresponding to the sliders’ ranges. When the Tracking button is pressed, a blue contour rectangle surrounds the biggest single portion of white, identifying it as the target. The coordinates of the centroid of the blue rectangle are shown on the screen and transmitted to the ESP32 and then to the main processor of the robot for Autonomous guidance (see picture below).
A diagram describing the Robot Browser Streaming System Diagram is pictured below.
For a complete description of the system, please download the complete PDF paper.
Python Streaming System
The ESP32-CAM Streaming Server (described in the paper) publishes its IP address immediately after download.
The python client program uses this IP address and displays the python window shown on the left in the picture below.
The XBOX controller program, (also described in the paper) also uses this IP address in port 82/ws to display the Graphical User Interface(GUI) shown at the right.
The white stripe displaying data in the left window of the above figure and the white data at the bottom of the GUI are data exchanged by the two programs. In other words, the XBOX Controller can control the program execution of the Client and the Client data can be used by the XBOX Controller and can be transmitted by the Controller to the ESP32 Server.
In the above figure, the XBOX Controller commands the Client program to identify the RED object. The Client program then transmits the centroid coordinated of the RED object to the XBOX Controller. If the human operator sets the Controller to Autonomous Mode, the centroid coordinates are transmitted to the Server and then to the main processor of the robot for guidance purposes.
The Python Streaming System Diagram is pictured below.
For a complete description of the system, please download the complete PDF paper.
Wrapping Up
This was just a quick description of the systems developed and described by Andrew Sass in his paper. You can access all his work on the following links:
- PDF: Introduction to ESP32 Robotics Systems
- Access all code on Github:
All questions should be posted on his Github page (ESP32 Robot SystemsTutorial repository) so you can have the chance to get your question answered by Andrew.
He had already written another tutorial about color detection and tracking using OpenCV.js that you can check on the following link:
We hope you found this project useful.
Thanks for reading.
Video of the work?
It looks very sophisticated and complex, but also very interesting. As always, you know how to find the right terms to make this kind of articles very understandable. I will follow with interest.
Thank you.
Just amazing project that open a lot of possibilities and uses on CV projects.
Thanks for sharing,
AH