ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller

In this article, we want to share a work developed by one of our readers that shows how to use the ESP32/ESP32-CAM streaming servers in robotic systems using OpenCV or OpenCV.js.

ESP32 ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller

This project/tutorial was created by Andrew R. Sass (one of our readers/followers).

This post is just an introduction to his work (Andrew Sass). The complete work is compiled in a PDF paper with full descriptions and explanations of the code. You can also have access to all the codes used throughout the paper. You can access and download all his work on the GitHub page:

Andrew Sass has already written a tutorial about OpenCV.js color detection and tracking on an ESP32-CAM video streaming server: ESP32-CAM Web Server with OpenCV.js: Color Detection and Tracking.

Introduction

The ESP32-CAM can act as a video streaming server for different clients. In his work, he covers the ESP32-CAM acting as a video streaming server for a browser client and for a Python client window. Then, he shows how to process the video using OpenCV.js or OpenCV to get output for robotics autonomous and manual operation. Additionally, he also created a user interface/control using an XBOX controller.

ESP32-CAM Video Streaming Server – Browser Client

When the ESP32-CAM acts as a video streaming server for a browser client, we can use OpenCV.js on the browser to process the image, and send the results back to the server.

ESP32-CAM Video Streaming Server – Python Client

On the other hand, when the ESP32-CAM acts as a video streaming server for a Python client, we have access to a more extensive library of OpenCV software than is currently available in OpenCV.js, because OpenCV is resident in the users’ laptop as opposed to the ESP32.

However, unlike the Browser Streaming System, data communication back to the server is not inherently available in the Python Streaming System. Since communication back to the server is essential for any complete robotic system, a novel method for this is presented in this work. It is, admittedly more complex, yet hopefully balanced by the advantages of the above-mentioned Python OpenCV library for those users not bound to a Browser application.

General Description of the Systems

Following is a brief introductory pictorial description of the Browser Streaming System operation.

Browser Streaming System

The Browser Server publishes its IP address and is accessed by using a web browser. After pressing the Color Track button on the Chrome screen, the Client video screen is seen below.

ESP32-CAM OpenCV Browser System

The red screen patch above shows sample sensor data transmitted by the ESP32 Server to the Client.

The XBOX Controller transmits Joystick, Button, Trigger, and Hat data to the ESP32. The ESP32 is the front end of the Robot and sends/receives data to the Robot’s main processor. This mode, in which a human operator controls the robot, is called the Teleop Mode.

When the list box selector below the left side of the video is set to NO, the view in the following picture is seen.

ESP32-CAM Robotics Teleop Operation with XBOX Controller

This is the Autonomous Mode in which the robot is under program control. In this mode, OpenCV (computer vision) is used to guide the robot. The top left view is the author wearing a red hat. The bottom right view shows a white dot approximately in the lower center screen. This dot is moveable and probes the position of the screen where the program can measure the RGB content of the picture. In this example, the white dot probe was initially moved to the red hat where the measurement was made and then moved back to its initial position.

The RGB sliders were set to the red hat RGB measurement. The top right view shows the binary image corresponding to the color range set by the sliders. The hat is the largest portion of the image corresponding to the sliders’ ranges. When the Tracking button is pressed, a blue contour rectangle surrounds the biggest single portion of white, identifying it as the target. The coordinates of the centroid of the blue rectangle are shown on the screen and transmitted to the ESP32 and then to the main processor of the robot for Autonomous guidance (see picture below).

A diagram describing the Robot Browser Streaming System Diagram is pictured below.

ESP32-CAM Robotics with OpenCV: how it works

For a complete description of the system, please download the complete PDF paper.

Python Streaming System

The ESP32-CAM Streaming Server (described in the paper) publishes its IP address immediately after download.

The python client program uses this IP address and displays the python window shown on the left in the picture below.

The XBOX controller program, (also described in the paper) also uses this IP address in port 82/ws to display the Graphical User Interface(GUI) shown at the right.

ESP32-CAM OpenCV python streaming system

The white stripe displaying data in the left window of the above figure and the white data at the bottom of the GUI are data exchanged by the two programs. In other words, the XBOX Controller can control the program execution of the Client and the Client data can be used by the XBOX Controller and can be transmitted by the Controller to the ESP32 Server.

In the above figure, the XBOX Controller commands the Client program to identify the RED object. The Client program then transmits the centroid coordinated of the RED object to the XBOX Controller. If the human operator sets the Controller to Autonomous Mode, the centroid coordinates are transmitted to the Server and then to the main processor of the robot for guidance purposes.

The Python Streaming System Diagram is pictured below.

ESP32 ESP32-CAM Robotics with OpenCV XBOX Controller Demonstration

For a complete description of the system, please download the complete PDF paper.

Wrapping Up

This was just a quick description of the systems developed and described by Andrew Sass in his paper. You can access all his work on the following links:

All questions should be posted on his Github page (ESP32 Robot SystemsTutorial repository) so you can have the chance to get your question answered by Andrew.

He had already written another tutorial about color detection and tracking using OpenCV.js that you can check on the following link:

We hope you found this project useful.

Thanks for reading.



Learn how to build a home automation system and we’ll cover the following main subjects: Node-RED, Node-RED Dashboard, Raspberry Pi, ESP32, ESP8266, MQTT, and InfluxDB database DOWNLOAD »
Learn how to build a home automation system and we’ll cover the following main subjects: Node-RED, Node-RED Dashboard, Raspberry Pi, ESP32, ESP8266, MQTT, and InfluxDB database DOWNLOAD »

Recommended Resources

Build a Home Automation System from Scratch » With Raspberry Pi, ESP8266, Arduino, and Node-RED.

Home Automation using ESP8266 eBook and video course » Build IoT and home automation projects.

Arduino Step-by-Step Projects » Build 25 Arduino projects with our course, even with no prior experience!

What to Read Next…


Enjoyed this project? Stay updated by subscribing our newsletter!

3 thoughts on “ESP32-CAM Robotics with OpenCV: Autonomous and Teleop Operation with XBOX Controller”

  1. It looks very sophisticated and complex, but also very interesting. As always, you know how to find the right terms to make this kind of articles very understandable. I will follow with interest.
    Thank you.

    Reply

Leave a Comment

Download Our Free eBooks and Resources

Get instant access to our FREE eBooks, Resources, and Exclusive Electronics Projects by entering your email address below.