Decoding and Encoding JSON with Arduino or ESP8266

In this blog post you’re going to learn how to decode (parse a JSON string) and encode (generate a JSON string) with the ArduinoJson library using the Arduino with the Ethernet shield. This guide also works with the ESP8266 and ESP32 Wi-Fi modules with small changes.

Important: this tutorial is only compatible with the ArduinoJSON library 5.13.5.

What is JSON?

JSON stands for JavaScript Object Notation. JSON is a lightweight text-based open standard design for exchanging data.

JSON is primarily used for serializing and transmitting structured data over network connection – transmit data between a server and a client. It is often used in services like APIs (Application Programming Interfaces) and web services that provide public data.

JSON syntax basics

In JSON, data is structured in a specific way. JSON uses symbols like { } , : ” ” [ ] and it has the following syntax:

• Data is represented in key/value pairs
• The colon (:) assigns a value to key
• key/value pairs are separated with commas (,)
• Curly brackets hold objects ({ })
• Square brackets hold arrays ([ ])

For example, to represent data in JSON, the key/value pairs come as follows:

{"key1":"value1", "key2":"value2", "key3":"value3"}

JSON examples

In a real world example, you may want structure data about a user:

{"name":"Rui", "country": "Portugal", "age":24}

Or in a IoT project, you may want to structure data from your sensors:

{"temperature":27.23, "humidity":62.05, "pressure":1013.25}

In JSON, the values can be another JSON object (sports) or an array (pets) . For example:

{
  "name": "Rui",
  "sports": {
    "outdoor": "hiking",
    "indoor": "swimming"
  },
  "pets": [
    "Max",
    "Dique"
  ]
}

Here we are structuring data about a user and we have several keys: “name”, “sports” and “pets”.

The name has the value Rui assigned. Rui may practice different sports relating to where they are practiced. So, we create another JSON object to save Rui’s favorite sports. This JSON object is the value of the “sports” key.

The “pets” key has an array that contains Rui’s pets’ names and it has the values “Max” and “Dique” inside.

Most APIs return data in JSON and most values are JSON objects themselves. The following example shows the data provided by a weather API.

{  
  "coord":{  
     "lon":-8.61,
     "lat":41.15
  },
  "weather":[  
    {  
      "id":803,
      "main":"Clouds",
      "description":"broken clouds",
      "icon":"04d"
    }
  ],
  "base":"stations",
  "main":{  
    "temp":288.15,
    "pressure":1020,
    "humidity":93,
    "temp_min":288.15,
    "temp_max":288.15
  },
  (...)
}

This API provides a lot of information. For example, the first lines store the coordinates with the longitude and latitude.

Arduino with Ethernet shield

The examples in this post use an Arduino with an Ethernet shield. Just mount the shield onto your Arduino board and connect it to your network with an RJ45 cable to establish an Internet connection (as shown in the figure below).

Note: the examples provided in this tutorial also work with the ESP8266 and ESP32 with small changes.

Preparing the Arduino IDE

The easiest way to decode and encode JSON strings with the Arduino IDE is using the ArduinoJson library 5.13.5 which was designed to be the most intuitive JSON library, with the smallest footprint and most efficiently memory management for Arduino.

It has been written with Arduino in mind, but it isn’t linked to Arduino libraries so you can use this library in any other C++ project. There’s also a documentation website for the library with examples and with the API reference.

Features

• JSON decoding (comments are supported)
• JSON encoding (with optional indentation)
• Elegant API, very easy to use
• Fixed memory allocation (zero malloc)
• No data duplication (zero copy)
• Portable (written in C++98)
• Self-contained (no external dependency)
• Small footprint
• Header-only library
• MIT License

Compatible with

• Arduino boards: Uno, Due, Mini, Micro, Yun…
• ESP8266, ESP32 and WeMos boards
• Teensy, RedBearLab boards, Intel Edison and Galileo
• PlatformIO, Particle and Energia

Installing the ArduinoJson library

For this project you need to install the ArduinoJson library in your Arduino IDE:

1. Click here to download the ArduinoJson version 5.13.5. You should have a .zip folder in your Downloads folder
2. Unzip the .zip folder and you should get ArduinoJson-master folder
3. Rename your folder from ArduinoJson-master to ArduinoJson
4. Move the ArduinoJson folder to your Arduino IDE installation libraries folder
5. Finally, re-open your Arduino IDE

Decoding JSON – Parse JSON string

Let’s start by decoding/parsing the next JSON string:

{"sensor":"gps","time":1351824120,"data":[48.756080,2.302038]}

Import the ArduinoJson library:

#include <ArduinoJson.h>

Arduino JSON uses a preallocated memory pool to store the JsonObject tree, this is done by the StaticJsonBuffer. You can use ArduinoJson Assistant to compute the exact buffer size, but for this example 200 is enough.

StaticJsonBuffer<200> jsonBuffer;

Create a char array called json[] to store a sample JSON string:

char json[] = "{\"sensor\":\"gps\",\"time\":1351824120,\"data\":[48.756080,2.302038]}"; 

Use the function parseObject() to decode/parse the JSON string to a JsonObject called root.

JsonObject& root = jsonBuffer.parseObject(json);

To check if the decoding/parsing was successful, you can call root.success():

if(!root.success()) {
  Serial.println("parseObject() failed");
  return false;
}

The result can be false for three reasons:

• the JSON string has invalid syntax;
• the JSON string doesn’t represent an object;
• the StaticJsonBuffer is too small – use ArduinoJson Assistant to compute the buffer size.

Now that the object or array is in memory, you can extract the data easily. The simplest way is to use the JsonObject root:

const char* sensor = root["sensor"];
long time = root["time"];
double latitude = root["data"][0];
double longitude = root["data"][1];

You can use the decoded variables sensor, time, latitude or longitude in your code logic.

OpenWeatherMap API

For a real example using an Arduino with an Ethernet shield, we’re going to use a free API from OpenWeatherMap to request the day’s weather forecast for your chosen location.

Learning to use APIs is a great skill because it allows you access to a wide variety of constantly-changing information, such as the current stock price, the currency exchange rate, the latest news, traffic updates, and much more.

Using the API

OpenWeatherMap’s free plan provides everything you need for thins example. To use the API you need an API key, known as the APIID. To get an APIID:

1. Open a browser and go to OpenWeatherMap
2. Press the Sign up button and create a free account
3. Once your account is created, you’ll be presented with a dashboard that contains several tabs (see figure below)
4. Select the API Keys tab and copy your unique Key

This is a unique key you need to pull information from the site. Copy and paste this key somewhere, you’ll need it in a moment.

To pull information on weather in your chosen location, enter the following URL with the sections in curly brackets replaced with your chosen location information and your unique API key:

http://api.openweathermap.org/data/2.5/weather?q={your city},{your country code}&APPID={your API Key}

Replace {your city} with the city you want data for, {your country code} with the country code for that city, and {your API key} with your unique API key we found previously. For example, our API URL for the town of Porto in Portugal, after replacing with the details, would be:

http://api.openweathermap.org/data/2.5/weather?q=Porto,PT&APPID=801d2603e9f2e1c70e042e4------

Note: more information on using the API to get weather information is available here.

Copy your URL into your browser and it should give you a bunch of information that corresponds to your local weather information.

In our case, it returns the weather in Porto, Portugal on the day of writing:

{
  "coord": {
    "lon": -8.61,
    "lat": 41.15
  },
  "weather": [
    {
      "id": 701,
      "main": "Mist",
      "description": "mist",
      "icon": "50d"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 290.86,
    "pressure": 1014,
    "humidity": 88,
    "temp_min": 290.15,
    "temp_max": 292.15
  },
  (...)
}

Making an API Request with Arduino

Now that you have a URL that returns your local weather data. You can automate this task and access that data in your Arduino or ESP8266 projects. Here’s the full script that you need to upload to your Arduino with Ethernet shield to return the temperature in Kelvin and humidity:

/*
 * Rui Santos 
 * Complete Project Details https://randomnerdtutorials.com
 * Based on the Arduino Ethernet Web Client Example
 * and on the sketch "Sample Arduino Json Web Client" of the Arduino JSON library by Benoit Blanchon (bblanchon.github.io/ArduinoJson)
 */

#include <ArduinoJson.h>
#include <Ethernet.h>
#include <SPI.h>

EthernetClient client;

// Name address for Open Weather Map API
const char* server = "api.openweathermap.org";

// Replace with your unique URL resource
const char* resource = "REPLACE_WITH_YOUR_URL_RESOURCE";

// How your resource variable should look like, but with your own COUNTRY CODE, CITY and API KEY (that API KEY below is just an example):
//const char* resource = "/data/2.5/weather?q=Porto,pt&appid=bd939aa3d23ff33d3c8f5dd1";

const unsigned long HTTP_TIMEOUT = 10000;  // max respone time from server
const size_t MAX_CONTENT_SIZE = 512;       // max size of the HTTP response

byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};

// The type of data that we want to extract from the page
struct clientData {
  char temp[8];
  char humidity[8];
};

// ARDUINO entry point #1: runs once when you press reset or power the board
void setup() {
  Serial.begin(9600);
  while (!Serial) {
    ;  // wait for serial port to initialize
  }
  Serial.println("Serial ready");
  if(!Ethernet.begin(mac)) {
    Serial.println("Failed to configure Ethernet");
    return;
  }
  Serial.println("Ethernet ready");
  delay(1000);
}

// ARDUINO entry point #2: runs over and over again forever
void loop() {
  if(connect(server)) {
    if(sendRequest(server, resource) && skipResponseHeaders()) {
      clientData clientData;
      if(readReponseContent(&clientData)) {
        printclientData(&clientData);
      }
    }
  }
  disconnect();
  wait();
}

// Open connection to the HTTP server
bool connect(const char* hostName) {
  Serial.print("Connect to ");
  Serial.println(hostName);

  bool ok = client.connect(hostName, 80);

  Serial.println(ok ? "Connected" : "Connection Failed!");
  return ok;
}

// Send the HTTP GET request to the server
bool sendRequest(const char* host, const char* resource) {
  Serial.print("GET ");
  Serial.println(resource);

  client.print("GET ");
  client.print(resource);
  client.println(" HTTP/1.1");
  client.print("Host: ");
  client.println(host);
  client.println("Connection: close");
  client.println();

  return true;
}

// Skip HTTP headers so that we are at the beginning of the response's body
bool skipResponseHeaders() {
  // HTTP headers end with an empty line
  char endOfHeaders[] = "\r\n\r\n";

  client.setTimeout(HTTP_TIMEOUT);
  bool ok = client.find(endOfHeaders);

  if (!ok) {
    Serial.println("No response or invalid response!");
  }
  return ok;
}

// Parse the JSON from the input string and extract the interesting values
// Here is the JSON we need to parse
/*{
    "coord": {
        "lon": -8.61,
        "lat": 41.15
    },
    "weather": [
        {
            "id": 800,
            "main": "Clear",
            "description": "clear sky",
            "icon": "01d"
        }
    ],
    "base": "stations",
    "main": {
        "temp": 296.15,
        "pressure": 1020,
        "humidity": 69,
        "temp_min": 296.15,
        "temp_max": 296.15
    },
    "visibility": 10000,
    "wind": {
        "speed": 4.6,
        "deg": 320
    },
    "clouds": {
        "all": 0
    },
    "dt": 1499869800,
    "sys": {
        "type": 1,
        "id": 5959,
        "message": 0.0022,
        "country": "PT",
        "sunrise": 1499836380,
        "sunset": 1499890019
    },
    "id": 2735943,
    "name": "Porto",
    "cod": 200
}*/

bool readReponseContent(struct clientData* clientData) {
  // Compute optimal size of the JSON buffer according to what we need to parse.
  // See https://bblanchon.github.io/ArduinoJson/assistant/
  const size_t bufferSize = JSON_ARRAY_SIZE(1) + JSON_OBJECT_SIZE(1) + 
      2*JSON_OBJECT_SIZE(2) + JSON_OBJECT_SIZE(4) + JSON_OBJECT_SIZE(5) + 
      JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(12) + 390;
  DynamicJsonBuffer jsonBuffer(bufferSize);

  JsonObject& root = jsonBuffer.parseObject(client);

  if (!root.success()) {
    Serial.println("JSON parsing failed!");
    return false;
  }

  // Here were copy the strings we're interested in using to your struct data
  strcpy(clientData->temp, root["main"]["temp"]);
  strcpy(clientData->humidity, root["main"]["humidity"]);
  // It's not mandatory to make a copy, you could just use the pointers
  // Since, they are pointing inside the "content" buffer, so you need to make
  // sure it's still in memory when you read the string

  return true;
}

// Print the data extracted from the JSON
void printclientData(const struct clientData* clientData) {
  Serial.print("Temp = ");
  Serial.println(clientData->temp);
  Serial.print("Humidity = ");
  Serial.println(clientData->humidity);
}

// Close the connection with the HTTP server
void disconnect() {
  Serial.println("Disconnect");
  client.stop();
}

// Pause for a 1 minute
void wait() {
  Serial.println("Wait 60 seconds");
  delay(60000);
}

View raw code

Note: make sure your replace the resource variable with your unique OpenWeatherMap URL resource:

const char* resource = "REPLACE_WITH_YOUR_URL_RESOURCE";

Modifying the code for your project

In this example, the Arduino performs an HTTP GET request to a desired service (in this case the OpenWeatherMap API), but you could change it to request any other web service. We won’t explain the Arduino code line by line.

For this project it’s important that you understand what you need to change in the Arduino code to decode/parse any JSON response. Follow these next three steps.

STEP #1 – struct

Create a data structure that can store the information that you’ll want to extract from the API. In this case, we want to store the temperature and humidity in char arrays:

struct clientData {
  char temp[8];
  char humidity[8];
};

STEP #2 – JsonBuffer size

Go to the ArduinoJson Assistant and copy the full OpenWeatherMap API response to the Input field.

Copy the Expression generated (see the preceding figure), in my case:

JSON_ARRAY_SIZE(1) + JSON_OBJECT_SIZE(1) + 2*JSON_OBJECT_SIZE(2) + JSON_OBJECT_SIZE(4) + JSON_OBJECT_SIZE(5) + JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(12)

You’ll need to edit the readReponseContent() function with your the generated JsonBuffer size from ArduinoJson Assistant to allocate the appropriate memory for decoding the JSON response from an API:

bool readReponseContent(struct clientData* clientData) {
  const size_t bufferSize = JSON_ARRAY_SIZE(1) + JSON_OBJECT_SIZE(1) + 
    2*JSON_OBJECT_SIZE(2) + JSON_OBJECT_SIZE(4) + JSON_OBJECT_SIZE(5) + 
    JSON_OBJECT_SIZE(6) + JSON_OBJECT_SIZE(12) + 390; 

  DynamicJsonBuffer jsonBuffer(bufferSize);
  JsonObject& root = jsonBuffer.parseObject(client);

Still inside the readReponseContent() function you need to copy the variables that you need for your project to your struct data:

strcpy(clientData->temp, root["main"]["temp"]);
strcpy(clientData->humidity, root["main"]["humidity"]);

STEP #3 – accessing the decoded data

Then, you can easily access the decoded JSON data in your Arduino code and do something with it. In this example we’re simply printing the temperature in Kelvin and humidity in the Arduino IDE serial monitor:

void printclientData(const struct clientData* clientData) {
  Serial.print("Temp = ");
  Serial.println(clientData->temp);
  Serial.print("Humidity = ");
  Serial.println(clientData->humidity);
}

Demonstration

Open the Arduino IDE serial monitor at a baud rate of 9600 and you’ll see the temperature in Kelvin and the humidity in percentage being printed in the Serial monitor every 60 seconds.

You can access the rest of the information in the OpenWeatherMap API response, but for demonstration purposes we only decoded the temperature and humidity.

Encoding JSON – Generate JSON string

Let’s learn how to encode/generate the next JSON string:

{"sensor":"gps","time":1351824120,"data":[48.756080,2.302038]}

You can read the docs about enconding here.

Import the ArduinoJson library:

#include <ArduinoJson.h>

Arduino JSON uses a preallocated memory pool to store the object tree, this is done by the StaticJsonBuffer. You can use ArduinoJson Assistant to compute the exact buffer size, but for this example 200 is enough.

StaticJsonBuffer<200> jsonBuffer;

Create a JsonObject called root that will hold your data. Then, assign the values gps and 1351824120 to the sensor and time keys, respectively:

JsonObject& root = jsonBuffer.createObject();
root["sensor"] = "gps";
root["time"] = 1351824120;

Then, to hold an array inside a data key, you do the following:

JsonArray& data = root.createNestedArray("data");
data.add(48.756080);
data.add(2.302038);

It is very likely that you’ll need to print the generated JSON in your Serial monitor for debugging purposes, to do that:

root.printTo(Serial);

After having your information encoded in a JSON string you could post it to another device or web service as shown in the next example.

Encoding example with Arduino and Node-RED

For this example you need to Node-RED or a similar software that can receive HTTP POST requests. You can install Node-RED on your computer, but I recommend running Node-RED on a Raspberry Pi.

Creating the flow

In this flow, you’re going to receive an HTTP POST request and print the receive data in the Debug window. Follow these next 6 steps to create your flow:

1) Open the Node-RED software in your browser

2) Drag an HTTP input node and a debug node

3) Edit the HTTP input by adding the POST method and the /json-post-example URL

4) You can leave the default settings for the debug node

5) Connect your nodes

6) To save your application, you need to click the deploy button on the top right corner

Your application is saved and ready.

Send JSON data with Arduino

After having Node-RED prepared to receive POST requests at the /json-post-example URL, you can use the next code example on an Arduino with an Ethernet shield to send data to it.

/*
 * Rui Santos 
 * Complete Project Details https://randomnerdtutorials.com
 * Based on the Arduino Ethernet Web Client Example
 * and on the sketch "Sample Arduino Json Web Client" of the Arduino JSON library by Benoit Blanchon (bblanchon.github.io/ArduinoJson)
 */

#include <ArduinoJson.h>
#include <Ethernet.h>
#include <SPI.h>

EthernetClient client;

// Replace with your Raspberry Pi IP address
const char* server = "REPLACE_WITH_YOUR_RASPBERRY_PI_IP_ADDRESS";

// Replace with your server port number frequently port 80 - with Node-RED you need to use port 1880
int portNumber = 1880;

// Replace with your unique URL resource
const char* resource = "/json-post-example";

const unsigned long HTTP_TIMEOUT = 10000;  // max respone time from server
const size_t MAX_CONTENT_SIZE = 512;       // max size of the HTTP response

byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED};

// ARDUINO entry point #1: runs once when you press reset or power the board
void setup() {
  Serial.begin(9600);
  while(!Serial) {
    ;  // wait for serial port to initialize
  }
  Serial.println("Serial ready");
  if(!Ethernet.begin(mac)) {
    Serial.println("Failed to configure Ethernet");
    return;
  }
  Serial.println("Ethernet ready");
  delay(1000);
}

// ARDUINO entry point #2: runs over and over again forever
void loop() {
  if(connect(server, portNumber)) {
    if(sendRequest(server, resource) && skipResponseHeaders()) {
      Serial.print("HTTP POST request finished.");
    }
  }
  disconnect();
  wait();
}

// Open connection to the HTTP server (Node-RED running on Raspberry Pi)
bool connect(const char* hostName, int portNumber) {
  Serial.print("Connect to ");
  Serial.println(hostName);

  bool ok = client.connect(hostName, portNumber);

  Serial.println(ok ? "Connected" : "Connection Failed!");
  return ok;
}

// Send the HTTP POST request to the server
bool sendRequest(const char* host, const char* resource) {
  // Reserve memory space for your JSON data
  StaticJsonBuffer<200> jsonBuffer;
  
  // Build your own object tree in memory to store the data you want to send in the request
  JsonObject& root = jsonBuffer.createObject();
  root["sensor"] = "dht11";
  
  JsonObject& data = root.createNestedObject("data");
  data.set("temperature", "30.1");
  data.set("humidity", "70.1");
  
  // Generate the JSON string
  root.printTo(Serial);
  
  Serial.print("POST ");
  Serial.println(resource);

  client.print("POST ");
  client.print(resource);
  client.println(" HTTP/1.1");
  client.print("Host: ");
  client.println(host);
  client.println("Connection: close\r\nContent-Type: application/json");
  client.print("Content-Length: ");
  client.print(root.measureLength());
  client.print("\r\n");
  client.println();
  root.printTo(client);

  return true;
}

// Skip HTTP headers so that we are at the beginning of the response's body
bool skipResponseHeaders() {
  // HTTP headers end with an empty line
  char endOfHeaders[] = "\r\n\r\n";

  client.setTimeout(HTTP_TIMEOUT);
  bool ok = client.find(endOfHeaders);

  if(!ok) {
    Serial.println("No response or invalid response!");
  }
  return ok;
}

// Close the connection with the HTTP server
void disconnect() {
  Serial.println("Disconnect");
  client.stop();
}

// Pause for a 1 minute
void wait() {
  Serial.println("Wait 60 seconds");
  delay(60000);
}

View raw code

Note: make sure your replace the server variable with your Raspberry Pi IP address:

const char* server = "REPLACE_WITH_YOUR_RASPBERRY_PI_IP_ADDRESS";

Modifying the code for your project

In this example, the Arduino performs an HTTP POST request to Node-RED, but you could change it to make request another web service or server. We won’t explain the Arduino code line by line. For this project it’s important that you understand what you need to change in the Arduino code to encode/generate a JSON request.

sendRequest() function

For this project you can modify the sendRequest() function with your own JSON data structure:

bool sendRequest(const char* host, const char* resource) {

Start by reserving memory space for your JSON data. You can use ArduinoJson Assistant to compute the exact buffer size, but for this example 200 is enough.

StaticJsonBuffer<200> jsonBuffer;

Create a JsonObject called root that will hold your data and assign the values to your keys (in this example we have the sensor key):

JsonObject& root = jsonBuffer.createObject();
root["sensor"] = "dht11";

To hold data inside an array, you do the following:

JsonObject& data = root.createNestedObject("data");
data.set("temperature", "30.1");
data.set("humidity", "70.1");

Print the generated JSON string in the Arduino IDE serial monitor for debugging purposes:

root.printTo(Serial);

The rest of the sendRequest() function is the POST request.

client.print("POST ");
client.print(resource);
client.println(" HTTP/1.1");
client.print("Host: ");
client.println(host);
client.println("Connection: close\r\nContent-Type: application/json");
client.print("Content-Length: ");
client.print(root.measureLength());
client.print("\r\n");
client.println();
root.printTo(client);

Note that you can use the root.measureLength() to determine the length of your generated JSON. The root.printTo(client) send the JSON data to the Ethernet client.

Demonstration

Open the Arduino IDE serial monitor at a baud rate of 9600 and you’ll see the JSON object printed in the serial monitor every 60 seconds.

In the Node-RED debug tab you’ll see the same JSON object being receive every 60 seconds:

Finally, you create functions in Node-RED that do something useful with the received data, but for demonstration purposes we’re just printing the sample data.

Wrapping up

In this tutorial we’ve shown you several examples on how to decode and encode JSON data. You can follow these basic steps to build more advanced projects that require exchanging data between devices.

We hope you’ve found this tutorial useful.

If you liked this project and Home Automation make sure you check our course: Build a Home Automation System for $100.