Click here to Skip to main content
15,867,568 members
Articles / Programming Languages / C++

ZeroMQ: Diving into the Wire

Rate me:
Please Sign up or sign in to vote.
5.00/5 (9 votes)
11 Jan 2015CC (ASA 3U)9 min read 38.1K   20   1
Examining packets exchanged between ZeroMQ sockets using Wireshark; and studying ZMTP protocol and its grammar that governs these packets through a simple PUSH-PULL example.

My articles about ZeroMQ

Introduction

ZeroMQ defines a Protocol called ZMTP (ZeroMQ Message Transport Protocol) as a transport layer protocol for exchanging messages between two peers over a connected transport layer such as TCP (see RFCs: 23/ZMTP and 37/ZMTP).

A Protocol is a set of rules (from Greek protocollon, first leaf glued to a manuscript describing its consents) that governs data exchanging between two end points. Each protocol has its own set of rules of how data is formatted, when to send, how to manage it once received, etc.

In this article, we will explain the ZMTP protocol using a simple ZeroMQ Push-Pull example. In order to capture the data exchanged between the two sockets, we will use Wireshark.

Wireshark

Wireshark is a free and open source network protocol analyzer. It captures and decodes network packets. One of its intended purposes is to learn network protocol internals.

Wireshark's decoding process uses dissectors that can identify and display protocol's fields and values into human-readable format frames. It supports thousands of dissectors that parse and decode common protocols.

The following Wireshark screenshot shows captured packets exchanged between a ZeroMQ Push and Pull sockets.

Image 1

The main window is composed of three panes:

  1. Packet List Pane: displays a summary of captured packets. By clicking on a packet in this pane, the contents of other two panes change.
  2. Packet Details Pan: displays the protocols and protocol's fields of the selected packet in the packet list pane.
  3. Packet Bytes Pane: displays the content of the selected packet in both hexadecimal and ASCII format and highlights the field selected in the details pane.

The protocol hierarchy in the above Wireshark screenshot is Ethernet-Internet Protocol-Transmission Control Protocol. In each protocol level, the protocol dissector decode its part of the protocol and it passes the data on to the lowest-level data dissector.

One of the key strengths of Wireshark is the ability to add new custom dissectors to it, either as plugins or built into the source.

We notice that our socket message's data displayed in the TCP level is not decoded because the ZMTP dissector has not yet been installed. The ZMTP dissector (zmtp-wireshark) is a Wireshark plugin written in Lua and supports ZMTP 3.0 and later.

After installing this dissector, here is the same conversation between the Push and Pull sockets as shown above but with ZMTP dissector this time.

Image 2

We notice that the packets in the “Packet List” pane has been decoded and displayed ZMTP informations in the columns. We notice also how this dissector decoded the ZMTP data and displayed its fields in a readable and elegant format in the “Packet Details” pane.

We will explain each ZMTP element in the following sections.

PUSH-PULL Pattern Example

In this example, we have one push socket connected to one pull socket.

Image 3

Here is the code:

C++
#include "zhelpers.hpp"

int main() {
    //  Create context
    zmq::context_t context(1);

    // Create server socket
    zmq::socket_t server (context, ZMQ_PUSH);
    server.bind("tcp://*:5557");

    // Create client socket
    zmq::socket_t client (context, ZMQ_PULL);
    client.connect("tcp://localhost:5557");

    // Send message from server to client
    s_send (server, "My Message");
    std::string msg = s_recv (client);
    std::cout << "Received: " << msg << std::endl;

    server.close();
    client.close();
    context.close();
}

The Push socket will send a single message to Pull socket.

Capturing Packets

Start Wireshark and add 'zmtp' filter to filter only the zmpt packets, then run the Push-Pull example. The captured packets are shown in the following screenshot:

Image 4

The ZMTP dissector has recognized its packets, decoded and displayed them in a readable format in the 'Packet List' and 'Packet Details' panes.

ZMTP packets exchanged between two sockets are called 'connection'. A 'connection' is composed of three groups of packets: greeting, handshake and traffic as shown in the above screenshot.

The ABNF grammar that defines ZMTP is:

zmtp = *connection
connection = greeting handshake traffic

Now, let us examine each packet:

Greeting

A greeting is composed of 64 octets containing data sent by peers in order to agree on version and security mechanism of the connection.

A greeting consists of signature, version, mechanism, as-server and filler. The ABNF grammar that defines the greeting is:

greeting = signature version mechanism as-server filler

Image 5

Each greeting displayed by the dissector is composed (in this example) of three packets exchanged between the two sockets, to see these packets:

  1. Click on any packet with right hand mouse button (in 'Packet List' pane) and select 'Follow TCP Stream' from the context menu.
  2. Select 'Hex Dump' radio button.

Image 6

In the 'Follow TCP Stream' dialog box, the packets data are colored with blue and red. The blue color indicates packets from Push to Pull socket while packets from Pull to Push socket is marked in red.

Signature & Version

The ZMTP signature is 10 bytes followed by the ZMTP version of two bytes. The ABNF grammar that defines the signature & version is:

signature = %xFF padding %x7F
padding = 8OCTET
version = version-major version-minor
version-major = %x03
version-minor = %x00

The ZMTP signature and version are partial parts of the greeting enabling a peer to detect and work with older versions of the protocol, that means a peer may downgrade its protocol to talk to a lower protocol peer. But, if a peer cannot downgrade its protocol to match its peer, it will close the connection.

In our example, the ZMTP version used by the two peers is 3.0 (major = 3, minor = 0).
Noting that padding field in the signature may be used for older protocol detection.

Mechanism

The security mechanism is an ASCII string null-padded as needed to fit 20 octets. The ABNF grammar that defines the security mechanism is:

mechanism = 20mechanism-char
mechanism-char = "A"-"Z" | DIGIT | "-" | "_" | "." | "+" | %x0 

The security mechanism ensures for a peer the identity of the other peer it talks to, so that messages cannot be tampered with, nor inspected, by third parties. The security mechanism defines also the handshake phase which is composed of some packets exchanged between peers after greetings. If a peer receives a security mechanism that does not exactly match the security mechanism it sent, it will close the connection.

In our example, the sockets define no security mechanism “NULL”, that means there is no authentication and no confidentiality.

The “NULL” security mechanism defines one command exchanged between the peers forming the handshaking phase which we will see later in this article.

AS-Server

The 'as-server' is composed of one byte. The ABNF grammar that defines the as-server is:

as-server = %x00 | %x01 

The “as-server” indicates if the peer is acting as a server (value is 1) or as a client (value is 0). These values are defined by the security mechanism and they are not related to socket bind/connection direction (for example in the 'PLAIN' security mechanism the peer defined as a client authenticates itself to the peer defined as server by sending a HELLO command. The server accepts or rejects this authentication).

The NULL security mechanism dose not specify a client and a server topology, so “as-server” field should be always zero for all peers.

Filler

The “filler” extends the greeting to 64 octets with zeros and its grammar is:

filler = 31%x00 

Handshake


Framing Data

After greetings, all data is sent as frames. A frame consists of a flags field (1 octet), followed by a size field (one octet or eight octets) and a frame body of size octets. The size does not include the flags field, nor itself, so an empty frame has a size of zero.

The flags consists of a single octet containing various control flags:

  • Bit 0 (MORE)

    • 0 : indicates that there are no frames to follow

    • 1 : indicates that there is another frame to follow.

  • Bit 1 (LONG):

    • 0 : indicates that the frame size is encoded as a 64-bit unsigned integer in network byte order

    • 1 : indicates that the frame size is encoded as a single octet

  • Bit 2 (COMMAND)

    • 0 : indicates that the frame is a message frame

    • 1 : indicates that this frame is a command frame

  • Bits 3-7: reserved for future use and must be zero.

Examples of frames are discussed in following sections.


Handshake is composed of zero or more commands defined by the security mechanism in the greetings. A command is a single long or short frame. The ABNF grammar that defines any command is:

command = command-size command-body
command-size = %x04 short-size | %x06 long-size
short-size = OCTET          ; Body is 0 to 255 octets
long-size = 8OCTET          ; Body is 0 to 2^63-1 octets
command-body = command-name command-data
command-name = OCTET 1*255command-name-char
command-name-char = ALPHA
command-data = *OCTET

Handshake is an extension protocol allowing peers to create a secure connection. If the security handshake is successful, the peers continue the discussion, otherwise one or both peers closes the connection.

We can see that the rule “command-size” in the above grammar starts either by 0x04 or by 0x06 which represents the flags field of a frame. The flags 0x04 has only Bit2 set to 1, which means that this frame is a single short command frame. The flags 0x06 has Bit1 and Bit2 set to 1, which means that this frame is a single large frame.

The NULL security mechanism defines a READY command exchanged between peers. A READY command consists of a list of properties. Each property consists of a name-value pair.

The ABNF grammar that defines the NULL security mechanism is:

null = ready *message | error
ready = command-size %d5 "READY" metadata
command-size = %x04 short-size | %x06 long-size
short-size = OCTET          ; Body is 0 to 255 octets
long-size = 8OCTET          ; Body is 0 to 2^63-1 octets
metadata = *property
property = name value
name = OCTET 1*255name-char
name-char = ALPHA | DIGIT | "-" | "_" | "." | "+"
value = 4OCTET *OCTET       ; Size in network byte order

Image 7

The READY command in our example contains a property named “Socket-Type” which defines the sender's socket type. The value of this property is “PUSH” when the push socket sends this command and “PULL” when the pull socket sends it.

A peer validates that the other peer is using a valid socket type (valid combination of sockets). In our example, Push peer validates that the other peer has a Pull socket type and vice versa. If the validation is not succeeded, then the connection will be closed.

Traffic

A traffic consists of commands and messages intermixed.

The ABNF grammar that defines the traffic is:

traffic = *(command | message) 

The grammar of “command” is already defined above, and here is the ABNF grammar of a message:

message = *message-more message-last
message-more = ( %x01 short-size | %x03 long-size ) message-body
message-last = ( %x00 short-size | %x02 long-size ) message-body
short-size = OCTET          ; Body is 0 to 255 octets
long-size = 8OCTET          ; Body is 0 to 2^63-1 octets
message-body= *OCTET 

The flags byte of the message frame is defined in the rules “message-more” and “message-last”. This field can take four values:

0x00: indicates that this frames is a short last-frame message
0x02: indicates that this frames is a long last-frame message
0x01: indicates that this frames is a short more-frame message
0x03: indicates that this frames is a long more-frame message

In our example, the traffic consists of one message:

Image 8

Now, we will send a multi frame message. The first frame is a long message (its length > 255 bytes) and the second frame is a short message (the same message that we sent before).

C++
s_sendmore (server, std::string(256, 'a'));
s_send (server, "My Message");

Image 9

In the above screenshot, I didn't display all long message bytes in the 'Packet Bytes” pane. We notice that the first frame's flags indicate that this frame is followed by another one (More) and it's a long frame (bit 1 is set to one). The payload length is encoded as a 64-bit unsigned integer (8 bytes) because it's greater that 255 bytes.

The second frame is followed directly after the first one. Its flags indicate that it's the last frame (bit 0 is set to zero) and that it's a short frame (bit 1 is set to zero) since the payload length is less than 256 bytes.

Conclusion

ZMTP is a protocol that governs data exchanging between ZeroMQ sockets. It defines a certain number of rules: protocol version, security mechanism, defining discrete messages (frames), metadata (single/multi frames, short/long message), etc.

License

This article, along with any associated source code and files, is licensed under The Creative Commons Attribution-Share Alike 3.0 Unported License


Written By
Architect
France France
Software Architect, Consultant and Solution Initiator with over 20 years of industry experience designing and developing software in different domains: semiconductor manufacturing, electrical energy distribution, solar energy production, medical device manufacturing, PC manufacturing etc.

Comments and Discussions

 
QuestionUnable to Decode Packets Pin
Member 1468451910-Dec-19 0:57
Member 1468451910-Dec-19 0:57 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.