General

How do I start the ØMQ daemon?

There are no services or daemons to start: ØMQ is a library. Compile and link an example and run it. ØMQ applications speak to each other directly.

How do I use a regular BSD socket to communicate with a ØMQ socket?

ØMQ has its own wire-level protocol. In order for a regular socket to communicate with it, that regular socket would need to send messages conforming to that protocol. Additionally, please read the section in the guide detailing how ØMQ is not a neutral carrier for your data.

Does ØMQ support IP multicast?

Yes, ØMQ includes Pragmatic General Multicast (RFC 3208) support using the excellent OpenPGM implementation.

Does ØMQ store messages on disk?

No. Do not even think about that. (Even when 2.1.x has a feature that looks like that)

Does ØMQ include APIs for serializing data to/from the wire representation?

No. This design decision adheres to the UNIX philosophy of "do one thing and do it well". In the case of ØMQ, that one thing is moving messages, not marshalling data to/from binary representations.

Some middleware products do provide their own serialization API. We believe that doing so leads to bloated wire-level specifications like CORBA (1055 pages). Instead, we've opted to use the simplest wire formats possible which ensure easy interoperability, efficiency and reduce the code (and bug) bloat.

If you wish to use a serialization library, there are plenty of them out there. See for example Google Protocol Buffers.

I'm trying to send data using the pgm/epgm transport and I get "Protocol not supported". Why?

You need to have ØMQ built with the —with-pgm option for this to be enabled.

How can I integrate ØMQ sockets with normal sockets? Or with a GUI event loop?

You can use the zmq_poll() function to poll for events on both ØMQ and normal sockets. The zmq_poll() function accepts a timeout so if you need to poll and process GUI events in the same application thread you can set a timeout and periodically poll for GUI events. See the zmq-camera example for a demonstration.

Why can't I use standard I/O multiplexing functions such as select() or poll() on ØMQ sockets?

ØMQ socket is not a standard POSIX socket. It would be great if it was, however, POSIX doesn't provide a mechanism to simulate file descriptors in user space. To convert ØMQ sockets into POSIX file descriptors we would have either to move ØMQ to kernel-space or hack the kernel to provide the functionality needed. In both cases we would have to sacrifice portability and stick to a single operating system. Note that there's a way to retrieve a file descriptor from ØMQ socket (ZMQ_FD socket option) that you can poll on from version 2.1 onwards, however, there are some serious caveats when using it. Check the documentation carefully before using this feature.

When sending a multipart message, my receiver doesn't get any message parts until the last part is sent. Is ØMQ broken?

The ØMQ library guarantees that multipart messages are sent and received atomically. A message cannot be delivered by the library to your application until all frames have been received. Obviously, this cannot happen while the sender still has parts pending or in queue.

There are two reasons for multipart message support in the library.

1. This function supports a scatter/gather-style for message transmission so the application can avoid memcpy'ing the (potentially) large data to a single message. Zero copy is a huge performance win particularly for large messages.
2. Multipart support may help a protocol designer to logically separate their protocol into separate frames at the application level. This may ease parsing on the receiving end since not all languages support fast bit-twiddling for binary structures like C.

How do I use the ZMQ_SWAP socket feature to persist my data to disk?

It is not the responsibility of the library to handle data persistence. If you need to make sure no messages are ever lost, you need to write application-level code to persist your data to non-volatile memory and handle the recovery modes of your application. Also, ZMQ_SWAP is deprecated and will be removed in the 3.x series of releases since it confuses many people over proper usage.

Is it possible to receive EAGAIN or to block when sending a multi-part message?

Multipart messages are sent as an atomic unit. If one part is sent successfully, then the socket is guaranteed to not block or return EAGAIN until all parts have been sent. Of course, if you run out of memory in the meantime then the guarantee doesn't mean much, so don't try to send messages that exceed available memory.

Are you trying to tell me that ØMQ won't magically save me from out-of-memory conditions?

You are a perfect candidate for using AMQP or JMS. Please use google to find more information on libraries that support those platforms and protocols.

Platform and Patterns

I want to write a program using ØMQ sockets. Which socket types should I use?

Please read the guide. It covers each socket pattern along with their use-cases. For further assistance, please join the mailing list or IRC and ask for help. Please do not be upset if the first question asked is, "Have you read the Guide yet?"

My multi-threaded program keeps crashing in weird places inside the ØMQ library. What am I doing wrong?

ØMQ sockets are not thread-safe. This is covered in some detail in the Guide.

The short version is that sockets should not be shared between threads. We recommend creating a dedicated socket for each thread.

For those situations where a dedicated socket per thread is infeasible, a socket may be shared if and only if each thread executes a full memory barrier before accessing the socket. Most languages support a Mutex or Spinlock which will execute the full memory barrier on your behalf.

For more information, please read the Guide.

Why do I see different behavior when I bind a socket versus connect a socket?

ØMQ creates queues per underlying connection, e.g. if your socket is connected to 3 peer sockets there are 3 messages queues.

With bind, you allow peers to connect to you, thus you don't know how many peers there will be in the future and you cannot create the queues in advance. Instead, queues are created as individual peers connect to the bound socket.

With connect, ØMQ knows that there's going to be at least a single peer and thus it can create a single queue immediately.

Consequently, when sending a message to bound socket with no peers, there's no queue to store the message to. Not so with connected socket.

I'm worried about my application heartbeats queueing up behind lower-priority data and causing a disconnect. How can I send a message with higher priority so it jumps forward in the queue?

There is no concept of message priority enforced by the library. We suggest opening a second set of sockets for the purposes of passing heartbeats. Alternately, modify your application so that it considers any incoming message as a renewal of the heartbeat from its source. This allows the application to save bandwidth and heartbeat only when there is no other traffic on the network.

I need to have multiple sockets share a single TCP connection (host + port). How can I accomplish this?

ØMQ cannot multiplex streams over a single connection. This is being investigated as a future enhancement. Join the mailing list or IRC and let yourself be heard.

I set a HWM (high water mark) for a socket but it isn't working!

That's not a question. Also, please be certain to read the man page for zmq_setsockopt() closely. Certain socket options only take effect for subsequent zmq_bind/zmq_connect calls. We recommend setting all socket options before making any calls to zmq_bind/zmq_connect; that way we don't have to remember these tiny implementation details and can focus on writing great code.

Also, read the next question about HWM below.

How does the HWM (high water mark) work with any socket type?

It works the following way right now:

The I/O thread reads messages from the pipe and pushes them to the network. If network is not able to accept more data (e.g. TCP backpressure is applied) it stops reading messages from the pipe and waits until the network is ready for accepting more data.

In the application thread, messages are simply pushed to the pipe when zmq_send() is called. If the pipe is full (HWM is reached) the message is dropped.

The problem with the above approach is that when you send a lot of messages is a quick sequence (e.g. sending small messages in a tight loop) the messages are stored in the pipe until it is full and the subsequent messages are simply dropped. The sender is not even notified about the fact that messages are disappearing.

The main core developer is hopeful that some community members will volunteer to assist in replacing this mechanism with a rate flow control mechanism.

When running "make test" on OSX, how do I fix the failures?

Please refer to the tuning guide for platform-specific tuning. In short, OSX has some low defaults that the tests overrun so they need to be tuned.

Performance

What is the optimal number of I/O threads for best performance?

The basic heuristic is to allocate 1 I/O thread in the context for every gigabit per second of data that will be sent and received (aggregated). Further, the number of I/O threads should not exceed (number_of_cpu_cores - 1).

The graph in the test results shows that ØMQ is slower than TCP/IP. What's the point then?

Obviously, you would expect system working on top of TCP to have higher latencies than TCP. Anything else would be - simply speaking - supernatural. However, throughput is a different matter. ØMQ gets you more throughput than TCP has using intelligent batching algorithms. Moreover ØMQ delivers value-add over the TCP. Asynchronicity, message queueing, routing based on business logic, multicast etc.

How come ØMQ has higher throughput than TCP although it's built on top of TCP?

Avoiding redundant networking stack traversals can improve throughput significantly. In other words, sending two messages down the networking stack in one go takes much less time then sending each of them separately.This technique is known as message batching.

When sending messages in batches you have to wait for the last one to send the whole batch. This would make the latency of the first message in the batch much worse, wouldn't it?

ØMQ batches messages in opportunistic manner. Rather than waiting for a predefined number of messages and/or predefined time interval, it sends all the messages available at the moment in one go. Imagine the network interface card is busy sending data. Once it is ready to send more data it asks ØMQ for new messages. ØMQ sends all the messages available at the moment. Does it harm the latency of the first message in the batch? No. The message won't be sent earlier anyway because the networking card was busy. On the contrary, latency of subsequent messages will be improved because sending single batch to the card is faster then sending lot of small messages. On the other hand, if network card isn't busy, the message is sent straight away without waiting for following messages. Thus it'll have the best possible latency.

ØMQ's latency is nice, but is there a way to make it even lower?

We are working on delivering ØMQ over alternative networking stacks, thus having advantage of features like kernel bypass, avoiding TCP/IP overhead, using high-performance networking hardware etc. That way we can get the latency as low as 10 microseconds.

Why am I only getting 40 Mbps performance when sending messages using PGM?

PGM uses rate-limiting on the sender side. By default this limit is set to 40 Mbps. You can set it using the ZMQ_RATE option to zmq_setsockopt().

Does the ØMQ library disable the Nagle algorithm (TCP NODELAY)?

Yes.

Monitoring

How do I determine how many messages are in queue?

This isn't possible. At any given time a message may be in the ØMQ sender queue, the sender's kernel buffer, on the wire, in the receiver's kernel buffer or in the receiver's ØMQ receiver queue. Furthermore, a ØMQ socket can bind and/or connect to many peers. Each peer may have different performance characteristics and therefore a different queue depth. Any "queue depth" number is almost certainly wrong, so rather than provide incorrect information the library avoids providing any view into this data.

How can I be notified that a peer has connected/disconnected from my socket?

ØMQ sockets can bind and/or connect to multiple peers simultaneously. The sockets also transparently provide asynchronous connection and reconnection facilities. At this time, none of the sockets will provide notification of peer connect/disconnect. This feature is being investigated for a future release.

How can I retrieve a list of all connected peers?

This is not supported.

How can I auto-discover services provided by a ØMQ-based application?

This type of facility is not supported by the library. Such a tool could be built on top of ØMQ.

Security

Is it true that it is not safe to use ØMQ over the internet because it will crash?

Earlier versions of the ØMQ library (before 2.1) were not very resilient against "fuzzing" attacks. A malformed packet or garbage data could cause an old version of the library to assert and exit. Since the release of 2.1, all reported cases of assertions caused by bad data have been fixed. If your testing uncovers a problem in this area, please file a bug report.

What security features does ØMQ support?

None at the moment. ØMQ does not deal with security by design but concentrates on getting your bytes over the network as fast as possible. Solutions exist for security at the transport layer which are well understood and have had many man-years of development invested in them, such as IPsec or OpenVPN.

Having said that, if you have a use case which precludes the use of transport layer security please let us know and we can discuss your needs.

I read somewhere that I have to run my application as root if I want to use PGM, is this true?

The epgm:// transport uses PGM encapsulated in UDP packets and does not require any special permissions.

If you need to use the raw pgm:// transport then your application must be able to create raw sockets, which means either running as root or with capabilities to do so. On a modern Linux distribution with capabilities enabled you can use the following to run an application with the CAP_NET_RAW capability enabled:

$  sudo execcap 'cap_net_raw=ep' pgmsend moo

For more details please see the relevant OpenPGM wiki page.

Building

What packages do I need to build ØMQ on Ubuntu?

You need build-essential and, for pre-ØMQ/3.0 versions, uuid-dev. So sudo apt-get install build-essential uuid-dev. You can also run sudo apt-get build-dep libzmq0 which installs all the build-dependencies of the libzmq0 ubuntu package.

After cloning the github repository, I can't build the library because the 'configure' script doesn't exist! What do I do?

You need autotools installed for your OS so that the configure script can be generated. Run this code to generate that script.

$ ./autogen.sh