Publisher Side Message Filtering

Discussion topics » Publisher Side Message Filtering

Summary

The current state of message filtering in the PUB/SUB topology relies on Subscriber side filtering of messages. The goal of this topic is to address this inefficiency, specifically the sending of unwanted messages, and determine the best path to a solution(s). At the bottom there is a list of posts to the mailing list regarding this topic.

The most straight forward way to address this may be to simply collect use cases. At least this will provide a simple test for proposed solutions, ie does this solution solve all of these use cases.

Use Cases

Example Use Case: Market Data Quote Server
Topology of this case is a simple Server-Client. One server distributes market data to 5-50 clients each subscribed to 10-1000 topics. Number of topics essentially equals the number of symbols, ie for US equities and equity options ~ 450k with up to 500k msg/sec. The number of clients and subscriptions are just made up to make the case more concrete, but the number of symbols and rate of messages correspond to real market data. One of the interesting things about this case is that while PUB side message filtering at least makes this possible with 0MQ, the implementation I am used to follows an Observer pattern. So there is no filtering required, every topic/symbol has a list of subscribers. Message filtering efficiency will obviously be important, which has been addressed here:

http://www.zeromq.org/whitepapers:message-matching

Questions

Should the PUB/SUB topology be simply changed to use PUB side filtering or is it better to have multiple types of PUB/SUB topologies? Is PUB side filtering simply inefficient and to be replaced or are there cases where it is preferred, if only because the inefficiencies in some cases are so small that the simplicity of the current implementation is more important?
How does PUB side filtering compare to using an Observer pattern where every topic has a list of subscribers/listeners?