zeromq

Abstract

This article discusses our strategy for approaching the market served by ØMQ as described in our market analysis whitepaper. We explain our vision of the market, and our guidelines in deciding what to build to serve that market. Centrally, we argue that the market grows from a community, and that the community grows from a set of good standards. Our goal, therefore, is to build ØMQ as a set of new standards that build on existing standards, along with technology that "seeds" the market.

The Rising Tide

In a changing market, many established businesses try to survive by reducing competition, handicapping the standards processes, and creating customer lock-in. This strategy works for established vendors with a captive market who are unable to compete on technological grounds.

Small players like us with sufficiently advanced technology and no captive market, can profitably base their business strategy on increasing competition, strengthening the standards processes, and breaking customer lock-in.

Typically, this creative destruction is impossible for larger firms, but easy for smaller ones. However a rising tide lifts all ships, except those with broken hulls, and we expect many innovative larger vendors to engage with us in making the tide rise.

The Community Makes the Market

When we look at high-performance messaging today we see a fragmented market in which vendors do not truly compete, customers do not have free choice, and large parts of the stack are proprietary and closed. Lock-in, not competition, is the main strategy from vendors. This creates niche markets which may be individually large but are overall insignificant compared to the potential market size.

We can consider a market to consist of:

• Customers seeking for solutions to their business demands.
• Vendors looking to provide technologies, products, and services.
• Brokers and authorities who regulate and moderate the market.

A successful technology market is typified by:

• Standardization right down the stack, ensuring customer choice and vendor competition at all levels;
• Aggressive lowering of cost and risk in stable areas, following the same curve as Moore's Law;
• Rapid growth into new areas built on stable standards, driven by competition between customers in their business areas.

Our mantra is "the Community Makes the Market". We see the customers, vendors, brokers, and authorities as an ecosystem, a community. The community thrives or withers mainly according to the quality, openness, and authority of its standards.

Standards and Communities

We identify several key areas where we can use a standards-based approach to drive interoperability and community growth. These communities merge into and sustain a single market.

The Framework Standard

Our first standard solves the problem of combining messaging components on a single computer system. We can view messaging middleware to be a composite product, a framework that combines third-party components. Rather than provide functionality in a monolithic product, messaging middleware combines transport functionality (sockets, network interfaces, routers), storage functionality (databases, storage devices), scheduling functionality (operating systems), and so on.

In the ideal market, vendors can offer substitutes for any component, and customers can choose these freely, creating overall messaging middleware composites that are highly tuned for performance, functionality and cost.

When we formalize this framework as a set of clear standard specifications, we get a Framework Standard. A Framework Standard can create a significant community. Some notable examples are the Eclipse framework and the Linux kernel framework.

We identify these characteristics of a successful Framework Standard:

1. The Framework Standard consists of a set of clearly-written specifications that define the interfaces between components of a certain class and the framework, or between components of specific classes.
2. The specifications are freely available and usable for any purpose with no license costs.
3. The specifications are strongly enforced by a competent authority.
4. The specifications are fully agnostic with respect to hardware and software choices.
5. The cost of developing, testing, and delivering components kept as low as possible.
6. A standard set of components are built to "seed" the market.

The Protocol Standard

Our second standard solves the problem of combining messaging components that span multiple computer systems. Messaging middleware must solve several problems, like addressing and subscription propagation, that involve software running on more than one system, thus distributed components.

In our ideal market any of these distributed components is fully substitutable by customers. The established way to deliver this substitutability is as a set of clear standard specifications, a Protocol Standard. A Protocol standard can create a significant community, indeed most of the Internet is built upon such standards.

The characteristics of a successful Protocol Standard are similar to those of a successful Framework Standard:

1. The Protocol Standard consists of a set of clearly-written specifications that define the interfaces between distributed components of a specific classes.
2. The specifications are freely available and usable for any purpose with no license costs.
3. The specifications are strongly enforced by a competent authority.
4. The specifications are fully agnostic with respect to hardware and software choices.
5. The cost of developing, testing, and delivering components kept as low as possible.
6. A standard set of components are built to "seed" the market.

The Collaboration Standard

Our final problem is how to create a community that actively collaborates on developing the actual technology we seek to deliver. While competition is essential in some areas, collaboration is just as important in other areas.

The question of how to build a development community has been well tested and the most efficient approach has been called "Free software in an open source community".

Open source licenses have been accurately described as "constitutions for a community". The most successful license is the GPL, mainly because it effectively defines a contract for collaboration that prevents cheating from parties who try to "game the system".

Licensing the software we write under the GPL is a necessary but not sufficient step. We must also:

1. Ensure that copyrights are centralized so that licenses can be enforced, and the software easily relicensed.
2. Define clear processes for contribution, so that the provenance of the software is established without any doubt.
3. Apply the proper licenses for the standards texts we write.

The effect of the GPL is two-fold. First, it ensures share-alike by teams who would otherwise work in secret. Second, it offers the project a funding model, through commercial (opt-out) licenses. We can demonstrate this with a simple example.

A team wishing to build a closed-source software component that plugs-in to the ØMQ framework can write such a component from scratch, or derive it from a standard component. In the first case the team owns the whole copyright; in the second the ownership is mixed. In both cases, if they deliver the component together with the framework as a single product, they must use the GPL license (since the resulting whole is considered a derived product).

In FOSS projects with unclear provenance, the firm has no route to building a proprietary product, since it is not feasible to contact each of the many copyright holders to negotiate a commercial option. With centralized copyright and guaranteed provenance, the firm negotiates a commercial license with the ØMQ copyright holder (FastMQ Inc.) and is thus able to build a classic closed product using the ØMQ technology.

Tuning and Benchmark Standards

Benchmarking lets customers measure the performance of competing components and make an informed choice.

Tuning lets customers configure their software and hardware to get maximum performance.

In both areas, a standards-driven approach should help the community and thus the market. We will develop this area further over time.

Existing Standards

ØMQ of course leverages a whole set of existing standards, and builds on these wherever possible.

The ØMQ Delivery

We thus see the ultimate product of the ØMQ project to be a set of standards and a reference product second. Our initial prototype delivery consists of a proof of concept, without framework or protocol specifications. Our target product delivery consists of:

1. A set of documents that describe the Framework Standard.
2. A set of documents that describe the Protocol Standard.
3. A set of standards for tuning and benchmarking.
4. A lightweight kernel that implements the framework.
5. A set of standard components that complete the framework.

Over time we expect that teams will derive new kernels and components from our reference implementations, adhering to the terms of the GPL, and thus creating the collaboration community. We expect that teams will build competing components that plug-in to the framework ), and distributed components that use the protocol standard to interoperate. We expect that many of these new components will be contributed back to the project but this is not necessary to the creation of a successful community, and market.

Appendix - Community rules

We define a number of basic rules for a healthy and growing community¹:

• Quality of mission. A clear statement of goals ensures overall focus and reduces conflict. A bold statement attracts and holds the very best participants.
• Appropriate funding. Sustainable long-term funding for organizational infrastructure enables contributions from more and better participants.
• Freedom of access. Financial, geographic, or other barriers to entry reduce the diversity of participants, which reduces overall creativity.
• Well-written rules. Clear rules backed by neutral authority resolve most disagreement before it happens, and allow people to trust each other implicitly.
• Vandal resistance. The ability to detect and punish vandals, and to recover from damage ensures that groups can survive the often hostile real world.
• Tools and processes. The quality of tools and processes defines how efficiently participants can aggregate information into knowledge.
• Freedom to organise. Identifying the problems, allocating resources, and monitoring success are better done by the participants than by top-down managers.
• Competent authority. A strong authority that is ultimately responsible for the mission ensures that participants trust the rules and each other.
• Organic growth. Groups need to attract new participants over time, or they become stagnant. Growth requires marketing, especially of competence.
• Decentralised work. Geographic separation and diversity ensures participants think independently, free from the influence of strong opinion makers.
• Transparency. In general, secrecy enables incompetence, and transparency promotes competence. The more public the organisation's work, the better.
• Free workspaces. If the organization offers free workspaces to anyone who wants them, individuals will experiment and invest more.
• Strong identities. Clear identification of contributors is a key tool for detecting and recovering from vandalism.
• Local ownership. When teams and individuals have full control over ("own") their work, they have much more incentive to invest.
• Proper attribution. When contributors are be credited for their work this creates a strong incentive to pro-bono contributors.
• Meritocratic structure. With proper tools to rank and promote individual members based on contribution, the best participants have more influence over the collective process.
• Metrics for success. Between individuals and between subgroups, competition for public recognition is a key motivator.
• Work is easily remixable. It is important to formally promote sharing and discourage privatisation of work, though the use of appropriate licenses.
• Neat structures. Communities need a predictable structure so that new entrants can easily identify the area where they can be most effective.