Ultra low-latency with CoralFIX and CoralReactor

Please note that this benchmark sends real ExecutionReport FIX messages. It simulates the very real trading scenario where your order is filled and your strategy needs to react as soon as possible. Furthermore, our clients run our benchmarks in their own environment. We provide all the source code of our benchmarks so clients can modify and adapt them in order to reach their own conclusions. They also run their own and very specific benchmarks and tests with similar results. This benchmark not only measures the FIX parser performance, but also and most importantly the network I/O performance. If you want to check our tick-to-trade latencies you can click here.

To test the performance of CoralFIX + CoralReactor we have developed a simple test with a fix server and a fix client. The client connects to the server, the standard FIX handshake through LOGON messages happens and the server proceeds to send 2 million FIX messages to the client. So we are measuring one-way latency from FIX server to FIX client over loopback. For a throughput benchmark instead you can check this article. Note that the server only sends the next message to the client when it gets a message back from it so we can correctly measure the time it takes for each message in isolation to travel from the server to the client. We then compute the average latency, which includes the full CoralFIX parsing time (i.e. encoding and decoding of the FIX message) plus the full CoralReactor TCP network I/O time. Below we present the latency results and the source code.

NOTE: If you want to check the benchmark of just the CoralFIX parser without any network I/O code (i.e. without CoralReactor) you can check this article.

The Results

Messages: 1,800,000 (one-way)
Avg Time: 4.774 micros
Min Time: 4.535 micros
Max Time: 69.516 micros
Garbage created: ZERO
75% = [avg: 4.712 micros, max: 4.774 micros]
90% = [avg: 4.726 micros, max: 4.825 micros]
99% = [avg: 4.761 micros, max: 5.46 micros]
99.9% = [avg: 4.769 micros, max: 7.07 micros]
99.99% = [avg: 4.772 micros, max: 9.481 micros]
99.999% = [avg: 4.773 micros, max: 24.017 micros]

The machine used for the benchmarks was an Intel i7 (4 x 3.5GHz) Ubuntu box overclocked to 4.5GHz.

The Source Code

The server:

package com.coralblocks.coralfix.bench.latency;

import com.coralblocks.coralfix.FixConstants;
import com.coralblocks.coralfix.FixMessage;
import com.coralblocks.coralfix.FixTags;
import com.coralblocks.coralfix.server.FixApplicationServer;
import com.coralblocks.coralreactor.client.Client;
import com.coralblocks.coralreactor.nio.NioReactor;
import com.coralblocks.coralreactor.server.Server;
import com.coralblocks.coralreactor.util.Configuration;
import com.coralblocks.coralreactor.util.MapConfiguration;

public class FixLatencyOneWayServer extends FixApplicationServer {
	
	// java -Xms6g -Xmx8g -server -verbose:gc -Xbootclasspath/p:../CoralReactor-boot-jdk7/target/coralreactor-boot-jdk7.jar -cp target/coralfix-all.jar:lib/jna-3.5.1.jar -DnioReactorProcToBind=2 com.coralblocks.coralfix.bench.latency.FixLatencyOneWayServer
	
	private long id;
	
	public FixLatencyOneWayServer(NioReactor nio, int port, Configuration config) {
	    super(nio, port, config);
    }

	@Override
    protected void handleConnectionOpened(Client client) {
		this.id = 1;
		send(client);
	}
	
	@Override
    protected void handleFixApplicationMessage(Client client, FixMessage fixMsg, boolean possDupe) {
	    send(client);
    }

	private final void send(Client client) {
		
		FixMessage fixOutMsg = getOutFixMessage(client, FixConstants.MsgTypes.ExecutionReport);
		fixOutMsg.addTimestamp(FixTags.TransactTime, nio.currentTimeMillis());
		fixOutMsg.add(FixTags.Account, "01234567");
		fixOutMsg.add(FixTags.OrderQty, 50);
		fixOutMsg.add(FixTags.Price, 400.5);
		fixOutMsg.add(FixTags.ClOrdID, id++);
		fixOutMsg.add(FixTags.HandlInst, '1');
		fixOutMsg.add(FixTags.OrdType, '2');
		fixOutMsg.add(FixTags.Side, '1');
		fixOutMsg.add(FixTags.Symbol, "OC");
		fixOutMsg.add(FixTags.Text, "NIGEL");
		fixOutMsg.add(FixTags.TimeInForce, '0');
		fixOutMsg.add(FixTags.SecurityDesc, "AOZ3 C02000");
		fixOutMsg.add(FixTags.SecurityType, 'O');
		// Note: contrary to popular belief, System.nanoTime() is consistency across CPU cores
		// and can be used to measure one-way latency without any problems. We have also used
		// native rdtsc timers with very similar latency results.
		fixOutMsg.add(7676, System.nanoTime());
	
		send(client, fixOutMsg);
	}
	
	public static void main(String[] args) {
		NioReactor nio = NioReactor.create();
		MapConfiguration config = new MapConfiguration();
		config.add("fixVersion", 44);
		Server server = new FixLatencyOneWayServer(nio, 55555, config);
		server.open();
		nio.start();
	}
}

The client:

package com.coralblocks.coralfix.bench.latency;

import com.coralblocks.coralbits.bench.Benchmarker;
import com.coralblocks.coralbits.util.SystemUtils;
import com.coralblocks.coralfix.FixConstants;
import com.coralblocks.coralfix.FixMessage;
import com.coralblocks.coralfix.client.FixApplicationClient;
import com.coralblocks.coralreactor.client.Client;
import com.coralblocks.coralreactor.nio.NioReactor;
import com.coralblocks.coralreactor.util.Configuration;
import com.coralblocks.coralreactor.util.MapConfiguration;

public class FixLatencyOneWayClient extends FixApplicationClient {

	// java -Xms6g -Xmx8g -server -verbose:gc -Xbootclasspath/p:../CoralReactor-boot-jdk7/target/coralreactor-boot-jdk7.jar -cp target/coralfix-all.jar:lib/jna-3.5.1.jar -DdetailedBenchmarker=true -DnioReactorProcToBind=3 com.coralblocks.coralfix.bench.latency.FixLatencyOneWayClient
	
	private int count = 0;
	private final int messages;
	private final Benchmarker bench;
	
	public FixLatencyOneWayClient(NioReactor nio, String localhost, int port, Configuration config) {
		super(nio, localhost, port, config);
		int warmup = SystemUtils.getInt("warmup", 200000);
		this.messages = SystemUtils.getInt("messages", 2000000);
		this.bench = Benchmarker.create(warmup);
    }
	
	@Override
	protected void handleFixApplicationMessage(FixMessage fixMsg, boolean possDupe) {
		
		long ts = fixMsg.getLong(7676);
		// Note: contrary to popular belief, System.nanoTime() is consistency across CPU cores
		// and can be used to measure one-way latency without any problems. We have also used
		// native rdtsc timers with very similar latency results.
		bench.measure(System.nanoTime() - ts);
		
		if (++count == messages) {
			// done...
			bench.printResults();
			close();
		} else {
			// send a dummy message back, just to trigger a new one...
			FixMessage fixOutMessage = getOutFixMessage(FixConstants.MsgTypes.ExecutionReport);
			fixOutMessage.add(333, 3);
			send(fixOutMessage);
		}
	}
	
	public static void main(String[] args) {
		NioReactor nio = NioReactor.create();
		MapConfiguration config = new MapConfiguration();
		config.add("fixVersion", "44");
		config.add("senderComp", "testClient");
		Client client = new FixLatencyOneWayClient(nio, "localhost", 55555, config);
		client.open();
		nio.start();
	}
}

Conclusion

CoralFIX + CoralReactor can easily deliver one-way FIX message latencies below 5 microseconds.

Coral Blocks

The Results

The Source Code

Conclusion