One of the most common signs of a non-smooth model is the presence of triggers or switches that toggle the behavior between different modes.  For example, consider a simple model trading a single stock with one parameter p: the model sells one hundred shares when the stock price reaches p but doesn’t do anything else.  If the price of the stock starts at p-a and goes to p-1 then back down to p-a, the model makes 0 dollars profit.  If instead the price does something slightly different and goes from p-a to p then back to p-a, the model makes 100*a dollars.  This model is therefore very sensitive to both small variations in the market and the parameter p, which will make deciding what p should be difficult.

In analysis of models with non-smooth surfaces, very often the “optimal” region of the parameter space will be near the edge of one of these discontinuities because this is where the profit potential is often highest.  The discontinuity makes the model unpredictable, but it can also make the model very profitable when it happens to be correct.  Naive optimizations on these non-smooth surfaces will simply pick which points along the edges of a discontinuity happen to have performed best, turning what should be a well-behaved continuous optimization that considers the shape of the parameter space into an operation that simply totals up a number of discrete events.

If we modify the model above to sell shares continuously as the price moves from p-a to p, we suddenly have a much smoother model.  It will not make as much money as the price moves from p-a to p back to p-a, but it will make money for any p > p-a and price movement from p-a to p-b back to p-a, with b < a.  This will result in a much broader “optimal” region in the parameter space and give it a well defined shape that can be analyzed.

In general, it is best to make the state of a trading system be as much as possible a smooth function of market variables rather than a piecemeal summation of discrete events.  This tends to both make the parameter space easier to analyze and give the model a more stable behavior.

Because of the massive data volumes, a high frequency trading system is very likely to have to deal with extraordinary or unanticipated situations that a system under less stringent performance requirements might be able to ignore or handle outside of the normal mode of operation.  Some typical “unanticipated” situations are:

• Messages arriving out of order from the market and/or brokerage
• Race conditions between in-transit messages
• Missing messages
• Unintended concurrent states among components of the trading system

It is very difficult to anticipate these situations and handle them correctly. But if they are not handled, then the system will (probably) lose a lot of money when they’re encountered.  That a trading system can be tested for its response to unanticipated situations without money on the line is therefore an absolute requirement.

(The system should also, of course, have very good unit test coverage, but unit tests only test anticipated situations.)

How does one test for unanticipated situations?  Automated trading systems are driven by market data and, fortunately, most data feeds provide many years of historical data that can be used in place of a live feed.  So building a trading system to be able to run offline against historical data is a very good way to do it.  Random data can also be generated from a limit order book model (see this paper, for example).

Data feeders for every source of trading system input can be built to simulate a real market day as realistically as possible.  For systems that take in multiple data feeds, this means joining the feeds and delivering the messages in the order they would appear live. Simulated execution messages can also be generated and delivered in a realistic way.  Features like the ability to play back a feed at real time speeds (so events that happened two seconds apart arrive at the trading system two seconds apart) can also be very useful.

Ideally, the offline trading system will be identical to the online system, running through all the same code, just with different sources of input.  To be efficient, though, so that many days of data can be tested very quickly, it is usually best to run offline tests in a single threaded mode.  If information is passed between system components with message queues, this can be done by reimplementing the message queue interfaces so that messages are just passed through and processed instead of enqueued.  These singly-threaded processes can then be run concurrently on a multiprocessor machine to rapidly test many days of data.

It is easiest to make a trading system like this testable if the state and behavior of the trading logic are fully determined by a set of incident messages, that is, the message queue is the only source of information for making trading decisions.  If it is not, it will be more difficult to have confidence that a given testing setup will present realistic scenarios to the trading system.

Trading system testability is a large topic, however, and I’ve only talked about some of the key ideas in this post.  Future posts will go into more detail and tie them to discussions of market modeling and data analysis.

The nature of a HFT startup is such that the trading technology and the business agreements must evolve together–which is a point I touched on in my first post–because one very rigidly defines the constraints on the other. For this reason, I believe that starting a HFT firm, without a large amount of capital to burn, requires beginning with a lower frequency model that can be made economical with whatever trading costs are presently achievable and then scaling up to higher frequencies as they become feasible. Fortunately, there is more than one way to scale up a model’s volume: it can either trade the same instruments more frequently (or with larger trade sizes), or it can trade more instruments at the same frequency. Unfortunately, the differences between trading at low and high frequencies make it unlikely that a single algorithm, however parameterized, will be able to achieve this, which implies that bootstrapping a firm’s trading to high frequencies will require several fundamental shifts in the trading technology–each of which must be successful.

The goal at each technology iteration is to build a track record of success while trading at the highest speeds that are economical because, unsurprisingly, acquiring lower trading costs means convincing those that have them to give that they will benefit from providing them. “Economical” in this case, though, does not necessarily mean profitable; arguing, with the benefit of a sizable trading record, that a strategy would have been profitable if trading costs were lower is a good way to get trading costs lowered. It may in fact be a prudent business strategy to trade “too fast” for the extant commission structure so that this argument can be made.

Smaller brokerages will be especially receptive to this kind of argument. They will often be more willing to take risks to increase the volume they handle because having more volume to offer will in turn allow them to negotiate better terms with clearing firms and exchanges and allow them to attract higher quality clients. It is important, though, to not get too entangled with any particular brokerage because a primary source of negotiating leverage will be the threat to take one’s business elsewhere.

Of course arguments based on a real track record will also be convincing to those that have trading capital to invest, especially if the track record is built with an eye to demonstrating scalability. Scalability across instruments is the most straightforward to demonstrate, arguing, for example, “we traded instruments A and B for a month and then traded C and D another month and were profitable both months, so we’ll be even more profitable if we can trade all four.” Scalability within a strategy is also valuable to demonstrate: it can be done by varying the capital allocation between strategies even when it would be more prudent from a financial perspective to allocate the capital more evenly. Ultimately the only way to be able to trade with a large amount of capital is to trade successfully with a smaller amount.

Building a track record of success while bootstrapping the trading technology is the key to solving the “chicken and egg” problems of starting a HFT firm. This requires that one trade tactically, however, not just with respect to the market but also with respect to ongoing business relationships. Depending on the profitability of the trading strategies, the amount of capital a firm has available to trade may not be enough to make focusing on profit worthwhile; the sole purpose of trading, then, should be to make the strongest case possible that trading costs should be lowered and more capital provided. Making this case repeatedly is how to build a HFT firm.

Using a standard blocking queue implementation may not meet performance requirements because of the synchronization that will happen every time a message is enqueued, though using an implementation that allows blocks of messages to be enqueued or dequeued at one time can help. An alternative is to build lock-free implementation of the required queues using atomic operations. A very good, clear explanation of how to implement a lock-free single reader/single writer queue was written by Herb Sutter here. This implementation can be modified to handle multiple writers by using an atomic swap operation in the enqueue method.

One problem with using lock-free queues, however, is that the reading thread may spin waiting for the next element(s) to be added, wasting cpu time. For queues that are being constantly filled this may still be efficient, but for queues written to in bursts, a good implementation can be to let the reading thread toggle the queue behavior, switching from lock-free to blocking after a set number of read attempts return nothing, and switching back to lock-free mode after a blocking read succeeds. Special care must be taken in designing the mechanism for switching queue modes, however, as a poor implementation can leave the reader blocked while messages sit in the queue.

Another problem with lock-free queues is that their performance tends to degrade relative to blocking queues as the number of writers increases so, as with limit order books, which implementation is best very much depends on the context.

Regardless of the specific queueing implementation, it is important to be able to enqueue and dequeue blocks of multiple messages at once because of a point I mentioned in an earlier post: market events arrive serially but some are actually simultaneous, so it is important for the correctness of a trading model that simultaneous events appear in the model’s message queue all at the same time.

Update 2: tr8dr in the comments makes the important point that accessing the system clock has a cost that should be considered, and also highlights a couple more points I should address.  One is that data feeds from multiple venues will have different latencies, so obtaining a picture of the current total market state across all venues is a not a trivial one, and a topic for another post.  Another is that, generally speaking, trading models are going to be responding to market ticks directly, not the values read off a clock.  When trying to get a “jump” on particular events that happen at a certain time, like market open or perhaps another pattern that has shown up in analysis, then an absolute measure of time is most useful.  tr8dr also has an interesting post on his blog about order book simulation, something I’d planned to talk about myself.

Standard computer clocks are generally terrible at maintaining an accurate time over long periods.  This is because a clock on a computer is just a counter, and the counter needs to be updated at exact time intervals to be accurate; computers are incapable of doing this, however, because they only perform operations at discrete times defined by their chip clock frequency, the period of which will never be an exact multiple of the required time interval.  Here is a link to a more complete description of this and related issues.

One way to deal with this is to ignore the system clock and use timestamps from a data feed as the “clock.”  Of course, the timestamps will be irregular and when the data is sparse achieving a required precision may be impossible.

Recognizing the fact that while computers are bad at maintaining clock accuracy over large time periods because of accumulating error, they are good at measuring time over short intervals, the best way I’ve found to build a trading clock is to measure time as the last timestamp from a data feed plus the time delta since that timestamp was received, measured by the system clock.  This can be implemented by simply recording the data feed timestamp and the system clock at once, with the only tricky part being to make sure that these two values are written in a single atomic operation so that calls to access the trading clock don’t need to be synchronized.  Also note that the timestamp/system time pair does not need to be updated with every market tick; how often it needs to be updated is a function of the system clock drift, which should definitely be measured on any machine that trades.

Keeping a log of the timestamp/system time pairs is a good idea, as well, because the difference between these times should grow at the system clock drift rate.  If it does not, this usually indicates a backlog in the data feed and should be investigated.  When handling multiple data feeds, comparing the timestamps and drift rates between the feeds will give even more information about potential problems.  In fact maintaining a matrix of the different measures of time at different points in the system is a very good way to monitor the health of the system.

Data latency, the time it takes for data to get to the trading system from the market, is one of the primary latencies of a HFT system that needs to be minimized, and the other is trading latency.  For models that are tick driven (models that trade as fast as possible after receiving a particular data event), I measure trading latency as the timestamp of the resulting event on the data feed minus the timestamp of the tick that sparked the trade.  This is a measure of the true system responsiveness.  Note that no call to the system clock is involved in the calculation, and it also does not include the data latency.

When trading directly with the market, the reference number required to identify an order sent by the trading system on the market data feed will of course be known.  If trading through an intermediary brokerage, probably via FIX, the order reference number can usually be provided as a field in the FIX execution reports. Either way it’s very important to have this information.

One last point I want to make regarding latency is that the dynamics of latency for similarly capable market participants sets up a kind of sub-”poker game”, separate from the “poker game” of the larger price movements.  When a trading system has sent off an order in response to the (delayed) state of the market, it knows before anyone else the “card” it is going to play and so has some knowledge of the future state of the market.  A good trading model will work to anticipate other’s responses to this “card” as well as anticipate the “cards” of others.  Assuming competitors have similar latencies, the “cards” of others will show up on the data feed within one round trip time (data latency plus trading latency) and their responses to the original “card” will show up within two round trip times.

If you’re interested in building a trading system I recommend first reading my previous post on general ideas to keep in mind.

My first really technical post will be on how to build a limit order book, probably the single most important component of a trading system.  Because the data structure chosen to represent the limit order book will be the primary source of market information for trading models, it is important to make it both absolutely correct and extremely fast.

To give some idea of the data volumes, the Nasdaq TotalView ITCH feed, which is every event in every instrument traded on the Nasdaq, can have data rates of 20+ gigabytes/day with spikes of 3 megabytes/second or more.  The individual messages average about 20 bytes each so this means handling 100,000-200,000 messages per second during high volume periods.

There are three main operations that a limit order book (LOB) has to implement: add, cancel, and execute.  The goal is to implement these operations in O(1) time while making it possible for the trading model to efficiently ask questions like “what are the best bid and offer?”, “how much volume is there between prices A and B?” or “what is order X’s current position in the book?”

The vast majority of the activity in a book is usually made up of add and cancel operations as market makers jockey for position, with executions a distant third (in fact I would argue that the bulk of the useful information on many stocks, particularly in the morning, is in the pattern of adds and cancels, not executions, but that is a topic for another post).  An add operation places an order at the end of a list of orders to be executed at a particular limit price, a cancel operation removes an order from anywhere in the book, and an execution removes an order from the inside of the book (the inside of the book is defined as the oldest buy order at the highest buying price and the oldest sell order at the lowest selling price).  Each of these operations is keyed off an id number (Order.idNumber in the pseudo-code below), making a hash table a natural structure for tracking them.

Depending on the expected sparsity of the book (sparsity being the average distance in cents between limits that have volume, which is generally positively correlated with the instrument price), there are a number of slightly different implementations I’ve used.  First it will help to define a few objects:

Order
  int idNumber;
  bool buyOrSell;
  int shares;
  int limit;
  int entryTime;
  int eventTime;
  Order *nextOrder;
  Order *prevOrder;
  Limit *parentLimit;

Limit  // representing a single limit price
  int limitPrice;
  int size;
  int totalVolume;
  Limit *parent;
  Limit *leftChild;
  Limit *rightChild;
  Order *headOrder;
  Order *tailOrder;

Book
  Limit *buyTree;
  Limit *sellTree;
  Limit *lowestSell;
  Limit *highestBuy;

The idea is to have a binary tree of Limit objects sorted by limitPrice, each of which is itself a doubly linked list of Order objects.  Each side of the book, the buy Limits and the sell Limits, should be in separate trees so that the inside of the book corresponds to the end and beginning of the buy Limit tree and sell Limit tree, respectively.  Each order is also an entry in a map keyed off idNumber, and each Limit is also an entry in a map keyed off limitPrice.

With this structure you can easily implement these key operations with good performance:

Add – O(log M) for the first order at a limit, O(1) for all others
Cancel – O(1)
Execute – O(1)
GetVolumeAtLimit – O(1)
GetBestBid/Offer – O(1)

where M is the number of price Limits (generally << N the number of orders).  Some strategy for keeping the limit tree balanced should be used because the nature of markets is such that orders will be being removed from one side of the tree as they’re being added to the other.  Keep in mind, though, that it is important to be able to update Book.lowestSell/highestBuy in O(1) time when a limit is deleted (which is why each Limit has a Limit *parent) so that GetBestBid/Offer can remain O(1).

A variation on this structure is to store the Limits in a sparse array instead of a tree.  This will give O(1) always for add operations, but at the cost of making deletion/execution of the last order at the inside limit O(M) as Book.lowestSell/highestBuy have to be updated (for a non-sparse book you will usually get much better than O(M) though).  If you store the Limits in a sparse array and linked together in a list then adds become O(log M) again while deletes/executions stay O(1).  These are all good implementations; which one is best depends mainly on the sparsity of the book.

Generally, it’s also wise to use batch allocations, or if using a garbage collecting language like Java, object pools for these entities.  Java can be made fast enough for HFT as long as the garbage collector isn’t allowed to run.

Strategies for safely and robustly providing access to the book’s data from multiple threads will be the subject of another post.

To conclude, this is how I’ve learned to build a high performance limit order book.  If anyone has questions about how specifically to implement some of the operations I talk about, please post in the comments.

One of the biggest lessons I’ve learned is the important of separating the trading models–the mathematical or algorithmic logic that analyzes data and decides how to trade–from the the trading system–the general framework which provides interfaces for receiving data and interacting with the market to the trading models.  The nature of high frequency trading is such that it is critical to be able to rapidly iterate to adapt models to market conditions.  If a trading system has what is truly model logic strewn throughout, it can become a nightmare to modify model behavior; new, fundamentally different, models may have to be foregone because implementing them will take too much time.

While my focus will be mainly on the technical challenges I faced as the engineering half of a two person HFT startup, the business challenges, such as acquiring trading capital and negotiating ever-lower trading costs, were similarly massive.  With many tech startups, the “business” and “technical” sides can be separated to some degree: a minimum product can be created and then a company built around it as the product is developed further.  With a trading firm, on the other hand, no one will provide the capital required to trade without a track record, and there can be no track record without trading capital; even if a trading strategy could be shown to work with a small amount of money, it is difficult to show that it would scale to the capital levels needed to make it an attractive investment.  So while my work was mainly in developing trading algorithms and the systems to run them, I also had to build models to convince investors that the algorithms would scale.

Another problem, especially acute in high frequency trading, is that the cost per trade directly impacts the profitability of a trading strategy so, for a given per-trade cost, strategies that would otherwise be profitable are not. Clearing firms will only provide a low per-trade cost to customers that execute many trades, but to execute many trades one needs a low per-trade cost.  So in addition to the purely technical challenges, building a HFT firm from scratch means solving two chicken-and-egg problems, that of building a track record and that of negotiating a low per-trade cost, simultaneously.

The mid 2000′s when I finished college turned out to be a kind of “wild west” environment for high  frequency trading.  Exchanges were at that point almost entirely computer driven and this meant trading at sub-second intervals was becoming feasible.  At the same time, the old ways of trading and the inefficiencies they produced were still extant and could be captured by HFT.  When the opportunity to build a startup doing HFT presented itself, I jumped at the chance.

Early on in the company there wasn’t any money to hire more people and later on, when it was apparent that the technology was viable, the risk of the technology getting out was decided to be too great to hire more people.  This left me as the sole developer and “quant” analyst and I had to learn very quickly how to do those jobs.  At most HFT firms, these are considered separate jobs done to be done by different people, which is a mistake, I believe.  Having built the systems for handling the market data and placing the orders has given me a greater understanding of how the market functions at the micro level and so improved the quality of my financial models, which is something I plan to talk about in a future post.

My firm finally went out of business in 2010 for a number of reasons.  A post-mortem will also be the subject of a future post.

Posts on this blog will fall into three main categories: anecdotes of my experiences as a developer of automated HFT systems, building and validating financial (mathematical and algorithmic) models, and building the trading systems to reliably execute those models.  The secretive nature of trading firms in general, and HFT firms in particular, makes good sources on the workings of real HFT systems pretty hard to come by.  Hopefully this blog will become a good source for practical information on the subject.

Please post in the comments questions you have or things you’d like to see me address or expand on.  As the purpose of this blog is to illuminate a cryptic subject, I will be happy to discuss whatever readers will find most interesting.