by Rick Martinelli, Haiku Laboratories, December 1995
Copyright (c) 1995


Prediction science has its foundations in mathematical statistics where, until recently, predictions involved a large number of calculations based on complicated mathematical models and had few practical applications.  In the 1950's, when large amounts of radar and other data were being collected, and just as computers were becoming available, the need for different prediction methods that were more suited to the new technologies became apparent.  New linear prediction algorithms were introduced by scientists and engineers to satisfy this need.  One of these has become known as the  Kalman Filter, named for its author, R.E. Kalman, who introduced it in 1960 (see reference [1] ). 

Market data is usually available as a chart, or time-series, of prices of a particular market item.  A Kalman filter is a device that can separate a time-series into two components, one called signal and the other called noise.  The terms 'signal' and 'noise' are used because usual tracking data from missiles, submarines, etc., consists of a 'true' value for the object's location (the signal) that has been contaminated with extraneous 'noise' from a variety of sources.  It is only when estimates of true values are available that the next location may be predicted with any accuracy.  In applying these methods to the markets, we are making certain tacit assumptions about our time-series.  We are assuming there is a smooth trend line within the data which, in some sense, represents the 'true' value of our market item before it was perturbed by 'market noise'.  In this way we are mimicking the behavior of, say, a submarine sending noisy sonar signals to an enemy whose job is to predict the submarine's next position.  By fitting the last few trend-line values to a suitable model, we can then extend the model to the next time-value, and generate a prediction.

A Kalman filter tracks a time-series using a two-stage process:

1. At every point in the time-series, a prediction is made of the next value based a few of the most recent estimates, and on the data-model contained in the Kalman filter equations.

2. Then, the next actual data point is read, and a compromise value between the predicted and actual values is calculated based on the amount of noise in the time-series.

This compromise value serves as the next 'true'  value for prediction.  For "today's" data point, "tomorrow's" value is unavailable and we take the predicted value from the first stage as our prediction of tomorrow's price.  The algorithm also maintains an estimate of the accuracy of this prediction, which is based on its historical accuracy, and is reported as the standard deviation of the prediction. For applications in market-data prediction,  the Alfa indicator is introduced as a way of generating buy/sell signals.  This indicator incorporates a market item's predicted price change, and the uncertainty of the prediction, into a single number.  A portfolio of market items may be sorted by their Alfa values for further consideration, or the Alfa's may be used in conjunction with other market indicators in a trading scheme.   A second indicator, called the Fortune, is introduced for accessing the algorithm's historical success in predicting the correct direction an item will move in the next tick.  The Fortune indicator can be used on any scheme that makes a prediction at every tick.

Time-Series and Signal Processing
A time-series is defined as any set of numbers arranged in chronological order, with the same time interval between any neighboring pair of numbers.  As a concrete example we will use one year of daily closes (264 data points) of IBM stock ending 10/23/95 (see Figure 1).   For our purposes the date range of the chart is unimportant; we are concerned only with predicting "tomorrow's" price from data through today.  As mentioned in the Introduction, a Kalman filter separates a time-series into two components, one called trend, or signal, and the other called noise. 

Figure 1.  IBM daily closing prices for a 264 day period ending 10/23/95

Let's represent our time-series by the sequence of numbers

Z1, Z2, ..., ZN

where N = 264 in our case, and assume it has been produced by a model of the form

Zn = C Xn + Dn

Xn = A Xn-1 + Bn

where the subscript n is the time variable and takes values from 1 to N (see reference [2] for details).

The letter X is called a state vector and, in our case, holds the last few trend-line (signal) estimates.  The letter A represents a matrix that contains the model by which trend values are updated.  In theory, these updates are then corrupted by Bn, called model noise, to produce the next state vector.  The geometry matrix C converts the state vector to a number and finally Dn, called observation noise, is added to produce the observed Z value.   The state model-update matrix A, together with its associated state vector X, are what makes this Kalman filter unique among all other Kalman filters.  A derivation of matrix A and a discussion of its properties are contained in [3].   Basically, the signal is assumed to be a (different) polynomial on every sub-segment of fixed length,  where the degree of A determines the degree of the polynomial and the length of the sub-segments.  Simulation results on this filter are contained in [4], and results on the use of this filter to track military targets are contained in [5].

For suitable startup state X0, the filter may be thought of as a black box that generates Z from the fixed data model contained in A and C, and the noise terms.  These noise terms each have a certain unknown variance, or intensity, that is ultimately reflected in the volatility of the market item. If they were known, the Kalman filter could be adjusted, or tuned, to the data volatility by initially specifying the ratio of the noise variances.  If VB represents the variance of Bn and VD is the variance of Dn, their ratio is called the Tracking Parameter T and is defined as T = VB/VD. (See also [6].)   As the filter passes over the data generating estimates of X, it also estimates the values of VBand VD.  At each data point then, the filter uses these estimates to adjust T.  This ability to adjust the tracking parameter to match the data during tracking is what makes the filter 'adaptive'.  To find the best T value for each market item, the algorithm is allowed to iterate through 100 values of T from 0.1 to 10.0 and find the best value in the sense that predicted prices overall are closest to the item's actual, historical prices.  In general, smaller T values are found with more volatile market items.

The data in Figure 1 was tracked with a third degree Kalman filter, with the tracking parameter automatically chosen as described above.  Figure 2 shows the last 25 data points in detail, together with the Kalman predictions at each point.  Notice that the filter performs well when the data is relatively smooth , but fails when the data makes an abrupt direction change. This is because the filter cannot “see” into the future and expects the data to maintain its current trend.  Future predictions, like “tomorrow’s” price, are extrapolated, noise-free state values:

XN+1 = A XN

XN+2 = A XN+1 = A2 XN

XN+3 = A XN+2 = A3 XN

and so on, together with their standard deviations.  Three future predictions are also shown in Figure 2.

Figure 2.  Kalman predictions (+) for the last 25 IBM data points, shownwith the data (solid line), and predictions for 3 points into the future.

Buy-Sell Signals and The ALFA Indicator
The Kalman scheme provides predictions at every time point in the data range (except the first few startup points), and standard deviations for every prediction.  As the filter passes over the data, these standard deviations tend to an average value that is related to the variances that defined T in Section 2, above.  Consequently, whenever a predicted price change exceeds this average by a significant amount, it becomes worthy of the trader’s attention.   The predicted price change and its accompanying standard deviation may be incorporated into one statistic, called ALFA, and denoted by a, which we define here as the  predicted price change divided by the standard deviation of the prediction

Defined in this way, a is a normalized indicator and so can be used to compare between different market items, rather than just within one item historically.  A portfolio of market items may be sorted by their current a values (with the minus sign removed for decreasing items) to reveal the largest predicted changes and/or smallest uncertainties.  Items near the top of the list could then be studied further.  Whenever a exceeds one, for example, the predicted price change exceeds one standard deviation, that is, the average standard deviation that have been “experienced’ by the filter on during its scan.

Figure 3. Historical Alfa values for IBM data.

Figure 3. Shows historical a’s for the IBM data.  If the first 50 points are ignored, due to filter startup, we see that a seldom exceeds one, and only once reached a value of two.  In practice, a’s greater than ten have been found.  Such large values usually mean the item has been trending in the same direction for several days 

Figure 3 suggests the following buy/sell flags: buy when a exceeds a predefined value; sell when it is less than the another, pre-defined value.  In the example below, these values are set at 1.8 and -1.8, respectively, and the buys and sells recorded.  A summary of the filter’s performance under these conditions is provided by the Fortune indicator, defined next.

Back-testing and the Fortune Indicator
A trading scheme can be thought of here as input-output device for generating buy/sell signals when presented with a market chart.  Back-testing is a general procedure for testing the potential effectiveness of a trading scheme.  The idea is to apply the scheme to historical data on representative portfolio charts.  That is, simply record the buys and sells indicated by the scheme, without actually investing, and keep track of the profits and losses.  To provide a visual record of the historical performance of the scheme, we define an indicator called the Fortune as follows.

We begin by modeling a trading scheme as a simple gambling system.  Suppose each market item has its own "roulette wheel", where the values on the wheel represent possible closing prices, and the wheel is spun once every day to determine the next price.  At each spin the trader may place a single bet of one betting unit on either black (for price increase), or red (for a decrease), depending on the buy/sell signals coming from the trading scheme..  If a bet is place, and if the filter has guessed the correct direction of price movement, we are returned our investment plus whatever amount the increase or decrease represents in fractions of a betting unit.  If not, we get our investment back minus that same amount.   Notice that the trader shows a profit whenever the filter guesses correctly, regardless of the direction of price movement.  After each bet, we record the accumulated number of betting units as our Fortune to date.

 A graph of the Fortune indicator for the IBM data is shown in Figure 4.  For this scan, the value of a indicating a buy (the buy flag) was set at 1.8, and the sell flag was set at -1.8.  Also, for the purpose of investing, the first 50 days of the scan are ignored, while the filter is stabilizing.

Figure 4. The Fortune indicator for IBM data.  Alfa buy flags = 1.8,sell flag = -1.8.

At day 102 , a exceeded the buy flag at 2.01 (see Figure 3), and one unit was invested.  However, the price closed lower the following day (as can be seen on careful inspection of Figure 1) and the resulting loss is shown as a drop of about 0.6% in the Fortune.  At day 139, a just exceeded the buy flag at 1.86.  This time the investment resulted in a profit of 1.7%.  Finally, for two consecutive days, 222 and 223, a ‘s below the sell flag at -1.82 and -1.86 resulted in profits of 0.9% and 1.6%, respectively.  The corresponding down-trend can be seen in Figure 1, and is shown again in Figure 5 together with the Kalman predictions.

Figure 5.  IBM data (solid line) and Kalman predictions (+) for a 20-day period containing a profitable down-trend.

Summary & Conclosions
Adaptive Kalman filters that have proved so useful in military applications, have been shown here to also be effective in market predictions, when used in conjunction with suitable indicators, and when properly adjusted.  To evaluate the filters in an investment setting, the a indicator defined in Section 3 was designed to be a buy/sell indicator in a trading scheme that employs the Kalman predictions.  The Fortune indicator defined in Section 4 was designed to provide a graphical display of the results.  An arbitrarily selected market item was tested and a-values were found that resulted profitable investments three out of four times.

However, this scheme is of limited value as a practical investment scheme.  In practice, a trader employing a prediction scheme should always intervene when an apparently trending item is found, and search for additional news on the item.  Also, the scheme described here requires a single day turn-around, and no commissions on trades, both non-realistic conditions. 

Lastly, given the nature of the markets, the value of this, or any other prediction scheme, is questionable when applied to only a handful of market items.  However, current technology has made data for many thousands of market items available to the public at reasonable prices.  The results in this memo suggest that, by searching such large portfolios and placing sufficient restrictions on the selection process, significantly trending market items may be found with adaptive Kalman filters.

[1]. Kalman, R.E., "A new approach to linear filtering and prediction problems", ASME Transactions, Series D, 82, 35-45, (1961).
[2].  Luenberger, D.G., Optimization by Vector Space Methods, Wiley, 1969, Chapters 3 and 4.
[3]. Martinelli, R., "Curve Fitting and the Kalman Filter", Kentron International Departmental Correspondence, December 2, 1978.
[4]. Martinelli,R., "Mathematical description of 3D track preprocessor",  Honeywell Marine Systems Center  Technical Memorandum, H476-90-001, 30 March 1979.
[5].  Parker, J.D., "Tracking filters for Precision Electronic Support Systems", Defense Technical Information Center, Cameron Station, Alexandria, Virginia 22304, DTIC# A264 089, 1992.
[6].  Kalata, P.R., "The Tracking Index: A generarized parameter for ab and abg Target Trackers”, IEEE Transactions on aerospace and electronic systems, Vol AES-20, p174, 1984.