MARKET DATA PREDICTION
WITH AN ADAPTIVE KALMAN FILTER
by Rick Martinelli, Haiku
Laboratories, December 1995
Copyright (c) 1995
INTRODUCTION
TIME-SERIES AND SIGNAL PROCESSING
BUY-SELL SIGNALS AND THE ALFA INDICATOR
BACK-TESTING AND THE FORTUNE INDICATOR
SUMMARY & CONCLUSIONS
REFERENCES
Introduction
Prediction science has its
foundations in mathematical statistics where, until recently, predictions
involved a large number of calculations based on complicated mathematical
models and had few practical applications. In the 1950's, when large
amounts of radar and other data were being collected, and just as computers
were becoming available, the need for different prediction methods that were
more suited to the new technologies became apparent. New linear
prediction algorithms were introduced by scientists and engineers to satisfy
this need. One of these has become known as the
Kalman Filter, named for its author, R.E. Kalman, who introduced it in
1960 (see reference [1] ).
Market data is usually available as a chart, or time-series, of prices
of a particular market item. A Kalman filter is a device that can
separate a time-series into two components, one called signal and the other
called noise. The terms 'signal' and 'noise' are used because usual
tracking data from missiles, submarines, etc., consists of a 'true' value for
the object's location (the signal) that has been contaminated with extraneous
'noise' from a variety of sources. It is only when estimates of true
values are available that the next location may be predicted with any
accuracy. In applying these methods to the markets, we are making certain
tacit assumptions about our time-series. We are assuming there is a
smooth trend line within the data which, in some sense, represents the 'true'
value of our market item before it was perturbed by 'market noise'. In
this way we are mimicking the behavior of, say, a submarine sending noisy sonar
signals to an enemy whose job is to predict the submarine's next
position. By fitting the last few trend-line values to a suitable model,
we can then extend the model to the next time-value, and generate a prediction.
A Kalman filter tracks a time-series using a two-stage process:
1. At every point in the time-series, a prediction is made of the next value
based a few of the most recent estimates, and on the data-model contained in
the Kalman filter equations.
2. Then, the next actual data point is read, and a compromise value between the
predicted and actual values is calculated based on the amount of noise in the
time-series.
This compromise value serves as the next 'true' value
for prediction. For "today's" data point,
"tomorrow's" value is unavailable and we take the predicted value
from the first stage as our prediction of tomorrow's price. The algorithm
also maintains an estimate of the accuracy of this prediction, which is based
on its historical accuracy, and is reported as the standard deviation of the
prediction. For applications in market-data prediction,
the Alfa indicator is introduced as a way of generating buy/sell
signals. This indicator incorporates a market item's predicted price
change, and the uncertainty of the prediction, into a single number. A
portfolio of market items may be sorted by their Alfa values for further
consideration, or the Alfa's may be used in conjunction with other market indicators
in a trading scheme. A second indicator, called the Fortune, is
introduced for accessing the algorithm's historical success in predicting the
correct direction an item will move in the next tick. The Fortune
indicator can be used on any scheme that makes a prediction at every tick.
Time-Series
and Signal Processing
A time-series is defined as any set
of numbers arranged in chronological order, with the same time interval between
any neighboring pair of numbers. As a concrete example we will use one
year of daily closes (264 data points) of IBM stock ending 10/23/95 (see Figure
1). For our purposes the date range of the chart is unimportant; we
are concerned only with predicting "tomorrow's" price from data
through today. As mentioned in the Introduction, a Kalman filter
separates a time-series into two components, one called trend, or signal, and
the other called noise.
Figure 1. IBM daily closing prices for a 264 day period ending 10/23/95
Let's
represent our time-series by the sequence of numbers
Z1, Z2, ...,
ZN
where N = 264 in our case, and assume it has been produced by a
model of the form
Zn = C Xn + Dn
Xn = A Xn-1 + Bn
where the subscript n is the time
variable and takes values from 1 to N (see reference [2] for details).
The letter X is called a
state vector and, in our case, holds the last few trend-line (signal)
estimates. The letter A represents a matrix that contains the model by
which trend values are updated. In theory, these updates are then
corrupted by Bn, called model noise, to produce the next state
vector. The geometry matrix C converts the state vector to a number and
finally Dn, called observation noise, is added to produce the
observed Z value. The state model-update matrix
A, together with its associated state vector X, are what makes this
Kalman filter unique among all other Kalman filters. A derivation of
matrix A and a discussion of its properties are contained in [3].
Basically, the signal is assumed to be a (different) polynomial on every
sub-segment of fixed length, where the degree of
A determines the degree of the polynomial and the length of the
sub-segments. Simulation results on this filter are contained in [4], and
results on the use of this filter to track military targets are contained in
[5].
For suitable startup
state X0, the filter may be thought of as a black box that generates
Z from the fixed data model contained in A and C, and the noise terms.
These noise terms each have a certain unknown variance, or intensity, that is
ultimately reflected in the volatility of the market item. If they were known,
the Kalman filter could be adjusted, or tuned, to the data volatility by initially
specifying the ratio of the noise variances. If VB represents
the variance of Bn and VD is the variance of Dn,
their ratio is called the Tracking Parameter T and is defined as T = VB/VD.
(See also [6].) As the filter passes over the data generating
estimates of X, it also estimates the values of VBand VD.
At each data point then, the filter uses these estimates to adjust T.
This ability to adjust the tracking parameter to match the data during tracking
is what makes the filter 'adaptive'. To find the best T value for each
market item, the algorithm is allowed to iterate through 100 values of T from
0.1 to 10.0 and find the best value in the sense that predicted prices overall
are closest to the item's actual, historical prices. In general, smaller
T values are found with more volatile market items.
The data in Figure 1 was
tracked with a third degree Kalman filter, with the tracking parameter
automatically chosen as described above. Figure 2 shows the last 25 data
points in detail, together with the Kalman predictions at each point.
Notice that the filter performs well when the data is relatively smooth , but fails when the data makes an abrupt direction
change. This is because the filter cannot “see” into the future and expects the
data to maintain its current trend. Future predictions, like “tomorrow’s”
price, are extrapolated, noise-free state values:
XN+1 = A XN
XN+2 = A XN+1 = A2 XN
XN+3 = A XN+2 = A3 XN
and so on, together with their
standard deviations. Three future predictions are also shown in Figure 2.
Figure 2. Kalman predictions (+) for the last 25 IBM data points, shownwith the data (solid line), and predictions for 3 points into the future.
Buy-Sell
Signals and The ALFA Indicator
The Kalman scheme provides predictions at every time point in the data range
(except the first few startup points), and standard deviations for every
prediction. As the filter passes over the data, these standard deviations
tend to an average value that is related to the variances that defined T in
Section 2, above. Consequently, whenever a predicted price change exceeds
this average by a significant amount, it becomes worthy of the trader’s
attention. The predicted price change and its accompanying standard
deviation may be incorporated into one statistic, called ALFA, and denoted by
a, which we define here as the predicted price change divided by the
standard deviation of the prediction
Defined in this way, a is a normalized indicator and so can be used to compare
between different market items, rather than just within one item
historically. A portfolio of market items may be sorted by their current
a values (with the minus sign removed for decreasing items) to reveal the
largest predicted changes and/or smallest uncertainties. Items near the
top of the list could then be studied further. Whenever a exceeds one,
for example, the predicted price change exceeds one standard deviation, that
is, the average standard deviation that have been “experienced’ by the filter
on during its scan.
Figure 3. Historical Alfa values for IBM
data.
Figure
3. Shows historical a’s for the IBM data. If the first 50
points are ignored, due to filter startup, we see that a seldom exceeds one,
and only once reached a value of two. In practice, a’s greater than ten
have been found. Such large values usually mean the item has been trending
in the same direction for several days
Figure 3 suggests the
following buy/sell flags: buy when a exceeds a predefined value; sell when it
is less than the another, pre-defined value. In the
example below, these values are set at 1.8 and -1.8, respectively, and the buys
and sells recorded. A summary of the filter’s performance under these
conditions is provided by the Fortune indicator, defined next.
Back-testing
and the Fortune Indicator
A trading scheme can be thought of here as input-output device for generating
buy/sell signals when presented with a market chart. Back-testing is a
general procedure for testing the potential effectiveness of a trading
scheme. The idea is to apply the scheme to historical data on
representative portfolio charts. That is, simply record the buys and
sells indicated by the scheme, without actually investing, and keep track of
the profits and losses. To provide a visual record of the historical
performance of the scheme, we define an indicator called the Fortune as
follows.
We begin by modeling a
trading scheme as a simple gambling system. Suppose each market item has
its own "roulette wheel", where the values on the wheel represent
possible closing prices, and the wheel is spun once
every day to determine the next price. At each spin the trader may place
a single bet of one betting unit on either black (for price increase), or red
(for a decrease), depending on the buy/sell signals coming from the trading
scheme.. If a bet is place, and if the filter
has guessed the correct direction of price movement, we are returned our
investment plus whatever amount the increase or decrease represents in
fractions of a betting unit. If not, we get our investment back minus
that same amount. Notice that the trader shows a profit whenever
the filter guesses correctly, regardless of the direction of price
movement. After each bet, we record the accumulated
number of betting units as our Fortune to date.
A graph of the
Fortune indicator for the IBM data is shown in Figure 4. For this scan,
the value of a indicating a buy (the buy flag) was set at 1.8,
and the sell flag was set at -1.8. Also, for the purpose of investing,
the first 50 days of the scan are ignored, while the filter is stabilizing.
Figure 4. The Fortune
indicator for IBM data. Alfa buy flags = 1.8,sell
flag = -1.8.
At day 102
, a exceeded the buy flag at 2.01 (see Figure 3), and one unit was
invested. However, the price closed lower the following day (as can be
seen on careful inspection of Figure 1) and the resulting loss is shown as a
drop of about 0.6% in the Fortune. At day 139, a just exceeded the buy
flag at 1.86. This time the investment resulted in a profit of
1.7%. Finally, for two consecutive days, 222 and 223, a ‘s below the sell
flag at -1.82 and -1.86 resulted in profits of 0.9% and 1.6%,
respectively. The corresponding down-trend can be seen in Figure 1, and
is shown again in Figure 5 together with the Kalman predictions.
Figure 5. IBM data (solid line) and Kalman predictions (+) for a 20-day period containing a profitable down-trend.
Summary
& Conclosions
Adaptive Kalman filters that have proved so useful in military applications,
have been shown here to also be effective in market predictions, when used in
conjunction with suitable indicators, and when properly adjusted. To
evaluate the filters in an investment setting, the a
indicator defined in Section 3 was designed to be a buy/sell indicator in a
trading scheme that employs the Kalman predictions. The Fortune indicator
defined in Section 4 was designed to provide a graphical display of the
results. An arbitrarily selected market item was tested and a-values were
found that resulted profitable investments three out of four times.
However, this scheme is
of limited value as a practical investment scheme. In practice, a trader
employing a prediction scheme should always intervene when an apparently
trending item is found, and search for additional news on the item. Also,
the scheme described here requires a single day turn-around, and no commissions
on trades, both non-realistic conditions.
Lastly, given the nature
of the markets, the value of this, or any other prediction scheme, is
questionable when applied to only a handful of market items. However,
current technology has made data for many thousands of market items available
to the public at reasonable prices. The results in this memo suggest
that, by searching such large portfolios and placing sufficient restrictions on
the selection process, significantly trending market items may be found with
adaptive Kalman filters.
References
[1]. Kalman, R.E., "A new approach to linear filtering
and prediction problems", ASME Transactions, Series D, 82, 35-45, (1961).
[2]. Luenberger, D.G., Optimization
by Vector Space Methods, Wiley, 1969, Chapters 3 and 4.
[3]. Martinelli, R., "Curve Fitting and the Kalman Filter", Kentron
International Departmental Correspondence, December 2, 1978.
[4]. Martinelli,R., "Mathematical description of
3D track preprocessor", Honeywell Marine Systems Center
Technical Memorandum, H476-90-001, 30 March 1979.
[5]. Parker, J.D., "Tracking filters for
Precision Electronic Support Systems",
[6]. Kalata, P.R., "The Tracking Index: A
generarized parameter for ab and abg Target Trackers”, IEEE Transactions on
aerospace and electronic systems, Vol AES-20, p174, 1984.