PREDICTING MARKET DATA WITH A KALMAN
FILTER
by Rick Martinelli & Neil Rhoads
Haiku Laboratories, 2009
Note: This
article appears in Technical
Analysis of Stocks and Commodities in two parts as Predicting Market Data Using The Kalman Filter, January 2010 and Part 2, February
2010.
Available Profit and the Fortune Chart
Introduction
The chart below shows daily opens
for one year (252 days) of Ford Motor Co. (F).
Figure 1. Daily opens
for Ford Motor Co. (F) ending 07/30/09.
According
to modern financial engineering principals, market data such as this is
supposed to be a Brownian motion, which means that the daily price changes form
a white-noise process. A white-noise is
a random process in which consecutive values are independent of each other
(among other things), meaning a price increase is just as likely as a decrease
each day. However, in reality, it is not
uncommon for a particular market item to have several consecutive down days, or
up days, over a short time span. During
such spans the prices are said to be correlated. The objective is to harness these
correlations with a Kalman filter for prediction.
In a
previous article, a simple linear extrapolation was employed to predict
tomorrow’s price-change; the prediction was then used to calculate the Alpha
statistic which compares the predicted price-change to a recent average of
price-changes. (See[1]) Relatively
large, positive values of Alpha indicate a long position, and relatively large,
negative values indicate a short position.
To test the effectiveness of the procedure as a trading scheme, it was
back-tested on a random selection of stocks and indexes. Indicated positions were taken, then closed
out the next trading day, and all profits and losses were accumulated in a
chart called the Fortune. Surprisingly,
of the 28 items tested, 20 produced greater profits than a simple buy&hold
position for the same time period.
The
current article expands on the previous work, replacing the simple one-day
predictor with a Kalman Filter. The
Kalman Filter, as applied here, is a two-stage algorithm which assumes there is
a smooth trend-line within the data that represents the 'true' value of the market
item before being perturbed by 'market noise'.
In the first stage, a few previous trend-line values are fit to a
suitable model, which is then extrapolated to the next time value to generate a
prediction and its error variance. In
the second stage, the corresponding data value is read and a new trend value is
computed as a compromise between the prediction and actual data value. The compromise is based on the relative
amounts of noise in the data and predictions.
The filter then repeats the cycle of prediction and correction as each
new data value is read.
The predicted price change and its standard deviation from the filter’s first-stage are combined to produce the Alpha statistic which is then used to determine buy-sell signals. A simulated trading scheme executes those signals and, as before, profits and/or losses are accumulated in the Fortune. Items are selected for simulation based on a new data-property called the available profit. The ratio of the Fortune to available profit defines a characteristic of the filter called its efficiency. The next sections discuss the Kalman filter and details of the simulation method. The last sections discuss results of simulations and offer some conclusions.
The
Kalman filter is a recursive algorithm invented in the 1960’s to track a moving
target from noisy measurements of its position, and predict its future position
(See [2] for details). Applying this
technology to financial market data, the noisy measurements become the sequence
of prices
y1, y2,…,yN
where yk
is the price of a market item on day k and N is the total number of days; for
the Ford data in Figure 1, N=252. The
simple equation
yk = xk + ek
then
says that the each data value yk is composed of a smooth trend xk
and a random noise component ek which is assumed to have zero
mean. The objective here is to estimate
the trend and use it to predict future data values. To do this, the Kalman filter employs an
auxiliary equation to predict a future trend value from previous trend
values. We chose
xk|k-1 = 3 (xk-1 - xk-2) + xk-3,
where xk|k-1
is the predicted trend value on day
k, given data through day k-1, and the xk’s are the previous
filtered trend values. This is the model, or process, by which trend values are predicted. The resulting filter is called the quadratic filter, because it assumes
that every four consecutive trend values fit a quadratic curve. Associated with the model is a process-noise sequence qk,
which in this case represents a correction term in the quadratic assumption.
A final trend estimate is made after the current
data value yk is input to the system:
xk|k = xk|k-1
+ Gk(yk – xk|k-1)
is
called the filtered trend value on
day k. The difference between actual and filtered values on day k is called the
residual: rk = yk
– xk|k. The innovation vk = yk
– xk|k-1 (above in parentheses) represents the new information in yk
that is not available in xk|k-1.
The filter gain Gk determines
the contribution made by the innovation in the final estimate and thus controls
the tradeoff between adherence to the model and fidelity to the data. The gain, in turn, is determined by the
relative amounts of noise in the data and in the model, which are maintained in
the filter.
The
amount of noise in the data is denoted by R and is the variance of the residual
sequence rk defined above.
The amount of noise in the model, or process, is denoted by Q and is the
variance of the process noise qk.
While R may be calculated from the residuals, Q is completely
unknown. To estimate Q, we define the tracking parameter T by the equation
T = -log(Q/R).
T
will be positive whenever Q < R and negative when Q > R. A negative T indicates the model has more
noise than the original data and may need to be changed. In the simulations below, values of T ranging
between -5 and 5 are tested and the value that minimizes the variance of the
innovation sequence vk is taken as the optimal T value.
The
data in Figure 1 was tracked with the quadratic filter and an optimal T
= 1.86 was determined. This corresponds
to measurement noise that is about 72 times greater than model noise. Figure 2 shows how the innovation variance
behaves as T varies and shows the minimum value at 1.86. The data was tracked again using optimal T and
the resulting trend (filtered version of Figure 1) is shown in Figure 3. The gain for the track is shown in Figure 4;
the fact that it approaches a constant value (about 0.37) is typical of a
well-modeled system.
Figure 2.
Innovation variance versus Tracking Parameter for the Ford data in
Figure 1. Optimal T is 1.86.
Figure 3.
Data from Figure 1 (green dots) and the Kalman estimated trend line
(red). Tracking Parameter T=1.86
Figure 4. Kalman gain corresponding to the
track in Figure 3.
How
well the filter and its intrinsic model perform depends on the filter’s
intended application. The current
application is one-day prediction of financial market price-changes. To evaluate the filter in this context, the
following section describes an idealized trading scheme where the predictions xk|k-1
and their standard deviation are used to calculate buy/sell signals.
The
Kalman filter provides predictions for each day in the data range (except the
first few startup points). Figure 5
shows predictions for a short portion of the data in Figure 1 (green
triangles).
Figure 5.
Kalman predictions for a portion of the data from 11/18/08 to 12/09/08
(green) together with the data.
The
filter also provides standard deviations of these predictions. Letting Δyk = xk|k-1
– yk-1 denote the predicted price change on day k, and σk
the standard deviation of that prediction, the Alpha indicator for day k is defined as
ak= Δyk / σk .
Figure
6 shows the Alpha sequence for the data in Figure 1, where the horizontal lines
indicate the two Alpha cutoff values.
The irregular cyclic component visible in Alpha is typical of Kalman
tracks of market data, likely due to inherent correlations in the data.
Figure 6.
Alpha values for the Ford data in Figure 1. The horizontal lines indicate the Alpha
cutoff values 0.38 and -0.38.
If
Alpha is positive then the predicted movement is upward, and if Alpha is also
greater than one, the predicted change exceeds the average. Conversely, if Alpha is negative, the
predicted movement is downward, and if Alpha is less than negative one, the
predicted change once again exceeds the average. In the previous article, ak
> 1 indicated a long position and ak < -1 a short position. The present article generalizes the situation
somewhat. Here, a critical value C, called the Alpha cutoff, is determined for each track, where ak
> C indicates a long position, ak
< -C indicates a short position,
and no position is taken otherwise.
Positions are cancelled the following day and the associated profit/loss
is accumulated in the Fortune. The cutoff C is determined by testing values between 0 and 3 in small
increments, and choosing the value that either yields the largest last-day-fortune, or yields the Fortune
chart closer to the available profit line than any other value. These
techniques are described next.
AVAILABLE PROFIT AND THE FORTUNE CHART
The
profit/loss on day k may be written
Pk = A Wk (yk/yk-1
– 1),
where
the quantity in parentheses is the relative price-change, or return, on day k, A is the trade amount
in dollars, and Wk = 1 if ak > C,
Wk = -1 if ak
< -C and Wk = 0
otherwise (W for wager). Note that Wk = 0
corresponds to no trade on day k and so Pk = 0 as well. Note further that A is the same for every
trade. For the Ford data, C was found to be 0.38, and Figure 7
shows its Pk values, where the trade amount A was set to one dollar
(zero values are not shown). The red
point at 11/28/08 represents the trade having the largest profit. The reason for the large profit can be seen
by recalling Figure 5 above, where the red point in Figure 7 corresponds to
point 8 in Figure 5. The prediction of 1.84 was nowhere near the actual at
2.47, but it was in the correct direction,
up from 1.72, resulting in a 0.436 profit.
Figure 7.
Pk values for the Ford data in Figure 1. T=1.86, C=0.38, A=1.
The Fortune sequence is the accumulation of the Pk’s:
Its
last value FN is called the last-day
fortune (LDF) and represents the amount of profit/loss realized at the end
of the simulation. A graph of Fn
versus day n, corresponding to the Pk data in Fig 7, is shown in
Figure 8. The LDF is 1.37 on 124 trades,
or about 1.1% average profit per trade.
This is, of course, an idealized fortune in which there are no trade
commissions, and trades can be transacted at the prescribed buy/sell prices (no
slippage). It is used here primarily to
evaluate the Kalman filter’s ability to predict the direction a stock price
will take.
Figure 8.
Fortune chart for a static Kalman track of the Ford data in Figure 1.
Quadratic Model.
T=1.86, C=0.38, A=1,
LDF=1.37.
The
Fortune plot is one indicator of that ability, and so is the profit ratio, defined as the ratio of
number of profitable trades to total trades.
In the Ford simulation it was 0.59.
In Figure 7, the profit ratio can be roughly visualized as the ratio of
number of points above the zero line to total points. A third way to evaluate the filter is by its efficiency. Suppose the scheme was able to capture every price change in the data, that is,
correctly predict direction each trading day. Assuming A=1, the sum
of
the absolute values of the returns represents the maximum amount of profit that
can be realized by day n (using this scheme), and is called the available profit (AP) on day n. The last value, SN, is the total
AP in the data, and the ratio FN/SN is taken as the
filter’s efficiency. For the Ford
example, SN is 13.15 making the efficiency 0.104, or about 10%. Figure 9 plots Fn, the evolution
of the Fortune (green line) compared with Sn, the evolution of the
AP, providing a view of the filter’s efficiency over time.
Figure 9. A comparison of the Fortune chart (green)
with the AP line for the Ford data.
The
AP line has a desirable property, called zero
downside volatility which we would like to see in the Fortune. Hence we should choose the Fortune line that
is ‘nearest’ the AP line, where the distance
D between the two lines is calculated as
.
The
value of C that minimizes the distance
D can be taken as optimal C, which is
the method used with the Ford data. The
other method, maximizing the LDF, usually yields the same C value, but not always.
Given a choice, we take the largest C
value because, although the LDF may be smaller, the Fortune line is smoother.
The
filter was tested on one year of daily opens for a large group of selected
stocks (all data obtained from Yahoo
Finance). The simulations involved two
optimizations. The first optimization
determines the best Kalman tracking parameter and the second finds the best
Alpha cutoff. The simulation proceeds in
five steps as follows:
- Track one year of data multiple times to find
the optimal tracking parameter T
- Track the data once more using optimal T
- Use the resulting Kalman predictions and their
standard deviations to find the Alpha sequence ak
- Use that Alpha sequence to find the optimal
cutoff C
- Use the optimal cutoff C to calculate the Fortune chart
While
the Ford data was chosen more-or-less at random, and for being well-known, the
items used in the simulations were carefully selected to have the largest
AP’s. Opening prices for approximately
five thousand stocks were scanned for their one-year available profit and many
of the stocks having AP’s greater than 7 were tracked with the quadratic
filter. One reason for choosing opens is
that they usually have the largest AP of the four daily prices. A representative sample of the results is
shown in the table below sorted by AP. Results for the Ford data are included and
have been highlighted. The last column
is the average profit/loss in dollars, or dollar
return, based on a trade amount A=$1000. To get a better feel for the nature of these
results, key features are plotted separately, starting with the sorted AP’s in
Figure 10.
Figure 10. Sorted AP’s for the 36 items in the
Table.
Figure 11. LDF’s corresponding to the AP’s in
Figure 10.
Figure 12. Filter efficiencies corresponding to the
AP’s in Figure 10.
Figures
10 and 11 show that filter efficiency is essentially uncorrelated with AP
(actual correlation is 0.07). The
obvious similarity between Figures 11 and 12 is artificial, because LDF values
are the product of AP and efficiency.
The average LDF is 2.2±1.9, a fairly wide range, and the average
efficiency is 12.0%±9.5%, a slightly narrower range than the LDF’s, but both
are still clearly dependent on the data.
Available profit is a property of the data only, and is independent of
the filter. Only the filter’s profit
ratio (Figure 13) seems to be (nearly) independent of the data being tracked,
its average being 0.51±0.07.
Figure 13.
Profit Ratios for the 36 test items and their average line (red).
The
highest efficiency is for number 18 in the table, BASI, at an incredible 52%,
with an accompanying LDF of 8.9. Data
and Fortune for BASI are shown in Figure 14.
BASI has the highest optimal T
at 4.8, implying the quadratic model is well-suited to this data. It also has the smallest optimal C at 0.0, meaning every possible trade
is executed. The least efficient is PWR
at 1.1% with an LDF of 0.105, shown in Figure 15. Optimal T
and C values, AP and profit ratio
values are all near average, suggesting there is something inherent in this
data responsible for the filter’s relatively poor performance in this
case.
The
stock having the fewest trades and best dollar return is HEB at about $88 per
trade on 18 trades, shown in Figure 16. Its Fortune
chart differs from the others in that there are relatively long time frames
where the chart is flat, i.e., no trades occur.
This is due to the large optimal C value at 2.38, the largest in the
table. Results like HEB are attractive
for their minimal downward movement. TCIC has the smallest tracking parameter
at T=-0.66, meaning model noise is about
double data noise suggesting another model should possibly be considered.
Figure 14.
Data (blue) and Fortune for BASI ending 07/10/09. Optimal T=4.80,
optimal C=0.0, AP=17.0, LDF=8.9,
profit ratio 0.47, 247 trades and about $36 per trade.
Figure 15.
Data and Fortune for PWR ending 06/10/09. Optimal T=1.80,
optimal C=0.12, AP=9.7, LDF=0.105,
profit ratio 0.51, 206 trades and $0.51 per trade.
Figure 15.
Data and Fortune for HEB ending 07/29/09. Optimal T=3.94,
optimal C=2.38, AP=18.9, LDF=1.59,
profit ratio 0.61, 18 trades and $88.28 per trade.
The goal here was to determine if a Kalman filter
could be exploited to predict the direction of stock price movements. Based on the LDF’s of the selected stocks in
the table, the implementation described here is remarkably effective in some
cases (e.g. BASI). The effectiveness of
the filter is traced to two primary factors: the nature of the data, namely,
its available profit and amount of correlation, and the efficiency of the
filter, which is determined by the filter’s model in combination with a
particular data set. Only the profit
ratio appears to be (nearly) independent of the data. It is apparent that selecting only the top
performers from the table would lead to a highly profitable portfolio. Unfortunately, the simulation presented here
could never be implemented in the real world because it requires all of the data on day one to determine
the optimal C and T values responsible for the filter’s
performance. However, application of
this scheme to a shorter time period, say the previous quarter or two, should
yield the best values for tomorrow’s
prediction! A simulation of this scheme
must await another article.
[1] Martinelli, Rick, "Harnessing
the (mis)Behavior of Markets", Technical Analysis of Stocks & Commodities,
June 2006
[2] Martinelli, Rick and
Rhoads, Neil, "Linear
Estimation and the Kalman Filter", Haiku Laboratories, 2008.
|
|
Symbol |
End
Date |
Last
Price |
AP |
T |
Optimal
C |
LDF |
Efficiency |
Profit
Ratio |
#
Trades |
Dollar
Return |
|
1 |
MECA |
10/30/08 |
1.62 |
39.15 |
4.720 |
0.08 |
1.901 |
4.9% |
0.489 |
221 |
8.60 |
|
2 |
BUTL |
07/13/09 |
0.02 |
32.92 |
3.380 |
0.06 |
5.926 |
18.0% |
0.354 |
206 |
28.77 |
|
3 |
CTIC |
08/18/09 |
1.54 |
31.30 |
-0.660 |
0.02 |
1.610 |
5.1% |
0.390 |
187 |
8.61 |
|
4 |
ANPI |
07/10/09 |
1.41 |
27.59 |
4.000 |
0.12 |
6.441 |
23.3% |
0.521 |
192 |
33.55 |
|
5 |
BJCT |
07/13/09 |
0.20 |
27.02 |
3.280 |
0.04 |
5.048 |
18.7% |
0.358 |
232 |
21.76 |
|
6 |
BPUR |
07/10/09 |
0.16 |
25.12 |
3.380 |
0.26 |
2.452 |
9.8% |
0.493 |
150 |
16.35 |
|
7 |
TKO |
07/29/09 |
0.14 |
23.87 |
3.540 |
0.36 |
1.653 |
6.9% |
0.443 |
115 |
14.38 |
|
8 |
BBGI |
07/10/09 |
2.23 |
22.03 |
2.020 |
0.34 |
3.176 |
14.4% |
0.509 |
116 |
27.38 |
|
9 |
CENX |
08/18/09 |
11.00 |
20.40 |
1.460 |
0.64 |
1.003 |
4.9% |
0.632 |
38 |
26.39 |
|
10 |
RAE |
07/29/09 |
1.75 |
20.34 |
3.140 |
0.36 |
1.836 |
9.0% |
0.515 |
130 |
14.12 |
|
11 |
MDH |
07/29/09 |
1.26 |
19.54 |
4.120 |
0.18 |
5.536 |
28.3% |
0.555 |
182 |
30.42 |
|
12 |
ACAS |
07/29/09 |
3.35 |
19.47 |
0.640 |
0.06 |
1.787 |
9.2% |
0.500 |
192 |
9.31 |
|
13 |
HEB |
07/29/09 |
2.30 |
18.86 |
3.940 |
2.38 |
1.589 |
8.4% |
0.611 |
18 |
88.28 |
|
14 |
CLZR |
08/18/09 |
1.38 |
18.64 |
1.940 |
0.24 |
1.798 |
9.6% |
0.467 |
120 |
14.98 |
|
15 |
NG |
07/29/09 |
3.94 |
18.22 |
1.480 |
0.06 |
4.907 |
26.9% |
0.563 |
222 |
22.10 |
|
16 |
GAN |
07/29/09 |
14.24 |
18.16 |
3.300 |
0.40 |
1.456 |
8.0% |
0.451 |
122 |
11.93 |
|
17 |
PSTI |
08/20/09 |
1.63 |
17.04 |
2.000 |
0.22 |
1.343 |
7.9% |
0.425 |
160 |
8.39 |
|
18 |
BASI |
07/10/09 |
0.85 |
16.97 |
4.800 |
0.00 |
8.898 |
52.4% |
0.470 |
247 |
36.02 |
|
19 |
PAL |
07/29/09 |
2.54 |
16.93 |
1.500 |
0.08 |
1.327 |
7.8% |
0.507 |
217 |
6.11 |
|
20 |
BCRX |
07/13/09 |
3.86 |
16.68 |
1.600 |
0.40 |
0.463 |
2.8% |
0.538 |
106 |
4.37 |
|
21 |
TIV |
07/29/09 |
0.99 |
16.53 |
1.840 |
0.00 |
1.306 |
7.9% |
0.498 |
247 |
5.29 |
|
22 |
BANR |
07/10/09 |
2.93 |
16.31 |
1.960 |
0.32 |
1.522 |
9.3% |
0.579 |
95 |
16.02 |
|
23 |
ABG |
07/29/09 |
11.99 |
16.27 |
1.720 |
1.04 |
1.116 |
6.9% |
0.611 |
36 |
30.99 |
|
24 |
FRG |
07/29/09 |
3.68 |
15.70 |
1.540 |
0.16 |
1.316 |
8.4% |
0.585 |
200 |
6.58 |
|
25 |
CNXT |
08/18/09 |
2.35 |
14.61 |
0.860 |
0.16 |
2.201 |
15.1% |
0.525 |
160 |
13.76 |
|
26 |
BPFH |
07/13/09 |
4.48 |
13.92 |
1.280 |
0.74 |
0.122 |
0.9% |
0.514 |
37 |
3.29 |
|
27 |
FAC |
06/10/09 |
2.11 |
13.72 |
2.540 |
0.04 |
1.368 |
10.0% |
0.465 |
228 |
6.00 |
|
28 |
BPOP |
07/13/09 |
1.40 |
13.16 |
1.560 |
0.10 |
1.829 |
13.9% |
0.543 |
173 |
10.57 |
|
29 |
F |
07/29/09 |
7.08 |
13.15 |
1.860 |
0.38 |
1.367 |
10.4% |
0.589 |
124 |
11.02 |
|
30 |
BELFA |
07/10/09 |
13.39 |
13.12 |
2.920 |
0.28 |
3.131 |
23.9% |
0.546 |
152 |
20.60 |
|
31 |
CACH |
08/18/09 |
4.43 |
12.82 |
1.820 |
0.18 |
1.160 |
9.1% |
0.522 |
113 |
10.26 |
|
32 |
PWR |
06/10/09 |
25.66 |
9.76 |
1.800 |
0.12 |
0.105 |
1.1% |
0.505 |
206 |
0.51 |
|
33 |
CMCO |
08/18/09 |
14.03 |
9.60 |
1.580 |
0.12 |
1.019 |
10.6% |
0.539 |
206 |
4.95 |
|
34 |
CAKE |
08/18/09 |
17.91 |
9.18 |
2.600 |
0.94 |
0.489 |
5.3% |
0.545 |
33 |
14.82 |
|
35 |
GE |
06/11/09 |
13.43 |
8.38 |
2.740 |
0.66 |
0.717 |
8.6% |
0.614 |
57 |
12.58 |
|
36 |
CALM |
08/18/09 |
28.72 |
7.50 |
1.180 |
0.28 |
0.941 |
12.5% |
0.532 |
126 |
7.47 |
Results of simulations of 36 large AP stocks.