Polaroid Photo

Pictures from 猴若斯筆記本

Choose a Topic:

週三
27
二月 '13

AR Model with Java Implementation III

Yule-Walker estimation is simple, but it has some problems:

To obtain the Yule-Walker estimates for an AR model of order m, it is necessary to solve a system of linear equations with m unknowns. In addition, to select the order of the AR model by the minimum AIC method, we need to evaluate the AIC values of the models with orders up to M, the maximum order. Namely, we have to estimate the coefficients by solving systems of linear equations with one unknown, . . . , M unknowns.

So we have Durbin-Levinson Algorithm to deal with this problem. Hereinafter, the AR coefficients and the innovation variance of the AR model of order m are denoted as and , respectively.The algorithm is defined as follows:

  1. Set and
  2. For m = 1, …, M, repeat the following steps:
    1. \hat{a}_m^m = (\hat{C}_m - \sum_{j=1}^{m-1}{\hat{a}_j^{m-1}\hat{C}_{m-j}} )(\hat{\sigma}_{m-1}^2)^{-1},
    2. \hat{a}_i^m = \hat{a}_i^{m-1} - \hat{a}_m^{m}\hat{a}_{m-i}^{m-1} for i=1, …, m-1
    3. \hat{\sigma}_m^2 = \hat{\sigma}_{m-1}^2 \left \{ 1 - (\hat{a}_m^m)^2 \right \},
    4. AIC_m = N(log2\pi\hat{\sigma}_m^2 + 1) + 2(m+1)

Here we see a for the first, which means the parameter in the order m. With this algorithm, we can easily estimate all parameters from order 1 to M in loop. The sample code is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
long N = this.getSampleCount();
this.v[0] = getGamma(0);
this.aic[0] = N * (Math.log(2 * Math.PI * this.v[0]) + 1) + 2;
for (int m=1; m<=p; m++) {
// estimate phi[m][m]
double sum = 0;
for (int j=1; j<=m-1;j++) {
sum += this.phi[m-1][j] * getGamma(m-j);
}
this.phi[m][m] = (getGamma(m) - sum)/this.v[m-1];
// estimate phi[m][i], i=1,...,m-1
for (int i=1; i<=m-1; i++) {
this.phi[m][i] = this.phi[m-1][i] - this.phi[m][m] * this.phi[m-1][m-i];
}
// estimate v[m]
this.v[m] = this.v[m-1] * (1 - this.phi[m][m] * this.phi[m][m]);
this.aic[m] = N * (Math.log(2 * Math.PI * this.v[m]) + 1) + 2 * (m + 1);
this.pacf[m] = this.phi[m][m];
}

Because Durbin-Levinson Algorithm is tend to solve the same formula as Yule-Walker method, so it’s forecasting method is just the same as previous post.

The last implementation is Burg’s algorithm based on the maximum entropy method (MEM) to estimate. The process is:

  1. Set , for n = 1, …, N. In addition, for the AR model of order 0, compute \hat{\sigma}_0^2 = N^{-1}\Sigma_{n=1}^Ny_n^2, and
  2. For m = 1, …, M, repeat the following steps:
    1. Estimate the by any of the formulae:
      • \hat{a}_m^m = \sum_{n=m+1}^N{v_n^{m-1}w_{n-m}^{m-1}} \left \{ \sum_{n=m+1}^N{(w_{n-m}^{m-1})^2} \right \}^{-1}
      • \hat{a}_m^m = \sum_{n=m+1}^N{v_n^{m-1}w_{n-m}^{m-1}} \left \{ \sum_{n=m+1}^N{(w_{n-m}^{m-1})^2}\sum_{n=m+1}^{N}{(v_n^{m-1})^2} \right \}^{-\frac{1}{2}}
      • \hat{a}_m^m = 2 \sum_{n=m+1}^{N}{v_n^{m-1}w_{n-m}^{m-1}}\left \{ \sum_{n=m+1}^{N}{(w_{n-m}^{m-1})^2} + \sum_{n=m+1}^{N}{(v_{n}^{m-1})^2} \right \}^{-1}
    2. Obtain the AR coefficients , …,  by a_j^m = a_j^{m-1} - a_m^ma_{m-j}^{m-1}
    3. For n = m+1,…,N, obtain the forward prediction error as v_n^m = v_n^{m-1} - \hat{a}_m^m w_{n-m}^{m-1}
    4. For n = m + 1, …, N, obtain the backward prediction error as w_{n-m}^m = w_{n-m}^{m-1} - \hat{a}_m^m w_n^{m-1}
    5. Estimate the innovation variance of the AR model of order m by \sigma_m^2 = \sigma_{m-1}^2\left \{ 1 - (\hat{a}_m^m)^2 \right \}
    6. Obtain AIC by AIC_m = N(log2\pi\hat{\sigma}_m^2 + 1) + 2(m + 1)

The following is sample code. Note here we use the third formula in step 2a.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
double[] y = getSamples();
int N = y.length;
double mean = this.getMean();
// let the mean to 0,
double sumOfSquares = 0;
double[] yy = new double[y.length];
for (int i=0;i<y.length;i++) {
yy[i] = y[i] - mean;
sumOfSquares += yy[i] * yy[i];
}
double[][] vv = new double[this.p+1][y.length+1];
double[][] w = new double[this.p+1][y.length+1];
// initialize
for (int i = 1; i<=yy.length; i++) {
vv[0][i] = yy[i-1];
w[0][i] = yy[i-1];
}
// estimate v[0] and aic[0]
this.v[0] = sumOfSquares / N;
//System.out.println(sumOfSquares);
//System.out.println(this.getAcf().getStatistics().getSumsq());
//System.out.println(this.v[0]);
this.aic[0] = N * (Math.log(2 * Math.PI * this.v[0] * this.v[0]) + 1) + 2;
for (int m=1;m<=this.p; m++) {
// estimate phi[m][m]
double sum1 = 0;
double sum2 = 0;
double sum3 = 0;
for (int n=m+1;n<=N;n++) {
sum1 += vv[m-1][n] * w[m-1][n-m];
}
for (int n=m+1;n<=N;n++) {
sum2 += w[m-1][n-m] * w[m-1][n-m];
}
for (int n=m+1;n<=N;n++) {
sum3 += vv[m-1][n] * vv[m-1][n];
}
this.phi[m][m] = 2 * (sum1 / (sum2 + sum3));
// estimate phi[m] .. phi[m][m-1]
for (int j=1;j<=m-1;j++) {
this.phi[m][j] = this.phi[m-1][j] - this.phi[m][m] * this.phi[m-1][m-j];
}
// For n = m+1, ···,N, obtain the forward prediction
for (int n=m+1; n<=N;n++) {
vv[m][n] = vv[m-1][n] - this.phi[m][m] * w[m-1][n-m];
}
// For n = m+1, ···,N, obtain the backward prediction error
for (int n=m+1; n<=N;n++) {
w[m][n-m] = w[m-1][n-m] - this.phi[m][m] * vv[m-1][n];
}
// Estimate the innovation variance of the AR model of order m
this.v[m] = this.v[m-1] * (1 - this.phi[m][m] * this.phi[m][m]);
// Estimate aic
this.aic[m] = N * (Math.log(2 * Math.PI * this.v[m]) + 1) + 2 * (m + 1);
}

Here is another sample implementation based on http://emptyloops.com/technotes/A%20tutorial%20on%20Burg’s%20method,%20algorithm%20and%20recursion.pdf.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
double[] x = getSamples();
int N = x.length -1;
int m = this.p;
// initialize Ak
double[] Ak = new double[this.p+1];
Ak[0] = 1;
// initialize f and b
double[] f = ArrayUtils.clone(x);
double[] b = ArrayUtils.clone(x);
// initialize Dk
double Dk = 0;
for (int j=0;j<=N;j++) {
Dk += 2 * f[j] * f[j];
}
Dk -= f[0] * f[0] + b[N] * b[N];
// Burg Recursion
for (int k=0; k< m; k++) {
// compute mu
double mu = 0;
for (int n=0; n<=N-k-1;n++) {
mu += f[n+k+1]*b[n];
}
mu *= -2/Dk;
// update Ak
for (int n=0;n<=(k+1)/2;n++) {
double t1 = Ak[n] + mu * Ak[k+1-n];
double t2 = Ak[k+1-n] + mu*Ak[n];
Ak[n] = t1;
Ak[k+1-n] = t2;
}
// update f and b
for (int n=0;n<=N-k-1;n++) {
double t1 = f[n+k+1] + mu*b[n];
double t2 = b[n] + mu*f[n+k+1];
f[n+k+1] = t1;
b[n] = t2;
}
// update Dk
Dk = (1-mu*mu)*Dk - f[k+1]*f[k+1] - b[N-k-1]*b[N-k-1];
}
//assign coefficients
for (int i=1;i<=this.p;i++) {
this.phi[this.p][i] = Ak[i] * -1; // remember to times -1 because this algorithm model is reversed
}

Note the sign of coefficients because the model is different.

Finally, we need to know the performance of them. In my simple simulation, the Burg’s algorithm has the best performance and Durbin-Levinson algorithm has the worst, but only a 1-2 ms difference in around 5,000 – 7,000 points. Yule-Walker method can only estimate an order at time, so if you need to test different orders to select the best one, it will cost another loop to estimate and might cost more time. As prediction, their 1 and 2 step  forecasting is not significantly different.

Reference:

Genshiro Kitagawa, “Introduction to Time Series Modeling", http://www.amazon.com/Introduction-Modeling-Monographs-Statistics-Probability/dp/1584889217/ref=sr_1_1?ie=UTF8&qid=1361603954&sr=8-1&keywords=introduction+to+time+series+modeling





Related Free Lance Projects on GAF:


週三
27
二月 '13

AR Model with Java Implementation II

The second estimation is based on Yule-Walker Method. This method is by solving Yule-Walker equation:

C_0 = \sum_{i=1}^m{\beta_i C_i} + \sigma^2

C_j = \sum_{i=1}^m{\beta_i C_{j-i}}

We can transfer the equation to a linear equations as following:

\begin{bmatrix}\^{C}_0 & \^{C}_1 & \cdots & \^{C}_{m-1} \\ \^{C}_1 & \^{C}_0 & \cdots & \^{C}_{m-2} \\ \vdots & \vdots & \ddots & \vdots \\ \^{C}_{m-1} & \^{C}_{m-2} & \cdots & \^{C}_0 \end{bmatrix} \begin{bmatrix} \beta_1 \\ \beta_2 \\ \vdots \\ \beta_m\end{bmatrix} = \begin{bmatrix}\^{C}_1 \\ \^{C}_2 \\ \vdots \\ \^{C}_m\end{bmatrix}

By solving above equation, we can get our estimations of to .

Here we see a new symbol , which is autocovariance function of the time series. So we need to implement a method to estimate autocovariance function. The equation is:

\hat{C}_k = \frac{1}{N}\sum_{n=k+1}^N{(y_n - \hat{\mu}})(y_{n-k} - \hat{\mu})

The sample code is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
/**
* autocovariance function of lag k
* @param k
* @return
*/

public double getGamma(int k) {
double m = stats.getMean(); // mean of time series
double sum = 0;
for (int n=k;n<stats.getN();n++) {
double yn = stats.getElement(n);
double ynk = stats.getElement(n-k);
sum += (yn-m)*(ynk-m);
}
this.gamma[k] = sum/stats.getN(); // store this value in an array
return this.gamma[k];
}

Now we can start our estimation method using Jama library:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
private void YuleWalkerMethod() {
double[][] Cj = new double[this.p][this.p];
double[][] Cp = new double[this.p][1];
for (int i=0; i<this.p; i++) {
for (int j=0;j<this.p; j++) { if (i == j) Cj[i][j] = getAcf().getGamma(0); else if (i > j) {
Cj[i][j] = Cj[j][i];
} else {
Cj[i][j] = getAcf().getGamma(j-i);
}
}
Cp[i][0] = getAcf().getGamma(i+1);
}
// solve matrix MCj * a = Mcp
Matrix MCj = new Matrix(Cj);
Matrix MCp = new Matrix(Cp);
Matrix b = MCj.solve(MCp);
for (int i=0 ;i< this.p; i++) {
this.phi[this.p][i+1] = b.get(i, 0);
}
}

Because it’s a zero mean model, so we don’t have to estimate , but we need to modify our forecast method. In a zero-mean model, the AR(p) can write as:

\underbrace{ \begin{bmatrix} y_t \\ y_{t-1} \\ \vdots \\ y_{t-p+1} \end{bmatrix} }_{Y_t} = \underbrace{ \begin{bmatrix} \beta_1 & \beta_2 & \cdots & \beta_{p-1} & \beta_p \\ 1 & 0 & \cdots & 0 & 0\\ 0 & 1 & \cdots & 0 & 0\\ \vdots & \vdots & \ddots & \vdots & \vdots\\ 0 & 0 & \cdots & 1 & 0 \end{bmatrix} }_{\Phi } \underbrace{ \begin{bmatrix} y_{t-1}\\ y_{t-2}\\ \vdots\\ y_{t-p} \end{bmatrix} }_{Y_{t-1}} + \underbrace{ \begin{bmatrix} \varepsilon_t \\ 0\\ \vdots\\ 0 \end{bmatrix} }_{\varepsilon_t}

So, the t+k forecasting is:

E_t(Y_{t+k}) = \Phi^k Y_t

The sample code is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
double[][] b = new double[p][p];
double[][] yt = new double[p][1];
for (int i=0; i<p;i++) {
for (int j=0; j<p;j++) {
if (i == 0) b[i][j] = this.phi[p][j+1];
else if (i == j+1) b[i][j] = 1;
}
}
for (int i=0; i<p; i++) {
yt[i][0] = samples[N-i-1] - mean;
}
Matrix B = new Matrix(b);
Matrix Y = new Matrix(yt);
for (int i=0; i<k; i++) {
B = B.times(B);
}
Matrix Ytk = B.times(Y);
double e = mean + Ytk.get(0, 0);

Reference:

Genshiro Kitagawa, “Introduction to Time Series Modeling", http://www.amazon.com/Introduction-Modeling-Monographs-Statistics-Probability/dp/1584889217/ref=sr_1_1?ie=UTF8&qid=1361603954&sr=8-1&keywords=introduction+to+time+series+modeling





Related Free Lance Projects on GAF:


週六
23
二月 '13

AR Model with Java Implementation I

After a few weeks searching on Google, I cannot find any information about free Java implementation of AR model. So I decide to implement it by myself.

Because there are several methods to solve this problem, I will implement some of them and compare the performance and outcomes.

First, consider the following AR(p) model:

The first way to estimate parameters is Least Squares method by solving the following formula:

Here, Y is a N * 1 matrix, X is a N * p matrix, and is a p * 1 matrix.

Our first implementation is using Jama library (http://math.nist.gov/javanumerics/jama/) to solve this formula. Here is the sample code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
double[] samples = getOriginalSamples(); // implement your own method to get sample data
int n = samples.length - this.p;
double[][] Y = new double[n][1];
for (int i=0;i<n; i++) {
  y[i] = new double[]{samples[i+p]};
}
double[][] x = new double[n][this.p];
for (int i=0;i<n;i++) {
  for (int j=0;j<p;j++) {
    x[i][j] = samples[i+j];
  }
}
Matrix Y = new Matrix(y);
Matrix X = new Matrix(x);
Matrix b = X.solve(Y);
for (int i=1;i<=this.p;i++) {
  System.out.println(b.get(i-1, 0));
}

Notice:

  • Here I let sample size (n) reduce to (sample length) - p.
  • This method didn't consider the constant , so we need to estimate  by , which is the mean.

The code is quite simple and some optimization is done by library.

Although Jama is a good matrix computation implementation,  for some issues, maybe you want to use other libraries maintained by reliable open source community.

Here we can use Apache's commons math library to solve the same formula:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
double[] samples = getOriginalSamples(); // implement your own method to get sample data
int n = samples.length - this.p;
double[][] Y = new double[n][1];
for (int i=0;i<n; i++) {
  y[i] = samples[i+p];
}
double[][] x = new double[n][this.p];
for (int i=0;i<n;i++) {
  for (int j=0;j<p;j++) {
    x[i][j] = samples[i+j];
  }
}
OLSMultipleLinearRegression regression = new OLSMultipleLinearRegression();
regression.newSampleData(y, x);
double[] params = regression.estimateRegressionParameters();
for (int i=0;i<params.length;i++) {
  System.out.println(params[i]);
}

Notice:

  • We use OLS to estimate the parameters. You can change to GLS if you think it's better.
  • We can get params 0 ... p, which is to

Basically, the 2 methods solve the same formula, but their implementation are different. In my simulation, they will have almost the same estimation in more than 200 points.

Another question is which one has better performance?

In my simulation with p =3, they both perform less than 1 ms in < 10,000 points. When >10,000 points, commons math library is more stable than Jama, but the difference is not very significant(like 1 ms v.s. 4ms). I think they are both good choice if points <= 20,000. If points > 20,000, I will suggest commons math library.

At last, we implement a k-step forecasting which is what we build this model for. The formula is: . Obviously, it is a recursive function. The sample code is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public double predict(int k) {
  double[] samples = getSamples();
  int N = samples.length;
  double e = 0;
  if (k&lt;=0) {
    return samples[N+k-1];
  } else {
    for (int i=1;i&lt;=this.p;i++) {
      e += this.phi[this.p][i] * predict(k-i);
    }
  }
  e += this.phi[this.p][0];
  return e;
}

So far, we have a simple AR(p) model with LS implementation and a k-step forecasting method. You can use it to analyze your time series data. We will implement other estimation method in next article. Have fun!!

Reference:

Genshiro Kitagawa, "Introduction to Time Series Modeling", http://www.amazon.com/Introduction-Modeling-Monographs-Statistics-Probability/dp/1584889217/ref=sr_1_1?ie=UTF8&qid=1361603954&sr=8-1&keywords=introduction+to+time+series+modeling





Related Free Lance Projects on GAF:


週一
10
九月 '12

【但唐謨:危險東南亞】讀後

我不太看恐怖片,但是這篇文章十分有趣,梳理了東南亞恐怖片以及西方/東方世界對於東南亞的想像與偏見:

http://okapi.books.com.tw/index.php/p3/p3_detail/sn/1297

只是對於【飛頭魔女】源於東南亞的penanggalan,我是有點小小的保留。記得看過【搜神記】裡面也有所謂飛頭蠻的故事,基本架構非常地像,以年代來看,個人還是比較傾向於從中國鬼故事取材的機會大了點,畢竟港台過去有不少改編聊齋的故事。





Related Free Lance Projects on GAF:


週二
3
五月 '11

A strange SQL question with min(xxx) and max(xxx)

Consider the following SQL:

1
 SELECT MIN(xxx), MAX(xxx) FROM TABLE

It's absolutely a legal SQL statement, and you wouldn't get any error when you execute it.

But when you execute it on a big table, you will find that the performance is quite slow.

The reason is, when you select a min and max value at the same time, database will confuse about how to walk through the index tree. Should it go from right or left? In this confusing case, database will give up walking the index tree and start to scan the whole table, and you will have to wait for table scanning — that means terrible performance in big table.

The solution is quite simple. You can union the result:

1
 SELECT MIN(xxx) FROM TABLE UNION SELECT MAX(xxx) FROM TABLE

or you can join the result:

1
 SELECT MIN(a.xxx), MAX(b.xxx) FROM TABLE a, TABLE b





Related Free Lance Projects on GAF:


週三
6
四月 '11

實作Infinispan的ClassLoader

最近在研究Infinispan,由於有個需求是必須要做Preload,資料從資料庫系統取出。雖然Infinispan有提供JDBC的CacheLoader,但是他的概念其實是把物件儲存到指定的地方。因此要嘛就是整個物件序列化到DB(binary),不然就是一個字串值存資料庫抓取。不過其實我們需要的是把資料庫的部份欄位資料填到物件後放到cache,因此這兩種預設的Loader都不合用。

Infinispan其實也可以自己寫Loader,不過文件並沒有提到, Google爬了一下資料也不多,只好自己去看幾個範例還有Infinispan提供的幾個Loader,終於讓我理出一個頭緒,在這邊記錄一下心得。

Infinispan提供了一個AbstractClassLoader的物件,我們可以直接先繼承這個物件,然後實作幾個方法即可:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;

import org.infinispan.Cache;
import org.infinispan.container.entries.InternalCacheEntry;
import org.infinispan.container.entries.InternalEntryFactory;
import org.infinispan.loaders.AbstractCacheLoader;
import org.infinispan.loaders.AbstractCacheLoaderConfig;
import org.infinispan.loaders.CacheLoaderConfig;
import org.infinispan.loaders.CacheLoaderException;
import org.infinispan.marshall.StreamingMarshaller;
import org.infinispan.util.logging.Log;
import org.infinispan.util.logging.LogFactory;

public class SimpleCacheLoader extends AbstractCacheLoader {
    private static final Log log = LogFactory.getLog(SimpleCacheLoader.class);
    SimpleCacheLoaderConfig config;
    Map _store = new TreeMap();

    @Override
    public Class getConfigurationClass() {
        return SimpleCacheLoaderConfig.class;
    }

    @Override
    public void init(CacheLoaderConfig config, Cache cache,
            StreamingMarshaller m) throws CacheLoaderException {
        super.init(config, cache, m);
        this.config = (SimpleCacheLoaderConfig) config;
        System.out.println("init ......");
    }

    @Override
    public InternalCacheEntry load(Object key) throws CacheLoaderException {
        System.out.println("load object....");
        if (key == null)
            return null;
        InternalCacheEntry se = _store.get(key);
        if (se == null)
            return null;
        if (se.isExpired()) {
            log
                    .debug("Key {0} exists, but has expired.  Entry is {1}",
                            key, se);
            _store.remove(key);
            return null;
        }

        return se;
    }

    @Override
    public Set<internalcacheentry> load(int numEntries)
            throws CacheLoaderException {
        System.out.println("load n....");
        if (numEntries < 0)
            return loadAll();
        Set<internalcacheentry> s = new HashSet<internalcacheentry>(numEntries);
        for (Iterator<internalcacheentry> i = _store.values().iterator(); i
                .hasNext()
                && s.size() < numEntries;) {
            InternalCacheEntry se = i.next();
            if (se.isExpired()) {
                log.debug("Key {0} exists, but has expired.  Entry is {1}", se
                        .getKey(), se);
                i.remove();
            } else if (s.size() < numEntries) {
                s.add(se);
            }
        }
        return s;
    }

    @Override
    public Set<internalcacheentry> loadAll() throws CacheLoaderException {
        Set<internalcacheentry> s = new HashSet<internalcacheentry>();
        for (Iterator<internalcacheentry> i = _store.values().iterator(); i
                .hasNext();) {
            InternalCacheEntry se = i.next();
            if (se.isExpired()) {
                log.debug("Key {0} exists, but has expired.  Entry is {1}", se
                        .getKey(), se);
                i.remove();
            } else
                s.add(se);
        }
        return s;
    }

    @Override
    public Set<object> loadAllKeys(Set<object> keysToExclude)
            throws CacheLoaderException {
        Set<object> set = new HashSet<object>();
        for (Object key : _store.keySet()) {
            if (keysToExclude == null || !keysToExclude.contains(key))
                set.add(key);
        }
        return set;
    }

    @Override
    public void start() throws CacheLoaderException {
        System.out.println("start......");
        // do some configuration
        for (int i = 0; i <= 10; i++) {
            Integer _obj = new Integer(i);
            InternalCacheEntry _entry = InternalEntryFactory.create(i + "",
                    _obj);
            _store.put(i + "", _entry);
        }
    }

    @Override
    public void stop() throws CacheLoaderException {
        _store.clear();
    }

    public static class SimpleCacheLoaderConfig extends
            AbstractCacheLoaderConfig {

    }

}

 





Related Free Lance Projects on GAF:


週一
21
三月 '11

[剝削好萊塢]讀後感

這本書是美國獨立電影大老Roger Corman談論自己的經歷,除了第一章談他自己的生活背景外,後面主要著墨的是他從1943開始的電影生涯。

初知道Roger Corman是很久以前在一個網站上讀一篇文章時看到的,裡面提到Corman訓練出無數的美國電影大師,也提到Corman被人稱為B級片之王。當時很喜歡讀這些非主流的電影資訊,所以對於所謂的王當然會想特別關注一下了。無奈後來工作之故,越來越沒時間到處去逛網站看這些非工作的東西,台灣也沒有相關的資料,就一直只是停留在有印象的階段。 近年台灣已經很容易買到簡體書(不得不承認大陸電影書的翻譯確實比台灣熱絡許多),突然看到這一本,就很有興致地買回來啃。

Corman的電影生涯特殊之處在於,他很少與主流公司合作拍片,也幾乎不拍大成本的片(幾百萬美元對他來說已經是極限,基本上他的片子成本都幾乎都是一百萬美元以內的,早期甚至都控制在十萬美元左右)。他是基於他對電影工業的一套獨特商業邏輯來看待這個規則。他認為成本大,就很難回收,基於商業邏輯,他不認為做賠本生意是應該的。Corman認為這是常理,他一直不懂為何大片廠總是看不清這樣的事。而依他的邏輯操作,幾乎每部片子最後都是賺錢的。 另一方面,他認為關鍵在於拍片效率的問題。他認為大片廠總是花很多時間在閒置的事情上,因為一旦片子開拍,即使今天都沒有拍任何鏡頭,這一天還是得花錢出去(這也跟美國獨特的工會制度有關),這是Corman拍片的大忌,所以Corman拍片非常緊湊,絕不會浪費一分鐘在沒有產出的事情上面,所以他都能在既有的資源下把片子拍出來。此外,彈性也是另一個問題。大片廠遇到計畫外的狀況,最常做的通常就是等,等拍片條件到了才拍,所以錢就這樣不斷地消耗。Corman的作法是條件變了就改劇本,改完馬上拍。這是獨立電影的好處,充滿了彈性。但是大片廠想改劇本,基於流程的關係得花一段時間,原則上時間跟金錢也就消耗掉了。 就電影內容而言,Corman不諱言他為了成本的關係,經常重複利用既有的東西。像是一個佈景道具拍兩部片,或是插入之前拍的鏡頭之類的。或是買別人拍的東西,重新剪接之後,補幾個鏡頭重新變成新片。這些大片廠可能完全不會想用的方法,Corman只要能用就用,沒有一定規則可言。對Corman來說,只要是能交代劇情,產生效果,方法不拘。 Corman也提到,這些剝削片儘管目的在賺錢,但是他多少也有想要表達的東西。有些可能是純粹一些技巧上的嘗試,有些是想要發掘新的類型,也有些想要討論一些嚴肅的議題。既然是剝削片,當然他會拍大量的跟風,有些是他自創類型的跟風,有些是其他既有類型的跟風。不過他也會試著在跟風裡面加入一些新的元素,讓觀眾有新鮮感。不過只要他察覺該類型已經玩得差不多,他會毫不遲疑地改換另一種類型。 說到底,從表面上看,Corman的很多作法跟拍小成本的導演並沒有太大的差異,但是Corman會因此受人尊崇,我想是基於他個人的獨特性。這樣的獨特性我認為有幾點:

  1. Corman對於電影藝術還是有他獨到的眼光。Corman不介意重複拍攝一個題材,因為這樣可以省錢,但是他會讓相同題材的電影有其獨特性存在。此外,Corman並不會因為拍的時間短,或是成本太低就不注意細節。他很注重情節的合理性,也會注意所有拍電影該注意的事。往後他介入藝術電影的市場並取得不錯的成績,更是可以說明他對於電影藝術其實是有過人的掌握。
  2. Corman其實也是勇於嘗試的人。他並不是一味地跟拍,他也會看準一個新的類型後去發掘新的題材。之所以有這樣的條件,一方面是對於電影市場的敏銳度,另一方面也是因為他拍片的成本夠低,可以放手一試。
  3. Corman許多電影都有他個人的性格在裡面,而正是這樣獨特Corman風格的電影,和其他的剝削片產生了差異。正如最後Corman總結自己的幾部電影,他也認為這些電影都表現了一些他自己的性格,也因此傳遞了他自己的一些想法。
  4. Corman最為人稱道的,是他擅於發掘新人。由於Corman的拍片計畫成本都不大,所以他很願意給新人嘗試的機會。當然要過他那關也不是說很容易,基本上Corman一定會先看過劇本和拍片計畫等,也會提出他的意見。而且最重要的是,一定要在他給定的限制條件下完成。

Corman在最後一章也反省過他的事業,他認為或許是因為介入製片的事業導致他後來無法分身去拍大成本的電影。但也有可能是他有意為之也說不定,因為他始終刻意與大公司保持一定的距離。Corman喜歡對自己的片子有絕對的主導權,不論是不是他當導演,而這點在大片廠是很難的(他很不喜歡大公司未經他許可就擅自剪接他的影片)。但是他也認為人生有一點小小的不完美也不是什麼太嚴重的事。Corman的許多影片,常常得到兩極化的評價,或許是簡陋的道具很難滿足看慣好萊塢特效的觀眾。我覺得Corman電影的藝術成就必然有被人超越的一天,但是Corman的地位,並不僅止於藝術方面,而是在於其獨到的創作觀點以及識人之明,這部分我想是無法撼動的。看完這本書,不僅對於Corman獨特的創作歷程可以有個輪廓,也可以看看Corman如何以一個獨立製片的身分,在藝術與商業下求得生存之道,而這應該就是許多獨立製片者的難處吧。





Related Free Lance Projects on GAF:


週三
23
六月 '10

ScribeFire測試

這一篇文章是用Chrome的Extension:ScribeFire發送出來的。

基本上還蠻好用的,不需要登入系統就可以直接發文章。
不過他的編輯器有時lag會有點嚴重,不曉得為什麼。
不過整體來說仍然不錯。




Related Free Lance Projects on GAF:


週五
7
五月 '10

整合CXF與Spring Security實現WS-Security(二)

在(一)中我們用的是最基本的寫法。這邊有另外一種寫法,就是當passwordType設定成PasswordDigest的時候,寫法稍有不同,主要的差別在實作的物件。

先看設定檔:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
    <import resource="classpath:META-INF/cxf/cxf.xml" />
    <import resource="classpath:META-INF/cxf/cxf-extension-soap.xml" />
    <import resource="classpath:META-INF/cxf/cxf-servlet.xml" />
    <bean id="hello" class="com.rex.message.ws.impl.HelloWorldImpl"/>
    <jaxws:endpoint id="helloWorld" implementor="#hello" address="/HelloWorld" >        
        <jaxws:inInterceptors>
            <bean class="org.apache.cxf.binding.soap.saaj.SAAJInInterceptor"/>
            <bean class="org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor">
                <constructor-arg>
                    <map>
                        <entry key="action" value="UsernameToken"/>
                        <entry key="passwordType" value="PasswordDigest"/>
                        <entry key="passwordCallbackRef">
                            <ref bean="myPasswordCallback"/>
                        </entry>
                    </map>
                </constructor-arg>
            </bean>
        </jaxws:inInterceptors>
    </jaxws:endpoint>
    <bean id="myPasswordCallback" class="com.rex.message.ws.ServerPasswordCallback"/>

基本上只是把passwordType的值換掉而已。

接下來看實作的程式碼:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public class ServerPasswordCallback implements CallbackHandler {

    @Autowired
    private UserManager userManager;

    public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
        WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
        String userName = pc.getIdentifier();
        // System.out.println(pc.getPassword());
        User user = userManager.findUniqueBy("userName", userName);
        if (user != null) {
            pc.setPassword(user.getPassword());
        }
    }
}

這邊可以看到,只有做一個動作,就是把正確的apssword設定到WSPasswordCallback的物件中。Callback物件會自動去比對傳入的值與client端送過來的值。個人以為,就程式風格上來說,這寫法比較簡潔,不過不是很直覺,至少對Java風格來說,setter做這麼多事確實是有一點讓人不習慣。至於處理上流程上是否有差別,目前因為沒有研究過程式碼,感覺不出來有什麼大的差別,不過我猜測頂多就是會多一個type的檢查吧,至於比對應該差不了多少。





Related Free Lance Projects on GAF:


週四
6
五月 '10

整合CXF與Spring Security實現WS-Security(一)

這邊的例子是實現UserName/Password的認證方法,讓Client必須提供帳密才能呼叫Web Service。

參考的網站主要是CXF的官網(http://cxf.apache.org/docs/ws-security.html),其實寫的算蠻清楚的,這邊做個整理。

首先我們在要做控管的Web Service裡面安插Interceptor:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
    <import resource="classpath:META-INF/cxf/cxf.xml" />
    <import resource="classpath:META-INF/cxf/cxf-extension-soap.xml" />
    <import resource="classpath:META-INF/cxf/cxf-servlet.xml" />
    <bean id="hello" class="com.rex.message.ws.impl.HelloWorldImpl"/>
    <jaxws:endpoint id="helloWorld" implementor="#hello" address="/HelloWorld" >        
        <jaxws:inInterceptors>
            <bean class="org.apache.cxf.binding.soap.saaj.SAAJInInterceptor"/>
            <bean class="org.apache.cxf.ws.security.wss4j.WSS4JInInterceptor">
                <constructor-arg>
                    <map>
                        <entry key="action" value="UsernameToken"/>
                        <entry key="passwordType" value="PasswordText"/>
                        <entry key="passwordCallbackRef">
                            <ref bean="myPasswordCallback"/>
                        </entry>
                    </map>
                </constructor-arg>
            </bean>
        </jaxws:inInterceptors>
    </jaxws:endpoint>
    <bean id="myPasswordCallback" class="com.rex.message.ws.ServerPasswordCallback"/>

這邊要注意的地方有幾個:

  1. 這裡需要Apache提供的wss4j以及CXF裡面的cxf-rt-ws-security這兩個函式庫,Maven設定是:
    
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
            <dependency>
                <groupId>org.apache.ws.security</groupId>
                <artifactId>wss4j</artifactId>
                <version>1.5.8</version>
                <type>jar</type>
            </dependency>
            <dependency>
                <groupId>org.apache.cxf</groupId>
                <artifactId>cxf-rt-ws-security</artifactId>
                <version>2.2.7</version>
                <type>jar</type>
            </dependency>
  2. action的設定是UsernameToken。這個字串是WS-Security預設好的,如果打錯的話,server端解析的時候會match不到對應的認證方法,就會直接回覆解析錯誤。Server端也可以自行定義action,不過這邊不討論。
  3. passwordType設定為PasswordText,表示取得的password是文字格式。這邊因為SOAP封包可能會被攔截,因此採用這種模式的話,最好是要求client端把密碼編碼過後再丟出來。
  4. passwordCallbackRef是等一下要實作的class,因為我們會用到autowired,需要Spring自動注入物件,所以用參考的方式設定。如果沒有這個需求,就用passwordCallback,然後value給定實作物件的完整名稱(com.rex.message.ws.ServerPasswordCallback)就好。
  5. 最後一行就是實作物件的設定,就跟一般的Bean設定方式相同。

接下來就是實作的部份。這邊只要實作CallbackHandler這個介面就可以,程式碼如下:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public class ServerPasswordCallback implements CallbackHandler {

    @Autowired
    private UserManager userManager;

    public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {
        WSPasswordCallback pc = (WSPasswordCallback) callbacks[0];
        String userName = pc.getIdentifier();
        User user = userManager.findUniqueBy("userName", userName);
        // 比對
        if (!pc.getPassword().equals(user.getPassword())){
            throw new IOException("wrong password");
        }
    }

}

這個實作很簡單,幾個小地方說明一下:

  1. 從callbacks裡面取得WSPasswordCallback物件,裡面就可以讀到client傳送過來的username和password。
  2. 比對成功的話不用特別作什麼事,失敗的話就直接輸出exception就可以了。

基本上這樣就完成了。用兩個範例做測試:

  1. 成功的範例: client傳送的SOAP訊息:
    
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    <soapenv:envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.message.rex.com/"><soapenv:envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.message.rex.com/">
       <soapenv:header>
          <wsse:security soapenv:mustunderstand="1" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
             <wsse:usernametoken wsu:id="UsernameToken-26" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
                <wsse:username>jhcheng</wsse:username>
                <wsse:password type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">e10adc3949ba59abbe56e057f20f883e</wsse:password>
                <wsse:nonce encodingtype="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">uUPNozTlx/2p6/aLb3Fjtw==</wsse:nonce>
                <wsu:created>2010-05-06T08:54:49.288Z</wsu:created>
             </wsse:usernametoken>
          </wsse:security>
       </soapenv:header>
       <soapenv:body>
          <ws:sayhi>
             <text>123</text>
          </ws:sayhi>
       </soapenv:body>
    </soapenv:envelope>

    Server回傳:

    
    
    1
    2
    3
    4
    5
    6
    7
    <soap:envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
       <soap:body>
          <ns1:sayhiresponse xmlns:ns1="http://ws.message.rex.com/">
             <return>Hello 123</return>
          </ns1:sayhiresponse>
       </soap:body>
    </soap:envelope>
  2. 失敗的範例: client傳送的SOAP訊息:
    
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    <soapenv:envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.message.rex.com/"><soapenv:envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ws="http://ws.message.rex.com/">
       <soapenv:header>
          <wsse:security soapenv:mustunderstand="1" xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
             <wsse:usernametoken wsu:id="UsernameToken-26" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
                <wsse:username>jhcheng</wsse:username>
                <wsse:password type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">123456</wsse:password>
                <wsse:nonce encodingtype="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary">uUPNozTlx/2p6/aLb3Fjtw==</wsse:nonce>
                <wsu:created>2010-05-06T08:54:49.288Z</wsu:created>
             </wsse:usernametoken>
          </wsse:security>
       </soapenv:header>
       <soapenv:body>
          <ws:sayhi>
             <text>123</text>
          </ws:sayhi>
       </soapenv:body>
    </soapenv:envelope>

    Server回傳:

    
    
    1
    2
    3
    4
    5
    6
    7
    8
    9
    <soap:envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
       <soap:Body>
          <soap:Fault>
             <faultcode xmlns:ns1="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">ns1:FailedAuthentication</faultcode>
             <faultstring>The security token could not be authenticated or authorized; nested exception is:
        java.io.IOException: wrong password</faultstring>
          </soap:Fault>
       </soap:Body>
    </soap:Envelope>




Related Free Lance Projects on GAF: