We present a novel algorithm for point pattern matching by means of spectra of directed graphs. Given a feature point-set, we construct a weighted directed graph and skew-symmetric matrix associated with the graph. By using spectral decomposition of the matrix, we give a spectral representation of the feature points with half of the eigenvectors. We theoretically analyze that our method can well deal with the matching problem under affine transformation. The expreiments applied to synthetic data and real-world images show the effectiveness of our method.