A panchromatic image based spectral imagery super resolution algorithm is proposed to resolve the problem of remote sensing image resolution enhancement in this paper. A simple gray scale mapping method is first presented to better use high frequency details distilled from a high resolution reference panchromatic image, then a Maximum a posterior (MAP) based iterative optimization process is introduced to realize super resolution while keeping spectral characters. Evaluated by both simulated and really acquired images, the proposed method effectively improved the resolution of spectral imagery. PSNR (Peak signal to noise ratio) of the reconstructed images has improved 1~5dB compared with the traditional bilinear interpolation and IHS (Intensity hue saturation) based method. Subjective quality of the enhanced images has obviously exceeded the other twos as well.