In visual object tracking area, exploiting correlation filter to track the target of interest opened a new chapter. Moreover, by adopting a circulant form of an image or feature descriptors jointed with the convolution theorem, this correlation filter tracker surpassed many of the previous state-of-the-art trackers in both tracking speed and stability. Nevertheless, when the appearance of the target object abruptly changes due to occlusion, background cluttering or viewpoint variation, even the aforementioned correlation filter tracker fails to compute a plausible correlation output. Concerned by this problem, we propose a method that observes the locational drift of the correlation peak from the desired location. Utilizing this information, we penalize the searching range of the correlation peak. Thus, the accuracy of the tracker increases. Our proposed method is verified by using 2014 Visual Object Tracking Challenge benchmark dataset.