Abstract We propose a robust object recognition system where patch-based pyramid images and the spatial relationships among patches are utilized for our image model. In particular, both a color histogram (CH) and a color co-occurrence histogram (CCH) are applied to obtain image features for each patch. The locations of subregions to be tested are decided by a particle filter in our matching process. We show that the performance of object recognition can be improved by using the spatial relationships among patches. To show the validity of our proposed method, we employ input images from various environments as test images.