Automatic polyp detection is to detect and to localize polyps in endoscopy (or colonoscopy) images. It is challenging because the of polyps and their appearances are diverse, and furthermore, the colors and textures of polyps are quite similar with those of normal tissues in many cases. For these reasons, it is often very hard to distinguish polyps from normal tissues by using the conventional methodology. To effectively resolve these challenges, we propose a new framework based on multi-classifier learning and a contour intensity difference (CID) measure. To successfully detect polyps of diverse appearances, we first classify polyps into K different types according to their shapes via unsupervised learning. We then subsequently learn K different classifiers to detect K different types of polyps. This multi-classifier learning improves a polyp detection rate. However, in return, false positives also increase because of the colon structures that look similar to polyps. For reducing false positives while conserving the high detection rate, we also propose a new CID measure. Experimental results using public and our own datasets show that the proposed methods are promising for detecting polyps with diverse appearances.