The calibration of IMU-camera is a crucial issue for fusion of IMU and camera. Recently popularized low-cost Inertial Measurement Unit (IMU) used in common mobile devices includes large uncertainty such as severe noise and problem in data acquisition. The uncertainty causes the enormous problem in IMU-camera calibration. In this paper, we handle the uncertainty of IMU. The main idea is to impose geometric constraints into existing filter-based calibration method. To incorporate the constraints of states with Kalman filter, we adopt perfect measurement scheme. Experimental results show that the proposed method outperforms existing filter-based method with respect to sensitivity to IMU uncertainty.