High-finesse ultra-low expansion (ULE) cavities are important components in many precision optical experiments, and the cavity length noise in ULE cavities has been intensively studied. However, some experiments, such as vacuum magnetic birefringence measurements, have focused on the phase difference between orthogonal polarizations. To determine the effect of high-reflectivity mirrors on birefringence noise, we investigated the birefringence noise of a ULE cavity with the finesse of 531000(1000) and obtained the noise spectrum of the intrinsic birefringence of the cavity. The single-pass phase difference due to the birefringence of the cavity mirror is (21.3 ± 0.2) μrad and the noise for the birefringent cavity is 81.3 prad∕√Hz at 0.1 Hzand 2.6 prad∕√Hz at 40 Hz. These results may draw attention when considering noise in cavity-based optical experiments.