It is widely believed that integer quantum Hall systems do not have fractional excitations. Here we show the converse to be true for a class of systems where integer quantum Hall effect emerges spontaneously due to the interplay of itinerant electrons and noncoplanar magnetic ordering. We show that magnetic $Z_2$ vortices in such systems, whose topological stability is guaranteed by the structure of the order parameter, indeed carry fractional charge.