We present a projective symmetry group (PSG) analysis of the spinless excitations of Z2 spin liquids on the kagome lattice. In the simplest case, vortices carrying Z2 magnetic flux ('visons') are shown to transform under the 48 element group GL(2, Z3). Alternative exchange couplings can also lead to a second case with visons transforming under 288 element group GL(2, Z3) \\times D3. We study the quantum phase transition in which visons condense into confining states with valence bond solid order. The critical field theories and confining states are classified using the vison PSGs.