Trust me lads, if you go around without the panniers, REMOVE THE BOLTS AND KEEP THEM SAFE.
Or just do them up properly and use threadlock? Mine haven't fallen out in 26k miles (mostly riding without the panniers fitted) and I'm sure very few others will have lost them either.