Leave em, the axle will stop sliding and start binding very quicky. It will affect the way your suspension works.
Edit;You talking about the pivot ball dust boots? Leave those too, I tried running without them. Things got gritty real quick. If you are running on pavement you can take em off without worry but off-road, nope. Not worth it IMO. I had to tear off the carriers after every race, clean and relube them to keep things moving smoothly.