So, just 4AA rechargeable cells in the AA holder? If so, try getting some name brand (energizer/duracell) normal alkaline batteries and see how it does.
4 rechargeable AA's have a running voltage of around 5V (each cell rated at 1.2V, fully charged under load puts them around 5V). 4.8V is the cut off for things to stop working. Combine the 4 cells only with cold temps and odds are, your riding the line. Getting a proper 6V hump pack and charger will help you out in the long run. I just run cheap
integy/intellect/tenergy ones I get on ebay for $12-$15 and they hold up for a long time.
That said, the switch may still be the bad link. You can bypass the switch and plug the battery directly into the receiver to rule it out, but I'd do so with 4 alkaline cells instead.