I'd be willing to bet it's a combination of the ESC, motor, and gearing. The ESC and Motor you're running are both small for that rig. Will they work? Yes. Are they being pushed close to their limits? Yes. Now you go and replace the stock Mod1 gears with Mod1.5, and now you're asking the motor and ESC to work even harder. I'd be willing to bet dropping teeth on the pinion would fix the issue, going back to the original stock mod1 gearing could possibly fix the issue, or using a bigger motor/ESC combo could also possibly fix the issue. You might even be able to get away with just using a torque-ier motor like the 1100kv EZRUN MAX 56118 SD G2 motor with the Max6 and mod1.5 gearing, but that motor is recommended to be run with the MAX5.
Personally, if it were me, the first thing I'd have done was went back to mod1 gearing but since you already ordered the MAX5, I'd go ahead and order the 1100kv motor and call it a day. That combo will push that mod1.5 gearing NP.
Just as a side note, I've been using Castle since I started in this hobby and Id buy a Castle before a HW any day of the week and twice on Sunday. I've never had an issue with a Castle ESC unless it was defective out of the box (happened once), or unless it was just really old, and I was pushing it hard which will kill any ESC eventually. Hell, I've still got a Castle MMM in one of my rigs, and It's still running. That ESC is from 2012-ish. Anyone that says Castle is junk is an idiot IMO.