The distance sensor needs a good 5V supply to work reliably. I no longer use the 4.5V Crumble battery pack, it’s not reliable enough with the distance sensor, especially if driving motors at the same time.
I use a Poundland, £1 lithium battery pack and hack the USB cable by removing the micro USB plug and putting crocks on instead. Works great. Two can be connected in parallel if you need more current.