Draughts From Underneath Kitchen Cabinets

I'm experiencing terrible draughts from underneath some kitchen base cabinets which makes the kitchen constantly cold. I removed the plinth/kick-board to try to see what the problem might be and discovered what looks like concrete/grey plaster which finishes in an uneven line about 6 inches above the solid floor. Beneath that is just bare brick down to floor level and what seems to be earth too when I manage to get my hand back there to feel whether there are holes where the draught may be coming from. It feels slightly damp under there too in places. I do sometimes find some slight mould appearing on the back panels of the corner base unit when I'm removing the contents to clean it out.
The cabinets are situated in an L-shape with an outside wall behind one part and a partition wall behind the other. It's an 1872 mid-terrace, so solid wall construction.
I've seen online lots of people experiencing similar problems and read that some have filled the space between the bottom of the cabinets and floor with insulation material in an attempt to alleviate the draught somewhat. I don't think that is going to be any good here based on what I can see. I'm sure it needs more than that but I really don't know what needs to be done to sort out the problem. Any sound advice would be very, very welcomed and appreciated.
Thank you