Update: The problem is still there.
It took some time to show up, but LMIC still starts to delay the sending occasionally. This keeps the RFM95 awake for anything between 2 and up to 5 minutes while using using 8 mA or so. With this behaviour, really low power applications with the 328P are impossible.
I can see here https://www.thethingsnetwork.org/forum/t/full-arduino-mini-lorawan-below-1ua-sleep-mode/8059 that @Charles obviously did something similar and I am wondering what I do fundamentally wrong?