Three possible workarounds for the I2C issues caused by (ill-behaving) external display:
- 
Use a different OLED display module. Assuming this is an incidental and not a more generic (SSD1306?) OLED module related issue.
 - 
Use a separate/second I2C bus (with different IO pins) of the ESP32 for the display.
 - 
Connect the display via an I2C bus isolator like TCA4311A.