Overclocking would worry me. I’m always amazed at how many ICs wedge (and/or wedge the bus) if they get a spurious start or address but invalid input (or whatever). I would either just test overclocking the bus (and then still be nervous) or isolate the low speed segment with FETs like @ToyBuilder suggested or use a repeater/isolator.