Yeahhhhhhh. Hate this.
Were we at a round table, I would nod & agree with everyone's statements as we went around, and when things got to me, I'd recall that, from my house (my point of world contact for 12+ hours of every day for the past 14 years), nearly every time I had similar issues, there would be a bump up in measured internet throughput within maybe two weeks' time. Like, 70|30-ish. Better than half.
My conclusion? (And this is still probabilistically constrained...) Your local ISP is tweaking/cranking, and when things get done, it will settle down, and possibly there might be a bump up on throughput, but not guaranteed.
Why is the real-time stuff not affected? A GUESS BUT, I'd guess that there's enough of a buffer -- "low-latency" read-backs not announced like TWS, that are at work. TOTAL guess.
Hope is doesn't persist for you. A pain if you don't know the answer....