fix: s/sliding-stream w/ certain executor configs#241
Conversation
Fixes an issue where, depending on where the source stream was running, the sliding stream would never emit later items downstream. We now move consumption onto `d/loop` rather than via the `connet-via`. Updates tests to a more imperative variant to try and mimic items coming in faster than consumption.
|
This PR is probably not worth merging. I am still seeing the behavior it was intended to fix. If you like the code it might be more "idiomatic" per other internal stream implementations, but the underlying issue remains. Will submit a PR once I have a real fix. Sorry for the noise. |
|
@rschmukler No worries, and thanks for the effort! It might help to open an issue to talk it through first. Either that or discuss it in #aleph/#manifold. Also, ping @DerGuteMoritz when it's ready, I'm not super-involved in Aleph/Manifold maintenance these days. Probably need to find a more active maintainer... |
|
The issue that I am running into is that under heavy production load, the stream that is producing items can end up cycling through the buffer (via Eg. with a buffer size This might have something to do with (a lack of) fairness guarantees around In its current form I lost two days debugging what was going on; it might be worth deprecating it (and I say that as the original author haha) |
For the record: those functions were introduced quite recently and you might not be the only one encountering this issue. I would be interested to have more feedback from the field. |
Fixes an issue where, depending on which thread the source stream was running, the sliding stream would never emit later items downstream.
We now move consumption onto
d/looprather than via theconnet-via.Updates tests to a more imperative variant to try and mimic items coming in faster than consumption.