pipeline does not stop consuming `from` when `to` is closed

Description

The docs for pipeline say "Will stop consuming the from channel if the to channel closes." however the following code will print 0..49 (and would continue consuming from forever).

Cause: Currently, when the output go loop puts to the to channel and gets a closed result indicator (logical false), it only stops the output go block, but lets the ingest and compute go loops continue, so the pipeline keeps consuming the from channel and computing values it can't use in the output. In other words, it does not do what it says it will do.

Alternatives:

  1. Change the docstring to not promise to stop consuming from(this seems wrong)

  2. When to is closed, signal the ingest and compute go blocks to stop, either via state (atom), or a control channel

  3. Push the signal back through the compute and ingest blocks somehow, maybe by closing the jobs and results channels and watching for that?

Proposed:

  • Create an atom indicating ingest processing should stop.

  • In the output loop, if the to channel is found to be closed, reset the stop atom to true.

  • In the ingest loop, check the stop atom - if set, stop ingesting and close the jobs chan (as we already do when the from channel is closed)

Note that some work may be in flight (1 may be in-process in the output go block, N may be in the results channel, and N may be parked in flight in process go blocks)

Patch: async-217-2.patch (same change in clj and cljs)

The example above usually prints 0…2 with the patch.

Screened by: Ghadi

Environment

None

Attachments

3

Activity

Show:

Alex Miller November 22, 2022 at 4:22 AM

-3 patch is alternate approach to use a control channel to shutdown the ingest go loop and job threads. have not yet replicated to cljs

Alex Miller February 3, 2022 at 2:58 PM

-2 patch is same, just merges context in the patch to make it easier to glance at

Alex Miller August 16, 2018 at 8:48 PM

Tianxiang: I'm inclined to agree with you, just being thorough in listing options.

import May 17, 2018 at 6:22 PM

Comment made by: xiongtx

One option would be to change the doc string and not promise this behavior.

The expectation should be that we don't endlessly process inputs when we're not consuming outputs, which would certainly be the case when the to channel is closed.

Details

Assignee

Reporter

Approval

Screened

Priority

Created May 17, 2018 at 5:26 PM
Updated November 22, 2022 at 4:22 AM