Unexpected multi-threading issues on Julia 1.8.5

[jakobnissen]

This might be me not understand that I do something thread-unsafe. Here is a small example, which should be run with 8 threads:

const BUFS = Vector{Vector{UInt8}}(undef, 8)

function foo(i)
    buf = IOBuffer()
    run(pipeline(`cat /my/large/file`; stdout=buf))
    BUFS[i] = copy(buf.data)
end

function bar()
    Threads.@threads for i in 1:8
        foo(i)
    end
end

for i in 1:10
    bar(); println(allequal(BUFS))
end

And I get

true
false
false
false
true
true
false
true
false
true

Adding sleep(1) after the command makes the issue go away. Removing the copy makes the issue go away.
This to me suggests that the run command wrongly reports it's finished before the data is actually inside the IOBuffer.

[vtjnash]

IOBuffer cannot be used for IO (edit: of process output). There is an open issue for me to implement an error for it.

[jakobnissen]

I see. So, the issue is that there is no mechanism to lock the IOBuffer, as essentially the main Julia process and the external command are two distinct threads both operating on it, which is inherently thread-unsafe?

Closing this issue. Is there a thread-safe alternative for an in-memory buffer I can use?

[vtjnash]

Base.BufferStream is the correct replacement for something that will read everything with back-pressure features disable, or a Base.PipeEndpoint to read incrementally

[jakobnissen]

For future references: The solution is #42424, possibly with a reference from the IOBuffer docstring to BufferStream