Fix immutable inputs DCL bug.

[jsirois]

Previously we used the double-checked-cell-async crate (See:
https://github.com/chrislearn/double-checked-cell-async/blob/46cd3b04eddddbe279282143fe8a936d5854588c/src/lib.rs#L228-L260),
which performed the second check of the double check lock with relaxed
ordering. The relevant code is snipped below and annotated:

pub struct DoubleCheckedCell<T> {
    value: UnsafeCell<Option<T>>,
    initialized: AtomicBool,
    lock: Mutex<()>,
}

impl<T> DoubleCheckedCell<T> {
    pub fn new() -> DoubleCheckedCell<T> {
        DoubleCheckedCell {
            value: UnsafeCell::new(None),
            initialized: AtomicBool::new(false),
            lock: Mutex::new(()),
        }
    }

    pub async fn get_or_try_init<Fut, E>(&self, init: Fut) -> Result<&T, E>
    where
        Fut: Future<Output = Result<T, E>>
    {
        // 1.) 1st load & check.
        if !self.initialized.load(Ordering::Acquire) {
            // 2.) Lock.
            let _lock = self.lock.lock().await;
            // 3.) 2nd load & check.
            if !self.initialized.load(Ordering::Relaxed) {
                {
                    // 4.) Critical section.
                    let result = init.await?;
                    let value = unsafe { &mut *self.value.get() };
                    value.replace(result);
                }
                // 5.) Store with lock held.
                self.initialized.store(true, Ordering::Release);
            }
        }
        let value = unsafe { &*self.value.get() };
        Ok(unsafe { value.as_ref().unchecked_unwrap() })
    }
}

Per the C++11 memory model used by Rust (See:
https://en.cppreference.com/w/cpp/atomic/memory_order), this would
seem to indicate the second load could be reordered to occur anywhere
after the 1st load with acquire ordering and anywhere before the store
with released ordering. If that second load was reordered to occur
before the lock was acquired, two threads could enter the critical
section in serial and the second thread would try to materialize to
paths that already were created and marked read-only. Switch to the
async-oncecell crate which performs both loads of the double-checked
lock with acquire ordering, ensuring they are not re-ordered with
respect to the interleaved non-atomics code.

Also fixup the materialization process to be atomic. We now cleanup
materialization chroots when materialization fails and only move their
contents to the destination path if the full materialization has
succeeded.

Fixes #13899

[ci skip-build-wheels]

[jsirois]

Einteresting. The test failure calls BS on me.

[jsirois]

I've unassigned #13899 and released back to the pool since I still don't understand this and my fix continues to be broken in the same way the current code is.

[jsirois]

Re-opened just for some CI debugging, not for review.

[jsirois]

So it turns out you cannot rename a 555 dir under macOS; thus the perm denied on macOS only in the tests right now. Who knew? I <3 macOS.

[jsirois]

Ok, reviewers - PTAL. There was a bugfix for macOS perms oddity and I've added more failure debug info in case my DCL bug fix (switch to OnceCell) was wrong. That debug info includes all the ways I could think of getting a perm denied in the wild and should help narrow this down more quickly if the issue persists.

[stuhood]

Thanks a lot for persisting here!

[stuhood]

Suggested change

	// If you try to rename a read-only directory (mode 0o555) under masOS you get permission
	// If you try to rename a read-only directory (mode 0o555) under macOS you get permission

[jsirois]

I'm going to save a tree since this is a comment and you've proved its understandable despite typos.

[stuhood]

If we end up needing to do this for any other materialized directories, we might be able to bake it into materialize_directory itself, before the root ends up marked readonly.

[jsirois]

Yeah, maybe. The materialize and move case that this use has would need a callback or an expclict parameter for a final dest.