Add Windows CI

[space-nuko]

No description provided.

[gradio-pr-bot]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-3628-all-demos

[abidlabs]

Thanks @space-nuko for making this addition! Quick question: why are these 3 tests flaky only on Windows?

Let me cc @freddyaboulton @pngwn as well for their thoughts. Quick context guys: as part of this #3608, we generally discussed adding CI to run on Windows, as some Python packages and dependencies behave differently on Windows vs. OSX, and @space-nuko went ahead and added the CI for Windows. I'm broadly in supportive of this PR as I've noticed some tests failing only on Windows (example: tempfile.NamedTemporaryFile requires delete=False to be set to work correctly), which would currently be uncaught.

[space-nuko]

Tests that didn't work when I ran them on CI (they succeed on my machine)

https://github.com/gradio-app/gradio/actions/runs/4526804333/jobs/7972195763

TestProgressBar.test_progress_bar_track_tqdm

self = <test.test_blocks.TestProgressBar object at 0x0000021CA3B75B[48](https://github.com/gradio- app/gradio/actions/runs/4526804333/jobs/7972195763#step:18:49)>

    @pytest.mark.asyncio
    async def test_progress_bar_track_tqdm(self):
        from tqdm import tqdm
    
        with gr.Blocks() as demo:
            name = gr.Textbox()
            greeting = gr.Textbox()
            button = gr.Button(value="Greet")
    
            def greet(s, prog=gr.Progress(track_tqdm=True)):
                prog(0, desc="start")
                time.sleep(0.25)
                for _ in prog.tqdm(range(4), unit="iter"):
                    time.sleep(0.25)
                time.sleep(1)
                for i in tqdm(["a", "b", "c"], desc="alphabet"):
                    time.sleep(0.25)
                return f"Hello, {s}!"
    
            button.click(greet, name, greeting)
        demo.queue(max_size=1).launch(prevent_thread_lock=True)
    
        async with websockets.connect(
            f"{demo.local_url.replace('http', 'ws')}queue/join"
        ) as ws:
            completed = False
            progress_updates = []
            while not completed:
                msg = json.loads(await ws.recv())
                if msg["msg"] == "send_data":
                    await ws.send(json.dumps({"data": [0], "fn_index": 0}))
                if msg["msg"] == "send_hash":
                    await ws.send(json.dumps({"fn_index": 0, "session_hash": "shdce"}))
                if msg["msg"] == "progress":
                    progress_updates.append(msg["progress_data"])
                if msg["msg"] == "process_completed":
                    completed = True
                    break
>       assert progress_updates == [
            [
                {
                    "index": None,
                    "length": None,
                    "unit": "steps",
                    "progress": 0.0,
                    "desc": "start",
                }
            ],
            [{"index": 0, "length": 4, "unit": "iter", "progress": None, "desc": None}],
            [{"index": 1, "length": 4, "unit": "iter", "progress": None, "desc": None}],
            [{"index": 2, "length": 4, "unit": "iter", "progress": None, "desc": None}],
            [{"index": 3, "length": 4, "unit": "iter", "progress": None, "desc": None}],
            [{"index": 4, "length": 4, "unit": "iter", "progress": None, "desc": None}],
            [
                {
                    "index": 0,
                    "length": 3,
                    "unit": "steps",
                    "progress": None,
                    "desc": "alphabet",
                }
            ],
            [
                {
                    "index": 1,
                    "length": 3,
                    "unit": "steps",
                    "progress": None,
                    "desc": "alphabet",
                }
            ],
            [
                {
                    "index": 2,
                    "length": 3,
                    "unit": "steps",
                    "progress": None,
                    "desc": "alphabet",
                }
            ],
        ]
E       AssertionError: assert [[{'desc': 's...e, ...}], ...] == [[{'desc': 's...e, ...}], ...]
E         Left contains one more item: []
E         Full diff:
E           [
E            [{'desc': 'start',
E              'index': None,
E              'length': None,
E              'progress': 0.0,
E              'unit': 'steps'}],
E            [{'desc': None,
E              'index': 0,
E              'length': 4,
E              'progress': None,
E              'unit': 'iter'}],
E            [{'desc': None,
E              'index': 1,
E              'length': 4,
E              'progress': None,
E              'unit': 'iter'}],
E            [{'desc': None,
E              'index': 2,
E              'length': 4,
E              'progress': None,
E              'unit': 'iter'}],
E            [{'desc': None,
E              'index': 3,
E              'length': 4,
E              'progress': None,
E              'unit': 'iter'}],
E            [{'desc': None,
E              'index': 4,
E              'length': 4,
E              'progress': None,
E              'unit': 'iter'}],
E            [{'desc': 'alphabet',
E              'index': 0,
E              'length': 3,
E              'progress': None,
E              'unit': 'steps'}],
E            [{'desc': 'alphabet',
E              'index': 1,
E              'length': 3,
E              'progress': None,
E              'unit': 'steps'}],
E            [{'desc': 'alphabet',
E              'index': 2,
E              'length': 3,
E              'progress': None,
E              'unit': 'steps'}],
E         +  [],
E           ]

test\test_blocks.py:1287: AssertionError
---------------------------- Captured stdout call -----------------------------
Running on local URL:  http://127.0.0.1:78[66](https://github.com/gradio-app/gradio/actions/runs/4526804333/jobs/7972195763#step:18:67)

To create a public link, set `share=True` in `launch()`.

https://github.com/gradio-app/gradio/actions/runs/4525655930/jobs/7970310041#step:18:515

test_queue_when_using_auth

@pytest.mark.asyncio
    async def test_queue_when_using_auth():
        sleep_time = 1
    
        async def say_hello(name):
            await asyncio.sleep(sleep_time)
            return f"Hello {name}!"
    
        with gr.Blocks() as demo:
            _input = gr.Textbox()
            _output = gr.Textbox()
            button = gr.Button()
            button.click(say_hello, _input, _output)
        demo.queue()
        app, _, _ = demo.launch(auth=("abc", "123"), prevent_thread_lock=True)
        client = TestClient(app)
    
        resp = client.post(
            f"{demo.local_url}login",
            data={"username": "abc", "password": "123"},
            follow_redirects=False,
        )
        assert resp.status_code == 200
        token = resp.cookies.get("access-token")
        assert token
    
        with pytest.raises(Exception) as e:
            async with websockets.connect(
                f"{demo.local_url.replace('http', 'ws')}queue/join",
            ) as ws:
                await ws.recv()
        assert e.type == websockets.InvalidStatusCode
    
        async def run_ws(_loop, _time, i):
            async with websockets.connect(
                f"{demo.local_url.replace('http', 'ws')}queue/join",
                extra_headers={"Cookie": f"access-token={token}"},
            ) as ws:
                while True:
                    try:
                        msg = json.loads(await ws.recv())
                    except websockets.ConnectionClosedOK:
                        break
                    if msg["msg"] == "send_hash":
                        await ws.send(
                            json.dumps({"fn_index": 0, "session_hash": "enwpitpex2q"})
                        )
                    if msg["msg"] == "send_data":
                        await ws.send(
                            json.dumps(
                                {
                                    "data": [str(i)],
                                    "fn_index": 0,
                                    "session_hash": "enwpitpex2q",
                                }
                            )
                        )
                        msg = json.loads(await ws.recv())
                        assert msg["msg"] == "process_starts"
                    if msg["msg"] == "process_completed":
                        assert msg["success"]
                        assert msg["output"]["data"] == [f"Hello {i}!"]
                        assert _loop.time() > _time
                        break
    
        loop = asyncio.get_event_loop()
        tm = loop.time()
        group = asyncio.gather(
            *[run_ws(loop, tm + sleep_time * (i + 1) - 1, i) for i in range(3)]
        )
>       await group

test\test_blocks.py:1482: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_loop = <ProactorEventLoop running=False closed=False debug=False>
_time = 1029.375, i = 2

    async def run_ws(_loop, _time, i):
        async with websockets.connect(
            f"{demo.local_url.replace('http', 'ws')}queue/join",
            extra_headers={"Cookie": f"access-token={token}"},
        ) as ws:
            while True:
                try:
                    msg = json.loads(await ws.recv())
                except websockets.ConnectionClosedOK:
                    break
                if msg["msg"] == "send_hash":
                    await ws.send(
                        json.dumps({"fn_index": 0, "session_hash": "enwpitpex2q"})
                    )
                if msg["msg"] == "send_data":
                    await ws.send(
                        json.dumps(
                            {
                                "data": [str(i)],
                                "fn_index": 0,
                                "session_hash": "enwpitpex2q",
                            }
                        )
                    )
                    msg = json.loads(await ws.recv())
                    assert msg["msg"] == "process_starts"
                if msg["msg"] == "process_completed":
                    assert msg["success"]
                    assert msg["output"]["data"] == [f"Hello {i}!"]
>                   assert _loop.time() > _time
E                   assert 1028.406 > 1029.375
E                    +  where 1028.406 = <bound method BaseEventLoop.time of <ProactorEventLoop running=True closed=False debug=False>>()
E                    +    where <bound method BaseEventLoop.time of <ProactorEventLoop running=True closed=False debug=False>> = <ProactorEventLoop running=True closed=False debug=False>.time

test\test_blocks.py:1474: AssertionError
---------------------------- Captured stdout call -----------------------------
Running on local URL:  http://127.0.0.1:78[67](https://github.com/gradio-app/gradio/actions/runs/4525655930/jobs/7970310041#step:18:68)

To create a public link, set `share=True` in `launch()`.

https://github.com/gradio-app/gradio/actions/runs/4525534957/jobs/7970100856

TestDisableFlagging.test_flagging_no_permission_error_with_flagging_disabled

self = <test.test_flagging.TestDisableFlagging object at 0x0000026F0CD68D08>

    def test_flagging_no_permission_error_with_flagging_disabled(self):
        with tempfile.TemporaryDirectory() as tmpdirname:
            os.chmod(tmpdirname, 0o444)  # Make directory read-only
            nonwritable_path = os.path.join(tmpdirname, "flagging_dir")
    
            io = gr.Interface(
                lambda x: x,
                "text",
                "text",
                allow_flagging="never",
                flagging_dir=nonwritable_path,
            )
            try:
>               io.launch(prevent_thread_lock=True)

test\test_flagging.py:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\tempfile.py:807: in __exit__
    self.cleanup()
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\tempfile.py:811: in cleanup
    _shutil.rmtree(self.name)
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\shutil.py:516: in rmtree
    return _rmtree_unsafe(path, onerror)
c:\hostedtoolcache\windows\python\3.7.9\x64\lib\shutil.py:[40](https://github.com/gradio-app/gradio/actions/runs/4525534957/jobs/7970100856#step:18:41)4: in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpsxrziwfp'
onerror = <function rmtree.<locals>.onerror at 0x0000026F16375E[58](https://github.com/gradio-app/gradio/actions/runs/4525534957/jobs/7970100856#step:18:59)>

    def _rmtree_unsafe(path, onerror):
        try:
            with os.scandir(path) as scandir_it:
                entries = list(scandir_it)
        except OSError:
            onerror(os.scandir, path, sys.exc_info())
            entries = []
        for entry in entries:
            fullname = entry.path
            try:
                is_dir = entry.is_dir(follow_symlinks=False)
            except OSError:
                is_dir = False
            if is_dir:
                try:
                    if entry.is_symlink():
                        # This can only happen if someone replaces
                        # a directory with a symlink after the call to
                        # os.scandir or entry.is_dir above.
                        raise OSError("Cannot call rmtree on a symbolic link")
                except OSError:
                    onerror(os.path.islink, fullname, sys.exc_info())
                    continue
                _rmtree_unsafe(fullname, onerror)
            else:
                try:
                    os.unlink(fullname)
                except OSError:
                    onerror(os.unlink, fullname, sys.exc_info())
        try:
>           os.rmdir(path)
E           PermissionError: [WinError 5] Access is denied: 'C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\tmpsxrziwfp'

c:\hostedtoolcache\windows\python\3.7.9\x[64](https://github.com/gradio-app/gradio/actions/runs/4525534957/jobs/7970100856#step:18:65)\lib\shutil.py:402: PermissionError
---------------------------- Captured stdout call -----------------------------
Running on local URL:  http://127.0.0.1:78[73](https://github.com/gradio-app/gradio/actions/runs/4525534957/jobs/7970100856#step:18:74)

To create a public link, set `share=True` in `launch()`.

[pngwn]

Why are certain tests classed as flaky only on windows? We should get those behaving the same across OSes.

I am definitely in favour of this addition (although windows machines are typically slo.wer on GH Actions, so we need to be ready for a slowdown in CI)

[pngwn]

Internal changes don't require a changelog entry.

@abidlabs @freddyaboulton We should remove this section from the changelog. Its adds noise for people who are interested in learning about bug fixes and new features. We can keep track of internal changes elsewhere if we want to but we should separate them from the changleog which is for users and should only contain changes with have a user-impact.

[abidlabs]

Sgtm we can remove in a separate PR

[freddyaboulton]

Only downside is that if a community member makes an internal change (modifying a guide), they would not get credited

[abidlabs]

That's true. I'll leave things as is in this PR, we can figure out a policy in a separate PR

[abidlabs]

Tests that didn't work when I ran them on CI (they succeed on my machine)

Just to confirm I'm not missing anything, @space-nuko, they are working now on CI, correct? I mean all of the tests (including) flaky test are passing on Windows CI

[space-nuko]

All the tests succeed on my computer and CI yes

[abidlabs]

A different test fails for me locally on Windows: test/test_components.py::TestImage::test_serialize_url

Let me see if I can get the tests to run work consistently across OS-es and then we should be good here to merge.

The CI isn't running all of the tests. Maybe because of this? Gonna dismiss but will fix the issue you raised

[abidlabs]

Made some updates to the tests:

• \test\test_blocks.py::test_queue_when_using_auth should indeed have been marked as flaky across all operating systems, as discussed in the original issue when this first came out (thanks @freddyaboulton for pointing me to that). But given that the test is flaky and very non-essential, I have opted to remove it altogther
• test.test_flagging.TestDisableFlagging::test_flagging_no_permission_error_with_flagging_disabled was written in a way that was not valid for Windows (Windows deletes temporary directories but the test changed the permissions on the directory so that it could not be deleted). This test has been rewritten to work across OS systems

With these two changes, the CI is passing locally and on GitHub so we should be good to merge.

[freddyaboulton]

Thank you @space-nuko and @abidlabs !

[abidlabs]

Merging, thanks for the contribution @space-nuko!