Skip to content

Commit

Permalink
skip windows GPU check if changes only in doc (#13248)
Browse files Browse the repository at this point in the history
### Description
Use Path filter and fake workflow to skip windows GPU check if there's
only changes in doc.
Refs:

https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks

The fake github yaml is generated by code.

### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->

###verifications:###
In this PR:
since the win-gpu-ci-pipeline.yml and .github are updated, so the real
Windows GPU workflows are always triggered.

in #13256
To avoid update win-gpu-ci-pipleline.yml, I added the path filter in
devops page. the fake win GPU workflows triggered, and the real
workflows are skipped.
  • Loading branch information
mszhanyi authored Oct 11, 2022
1 parent 4d25b9c commit ea128cd
Show file tree
Hide file tree
Showing 5 changed files with 121 additions and 8 deletions.
48 changes: 48 additions & 0 deletions .github/workflows/generate-skip-doc-change.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# generate_skip_doc_change.py

from pathlib import Path

from jinja2 import Environment, FileSystemLoader, Template

GITHUB_DIR = Path(__file__).resolve().parent.parent


class Skipped_Workflow:
def __init__(self, workflow_name: str, job_names: list, output_file_name: str):
self.workflow_name = workflow_name
self.job_names = job_names
self.fake_file_name = output_file_name


WIN_GPU_CI_WORKFLOW = Skipped_Workflow(
workflow_name="Windows GPU CI Pipeline",
job_names=[
"cuda build_x64_RelWithDebInfo",
"dml build_x64_RelWithDebInfo",
"training build_x64_RelWithDebInfo",
"kernelDocumentation build_x64_RelWithDebInfo",
],
output_file_name=str(GITHUB_DIR.joinpath("workflows/generated_fake_win_gpu_ci.yml")),
)


def generate_fake_ci_yaml(template: Template, workflow: Skipped_Workflow):
content = template.render(ci_workflow_name=workflow.workflow_name, job_names=workflow.job_names)

filename = workflow.fake_file_name
with open(filename, mode="w", encoding="utf-8") as output_file:
output_file.write(content)
if content[-1] != "\n":
output_file.write("\n")
print(f"... wrote {filename}")


def main() -> None:
environment = Environment(loader=FileSystemLoader(str(GITHUB_DIR.joinpath("workflows/"))))
template = environment.get_template("skip-doc-change.yml.j2")
skipped_workflows = [WIN_GPU_CI_WORKFLOW]
[generate_fake_ci_yaml(template, workflow) for workflow in skipped_workflows]


if __name__ == "__main__":
main()
36 changes: 36 additions & 0 deletions .github/workflows/generated_fake_win_gpu_ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Don't edit this file manully.
# Run generate_fake_workflows.py to generate it.

name: Windows GPU CI Pipeline
on:
push:
paths:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md

jobs:
job1:
name: cuda build_x64_RelWithDebInfo
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required, only documentation changed"'

job2:
name: dml build_x64_RelWithDebInfo
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required, only documentation changed"'

job3:
name: training build_x64_RelWithDebInfo
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required, only documentation changed"'

job4:
name: kernelDocumentation build_x64_RelWithDebInfo
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required, only documentation changed"'
20 changes: 20 additions & 0 deletions .github/workflows/skip-doc-change.yml.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Don't edit this file manully.
# Run generate_fake_workflows.py to generate it.

name: {{ ci_workflow_name }}
on:
push:
paths:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md

jobs:
{%- for name in job_names %}
job{{ loop.index }}:
name: {{ name }}
runs-on: ubuntu-latest
steps:
- run: 'echo "No build required, only documentation changed"'
{% endfor %}
13 changes: 5 additions & 8 deletions docs/ABI_Dev_Notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,22 +5,22 @@ Global variables may get constructed or destructed inside "DllMain". There are s
Onnxruntime must support explicit linking, where the operating system loads the DLL on demand at runtime, instead of process startup time. This is required by our language bindings like C#/Java.

However, there are some special restrictions on this, If a thread local variable need non-trivial construction, for the threads already exist before onnxruntime.dll is loaded, the variable won't get initialized correctly. So it's better to only access such variables from onnxruntime internal threads, or make these variables function local (Like the magic statics).


## No undefined symbols
On Windows, you can't build a DLL with undefined symbols. Every symbol must be get resolved at link time. On Linux, you can.
In order to simplify things, we require every symbol must get resolved at link time. The same rule applies for all the platforms. And this is easier for us to control symbol visibility.
In order to simplify things, we require every symbol must get resolved at link time. The same rule applies for all the platforms. And this is easier for us to control symbol visibility.


## Default visibility and how to export a symbol
On Linux, by default, at linker's view, every symbol is global. It's easy to use but it's also much easier to cause conflicts and core dumps. We have encountered too many such problems in ONNX python binding. Indeed, if you have a well design, for each shared lib, you only need to export **one** function. ONNX Runtime python binding is a good example. See [pybind11 FAQ](https://github.com/pybind/pybind11/blob/master/docs/faq.rst#someclass-declared-with-greater-visibility-than-the-type-of-its-field-someclassmember--wattributes) for more info.

For controlling the visibility, we use linker version scripts on Linux and def files on Windows. They work similar. That:
1. Only C functions can be exported.
1. Only C functions can be exported.
2. All the function names must be explicitly listed in a text file.
3. Don't export any C++ class/struct, or global variable.

Also, on Linux and Mac operating systems, all the code must be compiled with "-fPIC".
Also, on Linux and Mac operating systems, all the code must be compiled with "-fPIC".
On Windows, we don't use dllexport but we still need dllimport.

Therefore, our DLLEXPORT macro is like:
Expand All @@ -38,7 +38,4 @@ Therefore, our DLLEXPORT macro is like:
```

## static initialization order problem
It's well known C++ has [static initialization order problem](https://isocpp.org/wiki/faq/ctors#static-init-order). Dynamic linking can ensure that onnxruntime's static variables are already initialized before any onnxruntime's C API get called. The same thing applies to their destructors. It's good. But on the other side, static linking may have more usage restrictions on some of the APIs.



It's well known C++ has [static initialization order problem](https://isocpp.org/wiki/faq/ctors#static-init-order). Dynamic linking can ensure that onnxruntime's static variables are already initialized before any onnxruntime's C API get called. The same thing applies to their destructors. It's good. But on the other side, static linking may have more usage restrictions on some of the APIs.
12 changes: 12 additions & 0 deletions tools/ci_build/github/azure-pipelines/win-gpu-ci-pipeline.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,15 @@
trigger:
branches:
include:
- master
- rel-*
paths:
exclude:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md

parameters:
- name: RunOnnxRuntimeTests
displayName: Run Tests?
Expand Down

0 comments on commit ea128cd

Please sign in to comment.