ci: bundle devicetree Python package

dtsh implementation is tightly coupled to the edtlib (sometimes private) API, which: a) is part of the Zephyr project source tree (at zephyr/scripts/dts/python-devicetree): this is the "authoritative" source for the edtlib library source code b) has its own, work-in-progress, repository (as project zephyrproject-rtos/python-devicetree): it lacks behind (a) and is not intended for public use yet [1] c) is also available from PyPI (as project devicetree): this package also lacks behind a), and additionally its version numbers can't be easily matched with Zephyr tags The approach here is to: - NOT rely on PyPI to get edtlib (i.e.. remove 'devicetree' from the dtsh Python requirements in setup.py and friends) - bundle edtlib with dtsh, and update it with each new Zephyr stable version Limitation: bundling edtlib definitely plays against installing dtsh within the same Python virtual environment as west. [1]: https://github.com/zephyrproject-rtos/zephyr/tree/main/scripts/dts
dottspina · Apr 26, 2023 · 5e803eb · 5e803eb
1 parent 2111257
commit 5e803eb
Show file tree

Hide file tree

Showing 10 changed files with 5,465 additions and 152 deletions.
diff --git a/README.org b/README.org
@@ -70,6 +70,7 @@ All kinds of feedback and contribution are encouraged: please refer to the botto
 # Install dtsh in a dedicated Python virtual environment
 $ python -m venv --prompt dtsh .venv
 $ . .venv/bin/activate
+$ pip install --upgrade pip setuptools
 $ pip install --upgrade dtsh
 
 # Setting ZEPHYR_BASE will help dtsh in building a default bindings search path
@@ -142,12 +143,6 @@ It's recommended to install ~dtsh~ in a dedicated Python virtual environment.
 A Python /best practice/ is to always install a consistent set of /scripts/ and their dependencies in a dedicated
 [[https://peps.python.org/pep-0405/][virtual environment]], with up-to-date ~pip~, ~setuptools~ and ~wheel~ packages.
 
-#+begin_src sh
-python -m venv .venv
-. .venv/bin/activate
-pip install --upgrade pip setuptools wheel
-#+end_src
-
 See also [[https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/][Installing packages using pip and virtual environments]].
 
 *** Install from sources
@@ -159,7 +154,7 @@ git clone https://github.com/dottspina/dtsh.git
 cd dtsh
 python -m venv .venv
 . .venv/bin/activate
-pip install --upgrade pip setuptools wheel
+pip install --upgrade pip setuptools
 pip install .
 #+end_src
 
@@ -168,7 +163,7 @@ pip install .
 Install from [[https://pypi.org/project/dtsh/][PyPI]] in a dedicated Python virtual environment:
 
 #+begin_src sh
-python -m venv --prompt dtsh .venv
+python -m venv .venv
 . .venv/bin/activate
 pip install --upgrade pip setuptools
 pip install --upgrade dtsh
@@ -179,8 +174,9 @@ pip install --upgrade dtsh
 To remove ~dtsh~ and all its direct dependencies from a dedicated virtual environment:
 
 #+begin_src sh
-. /path/to/.venv/bin/activate
-pip uninstall dtsh rich Pygments devicetree
+cd dtsh
+. .venv/bin/activate
+pip uninstall dtsh rich Pygments
 #+end_src
 
 ** Run
@@ -242,64 +238,8 @@ See also issue [[https://github.com/dottspina/dtsh/issues/1#issuecomment-1278281
 
 ** Zephyr integration
 
-We'll assume a [[https://docs.zephyrproject.org/latest/develop/west/][west]]-managed Zephyr [[https://docs.zephyrproject.org/latest/develop/west/basics.html#example-workspace][workspace]] with a typical file layout
-(see [[https://docs.zephyrproject.org/latest/develop/getting_started/#get-zephyr-and-install-python-dependencies][Get Zephyr and install Python dependencies]]):
-
-#+begin_src
-zephyrproject/                 # Workspace topdir
-│
-│   # Per-workspace Python virtual environment, may be updated by west after manifest modification:
-├── .venv/
-│   └── bin                    # Python run-time and Zephyr tools (e.g. west, pylink, pyocd)
-│   └── lib                    # required Python libraries
-│
-├── .west/                     # marks the location of the workspace topdir
-│   └── config                 # per-workspace local configuration file
-│
-│   # The manifest repository, never modified by west after creation:
-├── zephyr/                    # .git/ repo
-│   └── west.yml               # manifest file
-│
-│   # Projects managed by west:
-├── modules/
-│   └── lib/
-│       └── tinycbor/          # .git/ project
-├── net-tools/                 # .git/ project
-└── [ ... other projects ...]
-#+end_src
-
-It's then possible to install ~dtsh~ in the same /command line development environment/ as ~west~:
-
-#+begin_src sh
-# Activate the Python venv as usual, e.g.:
-. /path/to/zephyrproject/.venv/bin/activate
-
-# Install latest dtsh release from PyPI
-pip install dtsh
-#+end_src
-
-And to simply run ~dtsh~ without any argument:
-
-#+begin_src sh
-# Activate the Python venv as usual, e.g.:
-. /path/to/zephyrproject/.venv/bin/activate
-# Set the Zephyr kernel environment as usual, e.g.:
-. /path/to/zephyrproject/zephyr/zephyr-env.sh
-
-# Build the Zephyr firmware as usual, e.g.:
-west build $ZEPHYR_BASE/samples/sensor/bme680
-# Open the generated DTS file build/zephyr/zephyr.dts using default bindings
-dtsh
-#+end_src
-
-To remove ~dtsh~ from a Zephyr workspace:
-
-#+begin_src sh
-. /path/to/zephyrproject/.venv/bin/activate
-pip uninstall dtsh rich
-#+end_src
-
-⚠ Be sure to NOT uninstall packages otherwise used within the Python virtual environment, e.g. ~rich~.
+*WARNING*: It's no longer advised to install ~dtsh~ within the same Python virtual environment
+as ~west~.
 
 * User's guide
 
@@ -698,7 +638,8 @@ git clone https://github.com/dottspina/dtsh.git
 cd dtsh
 python -m venv .venv
 . .venv/bin/activate
-pip install --upgrade pip setuptools wheel
+pip install --upgrade pip setuptools
+pip install -r requirements-dev.txt
 pip install --editable .
 #+end_src
 
@@ -711,9 +652,6 @@ To run a few unit tests:
 #+begin_src sh
 cd dtsh
 . .venv/bin/activate
-# install test requirements
-pip install ".[test]"
-# run unit tests
 python -m pytest tests
 #+end_src
 

diff --git a/pyrightconfig.json b/pyrightconfig.json
@@ -0,0 +1,25 @@
+{
+  "include": [
+    "src",
+    "tests"
+  ],
+
+  "exclude": [
+    "tests/bindings"
+  ],
+
+  "pythonVersion": "3.8",
+  "pythonPlatform": "All",
+
+  "venvPath": ".",
+  "venv": ".venv",
+
+   "executionEnvironments": [
+     {
+      "root": "tests",
+      "extraPaths": [
+          "src"
+        ]
+     }
+   ]
+}
diff --git a/pytest.ini b/pytest.ini
@@ -0,0 +1,9 @@
+# pytest configuration.
+#
+# See:
+# - https://docs.pytest.org/en/7.3.x/reference/customize.html#pytest-ini
+# - https://docs.pytest.org/en/7.3.x/reference/reference.html#ini-options-ref
+
+[pytest]
+testpaths = tests
+pythonpath = src
diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -0,0 +1,7 @@
+# Python requirements for development/tests.
+#
+mypy
+types-PyYAML
+pyright
+pylint
+pytest
diff --git a/setup.py b/setup.py
@@ -1,8 +1,4 @@
 """Project configuration (setuptools).
-
-See:
-https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/#setup-args
-https://github.com/pypa/sampleproject
 """
 
 # Always prefer setuptools over distutils
@@ -11,25 +7,13 @@
 
 here = pathlib.Path(__file__).parent.resolve()
 
-# Get the long description from the README file
 long_description = (here / "README.rst").read_text(encoding="utf-8")
 
-# Arguments marked as "Required" below must be included for upload to PyPI.
-# Fields marked as "Optional" may be commented out.
 setup(
-    # This is the name of your project. The first time you publish this
-    # package, this name will be registered for you. It will determine how
-    # users can install this project, e.g.:
-    #
-    # $ pip install sampleproject
-    #
-    # And where it will live on PyPI: https://pypi.org/project/sampleproject/
-    #
     # There are some restrictions on what makes a valid project name
     # specification here:
     # https://packaging.python.org/specifications/core-metadata/#name
     #
-    # Required.
     name="dtsh",
 
     # Versions should comply with PEP 440:
@@ -41,26 +25,17 @@
     #
     # See also: https://peps.python.org/pep-0440/
     #
-    # Required.
-    version="0.1.0a4",
+    version="0.1.0a5",
 
     # This is a one-line description or tagline of what your project does. This
     # corresponds to the "Summary" metadata field:
     # https://packaging.python.org/specifications/core-metadata/#summary
     #
-    # Optional.
     description="Shell-like interface with Zephyr devicetree and bindings",
 
-    # This is an optional longer description of your project that represents
-    # the body of text which users will see when they visit PyPI.
-    #
-    # Often, this is the same as your README, so you can just read it in from
-    # that file directly (as we have already done above)
-    #
     # This field corresponds to the "Description" metadata field:
     # https://packaging.python.org/specifications/core-metadata/#description-optional
     #
-    # Optional.
     long_description=long_description,
 
     # Denotes that our long_description is in Markdown; valid values are
@@ -74,39 +49,22 @@
     # This field corresponds to the "Description-Content-Type" metadata field:
     # https://packaging.python.org/specifications/core-metadata/#description-content-type-optional
     #
-    # Optional (see note above).
     #long_description_content_type="text/markdown",
 
-    # This should be a valid link to your project's main homepage.
-    #
     # This field corresponds to the "Home-Page" metadata field:
     # https://packaging.python.org/specifications/core-metadata/#home-page-optional
     #
-    # Optional.
     url="https://github.com/dottspina/dtsh",
 
-    # This should be your name or the name of the organization which owns the
-    # project.
-    #
-    # Optional.
     author="Chris Duf",
-
-    # This should be a valid email address corresponding to the author listed
-    # above.
-    #
-    # Optional.
     author_email="chris@openmarl.org",
 
-    # The license argument is more typically used to indicate differences from well-known licenses.
-    #
-    # Optional.
     license="Apache License version 2.0",
 
     # Classifiers help users find your project by categorizing it.
     #
     # For a list of valid classifiers, see https://pypi.org/classifiers/
     #
-    # Optional.
     classifiers=[
         # How mature is this project? Common values are
         #   3 - Alpha
@@ -122,46 +80,26 @@
         # that you indicate you support Python 3. These classifiers are *not*
         # checked by 'pip install'. See instead 'python_requires' below.
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3.10",
         "Programming Language :: Python :: 3 :: Only",
     ],
 
-    # This field adds keywords for your project which will appear on the
-    # project page. What does your project relate to?
-    #
     # Note that this is a list of additional keywords, separated
     # by commas, to be used to assist searching for the distribution in a
     # larger catalog.
     #
-    # Optional.
     keywords="devicetree, zephyr, dts, embedded",
 
-    # When your source code is in a subdirectory under the project root, e.g.
-    # `src/`, it is necessary to specify the `package_dir` argument.
-    #
-    # Optional.
     package_dir={"": "src"},
-
-    # You can just specify package directories manually here if your project is
-    # simple. Or you can use find_packages().
-    #
-    # Alternatively, if you just want to distribute a single Python file, use
-    # the `py_modules` argument instead as follows, which will expect a file
-    # called `my_module.py` to exist:
-    #
-    #   py_modules=["my_module"],
-    #
-    # Required.
     packages=find_packages(where="src"),
 
     # Specify which Python versions you support. In contrast to the
     # 'Programming Language' classifiers above, 'pip install' will check this
     # and refuse to install the project if the version does not match. See
     # https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires
-    python_requires=">=3.7, <4",
+    python_requires=">=3.8, <4",
 
     # This field lists other packages that your project depends on to run.
     # Any package you put here will be installed by pip when your project is
@@ -170,8 +108,8 @@
     # For an analysis of "install_requires" vs pip's requirements files see:
     # https://packaging.python.org/discussions/install-requires-vs-requirements/
     #
-    # Optional.
-    install_requires=["devicetree", "rich", "Pygments"],
+    # Requirements for both devicetree and dtsh.
+    install_requires=["PyYAML>=5.1", "rich", "Pygments"],
 
     # List additional groups of dependencies here (e.g. development
     # dependencies). Users will be able to install these using the "extras"
@@ -184,20 +122,17 @@
     #
     # Optional.
     extras_require={
+        "dev":  ["mypy", "types-PyYAML", "pyright", "pylint", "pytest"],
         "test": ["pytest"],
         "dist": ["build", "twine"],
     },
 
     # If there are data files included in your packages that need to be
     # installed, specify them here.
-    # package_data={  # Optional
-    #     "sample": ["package_data.dat"],
-    # },
     package_data={
          "dtsh": ["theme"],
     },
 
-
     # Although 'package_data' is the preferred approach, in some case you may
     # need to place data files outside of your packages. See:
     # http://docs.python.org/distutils/setupscript.html#installing-additional-files
@@ -212,10 +147,6 @@
     # `pip` to create the appropriate form of executable for the target
     # platform.
     #
-    # For example, the following would provide a command called `sample` which
-    # executes the function `main` from this package when invoked:
-    #
-    # Optional.
     entry_points={
         "console_scripts": [
             "dtsh=dtsh.cli:run",
@@ -227,12 +158,6 @@
     # This field corresponds to the "Project-URL" metadata fields:
     # https://packaging.python.org/specifications/core-metadata/#project-url-multiple-use
     #
-    # Examples listed include a pattern for specifying where the package tracks
-    # issues, where the source is hosted, where to say thanks to the package
-    # maintainers, and where to support the project financially. The key is
-    # what's used to render the link text on PyPI.
-    #
-    # Optional.
     project_urls={
         # 'Documentation': 'https://packaging.python.org/tutorials/distributing-packages/',
         "Bug Reports": "https://github.com/dottspina/dtsh/issues",