scripts: inline code generation with cogeno (and edts) #10885

b0661 · 2018-10-26T21:53:33Z

Overview

Extend Zephyr build system by inline code generation (cogeno) with Python snippets in source files. As most information for drivers is taken from the device tree an extended device tree database (edts) is added to allow easy retrieval with Python.

The PR is the sucessor to PR #6762. It includes all of #6762 and the relevant commits from #9876.

The PR is completely self contained. Besides the changes to the CMake build system to allow cogeno to be used as a script there are no other changes to the current code base.

Motivation for or Use Case

The prime motivation is to be able to configure drivers on build time by the information from the device tree in a standardized way. Therefor there must be means to easily generate structured data from device tree information at build time.

The EDTS database directly extracts the device tree information from the compiled DTS data. Inline code generation with Python snippets by cogeno is used to get this information and generate structured data to be compiled.

Design Details

Python snippets that are inlined in a source file are used as generators. The tool to scan the source file for the Python snippets and process them is cogeno. Cogeno is itself written in Python. Cogeno used Cog as it's starting point and has extended and rewritten it to provide specific generator functions.

The processing of source files is controlled by two CMake extension functions: zephyr_sources_cogeno(..) and zephyr_sources_cogeno_ifdef(..). During CMake configuration the source files are processed by cogeno and the generated source files are written to the CMake binary directory. The generated source files are added to the Zephyr sources.

The inlined Python snippets can contain any Python code, they are regular Python scripts. All Python snippets in a source file and all Python snippets of included template files are treated as a python script with a common set of global Python variables. Global data created in one snippet can be used in another snippet that is processed later on. This feature is e.g. used to customize included template files.

An inlined Python snippet can always access the cogeno module. The cogeno module encapsulates and provides all the functions to retrieve information (options, device tree properties, CMake variables, config properties) and to put out the generated code.

Test Strategy

A working proof of concept for driver instantiation is #10888 (Edit: Has to be updated).

For the EDTS database the sanity check script from #9876 is used with some adaptations.

PR History

This PR is the sucessor of several PRs:

drivers: pinctrl device interface (Linux-like) and STM32F0 driver #5799 'extract_dts_includes.py script'
- This PR is the source of codegen and the extended DTS database. Codegen was added to be able to generate device data structures for pin controllers at build time from DTS data. The EDTS database was added to provide a more consistent access to device tree info. To enable easier review PRs scripts: add Zephyr inline code generation with Python #6762 (codegen) and [WIP] scripts: extract_dts_includes.py: enhance #6761 (EDTS database/ DTS extraction) were split out. drivers: pinctrl device interface (Linux-like) and STM32F0 driver #5799 was also used as a test platform to prove the viability of inline code generation using a very complex driver - the pin controller driver for STM32F0. To have an easy review of the simpler prove of concept drivers also drivers: use codegen for device instantiation #9163 (Device instantiation) was split out.
[WIP] scripts: extract_dts_includes.py: enhance #6761 '[WIP] scripts: extract_dts_includes.py: enhance'
- This PR described a way to extract DTS info to the EDTS database using the extract_dts_includes.py script. It was superseded by script/dts: Generate Extended Device Tree database #9876 which still uses the extract_dts_includes.py script but improved the API of the extended DTS database that holds the DTS info after extraction.
scripts: add Zephyr inline code generation with Python #6762 'scripts: add Zephyr inline code generation with Python'
- This PR contained the inline code generation tool codegen. In the beginning codegen used the DTS info extraction from [WIP] scripts: extract_dts_includes.py: enhance #6761. Later on it used the sucessor script/dts: Generate Extended Device Tree database #9876.
drivers: use codegen for device instantiation #9163 'drivers: use codegen for device instantiation'
- As already descrived in this PR the easy instantiation of drivers using inline code generation was shown. It's pendant that fits to this PR is drivers: use codegen for device instantiation #10888.
script/dts: Generate Extended Device Tree database #9876 'script/dts: Generate Extended Device Tree database'
- This PR as the sucessor to [WIP] scripts: extract_dts_includes.py: enhance #6761 enhanced the EDTS database API and internal structure.
scripts: inline code generation with cogeno (and edts) #10885 - 2018: ''
- In this PR the codegen tool (scripts: add Zephyr inline code generation with Python #6762) and the extended DTS database (script/dts: Generate Extended Device Tree database #9876, [WIP] scripts: extract_dts_includes.py: enhance #6761) is re-integrated. The additions/ improvements that were done in script/dts: Generate Extended Device Tree database #9876 are also re-integrated. With the current PR the extraction of DTS info was made independent from the extract_dts_includes.py script to make this PR a self contained change that does not effect the current code base. The EDTS database now includes an internal DTS extraction function that extracts directly from the compiled DTS file.
scripts: inline code generation with cogeno (and edts) #10885 - NOW: ''
- Codegen transfered to cogeno, a standalone tool with it's own repository.

Signed-off-by: Bobby Noelte b0661n0e17e@gmail.com

dbkinder

Looks OK from a doc point of view, a couple of comments

doc/contribute/contribute_guidelines.rst

doc/subsystems/codegen/codegen.rst

b0661 · 2018-11-18T18:30:54Z

The codegen parts that are independent from Zephyr got their own repository https://gitlab.com/b0661/cogeno.

The inline code generation tool had to be renamed to cogeno because the name codegen is in use by several open source projects.

Cogeno is on PyPi now. You can install it by pip3 install cogeno. The documentation can now also be found on https://cogeno.readthedocs.io/en/latest/.

The Zephyr related documentation of cogeno is within the subsystems´s folder. Also the Zephyr specific Python modules and code generation templates are in Zephyrs templates` folder.

This is a big step to fulfill the requirements given in #10904 for code generation:

* User's acceptance/Ease of use:       
  * Code generation solution should be easy to read and use. User should identify main elements and understand behavior with minimum effort.
  * User should be able to easily map elements from the EDTS file to the code generation template

This is Zephyr specific. The related modules and templates are in Zephyr´s templates folder. Cogeno supports Python and Jinja2 scripting. Scripts may be inlined to source code as 'script snippets' or may be a complete file.

  * Ideally, syntax should be easy to edit in IDE

Cogeno scripts are in standard script languages. IDEs should easily identify complete script files. For inline code this will be different.

* Versatility:
  * Code generation solution should fit to most of zephyr drivers, and at least in a first iteration work with all drivers using `DEVICE_AND_API_INIT` macro.

Cogeno is as versatile as Python/ Jinja2. The solution for Zephyr can be implemented using Zephyr specific templates and modules.

* Ease of review:
  * Solution should not bring significant overhead to the review process. It should be constrained and sufficiently defined so we don't loose time to comment on various possible ways on using the solution for a particular driver.

Constraints are part of the Zephyr specfic templates.

* Maintainability:
  * Code generation provided should be easy to maintain.

Zephyr specifc stuff is in Zephyr. Generic code aka. cogeno is maintained and improved outside of Zephyr with the potential effect of contributions from other projects too.

* Doc:
  * Solution should be fully documented

Cogeno´s documentation is available online on Read the Docs. Zephyr specific documentation is in the Zephyr doc tree. There is still room to improve the documentation, but it already contains most of the basics.

* Implementation basics:
  * Input:    
    * EDTS Python library API (cf #9876, and zephyr/topic-EDTS branch) exclusively. No use of the `edts.json` directly in order to avoid reliance on the JSON formatting

The edtlib.py Python library is include in cogeno. After installation of cogeno it can also be used standalone as a command edtsdatabase.

    * Clean, non-templated `i2c_foo.c` file with the driver implementation (not used by the templating engine, here for completeness)
    * Single template file `i2c_foo_devices.h.in` with the templated device structures.
  * Output:         
    * Generated file : `i2c_foo_devices.h` in `build/zephyr/drivers/<driver>/`
    * Included in `i2c_foo.c`( `#include i2c_foo_devices.h`)

Zephyr specifc design rules for drivers.

  * Invocation: Solution should be invoked from one liner, simple cmake macro:
    `zephyr_library_sources_codegen_ifdef(CONFIG_I2C_STM32  i2c_ll_stm32.c)`

Cogeno integration in Zephyr is done by the codegen macros.

  * Rendering
    * Time: Needs to be rendered at `ninja` time
    * Performance: Needs to render files one by one in parallel

Cogeno can render concurrently if cogeno macros are invoked concurrently. Cogeno macros are invoked at Ninja time.

* Standalone:
  * The tooling to take a device tree & yaml, plus some template description should not be zephyr specific or zephyr aware (ie any invocation of the tools should be able to exist and run outside of Zephyr)

Cogeno takes the device tree & yaml, and bindings, and generates the EDTS database. Templates and the file pathes for the device tree are provided to cogeno by the build system. Cogeno is build system agnostic. Cogeno and the edtsdatabase are provided as commands and can be run outside of Zephyr.

  * The integration (cmake) and templates helpers can be zephyr aware.

The cogeno CMake macros and the device declare templates are within Zephyr.

Thanks
Bobby

Edit: codegen -> cogeno

Integrate code generation by cogeno into Zephyr build system. If a source file is added by zephyr_sources_cogeno(..) or zephyr_sources_cogeno_ifdef(..) cogeno will be executed and the generated file stored to the build directory. Compilation will run on the generated file only. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>

Document inline code generation with cogeno. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>

Not in project anymore

codecov-commenter · 2020-07-03T17:49:32Z

Codecov Report

Merging #10885 into master will decrease coverage by 4.17%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #10885      +/-   ##
==========================================
- Coverage   52.55%   48.38%   -4.18%     
==========================================
  Files         340      265      -75     
  Lines       49341    42193    -7148     
  Branches    11463    10137    -1326     
==========================================
- Hits        25929    20413    -5516     
+ Misses      18152    17703     -449     
+ Partials     5260     4077    -1183

Impacted Files	Coverage Δ
subsys/bluetooth/host/keys.h	`0.00% <0.00%> (-100.00%)`	⬇️
boards/posix/nrf52_bsim/k_busy_wait.c	`0.00% <0.00%> (-100.00%)`	⬇️
subsys/net/lib/sockets/sockets_internal.h	`28.57% <0.00%> (-71.43%)`	⬇️
drivers/console/native_posix_console.c	`22.36% <0.00%> (-50.36%)`	⬇️
subsys/usb/usb_descriptor.c	`0.00% <0.00%> (-45.57%)`	⬇️
subsys/net/ip/dhcpv4.c	`6.16% <0.00%> (-35.36%)`	⬇️
subsys/bluetooth/host/crypto.c	`0.00% <0.00%> (-32.15%)`	⬇️
subsys/bluetooth/host/keys.c	`10.22% <0.00%> (-24.19%)`	⬇️
subsys/net/ip/ipv6_fragment.c	`25.58% <0.00%> (-20.79%)`	⬇️
subsys/net/ip/tcp_internal.h	`48.48% <0.00%> (-15.52%)`	⬇️
... and 343 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5372e78...f536020. Read the comment docs.

b0661 · 2020-09-08T17:38:43Z

Cogeno does not need to be part of Zephyr anymore.

Inline code generation with Cogeno is now available as a Zephyr module.

The repository is here: https://gitlab.com/b0661/cogeno
The documentation may be read online at https://cogeno.readthedocs.io/en/latest/index.html.
A python package is available on PyPI: https://pypi.org/project/cogeno/

For integration into the Zephyr build system see https://cogeno.readthedocs.io/en/latest/cogeno/build.html#build-with-zephyr-and-cogeno.

b0661 requested review from dbkinder, galak, MaureenHelm, nashif and SebastianBoe as code owners October 26, 2018 21:53

This comment has been minimized.

Sign in to view

b0661 force-pushed the pr_codegen branch from c2b2987 to d98df0a Compare October 27, 2018 15:14

b0661 mentioned this pull request Oct 27, 2018

drivers: use codegen for device instantiation #10888

Closed

b0661 force-pushed the pr_codegen branch from d98df0a to 46256d0 Compare October 27, 2018 17:19

b0661 mentioned this pull request Oct 27, 2018

script/dts: Generate Extended Device Tree database #9876

Closed

b0661 force-pushed the pr_codegen branch 2 times, most recently from f80d1b5 to d51ad45 Compare October 28, 2018 08:42

galak added the area: Devicetree label Oct 28, 2018

b0661 force-pushed the pr_codegen branch from d51ad45 to 91f82dd Compare October 28, 2018 18:26

carlescufi requested a review from evenl October 29, 2018 12:51

b0661 mentioned this pull request Oct 30, 2018

Requirements for driver devices generation using device tree #10904

Closed

dbkinder previously requested changes Nov 1, 2018

View reviewed changes

doc/contribute/contribute_guidelines.rst Outdated Show resolved Hide resolved

doc/subsystems/codegen/codegen.rst Outdated Show resolved Hide resolved

SebastianBoe removed their request for review November 13, 2018 13:49

carlescufi mentioned this pull request Nov 13, 2018

Device Tree support overhaul #8499

Closed

6 tasks

b0661 force-pushed the pr_codegen branch from 91f82dd to 370c241 Compare November 18, 2018 17:10

b0661 force-pushed the pr_codegen branch 2 times, most recently from c166b28 to e800b6b Compare November 20, 2018 12:30

b0661 force-pushed the pr_codegen branch from e800b6b to a34249b Compare November 26, 2018 15:00

This comment has been minimized.

Sign in to view

b0661 mentioned this pull request Nov 27, 2018

ci: 'MIT' license verification fail on permissive license #11671

Closed

b0661 force-pushed the pr_codegen branch from a34249b to 3f4c9c0 Compare November 27, 2018 08:45

galak added area: Devicetree Tooling PR modifies or adds a Device Tree tooling and removed area: Devicetree labels Jan 30, 2020

tbursztyka mentioned this pull request Feb 19, 2020

Refining Zephyr's Device Driver Model - Take II #22941

Open

13 tasks

b0661 added 2 commits March 1, 2020 19:48

doc: add inline code generation documentation

f536020

Document inline code generation with cogeno. Signed-off-by: Bobby Noelte <b0661n0e17e@gmail.com>

b0661 force-pushed the pr_codegen branch from 3f4c9c0 to f536020 Compare March 1, 2020 19:09

zephyrbot added area: Documentation area: Build System labels Mar 1, 2020

b0661 changed the title ~~scripts: codegen with edts - inline code generation~~ scripts: inline code generation with cogeno (and edts) Mar 1, 2020

github-actions bot added has-conflicts Issue/PR has conflicts with another issue/PR and removed has-conflicts Issue/PR has conflicts with another issue/PR labels Jun 29, 2020

nashif added the Stale label Sep 4, 2020

b0661 closed this Sep 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scripts: inline code generation with cogeno (and edts) #10885

scripts: inline code generation with cogeno (and edts) #10885

b0661 commented Oct 26, 2018 •

edited

Loading

This comment has been minimized.

dbkinder left a comment

b0661 commented Nov 18, 2018 •

edited

Loading

This comment has been minimized.

codecov-commenter commented Jul 3, 2020

b0661 commented Sep 8, 2020

scripts: inline code generation with cogeno (and edts) #10885

scripts: inline code generation with cogeno (and edts) #10885

Conversation

b0661 commented Oct 26, 2018 • edited Loading

Overview

Motivation for or Use Case

Design Details

Test Strategy

PR History

This comment has been minimized.

dbkinder left a comment

Choose a reason for hiding this comment

b0661 commented Nov 18, 2018 • edited Loading

This comment has been minimized.

codecov-commenter commented Jul 3, 2020

Codecov Report

b0661 commented Sep 8, 2020

b0661 commented Oct 26, 2018 •

edited

Loading

b0661 commented Nov 18, 2018 •

edited

Loading