Move to ninja build system #2902
Comments
|
For anyone working on this, switch to the $ python3 tools/build/circuitpython.py feather_m0_express
$ ninja -C build-feather_m0_expressAt the minute it fails with: ...probably because it's not yet running the equivalent $ make -Bn -C ports/atmel-samd BOARD=feather_m0_express
...
: "GEN build-feather_m0_express/genhdr/qstrdefs.enum.h"
python3 ../../py/makeqstrdata.py build-feather_m0_express/genhdr/qstrdefs.preprocessed.h > build-feather_m0_express/genhdr/qstrdefs.enum.h
...Note that you can't run the above |
|
Yesterday I read through the short ninja build manual and wonder if it's worth using the Besides eliminating the need to specify the implicit header dependencies for an given C file using rule compile
command = arm-none-eabi-gcc -I../. -I../ports/atmel-samd -I../build-feather_m0_express -I../build-feather_m0_express/qstr -I../build-feather_m0_express/version -I../ports/atmel-samd/asf4/samd21 -I../ports/atmel-samd/asf4/samd21/CMSIS/Include -I../ports/atmel-samd/boards/feather_m0_express -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -D__ATSAMD21G18A__ -DSAMD21 -DSPI_FLASH_FILESYSTEM -DEXTERNAL_FLASH_DEVICE_COUNT=2 -DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" -c $in -o $out
build qstr/pp/main.c: qstr_src_pp ../main.c | version/genhdr/mpversion.hUsing rule compile
command = arm-none-eabi-gcc -MM -MF $out.d -I../. -I../ports/atmel-samd -I../build-feather_m0_express -I../build-feather_m0_express/qstr -I../build-feather_m0_express/version -I../ports/atmel-samd/asf4/samd21 -I../ports/atmel-samd/asf4/samd21/CMSIS/Include -I../ports/atmel-samd/boards/feather_m0_express -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -D__ATSAMD21G18A__ -DSAMD21 -DSPI_FLASH_FILESYSTEM -DEXTERNAL_FLASH_DEVICE_COUNT=2 -DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" -c $in -o $out
depfile = $out.d
build qstr/pp/main.c: qstr_src_pp ../main.cThis is what the |
|
I've been experimenting by directly modifying the build.ninja file and can get past the ninja_required_version = 1.1
# These variables are procedurally generated by tools/build/circuitpython.py
includes = $
-I.. $
-I../build-feather_m0_express $
-I../build-feather_m0_express/qstr $
-I../build-feather_m0_express/version $
-I../ports/atmel-samd $
-I../ports/atmel-samd/asf4/samd21 $
-I../ports/atmel-samd/asf4/samd21/CMSIS/Include $
-I../ports/atmel-samd/boards/build-feather_m0_express $
-I../ports/atmel-samd/boards/feather_m0_express
defines = $
-DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" $
-DEXTERNAL_FLASH_DEVICE_COUNT=2 $
-DFFCONF_H=\"lib/oofatfs/ffconf.h\" $
-DSAMD21 $
-DSPI_FLASH_FILESYSTEM $
-D__ATSAMD21G18A__
# These variables were reverse engineered from make output and may need to be
# merged with the procedurally generated variables above.
includes_pre = $
-I. $
-I../.. $
-I../../lib/tinyusb/src $
-I../../supervisor/shared/usb $
-I../lib/mp-readline $
-I../lib/timeutils $
-Iasf4/samd21 $
-Iasf4/samd21/CMSIS/Include $
-Iasf4/samd21/hal/include $
-Iasf4/samd21/hal/utils/include $
-Iasf4/samd21/hpl/core $
-Iasf4/samd21/hpl/gclk $
-Iasf4/samd21/hpl/pm $
-Iasf4/samd21/hpl/port $
-Iasf4/samd21/hpl/rtc $
-Iasf4/samd21/hpl/tc $
-Iasf4/samd21/hri $
-Iasf4/samd21/include $
-Iasf4_conf/samd21 $
-Iboards/ $
-Iboards/feather_m0_express $
-Ibuild-feather_m0_express $
-Ifreetouch $
-Iperipherals
defines_pre = $
-DCFG_TUD_CDC_RX_BUFSIZE=128 $
-DCFG_TUD_CDC_TX_BUFSIZE=128 $
-DCFG_TUD_MIDI_RX_BUFSIZE=128 $
-DCFG_TUD_MIDI_TX_BUFSIZE=128 $
-DCFG_TUD_MSC_BUFSIZE=512 $
-DCFG_TUSB_MCU=OPT_MCU_SAMD21 $
-DCIRCUITPY_ANALOGIO=1 $
-DCIRCUITPY_AUDIOBUSIO=1 $
-DCIRCUITPY_AUDIOCORE=1 $
-DCIRCUITPY_AUDIOIO=1 $
-DCIRCUITPY_AUDIOIO_COMPAT=1 $
-DCIRCUITPY_AUDIOPWMIO=0 $
-DCIRCUITPY_BITBANGIO=0 $
-DCIRCUITPY_BITBANG_APA102=0 $
-DCIRCUITPY_BLEIO=0 $
-DCIRCUITPY_BOARD=1 $
-DCIRCUITPY_BUSIO=1 $
-DCIRCUITPY_CANARY_WORD=0xADAF00 $
-DCIRCUITPY_DIGITALIO=1 $
-DCIRCUITPY_DISPLAYIO=1 $
-DCIRCUITPY_FREQUENCYIO=0 $
-DCIRCUITPY_FULL_BUILD=1 $
-DCIRCUITPY_GAMEPAD=1 $
-DCIRCUITPY_GAMEPADSHIFT=0 $
-DCIRCUITPY_I2CSLAVE=0 $
-DCIRCUITPY_MATH=1 $
-DCIRCUITPY_MICROCONTROLLER=1 $
-DCIRCUITPY_NEOPIXEL_WRITE=1 $
-DCIRCUITPY_NETWORK=0 $
-DCIRCUITPY_NVM=1 $
-DCIRCUITPY_OS=1 $
-DCIRCUITPY_PEW=0 $
-DCIRCUITPY_PIXELBUF=1 $
-DCIRCUITPY_PS2IO=0 $
-DCIRCUITPY_PULSEIO=1 $
-DCIRCUITPY_RANDOM=1 $
-DCIRCUITPY_ROTARYIO=1 $
-DCIRCUITPY_RTC=1 $
-DCIRCUITPY_SAFE_RESTART_WORD=0xDEADBEEF $
-DCIRCUITPY_SAMD=0 $
-DCIRCUITPY_SOFTWARE_SAFE_MODE=0x0ADABEEF $
-DCIRCUITPY_STAGE=0 $
-DCIRCUITPY_STORAGE=1 $
-DCIRCUITPY_STRUCT=1 $
-DCIRCUITPY_SUPERVISOR=1 $
-DCIRCUITPY_TIME=1 $
-DCIRCUITPY_TOUCHIO=1 $
-DCIRCUITPY_TOUCHIO_USE_NATIVE=1 $
-DCIRCUITPY_UHEAP=0 $
-DCIRCUITPY_USB_HID=1 $
-DCIRCUITPY_USB_MIDI=1 $
-DCIRCUITPY_USTACK=0 $
-DEXPRESS_BOARD $
-DEXTERNAL_FLASH_DEVICES="S25FL216K, GD25Q16C" $
-DEXTERNAL_FLASH_DEVICE_COUNT=2 $
-DFFCONF_H=$$"lib/oofatfs/ffconf.h$$" $
-DLONGINT_IMPL_MPZ -Os $
-DNDEBUG $
-DSAMD21 $
-DSPI_FLASH_FILESYSTEM=1 $
-DUSB_AVAILABLE $
-D__SAMD21G18A__
warnings = $
-Wall $
-Wcast-align $
-Wdouble-promotion $
-Werror $
-Werror-implicit-function-declaration $
-Wfloat-equal $
-Wmissing-format-attribute $
-Wnested-externs $
-Wno-deprecated-declarations $
-Wno-endif-labels $
-Wno-error=lto-type-mismatch $
-Wshadow $
-Wsign-compare $
-Wstrict-prototypes $
-Wundef $
-Wunreachable-code $
-Wwrite-strings
cflags_pre = $
--param max-inline-insns-single=500 $
-fdata-sections $
-ffunction-sections $
-finline-limit=60 $
-flto -flto-partition=none $
-fno-strict-aliasing $
-fshort-enums $
-fsingle-precision-constant $
-mabi=aapcs-linux $
-mcpu=cortex-m0plus $
-mfloat-abi=soft $
-msoft-float $
-mthumb $
-nostdlib $
-std=gnu11
rule compile
depfile = $out.d
command = arm-none-eabi-gcc -MM -MF $out.d $includes $defines -c $in -o $out
description = CC $out
rule gen-qstr-defs-pre
command = cat ../py/qstrdefs.h $in | $
sed 's/^Q(.*)/"&"/' | $
arm-none-eabi-gcc -E $includes_pre $defines_pre $warnings $cflags_pre - | $
sed 's/^"\(Q(.*)\)"/\1/' > $out
description = GEN $out
rule gen-qstr-data
command = python3 ../py/makeqstrdata.py $in > $out
description = GEN $out
rule gen-version-hdr
command = python3 ../py/makeversionhdr.py $out
description = GEN $out
build objs/main.o : compile ../main.c
build genhdr/qstrdefs.preprocessed.h : gen-qstr-defs-pre $
../ports/atmel-samd/qstrdefsport.h $
../ports/atmel-samd/build-feather_m0_express/genhdr/qstrdefs.collected.h
build genhdr/qstrdefs.enum.h : gen-qstr-data genhdr/qstrdefs.preprocessed.h
build version/genhdr/mpversion.h: gen-version-hdr |
I definitely want to use depfile for header dependencies. I don't think it'll work for mpversionhdr though because the file doesn't exist until built. I believe you need to be explicit for headers that are generated.
The sed stuff should definitely be merged into one of the QSTR build scripts. |
|
@omsai I pushed a new version that completes the QSTR stuff. Unfortunately, the next step is evolving the dynamic linkerscript generation that SAMD does. |
|
Zephyr's multiple repo & build management tool, west, may be useful. It manages both ninja & cmake builds, and has a nice clean way to manage multiple git repos. |
|
Thanks for the tip @durapensa! Can it do ninja without cmake? I don't really know want cmake to be a piece we use. |
|
@tannewt Peeking at the west docs, it uses cmake to generate ninja build files & board files. But since it's Python, that part can probably be replumbed. There's also a flashing component; could that be handy for priming/repairing boards for usb disk? |
|
@tannewt Can you again suggest some representative hardware target devices to focus this build system retooling effort? I don't mind buying several devices with bare minimum features if helps with quicker wins. A power outage damaged my Metro M4 Airlift which was plugged into a wall outlet at the time. Maybe a device that you're also using and that's still in production? |
Flashing is interesting but not very urgent. Most CircuitPython boards have UF2 bootloaders that are easy to manage separately. The main reason I'd like to rework our build system is to simplify configuration of different boards and generate a single build graph with all of the different configurations together. That will allow us to iterate towards sharing build artifacts across boards.
I want to caution against you taking this on. I have a lot of ideas about how I want to do it and don't want you to waste your time and money implementing something different than what I'm thinking. We could video chat about it all if you still want to try. Sorry to hear about your Metro! The ESP32-S2 metro is what I'd recommend as a replacement. |
|
Summary of the project discussion at Friday's Deep Dive video https://youtu.be/Vlxf8FRtV30?t=6418 for others following along this issue:
To build: board -> mcu sku -> soc -> cpu core Could put the flash things either under https://github.com/adafruit because it will be on adafruit time or it will go into https://github.com/baremetal-ninja There's a nice python library called tomlkit that explicitly preserves comments which would be handy for programmatic refactoring. |
|
@tannewt I can't make today's weekly meeting, but am interested in the next steps of ideas for milestones, a minimum viable product, and to pencil in dates for myself to meet those goals; I'm a fan of setting weekly goals with a few weeks sprinkled in as buffer. I feel like I have enough background now to work independently on some of these things, but in the call you mentioned maybe working on some of this during your week on Adafruit time so I thought it would make sense to coordinate more closely. Edit: I completely forgot to mention that it would be of particular interest to think about how tasks can be split up among multiple people / multiple GitHub issues because there were some enthusiastic people in the Deep Dive video call interested in contributing. |
|
Thanks for these notes @omsai! My plan with the flash is to do the flash toml bits, cascade toml and generators for it. I don't think I'll go so far as to do the boards. I'm not sure what a good small step for this is. One thing I'd do later is optimize across boards. For now we could just generate a giant ninja file to build everything separately. That still could be tricky because we likely want to factor out some of the compiler and board settings as we do that. This task feels like the initial work needs to be done by one person and once that skeleton is done, others can help it evolve. In particular, once we have cascade toml and the starts of the files, we can parallelize gathering info about more mcus/flashes. |
|
Ok WIP repos are here: https://github.com/adafruit/cascadetoml and https://github.com/adafruit/nvm.toml |
|
Two questions:
$ git clone -b ninja https://github.com/tannewt/circuitpython.git
$ cd circuitpython
$ git log --oneline origin/main..ninja
8d63932dd (HEAD -> ninja, origin/ninja) Fix qstr generation. Fails with linker script now
d2838e585 Merge remote-tracking branch 'adafruit/master' into ninja
55fb4f3f8 More qstr progress
8f66cf767 Build runs until needing qstr
$ git clone https://github.com/adafruit/cascadetoml.git
$ git clone https://github.com/adafruit/nvm.toml.git
$ cd cascadetoml
$ python3 cascadetoml.py check ../nvm.toml/
$ echo $?
0
$ python3 cascadetoml.py cascade ../nvm.toml/flash/gigadevice/GD25Q16C.toml cascade
(error below)Traceback (most recent call last):
File "cascadetoml.py", line 206, in <module>
app()
File "/home/omsai/.local/lib/python3.7/site-packages/typer/main.py", line 214, in __call__
return get_command(self)(*args, **kwargs)
File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/omsai/.local/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
return callback(**use_params) # type: ignore
File "cascadetoml.py", line 74, in cascade
if not parent.is_relative_to(root.parent):
AttributeError: 'PosixPath' object has no attribute 'is_relative_to' |
I'd probably start with the latest main and merge in the files from the old branch. I don't think there is much history there worth saving. I believe the QSTR stuff may have changed a bit. It's probably one of the larger things to move over.
That's one way to do it: # Data for path: flash/gigadevice/GD25Q16C.toml
# Data inferred from the path: {technology}/{manufacturer}/{sku}.toml
technology = "flash"
manufacturer = "gigadevice"
sku = "GD25Q16C"
# Data from flash/flash.toml
supports_fast_read = true
# Data from flash/gigadevice/gigadevice.toml
start_up_time_us = 5000
manufacturer_id = 0xc8
memory_type = 0x40
max_clock_speed_mhz = 104 # if we need 120 then we can turn on high performance mode
quad_enable_bit_mask = 0x02
has_sector_protection = false
supports_qspi = true
supports_qspi_writes = true
single_status_byte = false
# Data from flash/gigadevice/GD25Q16C.toml
# Settings for the Gigadevice GD25Q16C 2MiB SPI flash.
# Datasheet: http://www.gigadevice.com/datasheet/gd25q16c/
total_size = 0x200000 # 2 MiB
capacity = 0x15
write_status_register_split = falseThe other way is: |
|
I don't want to move to ninja anymore. Ideally it'd be an all-Python build system. No reason to have an issue for this work though. |
Prototype, WIP version of this is here: master...tannewt:ninja
The text was updated successfully, but these errors were encountered: