MCIMX93-EVK Board Support Package

1. About this document
======================

This document describes common and non-hardware specific information.
Please refer to README.hardware for hardware specific information.

Dependencies
------------
This layer depends on the following layers supplied with Wind River Linux:
  oe-core
  meta-imx
  meta-freescale

Maintenance
-----------
This layer is maintained by Wind River Systems, Inc.
Contact <support@windriver.com> or your support representative for more
information on submitting changes.

Building the nxp-imx9 layer
---------------------------
This layer and wr-kernel layer should be added to bblayers.conf. This
is done automatically when using the Wind River configure wrapper.

License
-------
Copyright (C) 2024 Wind River Systems, Inc.

Source code included in the tree for individual recipes is under the LICENSE
stated in the associated recipe (.bb file) unless otherwise stated.

The metadata is under the following license unless otherwise stated.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

2. BSP Kernel and Distros
=========================

The following table summarizes the valid Wind River Linux distros for this BSP.
'Y' in each content cell stands for supported; 'N' stands for not supported:

  +--------------+-------------+------------------+-------------------+--------------+
  | valid/distro |   wrlinux   | wrlinux-graphics | wrlinux-installer | wrlinux-tiny |
  +--------------+-------------+------------------+-------------------+--------------+
  |    valid     |      Y      |        Y         |         N         |       N      |
  +--------------+-------------+------------------+-------------------+--------------+

For the supported kernel type for this BSP, please check the TARGET_SUPPORTED_KTYPES
by running 'bitbake -e virtual/kernel | grep "^TARGET_SUPPORTED_KTYPES="'.

Note: The preempt-rt ktype can be used for this BSP/Machine.


3. Board Specific Patches
=========================
To get a list of patches applied to the kernel specific to this BSP,
along with patch descriptions, use git to see what changed on the default
kernel (git whatchanged <kernel_type>..<bsp_name>). For example:

	# cd tmp-glibc/work-shared/nxp-imx9/kernel-source
	# git whatchanged standard/base..


4. Boot Instructions
====================

4.1 NFS Root File System
------------------------

4.1.1 Prepare Kernel, DTB, NFS and TFTP servers
-----------------------------------------------
The files in the following example can be found in <buildprj>/tmp-glibc/deploy/images/nxp-imx9
and are copied to the export directory of the TFTP server.

4.1.2 Download kernel
---------------------

	# tftp $kernel_addr Image

4.1.3 Download DTB
------------------
For MCIMX93-EVK board:
	tftp $fdt_addr imx93-11x11-evk.dtb

	If you want to connect LVDS panel:
	tftp $fdt_addr imx93-11x11-evk-boe-wxga-lvds-panel.dtb

	If you want to use lpspi master:
	tftp $fdt_addr imx93-11x11-evk-lpspi.dtb

	If you want to use i3c:
	tftp $fdt_addr imx93-11x11-evk-i3c.dtb

	If you want to use MX93AUD-HAT:
	tftp $fdt_addr imx93-11x11-evk-aud-hat.dtb

	If you want to use flexspi:
	tftp $fdt_addr imx93-11x11-evk-flexspi-m2.dtb

	If you want to use lpuart:
	tftp $fdt_addr imx93-11x11-evk-lpuart.dtb

4.1.5 Set bootargs and boot the system
--------------------------------------

	# setenv bootargs console=<$console_port> root=/dev/nfs rw \
		 nfsroot=<nfs server>:<nfs root> \
		 ip=<target IP>::<gateway>:<netmask>::eth0:off \
		 video=HDMI-A-1:1920x1080-32@60
	# booti $kernel_addr - $fdt_addr

Note: for MCIMX93-EVK board, $kernel_addr="0x80400000", $fdt_addr="0x83000000"
	$console_port = "ttyLP0,115200"
      The kernel image will become larger if more kernel code is compiled into kernel image.
      So, please increase "fdt_addr" value by putting dtb file at higher address if you encounter with kernel booting failed.


5. Features
===========

5.1 HDMI
--------

To enable HDMI as the default framebuffer device, add:

	video=HDMI-A-1:1920x1080-32@60

to the bootargs parameter of U-Boot.

5.2 USB feature
---------------

5.2.1 USB OTG Usage and verification
------------------------------------

1) Verify the OTG device mode

Plug in cable B and connect to the host machine.

Board configuration:

	# ifconfig usb0 192.168.1.10

Host machine configuration:

	# sudo ifconfig usb0 192.168.1.100
	# ping 192.168.1.10 -c 2
	PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.
	64 bytes from 192.168.1.10: icmp_req=1 ttl=64 time=2.06 ms
	64 bytes from 192.168.1.10: icmp_req=2 ttl=64 time=0.286 ms
	...

2) Switch to OTG host mode

Replace cable B with cable A.

5.3 Static PM
-------------

The debug UART can be set as a wakeup source with:

For MCIMX93-EVK board:
	# echo enabled > /sys/class/tty/ttyLP0/power/wakeup
	# echo mem > /sys/power/state

5.4 Graphics
------------

5.4.1 Xwayland
--------------
If you want to build a xwayland image, please follow the steps as below:

<1> Setup program and create a build directory by using the following arguments:

	--machines nxp-imx9 --distros wrlinux-graphics --dl-layers

<2> Setup environment and create a build directory:

	$ . ./environment-setup-<host>-wrlinuxsdk-linux
	$ . ./oe-init-build-env <build>

<3> Use the command as below to build a xwayland image:

	bitbake wrlinux-image-std-sato

5.5 thermal
-----------

	To check current CPU temperature
	# cd /sys/class/thermal/thermal_zone0
	# cat temp
	51000
	# cat trip_point_0_type
	passive
	# cat trip_point_0_temp
	85000
	# cat trip_point_1_type
	critical
	# cat trip_point_1_temp
	95000

5.6 Optee-test
--------------

Because optee os is integrated into the bootloader image, and due to license reasons,
we don't support building the bootloader by default. Therefore, before using optee
features, you need to refer to README.proprietary ("Build imx-boot") to compile an WIC image
with the bootloader.

1) Burn WIC image
# bitbake wrlinux-image-std
# dd if=wrlinux-image-<std-sato/std>-nxp-imx8.wic of=/dev/your_sd_dev

2) run optee testing sample 

# xtest -l15

5.7 MX93AUD-HAT
---------------

Attach the MX93AUD-HAT to MCIMX93-EVK J1001, select imx93-11x11-evk-aud-hat.dtb to boot,
plug the headset into the MX93AUD-HAT output J10 and utilize the imxcs42448 for playback

root@nxp-imx9:~# aplay -l
card 3: imxcs42448 [imx-cs42448], device 0: cs42448 cs42448-0 [cs42448 cs42448-0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0

aplay -Dhw:3,0 stereo1648s.wav

5.8 FLEXCAN
-----------

Detail verification commands as below:

Currently, imx93 J1101 is a can interface, 1-4 are 5V, CAN_H, CAN_L, and GND
respectively. Please use a board that supports canfd function to connect, such as imx8qm.

imx93:
        # ip link set can0 up type can bitrate 500000 dbitrate 8000000 fd on
	# candump can0 &
	# ip -d -s link show can0
imx8qm:
        # ip link set can1 up type can bitrate 500000 dbitrate 8000000 fd on
        # cansend can1 5A1#11.22.33.44.55.66.77.88
        # ip -d -s link show can1

If can0 dumps data from can1, FLEXCAN feature works fine.

5.9 TSN
-------

Use eth1 of imx8mp to connect to eth1 of imx93

Add preparations before testing:

Load the following modules into the kernel.
 sch_multiq.ko sch_cbs.ko sch_etf.ko sch_taprio.ko sch_mqprio.ko sch_ingress.ko act_skbedit.ko pktgen.ko

1) Clock synchronization:

Run ptp4l on two boards:
ptp4l -i eth1 -p /dev/ptp1 -m -2

ptp4l[1732.242]: master offset       -120 s2 freq   +1004 path delay       648
ptp4l[1733.242]: master offset        -98 s2 freq    +990 path delay       730
ptp4l[1734.242]: master offset         38 s2 freq   +1096 path delay       730
ptp4l[1735.242]: master offset         24 s2 freq   +1094 path delay       764
ptp4l[1736.242]: master offset         45 s2 freq   +1122 path delay       763
ptp4l[1737.242]: master offset         50 s2 freq   +1140 path delay       762
ptp4l[1738.243]: master offset         26 s2 freq   +1131 path delay       762
ptp4l[1739.243]: master offset          8 s2 freq   +1121 path delay       764
ptp4l[1740.243]: master offset         -5 s2 freq   +1110 path delay       765
ptp4l[1741.243]: master offset        -17 s2 freq   +1097 path delay       769
ptp4l[1742.243]: master offset        -10 s2 freq   +1099 path delay       770
ptp4l[1743.243]: master offset         -5 s2 freq   +1101 path delay       769
ptp4l[1744.243]: master offset         -5 s2 freq   +1099 path delay       769
ptp4l[1745.243]: master offset        -18 s2 freq   +1085 path delay       770
ptp4l[1746.244]: master offset         18 s2 freq   +1115 path delay       770
ptp4l[1747.244]: master offset         10 s2 freq   +1113 path delay       770

2) Qav

imx93evk:

<1> Set a queue map handle:
 tc qdisc add dev eth1 root handle 1: mqprio num_tc 5 map 0 1 2 3 4 queues 1@0 1@1 1@2 1@3 1@4 hw 0
<2> Set bandwidth of queue 3 to be 20 Mbps: 
 tc qdisc replace dev eth1 parent 1:4 cbs locredit -1470 hicredit 30 sendslope -980000 idleslope 20000 offload 1
<3> Send a stream into queue 3:
 /home/root/samples/pktgen/pktgen_sample01_simple.sh -i eth1 -q 3 -s 500 -n 3000

Get the result, bandwidth is 19 Mbps.

root@nxp-imx9:~# /home/root/samples/pktgen/pktgen_sample01_simple.sh -i eth1 -q 3 -s 500 -n 3000
WARN : Missing destination MAC address
WARN : Missing destination IP address
Running... ctrl^C to stop
Done
Result device: eth1
Params: count 3000  min_pkt_size: 500  max_pkt_size: 500
     frags: 0  delay: 0  clone_skb: 0  ifname: eth1
     flows: 0 flowlen: 0
     queue_map_min: 3  queue_map_max: 3
     dst_min: 198.18.0.42  dst_max: 
     src_min:   src_max: 
     src_mac: 00:04:9f:08:78:b7 dst_mac: 02:5d:ae:ba:e0:00
     udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
     src_mac_count: 0  dst_mac_count: 0
     Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_RND  
Current:
     pkts-sofar: 3000  errors: 0
     started: 2089376122us  stopped: 2089994023us idle: 85250us
     seq_num: 3001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
     cur_saddr: 192.168.3.1  cur_daddr: 198.18.0.42
     cur_udp_dst: 9  cur_udp_src: 92
     cur_queue_map: 3
     flows: 0
Result: OK: 617900(c532650+d85250) usec, 3000 (500byte,0frags)
  4855pps 19Mb/sec (19420000bps) errors: 0

<4> Set bandwidth of queue 4 to be 40 Mbps:
 tc qdisc replace dev eth1 parent 1:5 cbs locredit -1440 hicredit 60 sendslope -960000 idleslope 40000 offload 1
<5> Send a stream into queue 4 and get the result:
root@nxp-imx9:~# /home/root/samples/pktgen/pktgen_sample01_simple.sh -i eth1 -q 4 -s 500 -n 3000
WARN : Missing destination MAC address
WARN : Missing destination IP address
Running... ctrl^C to stop
Done
Result device: eth1
Params: count 3000  min_pkt_size: 500  max_pkt_size: 500
     frags: 0  delay: 0  clone_skb: 0  ifname: eth1
     flows: 0 flowlen: 0
     queue_map_min: 4  queue_map_max: 4
     dst_min: 198.18.0.42  dst_max: 
     src_min:   src_max: 
     src_mac: 00:04:9f:08:78:b7 dst_mac: 02:5d:ae:ba:e0:00
     udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
     src_mac_count: 0  dst_mac_count: 0
     Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_RND  
Current:
     pkts-sofar: 3000  errors: 0
     started: 2291228139us  stopped: 2291536099us idle: 44961us
     seq_num: 3001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
     cur_saddr: 192.168.3.1  cur_daddr: 198.18.0.42
     cur_udp_dst: 9  cur_udp_src: 72
     cur_queue_map: 4
     flows: 0
Result: OK: 307960(c262999+d44961) usec, 3000 (500byte,0frags)
  9741pps 38Mb/sec (38964000bps) errors: 0

<6> Send two streams into queue 3 and queue 4 using the command below:
 /home/root/samples/pktgen/pktgen_twoqueue.sh -i eth1 -q 3 -s 1500 -n 0

imx8mp:

<1> use tcpdump to get tsn.pcap:
 tcpdump -i eth1 -e -n -t -xx -c 3000 -w tsn.pcap
<2> the frames sort by one Q3 frame and two Q4 frames:
 Use wireshark to open tsn.pcap and you will find the frames sorted by one Q3 frame and two Q4 frames

3) Qbv

<1>Enable the ptp device, and get the current ptp time:
 run 'ptp4l -i eth1 -p /dev/ptp1 -m' on two boad
 
<2> get current time on imx93evk:
 devmem2 0x428a0b08

<3> Get the basetime to be 2 minutes later:
 #Basetime = (currentime + 120) * 1000000000

<4> Set time schedule, open queue 1 in 100 µs and open queue 2 in 100 µs on imx93:
 tc qdisc replace dev eth1 parent root handle 100 taprio \
 num_tc 5 map 0 1 2 3 4 queues 1@0 1@1 1@2 1@3 1@4 base-time $Basetime \
 sched-entry S 1 100000 \
 sched-entry S 2 100000 \
 sched-entry S 4 100000 flags 2

<5> Capture the streams on imx8mp:
 tcpdump -i eth1 -e -n -t -xx -c 3000 -w tsn.pcap

<6> Send two streams into queue 1 and queue 2 on imx93:
 /home/root/samples/pktgen/pktgen_twoqueue.sh -i eth1 -q 1 -s 1000 -n 0 -m 90:e2:ba:ff:ff:ff

Use wireshark to open tsn.pcap, 100 µs queue 1 frames (length=1004) and 100 µs queue 2 frames
(length=1504) will be got.

4) Qbu

Since verifying the Qbu test case requires reading the Start mPacket Delimiter (SMD)
of the Ethernet frame to determine its type (e.g., the SMD value of an express message
is 0xd5, verify message is 0x7, respond message is 0x19), obtaining the SMD value
typically requires specialized network equipment like TestCenter. However, we can validate
Ethernet Frame Preamble using the following simple method:

Connect eth1 of imx8mp and imx93 boards back to back. Then,
run below command to enable the hardware verification:

 ethtool --set-frame-preemption eth1 fp on preemptible-queues-mask 0x04 min-frag-size 60

imx93 can see:
    imx-dwmac 428a0000.ethernet eth1: FPE: Verify mPacket is transmitted
    imx-dwmac 428a0000.ethernet eth1: Send Verify mPacket lo_state=1 lp_state=0

imx8mp can see:
    imx-dwmac 30bf0000.ethernet eth1: FPE: Verify mPacket is received

5.10 camera
-----------

1) Download the "ap1302_60fps_ar0144_27M_2Lane_awb_tuning.bin" file from
https://github.com/ONSemiconductor/ap1302_binaries/tree/main/NXP_i.MX93

2) Rename it to "ap1302.fw"

3) After booting the Linux OS, copy the file to the target board under "/lib/firmware/imx/camera" and reboot.

5.11 Low Drive Mode
-------------------

The Linux kernel configures the system to run in Overdrive mode (ODM) by default.
This Overdrive mode and associated performance are not necessary for some user
applications. Users can transition to Nominal or Low-drive mode as defined in
the i.MX 93 data sheet instead.

You can switch to Low Drive Mode with below command:

echo 1 > /sys/devices/platform/imx93-lpm/mode
System switching to ND mode...

cat /sys/devices/platform/imx93-lpm/mode

6. kexec/kdump
==============

For discussion purposes, some useful terminology will be described here.

* boot kernel - the first kernel that you start and supports kexec, from U-Boot
	      for instance
* capture kernel - the kernel that you reboot into after a boot kernel crash

To build the boot kernel, enable kexec and kdump in your local.conf file:

	WRTEMPLATE = "feature/kexec feature/kdump"

6.1 kdump
---------

For the boot kernel:
To reserve a memory region for the capture kernel, you need to pass the bootargs
argument "crashkernel" to the boot kernel as follow:

    crashkernel=512M

For the dump-capture kernel:
Before booting the dump-capture kernel, add "maxcpus=1 cma=128M" to the boot parameter.

NOTE: Use Image as a secondary kernel. It can be found in the
tmp-glibc/deploy/images/nxp-imx9/ directory.

For more detailed info about kdump, refer to Documentation/kdump/kdump.txt
in the kernel source tree.

7. Creating Partitioned Images(WIC)
===================================

User can use the OpenEmbedded Image Creator, wic, to create the properly
partitioned image on a SD card. The wic command
generates partitioned images from existing OpenEmbedded build artifacts.
User can refer to the below URL to get more WIC details:

https://docs.yoctoproject.org/singleindex.html#creating-partitioned-images-using-wic

This BSP supports disk images for SD card.
After build the project, user will get a WIC image under the directory
tmp/deploy/images/<bsp name>/ ,such as:

tmp-glibc/deploy/images/nxp-imx9/wrlinux-image-<std-sato/std>-nxp-imx9.wic

Then user can write the output image to a SD card.

7.1 Burn images to SD card
--------------------------

1) Burn WIC image

# dd if=wrlinux-image-<std-sato/std>-nxp-imx9.wic of=/dev/your_sd_dev

2) Set environment variable for booting system

For MCIMX93-EVK board:

u-boot=>setenv fdt_file imx93-11x11-evk.dtb
u-boot=>setenv bootfile Image
u-boot=>setenv bootargs 'root=/dev/mmcblk1p2 rw rootdelay=5 console=${console},${baudrate} earlycon ip=dhcp'
u-boot=>setenv bootcmd 'fatload mmc 1:1 $loadaddr $bootfile; fatload mmc 1:1 $fdt_addr $fdt_file; booti $loadaddr - $fdt_addr';
u-boot=>run bootcmd

8. USB OTG
==========

This section only describs how to verify usb otg feature on MCIMX93-EVK board.

8.1 Host Mode
-------------

Connect usb disk with usb typec port(J17) on MCIMX93-EVK board.
Usb disk is detected, and then confirm the speed with command as below:

	# lsusb -t
	/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ci_hdrc/1p, 480M
	    |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 480M