Status (part 2)

[luc-github]

This discussion is the following of #1
It should be about :

• 3D Printer support (done for Marlin👍 )
• GCode streaming implementation
• cnc UI / backend implementation
• code refactoring
• hardware support

[luc-github]

Some videos to show latest update - sorry for the quality
I connected my HMI V3 to my ender 5 plus using usb port ( not more need serial hacking 😸 )

Connection.mp4

The manual bed leveling:

manualbedleveling.mp4

The Auto bed leveling probing:

Autobedleveling.mp4

[3DSmitty]

Love it!

[luc-github]

I had some hard time with the table widget that does not like to be updated in same time as others widgets, some race conditions in refresh I guess, but I finaly found a workaround, and could update the table and show some text update for the context

[luc-github]

Thanks to @serisman optimization esp32 2.8 TFT without PSRAM is now working as expected using current FW, UI has been updated accordingly, so this TFT is no more deprecated (CMakeLists.txt and esp3d.io have been updated)
that said 2.8 screen is really small and PSRAM is still strongly recommended for performance improvement, so no PSRAM is currently ok because screen is small

[serisman]

The 3.5" 480x320 TFT without PSRAM (with either capacitive or resistive touch screen) is also fully working as expected using current FW. It has been working well enough that I haven't even tried the PSRAM hardware mod on them yet.

[3DSmitty]

Thank You @serisman !

[luc-github]

I have added a target system emulator: fw_simulator.py to test the screen without connect it to real printer/cnc, it currently simulate temperatures / jog / bed leveling/fan and feedrate

It just need to be launched in terminal, it will display the complete data exchange between emulator and TFT on terminal.

I have currently limited to existing command used by TFT and it is based on Marlin but will be extended to others commands and others FW (repetier, smoothieware, grbl, etc...) when necessary
I could use the Native Marlin Emulator but because it is necessary to emulate others FW to test easily the TFT, I jumped directly to a python emulator for all
The port detection is in theory automatic but I only tested it on Windows 11, so feedback on OSX /Linux are welcome.

[luc-github]

with colors to differenciate in and out it is more readable:

[luc-github]

I am actively working on Gcode Host streaming integration from @jamespearson04 code, I hope this to be done for next week because I am not every day on it but I did lot of progress so far, I am on 50% of the first pass which is pure code integration / refactoring / simplification / etc..., the next pass will be code review with test, and 3rd and last pass will be final code review + UI integration+ long test run
So far so good - nothing is blocking

[luc-github]

Quick update GCode streaming Integration is done for Marlin and under review by @jamespearson04
During this time I have also worked on Time support in TFT and WebDav Support, both are a rewrite from scratch and not a porting from ESP3D because I took time to study them carefully and found several gliches / not necessary code and some mess in algorythms (from me 😓 )
So a rewrite from scratch was necessary - when I tried to do minimum features for webdav, it was ok on Filezilla Pro, which support WebDav but Windows did not recognized the xml, for some reason namespaces are mandatory to be used by window when in RFC they are just recommanded, and even not used on some examples...took some trial to figure out what part is the issue, same for date format which supposed to be ok with this format :
<D:creationdate>1997-12-01T17:42:21-08:00</D:creationdate>
but filezilla pro and window actualy need to be in this format:
<esp3d:getlastmodified>Thu, 01 Jan 1970 00:04:34 GMT</esp3d:getlastmodified>

But I have learn a lot ^_^ so it is ok

Time module is already done, and WebDav module still miss 2 methods to be implemented : COPY and MOVE, which should be done this week hopefully, so I can push a PR.

This also means once done, I need to go back to ESP3D V3 for a couple of weeks to fix the mess I did in Time module and simplify WebDav Module...orz 🙈 🙉 🙊

[luc-github]

PR pushed and feature list updated

[luc-github]

Ho boy - I studied Webdav and took assumption that following Dav 1 specs, no PROPPATCH / LOCK/ UNLOCK would fit ESP3D-TFT needs
And that is true - on Filezilla Pro - upload a file use PUT and that is all.
Also I was assuming because in Header the server annnounce it only Allow: PUT, GET, HEAD,DELETE, COPY, MOVE, PROPFIND, MKCOL, OPTIONS and also announcing DAV:1 the client will follow the directive, like Filezilla Pro do.

Well Window do not care and use LOCK/ UNLOCK/ PROPPATCH even server say not allowed, I should suspect that with inital Date issue, but I am too naive ...orz

So I will have to add some support for PROPPATCH / LOCK / UNLOCK but not announce them in Allow and try to keep Dav 1 protocol (TBC) because using Lock mechanism make every upload very chatty and unnecessary slow down the communication with ESP3D

Back to business ^_^

[luc-github]

I ended simulating PROPPATCH / LOCK / UNLOCK, and so far so good, now drag and drop a file to webdav in window works as expected 👍

[luc-github]

I am currently switching to ESP3D code to port all improvements done in ESP3D-TFT
I have added the debug level same as ESP3D-TFT
I have already fixed the Time module and Webdav Module, which allowed me to find some error 1dd9ead, and I have updated doc accordingly https://github.com/luc-github/ESP3D-TFT/blob/main/docs/WebDavService.md which actually should be now pushed to esp3d.io...

I am now on Settings part, that include setttings validation which is better in ESP3D-TFT and rewrite of ESP400 in ESP3D
I am also working on documentation of each ESPXXX commands of ESP3D - once done it will be pushed to esp3d.io and I will then do same for ESPXXX command for ESP3D-TFT
I need also to port the Sticky authentication feature of ESP3D-TFT in serial/telnet/websocket

So I still need a couple of week I think before going back to ESP3D-TFT as the above list is time consuming and I hate to write docs ... but now there are ~100 ESPXXX commands and some vary depending project so it is important to make things clean

Once all done I will start the CNC UI which should be the main screen and probe pages also and any additionnal page that would be necessary once design is defined because I do not have any in mind for the moment => suggestion are always welome

if any major issue on current code I will switch back and fix it ASAP of course

[luc-github]

Implementation of new messenging API and porting the sticky authentication to ESP3D make me to write a quick doc to sum up authentication: https://github.com/luc-github/ESP3D/blob/Refactoring-internal-clients/docs/authentication.md that will be added to http://esp3d.io

So currently user and admin have same level, I need to update that part and I will sync with same on ESP3D-TFT as well as ESP command description documents done for ESP3D V3 : https://esp3d.io/esp3d/v3.x/documentation/commands/index.html

Here the scope of right for each authentication level for ESP3D, ESP3D-TFT should have almost same commands:

Feature	not authenticated	guest	user	admin
Web Interface	No	No	Yes	Yes
Telnet	No	No	Yes	Yes
WebSocket	No	No	Yes	Yes
WebDav	No	No	Yes	Yes
Bluetooth	No	No	Yes	Yes
ESP0	Yes	Yes	Yes	Yes
ESP100	No	No	Get/Set	Get/Set
ESP101	No	No	Set	Set
ESP102	No	No	Get/Set	Get/Set
ESP103	No	No	Get/Set	Get/Set
ESP104	No	No	Get/Set	Get/Set
ESP105	No	No	Get/Set	Get/Set
ESP106	No	No	Set	Set
ESP107	No	No	Get/Set	Get/Set
ESP108	No	No	Get/Set	Get/Set
ESP110	No	No	Get/Set	Get/Set
ESP111	No	No	Get	Get
ESP112	No	No	Get/Set	Get/Set
ESP114	No	No	Get/Set	Get/Set
ESP115	No	No	Get/Set	Get/Set
ESP120	No	No	Get/Set	Get/Set
ESP121	No	No	Get/Set	Get/Set
ESP130	No	No	Get/Set	Get/Set
ESP131	No	No	Get/Set	Get/Set
ESP140	No	No	Get/Set	Get/Set
ESP150	No	No	Get/Set	Get/Set
ESP160	No	No	Get/Set	Get/Set
ESP161	No	No	Get/Set	Get/Set
ESP170	No	No	Get/Set	Get/Set
ESP171	No	No	Get	Get
ESP180	No	No	Get/Set	Get/Set
ESP181	No	No	Get/Set	Get/Set
ESP190	No	No	Get/Set	Get/Set
ESP191	No	No	Get/Set	Get/Set
ESP200	No	No	Get/Set	Get/Set
ESP201	No	No	Get/Set	Get/Set
ESP202	No	No	Get/Set	Get/Set
ESP210	No	No	Get/Set	Get/Set
ESP214	No	No	Set	Set
ESP215	No	No	Set	Set
ESP220	No	No	Get	Get
ESP250	No	No	Set	Set
ESP290	No	No	Set	Set
ESP400	No	No	Get	Get
ESP401	No	No	Set	Set
ESP402	No	No	Get/Set	Get/Set
ESP410	No	No	Get	Get
ESP420	No	No	Get	Get
ESP444	No	No	Set	Set
ESP450	No	No	Get	Get
ESP500	Get/Set	Get/Set	Get/Set	Get/Set
ESP510	No	No	Get/Set	Get/Set
ESP550	No	No	No	Get/Set
ESP555	No	No	Get/Set	Get/Set
ESP600	No	No	Set	Set
ESP610	No	No	Get/Set	Get/Set
ESP620	No	No	Set	Set
ESP700	No	No	Set	Set
ESP701	No	No	Get/Set	Get/Set
ESP710	No	No	Set	Set
ESP715	No	No	Set	Set
ESP720	No	No	Get	Get
ESP730	No	No	Get/Set	Get/Set
ESP740	No	No	Get	Get
ESP750	No	No	Get/Set	Get/Set
ESP780	No	No	Get	Get
ESP790	No	No	Get/Set	Get/Set
ESP800	No	No	Get/Set	Get/Set
ESP900	No	No	Get/Set	Get/Set
ESP901	No	No	Get/Set	Get/Set
ESP910	No	No	Get/Set	Get/Set
ESP930	No	No	Get/Set	Get/Set
ESP931	No	No	Get/Set	Get/Set
ESP999	No	No	Set	Set

My initial plan was : user can only read (get) settings but not change them, can use any feature without be able to change any configuration,
admin can change any settings
I plan to apply it but feedback / suggestion are welcome

[luc-github]

Authentication per login is now implemented as well as some glitch found when sync code with ESP3D

I need to spend time on ESP3DLib now then will start to work on CNC UI

[luc-github]

I am done with ESP3DLib 3.0 so I will start to work on CNC UI next week which will be be next year ^_^

[3DSmitty]

Awesome, Happy New Years!

[luc-github]

I have added a command ([ESP216]SNAP)) to take snapshot of active screen and dump it to SD
the snapshot is a raw dump to avoid to overload the FW footprint, the raw dump file need a conversion tool conv.py to be converted as png file, this should be useful for documentation

Here some screenshot done on 800x480 HMI

[3DSmitty]

Great Idea! This will also be helpful for troubleshooting if someone needs to share what is happening on their screen.

[luc-github]

well for debugging it depend, it use a lot of memory to dump the current screen (750KB for the 800x480, limited to RGB, no alpha, 16bit depth = 2bytes/pixel) and the LVGL snapshot API seems not really to like to snap when there is some animation like a spinner or scrolling text - so it must be used carefully IMHO

[luc-github]

I am not really happy with the lvgl API crashing if any animation is active, and the need of such big amount of memory, so I have created a rewrite plan : #53
This one is not urgent , neither critical because snapshot is only a nice to have , and can be disabled (https://github.com/luc-github/ESP3D-TFT/blob/main/CMakeLists.txt#L224), that is why I do not plan to work on it right now, but opened a issue as memo

Right now I rather like to work on CNC UI

[3DSmitty]

I have a CNC router with FluidNC I would love to test this on :)

[luc-github]

I am not sure fluidnc is best target , it has already wifi interface , sdcard and also some pendant tft fw.
the target is more pure grbl fw or grblHAL on board without sd and network capability

[luc-github]

I am working on UI and workflow definition for grbl - it is not a FW I use everyday and I needed to refresh my knowledge to see what is relevant to be displayed and what control could be useful
I am still wondering if need to to keep or split laser UI and milling UI in same FW or just update common UI by setting =>TBD
I am also investigating the impact of supporting mm and inch on TFT and WebUI
Finally I am considering to limit current TFT UI support to X Y Z A, and put on hold B and C axis to cover 90% of users and not overload the UI =>TBC

• the first line should still be the TFT status
• the second line of controls should be the axis positions :MPos or WPos depending a setting, clicking on it will display the Jog screen where you can jog / home / zero the axis
• the third line of control should display the state (Idle/Run /Error) as button it should allow to unlock state, on same like should be the streaming status, if no streaming it will display the $G report or I may add a sliding option =>TBD
  The fourth line of control should display the override buttons to display with current values clicking on it should allow to change the override values, then the file selector button, the reset button and settings button

In settings screens you should have WiFi Setup like for 3D printers, Probe Button => TBD what it should do, the settings button and Information buttons like for 3D Printer, still have space for 2 others buttons , may be dedicated to function :
e.g: laser setting for laser system and spindle control for milling system =>TBD

As usual constructive feedback and suggestions are welcome

[3DSmitty]

@luc-github FYI, have you seen this: https://github.com/suchmememanyskill/CYD-Klipper

[luc-github]

it for klipper so it is a client based on https://klipperscreen.readthedocs.io

[3DSmitty]

Yeah I know but it uses same hardware.

[luc-github]

Many projects use this hardware : https://github.com/search?q=ESP32-2432S028R&type=repositories

What special feature / point would you emphasis on this project ?

[luc-github]

lvgl 9.0 is out I will also start to work on #54

[luc-github]

Working on porting USB-OTG from ESP3D-TFT to a ESP3D I have realized that IDF release updated the USB drivers
before:

## IDF Component Manager Manifest File
dependencies:
  usb_host_ch34x_vcp: "^1"
  usb_host_cp210x_vcp: "^1"
  usb_host_ftdi_vcp: "^1"
  usb_host_vcp: "^1"
  idf: ">=5.1.0"

to

## IDF Component Manager Manifest File
dependencies:
  usb_host_ch34x_vcp: "^2"
  usb_host_cp210x_vcp: "^2"
  usb_host_ftdi_vcp: "^2"
  usb_host_vcp: "^1"
  idf: ">=5.1.0"

which bring some minor API change but also a RX buffer with configurable size

So I will update the code to handle driver update

[3DSmitty]

This one looks promising without the relay board: https://www.aliexpress.us/item/3256806115962222.html

[luc-github]

I did a big refactoring for the configuration and drivers , I have started a doc to explain the files org / architecture : https://github.com/luc-github/ESP3D-TFT/blob/main/docs/project-arch.md

I have tried to use same API for all same drivers based on initial PR of @serisman which only covered some plateform
Only one platform is not yet updated, it is the esp32 rotrix screen, because my devt board allowing to flash by serial - is now dead ...orz- I still have a production one but need to do some smd soldering to add connector for Serial to allow board to be easily flashed - I will update the board later -

Board	Status
ESP32_2432S028R	Ok
ESP32_3248S035C	Ok
ESP32_3248S035R	Ok
ESP32_ROTRICS_DEXARM35	TBD
ESP32_CUSTOM	Ok
ESP32S3_4827S043C	Ok
ESP32S3_8048S043C	Ok
ESP32S3_8048S050C	Ok
ESP32S3_8048S070C	Ok
ESP32S3_BZM_TFT35_GT911	Ok
ESP32S3_HMI43V3	Ok
ESP32S3_ZX3D50CE02S_USRC_4832	Ok
ESP32S3_CUSTOM	Ok
ESP32S3_FREENOVE_1_1	Ok

So many board I took me some time to review each one , there is a PR from @liqijian101 for the camera support I will work on it next week

The file org is more modulare a easier to update in case of new board, now each board has dedicated cmake, any new cmake is automaticaly added, just add the board in the main CMakeLists.txt to let user to select it.
Drivers can be common or specific , same for partition and sdkconfig , so it it is very flexible
The modules to be enabled is separated to simplify as usually everything is enabled by default, dev_tool setting is also in another file (e.g for logging level) to also simplify access and avoid to scroll long CMakeLists.txt

[luc-github]

I have almost finish to integrate the camera support made by @liqijian101 should be ok in couple of days, doing test I found several glitches and issues in error management when doing streaming - I will work on it once Camera code is integrated

[luc-github]

Camera is now integrated and need to be tested - I will be back to UI improvement and grbl UI implementation

[luc-github]

Working on cnc UI - I have realized the style for components sizes/positions and color need some refactoring first:

The current styles is spread in several files, and is not enough atomic to allow optimized rendering - some size are too generic and used accross resolution, which result some UI compromise in some resolutions.

So I will still use define because define are the smallest foot print, and create: a file with colors for common components, a file for each screen and for each resolution , this should allow a good flexibility but also keep a good code readibility.

I am also thinking to add a namespace for each screen to avoid variables colisions

This refactoring should also make any UI change easier IMHO

Any suggestion how to simply things is welcome as usual

[luc-github]

Still doing a lot of refactoring but I start to have a begining of cnc UI



You can see up to 4 axis for the moment , there is a button to switch between Machine Positions and Working Positions
There is an always visible reset button
The bottom buttons are overload button to display overload screen, File Selection Button and Menu button to switch to extended menu
when processing a job the File Selection Button become a Pause/Resume Button and Stop Button for the streaming

I think the most needed controls for main screen are now identified
The advanced menu will allow to configure wifi / settings and get informations, it should also to configure Probe , laser or spindle according settings, and may be macros that is not yet implemented
any constructive suggestions is welcome as usual

[3DSmitty]

Normally on a CNC you have a Feedhold button as well as a estop (kill) button. The difference is feedhold just pauses the running program (maybe for adjustments)and you can press feedhold again to continue on with the program. The estop completely stops everything on the machine. Hope that helps.

[3DSmitty]

The safety door enable should be on a settings screen and maybe the kill alarm lock. Jog cancel should NOT be on the main screen it should be on the jog screen. You should NOT be able to jog the machine while a program is running.

[luc-github]

estop is mechanical so cannot be on software

the Door Open command is the most close to estop , it simulate Door Open and so stop everything or I am wrong ? This is what I use in webUI where Emergency Stop has been renamed as Quick Stop because it seems it may confuse users...

The jog screen will have the Jog cancel of course - but if there is a jog (Jog State) when user is on main screen the button power can send Jog Cancel command, on other states but Alarm State it will be Door Open simulation command that will be sent instead.
Because it make no sense to display a stop in Alarm State because everything should already be stopped, so display Unlock in Alarm state instead of Power button make sense I think. In Error mode it is different in some case the machine may still running unlike Alarm State
Am I wrong in state management ?

About FeedHold I am just wondering if it is really necessary - when streaming GCODE the stream is is controled by the TFT and when there is a stream the button pause/restore are displayed, in WebUI I display both because there is a lot of space and also the stream may not be controled by webui so displaying hold feed make sense, but in TFT I am not sure what would be the user case

[3DSmitty]

Ok, long as there is a pause button i think we are ok. That is basically what feedhold does. How would renaming emergency stop to quick stop confuse users? Anyone familiar with CNC machining knows what emergency stop is, they won't know what quick stop is.

[luc-github]

the discussion is in discord https://discord.com/channels/752822148795596940/752822598387367989/1220834166824898641

[luc-github]

Still working on cnc UI - very slowly due to others tasks

I have ordered a couple of 7' because it looks pretty complete : https://www.aliexpress.com/item/1005007096249725.html for 38Euros
https://www.waveshare.com/wiki/ESP32-S3-Touch-LCD-7

It seems having USB port exposed also , the only drawback is no external antenna so far - I will add the support soon because it use gpio expander (CH422G) for lcd / SD / Touch and USB selector, I need to see how to integrate it - fortunatly the doc is pretty complete, will see

[luc-github]

the waveshare 7.0 is now supported : https://esp3d.io/ESP3D-TFT/Version_1.X/hardware/esp32-s3/waveshare-70-8048/
The doc for the CH422G is in chinese and I did not understood the commands and it is not really documented, the sample codes are also little bit confusing and lack of explainations - but after some research and test I finally found the proper settings to support the usage maid for waveshare 7.0 which fortunatly is basic.

[luc-github]

while resuming work on tft I have noticed that :
1 - USB OTG is no more working 5.1.5 and 5.1.2 -> need to investigate as I though was due to 5.1.5 sdkconfig and new parameters but even reverting to latest 5.1.2 is not working ->TBD
2 - settings task failed to create randomly if debug is enabled or if network connection failed -> need to investigate
3 - the number of task creation is now up to 10 so there may be some refactoring to be done to use less tasks
4 - Doing sdkconfig edition using makemenu config is no more working - may be due to new CMakeLists.txt organisation, need a way to edit SDK Config properly -> need to investigate

[luc-github]

1 - i s now fixed and merged

In addition testing more the streaming to evaluate workload on buffered streaming, I have found out that :
a - resume after pause does not put position back to last position in pause -> need to investigate
b - abort seems ignored -> need to investigate