Distributed Multi-Printer Support

[smartin015]

Is your feature request related to a problem? Please describe.

It's not so unreasonable for people to have multiple 3D printers at home, or in the prototyping industry. Even more extreme are 3D printing services (e.g. Shapeways) managing hundreds/thousands of printers.

The continuous print queue is a plugin for a single instance of octoprint, and guides like this one suggest the easiest way to manage multiple printers is with multiple instances of octoprint. This means also multiple instances of the plugin, with independent queues.

But why have many queues, when you could have a single entry point for 3D printing?

Describe the solution you'd like

Imagine the power of a single continuous print queue driving multiple printers:

1. Add prints to the queue in one instance of octoprint (or a separate page)
2. Start queue management
3. Prints are automatically distributed to many printers, with new print commands issued as prints finish.

Distributed coordination can be done in a heavy-weight fashion with kazoo, or perhaps more nimbly with a custom election implementation.

Once a leader plugin instance is established, it should synchronize its queue state with the others so they all have the same prints queued. The individual printer statuses should appear in the plugin tab (connection state, sync state, print state). Presence or absence of print files will also be synchronized, so the leader knows which printer has which files.

When the queue is started, the leader will determine which printer will best accommodate the print. Considerations for "best for the job" may vary in the future, but it can start simply by rejecting busy printers and ensuring all printers are operational.

Printer event synchronization will be required as well - when prints finish, the printers should tell the leader about it. Failures and other events should be echoed to the leader as well for tracking - so it's probably easiest to just forward all printing events and filter them on the leader side. Similarly, health checking should be attempted periodically to ensure the print network is OK.

Failure handling across printers can also become tricky. If a print fails multiple times on one printer, we can attempt it on another printer. If it starts failing there, the issue is probably the print itself. This is something the leader will need to handle.

If the leader goes offline, another plugin instance takes over leadership - using the bully algorithm makes next-leader determination really quick and convenient.

Describe alternatives you've considered

• Somehow linking multiple printers to the same octoprint instance. This seems far harder to do than implementing cross-plugin syncing.
• Different types of printers will require different gcode for printing. Initial implementation will assume identical printer setups, but in the future users could perhaps upload multiple files and the selection algorithm choose which file to send to each printer. Print times might vary, which would also be a consideration for printer selection (time bin-packing of sorts).
• It may be worth implementing a "namespace" for print automation, so that multiple instances of print queue can still be allowed, with their own printer pools.
• A work stealing approach for scheduling might be relevant here, although with more sophistication we can avoid some of its pitfalls (e.g. excessive material changes)

Additional context

Implementing #32 will complicate queue item distribution, as different printers may have different materials. However, this doesn't block initial implementation. Followup will be needed when synchronizing printer state, and in selection considerations (assume changing filament is costly, so try to match colors to printers).

Hand-wavy visioneering

Multi-printer setups don't need to be limited to local-network... imagine a worldwide network of home printers where users can "earn" printing time on others' printers by performing work using their own printers. Printer selection could prioritize geographical closeness for "rapid" prototyping, or perhaps some shipping process for "volume" printing.

This might actually, unironically be a use case for creating a cryptocurrency. Proof of work is running and delivering others' prints - which can be used to generate coins for you to use to run your own jobs on the print network. https://3d-token.com/ is one example of something similar... but it seems to be more about using crypto to pay for a proprietary printing service. this article equates "supply chain" and "blockchain", but without any details. What I have in mind is more "p2p sharing, but for 3d printing".

[aburtonProto]

This.
The idea sounds awesome, and has many useful applications. #32 would need to be at least basically implemented first, in my opinion, specifically material type.
Also the idea of creating a currency around sharing printer time is fascinating. Completely useful work as Proof-of-Work.

[smartin015]

Progress update:

I've pushed a few commits to a separate branch - nothing is usable yet, but I currently have working solutions to a number of problems, including:

• auto-discovery of other octoprint instances on the network without having to list IP addresses
• distribution of print files to various printers in the queue, without a central server
• peer-to-peer state synchronization
• scheduling a next print across different queues attached to the same printer
• the flow for creating/modifying jobs and submitting to / revoking from network queues
• synchronizing/sharing print files across different networked printers
• observing individual printer constraints when scheduling (incl. which filament is currently loaded)
• optimizing which printer receives which job based on operator availability at particular times of days / days of week / etc.

It's been a really fun traipse through UDP broadcasting, linear optimization, dynamic programming, RAFT consensus, and a bunch of other stuff I hardly ever get to work with!

I'm now in the stages of making a basic "plugin-free" command line implementation so that I can prototype and load test without having to install octoprint many times. Also writing unit and integration tests to make sure there's no obvious mistakes.

When I have a serviceable version, I plan to release it as an "alpha" feature (read: can break at any time, for any reason). Also need to buy a couple more printers for real testing, as I've only got one printer at the moment.

A friend also shared Prusa Pro AFS with me - really awesome execution of networked printing (especially using drawer slides for back-panel access to swap out printers). But it appears once again that companies just can't help but create cloud services with attached subscription models 🙄

[smartin015]

Another minor update - I've split the code out into a separate repository I've been calling peerprint. In addition to keeping the continuous print repo clean, it's now possible to integrate with other 3D print services & tools - such as MatterControl, Repetier, AstroPrint etc. - while still sharing the same queue(s).

Not that any of that is likely in the near or even mid term - I need to finish hooking this into the CP plugin first. But it's fun to dream.

Next step is to attempt integration back into the continuous print plugin. Some features (e.g. material selection) may just be placeholder-ed so we can reach parity with current queue behaviors before continuing onward.

[smartin015]

Status update: haven't touched the ntework printing code for a bit, but color/material selection is available in the latest RC, so we should be good to proceed with that in network printing as well.

I'm now switching the CPQ storage setup to sqlite (from serialized JSON) which should make all this integration a lot more sane by the end of it all. CPQ's sqlite database will be separate from the data managed by peerprint, so that future integrations with peerprint won't have to bolt on a whole separate / redundant queuing system in order to use the library.

[smartin015]

Storage implementation is almost ready for the RC branch - I aim to have it ready by the end of this week.

Also purchased and octopi'd a MP Mini Delta V2 as a second network printer for initial shared queue testing:

[smartin015]

Another status update: architecture is starting to settle and the sqlite3 conversion is complete and unit tested. However, I'm still working on UX refinements and shaking out remaining bugs. This will take some time before it's RC-worthy.

It's actually not too much additional work to add basic support for #54 - including basic restrictions on sets to only print on certain printers, which allows for LAN queues with heterogeneous printers as long as you include a .gcode file for each category/type of printer in the queue. I have job export and printer profile metadata feature-complete and mostly untested right now; import and unpacking is yet to be fully implemented.

[smartin015]

Update time! And this time with a screenshot:

This marks the first successful (albeit virtual) print of a job that was issued by a remote peer elsewhere on the network. This confirms that:

• Peer discovery works
• Peer state synchronization works
• Job acquisition & locking works
• Bundling print files into a single job file works
• Network file transfer works
• Unbundling job files into individual print files for selection & printing works
• Job state updating works
• Driver code is general-purpose and can handle both LAN and local queues

I'm past the hardest parts... probably. Still more UX and unit testing work to be done, but the work should be quite a bit more straightforward now.

[smartin015]

I finished two usability tests with some printing friends and worked through the feedback - everything's running quite smoothly now.

Next steps:

• Refresh the unit tests so the build stops failing
• Deploy it on my two printers and let it soak for a few days
• Fix any issues that arise from physical testing
• Deploy to the RC branch (ETA: 1-2 weeks), collect feedback, fix bugs

After another 2 weeks of soaking, will finalize and publish 2.0.0 to the main branch

(Edit: after a couple small hotfixes, successfully started a print on my delta printer from another instance of OctoPrint! 🎉 )

[smartin015]

Candidate release created: https://github.com/smartin015/continuousprint/releases/tag/2.0.0rc1

This contains a lot of new features and changes, so I've posted a release video which is also available from the docs main page.

[smartin015]

I've moved the broader WAN/crypto aspirations in this issue to smartin015/peerprint#2. The remaining features discussed here are implemented in 2.0.0 (currently in RC), so I'm going to close this out in favor of new issues when bugs are discovered.