Handle canvases with "complex" scenes

[tomcrane]

Where a complex scene is anything in the 0.0..01% scenarios mentioned in #1, #2, #3.

That is, the canvas we are passing in to be rendered is not just one image filling the whole canvas, with (optionally) one image service available for that image.

The canvas has any combination of

• more than one image
• image(s) that target a particular region of the canvas, not all of it
• Choice (see #tbc)

(and more to come)

This depends on the developer "stance".

As a developer, I have loaded a manifest, obtained a canvas, and pass the canvas to a renderer:

<canvas iiif-content=... />

I don't want to evaluate the scene myself to determine what sub-layout I'll need to do... I want <canvas...> to do that for me.

I don't know what's on the canvas... my custom code stops before we get into that.
Even with Choice I can still be oblivious of what's actually on the canvas and expect the component to render a default scene.

I could choose to become aware that the Canvas has Choice present and react to that, if I want (see Choice #tbc)

[stephenwf]

I think this will be one of the last ones to work on. Likely pulling together all of the other DX into an opinionated render-everything component. Waits to be seen if temporal elements will come into this too - to drive experience like the fire demo.

Rendering

In terms of rendering, currently we can:

• Paint single static images
• Paint a single zoom level of a tiled image
• Paint a composite element, and switch based on zoom (described in Show a single responsive image #2 )
• Paint a box - with limited styling for background, border etc.
• Paint HTML content, should be considered an escape hatch

With some experiments:

• Painting polygons (but no donuts 🍩 )
• Paint an externally controlled texture (for example, a texture synced with a video element)
• Text - rasterisation issues not yet solved, currently falls back to HTML

Where painting here means adding to an object at x, y, w, h with zoom etc.

Data source

The presentation 3 parser holds off parsing annotations and leaves them intact. So currently when working with them, you are working directly with the source. The upgrade path that happens before will upgrade OA annotations to W3C annotations however.

When deciding what to render, the only thing we really need is the ordered list of annotations and then a set of strategies for understanding various types of annotations.

The universal viewer is really the only viewer that currently looks at the individual content in the annotations and will change which internal viewer it uses based on that. Here is the source for that:
https://github.com/UniversalViewer/universalviewer/blob/webpack/src/Viewer.ts#L381-L430

This works by checking the type and format in the annotation body and associating a renderer based on that. The first question I have is: Is this check enough?

If that is enough, there's a starting point here.

It helps to disambiguate the various valid values of annotation bodies.

const matcher = matchAnnotationBody({
  String: function () { ... }, 
  ChoiceBody: function () { ... }, 
  SpecificResource: function () { ... } ,
  ContentResource: function () { ... } ,
  OtherContentResource: function () { ... } ,
});

// ... 

matcher(annotation.body); // List of annotation parsed bodies

From this point we are really relying on all Sound or Image types looking roughly the same or having additional checks for format and having something that can render each. That might require extra tools - such as making HEAD requests to check for CORS, auth etc.

Composition

So far there have been broadly "right" answers to questions - how to identify data and how to render specific data. Composition is where we will have to make decisions. I don't have those answers, but do have some ideas on what we will run into:

• Authentication
• Choices
• Layers
• Annotations
• Text / SVG / Box styling, size
• Temporal elements (not painted with t=n)

These may manifest as default UI elements, possibly customisable, or as DOM events that can be picked up externally.

[tomcrane]

This is a tricky bit, yes.
The UV picks a canvas renderer based on looking at the media format, type and file extension (should really be solely on type - Image, Video, Sound - as dctypes, rather than formats - type is more fundamental). But this doesn't help if it's a canvas with a duration and a video in part of that duration and an image with deep zoom in another part of the duration. The canvas renderer is defined by the type of the content it looks at first.

Do we want to have something that can render anything, any mixture of stuff?
Should we simply not deal with canvases with durations for this initial phase?

You could say that if there is a duration at all, then use a component that doesn't do deep zoom but can still composite static images and text along with audio and video (like iiif-av-component), and if there is just w,h use a renderer that can do deep zoom but couldn't show a video. Is that good? Or bad?

If we do support AV, and it involves a different renderer, you still want to be able to hide this selection of renderer from the developer at code-time. Still want to be able to just pass canvases to a canvas renderer and have it do something.

[stephenwf]

@tomcrane is there an up-to-date P3 fire demo? I can put together a code sample that composes their elements (manually)

[tomcrane]

https://tomcrane.github.io/fire/dz.html
and
https://tomcrane.github.io/fire/fire3-dz.json

[stephenwf]

Code samples added based on that:
Data: #37
Atlas: #36

[tomcrane]

Proposal

From the developer point of view this is identical to the scenarios in #1, #2, #4. This is the point of canvas panel, it doesn't require you to evaluate the scene just make something visible.

So a canvas that had multiple image content resources in different places, each with image services, would require no more declarative or programmatic markup. If it has choices, or linking annos, or text, or other content that you want to present more UI for or handle user interaction with, then the developer needs to do more work. But these are handled in other discussions.