Fallback requirements for picture element

[mattgarrish]

Now that the picture element is valid for use, there is a question for epubcheck about how to determine when a manifest fallback needs to be applied. The media attribute makes determining whether a core media type will be available complicated, since it all depends on the runtime environment which will apply.

If we require that a core media type be specified in the img element, though, then it doesn't matter what combination of formats and media queries are used in the source elements. The img element also represents the default, so a reading system will still have the option to choose the foreign resource(s). As the natural fallback, it seems like it should always specify the CMT to avoid manifest fallback requirements.

(It's probably also going to be best practice to have a CMT in the img, too, since there's no guarantee of support for picture/source yet.)

But, we need to know if the CG agrees with this conclusion, and we should make this requirement explicit in the fallback requirements, if so.

If not, how else can we reliably determine if a CMT will be available? Do we require that at least one source element specify a CMT and not specify a media attribute? Do we ignore media attributes and just hope for the best so long as a CMT is listed?

[mattgarrish]

To resolve this one, I'd suggest we either add a paragraph like this to the end of section 2.5.5:

Foreign Resources MAY be referenced from a picture element without fallback provided a Core Media Type resource is specified in the element's child img element. Otherwise, a manifest fallback MUST be provided for the resource referenced from the img element.

or possibly the more flexible:

The picture element MAY reference Foreign Resources without the need for a fallback Core Media Type resource provided one of the following conditions is true:

• its child img element specifies a Core Media Type resource; or
• at least one child source element specifies a Core Media Type resource and does not define any conditions on its use in a media attribute.

If neither condition is met, a manifest fallback MUST be provided for the resource defined in the child img element.

[murata2makoto]

In my understanding, manifest fallback is not widely used or implemented. Why should we use it here?

[mattgarrish]

Why should we use it here?

The img element already requires manifest fallbacks for foreign resources, so it's not introducing anything new in that regard.

I'd rather not perpetuate it, but unless we require a usable CMT in the img element or a source without any conditions then how do you propose we satisfy the fallback requirement?

[murata2makoto]

I understand now. I prefer your first solution. It is simple. The second one does not add any practical advantages since the media attribute cannot be used as a condition.

[mattgarrish]

since the media attribute cannot be used as a condition

I'm not sure what you mean here. The attribute is a condition on use. If you omit it, then the referenced image can be used anywhere. For example, this is fine:

<picture>
<source src="cmt.jpg"/>
<img src="foreign.tiff"/>
</picture>

but this is problematic because the core media type is only available for small device screens:

<picture>
<source srcset="cmt.jpg" media="(max-width: 640px)"/>
<img src="foreign.tiff"/>
</picture>

It's sort of a question of best practice whether the CMT needs to be in the img. It makes sense because older reading systems won't process the other sources, but do we need to exclude other valid ways of specifying a fallback?

It's also what makes manifest fallbacks confusing. Technically this is about the picture element and manifest fallbacks are for elements without fallbacks. But you might still need to use a manifest fallback for older reading systems if you don't put the CMT in the img element. We probably should avoid mentioning them -- they don't satisfy the fallback requirement, but you could still add one for older reading systems if you're worried.

For example, if you have:

<picture>
<source src="cmt.jpg"/>
<img src="foreign.tiff"/>
</picture>

Then in the manifest you might also add a fallback attribute just to be safe:

<item src="foreign.tiff" fallback="cmt"/>
<item src="cmt.jpg id="cmt"/>

But it's this kind of complexity, plus EPUB's goal to have CMTs for interoperability, that makes the img option the simplest.

[murata2makoto]

@mattgarrish wrote:

at least one child source element specifies a Core Media Type resource and does not define any conditions on its use in a media attribute.

That child source element does not define any conditions and is thus always selected. This is what I mean.

[mattgarrish]

That child source element does not define any conditions and is thus always selected.

I haven't seen that stated in the HTML specification. Do you have a reference?

It is always possible to select it, but all I've seen is that the selection is user agent-defined. (The type attribute would be required, but I omitted it for simplicity.)

[mattgarrish]

The addition of srcset for img makes determining if a core media type is available, even outside of picture, a headache, too.

[murata2makoto]

@mattgarrish I stand corrected about non-mandatory selection. But I also wonder if we are making things unnecessarily complicated. Is it possible to lift the manifest-level fallback requirement for image elements within picture elements?

[mattgarrish]

But I also wonder if we are making things unnecessarily complicated. Is it possible to lift the manifest-level fallback requirement for image elements within picture elements?

Right, I'd prefer to require a core media type in the img element. I'm only raising the other option because it is technically valid to do, so I wonder if people will complain that we've been overly restrictive again. It also makes things a lot simpler for epubcheck if we don't have to check sources, but we couldn't implement this on our own.

If we require the core media type in the img, then the other nice thing is that manifest fallbacks never factor in. Technically, they are only for elements that don't have fallback capabilities, and in the case of picture the img is the fallback for graceful degradation, so I think we do have a strong case to make it a requirement.

[rdeltour]

Right, I'd prefer to require a core media type in the img element. I'm only raising the other option because it is technically valid to do, so I wonder if people will complain that we've been overly restrictive again.
(…)
If we require the core media type in the img, then the other nice thing is that manifest fallbacks never factor in. Technically, they are only for elements that don't have fallback capabilities, and in the case of picture the img is the fallback for graceful degradation, so I think we do have a strong case to make it a requirement.

A couple things to take into account if we want to make this restriction:

• it creates a new backward incompatibility, when a foreign type on img was previously allowed with a manifest fallback
• the ability to effectively provide a fallback depends on RS support for the picture element; granted, browser support is pretty good.

The backward compatibility issue is what worries me the most. Maybe an intermediary approach is to:

• when picture is used, require the img/@src to be a CMT.
• otherwise, the img can be a foreign type if it has a manifest fallback.

That way we get the "best" of both worlds.

Apart from that, I think that making the img the holder of the CMT (instead of allowing a source to do it) is perfectly acceptable, and it's also more robust.

It also makes things a lot simpler for epubcheck if we don't have to check sources, but we couldn't implement this on our own.

EPUBCheck will still have to parse source elements, and srcset attributes (on both source and img), as we need to check things like the image is not remote, exists in the container, is declared in the manifest, etc, etc. But that's another issue 😊

[mattgarrish]

• when picture is used, require the img/@src to be a CMT.

Yes, sorry if that's not clear. I'm only talking about img within the context of picture. When img isn't inside of a picture element, nothing changes in terms of existing requirements. epubcheck will have to be a little more context aware. ;)

[GeorgeKerscher]

Hi, Just checking on devices with no screen, we are still good? Best George From: Romain Deltour <notifications@github.com> Sent: Wednesday, March 6, 2019 6:24 AM To: w3c/publ-epub-revision <publ-epub-revision@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [w3c/publ-epub-revision] Fallback requirements for picture element (#1237) Right, I'd prefer to require a core media type in the img element. I'm only raising the other option because it is technically valid to do, so I wonder if people will complain that we've been overly restrictive again. (…) If we require the core media type in the img, then the other nice thing is that manifest fallbacks never factor in. Technically, they are only for elements that don't have fallback capabilities, and in the case of picture the img is the fallback for graceful degradation, so I think we do have a strong case to make it a requirement. A couple things to take into account if we want to make this restriction: * it creates a new backward incompatibility, when a foreign type on img was previously allowed with a manifest fallback * the ability to effectively provide a fallback depends on RS support for the picture element; granted, browser support is pretty good <https://caniuse.com/#search=picture> . The backward compatibility issue is what worries me the most. Maybe an intermediary approach is to: * when picture is used, require the img/@src to be a CMT. * otherwise, the img can be a foreign type if it has a manifest fallback. That way we get the "best" of both worlds. Apart from that, I think that making the img the holder of the CMT (instead of allowing a source to do it) is perfectly acceptable, and it's also more robust. It also makes things a lot simpler for epubcheck if we don't have to check sources, but we couldn't implement this on our own. EPUBCheck will still have to parse source elements, and srcset attributes (on both source and img), as we need to check things like the image is not remote, exists in the container, is declared in the manifest, etc, etc. But that's another issue 😊 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1237 (comment)> , or mute the thread <https://github.com/notifications/unsubscribe-auth/AG3HSBjFLSjim91QQLM38cR4f3FmZ1csks5vT8FVgaJpZM4bO8QH> . <https://github.com/notifications/beacon/AG3HSO0nwiDlWNnM-QvNsSHB4bC0AO7Kks5vT8FVgaJpZM4bO8QH.gif>

[mattgarrish]

Just checking on devices with no screen, we are still good?

@GeorgeKerscher Yes, the picture element is just a wrapper that allows other sources to be associated with the img tag. You still put an alt attribute on the img to provide alternative text, and you can use aria-* attributes to associate an extended description.

[mattgarrish]

Fixed by #1244.

[rdeltour]

So, I had a detailed look at HTLM's images processing model, and I think the resolution of this issue is actually wrong.

Basically, for browsers supporting picture and srcset (most of them), the image selection algorithm can be simplified to:

1. compute the image source set (all img elements are associated with a conceptual source set)
2. the UA can chose any of the image source in the resulting source set

To simplify again, the algorithm which returns an image’s source set says:

• if img is not in picture, return the source set is the one defined in srcset attribute, augmented with the source defined in src only under certain conditions (precisely: if no other source of pixel density 1 is present AND no source has a width descriptor)
• if img is in picture, return the source set of the first source or img child which media condition (if present) matches and type (if present) is supported, if not empty.

In other words, the HTML processing model does not use the src attribute as a fallback. So I don't think EPUB can rely on it to be a native fallback for images.

If we keep the current resolution, we're basically creating a willful violation of HTML, and I really doubt EPUB reading systems will bother to tweak the image processing model of underlying browser components.

I suggest we change the rules to:

• if an img element is not a picture child, both src and srcset can reference foreign resources as long as a manifest fallback is defined.
• if an img and source element is in a picture: require both src and srcset to only reference CMT resources unless srcset is defined on a source element with a type attribute set to a foreign type.

(in other words: only allow foreign types on source elements with an explicit type attribute, or on standalone img for backward compatibility).

Technically, we could loosen the latter point to requiring that the source set contains at least one CMT resource, but I don't know if browsers would use that for content negotiation. We'd better stick to source elements with type attributes, which are designed for the task.

Or, alternatively, we consider that image core media type are a thing of the past better left to best practices, and we make no restriction at all ¯\_(ツ)_/¯.