Add Description-Content-Type field

[msabramo]

This allows specifying the content type of the distribution's description (e.g.: text/plain, text/x-rst, text/markdown).

See https://github.com/msabramo/python-packaging-user-guide/blob/description-content-type/source/specifications.rst#description-content-type

Cc: @nicktimko, @ncoghlan, @dstufft, @jaraco, @xavfernandez

Related:

• https://mail.python.org/pipermail/distutils-sig/2016-May/028689.html
• Add Markdown support readme_renderer#3
• WIP: Add new long_description_content_type kwarg setuptools#708

[nicktimko]

Bikeshedding: CommonMark? Can that be the default?

[msabramo]

Maybe. I've never used CommonMark; in fact, I hadn't heard of it until you mentioned it. Have a Python implementation that you like? https://pypi.python.org/pypi/CommonMark perhaps?

[msabramo]

How is CommonMark different than the original Markdown? Why do you prefer it?

[nicktimko]

CommonMark (http://commonmark.org/) is rigidly defined unlike Markdown which has bits of ambiguity (true "original" Markdown.pl is properly broken in some nesting cases). It improves a bunch of quirks, like spaces around lists, and standardizes some of the more convenient parts of GFM like fenced code blocks.

[msabramo]

http://commonmark.org/ sounds interesting and has some big names behind it. I'm intrigued. I think I'll take a closer look and consider making CommonMark the default.

[msabramo]

BTW, the only reason that I made original Markdown the default is because it's listed as the initial variant in RFC 7763 at https://tools.ietf.org/html/rfc7763#section-6.1.4, but then CommonMark is referenced in RFC 7764 at https://tools.ietf.org/html/rfc7764#section-4.5 (which I hadn't noticed before), so I guess nothing prevents it from being the default.

Part of me thinks that because there are so many competing Markdown variants and so much confusion around Markdown, that perhaps there should be no default and we should require the variant to be specified. Thoughts on that?

[ncoghlan]

The overall default on the server side needs to be UTF-8 ReST (as in the current PR) for compatibility with existing projects that already correctly use ReST formatting.

However, when the media type is given as text/markdown, I'd actually suggest using either GFM (GitHub Flavored Markdown) or CommonMark (where GitHub are one of the backing organisations) as the default, as one of the more common reasons for using Markdown in README files these days is as a nice user-friendly front page for a GitHub repo.

So using CommonMark as the default (including falling back to it if asked to process an unknown variant) makes sense to me.

[ncoghlan]

I'd make charset optional, defaulting to UTF-8 - that way text/markdown is sufficient to get Markdown rendering.

[msabramo]

My first instinct was to default to charset to UTF-8.

But then I saw this in RFC 7763 section 2:

     charset: Per Section 4.2.1 of [RFC6838], charset is REQUIRED.
     There is no default value because neither [MDSYNTAX] nor many
     popular implementations at the time of this registration do
     either.

That said, we can disagree with that RFC and do what we want, but I thought deferring to previous work would be a good start, since they probably had reasons and discussions around this.

Intuitively, it seems to me to be a good idea to simply default to UTF-8. I'm not sure why the RFC didn't simply do that. I wonder if they've thought of something that we haven't?

[msabramo]

OK, I went ahead and made charset optional and defaulted it to UTF-8.

d454938

[ncoghlan]

In the general case, I agree that there's no sensible universal default, but Python project descriptions aren't the general case - they're a specific use case. That lets us make assumptions in the name of ease-of-use that wouldn't be valid given a broader context.

[ncoghlan]

+1 from me for the general idea, but what you have so far is missing guidance on expected behaviour for tools reading the field to render the description when:

• they don't recognise the content type
• they don't recognise the charset
• they don't recognise the specific variant

Those additions would be SHOULD/MAY type guidelines rather than MUST requirements, but I think we still want to give some thought to how PyPI should handle those cases, and how those considerations can be turned into general recommendations.

[msabramo]

Added mentions of CommonMark in 774d091.

[msabramo]

Made CommonMark the default variant for Markdown in 33dc8c5.

[ncoghlan]

Latest draft looks good to me, but I'll defer to @dstufft for actual sign-off.

While I don't expect this idea to be controversial, we should also give folks a chance to discuss the idea on distutils-sig before committing to it.

[msabramo]

@ncoghlan: I'm still planning to add some info on expected behavior when things are omitted or invalid, as you suggested in #258 (comment).

But certainly, I'd love to have @dstufft and anyone else's comments.

[pfmoore]

Maybe move the sentence "If charset is not provided..." down below? (See next comment)

[msabramo]

Does e4b39db help?

[ncoghlan]

I agree with @pfmoore that this can be dropped in favour of the more concise specification below.

[pfmoore]

Overall, LGTM (one minor comment in the diff).

[msabramo]

e4b39db: Clarified how to treat missing/invalid values for Description-Content-Type (and charset and variant) as suggested by @ncoghlan at #258 (comment)

[theacodes]

I imagine @dstufft is likely at pycon and will be a little slow to respond. :)

…

On Tue, May 16, 2017, 9:58 PM Marc Abramowitz ***@***.***> wrote: @dstufft <https://github.com/dstufft>: Any more comments on this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#258 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUcx46zf1GPwd7q8FHM2yNFg4416npks5r6n5zgaJpZM4JYgnk> .

[dstufft]

Sadly I'm not at PyCon, I just haven't gotten a chance to circle back around to this yet :(

[msabramo]

Yay, I see @dstufft merged this. Great news!

So what do we need to do next?

[dstufft]

Probably implement support in setuptools, then wheel, then twine, then PyPI/Warehouse.

[nicktimko]

Would it be better to have the server (PyPI/Warehouse) accepting it before the clients? Then newer clients wouldn't get errors if they support what the server doesn't

[ncoghlan]

I think there are a few aspects with a required order, then some that can happen in arbitrary order:

1. Check that pip will ignore the new field at install time. If not, update it, and document the minimum required version implied by using the new field. Also make note of whether or not easy_install will be able to install packages using the new field (we don't have to fix that if it does, but we should document the answer)
2. Update PyPI/Warehouse to allow the new field (if they don't already)
3. Update setuptools and the other build systems to allow setting the new field, and PyPI/Warehouse to actually pay attention to it

(We should probably document this somewhere on pypi.io as a standard practice, since it applies to any interoperability metadata changes we make)

[wimglenn]

@ncoghlan or anyone else who knows about this, could you please explain how to get it working properly? I added in setup.cfg:

[metadata]
long_description_content_type = text/x-rst; charset=UTF-8

This is syntax documented as supported in setuptools (here). But it doesn't seem to be respected or used for anything - the generated metadata in the distribution always just shows "UNKNOWN". Is the correct value getting dropped in the installation at some stage?

I use the version of setuptools v38.2.4, and I use "python setup.py bdist_wheel" to build the dist.

[di]

Hi @wimglenn, there's an open PR to setuptools to fix this: pypa/setuptools#1207

[wimglenn]

@di ah, yes, that’s exactly the spot I ended up at in pdb when I was looking at it. Thanks!