Updating the build system and fonts

[aaronbell]

Did a major overhaul of the repository structure and build system. Now split out the differentiated glyphs from the common glyphs and merges them together on build.

Also did some other updates to address FB errors.

[NightFurySL2001]

Hi @aaronbell, thank you for your help to proceed with the build process. However, I have communicated with the font author (which does not know about build process) which has pointed out some issues going forward:

He is planning to stop maintaining the font after CJK Ext-J block is released in Unicode next year, and there are pending changes to be integrated into the font now at his local sources (which are not based on UFO - UFO are generated artefacts after building the font). As such, the current build flow is proposed and maintained by me.

Here are a few questions regarding the build process and the future of other versions of LXGW Wenkai:

1. Is the font information stored in /sources/*.ufo along with the differentiated Latin parts, and shared data in /sources/*_common*.ufoz?
2. How are /sources/*_common*.ufoz different than the original ufoz?
3. Will you do similar build system updates to the other LXGW Wenkai series? Current submissions are this repo (TC), Add [LxgwWenKai-Lite] google/fonts#4601 (standard) and Add LXGW WenKai GB google/fonts#5790 (GB). It seems for the other 2 editions only -Lite edition will be accepted to Google Fonts, where there contains no low-quality zi2zi generated glyphs.
   It would be hard for the font author to maintain these repositories (esp. with the new build system) since there are 5 editions (standard, standard Lite, GB, GB Lite, TC), (I assumed) only 3 of them are pending for Google Fonts submission, and currently only this repo is updated with the new build system.
4. Is it possible to include GitHub Actions such that the font can be built on GitHub? Just setting it to be manually triggered will be fine.

[aaronbell]

Hello! I totally understand your apprehension. Let me address your questions.

1

Correct. the *.ufo are the differentiated Latins, and *_common.ufoz are the shared Hanzi forms

2

Essentially, the difference is that the _common files have the differentiated Latin glyphs removed, so it is just Hanzi and any other glyphs that are either 500 or 1000 units wide (since these remain the same between the Mono and non-Mono version).

3

At present I am only contracted to update Wenkai TC. This isn't to say that the others won't be updated as well, but TC is the only one that is currently prioritized.

4

Yes, I can do that, no problem.

My intention behind this is to actually make it easier to maintain — there is only one copy of each glyph, and one copy of the feature code for the entire set of fonts (stored in sources/features.fea). This way, should any modifications be needed to the shared components, those modifications will automatically be applied to both the standard and Mono versions. However, if you don't feel comfortable with this system, that's fine too.

[NightFurySL2001]

I've reviewed the changes and most seem reasonable. A few questions though:

1. Is it possible to retain the original ufoz instead of doing a subset? This way the font author just need to update the same ufoz file without affecting his local source files.
2. Any reasons for ascender/descender value changes?
3. Why is the removal of styleMapFamilyName necessary? Won't it affect the linking process?
4. This one is a fault on our end: the UFOZ files are missing the openTypeNameRecords for Chinese names when exporting. Can you please help to add these in?
   missing-fontinfo.zip

[aaronbell]

Is it possible to retain the original ufoz instead of doing a subset? This way the font author just need to update the same ufoz file without affecting his local source files.

Yes, though it is worth keeping in mind that the ufoz is acting like a glyph repository and anything that is in the differentiated ufo will by override the ufoz. So it is important to keep an eye on the changelog and update anything that should be in the differentiated ufo or common feature.fea file.

Any reasons for ascender/descender value changes?

There was a misalignment between winAscent / winDescent and the hhea values, so I aligned those. I also noted that the Mono version differed from the standard version, so I changed the metrics to align with the standard. Unless there was a particular reason to have the two versions differentiated? Happy to modify further, I just thought it best to keep everything aligned.

Why is the removal of styleMapFamilyName necessary? Won't it affect the linking process?

IIRC, it was actually preventing linking because the macStyle and other bits weren't set correctly. Fontmake can figure out how to set the font metadata correctly, so I just updated the source metadata to ensure it knew what to set :)

This one is a fault on our end: the UFOZ files are missing the openTypeNameRecords for Chinese names when exporting. Can you please help to add these in?

Sure

[lxgw]

I also noted that the Mono version differed from the standard version, so I changed the metrics to align with the standard. Unless there was a particular reason to have the two versions differentiated? Happy to modify further, I just thought it best to keep everything aligned.

The monospaced edition takes the mertics of Sarasa Sans Mono as reference, which are more suitable for coding (See lxgw/LxgwWenKai#56). As such, the metric data of the monospaced edition differs from that of the proportional edition.

[NightFurySL2001]

Is it required to update the lib.plist when adding new glyphs in the common UFOZ file? There's a GlyphOrder list there.

Also may I ask what's this key is for? com.schriftgestaltung.appVersion

[lxgw]

The following glyphs shouldn't be involved in common ufoz files due to the difference between proportional and monospaced editions of these glyphs.

asterisk
underscore
grave
x
y
macron
yacute
ydieresis
ycircumflex
florin
u01B4
u0233
u024F
u025B
u0284
u028E
u028F
u0296
u0443
u045E
u04EF
u04F1
u04F3
u1D08
u1E8B
u1E8D
u1E8F
u1E99
u1EF5
u1EF7
u1EF9
ygrave

[aaronbell]

@NightFurySL2001 @lxgw I've updated the ufoz to be a duplicate of the original ones that were supplied in the repro. Well, close to a duplicate, since I added a few additional glyphs and removed one (00AD). If you could ask the designer to use the new versions of the files, then they can update that file on the repro and it'll pull in the new version.

The following glyphs shouldn't be involved in common ufoz files due to the difference between proportional and monospaced editions of these glyphs.

Generally I'd agree, except that those glyphs are probably either 500 or 1000 ADW in both the proportional and monospace versions of the font. So for that reason it doesn't matter.

[aaronbell]

Is it required to update the lib.plist when adding new glyphs in the common UFOZ file? There's a GlyphOrder list there.

No. That's just the specific glyph order for the font, but when building it'll pull in anything that's in the glyphs folder

Also may I ask what's this key is for? com.schriftgestaltung.appVersion

That's an artifact of the conversion that I did (using Glyphs) that records the current version of Glyphs, as newer versions can affect how things work.

[NightFurySL2001]

Here is update flow in future:

1. Common UFOZ (non-differentiated glyphs) will remain the same.
2. Width difference will need to modify the UFO.
   1. editing glyphs = replacing glif directly
   2. adding glyphs = add glif, modify contents.plist
   3. remove glyphs = delete glif, modify contents.plist

[lxgw]

@aaronbell I have checked the new font files and UFO source files in the fork repo, and the vertical metrics have been lost. It should be because I exported the ufo file with FontForge in the first place. Could you help to add the vertical metrics back? The advanced height of each glyph is 1000 and the public.verticalOrigin value is 880.

The following glyphs shouldn't be involved in common ufoz files due to the difference between proportional and monospaced editions of these glyphs.

Generally I'd agree, except that those glyphs are probably either 500 or 1000 ADW in both the proportional and monospace versions of the font. So for that reason it doesn't matter.

Additionally, I'd still like to include the monospaced edition for these glyphs since there is a significant spacing change for monospaced edition for these glyphs (the outlines of these glyphs are narrower in the monospaced edition than in the proportional edition), also it'll be easier for me to maintain as essentially the whole ASCII region and LCG will be separated cleanly.

I'd appreciate it if you could address these issues. Looking forward to your follow-up actions.

[lxgw]

Moreover, I have also found that the powerline symbols in monospaced fonts had been lost.

In the image below, the twenty glyphs highlighted in red are powerline symbols I have added into the monospaced fonts.

Could you help to add these powerline symbols back into the monospaced fonts?

LXGWWenKaiMono-PowerlineSymbols.zip

[aaronbell]

Thanks for the review. I’ll look into these.

[aaronbell]

This has been a rather informative experience for me in understanding the limitations of Glyphs in managing and updating UFOs directly. It appears that anything Glyphs decides it doesn't understand / process properly is simply dumped on export. For example, the localized names. Also, you'd mentioned public.verticalOrigin has gone missing, but it appears not to even be present in your version of the UFO files either, which makes me wonder if there's a limitation in the capabilities of Font Forge too.

I suspect that this will be an issue with most applications that translate UFO into app-specific data structures and then back to UFO. I'd suggest using Robofont as it is a native UFO reader / writer, but it appears to have non-existent CJK / vertical typesetting support. Otherwise, it is probably best to store in a particular app's format, then use a reliable conversion tool (like glyphs2ufo) to obtain the desired output. It makes it harder for others to work on your sources, but at least you can avoid things being broken unexpectedly like we've observed here.

I've gone through and done the fun task of doing an overhaul of the differentiated UFO files by first converting them to Glyphs format, doing my usual clean up, and then converting them back to UFO / DesignSpace using glyphsLib. I know that the output of that library is quite reliable and in conjunction with FontTools will give us binaries with predictable results. As part of this, I also reduced the amount of duplicate / unnecessary data in the fontinfo section, and let the toolchain do its job in filling out that info.

I believe that these are looking pretty good, but if you could take another look through, I'd appreciate it! I'll do one more pass in the morning :)

[NightFurySL2001]

Also, you'd mentioned public.verticalOrigin has gone missing, but it appears not to even be present in your version of the UFO files either, which makes me wonder if there's a limitation in the capabilities of Font Forge too.

Yes, FontForge seem to be missing the vertical metrics after exporting to UFO.

I suspect that this will be an issue with most applications that translate UFO into app-specific data structures and then back to UFO

That is correct. FontForge has multiple issues with exporting UFO (and TTF files too, see lxgw/LxgwWenkaiGB#10) as you had experienced. The font author @lxgw is using FontCreator which do support exporting UFO in FC 14, but the exported UFO have some problems too (features.fea doesn't always follow Adobe syntax, exported in UTF-8 with BOM, missing critical localised names) which is why he's using FontForge to export UFO.

The best font editing software that directly exports UFO correctly is FontLab, but they don't plan to support vertical metrics yet. Personally I had written a simple program to extract UFO from binary font files (https://github.com/NightFurySL2001/py-fonttools-script/blob/main/convert_ufo.py), but it takes quite a long time and so he end up not using that too.

---

We have a few comments for the new UFO files:

1. The localised names are incorrect. 0x804 (2052) should be Simplified Chinese 霞鹜文楷 TC and 霞鹜文楷等宽 TC, while 0x404 (1028), 0xC04 (3076) and 0x1404 (5124) should be Traditional Chinese 霞鶩文楷 TC and 霞鶩文楷等寬 TC. The original files I've provided are using the correct localised names.
2. PANOSE information seem to be missing (all 0) in the binary font file. PANOSE value of (2-x-x-9-) and xAvgCharWidth are required for the font to be correctly identified as monospaced font in Windows (eg. in Command Prompt).
3. We need the openTypeOS2CodePageRanges values to be only CP1252 Latin 1 and CP950 Traditional Chinese for correctly identifying the font as Traditional Chinese font. Multiple codepages will affect Windows to treat it as generic language.

Additional question from @lxgw:

1. It seems that U+00AD SOFT HYPHEN is removed, is there any reason for the removal?
2. What are the 11 newly added characters in support for? (U+04F5, 2C64, 2C65, 2C66, A78D, A7AA, A7AD, A7AE, A7B2, A7B3, A7B4)

[aaronbell]

Sheesh. And here we thought UFO would be the true unifier of the font production world! I'm honestly not sure of the best way, then, to manage a UFO-based system, except to do what we've done here where the core files that contain the necessary metadata are, in a way, locked, and the UFO that @lxgw is working on acts more like a glyph repository than a source file. Anyway, I think we have something now that will work for our purposes.

---

localized names

Thanks for catching that. I'd missed the slight variation between the Simplified and Traditional names (I copied from the files you sent). Corrected.

PANOSE

Apparently this is a limitation of FontTools that clears PANOSE values from "interpolated" instances. I've had to re-add it in post-production since there is currently no way to implement it directly. I've also removed it from the UFO source to keep things cleaner.

codePages

Added. I expected that FT would add it automatically like it does with unicode ranges, but seems not. Added in the UFO source.

SOFT HYPHEN

Font Bakery recommends removal of this glyph:

it is recommended to not include it in the font at all, because discretionary hyphenation should be handled at the level of the shaping engine, not the font. Also, even if present, the software would not display that character.

11 newly added characters:
Another FB check looks to confirm that each capital or lowercase form has a corresponding lowercase or capital form (respectively). These 11 added glyphs is to complete the missing pairs.

Uploaded the updated files and double-checking the validations on my end.

[aaronbell]

Hi all, wanted to check in if there's any remaining feedback on the font. I need to do a PR to Google Fonts soon.

Thanks!

[NightFurySL2001]

The author has checked and noted that the separated UFO and UFOZ are using different contour direction in the final font file. Latin are clockwise and common are counter-clockwise.

We had a discussion on the codepage issue and he suggest to revert it back to automatic calculation since it provided more support for languages. This is a fault on my side.

[aaronbell]

No problem.

On the path direction issue, I will correct on my end but please let the designer know that his files have the path directions set incorrectly to standard and need to be reversed. Should the common file be updated without that correction done, then there will be a mismatch again.

[aaronbell]

Done. Thanks

[NightFurySL2001]

So just to confirm, use counter-clockwise/black on left?

The sources and editing are in TrueType (quadratic) outlines, and according to TrueType specs they should be clockwise/black on right, which is what the font author did. In the UFO before this merge request:

[aaronbell]

This was a bit confusing to me and with some research I discovered that Glyphs' default mode of setting path direction is CCW, and that Glyphs will correct the path direction on export. It doesn't assume one wants to have CW for quadratic curves, for some reason. My error on that.

I'll change the path direction in the differentiated ones to match the common files. No changes needed on your end.

[NightFurySL2001]

Thanks, is the files updated? We're ready to merge soon after verifying the path direction are fixed.

[aaronbell]

Oh. It looks like I never uploaded the new files. Sorry!

[NightFurySL2001]

I think we got a problem... did fontmake/gftools builder assumed all inputs are cubic/Bezier and CCW? Even when the source files are all CW, the output TTF is CCW.

My guess is fontmake/gftools builder uses cubic/Bezier internally, and didn't reverse contour direction when loading the UFO and converting it from quadratic to cubic, meaning internally fontmake is keeping cubic curves at CW. Then, when outputting TTF, the contour type and direction are both reversed, making it quadratic curve at CCW.

[lxgw]

As the author of this project, I'd like to sincerely thank you for all the effort you've put into this project. Now I merge this request.

[NightFurySL2001]

I think we got a problem... did fontmake/gftools builder assumed all inputs are cubic/Bezier and CCW? Even when the source files are all CW, the output TTF is CCW.

My guess is fontmake/gftools builder uses cubic/Bezier internally, and didn't reverse contour direction when loading the UFO and converting it from quadratic to cubic, meaning internally fontmake is keeping cubic curves at CW. Then, when outputting TTF, the contour type and direction are both reversed, making it quadratic curve at CCW.

@aaronbell we might need your help to check on this (non) issue.

[NightFurySL2001]

@aaronbell can you help to check why bumpfontversion in GitHub Actions isn't reflected in the build release font files? The test updates are in https://github.com/NightFurySL2001/LxgwWenkaiTC .

[aaronbell]

I’ll be able to look into these items probably in a few weeks or so once I roll back onto Google projects :)

[NightFurySL2001]

@aaronbell actually never mind, the problems are found: reverseOutlineDirection: False for TTF direction, and (quite silly) build fonts going into fonts/ttf by default in fontmake, but the repo uses fonts/TTF.

The updates are in the test-v1.333 branch of this repo now.