Add builder for Graphviz #351

NHDaly · 2019-12-30T16:07:32Z

So far, I've only managed to get a few basic linux platforms building.
I'll look into the rest, next.

G/Graphviz/build_tarballs.jl

NHDaly · 2019-12-30T16:12:23Z

I did have a question about this that hopefully someone (maybe @giordano?) can answer?:

I got the list of dependencies from the Requirements listed on Graphviz's site:
https://graphviz.gitlab.io/_pages/Download/Download_source.html

However, it seems that graphviz has listed recursive dependencies, so many of these (e.g. freetype) are only listed because they're a dependency of pango.

Do we usually follow that same structure in these packages, or is it better to just list the "top level" dependencies we have (which are I think just Pango, Cairo, and Expat)?

giordano · 2019-12-30T16:15:13Z

Do we usually follow that same structure in these packages, or is it better to just list the "top level" dependencies we have (which are I think just Pango, Cairo, and Expat)?

List only the direct dependencies, each of them will automatically bring in also its own dependencies.

giordano · 2019-12-30T16:20:07Z

Just by looking to what are the successful platforms my guess is that it's trying to run foreign excutables (i686-linux-gnu, x86_64-linux-gnu and x86_64-linux-musl are the only platforms for which the BB environment can run executables).

Edit: well, it looks like that my guess is wrong, but the coincidence is interesting 😅 It turned out my guess was correct.

G/Graphviz/build_tarballs.jl

giordano · 2020-01-01T15:38:36Z

As I suspected earlier, this is trying to run foreign executables 😞

For example, on aarch64-linux-gnu:

sandbox:${WORKSPACE}/srcdir/graphviz-2.42.3 # make 
make  all-recursive
[...]
Making all in gvpr
make[3]: Entering directory '/workspace/srcdir/graphviz-2.42.3/lib/gvpr'
./mkdefs gdefs.h < ../../lib/gvpr/gprdata
./mkdefs: line 1: ELF��
                       @@xG@8@@@@@@���@�@@����A�A����A�A���@�@: not found
./mkdefs: line 1: syntax error: unexpected end of file (expecting ")")
make[3]: *** [Makefile:1081: gdefs.h] Error 2
make[3]: Leaving directory '/workspace/srcdir/graphviz-2.42.3/lib/gvpr'
make[2]: *** [Makefile:571: all-recursive] Error 1
make[2]: Leaving directory '/workspace/srcdir/graphviz-2.42.3/lib'
make[1]: *** [Makefile:816: all-recursive] Error 1
make[1]: Leaving directory '/workspace/srcdir/graphviz-2.42.3'
make: *** [Makefile:623: all] Error 2
sandbox:${WORKSPACE}/srcdir/graphviz-2.42.3 # file lib/gvpr/mkdefs
lib/gvpr/mkdefs: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

giordano · 2020-01-01T16:14:26Z

After the patch I added, compilation is successful but make install is running one of the executables -- dot. 😑

giordano · 2020-01-01T16:19:44Z

For the record, it's likely that for Windows we'll need at least some of these patches: https://github.com/msys2/MINGW-packages/tree/master/mingw-w64-graphviz. For FreeBSD there are these patches: https://svnweb.freebsd.org/ports/head/graphics/graphviz/files/

giordano · 2020-01-01T18:09:31Z

G/Graphviz/build_tarballs.jl

+atomic_patch -p1 ../patches/gvpr-build-native-mkdefs.patch
+
+# This patch disable generation of dot's configuration
+atomic_patch -p1 ../patches/do-not-build-dot-config.patch


make install would try to run the built dot binary executable in order to generate the configuration file. This is a no-go within our environment as we can't run foreign executables apart from three platforms, so we necessarily have to skip this step.

The file could be generated by the Julia package using this JLL package, like what we do in Gtk.jl, however the location of the configuration is tricky: this file must be in the same directory as the plugin libraries -- they're in the lib/graphviz directory of the tarball.

The location of this directory is determined by the gvconfig_libdir function in lib/gvc/gvconfig.c of the source code. By default this directory should be something like ../lib/graphviz, but this can be overridden by setting the environment variable GVBINDIR. From dot manual:

GVBINDIR
Indicates which directory contains the Graphviz config file and plug-in libraries. If it is defined, the value overrides any other mechanism for finding this directory. If Graphviz is properly installed, it should not be needed, though it can be useful for relocation on platforms not running Linux or Windows.

I think we can generate the config file to a mutable artifact directory and copy there also the plugin libraries, then set GVBINDIR to this directory.

whistles yeah i think that makes sense. Reallly great finds! :D

Your patch for building mkdefs is really cool. This stuff is neat! I'm impressed at how it all comes together! 😁

giordano · 2020-01-01T23:18:49Z

G/Graphviz/bundled/patches/gvpr-build-native-mkdefs.patch

+ mkdefs$(EXEEXT): $(mkdefs_OBJECTS) $(mkdefs_DEPENDENCIES) $(EXTRA_mkdefs_DEPENDENCIES) 
+ 	@rm -f mkdefs$(EXEEXT)
+-	$(AM_V_CCLD)$(LINK) $(mkdefs_OBJECTS) $(mkdefs_LDADD) $(LIBS)
+	$(AM_V_CCLD)$(LIBTOOL) $(AM_V_lt) --tag=CC $(AM_LIBTOOLFLAGS) \


For the record, what I did here was to replace $(LINK) with its value in this Makefile and then replace $(CCLD) with $(CC_FOR_BUILD), which is the native C compiler. In addition I added a rule above to build mkdefs.o with the native compiler as well.

NHDaly · 2020-01-25T17:16:45Z

For the record, it's likely that for Windows we'll need at least some of these patches: https://github.com/msys2/MINGW-packages/tree/master/mingw-w64-graphviz. For FreeBSD there are these patches: https://svnweb.freebsd.org/ports/head/graphics/graphviz/files/

Do you have a usual way to handle patches like these that are already in source control? Can we add the patches repos as dependencies and then use them in the build somehow? Or just manually clone them in our build script?

giordano · 2020-01-25T17:20:42Z

Do you have a usual way to handle patches like these that are already in source control? Can we add the patches repos as dependencies and then use them in the build somehow? Or just manually clone them in our build script?

Those repo are huge, I much prefer to pick up the only patches we need, also because they may not apply cleanly.

For the specific case of Windows, I already have a bunch of patches, I'll need to push them here, but compilation still fails, there are a lot of more errors.

NHDaly · 2020-01-25T17:38:48Z

Those repo are huge

Yeah, it's true. I was just looking at https://stackoverflow.com/questions/7106012/download-a-single-folder-or-directory-from-a-github-repo, which seems useful (svn checkout https://github.com/msys2/MINGW-packages/trunk/mingw-w64-graphviz), but okay yeah manually copying the patches probably makes sense 👍)

For the specific case of Windows, I already have a bunch of patches, I'll need to push them here, but compilation still fails, there are a lot of more errors.

Awesome, thanks for the continued work on this! :) I'll try bringing in one of the freebsd patches now then!

NHDaly · 2020-01-25T17:45:05Z

Actually, for freebsd, it looks like it was failing to build fontconfig?:

[17:04:46] libtool: link: ( cd ".libs" && rm -f "libgvplugin_pango_C.la" && ln -s "../libgvplugin_pango_C.la" "libgvplugin_pango_C.la" )
[17:04:47] libtool: link: cc -shared  -fPIC -DPIC  .libs/gvplugin_pango.o .libs/gvrender_pango.o .libs/gvgetfontlist_pango.o .libs/gvtextlayout_pango.o .libs/gvloadimage_pango.o   -Wl,-rpath -Wl,/workspace/srcdir/graphviz-2.42.3/lib/gvc/.libs -Wl,-rpath -Wl,/workspace/srcdir/graphviz-2.42.3/lib/xdot/.libs -Wl,-rpath -Wl,/workspace/srcdir/graphviz-2.42.3/lib/cgraph/.libs -Wl,-rpath -Wl,/workspace/srcdir/graphviz-2.42.3/lib/cdt/.libs -Wl,-rpath -Wl,/workspace/srcdir/graphviz-2.42.3/lib/pathplan/.libs -Wl,-rpath -Wl,/workspace/destdir/lib -Wl,-rpath -Wl,/workspace/destdir/lib -L/workspace/srcdir/graphviz-2.42.3/lib/xdot/.libs -L/workspace/srcdir/graphviz-2.42.3/lib/cgraph/.libs -L/workspace/srcdir/graphviz-2.42.3/lib/cdt/.libs -L/workspace/srcdir/graphviz-2.42.3/lib/pathplan/.libs -L/workspace/destdir/lib ../../lib/gvc/.libs/libgvc.so /workspace/srcdir/graphviz-2.42.3/lib/xdot/.libs/libxdot.so /workspace/srcdir/graphviz-2.42.3/lib/cgraph/.libs/libcgraph.so /workspace/srcdir/graphviz-2.42.3/lib/cdt/.libs/libcdt.so /workspace/srcdir/graphviz-2.42.3/lib/pathplan/.libs/libpathplan.so /workspace/destdir/lib/libpangocairo-1.0.so /workspace/destdir/lib/libcairo.so /workspace/destdir/lib/libpixman-1.so -lpng16 /workspace/destdir/lib/libxcb-shm.so /workspace/destdir/lib/libX11-xcb.so /workspace/destdir/lib/libxcb-render.so /workspace/destdir/lib/libXrender.so /workspace/destdir/lib/libXext.so /workspace/destdir/lib/libX11.so /workspace/destdir/lib/libxcb.so /workspace/destdir/lib/libXau.so /workspace/destdir/lib/libXdmcp.so -lrt /workspace/destdir/lib/libpangoft2-1.0.so /workspace/destdir/lib/libharfbuzz.so /workspace/destdir/lib/libpango-1.0.so /workspace/destdir/lib/libgthread-2.0.so -lfribidi /workspace/destdir/lib/libgobject-2.0.so /workspace/destdir/lib/libffi.so /workspace/destdir/lib/libglib-2.0.so /workspace/destdir/lib/libpcre.so -lpthread /workspace/destdir/lib/libintl.so /workspace/destdir/lib/libiconv.so /workspace/destdir/lib/libfontconfig.so /workspace/destdir/lib/libexpat.so /workspace/destdir/lib/libuuid.so /workspace/destdir/lib/libfreetype.so -lz -lbz2 -lm  -O2   -pthread -Wl,-soname -Wl,libgvplugin_pango.so.6 -o .libs/libgvplugin_pango.so.6
[17:04:47] /workspace/destdir/lib/libfontconfig.so: error adding symbols: File in wrong format
[17:04:47] clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
[17:04:47] make[3]: *** [Makefile:705: libgvplugin_pango.la] Error 1
[17:04:47] make[3]: Leaving directory '/workspace/srcdir/graphviz-2.42.3/plugin/pango'

https://dev.azure.com/JuliaPackaging/Yggdrasil/_build/results?buildId=849&view=logs&j=11cc9737-c3b2-5b2e-49c1-3ecd4dd3473d&t=6864626c-b206-5daa-5d51-831805962223&l=31282

I notice that the last commit for Pango didn't have CI results (7228735). Do we know if Pango is currently working on freebsd?
Maybe it's not?

giordano · 2020-01-25T17:46:59Z

Uh, that's a known error, we only need to rebuild Fontconfig. Do you mind opening a PR with a dummy change to fontconfig? Some libraries built in December have this problem, now fixed. For context: JuliaPackaging/BinaryBuilder.jl#612

NHDaly · 2020-01-25T17:51:17Z

Uh, that's a known error, we only need to rebuild Fontconfig. Do you mind to open a PR with a dummy change to fontconfig?

Oh cool, thanks.
Done!
#435

giordano · 2020-01-25T19:33:48Z

[19:26:46] libtool: link: (cd .libs/libgvplugin_neato_layout_C.lax/libsfdpgen_C.a && x86_64-unknown-freebsd11.1-ar x "/workspace/srcdir/graphviz-2.42.3/plugin/neato_layout/../../lib/sfdpgen/.libs/libsfdpgen_C.a")
[19:26:46] /opt/x86_64-linux-musl/x86_64-linux-musl/lib64/libstdc++.so: error adding symbols: File in wrong format
[19:26:46] clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
[19:26:46] make[3]: *** [Makefile:740: libgvplugin_neato_layout.la] Error 1

Why the linker is trying to use a library of the host system? 😩

giordano · 2020-01-25T19:46:42Z

I ran

grep -r musl

in the top directory of Graphviz after running configure when building for FreeBSD. There are almost 4000 (yes, four thousands) results. I want to cry.

NHDaly · 2020-01-25T20:31:38Z

.......... hahaha omg that's amazing! 😂

Mayyyyyyyybe the configure patches for freebsd will help?
https://svnweb.freebsd.org/ports/head/graphics/graphviz/files/
And/or is this what the update_configure_scripts is for?

giordano · 2020-01-25T20:48:29Z

I'm not sure this the real problem, but I found that the libtool script has

# Compile-time system search path for libraries.
sys_lib_search_path_spec="/opt/x86_64-linux-musl/lib/clang/9.0.1 /opt/x86_64-unknown-freebsd11.1/x86_64-unknown-freebsd11.1/sys-root/usr/lib "

For comparison, x86_64-linux-gnu has

# Compile-time system search path for libraries.
sys_lib_search_path_spec="/opt/x86_64-linux-gnu/lib/gcc/x86_64-linux-gnu/4.8.5 /opt/x86_64-linux-gnu/lib/gcc /opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64 /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/lib64 /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/lib64 "

so no reference to the build system.

Edit: no, that's probably not too relevant, when building for macOS sys_lib_search_path_spec has also the musl directory in it

NHDaly · 2020-02-25T14:48:58Z

I'm sad this cross-compilation has been so difficult. Do you think it would make sense to ask for help in some graphviz forum somewhere? or maybe in a freebsd and/or mingw forum? Have you found success with reaching out to other communities like this in the past?

giordano · 2020-02-25T15:47:40Z

Last time I had a look at this, the mingw build was actually compiling all the expected products, but failing to compile something else, so we could probably be on the right track. It's just that the road ahead is still long 😅 I don't remember anymore what was the problem for FreeBSD

I'm sad this cross-compilation has been so difficult. Do you think it would make sense to ask for help in some graphviz forum somewhere? or maybe in a freebsd and/or mingw forum? Have you found success with reaching out to other communities like this in the past?

That greatly varies. Some upstream communities are more active in helping people out, other are less likely to give you a hand.

NHDaly · 2020-05-04T13:55:45Z

I can't remember if i've already asked this, but since we're having so much trouble getting windows and freebsd to build, is it possible to merge with just the architectures we have so far? Or do you really try to avoid that?

It would be really great if we could ship graphviz so that people don't have to install it themselves for PProf (JuliaPerf/PProf.jl#16), and i wanted to start casually exporting some graphs as output from other packages as well, and hoped to use this.

giordano · 2020-05-04T14:02:43Z

is it possible to merge with just the architectures we have so far? Or do you really try to avoid that?

I think in general the decision of the platforms to support is more on the maintainer of the Julia package that will use the library

Or do you really try to avoid that?

I personally don't care much 😅 It's just that it would be nice to make it straightforward to install packages especially on the platform where this is usually more difficult 😂

So far, I've only managed to get a few basic linux platforms building. I'll look into the rest, next.

giordano · 2020-05-04T22:36:53Z

@NHDaly shall we merge? I have the feeling that we should list the LibraryProducts otherwise the executable may not be able to find them

NHDaly · 2020-05-09T01:28:45Z

Sigh, it's too bad.. it would've been awesome to support windows, but yeah i think this is better than nothing!

I have the feeling that we should list the LibraryProducts otherwise the executable may not be able to find them

Sure! I have to admit it's been so long since I looked at this (I started this PR in 2019!) that I really don't remember how it all works.. If that's something easy, I would definitely appreciate you doing it! ❤️ if not, i can have a look at it with you, yeah

giordano · 2020-05-10T16:39:32Z

Sigh, it's too bad.. it would've been awesome to support windows, but yeah i think this is better than nothing!

We can still try this later 😉

NHDaly mentioned this pull request Dec 30, 2019

Ship graphviz? JuliaPerf/PProf.jl#16

Open

giordano reviewed Dec 30, 2019

View reviewed changes

G/Graphviz/build_tarballs.jl Outdated Show resolved Hide resolved

NHDaly commented Dec 30, 2019

View reviewed changes

G/Graphviz/build_tarballs.jl Outdated Show resolved Hide resolved

giordano added the long shot 🏹 This is going to be fun label Dec 30, 2019

giordano reviewed Dec 31, 2019

View reviewed changes

G/Graphviz/build_tarballs.jl Outdated Show resolved Hide resolved

giordano force-pushed the nhd-Graphviz branch from ed414fb to 0dd3782 Compare January 1, 2020 16:12

giordano force-pushed the nhd-Graphviz branch from 0dd3782 to f05df7d Compare January 1, 2020 16:34

giordano reviewed Jan 1, 2020

View reviewed changes

giordano force-pushed the nhd-Graphviz branch from adeaae3 to a49ffd3 Compare January 25, 2020 18:47

staticfloat force-pushed the master branch from e111927 to f645547 Compare February 8, 2020 08:53

NHDaly and others added 8 commits May 4, 2020 22:33

Add builder for Graphviz

c62ee8e

So far, I've only managed to get a few basic linux platforms building. I'll look into the rest, next.

[Graphviz] Remove indirect deps; better setting of CFLAGS

5ed360d

[Graphviz] Upgrade to latest release

6d8cade

[Graphviz] Add patch to build native utility instead of foreign one

4d4770e

[Graphviz] Do not build dot's configuration

5294852

[Graphviz] Add some patches for Windows

696e5b1

[Graphviz] Do not build with -ffast-math

a6a5a15

[Graphviz] Use new types for sources and dependencies

e010f21

giordano force-pushed the nhd-Graphviz branch from 22a8c3d to e010f21 Compare May 4, 2020 21:36

[Graphviz] Expand C++ string ABI, build only for successful platforms

7215a5d

NHDaly marked this pull request as ready for review May 9, 2020 01:27

[Graphviz] Add more library products

edf50f5

giordano added Windows 🙄 Doesn't build for Windows FreeBSD 👿 Doesn't build for FreeBSD labels May 10, 2020

giordano merged commit 58fb026 into JuliaPackaging:master May 10, 2020

giordano mentioned this pull request Aug 5, 2020

Add Graph tests SciML/Catalyst.jl#237

Closed

NHDaly deleted the nhd-Graphviz branch September 18, 2020 03:58

amontoison mentioned this pull request Mar 6, 2021

[SCOTCH] Add builder #2652

Merged

giordano mentioned this pull request May 20, 2021

Broken Pango in Graphviz_jll #1428

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add builder for Graphviz #351

Add builder for Graphviz #351

NHDaly commented Dec 30, 2019

NHDaly commented Dec 30, 2019

giordano commented Dec 30, 2019

giordano commented Dec 30, 2019 •

edited

Loading

giordano commented Jan 1, 2020

giordano commented Jan 1, 2020

giordano commented Jan 1, 2020 •

edited

Loading

giordano Jan 1, 2020 •

edited

Loading

NHDaly Jan 1, 2020

giordano Jan 1, 2020

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020

NHDaly commented Jan 25, 2020

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020 •

edited

Loading

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020

giordano commented Jan 25, 2020

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020 •

edited

Loading

NHDaly commented Feb 25, 2020

giordano commented Feb 25, 2020

NHDaly commented May 4, 2020

giordano commented May 4, 2020

giordano commented May 4, 2020

NHDaly commented May 9, 2020

giordano commented May 10, 2020

Add builder for Graphviz #351

Add builder for Graphviz #351

Conversation

NHDaly commented Dec 30, 2019

NHDaly commented Dec 30, 2019

giordano commented Dec 30, 2019

giordano commented Dec 30, 2019 • edited Loading

giordano commented Jan 1, 2020

giordano commented Jan 1, 2020

giordano commented Jan 1, 2020 • edited Loading

giordano Jan 1, 2020 • edited Loading

Choose a reason for hiding this comment

NHDaly Jan 1, 2020

Choose a reason for hiding this comment

giordano Jan 1, 2020

Choose a reason for hiding this comment

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020

NHDaly commented Jan 25, 2020

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020 • edited Loading

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020

giordano commented Jan 25, 2020

NHDaly commented Jan 25, 2020

giordano commented Jan 25, 2020 • edited Loading

NHDaly commented Feb 25, 2020

giordano commented Feb 25, 2020

NHDaly commented May 4, 2020

giordano commented May 4, 2020

giordano commented May 4, 2020

NHDaly commented May 9, 2020

giordano commented May 10, 2020

giordano commented Dec 30, 2019 •

edited

Loading

giordano commented Jan 1, 2020 •

edited

Loading

giordano Jan 1, 2020 •

edited

Loading

giordano commented Jan 25, 2020 •

edited

Loading

giordano commented Jan 25, 2020 •

edited

Loading