Add ccache as an optional spkg

[ohanar]

This is needed to eliminate the majority of the build time when switching between various branches in the git based workflow. It is also useful in the short run for dramatically speeding up the build process for many spkgs.

Installation:

• Add the spkg http://wstein.org/home/ohanar/spkgs/ccache-3.1.8.spkg to spkg/standard
• Apply attachment: trac13032_root.1.patch to the root repository
• Apply attachment: trac13032_scripts.patch to the scripts repository
• Apply attachment: trac13032_doc.patch to the sage library

CC: @kini @robertwb @ppurka @jdemeyer

Component: packages: optional

Keywords: sd40.5

Author: R. Andrew Ohana

Reviewer: Punarbasu Purkayastha

Merged: sage-5.6.beta2

Issue created by migration from https://trac.sagemath.org/ticket/13032

[ohanar]

comment:1

Not yet ready, the build scripts still need to be modified to set CC and CXX once this is installed.

[ohanar]

Dependencies: #13040

[ohanar]

Changed keywords from none to sd40.5

[ohanar]

Changed dependencies from #13040 to #13040 #13044

[ppurka]

comment:13

Two general comments:

1. We probably need to set the max cache size. On my system the default is 1G, which some people may dislike. It can be set using ccache -M <size>{G,M,K}, where the letters G, M, K can be used to specify the usual gigabytes, megabytes and kilobytes. This needs to be set only if CCACHE_DIR is empty, since the user may have his/her own ccache settings.
2. This fact about the ccache size should be mentioned in the documentation, so that the user knows how much of disk space will be used.

[ohanar]

comment:14

Replying to @ppurka:

Two general comments:

1. We probably need to set the max cache size. On my system the default is 1G, which some people may dislike. It can be set using ccache -M <size>{G,M,K}, where the letters G, M, K can be used to specify the usual gigabytes, megabytes and kilobytes.

This is set in the spkg-install to 3G.

This needs to be set only if CCACHE_DIR is empty, since the user may have his/her own ccache settings.
2. This fact about the ccache size should be mentioned in the documentation, so that the user knows how much of disk space will be used.

Both good points, I'll see about changing these.

[ohanar]

Changed dependencies from #13040 #13044 to none

[ohanar]

comment:21

1. The f90cache source code does not include a copy of its license. Since it is based off of and old version of ccache, it must be GPL licensed, so it is in violation of GPL license. For this reason, I'm going to remove it from this ticket.
2. ccache depends on zlib and includes its own copy to use if it can't find one. I've updated the spkg and build scripts to ensure that it uses our copy.

[ppurka]

comment:46

Replying to @jdemeyer:

Why not use the default directories?

Because it keeps the sage files in a fixed sage directory instead of the home directory of the user. The default sizes of those directories are huge (4G) and are needed because Sage itself is huge. I am already using 3.5G of that.

~/.sage/ccache» du -sh .
3.5G	.

[haraldschilly]

comment:48

ccache spkg is on the server

[jdemeyer]

Merged: sage-5.6.beta2

[jdemeyer]

comment:50

I still don't quite like how this patch uses a separate CCACHE_DIR inside .sage. Because now, I have:

$ ccache -s
cache directory                     /home/jdemeyer/.ccache
[...]
cache size                           3.2 Gbytes
max cache size                       4.0 Gbytes

$ ./sage --sh
(sage-sh) ccache -s
cache directory                     /home/jdemeyer/.sage//ccache
[...]
cache size                         935.1 Mbytes
max cache size                       1.0 Gbytes

I would like to have just one .ccache directory, especially since my default ccache dir is already used for Sage. I don't think we should add 1GB of data inside the home directories of unsuspecting users.

Also note that the default size of 1GB is too little anyway to be useful for Sage.

[ppurka]

comment:51

You can of course have only one ccache dir. In that case, simply define the environment variable CCACHE_DIR in your shell rc file. Then everything will use that directory. The default is set for those people who don't bother with setting up ccache variables.

~» sage --sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/punarbasu/Installations/sage-5.6.beta1
~» ccache -s
cache directory                     /home/punarbasu/.sage//ccache
cache hit (direct)                  1620
cache hit (preprocessed)           13825
cache miss                         16881
called for link                     3066
called for preprocessing            1553
multiple source files                 20
compiler produced stdout               2
compile failed                       389
preprocessor error                   236
bad compiler arguments               285
unsupported source language         1214
autoconf compile/link               3228
unsupported compiler option          786
no input file                       1345
files in cache                     53031
cache size                           2.7 Gbytes
max cache size                       4.0 Gbytes
~»
Exited Sage subshell.



~» export CCACHE_DIR=$HOME/.ccache
~» sage --sh

Starting subshell with Sage environment variables set.  Don't forget
to exit when you are done.  Beware:
 * Do not do anything with other copies of Sage on your system.
 * Do not use this for installing Sage packages using "sage -i" or for
   running "make" at Sage's root directory.  These should be done
   outside the Sage shell.

Bypassing shell configuration files...

Note: SAGE_ROOT=/home/punarbasu/Installations/sage-5.6.beta1
~» ccache -s
cache directory                     /home/punarbasu/.ccache
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
files in cache                         0
cache size                             0 Kbytes
max cache size                       1.0 Gbytes

[jdemeyer]

comment:52

Replying to @ppurka:

The default is set for those people who don't bother with setting up ccache variables.

Not really. I didn't bother setting CCACHE_DIR (since I like the default) but I don't like this ticket's defaults. It makes it harder to configure ccache for example (it needs to be done inside a Sage shell) and I still don't understand why somebody would want a separate ccache directory for Sage.

[jhpalmieri]

comment:53

Maybe there should be a new environment variable SAGE_CCACHE_DIR. If it's not set (the default), then use CCACHE_DIR. Otherwise, use it, in case someone wants to use a separate ccache directory for Sage.

ppurka, you also didn't address the comment about 1GB being "too little anyway to be useful for Sage".

[jdemeyer]

comment:54

I don't think we should introduce yet another environment variable. Users wishing to set a Sage-only ccache dir can use .sage/sagerc

[ppurka]

comment:55

I don't understand what the confusion is about. There are only two possibilities.

1. If you set CCACHE_DIR in your shell's rc file, then that variable will be used and none of your ccache settings will be changed. Here is an example on a test account where I exported the setting CCACHE_DIR=~/.ccache explicitly before compiling sage. Then I stopped the compilation of sage midway and looked at the output of ccache -s from a different terminal. All of this is outside of sage shell.

test@ub2 ~ $ ccache -s
cache directory                     /tmp/test/.ccache
cache hit (direct)                     1
cache hit (preprocessed)               2
cache miss                           139
called for link                        3
called for preprocessing              46
compile failed                         5
preprocessor error                     5
bad compiler arguments                 4
autoconf compile/link                 75
no input file                         18
files in cache                       288
cache size                           3.6 Mbytes
max cache size                       1.0 Gbytes

2. If you do not provide a ccache setting at all, then the package sets the ccache size and the ccache directory to some defaults. Since, you are not showing any preference here, the ccache directory is set to ~/.sage/ccache and the size is set to 4G. All sage related stuff is in one single directory ~/.sage by default - I don't understand why we should go out of our way to mix it with user's own settings in their home directories.

[jdemeyer]

comment:56

Replying to @ppurka:

If you do not provide a ccache setting at all [...] the size is set to 4G.

That's already false, the default is 1GB.

All sage related stuff is in one single directory ~/.sage by default - I don't understand why we should go out of our way to mix it with user's own settings in their home directories.

Because I don't consider cache compilation files to be Sage-related.

Imagine I want to compile a package X outside of Sage, a package which happens to be part of Sage (I do that with PARI for example). Then it's a pity that the cached object files for package X appear twice in my home directory: once in .ccache and once in .sage.

My problem really boils down to this: why would you want a separate ccache directory for Sage?

[jdemeyer]

comment:57

Replying to @ppurka:

Since, you are not showing any preference here

I would say that, by not setting CCACHE_DIR, I am showing a preference for the default ccache directory $HOME/.ccache

I think that setting a variable explicitly to its default value should be the same as not setting it at all.

[ppurka]

comment:58

Replying to @jdemeyer:

Replying to @ppurka:

If you do not provide a ccache setting at all [...] the size is set to 4G.

That's already false, the default is 1GB.

I must reiterate. If you don't provide any ccache directory settings at all, then it is set to 4GB with this ticket/spkg.

Imagine I want to compile a package X outside of Sage, a package which happens to be part of Sage (I do that with PARI for example). Then it's a pity that the cached object files for package X appear twice in my home directory: once in .ccache and once in .sage.

Well, then you set your ccache environment variable appropriately. To use ccache while compiling with sage you need to set SAGE_INSTALL_CCACHE=yes in your environment variable. Then, it will use whatever ccache directory you have set and otherwise it will use the default of ~/.sage/ccache.

My problem really boils down to this: why would you want a separate ccache directory for Sage?

In fact, one question that I would ask is - what other sage setting is present outside the ~/.sage directory? If there is, then it makes sense to mix and match sage specific settings and folders with system/user specific settings.

[kini]

comment:59

You might want to put this question to sage-devel if you haven't already. I think there might be people on both sides of the argument.

On the one hand I agree that setting a variable explicitly to its default value should be the same as not setting it at all. On the other hand, I feel like I'd want a separate ccache directory for Sage because I would like all Sage artifacts to be segregated from other stuff on my system, as they currently are by default. Currently, Sage will never modify anything outside its directory except /tmp and $DOT_SAGE unless you specifically write code that does so (right?). Not having a separate Sage ccache directory would add ~/.ccache to the list.

A compromise/workaround would be to, say, symlink ~/.sage/ccache to ~/.ccache manually if you want to share caches. Is this too much of a burden, do you think, Jeroen? The SPKG installer could print a message about this possibility, or something.

[jdemeyer]

comment:60

Replying to @ppurka:

I must reiterate. If you don't provide any ccache directory settings at all, then it is set to 4GB with this ticket/spkg.

OK, so it's something specific to this spkg. When using system ccache, I still get 1GB of cache inside .sage, even if I manually configured ccache to use 4GB.

To use ccache while compiling with sage you need to set SAGE_INSTALL_CCACHE=yes in your environment variable.

Again false, as a system ccache might be used.

In fact, one question that I would ask is - what other sage setting is present outside the ~/.sage directory? If there is, then it makes sense to mix and match sage specific settings and folders with system/user specific settings.

Probably Sage's readline uses non-Sage-specific configuration files ($HOME/.inputrc).

[jdemeyer]

comment:61

Replying to @kini:

A compromise/workaround would be to, say, symlink ~/.sage/ccache to ~/.ccache manually if you want to share caches. Is this too much of a burden, do you think, Jeroen? The SPKG installer could print a message about this possibility, or something.

Of course it's not much burden for me, but what about other people in the same situation? This patch changes the ccache directory and the cache size behind people's back if they were using system ccache before. So this patch can make performance strictly worse without users being aware of it.

[jdemeyer]

comment:62

Replying to @kini:

Not having a separate Sage ccache directory would add ~/.ccache to the list.

Which is a problem because ...........?

[kini]

comment:63

Probably Sage's readline uses non-Sage-specific configuration files ($HOME/.inputrc).

Does Sage change ~/.inputrc or just read it? Sage of course reads tons of files all over the system. An example of another dotfile in ~ that Sage (or Sage's Mercurial anyway) reads is ~/.hgrc. But it will change ~/.ccache.

Of course it's not much burden for me, but what about other people in the same situation?

As I said, we can print a note at the end of the SPKG installation process, or otherwise document it. I suspect that most users will not mind that the ccache directory ~/.sage/ccache is used, and if they do, it is probably because they are already running ccache on their system and want to consolidate the cache directories together, so they're probably savvy enough to make a symlink.

This patch changes the ccache directory and the cache size behind people's back if they were using system ccache before.

Sage installs another copy of Python on the system "behind people's back" as well and nobody complains (or at least we don't listen to such complaints). I don't see the problem.

Also, maybe I am misunderstanding something. Are you saying that installing this SPKG into Sage will change the behavior of the system ccache running independently of Sage?

Which is a problem because ...........?

For one thing, it means more detritus around the system to clean up if you want to remove Sage from your system. And very large unwanted detritus at that, if the user doesn't have a system ccache. Of course, it's an optional package, so we can assume the user at least knows they have installed the ccache SPKG, but most users would expect that they can completely eradicate Sage and all its artifacts by deleting the Sage directory and $DOT_SAGE.

Also, this kind of lack of compartmentalization just seems unhygienic. I kind of don't like the idea of Sage interacting with and modifying data that belongs to other programs which I've installed systemwide. I like that Sage is a sandbox mini-distribution of software which is self-contained. (Or, at least that's the paradigm the official Sage distribution has conformed to so far, sage-on-gentoo and sage-on-debian notwithstanding.)

[jdemeyer]

comment:64

Replying to @kini:

Also, maybe I am misunderstanding something. Are you saying that installing this SPKG into Sage will change the behavior of the system ccache running independently of Sage?

No, that's not the case at all. The case I am considering is the opposite: not installing this SPKG into Sage but using a system-wide ccache.

[jdemeyer]

comment:65

Here is a proposal for a compromise: use $HOME/.ccache if using a system-wide ccache installation and use $DOT_SAGE/ccache if using Sage's ccache, which we test by

if [ -x "$SAGE_LOCAL/bin/ccache" ];

[jdemeyer]

comment:66

Replying to @kini:

I kind of don't like the idea of Sage interacting with and modifying data that belongs to other programs which I've installed systemwide.

I'd argue that in this case, the interaction is a feature, not a bug.

And very large unwanted detritus at that, if the user doesn't have a system ccache

But I do have system-wide ccache. For me, this patch adds 1GB of extra detritus in .sage/ccache and actually reduces performance by using a smaller cache.

[ppurka]

comment:67

I think what you are really asking about is

What should the zero-configuration scenario, even for advanced users, look like?

I count you as an advanced user since you are using a user-wide ccache, and you don't want sage to use any other ccache directory by default. On top of that, you want sage to use the ccache directory that you are already using, which may or may not be the default ccache directory.

I think this question should be asked to the general audience in sage-devel. There might be people who keep the ~/.sage directory separate, along with the rest of their sage installation. For a heavy user of sage or for someone running a server, I can contemplate that the ~/.sage (or DOT_SAGE) will not be small and there might be people who are using a different (maybe mounted) directory that is separate from their home directory.

[jdemeyer]

comment:68

Replying to @ppurka:

I think this question should be asked to the general audience in sage-devel.

Done.

[jhpalmieri]

comment:69

Since the spkg is pretty small, would it make sense to distribute it with Sage? It would still only be built if SAGE_INSTALL_CCACHE were set, so it would still be in some sense optional. But then one wouldn't need internet access when building Sage, and the install log for ccache would have a more informative name: it would include the version number.

[jdemeyer]

comment:70

Replying to @jhpalmieri:

Since the spkg is pretty small, would it make sense to distribute it with Sage? It would still only be built if SAGE_INSTALL_CCACHE were set, so it would still be in some sense optional.

I don't mind.

[jdemeyer]

comment:71

I created a patch at #13938 to not set CCACHE_DIR, needs review.