float type 'dd' unknown - qd already installed

[giblinjp]

I installed qd using make install, and then after that built g6k (I also modified bootstrap.sh to add the --with-qd argument. I confirmed that fpylll correctly grabbed it; inside of the config files, HAVE_QD is set to true after build. As far as I can tell from fplll's logs, it also recognized the presence of qd during build. However, when running bkz on a large lattice (yes, I increased MAX_SIEVING_DIM), I still get the error "Float type 'dd' unknown". I can't see what would cause that, as both libraries clearly recognize its presence.

Yes this is most likely a me problem not a library problem but I'm not sure where to go to resolve this issue.

[malb]

Do you have any error logs etc? As in "who" is complaining "Float type 'dd' unknown" ? Maybe G6K didn't pick it up? Have you tried constructing a MatGSO object with that float type?

[joerowell]

BTW: this is unlikely a G6K issue directly. I think we only use dd via fpylll for e.g LLL reduction.

[giblinjp]

Do you have any error logs etc? As in "who" is complaining "Float type 'dd' unknown" ? Maybe G6K didn't pick it up? Have you tried constructing a MatGSO object with that float type?

Traceback (most recent call last):
File "/home/giblinjp/g6k/401-fully-auto.sage.py", line 19, in
g6k = Siever(A, SieverParams(threads=_sage_const_24 ))
File "g6k/siever.pyx", line 85, in g6k.siever.Siever.init
M = self.MatGSO(M, float_type=float_type)
File "g6k/siever.pyx", line 127, in g6k.siever.Siever.MatGSO
M = GSO.Mat(A, float_type=float_type,
File "src/fpylll/fplll/gso.pyx", line 175, in fpylll.fplll.gso.MatGSO.init
File "src/fpylll/util.pyx", line 54, in fpylll.util.check_float_type
ValueError: Float type 'dd' unknown.

I am working on a Matrix of dimensions higher than 200 (yes it's technically "not supported" but it should just work; i properly rebuilt the library with an increased MAX_SIEVING_DIM). G6K has a conditional in siever.pyx that switches over to using float type 'dd' if the matrix is larger than 200 in dimension. The implication here is that fplll does not recognize float type 'dd', which is strange, since as far as I can tell, my system installation and the g6k installation both incorporated the optional addition of 'qd' float types (which includes 'dd' type). I can get you more specifics in a bit.

[malb]

Does, say, this work?

from fpylll import IntegerMatrix, GSO
B = IntegerMatrix.random(5, 'uniform', bits = 8)
M = GSO.Mat(B, float_type="dd")

[giblinjp]

Can confirm that produces the same error. To me this would imply that fpylll would have "HAVE_QD" set to false, but everything inside of fpylll has HAVE_QD set to true.

[malb]

Looks like there's a mismatch somewhere between what one copy/part of FPyLLL believes and what some other copy/part believes? Sorry, I'm clearly merely stating the obvious here.

[malb]

import fpylll
fpylll.config.have_qd

returns True or False?

[giblinjp]

I can confirm that the above command returns false. What confuses me, however, is that g6k's local build of fplll reports finding qd, the fpylll install within g6k also reports finding qd during building process, and I built and installed a copy of fplll before installing sagemath that reports finding qd installed. It's almost as if sagemath is installing its own copy of fpylll, and that version isn't finding qd. I am going to try building sagemath by hand next, instead of relying on apt-get.

Below is a "script" of commands / tasks I am performing in a clean Ubuntu 22.04 vm that reliably reproduces the False return from fpylll.config.have_qd:

sudo apt-get install -y libtool-bin
sudo apt-get install -y autoconf
sudo apt-get install -y build-essential
sudo apt-get install -y git

wget https://www.davidhbailey.com/dhbsoftware/qd-2.3.24.tar.gz
tar -xf qd-2.3.24.tar.gz
cd qd-2.3.24
./configure --enable-shared=yes
sudo make install
cd ..

git clone https://github.com/fplll/fplll
cd fplll
./configure --with-qd
sudo make install
cd ..

sudo apt-get install -y sagemath
sage
import fpylll
fpylll.config.have_qd

As can be seen, the issue very clearly doesn't stem from g6k at this point. If you want, I can close this and move over to fplll/fpylll for support.

[malb]

Mhhh, not sure we can do much about this also over at FPLLL/FPyLLL as it seems to be about how Debian/Ubuntu builds FPLLL?

[giblinjp]

That feels wrong to me, as building fplll / fpylll myself on ubuntu reports found qd as yes during build process. It almost feels like sagemath in specific is installing its own pre-built version of fpylll that isn't checking for qd, which makes me think I should try building sagemath by hand from source code.

However, If you think the distro is building things wrong, you probably know more about this than me. What should I try building on?

[malb]

FWIW I've installed SageMath in a conda environment and there it comes with qd:

$ ldd /home/malb/.conda/envs/sagemath/lib/libfplll.so.8.0.1
...
libqd.so.0 => /home/malb/.conda/envs/sagemath/lib/./libqd.so.0 (...)
...

Maybe you can check "who" installed what FPLLL without QD built in?

[giblinjp]

I finally got this resolved, it took me a good long while and multiple re-builds of sagemath by hand.

My initial setup was intentionally installing SageMath 9.5; inside of the build files, this has a discrete range of versions for each dependancy listed inside of the build files. If none are found, it will download and install itself, the latest version of each within that range. Sage 9.5's "latest" version of fplll will not accept QD, no matter what you do; build by hand, manually add configure arguments to the sage build paths, etc. I suppose it might've been possible to modify the range to accept a more recent version, but I chose an alternative route.

SageMath 9.8 and more recent have the fplll range set to more recent, and that version of fplll will recognize QD installed on the system. Actually, if you go recent enough, fplll seems to auto-bundle QD, meaning you don't even need it on your system to have QD support - however, if you go too recent, you run into a python issue with g6k under default configurations.

On SageMath 10.3, a standard installation will ALWAYS have QD built in to fplll. However, g6k will fail to run "SieverParams" correctly, because it relies on deprecated pkg_config methods. This will actually still run, but it will just drop any aruments given, meaning I could not input arguments, like multi-threaded.

My final solution was to install QD, and then download the source for sagemath 9.8, and build it. Then, install g6k on top of this. This alleviated all issues I was having.

Thank you for your support on this issue, I'm not particularly experienced on compiling and troubleshooting open-source niche software.