Segmentation fault: 11

[vicpara]

I was trying to follow the documentation from https://github.com/castorini/pyserini#dense-indexes .

On MacOS 12.6 Monterey, when running the script below I get a Segmentation fault error.

python -m pyserini.encode \
  input   --corpus data/collections/simple_cacm_corpus.json \
          --fields text \
  output  --embeddings  embeddings/small-dense-1668560951 \
  encoder --encoder  castorini/tct_colbert-v2-hnp-msmarco \
          --fields text \
          --batch 8 \
          --fp16

returns
Segmentation fault: 11

[lintool]

Faiss dense retrieval is memory intensive. How much RAM do you have on your machine?

[vicpara]

I run MacOS, 3.6 GHz 10-Core Intel Core i9, 128Gb, no cuda device.
Removed pip and conda, reinstalled as advised and to no success. I keep getting the same error.

[vicpara]

I tried to reproduce the error in a docker container.

FROM continuumio/anaconda3

RUN conda init bash && \ 
    conda create -n pyserini python=3.9

SHELL ["conda", "run", "-n", "pyserini", "/bin/bash", "-c"]

# RUN conda init bash && conda activate -y pyserini 

RUN pip install --upgrade pip
RUN conda install -y -c conda-forge openjdk=11

RUN conda install -y wget
RUN conda install -y -c conda-forge lightgbm

SHELL ["conda", "run", "--no-capture-output", "-n", "pyserini", "/bin/bash", "-c"]

RUN pip install pyserini
RUN conda install pytorch torchvision torchaudio cpuonly -c pytorch
RUN conda install faiss-cpu -c pytorch
RUN pip install torch

RUN mkdir /data && \
    mkdir /embeddings

COPY data/collections/my-small-dense/simple_cacm_corpus.json /data/
RUN curl https://github.com/castorini/pyserini/blob/fe92cc04ebea1bc1648b9bd9ff041db91ab01cd5/tests/resources/simple_cacm_corpus.json -o /data/simple_cacm_corpus.json

CMD USE_CUDA=0 python -m pyserini.encode \
  input   --corpus data/simple_cacm_corpus.json \
          --fields text \
  output  --embeddings  embeddings/small-dense-1668560951 \
  encoder --encoder  castorini/tct_colbert-v2-hnp-msmarco \
          --fields text \
          --batch 8

the error I'm getting is as follows. I'm wondering if they are connected?

Traceback (most recent call last):
  File "/opt/conda/envs/pyserini/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/pyserini/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/pyserini/encode/__main__.py", line 116, in <module>
    encoder = init_encoder(args.encoder.encoder, args.encoder.encoder_class, device=args.encoder.device)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/pyserini/encode/__main__.py", line 58, in init_encoder
    return encoder_class(**kwargs)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/pyserini/encode/_tct_colbert.py", line 38, in __init__
    self.model.to(self.device)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/torch/nn/modules/module.py", line 987, in to
    return self._apply(convert)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/torch/nn/modules/module.py", line 639, in _apply
    module._apply(fn)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/torch/nn/modules/module.py", line 639, in _apply
    module._apply(fn)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/torch/nn/modules/module.py", line 662, in _apply
    param_applied = fn(param)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/torch/nn/modules/module.py", line 985, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  File "/opt/conda/envs/pyserini/lib/python3.9/site-packages/torch/cuda/__init__.py", line 221, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

[lintool]

It appears that you have a version of Torch that expects GPU... and since you don't have one, that might be the issue?

[lintool]

Also, I don't think it'll make a difference, but I'm on Python 3.8, not 3.9...

[vicpara]

I've been installing pytorch with conda install pytorch torchvision torchaudio cpuonly -c pytorch
Pip equally should be able to detect correctly the architecture.

[lintool]

Can you please note the resolution on the GitHub discussion ticket so we can close it?

…

-Jimmy

On 11/16/22 8:07 AM, Victor Paraschiv wrote: I had |faiss-cpu| installed via pip. Interestingly tevatron was working fine. Run |conda install faiss-cpu -c pytorch| and solved the problem. Thanks for this. — Reply to this email directly, view it on GitHub <#1346 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACMT3NLDEQG3W54KB5UMILWITL77ANCNFSM6AAAAAASBUHOTM>. You are receiving this because you commented.Message ID: ***@***.***>

[vicpara]

Sorry for that. I rushed to reply but I was running the wrong script. The problem still persists after following the installation instructions.

[vicpara]

we can close this now

[vicpara]

For CPU only machines adding --device cpu seems to fix the problem. Even when all packages are installed as for CPU only for some reason, pytorch/ torch/ faiss forces the use of some CUDA enabled dependencies.

I installed all dependencies via pip directly and it works. No need to reach out to anaconda. In my case that kept throwing 'Segmentation fault 11' errors.

I managed to successfully run it on both python 3.8 and 3.9.

[lintool]

Yay, glad this got sorted out!

For CPU only machines adding --device cpu seems to fix the problem.

To be clear, adding --device cpu to what?

[vicpara]

To the pyserini encode call, that mentioned in the initial post.

python -m pyserini.encode \
  input   --corpus data/collections/simple_cacm_corpus.json \
          --fields text \
  output  --embeddings  embeddings/small-dense-1668560951 \
          --to-faiss \
  encoder --encoder  castorini/tct_colbert-v2-hnp-msmarco \
          --fields text \
          --batch 8  \
          --device cpu