You are using a model of type RefinedWeb to instantiate a model of type RefinedWebModel. This is not supported for all configurations of models and can yield errors.

[yuhai-china]

When I try to run generate command on a H100 server, found the following error ;

You are using a model of type RefinedWeb to instantiate a model of type RefinedWebModel. This is not supported for all configurations of models and can yield errors.
Traceback (most recent call last):
File "/usr/local/bin/falcontune", line 11, in
load_entry_point('falcontune==0.1.0', 'console_scripts', 'falcontune')()
File "/usr/local/lib/python3.8/dist-packages/falcontune-0.1.0-py3.8.egg/falcontune/run.py", line 87, in main
args.func(args)
File "/usr/local/lib/python3.8/dist-packages/falcontune-0.1.0-py3.8.egg/falcontune/generate.py", line 43, in generate
model, tokenizer = load_model(
File "/usr/local/lib/python3.8/dist-packages/falcontune-0.1.0-py3.8.egg/falcontune/model/init.py", line 33, in load_model
model, tokenizer = load_model(model_config, weights, half=half, backend=backend)
File "/usr/local/lib/python3.8/dist-packages/falcontune-0.1.0-py3.8.egg/falcontune/model/falcon/model.py", line 1147, in load_model
ql = importlib.import_module(f'falcontune.backend.{backend}.quantlinear')
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.8/dist-packages/falcontune-0.1.0-py3.8.egg/falcontune/backend/cuda/quantlinear.py", line 3, in
import quant_cuda
ImportError: /usr/local/lib/python3.8/dist-packages/quant_cuda-0.0.0-py3.8-linux-x86_64.egg/quant_cuda.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

[richardburleigh]

Try either removing and reinstalling the requirements, or starting again in a virtual Python environment.