Add support for LogitsTemperatureScaler in the new ModelOutput API

[sararb]

Goals:

• As a follow-up to the fixes needed to reproduce the research scripts with the new API (see Add L2-batch regularization to EmbeddingTable #812 and Add LogQ correction support to the new ModelOutput API #811), I defined integration tests for TwoTower and MatrixFactorization. I observed that TwoTower is still returning a smaller validation performance compared to the old API.

• After debugging, the issue is now related to LogitsTemperatureScaler not including the logic of scaling the logits if the class variable apply_on_call_outputs is set to True (which is the default).

• In fact, the logits scaler in ModelOutput is set here with the default value True, so the logits are never scaled with the provided temperature.

• apply_on_call_outputs is specific to Merlin supporting two versions of the API so I believe we should not expose this argument to the user. It might be confusing for the user to understand when to set apply_on_call_outputs to True or False.

• So I refactored LogitsTemperatureScaler to have similar logic we have in PopularityLogitsCorrection

• Figures below show the 3 runs of integration tests: Old API, New API, New API with logit scaler fix

Implementation Details 🚧

• Add a copy_with_updates to the Prediction NamedTuple class to update only necessary tensors in the post transforms blocks.
• Make sure the ModelOutputBase is always returning a Prediction object.
• Add applying logits-temperature inside the call method of the LogitsTemperatureScaler block.
• Add get_config method to LogitsTemperatureScaler to store the value temperature when saving the model.

Testing Details 🔍

• Add test of logit scaler with different temperatures in outputs/test_base.py`

[nvidia-merlin-bot]

Click to view CI Results

GitHub pull request #815 of commit 6ebbb78330f12a3e5bc47283faccee5d6fda506f, no merge conflicts.
Running as SYSTEM
Setting status of 6ebbb78330f12a3e5bc47283faccee5d6fda506f to PENDING with url https://10.20.13.93:8080/job/merlin_models/1552/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/815/*:refs/remotes/origin/pr/815/* # timeout=10
 > git rev-parse 6ebbb78330f12a3e5bc47283faccee5d6fda506f^{commit} # timeout=10
Checking out Revision 6ebbb78330f12a3e5bc47283faccee5d6fda506f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6ebbb78330f12a3e5bc47283faccee5d6fda506f # timeout=10
Commit message: "fix logit scaler for new ModelOutput API"
 > git rev-list --no-walk 36292c0ed34cdbc9c5d610ccd896e4ea09fd32ec # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins7044676518061846842.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 775 items
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  0%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  1%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  2%]

tests/unit/datasets/test_social.py .                                     [  2%]

tests/unit/datasets/test_synthetic.py ......                             [  3%]

tests/unit/implicit/test_implicit.py .                                   [  3%]

tests/unit/lightfm/test_lightfm.py .                                     [  3%]

tests/unit/tf/test_core.py ......                                        [  4%]

tests/unit/tf/test_loader.py ................                            [  6%]

tests/unit/tf/test_public_api.py .                                       [  6%]

tests/unit/tf/blocks/test_cross.py ...........                           [  8%]

tests/unit/tf/blocks/test_dlrm.py ..........                             [  9%]

tests/unit/tf/blocks/test_interactions.py ...                            [  9%]

tests/unit/tf/blocks/test_mlp.py .................................       [ 13%]

tests/unit/tf/blocks/test_optimizer.py s................................ [ 18%]

.....................                                                    [ 20%]

tests/unit/tf/blocks/retrieval/test_base.py .                            [ 21%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 21%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ............            [ 22%]

tests/unit/tf/blocks/sampling/test_cross_batch.py .                      [ 22%]

tests/unit/tf/blocks/sampling/test_in_batch.py .                         [ 23%]

tests/unit/tf/core/test_aggregation.py .........                         [ 24%]

tests/unit/tf/core/test_base.py ..                                       [ 24%]

tests/unit/tf/core/test_combinators.py s....................             [ 27%]

tests/unit/tf/core/test_encoder.py ..                                    [ 27%]

tests/unit/tf/core/test_index.py ...                                     [ 27%]

tests/unit/tf/core/test_prediction.py ..                                 [ 28%]

tests/unit/tf/core/test_tabular.py ......                                [ 28%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 29%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 29%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 29%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 29%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 29%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 29%]

tests/unit/tf/examples/test_07_train_traditional_models.py .             [ 29%]

tests/unit/tf/examples/test_usecase_accelerate_training_by_lazyadam.py F [ 29%]

[ 29%]

tests/unit/tf/examples/test_usecase_ecommerce_session_based.py .         [ 30%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 30%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 30%]

tests/unit/tf/inputs/test_embedding.py ................................. [ 35%]

.......                                                                  [ 36%]

tests/unit/tf/inputs/test_tabular.py ..................                  [ 38%]

tests/unit/tf/layers/test_queue.py ..............                        [ 40%]

tests/unit/tf/losses/test_losses.py .......................              [ 43%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 43%]

tests/unit/tf/metrics/test_metrics_topk.py ........................      [ 46%]

tests/unit/tf/models/test_base.py s.......................               [ 49%]

tests/unit/tf/models/test_benchmark.py ..                                [ 50%]

tests/unit/tf/models/test_ranking.py ..................................  [ 54%]

tests/unit/tf/models/test_retrieval.py ................................  [ 58%]

tests/unit/tf/outputs/test_base.py ......                                [ 59%]

tests/unit/tf/outputs/test_classification.py ......                      [ 60%]

tests/unit/tf/outputs/test_contrastive.py ..............                 [ 62%]

tests/unit/tf/outputs/test_regression.py ..                              [ 62%]

tests/unit/tf/outputs/test_sampling.py ....                              [ 62%]

tests/unit/tf/outputs/test_topk.py .                                     [ 62%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 63%]

tests/unit/tf/prediction_tasks/test_multi_task.py ................       [ 65%]

tests/unit/tf/prediction_tasks/test_next_item.py .....                   [ 65%]

tests/unit/tf/prediction_tasks/test_regression.py .....                  [ 66%]

tests/unit/tf/prediction_tasks/test_retrieval.py .                       [ 66%]

tests/unit/tf/prediction_tasks/test_sampling.py ......                   [ 67%]

tests/unit/tf/transformers/test_block.py ....................            [ 70%]

tests/unit/tf/transformers/test_transforms.py ......                     [ 70%]

tests/unit/tf/transforms/test_bias.py ..                                 [ 71%]

tests/unit/tf/transforms/test_features.py s............................. [ 74%]

....................s......                                              [ 78%]

tests/unit/tf/transforms/test_negative_sampling.py .........             [ 79%]

tests/unit/tf/transforms/test_noise.py .....                             [ 80%]

tests/unit/tf/transforms/test_sequence.py ....................           [ 82%]

tests/unit/tf/transforms/test_tensor.py ...                              [ 83%]

tests/unit/tf/utils/test_batch.py ....                                   [ 83%]

tests/unit/tf/utils/test_dataset.py ..                                   [ 84%]

tests/unit/tf/utils/test_tf_utils.py .....                               [ 84%]

tests/unit/torch/test_dataset.py .........                               [ 85%]

tests/unit/torch/test_public_api.py .                                    [ 85%]

tests/unit/torch/block/test_base.py ....                                 [ 86%]

tests/unit/torch/block/test_mlp.py .                                     [ 86%]

tests/unit/torch/features/test_continuous.py ..                          [ 86%]

tests/unit/torch/features/test_embedding.py ..............               [ 88%]

tests/unit/torch/features/test_tabular.py ....                           [ 89%]

tests/unit/torch/model/test_head.py ............                         [ 90%]

tests/unit/torch/model/test_model.py ..                                  [ 90%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 92%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 92%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 93%]

tests/unit/utils/test_schema_utils.py ................................   [ 97%]

tests/unit/xgb/test_xgboost.py ....................                      [100%]
=================================== FAILURES ===================================

_________________ test_usecase_accelerate_training_by_lazyadam _________________
tb = <testbook.client.TestbookNotebookClient object at 0x7fcc9ab9b670>
@testbook(
    REPO_ROOT / p,
    timeout=180,
    execute=False,
)
def test_usecase_accelerate_training_by_lazyadam(tb):
    tb.inject(
        """
        import os
        os.environ["NUM_ROWS"] = "1000"
        """
    )


  tb.execute()


tests/unit/tf/examples/test_usecase_accelerate_training_by_lazyadam.py:22:

/usr/local/lib/python3.8/dist-packages/testbook/client.py:147: in execute

super().execute_cell(cell, index)

/usr/local/lib/python3.8/dist-packages/nbclient/util.py:85: in wrapped

return just_run(coro(*args, **kwargs))

/usr/local/lib/python3.8/dist-packages/nbclient/util.py:60: in just_run

return loop.run_until_complete(coro)

/usr/lib/python3.8/asyncio/base_events.py:616: in run_until_complete

return future.result()

/usr/local/lib/python3.8/dist-packages/nbclient/client.py:1025: in async_execute_cell

await self._check_raise_for_error(cell, cell_index, exec_reply)

self = <testbook.client.TestbookNotebookClient object at 0x7fcc9ab9b670>

cell = {'cell_type': 'code', 'execution_count': 7, 'id': '0500ad25-29e0-40c8-85bc-6e3864107c6a', 'metadata': {'execution': {'...e_train_function_3849]']}], 'source': 'model1.compile(optimizer="adam")\nmodel1.fit(train, batch_size=1024, epochs=1)'}

cell_index = 12

exec_reply = {'buffers': [], 'content': {'ename': 'ResourceExhaustedError', 'engine_info': {'engine_id': -1, 'engine_uuid': '74f0f5...e, 'engine': '74f0f506-94a2-412f-809b-86cb0b016fa3', 'started': '2022-10-20T18:38:28.764793Z', 'status': 'error'}, ...}
async def _check_raise_for_error(
    self, cell: NotebookNode, cell_index: int, exec_reply: t.Optional[t.Dict]
) -> None:

    if exec_reply is None:
        return None

    exec_reply_content = exec_reply['content']
    if exec_reply_content['status'] != 'error':
        return None

    cell_allows_errors = (not self.force_raise_errors) and (
        self.allow_errors
        or exec_reply_content.get('ename') in self.allow_error_names
        or "raises-exception" in cell.metadata.get("tags", [])
    )
    await run_hook(
        self.on_cell_error, cell=cell, cell_index=cell_index, execute_reply=exec_reply
    )
    if not cell_allows_errors:


      raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)


E           nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:

E           ------------------

E           model1.compile(optimizer="adam")

E           model1.fit(train, batch_size=1024, epochs=1)

E           ------------------

E

E           �[0;31m---------------------------------------------------------------------------�[0m

E           �[0;31mResourceExhaustedError�[0m                    Traceback (most recent call last)

E           Cell �[0;32mIn [7], line 2�[0m

E           �[1;32m      1�[0m model1�[38;5;241m.�[39mcompile(optimizer�[38;5;241m=�[39m�[38;5;124m"�[39m�[38;5;124madam�[39m�[38;5;124m"�[39m)

E           �[0;32m----> 2�[0m �[43mmodel1�[49m�[38;5;241;43m.�[39;49m�[43mfit�[49m�[43m(�[49m�[43mtrain�[49m�[43m,�[49m�[43m �[49m�[43mbatch_size�[49m�[38;5;241;43m=�[39;49m�[38;5;241;43m1024�[39;49m�[43m,�[49m�[43m �[49m�[43mepochs�[49m�[38;5;241;43m=�[39;49m�[38;5;241;43m1�[39;49m�[43m)�[49m

E

E           File �[0;32m~/workspace/merlin_models/models/merlin/models/tf/models/base.py:831�[0m, in �[0;36mBaseModel.fit�[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing, train_metrics_steps, pre, **kwargs)�[0m

E           �[1;32m    828�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39m_reset_compile_cache()

E           �[1;32m    829�[0m     �[38;5;28mself�[39m�[38;5;241m.�[39mtrain_pre �[38;5;241m=�[39m pre

E           �[0;32m--> 831�[0m out �[38;5;241m=�[39m �[38;5;28;43msuper�[39;49m�[43m(�[49m�[43m)�[49m�[38;5;241;43m.�[39;49m�[43mfit�[49m�[43m(�[49m�[38;5;241;43m�[39;49m�[38;5;241;43m�[39;49m�[43mfit_kwargs�[49m�[43m)�[49m

E           �[1;32m    833�[0m �[38;5;28;01mif�[39;00m pre:

E           �[1;32m    834�[0m     �[38;5;28;01mdel�[39;00m �[38;5;28mself�[39m�[38;5;241m.�[39mtrain_pre

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py:67�[0m, in �[0;36mfilter_traceback..error_handler�[0;34m(*args, **kwargs)�[0m

E           �[1;32m     65�[0m �[38;5;28;01mexcept�[39;00m �[38;5;167;01mException�[39;00m �[38;5;28;01mas�[39;00m e:  �[38;5;66;03m# pylint: disable=broad-except�[39;00m

E           �[1;32m     66�[0m   filtered_tb �[38;5;241m=�[39m process_traceback_frames(e�[38;5;241m.�[39m__traceback_)

E           �[0;32m---> 67�[0m   �[38;5;28;01mraise�[39;00m e�[38;5;241m.�[39mwith_traceback(filtered_tb) �[38;5;28;01mfrom�[39;00m �[38;5;28mNone�[39m

E           �[1;32m     68�[0m �[38;5;28;01mfinally�[39;00m:

E           �[1;32m     69�[0m   �[38;5;28;01mdel�[39;00m filtered_tb

E

E           File �[0;32m/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py:54�[0m, in �[0;36mquick_execute�[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)�[0m

E           �[1;32m     52�[0m �[38;5;28;01mtry�[39;00m:

E           �[1;32m     53�[0m   ctx�[38;5;241m.�[39mensure_initialized()

E           �[0;32m---> 54�[0m   tensors �[38;5;241m=�[39m pywrap_tfe�[38;5;241m.�[39mTFE_Py_Execute(ctx�[38;5;241m.�[39m_handle, device_name, op_name,

E           �[1;32m     55�[0m                                       inputs, attrs, num_outputs)

E           �[1;32m     56�[0m �[38;5;28;01mexcept�[39;00m core�[38;5;241m.�[39m_NotOkStatusException �[38;5;28;01mas�[39;00m e:

E           �[1;32m     57�[0m   �[38;5;28;01mif�[39;00m name �[38;5;129;01mis�[39;00m �[38;5;129;01mnot�[39;00m �[38;5;28;01mNone�[39;00m:

E

E           �[0;31mResourceExhaustedError�[0m: Graph execution error:

E

E           Detected at node 'Adam/Adam/update_17/mul_4' defined at (most recent call last):

E               File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main

E                 return _run_code(code, main_globals, None,

E               File "/usr/lib/python3.8/runpy.py", line 87, in _run_code

E                 exec(code, run_globals)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py", line 17, in 

E                 app.launch_new_instance()

E               File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 978, in launch_instance

E                 app.start()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelapp.py", line 712, in start

E                 self.io_loop.start()

E               File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

E                 self.asyncio_loop.run_forever()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

E                 self._run_once()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

E                 handle._run()

E               File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

E                 self._context.run(self._callback, *self._args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue

E                 await self.process_one()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 499, in process_one

E                 await dispatch(*args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell

E                 await result

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 730, in execute_request

E                 reply_content = await reply_content

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/ipkernel.py", line 383, in do_execute

E                 res = shell.run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/zmqshell.py", line 528, in run_cell

E                 return super().run_cell(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2885, in run_cell

E                 result = self._run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2940, in _run_cell

E                 return runner(coro)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/async_helpers.py", line 129, in pseudo_sync_runner

E                 coro.send(None)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3139, in run_cell_async

E                 has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3318, in run_ast_nodes

E                 if await self.run_code(code, result, async=asy):

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3378, in run_code

E                 exec(code_obj, self.user_global_ns, self.user_ns)

E               File "/tmp/ipykernel_32103/3741080137.py", line 2, in 

E                 model1.fit(train, batch_size=1024, epochs=1)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 831, in fit

E                 out = super().fit(**fit_kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler

E                 return fn(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1409, in fit

E                 tmp_logs = self.train_function(iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1051, in train_function

E                 return step_function(self, iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1040, in step_function

E                 outputs = model.distribute_strategy.run(run_step, args=(data,))

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1030, in run_step

E                 outputs = model.train_step(data)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 658, in train_step

E                 self.optimizer.minimize(loss, self.trainable_variables, tape=tape)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 539, in minimize

E                 return self.apply_gradients(grads_and_vars, name=name)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 678, in apply_gradients

E                 return tf.internal.distribute.interim.maybe_merge_call(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 723, in _distributed_apply

E                 update_op = distribution.extended.update(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 701, in apply_grad_to_update_var

E                 return self._resource_apply_sparse_duplicate_indices(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 1326, in _resource_apply_sparse_duplicate_indices

E                 return self._resource_apply_sparse(summed_grad, handle, unique_indices,

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/adam.py", line 214, in _resource_apply_sparse

E                 v_t = tf.compat.v1.assign(v, v * coefficients['beta_2_t'],

E           Node: 'Adam/Adam/update_17/mul_4'

E           Detected at node 'Adam/Adam/update_17/mul_4' defined at (most recent call last):

E               File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main

E                 return _run_code(code, main_globals, None,

E               File "/usr/lib/python3.8/runpy.py", line 87, in _run_code

E                 exec(code, run_globals)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py", line 17, in 

E                 app.launch_new_instance()

E               File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 978, in launch_instance

E                 app.start()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelapp.py", line 712, in start

E                 self.io_loop.start()

E               File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

E                 self.asyncio_loop.run_forever()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

E                 self._run_once()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

E                 handle._run()

E               File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

E                 self._context.run(self._callback, *self._args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue

E                 await self.process_one()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 499, in process_one

E                 await dispatch(*args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell

E                 await result

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 730, in execute_request

E                 reply_content = await reply_content

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/ipkernel.py", line 383, in do_execute

E                 res = shell.run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/zmqshell.py", line 528, in run_cell

E                 return super().run_cell(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2885, in run_cell

E                 result = self._run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2940, in _run_cell

E                 return runner(coro)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/async_helpers.py", line 129, in pseudo_sync_runner

E                 coro.send(None)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3139, in run_cell_async

E                 has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3318, in run_ast_nodes

E                 if await self.run_code(code, result, async=asy):

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3378, in run_code

E                 exec(code_obj, self.user_global_ns, self.user_ns)

E               File "/tmp/ipykernel_32103/3741080137.py", line 2, in 

E                 model1.fit(train, batch_size=1024, epochs=1)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 831, in fit

E                 out = super().fit(**fit_kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler

E                 return fn(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1409, in fit

E                 tmp_logs = self.train_function(iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1051, in train_function

E                 return step_function(self, iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1040, in step_function

E                 outputs = model.distribute_strategy.run(run_step, args=(data,))

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1030, in run_step

E                 outputs = model.train_step(data)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 658, in train_step

E                 self.optimizer.minimize(loss, self.trainable_variables, tape=tape)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 539, in minimize

E                 return self.apply_gradients(grads_and_vars, name=name)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 678, in apply_gradients

E                 return tf.internal.distribute.interim.maybe_merge_call(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 723, in _distributed_apply

E                 update_op = distribution.extended.update(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 701, in apply_grad_to_update_var

E                 return self._resource_apply_sparse_duplicate_indices(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 1326, in _resource_apply_sparse_duplicate_indices

E                 return self._resource_apply_sparse(summed_grad, handle, unique_indices,

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/adam.py", line 214, in _resource_apply_sparse

E                 v_t = tf.compat.v1.assign(v, v * coefficients['beta_2_t'],

E           Node: 'Adam/Adam/update_17/mul_4'

E           2 root error(s) found.

E             (0) RESOURCE_EXHAUSTED:  failed to allocate memory

E           	 [[{{node Adam/Adam/update_17/mul_4}}]]

E           Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

E

E           	 [[StatefulPartitionedCall/cond/pivot_t/_131/_53]]

E           Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

E

E             (1) RESOURCE_EXHAUSTED:  failed to allocate memory

E           	 [[{{node Adam/Adam/update_17/mul_4}}]]

E           Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

E

E           0 successful operations.

E           0 derived errors ignored. [Op:__inference_train_function_3849]

E           ResourceExhaustedError: Graph execution error:

E

E           Detected at node 'Adam/Adam/update_17/mul_4' defined at (most recent call last):

E               File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main

E                 return _run_code(code, main_globals, None,

E               File "/usr/lib/python3.8/runpy.py", line 87, in _run_code

E                 exec(code, run_globals)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py", line 17, in 

E                 app.launch_new_instance()

E               File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 978, in launch_instance

E                 app.start()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelapp.py", line 712, in start

E                 self.io_loop.start()

E               File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

E                 self.asyncio_loop.run_forever()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

E                 self._run_once()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

E                 handle._run()

E               File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

E                 self._context.run(self._callback, *self._args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue

E                 await self.process_one()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 499, in process_one

E                 await dispatch(*args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell

E                 await result

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 730, in execute_request

E                 reply_content = await reply_content

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/ipkernel.py", line 383, in do_execute

E                 res = shell.run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/zmqshell.py", line 528, in run_cell

E                 return super().run_cell(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2885, in run_cell

E                 result = self._run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2940, in _run_cell

E                 return runner(coro)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/async_helpers.py", line 129, in pseudo_sync_runner

E                 coro.send(None)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3139, in run_cell_async

E                 has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3318, in run_ast_nodes

E                 if await self.run_code(code, result, async=asy):

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3378, in run_code

E                 exec(code_obj, self.user_global_ns, self.user_ns)

E               File "/tmp/ipykernel_32103/3741080137.py", line 2, in 

E                 model1.fit(train, batch_size=1024, epochs=1)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 831, in fit

E                 out = super().fit(**fit_kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler

E                 return fn(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1409, in fit

E                 tmp_logs = self.train_function(iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1051, in train_function

E                 return step_function(self, iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1040, in step_function

E                 outputs = model.distribute_strategy.run(run_step, args=(data,))

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1030, in run_step

E                 outputs = model.train_step(data)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 658, in train_step

E                 self.optimizer.minimize(loss, self.trainable_variables, tape=tape)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 539, in minimize

E                 return self.apply_gradients(grads_and_vars, name=name)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 678, in apply_gradients

E                 return tf.internal.distribute.interim.maybe_merge_call(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 723, in _distributed_apply

E                 update_op = distribution.extended.update(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 701, in apply_grad_to_update_var

E                 return self._resource_apply_sparse_duplicate_indices(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 1326, in _resource_apply_sparse_duplicate_indices

E                 return self._resource_apply_sparse(summed_grad, handle, unique_indices,

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/adam.py", line 214, in _resource_apply_sparse

E                 v_t = tf.compat.v1.assign(v, v * coefficients['beta_2_t'],

E           Node: 'Adam/Adam/update_17/mul_4'

E           Detected at node 'Adam/Adam/update_17/mul_4' defined at (most recent call last):

E               File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main

E                 return _run_code(code, main_globals, None,

E               File "/usr/lib/python3.8/runpy.py", line 87, in _run_code

E                 exec(code, run_globals)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel_launcher.py", line 17, in 

E                 app.launch_new_instance()

E               File "/usr/local/lib/python3.8/dist-packages/traitlets/config/application.py", line 978, in launch_instance

E                 app.start()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelapp.py", line 712, in start

E                 self.io_loop.start()

E               File "/usr/local/lib/python3.8/dist-packages/tornado/platform/asyncio.py", line 215, in start

E                 self.asyncio_loop.run_forever()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever

E                 self._run_once()

E               File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once

E                 handle._run()

E               File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run

E                 self._context.run(self._callback, *self._args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 510, in dispatch_queue

E                 await self.process_one()

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 499, in process_one

E                 await dispatch(*args)

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 406, in dispatch_shell

E                 await result

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/kernelbase.py", line 730, in execute_request

E                 reply_content = await reply_content

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/ipkernel.py", line 383, in do_execute

E                 res = shell.run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/ipykernel/zmqshell.py", line 528, in run_cell

E                 return super().run_cell(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2885, in run_cell

E                 result = self._run_cell(

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 2940, in _run_cell

E                 return runner(coro)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/async_helpers.py", line 129, in pseudo_sync_runner

E                 coro.send(None)

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3139, in run_cell_async

E                 has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3318, in run_ast_nodes

E                 if await self.run_code(code, result, async=asy):

E               File "/usr/local/lib/python3.8/dist-packages/IPython/core/interactiveshell.py", line 3378, in run_code

E                 exec(code_obj, self.user_global_ns, self.user_ns)

E               File "/tmp/ipykernel_32103/3741080137.py", line 2, in 

E                 model1.fit(train, batch_size=1024, epochs=1)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 831, in fit

E                 out = super().fit(**fit_kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 64, in error_handler

E                 return fn(*args, **kwargs)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1409, in fit

E                 tmp_logs = self.train_function(iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1051, in train_function

E                 return step_function(self, iterator)

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1040, in step_function

E                 outputs = model.distribute_strategy.run(run_step, args=(data,))

E               File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1030, in run_step

E                 outputs = model.train_step(data)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 658, in train_step

E                 self.optimizer.minimize(loss, self.trainable_variables, tape=tape)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 539, in minimize

E                 return self.apply_gradients(grads_and_vars, name=name)

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 678, in apply_gradients

E                 return tf.internal.distribute.interim.maybe_merge_call(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 723, in _distributed_apply

E                 update_op = distribution.extended.update(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 701, in apply_grad_to_update_var

E                 return self._resource_apply_sparse_duplicate_indices(

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/optimizer_v2.py", line 1326, in _resource_apply_sparse_duplicate_indices

E                 return self._resource_apply_sparse(summed_grad, handle, unique_indices,

E               File "/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/adam.py", line 214, in _resource_apply_sparse

E                 v_t = tf.compat.v1.assign(v, v * coefficients['beta_2_t'],

E           Node: 'Adam/Adam/update_17/mul_4'

E           2 root error(s) found.

E             (0) RESOURCE_EXHAUSTED:  failed to allocate memory

E           	 [[{{node Adam/Adam/update_17/mul_4}}]]

E           Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

E

E           	 [[StatefulPartitionedCall/cond/pivot_t/_131/_53]]

E           Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

E

E             (1) RESOURCE_EXHAUSTED:  failed to allocate memory

E           	 [[{{node Adam/Adam/update_17/mul_4}}]]

E           Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.

E

E           0 successful operations.

E           0 derived errors ignored. [Op:__inference_train_function_3849]
/usr/local/lib/python3.8/dist-packages/nbclient/client.py:919: CellExecutionError

----------------------------- Captured stderr call -----------------------------

2022-10-20 18:38:24.988324: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA

To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

2022-10-20 18:38:27.310684: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 0

2022-10-20 18:38:27.310827: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1627 MB memory:  -> device: 0, name: Tesla P100-DGXS-16GB, pci bus id: 0000:07:00.0, compute capability: 6.0

2022-10-20 18:38:27.311499: I tensorflow/core/common_runtime/gpu/gpu_process_state.cc:222] Using CUDA malloc Async allocator for GPU: 1

2022-10-20 18:38:27.311554: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 13851 MB memory:  -> device: 1, name: Tesla P100-DGXS-16GB, pci bus id: 0000:08:00.0, compute capability: 6.0

2022-10-20 18:38:38.825570: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:288] gpu_async_0 cuMemAllocAsync failed to allocate 1083564064 bytes: CUDA error: out of memory (CUDA_ERROR_OUT_OF_MEMORY)

Reported by CUDA: Free memory/Total memory: 884539392/17069309952

2022-10-20 18:38:38.825632: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:293] Stats: Limit:                      1706033152

InUse:                      4225294840

MaxInUse:                   4225294840

NumAllocs:                         229

MaxAllocSize:               1083564064

Reserved:                            0

PeakReserved:                        0

LargestFreeBlock:                    0
2022-10-20 18:38:38.825651: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:56] Histogram of current allocation: (allocation_size_in_bytes, nb_allocation_of_that_sizes), ...;

2022-10-20 18:38:38.825660: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1, 4

2022-10-20 18:38:38.825668: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 4, 34

2022-10-20 18:38:38.825674: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 8, 7

2022-10-20 18:38:38.825681: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 40, 2

2022-10-20 18:38:38.825687: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 128, 6

2022-10-20 18:38:38.825693: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 160, 6

2022-10-20 18:38:38.825699: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 192, 4

2022-10-20 18:38:38.825705: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 256, 7

2022-10-20 18:38:38.825711: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 288, 3

2022-10-20 18:38:38.825717: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 480, 4

2022-10-20 18:38:38.825723: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1028, 1

2022-10-20 18:38:38.825729: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 3168, 5

2022-10-20 18:38:38.825736: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 61440, 3

2022-10-20 18:38:38.825742: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 65536, 3

2022-10-20 18:38:38.825748: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 584352, 5

2022-10-20 18:38:38.825755: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 823872, 3

2022-10-20 18:38:38.825761: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 4324736, 3

2022-10-20 18:38:38.825767: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 7426048, 3

2022-10-20 18:38:38.825773: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 15401440, 3

2022-10-20 18:38:38.825802: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 18678720, 3

2022-10-20 18:38:38.825810: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 39970560, 3

2022-10-20 18:38:38.825816: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 56589504, 3

2022-10-20 18:38:38.825824: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 135407776, 4

2022-10-20 18:38:38.825830: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1083564064, 3

2022-10-20 18:38:38.825857: W tensorflow/core/framework/op_kernel.cc:1733] RESOURCE_EXHAUSTED: failed to allocate memory

2022-10-20 18:38:38.855854: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:288] gpu_async_0 cuMemAllocAsync failed to allocate 1083564064 bytes: CUDA error: out of memory (CUDA_ERROR_OUT_OF_MEMORY)

Reported by CUDA: Free memory/Total memory: 515440640/17069309952

2022-10-20 18:38:38.855897: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:293] Stats: Limit:                      1706033152

InUse:                      4647134016

MaxInUse:                   4647134016

NumAllocs:                         253

MaxAllocSize:               1083564064

Reserved:                            0

PeakReserved:                        0

LargestFreeBlock:                    0
2022-10-20 18:38:38.855916: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:56] Histogram of current allocation: (allocation_size_in_bytes, nb_allocation_of_that_sizes), ...;

2022-10-20 18:38:38.855925: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1, 4

2022-10-20 18:38:38.855932: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 4, 34

2022-10-20 18:38:38.855938: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 8, 8

2022-10-20 18:38:38.855944: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 40, 2

2022-10-20 18:38:38.855950: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 128, 9

2022-10-20 18:38:38.855956: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 160, 9

2022-10-20 18:38:38.855962: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 192, 4

2022-10-20 18:38:38.855969: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 256, 7

2022-10-20 18:38:38.855975: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 288, 4

2022-10-20 18:38:38.855981: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 480, 5

2022-10-20 18:38:38.855987: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1028, 1

2022-10-20 18:38:38.855993: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 3168, 5

2022-10-20 18:38:38.855999: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 61440, 3

2022-10-20 18:38:38.856006: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 65536, 3

2022-10-20 18:38:38.856012: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 584352, 5

2022-10-20 18:38:38.856018: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 823872, 5

2022-10-20 18:38:38.856032: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 4324736, 5

2022-10-20 18:38:38.856039: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 7426048, 5

2022-10-20 18:38:38.856045: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 15401440, 5

2022-10-20 18:38:38.856051: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 18678720, 5

2022-10-20 18:38:38.856058: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 39970560, 5

2022-10-20 18:38:38.856064: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 56589504, 5

2022-10-20 18:38:38.856070: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 135407776, 5

2022-10-20 18:38:38.856096: E tensorflow/core/common_runtime/gpu/gpu_cudamallocasync_allocator.cc:59] 1083564064, 3

2022-10-20 18:38:38.856110: W tensorflow/core/framework/op_kernel.cc:1733] RESOURCE_EXHAUSTED: failed to allocate memory

Error in atexit._run_exitfuncs:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/init.py", line 2127, in shutdown

h.close()

File "/usr/local/lib/python3.8/dist-packages/absl/logging/init.py", line 934, in close

self.stream.close()

File "/usr/local/lib/python3.8/dist-packages/ipykernel/iostream.py", line 438, in close

self.watch_fd_thread.join()

AttributeError: 'OutStream' object has no attribute 'watch_fd_thread'

=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning

tests/unit/datasets/test_ecommerce.py: 2 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 6 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 26 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 5 warnings

tests/unit/tf/core/test_index.py: 8 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/inputs/test_continuous.py: 4 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 26 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 38 warnings

tests/unit/tf/models/test_retrieval.py: 60 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning

tests/unit/tf/transformers/test_block.py: 15 warnings

tests/unit/tf/transforms/test_bias.py: 2 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_noise.py: 1 warning

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 9 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 3 warnings

tests/unit/xgb/test_xgboost.py: 18 warnings

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 5 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 26 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 7 warnings

tests/unit/tf/core/test_index.py: 3 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/inputs/test_continuous.py: 4 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 26 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 36 warnings

tests/unit/tf/models/test_retrieval.py: 32 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/transformers/test_block.py: 9 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 7 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 2 warnings

tests/unit/xgb/test_xgboost.py: 17 warnings

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 2 warnings

tests/unit/tf/core/test_prediction.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 2 warnings

tests/unit/tf/inputs/test_embedding.py: 9 warnings

tests/unit/tf/inputs/test_tabular.py: 8 warnings

tests/unit/tf/models/test_ranking.py: 20 warnings

tests/unit/tf/models/test_retrieval.py: 4 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings

tests/unit/xgb/test_xgboost.py: 12 warnings

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:960: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/blocks/retrieval/test_two_tower.py: 1 warning

tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/models/test_retrieval.py: 54 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_fileq9jhqx5m.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/core/test_combinators.py::test_parallel_block_select_by_tags

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/tabular.py:614: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

elif isinstance(self.feature_names, collections.Sequence):
tests/unit/tf/core/test_index.py: 5 warnings

tests/unit/tf/models/test_retrieval.py: 26 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

tests/unit/tf/utils/test_dataset.py: 1 warning

/var/jenkins_home/workspace/merlin_models/models/merlin/models/utils/dataset.py:75: DeprecationWarning: unique_rows_by_features is deprecated and will be removed in a future version. Please use unique_by_tag instead.

warnings.warn(
tests/unit/tf/models/test_base.py::test_model_pre_post[True]

tests/unit/tf/models/test_base.py::test_model_pre_post[False]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]

tests/unit/tf/models/test_base.py::test_freeze_sequential_block

tests/unit/tf/models/test_base.py::test_freeze_unfreeze

tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks

/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.

super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_base.py::test_retrieval_model_query

tests/unit/tf/models/test_base.py::test_retrieval_model_query

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py:294: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

tensor_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(tensor)))
tests/unit/tf/models/test_ranking.py::test_deepfm_model_only_categ_feats[False]

tests/unit/tf/models/test_ranking.py::test_deepfm_model_categ_and_continuous_feats[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:569: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask/GatherV2:0", shape=(None, 48), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_3:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective

/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first

self.make_current()
tests/unit/xgb/test_xgboost.py: 14 warnings

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/xgb/test_xgboost.py: 11 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1183: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

mask = pd.Series(mask)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [5] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.

==== 1 failed, 762 passed, 12 skipped, 1211 warnings in 1514.07s (0:25:14) =====

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins15424315476906635711.sh

[sararb]

rerun tests

[nvidia-merlin-bot]

Click to view CI Results

GitHub pull request #815 of commit 6ebbb78330f12a3e5bc47283faccee5d6fda506f, no merge conflicts.
Running as SYSTEM
Setting status of 6ebbb78330f12a3e5bc47283faccee5d6fda506f to PENDING with url https://10.20.13.93:8080/job/merlin_models/1555/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/815/*:refs/remotes/origin/pr/815/* # timeout=10
 > git rev-parse 6ebbb78330f12a3e5bc47283faccee5d6fda506f^{commit} # timeout=10
Checking out Revision 6ebbb78330f12a3e5bc47283faccee5d6fda506f (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6ebbb78330f12a3e5bc47283faccee5d6fda506f # timeout=10
Commit message: "fix logit scaler for new ModelOutput API"
 > git rev-list --no-walk 2c2062dcb9c8ec545bf784415cf3acafca86d14d # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins7166896445195625129.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 775 items
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  0%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  1%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  2%]

tests/unit/datasets/test_social.py .                                     [  2%]

tests/unit/datasets/test_synthetic.py ......                             [  3%]

tests/unit/implicit/test_implicit.py .                                   [  3%]

tests/unit/lightfm/test_lightfm.py .                                     [  3%]

tests/unit/tf/test_core.py ......                                        [  4%]

tests/unit/tf/test_loader.py ................                            [  6%]

tests/unit/tf/test_public_api.py .                                       [  6%]

tests/unit/tf/blocks/test_cross.py ...........                           [  8%]

tests/unit/tf/blocks/test_dlrm.py ..........                             [  9%]

tests/unit/tf/blocks/test_interactions.py ...                            [  9%]

tests/unit/tf/blocks/test_mlp.py .................................       [ 13%]

tests/unit/tf/blocks/test_optimizer.py s................................ [ 18%]

.....................                                                    [ 20%]

tests/unit/tf/blocks/retrieval/test_base.py .                            [ 21%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 21%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ............            [ 22%]

tests/unit/tf/blocks/sampling/test_cross_batch.py .                      [ 22%]

tests/unit/tf/blocks/sampling/test_in_batch.py .                         [ 23%]

tests/unit/tf/core/test_aggregation.py .........                         [ 24%]

tests/unit/tf/core/test_base.py ..                                       [ 24%]

tests/unit/tf/core/test_combinators.py s....................             [ 27%]

tests/unit/tf/core/test_encoder.py ..                                    [ 27%]

tests/unit/tf/core/test_index.py ...                                     [ 27%]

tests/unit/tf/core/test_prediction.py ..                                 [ 28%]

tests/unit/tf/core/test_tabular.py ......                                [ 28%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 29%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 29%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 29%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 29%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 29%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 29%]

tests/unit/tf/examples/test_07_train_traditional_models.py .             [ 29%]

tests/unit/tf/examples/test_usecase_accelerate_training_by_lazyadam.py F [ 29%]

[ 29%]

tests/unit/tf/examples/test_usecase_ecommerce_session_based.py .         [ 30%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 30%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 30%]

tests/unit/tf/inputs/test_embedding.py ................................. [ 35%]

.......                                                                  [ 36%]

tests/unit/tf/inputs/test_tabular.py ..................                  [ 38%]

tests/unit/tf/layers/test_queue.py ..............                        [ 40%]

tests/unit/tf/losses/test_losses.py .......................              [ 43%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 43%]

tests/unit/tf/metrics/test_metrics_topk.py ........................      [ 46%]

tests/unit/tf/models/test_base.py s.......................               [ 49%]

tests/unit/tf/models/test_benchmark.py ..                                [ 50%]

tests/unit/tf/models/test_ranking.py ..................................  [ 54%]

tests/unit/tf/models/test_retrieval.py ...................Terminated

Build was aborted

Aborted by �[8mha:////4I6AZwo/1Z8Fal8AhZTEatjIwqNwCcqT21311HdysuK+AAAAlx+LCAAAAAAAAP9b85aBtbiIQTGjNKU4P08vOT+vOD8nVc83PyU1x6OyILUoJzMv2y+/JJUBAhiZGBgqihhk0NSjKDWzXb3RdlLBUSYGJk8GtpzUvPSSDB8G5tKinBIGIZ+sxLJE/ZzEvHT94JKizLx0a6BxUmjGOUNodHsLgAzWEgZu/dLi1CL9xJTczDwAj6GcLcAAAAA=�[0madmin

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins13186449769077777839.sh

[nvidia-merlin-bot]

Click to view CI Results

GitHub pull request #815 of commit 2c6405ce9fc2b75491206f235e4ca16685e1e4e2, no merge conflicts.
Running as SYSTEM
Setting status of 2c6405ce9fc2b75491206f235e4ca16685e1e4e2 to PENDING with url https://10.20.13.93:8080/job/merlin_models/1561/console and message: 'Pending'
Using context: Jenkins
Building on master in workspace /var/jenkins_home/workspace/merlin_models
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/815/*:refs/remotes/origin/pr/815/* # timeout=10
 > git rev-parse 2c6405ce9fc2b75491206f235e4ca16685e1e4e2^{commit} # timeout=10
Checking out Revision 2c6405ce9fc2b75491206f235e4ca16685e1e4e2 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 2c6405ce9fc2b75491206f235e4ca16685e1e4e2 # timeout=10
Commit message: "Merge branch 'main' into fix-logit-scaler"
 > git rev-list --no-walk 76f7861477466e2003bd4c37851d4b1a93234fd4 # timeout=10
[merlin_models] $ /bin/bash /tmp/jenkins14718599634250335050.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (5.4.0)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: importlib-resources>=1.4.0; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10; python_version < "3.9" in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: zipp>=3.1.0; python_version < "3.10" in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0; python_version < "3.9"->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.6.1, xdist-2.5.0, forked-1.4.0, cov-4.0.0
collected 775 items
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  0%]

tests/unit/datasets/test_ecommerce.py ..sss                              [  1%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  2%]

tests/unit/datasets/test_social.py .                                     [  2%]

tests/unit/datasets/test_synthetic.py ......                             [  3%]

tests/unit/implicit/test_implicit.py .                                   [  3%]

tests/unit/lightfm/test_lightfm.py .                                     [  3%]

tests/unit/tf/test_core.py ......                                        [  4%]

tests/unit/tf/test_loader.py ................                            [  6%]

tests/unit/tf/test_public_api.py .                                       [  6%]

tests/unit/tf/blocks/test_cross.py ...........                           [  8%]

tests/unit/tf/blocks/test_dlrm.py ..........                             [  9%]

tests/unit/tf/blocks/test_interactions.py ...                            [  9%]

tests/unit/tf/blocks/test_mlp.py .................................       [ 13%]

tests/unit/tf/blocks/test_optimizer.py s................................ [ 18%]

.....................                                                    [ 20%]

tests/unit/tf/blocks/retrieval/test_base.py .                            [ 21%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 21%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ............            [ 22%]

tests/unit/tf/blocks/sampling/test_cross_batch.py .                      [ 22%]

tests/unit/tf/blocks/sampling/test_in_batch.py .                         [ 23%]

tests/unit/tf/core/test_aggregation.py .........                         [ 24%]

tests/unit/tf/core/test_base.py ..                                       [ 24%]

tests/unit/tf/core/test_combinators.py s....................             [ 27%]

tests/unit/tf/core/test_encoder.py ..                                    [ 27%]

tests/unit/tf/core/test_index.py ...                                     [ 27%]

tests/unit/tf/core/test_prediction.py ..                                 [ 28%]

tests/unit/tf/core/test_tabular.py ......                                [ 28%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 29%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 29%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 29%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 29%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 29%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 29%]

tests/unit/tf/examples/test_07_train_traditional_models.py .             [ 29%]

tests/unit/tf/examples/test_usecase_accelerate_training_by_lazyadam.py . [ 29%]

[ 29%]

tests/unit/tf/examples/test_usecase_ecommerce_session_based.py .         [ 30%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 30%]

tests/unit/tf/inputs/test_continuous.py .....                            [ 30%]

tests/unit/tf/inputs/test_embedding.py ................................. [ 35%]

.......                                                                  [ 36%]

tests/unit/tf/inputs/test_tabular.py ..................                  [ 38%]

tests/unit/tf/layers/test_queue.py ..............                        [ 40%]

tests/unit/tf/losses/test_losses.py .......................              [ 43%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 43%]

tests/unit/tf/metrics/test_metrics_topk.py ........................      [ 46%]

tests/unit/tf/models/test_base.py s.......................               [ 49%]

tests/unit/tf/models/test_benchmark.py ..                                [ 50%]

tests/unit/tf/models/test_ranking.py ..................................  [ 54%]

tests/unit/tf/models/test_retrieval.py ................................  [ 58%]

tests/unit/tf/outputs/test_base.py ......                                [ 59%]

tests/unit/tf/outputs/test_classification.py ......                      [ 60%]

tests/unit/tf/outputs/test_contrastive.py ..............                 [ 62%]

tests/unit/tf/outputs/test_regression.py ..                              [ 62%]

tests/unit/tf/outputs/test_sampling.py ....                              [ 62%]

tests/unit/tf/outputs/test_topk.py .                                     [ 62%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 63%]

tests/unit/tf/prediction_tasks/test_multi_task.py ................       [ 65%]

tests/unit/tf/prediction_tasks/test_next_item.py .....                   [ 65%]

tests/unit/tf/prediction_tasks/test_regression.py .....                  [ 66%]

tests/unit/tf/prediction_tasks/test_retrieval.py .                       [ 66%]

tests/unit/tf/prediction_tasks/test_sampling.py ......                   [ 67%]

tests/unit/tf/transformers/test_block.py ....................            [ 70%]

tests/unit/tf/transformers/test_transforms.py ......                     [ 70%]

tests/unit/tf/transforms/test_bias.py ..                                 [ 71%]

tests/unit/tf/transforms/test_features.py s............................. [ 74%]

....................s......                                              [ 78%]

tests/unit/tf/transforms/test_negative_sampling.py .........             [ 79%]

tests/unit/tf/transforms/test_noise.py .....                             [ 80%]

tests/unit/tf/transforms/test_sequence.py ....................           [ 82%]

tests/unit/tf/transforms/test_tensor.py ...                              [ 83%]

tests/unit/tf/utils/test_batch.py ....                                   [ 83%]

tests/unit/tf/utils/test_dataset.py ..                                   [ 84%]

tests/unit/tf/utils/test_tf_utils.py .....                               [ 84%]

tests/unit/torch/test_dataset.py .........                               [ 85%]

tests/unit/torch/test_public_api.py .                                    [ 85%]

tests/unit/torch/block/test_base.py ....                                 [ 86%]

tests/unit/torch/block/test_mlp.py .                                     [ 86%]

tests/unit/torch/features/test_continuous.py ..                          [ 86%]

tests/unit/torch/features/test_embedding.py ..............               [ 88%]

tests/unit/torch/features/test_tabular.py ....                           [ 89%]

tests/unit/torch/model/test_head.py ............                         [ 90%]

tests/unit/torch/model/test_model.py ..                                  [ 90%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 92%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 92%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 93%]

tests/unit/utils/test_schema_utils.py ................................   [ 97%]

tests/unit/xgb/test_xgboost.py ....................                      [100%]
=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../../../usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41

/usr/local/lib/python3.8/dist-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning

tests/unit/datasets/test_ecommerce.py: 2 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 6 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 26 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 5 warnings

tests/unit/tf/core/test_index.py: 8 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/inputs/test_continuous.py: 4 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 26 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 38 warnings

tests/unit/tf/models/test_retrieval.py: 60 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning

tests/unit/tf/transformers/test_block.py: 15 warnings

tests/unit/tf/transforms/test_bias.py: 2 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_noise.py: 1 warning

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 9 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 3 warnings

tests/unit/xgb/test_xgboost.py: 18 warnings

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_ecommerce.py: 2 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 5 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 26 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 7 warnings

tests/unit/tf/core/test_index.py: 3 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/inputs/test_continuous.py: 4 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 26 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 36 warnings

tests/unit/tf/models/test_retrieval.py: 32 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/transformers/test_block.py: 9 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 7 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 2 warnings

tests/unit/xgb/test_xgboost.py: 17 warnings

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_entertainment.py: 1 warning

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 2 warnings

tests/unit/tf/core/test_prediction.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 2 warnings

tests/unit/tf/inputs/test_embedding.py: 9 warnings

tests/unit/tf/inputs/test_tabular.py: 8 warnings

tests/unit/tf/models/test_ranking.py: 20 warnings

tests/unit/tf/models/test_retrieval.py: 4 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings

tests/unit/xgb/test_xgboost.py: 12 warnings

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:960: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/blocks/retrieval/test_two_tower.py: 1 warning

tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/models/test_retrieval.py: 54 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_fileb_0twgld.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/core/test_combinators.py::test_parallel_block_select_by_tags

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/tabular.py:614: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

elif isinstance(self.feature_names, collections.Sequence):
tests/unit/tf/core/test_index.py: 5 warnings

tests/unit/tf/models/test_retrieval.py: 26 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

tests/unit/tf/utils/test_dataset.py: 1 warning

/var/jenkins_home/workspace/merlin_models/models/merlin/models/utils/dataset.py:75: DeprecationWarning: unique_rows_by_features is deprecated and will be removed in a future version. Please use unique_by_tag instead.

warnings.warn(
tests/unit/tf/models/test_base.py::test_model_pre_post[True]

tests/unit/tf/models/test_base.py::test_model_pre_post[False]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]

tests/unit/tf/models/test_base.py::test_freeze_sequential_block

tests/unit/tf/models/test_base.py::test_freeze_unfreeze

tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks

/usr/local/lib/python3.8/dist-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.

super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_base.py::test_retrieval_model_query

tests/unit/tf/models/test_base.py::test_retrieval_model_query

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py:294: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

tensor_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(tensor)))
tests/unit/tf/models/test_ranking.py::test_deepfm_model_only_categ_feats[False]

tests/unit/tf/models/test_ranking.py::test_deepfm_model_categ_and_continuous_feats[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:569: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask/GatherV2:0", shape=(None, 48), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_3:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:335: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective

/usr/local/lib/python3.8/dist-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first

self.make_current()
tests/unit/xgb/test_xgboost.py: 14 warnings

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/xgb/test_xgboost.py: 11 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1183: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

mask = pd.Series(mask)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:62: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:78: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:92: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [5] ../../../../../usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/test_util.py:2746: Not a test.

========= 763 passed, 12 skipped, 1211 warnings in 1513.19s (0:25:13) ==========

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[merlin_models] $ /bin/bash /tmp/jenkins3178266470611072434.sh