assert torch.all(all_layers[-1] == last_layer_features)

[HAWLYQ]

❓ Questions and Help

Before asking:

1. search the issues.
2. search the docs.

What is your question?

Hi, @myleott , I meet an AssertionError when I run the demo code to extract features from RoBERTa. It shows the last layer features is not the same as 'all_layers[-1]'.

Code

import torch
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')
tokens = roberta.encode('Hello world!')
last_layer_features = roberta.extract_features(tokens)
all_layers = roberta.extract_features(tokens, return_all_hiddens=True)
assert torch.all(all_layers[-1] == last_layer_features)

What have you tried?

There is nothing wrong with the size of last_layer_features and the length of all_layers.
The outputs of last_layers_features and all_layers[-1] are shown below:

What's your environment?

• fairseq Version (e.g., 1.0 or master):0.9.0
• PyTorch Version (e.g., 1.0) 1.4.0
• OS (e.g., Linux):linux
• How you installed fairseq (pip, source):pip
• Build command you used (if compiling from source):
• Python version:3.6
• CUDA/cuDNN version:10.0
• GPU models and configuration:-
• Any other relevant information:

[myleott]

You need to call roberta.eval() to disable dropout first, otherwise the calls will return slightly different results.

This works for me:

import torch
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')
roberta.eval()  # <-- this disables dropout
tokens = roberta.encode('Hello world!')
last_layer_features = roberta.extract_features(tokens)
all_layers = roberta.extract_features(tokens, return_all_hiddens=True)
assert torch.all(all_layers[-1] == last_layer_features)