Add VQA V2.0 and Visual Dialog V0.9. #54

jiasenlu · 2017-05-08T08:55:48Z

This pull request contains
1: add vqa_v2 task loader.
2: add VisDial_v0.9 task loader.
3: disable download COCO image as default
4: move COCO image path to "--download_path"

To test the new added task

For VQA 2.0: python examples/display_data.py -t vqa_coco2014_v2 --download-path 'path_to_COCO_img'

For Visual Dialog: python examples/display_data.py -t visdial

Currently, the Visual Dialog inherit from the default "DialogTeacher" class. There is no placeholder for the additional image information. The "DialogData" class with the format [(x, y, r, c), new_episode?], could we extend this to [(x, y, r, c, i), new_episode?] where "i" is some optional additional information such as image_id? If so, I can send another pull request to modify that.

facebook-github-bot · 2017-05-08T08:55:53Z

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla - and if you have received this in error or have any questions, please drop us a line at cla@fb.com. Thanks!

If you are contributing on behalf of someone else (eg your employer): the individual CLA is not sufficient - use https://developers.facebook.com/opensource/cla?type=company instead. Contact cla@fb.com if you have any questions.

facebook-github-bot · 2017-05-08T08:58:53Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

alexholdenmiller · 2017-05-08T17:28:30Z

Thanks Jiasen!! I think we'd like to keep the dataset downloading the images automatically (to datapath)--I know it's large, but users should only need to do it once anyways. Everything in downloads is also automatically downloaded (currently just the Memnn github repo).

As far as DialogTeacher/DialogData, I think we want to write a new one instead that includes some of the data loading from your vqa_coco2014 (v1) and prepares us to handle the preprocessing options that we talked about before. We don't want image_id as part of the action/observation dictionary--just images themselves--so we want to include the image loading as part of that data class.

alexholdenmiller · 2017-05-08T16:49:44Z

parlai/tasks/vqa_coco2014/agents.py

@@ -36,7 +35,7 @@ def _path(opt):
    annotation_path = os.path.join(opt['datapath'], 'VQA-COCO2014',
        annotation_suffix + '_annotations.json')

-    image_path = os.path.join(opt['datapath'], 'VQA-COCO2014', img_suffix)


actually, we'd like to keep the images in datapath

I see. Then I think we might not want the image stay in 'VQA-COCO2014' folder, other task such as Visual Dialog may also use the COCO image. How about put the image under COCO-IMG ? Then, multiple task can share the image data.

Sounds great! Yeah giving it the most general name for the data file is perfect, and then multiple tasks can depend on it (and it won't rebuild it if it's already there).

alexholdenmiller · 2017-05-08T16:51:56Z

parlai/tasks/vqa_coco2014_v2/agents.py

+            anno = self.annotation['annotations'][self.episode_idx]
+            answers = [ans['answer'] for ans in anno['answers']]
+        else:
+            answers = ['fake_answer']


actually, just leave the labels field of the dict empty (accessing it gives KeyError, not None)

ok, thanks. will change this.

facebook-github-bot · 2017-05-10T19:18:49Z

@jiasenlu updated the pull request - view changes

Summary: Fix the VQA, VisalDialog, VQA v2.0 based on last update of ParAI. Test Plan: python examples/display_data.py -t vqa_coco2014 python examples/display_data.py -t vqa_coco2014_v2 python examples/display_data.py -t visdial Reviewers: Subscribers: Tasks: Tags: Blame Revision:

facebook-github-bot · 2017-05-10T21:01:14Z

@jiasenlu updated the pull request - view changes

facebook-github-bot · 2017-05-10T21:09:01Z

@jiasenlu updated the pull request - view changes

facebook-github-bot · 2017-05-11T04:50:54Z

@jiasenlu updated the pull request - view changes

alexholdenmiller

looking really good overall! how are you testing if the images are loading properly?

would be awesome to get a model in next so that we can try to train it and see if we can reproduce the results from the paper

alexholdenmiller · 2017-05-11T15:31:19Z

parlai/core/dialog_teacher.py

@@ -265,7 +265,7 @@ def get(self, episode_idx, entry_idx=0):
                table['reward'] = entry[2]
                if len(entry) > 3:
                    table['label_candidates'] = entry[3]
-                    if len(entry) > 4 and not opt.get('no_images', False):
+                    if len(entry) > 4 and not self.opt.get('no_images', False):


ah good catch thank you

alexholdenmiller · 2017-05-11T15:36:16Z

parlai/tasks/vqa_coco2014_v2/agents.py

+        self.len = len(self.ques['questions'])
+
+class DefaultTeacher(OeTeacher):
+    pass


does v2 have a multiple-choice version?

no, VQA v2.0 doesn't have the multiple-choice now.

jaseweston · 2017-05-11T15:43:54Z

yes, we need to have at least a simple example of visualizing the images..

…

On Thu, May 11, 2017 at 11:37 AM, Alexander Miller ***@***.*** > wrote: ***@***.**** approved this pull request. looking really good overall! how are you testing if the images are loading properly? would be awesome to get a model in next so that we can try to train it and see if we can reproduce the results from the paper ------------------------------ In parlai/core/dialog_teacher.py <#54 (comment)> : > @@ -265,7 +265,7 @@ def get(self, episode_idx, entry_idx=0): table['reward'] = entry[2] if len(entry) > 3: table['label_candidates'] = entry[3] - if len(entry) > 4 and not opt.get('no_images', False): + if len(entry) > 4 and not self.opt.get('no_images', False): ah good catch thank you ------------------------------ In parlai/tasks/vqa_coco2014_v2/agents.py <#54 (comment)> : > + return shared + + def _setup_data(self, data_path, annotation_path): + print('loading: ' + data_path) + with open(data_path) as data_file: + self.ques = json.load(data_file) + + if self.datatype != 'test': + print('loading: ' + annotation_path) + with open(annotation_path) as data_file: + self.annotation = json.load(data_file) + + self.len = len(self.ques['questions']) + +class DefaultTeacher(OeTeacher): + pass does v2 have a multiple-choice version? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#54 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKjk-HHAo67A1CAmU1TddOwo2BCjWHSXks5r4ysogaJpZM4NTqgb> .

alexholdenmiller · 2017-05-11T16:20:47Z

simple example which prints ascii version of images (after pip install asciimatics), or we could write our own similar one

import sys
from asciimatics.renderers import ImageFile
img = ImageFile(sys.argv[1], height=30)
print(img)

maybe we want something like this in display_data?

jiasenlu · 2017-05-11T22:44:27Z

Yeah, I think we should do that. Maybe we can add this in a separate pull request?

alexholdenmiller · 2017-05-12T14:37:52Z

Just pushed this to get that fix in

Jiasen Lu added 3 commits May 7, 2017 19:09

Move COCO image path to --download_path

75ca32d

add VQA_v2.0

7511160

add VisDial_V0.9

a134a3d

facebook-github-bot added the CLA Signed label May 8, 2017

alexholdenmiller suggested changes May 8, 2017

View reviewed changes

Jiasen Lu added 3 commits May 8, 2017 10:33

Merge remote-tracking branch 'upstream/master'

911740c

Merge branch 'master' of https://github.com/facebookresearch/ParlAI

a5dd2d3

Merge remote-tracking branch 'upstream/master'

75d7145

forget to add some file last pull request, fix in this one

6255107

fix v2 agents

7abd701

alexholdenmiller approved these changes May 11, 2017

View reviewed changes

jiasenlu mentioned this pull request May 12, 2017

There may be an error in line 268 in dialog_teacher.py #64

Closed

alexholdenmiller merged commit 10453fd into facebookresearch:master May 12, 2017

snyk-bot mentioned this pull request Jan 17, 2023

[Snyk] Upgrade async-lock from 1.0.0 to 1.4.0 Kendralabs/ParlAI#3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add VQA V2.0 and Visual Dialog V0.9. #54

Add VQA V2.0 and Visual Dialog V0.9. #54

jiasenlu commented May 8, 2017

facebook-github-bot commented May 8, 2017

facebook-github-bot commented May 8, 2017

alexholdenmiller commented May 8, 2017

alexholdenmiller May 8, 2017

jiasenlu May 9, 2017

alexholdenmiller May 9, 2017

alexholdenmiller May 8, 2017

jiasenlu May 9, 2017

facebook-github-bot commented May 10, 2017

facebook-github-bot commented May 10, 2017

facebook-github-bot commented May 10, 2017

facebook-github-bot commented May 11, 2017

alexholdenmiller left a comment

alexholdenmiller May 11, 2017

jiasenlu May 11, 2017

alexholdenmiller May 11, 2017

jiasenlu May 11, 2017 •

edited

Loading

jaseweston commented May 11, 2017 via email

alexholdenmiller commented May 11, 2017 •

edited

Loading

jiasenlu commented May 11, 2017

alexholdenmiller commented May 12, 2017

Add VQA V2.0 and Visual Dialog V0.9. #54

Add VQA V2.0 and Visual Dialog V0.9. #54

Conversation

jiasenlu commented May 8, 2017

facebook-github-bot commented May 8, 2017

facebook-github-bot commented May 8, 2017

alexholdenmiller commented May 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented May 10, 2017

facebook-github-bot commented May 10, 2017

facebook-github-bot commented May 10, 2017

facebook-github-bot commented May 11, 2017

alexholdenmiller left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiasenlu May 11, 2017 • edited Loading

Choose a reason for hiding this comment

jaseweston commented May 11, 2017 via email

alexholdenmiller commented May 11, 2017 • edited Loading

jiasenlu commented May 11, 2017

alexholdenmiller commented May 12, 2017

jiasenlu May 11, 2017 •

edited

Loading

alexholdenmiller commented May 11, 2017 •

edited

Loading