Form preview fails with error argument of type 'int' is not iterable

[tinok]

Description

When previewing specific form, preview fails to load (argument of type 'int' is not iterable).

Steps to Reproduce

1. Open this project in the formbuilder (user: tinokreutzer)
2. Click on preview

Expected behavior

Form previews

Actual behavior

Form preview fails. Error:

Additional details

When exporting the XLS and importing as new project, the form preview succeeds somehow.

[jnm]

@noliveleger, here's an example of how trying to handle exceptions gracefully can make debugging much harder:

kpi/kpi/models/asset.py

Lines 885 to 901 in 550dc43

	except Exception as err:
	err_message = unicode(err)
	logging.error('Failed to generate xform for asset', extra={
	'src': source,
	'id_string': id_string,
	'uid': self.uid,
	'_msg': err_message,
	'warnings': warnings,
	})
	xml = ''
	details.update({
	u'status': u'failure',
	u'error_type': type(err).__name__,
	u'error': err_message,
	u'warnings': warnings,
	})
	return (xml, details)

If the exception weren't caught, we'd get a stack trace with local variables in Sentry. Instead, we get src, id_string, uid, _msg, and warnings (and _msg isn't even shown?): https://sentry.kbtdev.org/kobo/kpi-backend/issues/2837/.

I think this is basically the same situation as #1705, but we'll have to figure out whether some of the exceptions that could be raised here have to do with user error, e.g. invalid XLSForms.

For now, I'll try calling FormPack.to_xml() manually inside the Django shell to get a useful traceback.

[jnm]

Related to the Pyxform upgrade (specifically https://github.com/XLSForm/pyxform/pull/118/files#diff-b29aeb282b118614b8333cdfc19f700dR76):

>>> from formpack import FormPack
>>> a = Asset.objects.get(uid='agzXjv6TETFuJ8ibpV4GGk')
>>> FormPack({'content': a.content})[0].to_xml()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/srv/pip/src/formpack/src/formpack/version.py", line 287, in to_xml
    self.to_dict(remove_sheets=['translations', 'translated'],
  File "/srv/pip/src/formpack/src/formpack/utils/xform_tools.py", line 39, in formversion_pyxform
    imported_survey_json = workbook_to_json(content)
  File "/usr/local/lib/python2.7/dist-packages/pyxform/xls2json.py", line 395, in workbook_to_json
    replace_smart_quotes_in_dict(choice_item)
  File "/usr/local/lib/python2.7/dist-packages/pyxform/xls2json.py", line 80, in replace_smart_quotes_in_dict
    if smart_quote in value:
TypeError: argument of type 'int' is not iterable

[jnm]

replace_smart_quotes_in_dict() attempts to check if smart_quote in value without first making sure that value is a string (and in this case, it's an int from 'order': 0). Troublesome dict:

{u'$autovalue': u'option_1',
 u'$kuid': u'sqUjrGI0R',
 u'label': u'Option 1',
 u'list_name': u'wj5sl24',
 u'name': u'option_1',
 u'order': 0}

[jnm]

Why does it work when exporting the form as XLS and re-importing? The order value is read as a string when importing, e.g. for the following choices sheet, where name is a text cell and order is a number:

list_name	name	label	order
rx3hr66	1	Option 1	1

...the resulting JSON in KPI is:

{
    "$autovalue": "1",
    "name": "1",
    "label": [
        "Option 1"
    ],
    "$kuid": "zIM7x1gzk",
    "list_name": "rx3hr66",
    "order": "1"
}

[jnm]

Since @tinok reports this form was created entirely in the form builder, I was curious as to how the order column appeared, and why it was present for this choice list:

{
    "$autovalue": "option_1",
    "name": "option_1",
    "label": [
        "Option 1"
    ],
    "$kuid": "sqUjrGI0R",
    "list_name": "wj5sl24",
    "order": 0
},

...but not others:

{
    "$autovalue": "option_1",
    "list_name": "aa9wy77",
    "$kuid": "iug7hoytZ",
    "name": "option_1",
    "label": [
        "Option 1"
    ]
},

It seems that the form builder adds order only after changing the order of the choices by dragging and dropping.

[jnm]

PR'd a fix to Pyxform for this XLSForm/pyxform#237
PR'd a workaround to kobotoolbox/formpack#180