"ValueError: 'col1' is not in list" when using "select" argument in DataPortal().load()

[fuma900]

Context

I'm using Python 3.6.5 and pyomo 5.5.0.

data = DataPortal(model=m)
data.load(
    filename='data.csv',
    index=('hours', 'UPs'),
    select=('col1', 'col2', 'col3', 'col4', 'col5', 'col6'),
    param=('availability', 'pun', 'pz', 'msdDownPrice', 'msdDownMaxQty', 'msdUpPrice')
)

Problem

Trying to load a csv file into the model through DataPortal i'm getting this error:
ValueError: 'col1' is not in list
The csv is well formatted and its headers are successfully parsed using 3rd parties libraries (e.g. pandas' read_csv).

Solution

Debugging the problem i found out that the headers were parsed as ['col1', 'col2', 'col3', ... ].
Thanks to this SO post i was able to address the issue by changing the encoding from python default to encoding='utf-8-sig' in csv_table.py - line 37.
I guess the issue was primarily caused by excel's "save as csv" function which most probably prepend a UTF-8 Byte Order Mark.

I think it could be useful to expose an optional encoding argument in the load function to let developers manage encoding-related issues on their own.

[mrmundt]

In an effort to clean up issues, we are moving some discussions to our Archived Design Discussions wiki.