Document arrow data types to perspective data schema load init mapping

[RandomFractals]

Support Question

I had to do this fine dance after looking at your utils.js and perspective.js#ArrowColumnLoader to figure out how to inject arrow data types schema and translations into perspective.js from arrow data schema in my config.ts: https://github.com/RandomFractals/vscode-data-preview/blob/master/src/config.ts

// arrow to data view type mappings
// see: https://github.com/finos/perspective/blob/master/packages/perspective/src/js/utils.js
// and https://github.com/finos/perspective/blob/master/packages/perspective/src/js/perspective.js#ArrowColumnLoader
export const dataTypes = { 
  "Binary": "string",
  "Bool": "boolean",
  "Date": "date",
  "Dictionary": "string",
  "Float32": "float",
  "Float64": "float",
  "Int8": "integer",
  "Int16": "integer",
  "Int32": "integer",
  "Int64": "integer",
  "Timestamp": "datetime",
  "Utf8": "string",
};

after hours of code browsing, I found that your load({object, object} method lets me set data schema and utils.js#get_column_types() helped me infer what your corresponding arrow data types are.

Could use better enum or docs on this.

I ended up doing this in my data.preview.ts for arrow data types mapping:

    // remap arrow data schema to columns for data viewer
    this._schema = {};
    dataTable.schema.fields.map(field => {
      let fieldType: string = field.type.toString();
      const typesIndex: number = fieldType.indexOf('<');
      if (typesIndex > 0) {
        fieldType = fieldType.substring(0, typesIndex);
      }
      this._schema[field.name] = config.dataTypes[fieldType];
    });

and then this tricky biz in data.preview.html:

            viewer.restore(viewConfig);
            if (data.schema !== undefined && Object.keys(data.schema).length > 0) {
              // viewer.columns = viewConfig['columns'];
              viewer.load(data.schema);
              viewer.update(tableData);
            }
            else {
              viewer.load(tableData);
            }

It worked! I just wish I did not have to spend hours to figure it out. Thx!

BTW, your arrow data support is awesome! Check out mine too! :)

[RandomFractals]

@texodus & @timkpaine any updates on this?

[texodus]

No update; this is a valid issue, but not in my immediate dev plan. For now, perspective supports well only the base types supported by Javascript: 32bit signed int, double, string, bool, datetime. PRs welcome, as always.

[RandomFractals]

you keep saying that yet dates still could use some work :(

Can you at least address this 1 pesky issue with dates in your next v0.3.7 rc release?

#524

You end all your close tickets with PRs are welcomed, yet you don't merge them due to CLA or some other not developed in house reasons. That 1 is kind of a sticky point for most devs working with data and your lib.

Thx in advance!

[RandomFractals]

you can close this ticket. I can find my ways around your repo :)