Feature request: top-level datasets to be more-expressive

[ijlyttle]

This may be a follow-on to #3806, and might also affect vega/altair#951.

My suggested implementation may be problematic, but I hope it will be enough to begin a discussion.

Problem

Currently we can use top-level datasets to map names to inline datasets, see spec-datasets. This is convenient for keeping the spec lightweight and comprehensible, especially for layered charts.

If I want to do the same thing, instead referring to a dataset using a url, I (think I) have to use the format shown in spec-csv. The issue is that if I want to switch out my dataset, I need to do it in two places.

What I would like to be able to do

I would like to be able to do something like shown in spec-proposed, where datasets maps names to objects containing data specifications (not just inline values).

As a further request (which may be completely unreasonable), I wonder if datasets could map names to objects that would contain data and, optionally, transform specifications, as transformed data remains data. For example:

{
  ...,
  "datasets": {
    "data-001": {
      "data": {...},
      "transform": {...}
    }
  },
  ...
}

spec-datasets

{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.4.1.json",
  "config": {"view": {"height": 300, "width": 400}},
  "datasets": {
    "data-001": [
      {"a": "A","b": 28}, {"a": "B","b": 55}, {"a": "C","b": 43},
      {"a": "D","b": 91}, {"a": "E","b": 81}, {"a": "F","b": 53},
      {"a": "G","b": 19}, {"a": "H","b": 87}, {"a": "I","b": 52}
    ]
  },
  "layer": [
    {
      "data": {"name": "data-001"},
      "encoding": {
        "x": {"field": "a", "type": "nominal"},
        "y": {"field": "b","type": "quantitative"},
        "opacity": {"value": 0.5}
      },
      "mark": "bar"
    },
    {
      "data": {"name": "data-001"},
      "encoding": {
        "x": {"field": "a", "type": "nominal"},
        "y": {"field": "b","type": "quantitative"}
      },
      "mark": "line"
    }
  ]
}

spec-csv

{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.4.1.json",
  "config": {"view": {"height": 300, "width": 400}},
  "layer": [
    {
      "data": {
        "url": "https://ijlyttle.github.io/vega-lite-demo/data-raw/data_01.csv",
        "format": {"type": "csv"}
      },
      "encoding": {
        "x": {"field": "category", "type": "nominal"},
        "y": {"field": "number","type": "quantitative"},
        "opacity": {"value": 0.5}
      },
      "mark": "bar"
    },
    {
      "data": {
        "url": "https://ijlyttle.github.io/vega-lite-demo/data-raw/data_01.csv",
        "format": {"type": "csv"}
      },
      "encoding": {
        "x": {"field": "category", "type": "nominal"},
        "y": {"field": "number","type": "quantitative"}
      },
      "mark": "line"
    }
  ]
}

spec-proposed

{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.4.1.json",
  "config": {"view": {"height": 300, "width": 400}},
  "datasets" : {
    "data-001": {
      "data": {
        "url": "https://ijlyttle.github.io/vega-lite-demo/data-raw/data_01.csv",
        "format": {"type": "csv"}
      },
    }
  },
  "layer": [
    {
      "data": {"name": "data-001"},
      "encoding": {
        "x": {"field": "category", "type": "nominal"},
        "y": {"field": "number","type": "quantitative"},
        "opacity": {"value": 0.5}
      },
     "mark": "bar"
    },
    {
      "data": {"name": "data-001"},
      "encoding": {
        "x": {"field": "category", "type": "nominal"},
        "y": {"field": "number","type": "quantitative"},
      },
      "mark": "line"
    }
  ]
}

[kanitw]

I can see this being useful. Btw, I think we don't need additional data nesting in

    "data-001": {
      "data": {
        "url": "https://ijlyttle.github.io/vega-lite-demo/data-raw/data_01.csv",
        "format": {"type": "csv"}
      },
    }

This should be sufficient:

    "data-001": {
      "url": "https://ijlyttle.github.io/vega-lite-demo/data-raw/data_01.csv",
      "format": {"type": "csv"}
    }

[domoritz]

I think we could support top level URL data but I'm hesitant about transforms.

[ijlyttle]

For me, being able to use top-level URL data is much more important than the transforms - I just thought to bring it up while the topic might be "open".

If transforms will not be available at the top-level, then there is no need for the data nesting, so I would agree completely with @kanitw's suggestion. (Congrats btw!)

[kanitw]

I think we could revise if we want the support for transform later.