optionally hit worker even if some sources didn't hold data for the tile

[tcql]

Per this code:

for (var i = 0; i < results.length; i++) {
  data[sources[i].name] = results[i];
  if (!results[i]) return process.send({reduce: true});
}

the worker bails out and returns a reduce event if any source doesn't have data for the requested tile. This is usually great, but in some cases where you want to compare disparate data sources and are relying on reduce events to send back information about how much data each source does or doesn't exist in a tile, you end up losing information.

For example, if I want to find the length of roads in San Francisco that are matched by GPS datapoints. I would like to keep a tally of the total length of road in the bbox, as well as how much is matchable by GPS points. Right now, if there is no GPS data in the tile, we bail out, so I'm missing some of the total length information.

To maintain compatibility and provide optimization for the usual cases where you want this bail-out behavior, I'm proposing we add a tile-reduce option for this, maybe requireAllSources: false (defaulted true).

cc @morganherlocker @aaronlidman @mourner

[mourner]

👍 makes sense, let's add an option.

[morganherlocker]

I'm proposing we add a tile-reduce option for this, maybe requireAllSources: false (defaulted true).

I like this proposal, but might counter with something slightly more fine-grained:

val	state
requireData:'all'	all sources must have data (current defualt)
requireData:'any'	at least 1 source must have data
requireData:'none'	function is called even on empty tiles

The any option is actually what I thought the current default was, and would cover cases like Tim's road analysis. The none option is one I have wanted to have for certain unusual cases, such as jobs that request data from a non-tiled source (I have done this with satellite data and can think of other cases), and in cases where results might need to be logged out somewhere, even (especially?) in cases of no data.

Thoughts?

[tcql]

bump. any further thoughts here? I'm fine with having 3 modes per @morganherlocker's

val	state
requireData:'all'	all sources must have data (current default)
requireData:'any'	at least 1 source must have data
requireData:'none'	function is called even on empty tiles

@mourner @aaronlidman

[mourner]

I'm fine with the API proposed above too.

[jenningsanderson]

👋 Any update on this? I'd love to see this feature supported as well.

[iandees]

Hello from the future! I am running into this today and would also love to have this feature. I'm hacking in a workaround for now, but excited for #110.