Table function for supporting prometheus query_range function

[vamsimanohar]

Description

• This CR includes changes for initial implementation of table functions targeting prometheus API.
• Table functions are functions which return tables as output. Table functions can be defined for any of the connector and will be pushed down.
• New query_range table function is defined to support pass through promQL query directly to prometheus.

These functions can be called either with named parameters or positioned parameters.

Example queries for query_range.
Named parameters query.
source = prometheus.query_range(query="promQL", start=123, end=145, step=45)

Positioned parameters query.
source = prometheus.query_range("promQL", 123, 145, 45)

Issues Related:

• Implement promQl for range queries in Prometheus #623

Check List

• [x ] New functionality includes testing.
  • [x ] All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• [ x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.18%. Comparing base (b30d156) to head (3526376).
Report is 528 commits behind head on 2.x.

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x     #875      +/-   ##
============================================
- Coverage     97.90%   95.18%   -2.72%     
- Complexity     3072     3126      +54     
============================================
  Files           293      309      +16     
  Lines          7588     8392     +804     
  Branches        490      618     +128     
============================================
+ Hits           7429     7988     +559     
- Misses          158      350     +192     
- Partials          1       54      +53     

Flag	Coverage Δ
query-workbench	62.76% <ø> (?)
sql-engine	97.94% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vamsimanohar]

I have some suggestion about the table function. In current implementation, it is not part of the existing expression system. In particular, the table function has its own interface ConnectorTableFunction and its signature is resolved in StorageEngine.getTableFunction. Can we introduce it into the existing expression system instead of making it a special case?

I checked our code and found the resolution logic is captured by FunctionResolver interface. There is a default one and RelevanceFunctionResolver which is similar to what table function wants.

// During initialization:
for storage : allStorageEngines:
    functionRepository.register(storage.getFunctions())

class PrometheusStorageEngine:
    def List<FunctionResolver> getFunctions():
        return [queryRange(), query() ...]

class Analyzer:
    def visitTableFunction(astNode):
        tableFunc = functionRepository.compile(astNode)
        env = { (arg.name, arg.value) for astNode.argument() }
        return LogicalRelation(tableFunc.valueOf(env)).  // use existing valueOf or add new interface if inappropriate

class PrometheusFunctions:
    def queryRange():
        return functionBuilder.build...

    def query():
        return functionBuilder.build...

Let me know if this makes sense. Thanks!

Similar to first iteration except that functions are exposed via storageengine. Will try to incorporate.

[penghuo]

please update the docs.

[vamsimanohar]

please update the docs.

I will raise one this week for catalogs and all.

[dai-chen]

Thanks for the changes!