Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time after stateful and refactor the old calculation #164

Closed
compae opened this issue May 18, 2015 · 0 comments
Closed

Time after stateful and refactor the old calculation #164

compae opened this issue May 18, 2015 · 0 comments

Comments

@compae
Copy link
Member

compae commented May 18, 2015

in the last release, the outputs had all the logic to calculate the time of the event and granularity, if there is not calculated and added to the row.

With the new functionality stateful time aggregation is calculated and sent to subsequent stages of the aggregation key pair that make up the RDD.

Now all data are aggregated by time and this is what has to be stored at the outputs.
Be especially careful with the order of the fields, in several places we have to eliminate the time if there dimensionValue and insert it in the right order with the value of the key. Thus we have clearly identified the schemas with rows and values.
Special care must be taken with the process of multiplexing of a rollup.

Without this refactor and implementation the old aggregations values is wrong

@compae compae self-assigned this May 18, 2015
@danielcsant danielcsant added this to the Release 0.6.0 milestone Jul 1, 2015
@compae compae closed this as completed Nov 11, 2016
stratiocommit pushed a commit that referenced this issue Oct 23, 2019
* Upsert in jdbc not update search fields (#155)

* [SPARTA-1247] Fix/java heap size (#164)

* [SPARTA-1247] Added OS memory to heap size

* Leave 512MB as min

* Improved readability

* Set newMemorySize only to container

* Control empty rdd partitions in JDBC save/upsert (#179)

* Minor changes from branch incompatibles with new Sparta Version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants