Add Akumuli database support #73

Lazin · 2019-05-17T11:04:29Z

No description provided.

CLAassistant · 2019-05-17T11:04:37Z

All committers have signed the CLA.

codecov-io · 2019-05-17T11:08:39Z

Codecov Report

Merging #73 into master will decrease coverage by 0.03%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master      #73      +/-   ##
==========================================
- Coverage   69.61%   69.58%   -0.04%     
==========================================
  Files         103      104       +1     
  Lines        5704     5810     +106     
==========================================
+ Hits         3971     4043      +72     
- Misses       1658     1689      +31     
- Partials       75       78       +3

Impacted Files	Coverage Δ
internal/inputs/utils.go	`78.26% <ø> (ø)`	⬆️
internal/inputs/generator_data.go	`88.81% <0%> (-1.19%)`	⬇️
internal/inputs/generator_queries.go	`73.17% <100%> (+0.16%)`	⬆️
cmd/tsbs_generate_data/serialize/akumuli.go	`92.85% <92.85%> (ø)`
query/stat_processor.go	`16.12% <0%> (-5.62%)`	⬇️
query/stats.go	`100% <0%> (ø)`	⬆️
query/benchmarker.go	`75.78% <0%> (+1.39%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 332c6cf...3429e43. Read the comment docs.

RobAtticus · 2019-06-17T19:06:08Z

Hi @Lazin , I see you've updated this (though it does need a few more conflicts fixed). I will try to look into this this week so we can help move it forward, now that CrateDB is merged in.

Lazin · 2019-06-18T20:48:51Z

Looking forward to it! I fixed merge conflicts but it looks like something is broken on CI now. I'll investigate this tomorrow.

RobAtticus · 2019-08-28T15:55:53Z

Hi @Lazin , I apologize for not getting back to you sooner. We've been really heads down with a new use case that we are just now releasing:
#87

It does change a lot of things structurally that might affect your PR...would you mind looking into that and pinging me when the PR is ready for re-review?

Lazin · 2019-09-11T16:08:12Z

Hi @RobAtticus ,
I updated and tested PR. It can be reviewed now.

atanasovskib · 2019-09-18T14:02:48Z

Hello @Lazin
The akumuli_test.go file contains some errors (tag values can be any type now, no need to have strings only, and the Point struct accepts an interface{} now)

Lazin · 2019-09-18T15:38:53Z

Hi @blagojts
I updated akumuli_test.go and merged with master.

atanasovskib

Most of the files look good. I have some minor change requests.
I also couldn't test out the data loading and query execution. But that may be because of me (at the moment of writing this https://akumuli.gitbook.io/docs is unavailable)

I ran Akumuli with docker
docker run -d -p 8181:8181 -p 8282:8282 -p 8383:8383 -p 4242:4242 akumuli/akumuli:0.7.45-skylake
And when I try to run tsbs_load_akumuli -file=aku.in I get the error Can't establish connection with http://localhost:8282 but the db is up and running and curl http://localhost:8181/api/stats returns an ok response

cmd/tsbs_generate_data/serialize/akumuli.go

atanasovskib · 2019-09-18T14:25:35Z

cmd/tsbs_generate_data/serialize/akumuli.go

+	series := string(buf[HeaderLength:])
+	if !s.bookClosed {
+		// Save point for later
+		if id, ok := s.book[series]; ok {


this if/else is a bit long, maybe you can extract parts of it to small functions

cmd/tsbs_generate_data/serialize/akumuli_test.go

atanasovskib · 2019-09-18T14:28:17Z

cmd/tsbs_generate_data/serialize/akumuli_test.go

+
+	for _, c := range cases {
+		actualCnt := strings.Count(got, c.expValue)
+		if actualCnt != c.expCount {


the serialized content is not tested/asserted

It's checked on line 73. If one of the test samples is not present in the output required number of times it will fail.

cmd/tsbs_generate_queries/databases/akumuli/devops.go

atanasovskib · 2019-09-19T11:35:36Z

cmd/tsbs_generate_queries/databases/akumuli/devops.go

@@ -0,0 +1,294 @@
+package akumuli


I can't find the GroupByOrderByLimit query generation method

It's not possible to implement using Akumuli query API. Date/time truncation is not supported.

atanasovskib · 2019-09-19T12:06:08Z

cmd/tsbs_load_akumuli/creator.go

@@ -0,0 +1,32 @@
+package main


If I understood correctly, Akumuli does not have the concept of databases and schemas like PostgreSQL or InfluxDB has. And the db creator is not needed to create a database. But the Creator also handles cleaning of old/stale data (check out tsbs_load_timescaledb/creator.go). I would also change it so that the DBExists method always returns true so the RemoveOldDb gets called by tsbs (where you can put the cleanup code). The calling of RemoveOldDb is controlled by the do-create-db flag, which by default is true, but if you want to keep the old data you can set it to false

Yes, it doesn't have a notion of database/table/schema. And it's not possible to clean up using the database API. You have to run command in terminal to crate or delete the database.

Lazin · 2019-10-09T15:13:24Z

@blagojts thanks, I appreciated the feedback!
To run the test you should use recent docker image. 0.7.45 is way too old. The latest tag 0.8.72-skylake should work fine.

RobAtticus · 2019-10-11T19:59:37Z

Hey @Lazin , we're seeing failures in CI:

# github.com/timescale/tsbs/cmd/tsbs_load_akumuli
cmd/tsbs_load_akumuli/main.go:33:34: not enough arguments in call to load.GetBenchmarkRunner
	have ()
	want (load.BenchmarkRunnerConfig)

and

# github.com/timescale/tsbs/cmd/tsbs_run_queries_akumuli
cmd/tsbs_run_queries_akumuli/main.go:26:35: not enough arguments in call to query.NewBenchmarkRunner
	have ()
	want (query.BenchmarkRunnerConfig)

Lazin · 2019-10-21T13:00:11Z

Hi @RobAtticus,
I fixed this errors and ran all benchmark apps to check if anything else was broken. It all worked for me. Also, updated the doc since the command line interface changed slightly.

RobAtticus

Just a few last nits if you want, otherwise I might fix them after merge.

Can you verify which queries this works for? Is this every query in devops except GroupByOrderByLimit?

RobAtticus · 2019-11-25T21:04:58Z

cmd/tsbs_generate_data/serialize/akumuli.go

+	buf := make([]byte, 0, 1024)
+	// Add cue
+	const HeaderLength = 8
+	buf = append(buf, "AAAAFFEE+"...)


can you make this a named constant? I am unclear what this magic value is.

Done. The value is just a placeholder that gets replaced by the binary value later (all that binary.LittleEndian.Putxxx calls).

RobAtticus · 2019-11-25T21:05:08Z

cmd/tsbs_generate_data/serialize/akumuli.go

+			// Shortcut
+			s.index++
+			tmp := make([]byte, 0, 1024)
+			tmp = append(tmp, "AAAAFFEE*2\n"...)


same thing as above

RobAtticus · 2019-11-25T21:06:18Z

cmd/tsbs_generate_data/serialize/akumuli.go

+	// Update cue
+	binary.LittleEndian.PutUint16(buf[4:6], uint16(len(buf)))
+	binary.LittleEndian.PutUint16(buf[6:HeaderLength], uint16(len(p.fieldValues)))
+	if deferPoint {


this is probably more go-like written as:

if deferPoint { s.deferred = append(s.deferred, buf...) return nil } _, err = w.Write(buf) return err

RobAtticus · 2019-11-25T21:07:02Z

cmd/tsbs_generate_data/serialize/akumuli.go

+			binary.LittleEndian.PutUint16(buf[6:HeaderLength], uint16(0))
+			binary.LittleEndian.PutUint32(buf[:4], id)
+		} else {
+			panic("Unexpected series name")


let's just return an error here? A long term goal I have is to reduce the amount of panics in the codebase, and given this func already returns errors, we should probably just do that

handled

RobAtticus · 2019-12-28T00:25:59Z

Sorry it took so long, but merged!

Lazin added 19 commits April 14, 2019 13:13

Add akumuli data generator

6695176

Update akumuli serializer

bae2b53

Add cueues for data loader

35a1be2

Remove SerializerCloser interface

3c0e354

Use encoding/binary package instead of unsafe

e99f2f4

Add tsbs_load_akumuli

d8900b1

Add akumuli_writer component

f5cd2c9

Implement processor

6d7dafb

Add load script

827a211

Fix load generator for akumuli

5b6ff8b

Improve akumuli loader script

d291b76

Fix concurrency issue

b1ec9a4

Simplify loader

b7edd9e

Add Akumuli load generator

ddc0d1c

Add query generator for Akumuli

8819ac8

Use json package to generate queries

f3a1bcd

Add lastpoint query support

c3df4dd

Add double-groupby query generator

4086451

Add query runner for Akumuli

f3ac7e5

Lazin added 6 commits May 18, 2019 03:26

Add serializer unit-test

bdbee20

Add akumuli.md and clean up flags

72407ba

Add more details to docs/akumuli.md

ca91dc6

Update akumuli.md

6bab456

Remove unused parameter

e96d10a

Merge branch 'master' of github.com:Lazin/tsbs

8e5bcfc

Lazin added 2 commits June 18, 2019 12:32

Merge branch 'master' of github.com:timescale/tsbs

f876756

Update akumuli query gen

4561616

Fix format string

b01fbba

Lazin added 2 commits September 9, 2019 17:21

Merge branch 'master' of github.com:timescale/tsbs

7d07611

Use utils.QueryGenerator interface

0a166e8

Lazin added 2 commits September 18, 2019 17:55

Fix akumuli unit-test

bfee78c

Merge branch 'master' of github.com:timescale/tsbs

3b5d592

atanasovskib previously requested changes Sep 19, 2019

View reviewed changes

Fix review issues

c118f47

RobAtticus requested a review from atanasovskib October 11, 2019 19:56

Lazin added 4 commits October 17, 2019 18:23

Update tsbs_load_akumuli

d4c1091

Update tsbs_run_queries_akumuli

b8a9639

Merge branch 'master' of github.com:timescale/tsbs

ec2f47c

Update akumuli.md

d246c46

Merge branch 'master' into master

e4ace97

RobAtticus approved these changes Nov 25, 2019

View reviewed changes

Fix timescale#73 code-review issues

3429e43

RobAtticus merged commit 938f015 into timescale:master Dec 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Akumuli database support #73

Add Akumuli database support #73

Lazin commented May 17, 2019

CLAassistant commented May 17, 2019 •

edited

Loading

codecov-io commented May 17, 2019 •

edited

Loading

RobAtticus commented Jun 17, 2019

Lazin commented Jun 18, 2019

RobAtticus commented Aug 28, 2019

Lazin commented Sep 11, 2019

atanasovskib commented Sep 18, 2019

Lazin commented Sep 18, 2019

atanasovskib left a comment •

edited

Loading

atanasovskib Sep 18, 2019

atanasovskib Sep 18, 2019

Lazin Oct 9, 2019

atanasovskib Sep 19, 2019

Lazin Oct 9, 2019

atanasovskib Sep 19, 2019

Lazin Oct 9, 2019

Lazin commented Oct 9, 2019 •

edited

Loading

RobAtticus commented Oct 11, 2019

Lazin commented Oct 21, 2019

RobAtticus left a comment

RobAtticus Nov 25, 2019

Lazin Nov 26, 2019

RobAtticus Nov 25, 2019

Lazin Nov 26, 2019

RobAtticus Nov 25, 2019

Lazin Nov 26, 2019

RobAtticus Nov 25, 2019

Lazin Nov 26, 2019

RobAtticus commented Dec 28, 2019

Add Akumuli database support #73

Add Akumuli database support #73

Conversation

Lazin commented May 17, 2019

CLAassistant commented May 17, 2019 • edited Loading

codecov-io commented May 17, 2019 • edited Loading

Codecov Report

RobAtticus commented Jun 17, 2019

Lazin commented Jun 18, 2019

RobAtticus commented Aug 28, 2019

Lazin commented Sep 11, 2019

atanasovskib commented Sep 18, 2019

Lazin commented Sep 18, 2019

atanasovskib left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Lazin commented Oct 9, 2019 • edited Loading

RobAtticus commented Oct 11, 2019

Lazin commented Oct 21, 2019

RobAtticus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RobAtticus commented Dec 28, 2019

CLAassistant commented May 17, 2019 •

edited

Loading

codecov-io commented May 17, 2019 •

edited

Loading

atanasovskib left a comment •

edited

Loading

Lazin commented Oct 9, 2019 •

edited

Loading