Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LogQL: Labels and Metrics Extraction #2769

Merged
merged 49 commits into from
Oct 20, 2020
Merged

LogQL: Labels and Metrics Extraction #2769

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
03279b1
Adds logfmt, regexp and json logql parser
cyriltovena Sep 15, 2020
31c2f3f
hook the ast with parsers.
cyriltovena Sep 16, 2020
eaf72bd
hook parser with memchunk.
cyriltovena Sep 17, 2020
7d1dc7b
hook parser with the storage.
cyriltovena Sep 17, 2020
484afc1
hook parser with ingesters
cyriltovena Sep 17, 2020
0121a3c
fixes all tests
cyriltovena Sep 17, 2020
62f2829
Refactor to pipeline and implement ast parsing.
cyriltovena Sep 21, 2020
89c489c
Fixes the lexer for duration and range
cyriltovena Sep 22, 2020
4238173
Fixes all tests and add some for label filters
cyriltovena Sep 23, 2020
88ad104
Add label and line format.
cyriltovena Sep 23, 2020
c7791a4
Add tests for fmt label and line with validations.
cyriltovena Sep 24, 2020
0651e25
Polishing parsers and add some more test cases
cyriltovena Sep 25, 2020
4c0570d
Finish the unwrap parser, still need to add more tests
cyriltovena Sep 29, 2020
01e93c0
Indent this hell.
cyriltovena Sep 29, 2020
e455c88
Moar tests and it works.
cyriltovena Sep 29, 2020
8bc18e5
Add more tests which lead me to find a bug in the lexer
cyriltovena Sep 30, 2020
08d2cf7
Add more tests and fix all engine tests
cyriltovena Sep 30, 2020
b801417
Fixes match stage in promtail pipelines.
cyriltovena Sep 30, 2020
850b003
Hook Pipeline into ingester, tailer and storage.
cyriltovena Oct 1, 2020
31c26c0
Correctly setup sharding for logqlv2
cyriltovena Oct 1, 2020
b5e11d0
Fixes precedences issue with label filters and add moar tests :v:
cyriltovena Oct 2, 2020
0fd6018
Adds quantile_over_time, grouping for non associate range aggregation…
cyriltovena Oct 2, 2020
2ca6677
Extract with grouping
cyriltovena Oct 3, 2020
4effb67
Adds parsing duration on unwrap
cyriltovena Oct 5, 2020
832a977
Improve the lexer to support more common identifier as functions.
cyriltovena Oct 6, 2020
6563d6e
Fixes the frontend logs to include org_id.
cyriltovena Oct 6, 2020
92f7c39
Merge branch 'fix-orgid-frontend' into logql-parser
cyriltovena Oct 6, 2020
5578dbb
Support byte sizes in label filters.
jeschkies Oct 9, 2020
13132ad
Wip on error handling.
cyriltovena Oct 12, 2020
db07446
Fixes json parser with prometheus label name rules.
cyriltovena Oct 12, 2020
78973cf
fixup! Support byte sizes in label filters.
jeschkies Oct 12, 2020
bbacba7
Merge remote-tracking branch 'cyril/logql-parser' into karsten/bytes-…
jeschkies Oct 12, 2020
25dd730
Merge pull request #5 from jeschkies/karsten/bytes-filter
cyriltovena Oct 12, 2020
c054a5d
Wip error handling, commit before big refactoring.
cyriltovena Oct 13, 2020
e7d8234
Merge branch 'logql-parser' of github.com:cyriltovena/loki into logql…
cyriltovena Oct 13, 2020
5ab8b5c
Refactoring in progress.
cyriltovena Oct 13, 2020
5272d91
Work in progress.
cyriltovena Oct 13, 2020
1aa1609
Got something that builds and throw __error__ labels properly now.
cyriltovena Oct 14, 2020
1af9c14
Add error handling + fixes groupins and post filtering.
cyriltovena Oct 14, 2020
960ef5a
400 on pipeline errors.
cyriltovena Oct 14, 2020
de83465
Fixes a races in the log pipeline.
cyriltovena Oct 15, 2020
87c4f00
Unsure the key is parsable and valid.
cyriltovena Oct 15, 2020
50315b1
Cleanup and code documentation.
cyriltovena Oct 15, 2020
54dd6c2
Lint.
cyriltovena Oct 15, 2020
a50490f
Lint.
cyriltovena Oct 15, 2020
cbbc37c
Merge remote-tracking branch 'upstream/master' into logql-parser
cyriltovena Oct 15, 2020
801b721
Fixes frontend handler.
cyriltovena Oct 15, 2020
1aee415
Fixes old test.
cyriltovena Oct 15, 2020
1ea917f
Fix go1.15 local failing test.
cyriltovena Oct 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions pkg/chunkenc/dumb_chunk.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import (
"sort"
"time"

"github.com/prometheus/prometheus/pkg/labels"

"github.com/grafana/loki/pkg/iter"
"github.com/grafana/loki/pkg/logproto"
"github.com/grafana/loki/pkg/logql"
Expand Down Expand Up @@ -68,7 +70,7 @@ func (c *dumbChunk) Utilization() float64 {

// Returns an iterator that goes from _most_ recent to _least_ recent (ie,
// backwards).
func (c *dumbChunk) Iterator(_ context.Context, from, through time.Time, direction logproto.Direction, _ logql.LineFilter) (iter.EntryIterator, error) {
func (c *dumbChunk) Iterator(_ context.Context, from, through time.Time, direction logproto.Direction, _ labels.Labels, _ logql.Pipeline) (iter.EntryIterator, error) {
i := sort.Search(len(c.entries), func(i int) bool {
return !from.After(c.entries[i].Timestamp)
})
Expand All @@ -93,7 +95,7 @@ func (c *dumbChunk) Iterator(_ context.Context, from, through time.Time, directi
}, nil
}

func (c *dumbChunk) SampleIterator(_ context.Context, from, through time.Time, _ logql.LineFilter, _ logql.SampleExtractor) iter.SampleIterator {
func (c *dumbChunk) SampleIterator(_ context.Context, from, through time.Time, _ labels.Labels, _ logql.SampleExtractor) iter.SampleIterator {
return nil
}

Expand Down Expand Up @@ -137,7 +139,7 @@ func (i *dumbChunkIterator) Entry() logproto.Entry {
}

func (i *dumbChunkIterator) Labels() string {
panic("Labels() called on chunk iterator")
return ""
}

func (i *dumbChunkIterator) Error() error {
Expand Down
10 changes: 6 additions & 4 deletions pkg/chunkenc/interface.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import (
"strings"
"time"

"github.com/prometheus/prometheus/pkg/labels"

"github.com/grafana/loki/pkg/iter"
"github.com/grafana/loki/pkg/logproto"
"github.com/grafana/loki/pkg/logql"
Expand Down Expand Up @@ -98,8 +100,8 @@ type Chunk interface {
Bounds() (time.Time, time.Time)
SpaceFor(*logproto.Entry) bool
Append(*logproto.Entry) error
Iterator(ctx context.Context, from, through time.Time, direction logproto.Direction, filter logql.LineFilter) (iter.EntryIterator, error)
SampleIterator(ctx context.Context, from, through time.Time, filter logql.LineFilter, extractor logql.SampleExtractor) iter.SampleIterator
Iterator(ctx context.Context, mintT, maxtT time.Time, direction logproto.Direction, lbs labels.Labels, pipeline logql.Pipeline) (iter.EntryIterator, error)
SampleIterator(ctx context.Context, from, through time.Time, lbs labels.Labels, extractor logql.SampleExtractor) iter.SampleIterator
// Returns the list of blocks in the chunks.
Blocks(mintT, maxtT time.Time) []Block
Size() int
Expand All @@ -122,7 +124,7 @@ type Block interface {
// Entries is the amount of entries in the block.
Entries() int
// Iterator returns an entry iterator for the block.
Iterator(context.Context, logql.LineFilter) iter.EntryIterator
Iterator(ctx context.Context, lbs labels.Labels, pipeline logql.Pipeline) iter.EntryIterator
// SampleIterator returns a sample iterator for the block.
SampleIterator(context.Context, logql.LineFilter, logql.SampleExtractor) iter.SampleIterator
SampleIterator(ctx context.Context, lbs labels.Labels, extractor logql.SampleExtractor) iter.SampleIterator
}
Loading