Introduce sequence-number-aware translog #22822

jasontedor · 2017-01-26T19:13:15Z

Today, the relationship between Lucene and the translog is rather simple: every document not in Lucene is guaranteed to be in the translog. We need a stronger guarantee from the translog though, namely that it can replay all operations after a certain sequence number. For this to be possible, the translog has to made sequence-number aware. As a first step, we introduce the min and max sequence numbers into the translog so that each generation knows the possible range of operations contained in the generation. This will enable future work to keep around all generations containing operations after a certain sequence number (e.g., the global checkpoint).

Relates #10708

nik9000 · 2017-01-26T19:21:20Z

core/src/main/java/org/elasticsearch/index/translog/BaseTranslogReader.java

-    public abstract  int totalOperations();
+    public abstract int totalOperations();
+
+    abstract long getMinSeqNo();


I think it'd be nice to have javadocs for these.

I pushed abd491cbf3a62f74d2c391876aee7b3e57166b0c.

nik9000 · 2017-01-26T19:25:20Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

        this.offset = offset;
        this.numOps = numOps;
        this.generation = generation;
+        this.minSeqNo = minSeqNo;
+        this.maxSeqNo = maxSeqNo;


Is it worth checking that minSeqNo <= maxSeqNo?

Or maybe something like minSeqNo <= maxSeqNo || minSeqNo == Long.MAX_VALUE && maxSeqNo == Long.MIN_VALUE if we want to keep the merging easier.

+1 for assertions.

I pushed e9370c84925e00d11d74e6785004c9de7f9feefc.

nik9000 · 2017-01-26T19:26:34Z

core/src/main/java/org/elasticsearch/index/translog/Translog.java

@@ -198,7 +198,7 @@ public Translog(
                logger.debug("wipe translog location - creating new translog");
                Files.createDirectories(location);
                final long generation = 1;
-                Checkpoint checkpoint = new Checkpoint(0, 0, generation, globalCheckpointSupplier.getAsLong());
+                Checkpoint checkpoint = new Checkpoint(0, 0, generation, Long.MAX_VALUE, Long.MIN_VALUE, globalCheckpointSupplier.getAsLong());


Negative values for translog seem to have a special meaning. Maybe this should be 0, 0?

Or one of the constants in SequenceNumberService?

+1 to constants. I'll comment about it at the end.

nik9000 · 2017-01-26T19:29:33Z

core/src/main/java/org/elasticsearch/index/translog/Translog.java

@@ -419,22 +419,21 @@ public Location add(Operation operation) throws IOException {
            out.writeInt(operationSize);
            out.seek(end);
            final ReleasablePagedBytesReference bytes = out.bytes();
-            try (ReleasableLock lock = readLock.acquire()) {
+            try (final ReleasableLock ignored = readLock.acquire()) {


These are implicitly final already. I do like the name change though.

I opened #22960 so this travesty does not happen again.

I wouldn't go so far as to say it is a travesty, but, thanks!

I pushed 4311c1fc4bfeb9830e688d47ac3729b9d7acf8e3.

nik9000 · 2017-01-26T19:31:42Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

    protected final AtomicBoolean closed = new AtomicBoolean(false);

    /**
     * Create a reader of translog file channel. The length parameter should be consistent with totalOperations and point
     * at the end of the last operation in this snapshot.
     */
-    public TranslogReader(long generation, FileChannel channel, Path path, long firstOperationOffset, long length, int totalOperations) {
+    public TranslogReader(
+        final long generation,


Can you indent these so they don't look like method body?

++ I hate this - sorry

I pushed 8786360f5f4f40fa31ab273464128abb34c440a4 and 48403f4baa62d1bd6223ea26a547ccb6b63ef6eb.

bleskes · 2017-01-26T22:26:21Z

cool.. I quickly read through it and the basics look good. I will give it a careful look tomorrow morning (I want to look critically at the initial min/max values and also w.r.t the BWC situation where incoming seqnos can be -1)

s1monw

left some comments

s1monw · 2017-01-27T13:41:03Z

core/src/main/java/org/elasticsearch/index/translog/BaseTranslogReader.java

-    public abstract  int totalOperations();
+    public abstract int totalOperations();
+
+    abstract long getMinSeqNo();


s1monw · 2017-01-27T13:41:14Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

@@ -35,11 +36,13 @@
 import java.nio.file.OpenOption;
 import java.nio.file.Path;

-class Checkpoint {
+final class Checkpoint {


s1monw · 2017-01-27T13:42:04Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

    }

    // reads a checksummed checkpoint introduced in ES 5.0.0
    static Checkpoint readChecksummedV1(DataInput in) throws IOException {
-        return new Checkpoint(in.readLong(), in.readInt(), in.readLong(), SequenceNumbersService.UNASSIGNED_SEQ_NO);
+        return new Checkpoint(


can we wrap this in 2 lines instead of N this makes my eyes bleed

That sounds horrible, you should really get that checked out. 😛

I pushed a commit that should address your concerns.

s1monw · 2017-01-27T13:48:53Z

core/src/main/java/org/elasticsearch/index/translog/Translog.java

@@ -778,6 +777,8 @@ public static Type fromId(byte id) {

        Source getSource();

+        long seqNo();


can this be getSeqNo?

s1monw · 2017-01-27T13:49:12Z

core/src/main/java/org/elasticsearch/index/translog/Translog.java

@@ -1147,6 +1150,7 @@ public String toString() {
        private final long primaryTerm;
        private final String reason;

+        @Override
        public long seqNo() {


maybe getSeqNo?

s1monw · 2017-01-27T13:49:44Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

    protected final AtomicBoolean closed = new AtomicBoolean(false);

    /**
     * Create a reader of translog file channel. The length parameter should be consistent with totalOperations and point
     * at the end of the last operation in this snapshot.
     */
-    public TranslogReader(long generation, FileChannel channel, Path path, long firstOperationOffset, long length, int totalOperations) {
+    public TranslogReader(
+        final long generation,


++ I hate this - sorry

s1monw · 2017-01-27T13:49:57Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

@@ -116,7 +143,15 @@ public static TranslogReader open(FileChannel channel, Path path, Checkpoint che
                            throw new TranslogCorruptedException("expected shard UUID " + uuidBytes + " but got: " + ref +
                                            " this translog file belongs to a different translog. path:" + path);
                        }
-                        return new TranslogReader(checkpoint.generation, channel, path, ref.length + CodecUtil.headerLength(TranslogWriter.TRANSLOG_CODEC) + Integer.BYTES, checkpoint.offset, checkpoint.numOps);
+                        return new TranslogReader(
+                            checkpoint.generation,


please stop doing this.

maybe make the constructor accept a checkpoint and that will make many of these parameters go away..

s1monw · 2017-01-27T13:54:13Z

core/src/test/java/org/elasticsearch/index/translog/TranslogVersionTests.java

+                channel,
+                path,
+                new Checkpoint(
+                    Files.size(path),


This is better now.

s1monw · 2017-01-27T13:58:31Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

                ex.addSuppressed(inner);
            }
            throw ex;
        }
        totalOffset += data.length();
        operationCounter++;
+        minSeqNo = Math.min(minSeqNo, seqNo);


can we add a comment that the order we assign these values matters to ensure we always have a >= relationship?

we need to take BWC aspect here into account. The incoming minSeqNo can be unassigned (if the ops comes from an old primary) but as soon as we start receiving valid seqNo we should never go back. At least, I hope that's where end up being and I want to start building the assertions based on that.

s1monw · 2017-01-27T14:23:28Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -191,6 +205,14 @@ public int totalOperations() {
        return operationCounter;
    }

+    public long getMinSeqNo() {


I wonder if it's possible to read these out of order and therefore get inconsistent values? like if you read max first and then min? Also I wonder if the default value can be confusing?

to be really consistent I think we need to return a tuple and read it under the lock?

I think we can wait and see the usage (but document the caveat - other stats are not consistent either). At the moment it's only used for snapshots which is created under lock. I think that the future for recovery/translog trimming will not care about the the writer's values. An alternative is to not expose these but rather the last checkpoint, which is consistent.

Yeah, I'd prefer not to expose these unless we're sure we need them because of the concurrency issues. Or just expose them for testing if we need them there.

@jasontedor it seems the consensus leans towards exposing/use the last checkpoint. I presume you had a reason to leave as is. Can you elaborate?

I pushed bdbbe83399a22d2df91aa5ed684d86d3579dc023.

bleskes

I did a more detailed pass and left a bunch of comments.

On top of it, I think we should extend the translog stats with the min max (which might need the lock/tuple that Simon discussed)

About those constants. Without BWC I would argue we should initialize the min/max with the no ops performed constants. The tricky part that with BWC the translog can up full with ops and still the min/max would be no_ops_performed which is strange and will lead to bugs imo. As such, I would recommend using unassigned until the first operation with a seq# enters the system, at which point we switch to normal min/max.

bleskes · 2017-01-27T15:34:14Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

        this.offset = offset;
        this.numOps = numOps;
        this.generation = generation;
+        this.minSeqNo = minSeqNo;
+        this.maxSeqNo = maxSeqNo;


+1 for assertions.

bleskes · 2017-01-27T15:35:25Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

+            in.readLong(),
+            in.readInt(),
+            in.readLong(),
+            SequenceNumbersService.NO_OPS_PERFORMED,


I think we should use unassigned here too.

This is addressed.

bleskes · 2017-01-27T15:35:51Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

    }

    // reads checkpoint from ES < 5.0.0
    static Checkpoint readNonChecksummed(DataInput in) throws IOException {
-        return new Checkpoint(in.readLong(), in.readInt(), in.readLong(), SequenceNumbersService.UNASSIGNED_SEQ_NO);
+        return new Checkpoint(


we can trash this in master, right?

I pushed 8e9091bf41d2d28cd46485e760c235b895e8a65f.

bleskes · 2017-01-27T15:39:19Z

core/src/main/java/org/elasticsearch/index/translog/Translog.java

@@ -198,7 +198,7 @@ public Translog(
                logger.debug("wipe translog location - creating new translog");
                Files.createDirectories(location);
                final long generation = 1;
-                Checkpoint checkpoint = new Checkpoint(0, 0, generation, globalCheckpointSupplier.getAsLong());
+                Checkpoint checkpoint = new Checkpoint(0, 0, generation, Long.MAX_VALUE, Long.MIN_VALUE, globalCheckpointSupplier.getAsLong());


+1 to constants. I'll comment about it at the end.

bleskes · 2017-01-27T15:41:27Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

@@ -116,7 +143,15 @@ public static TranslogReader open(FileChannel channel, Path path, Checkpoint che
                            throw new TranslogCorruptedException("expected shard UUID " + uuidBytes + " but got: " + ref +
                                            " this translog file belongs to a different translog. path:" + path);
                        }
-                        return new TranslogReader(checkpoint.generation, channel, path, ref.length + CodecUtil.headerLength(TranslogWriter.TRANSLOG_CODEC) + Integer.BYTES, checkpoint.offset, checkpoint.numOps);
+                        return new TranslogReader(
+                            checkpoint.generation,


maybe make the constructor accept a checkpoint and that will make many of these parameters go away..

bleskes · 2017-01-27T16:03:17Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -191,6 +205,14 @@ public int totalOperations() {
        return operationCounter;
    }

+    public long getMinSeqNo() {


I think we can wait and see the usage (but document the caveat - other stats are not consistent either). At the moment it's only used for snapshots which is created under lock. I think that the future for recovery/translog trimming will not care about the the writer's values. An alternative is to not expose these but rather the last checkpoint, which is consistent.

bleskes · 2017-01-27T16:04:38Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -295,7 +331,7 @@ public boolean syncUpTo(long offset) throws IOException {
                    try {
                        channel.force(false);
                        checkpoint =
-                            writeCheckpoint(channelFactory, offsetToSync, opsCounter, globalCheckpoint, path.getParent(), generation);
+                            writeCheckpoint(channelFactory, offsetToSync, opsCounter, minSeqNo, maxSeqNo, globalCheckpoint, path.getParent(), generation);


we should capture these under lock.

I pushed 7802d6ec2d2f3fdb2dbb1c79c60b3fe6c605291d.

This is done in 7802d6ec2d2f3fdb2dbb1c79c60b3fe6c605291d.

bleskes · 2017-01-27T16:06:23Z

core/src/main/java/org/elasticsearch/index/translog/TruncateTranslogCommand.java

@@ -168,7 +168,8 @@ protected void execute(Terminal terminal, OptionSet options, Environment env) th

    /** Write a checkpoint file to the given location with the given generation */
    public static void writeEmptyCheckpoint(Path filename, int translogLength, long translogGeneration) throws IOException {
-        Checkpoint emptyCheckpoint = new Checkpoint(translogLength, 0, translogGeneration, SequenceNumbersService.UNASSIGNED_SEQ_NO);
+        Checkpoint emptyCheckpoint =
+            new Checkpoint(translogLength, 0, translogGeneration, Long.MAX_VALUE, Long.MIN_VALUE, SequenceNumbersService.UNASSIGNED_SEQ_NO);


I think we should use a static method on the translog class to make sure this is consistent with what the translog does when it creates an empty translog.

I pushed e6b356d885d22524bc899987525d6e655fffd31a although I left unsettled the issue of the default values for the min and max sequence number (I will address these later).

bleskes · 2017-01-27T16:08:38Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

@@ -1208,12 +1220,12 @@ public void testRecoveryUncommittedCorruptedCheckpoint() throws IOException {
        TranslogConfig config = translog.getConfig();
        Path ckp = config.getTranslogPath().resolve(Translog.CHECKPOINT_FILE_NAME);
        Checkpoint read = Checkpoint.read(ckp);
-        Checkpoint corrupted = new Checkpoint(0, 0, 0, SequenceNumbersService.UNASSIGNED_SEQ_NO);
+        Checkpoint corrupted = new Checkpoint(0, 0, 0, SequenceNumbersService.NO_OPS_PERFORMED, SequenceNumbersService.NO_OPS_PERFORMED, SequenceNumbersService.UNASSIGNED_SEQ_NO);


this another example we need the "empty" checkpoint utility - this is different than what the translog does.

I pushed e6b356d885d22524bc899987525d6e655fffd31a although I left unsettled the issue of the default values for the min and max sequence number (I will address these later).

bleskes · 2017-01-27T16:09:22Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

        for (int i = 0; i < numOps; i++) {
            out.reset(bytes);
            out.writeInt(i);
-            writer.add(new BytesArray(bytes));
+            long seqNo;
+            do {


once we add it, can we test that we do the right things with unassigned seq nos due to BWC?

I pushed 390ee829db21bae606446a5393c35af055a47c02.

jasontedor · 2017-02-07T21:54:44Z

@nik9000 @bleskes @s1monw I think that this is ready for another round of reviews.

nik9000 · 2017-02-08T15:45:33Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

-    // reads checkpoint from ES < 5.0.0
-    static Checkpoint readNonChecksummed(DataInput in) throws IOException {
-        return new Checkpoint(in.readLong(), in.readInt(), in.readLong(), SequenceNumbersService.UNASSIGNED_SEQ_NO);
+        final long minSeqNo = Translog.INITIAL_MIN_SEQ_NO;


I don't think this should have the same number as an empty checkpoint. That way we can tell the difference between a checkpoint that didn't have any sequence numbers and one that was empty. I guess we could just look at the size, but still, I think it'd be more clear to reserve some int for this.

nik9000 · 2017-02-08T15:47:14Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

        out.writeLong(globalCheckpoint);
    }

+    static Checkpoint emptyTranslogCheckpoint(final long offset, final long generation, final long globalCheckpoint) {
+        return new Checkpoint(offset, 0, generation, Translog.INITIAL_MIN_SEQ_NO, Translog.INITIAL_MAX_SEQ_NO, globalCheckpoint);


Hmmm - now that you have it this way and those are the same numbers I think it'd be more clear to use UNASSIGNED_SEQ_NO. I think it is confusing to have three names for -2 because they'll all be unexpectedly equal, you know?

nik9000 · 2017-02-08T15:57:21Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

                ex.addSuppressed(inner);
            }
            throw ex;
        }
        totalOffset += data.length();
        operationCounter++;
+
+        if (minSeqNo == Translog.INITIAL_MIN_SEQ_NO) {


I'd be happy with Long.MAX_VALUE here so long as we're careful not to let them escape from this class.

can we assert operation counter is 0? (and move the counter increment?)

I pushed e8f11578cade5be8fdbb9c96d3478e07c0532ebf.

nik9000 · 2017-02-08T15:57:26Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

+            minSeqNo = Math.min(minSeqNo, seqNo);
+        }
+
+        if (maxSeqNo == Translog.INITIAL_MAX_SEQ_NO) {


I pushed e8f11578cade5be8fdbb9c96d3478e07c0532ebf.

nik9000 · 2017-02-08T15:59:16Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -191,6 +205,14 @@ public int totalOperations() {
        return operationCounter;
    }

+    public long getMinSeqNo() {


Yeah, I'd prefer not to expose these unless we're sure we need them because of the concurrency issues. Or just expose them for testing if we need them there.

bleskes

Thx @jasontedor . I went through it again and I think we're close. Like Nik, I have concerns around have the new constants INITIAL_MIN_SEQ_NO. If we go the extra mile (which I think is good), I would go with setting the initial values to the intuitive NO_OPS_PERFORMED. Any incoming unassigned seq# should set it to UNASSINGED_SEQ_NO and I think that from that point on all the logic works.

Also - did you see my ask for extending the translog stats?

bleskes · 2017-02-12T07:50:50Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

-            final TranslogWriter writer =
-                new TranslogWriter(channelFactory, shardId, checkpoint, channel, file, bufferSize, globalCheckpointSupplier);
-            return writer;
+                writeCheckpoint(channelFactory, headerLength, 0, Translog.INITIAL_MIN_SEQ_NO, Translog.INITIAL_MAX_SEQ_NO, globalCheckpointSupplier.getAsLong(), file.getParent(), fileGeneration);


shouldn't we sue the createEmptyCheckpoint method?

I pushed fc40b8f8e77b8cab1dd0544334e9d01f57b5f837.

bleskes · 2017-02-12T07:52:27Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

                ex.addSuppressed(inner);
            }
            throw ex;
        }
        totalOffset += data.length();
        operationCounter++;
+
+        if (minSeqNo == Translog.INITIAL_MIN_SEQ_NO) {


can we assert operation counter is 0? (and move the counter increment?)

bleskes · 2017-02-12T07:54:37Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

+                minSeqNo = seqNo;
+            }
+        } else {
+            assert seqNo != SequenceNumbersService.UNASSIGNED_SEQ_NO;


I think this is confusing (I'm leaving my original comment below, to show how I got confused). IMO we should always set the minSeqNo / maxSeqNo upon the first value we see. If it is UNASSIGNED_SEQ_NO, then so be it. It tells us that the translog contains operations without seq#, which is currently impossible to know.

hmm... is this correct? what want to check here is that if the incoming value is UNASSIGNED_SEQ_NO then the current value of minSeqNo must be UNASSIGNED_SEQ_NO . This is to say - once seq# are added to the mix, we never go back (we may need to change some logic somewhere else to do so, but I think it's the right move).

I pushed e8f11578cade5be8fdbb9c96d3478e07c0532ebf.

bleskes · 2017-02-12T07:55:42Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

+                maxSeqNo = seqNo;
+            }
+        } else {
+            assert seqNo != SequenceNumbersService.UNASSIGNED_SEQ_NO;


same comments rely to here.

I pushed e8f11578cade5be8fdbb9c96d3478e07c0532ebf.

bleskes · 2017-02-12T07:57:12Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -191,6 +205,14 @@ public int totalOperations() {
        return operationCounter;
    }

+    public long getMinSeqNo() {


@jasontedor it seems the consensus leans towards exposing/use the last checkpoint. I presume you had a reason to leave as is. Can you elaborate?

bleskes · 2017-02-12T08:00:52Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

@@ -1008,42 +1032,6 @@ public void testTranslogWriter() throws IOException {
        IOUtils.close(writer);
    }

-    public void testFailWriterWhileClosing() throws IOException {


good that we have one thing less to worry about, but I wonder if it's a good idea to remove the test. I mean these transitions are always tricky and it's good to have it well tested?

This test was testing a specific scenario: constructing the new translog reader could fail because we did a file operation (reading the channel position) in that construction. That file operation was removed, so that failure scenario can not occur anymore. Are you saying that you want the test to remain, just without testing for the failure scenario?

yeah, I get why you removed it but It think it's indeed good to keep around - it seems we don't have any other test for the closeIntoReader method.

I pushed introduce-sequence-number-aware-translog.

bleskes · 2017-02-12T08:02:09Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

+        long minSeqNo = Translog.INITIAL_MIN_SEQ_NO;
+        long maxSeqNo = Translog.INITIAL_MAX_SEQ_NO;
+        final Set<Long> seenSeqNos = new HashSet<>();
+        boolean opsHaveValidSequenceNumbers = false;


should we randomize the start to simulate "no BWC mode"?

I pushed e8f11578cade5be8fdbb9c96d3478e07c0532ebf.

bleskes · 2017-02-12T08:03:30Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

+            } while (seenSeqNos.contains(seqNo));
+            if (seqNo != SequenceNumbersService.UNASSIGNED_SEQ_NO) {
+                seenSeqNos.add(seqNo);
+                if (minSeqNo == Translog.INITIAL_MIN_SEQ_NO) {


wouldn't it be simpler to calculate those in the end based on seenSeqNos? we can also wrap this whole thing in a opsHaveValidSequenceNumbers chcek and then the inner loop won't need to change for UNASSIGNED_SEQ_NO

I pushed e8f11578cade5be8fdbb9c96d3478e07c0532ebf.

jasontedor · 2017-02-16T22:56:18Z

Also - did you see my ask for extending the translog stats?

Certainly this makes sense at the shard level, but it does not make sense when we report node level or index level stats. Thus, I think they should be left out?

jasontedor · 2017-02-16T22:57:19Z

Thanks @nik9000 and @bleskes; I think I've addressed your feedback.

bleskes · 2017-02-17T07:15:28Z

test this please

bleskes

Thx @jasontedor for the hard work. This LGTM. I left some minor comments but there is no need for an extra review on my end. @s1monw do you want to do another pass?

Re the stats - I see what you're saying. I'm not happy with it as I think this will give us a way to debug this and it will be good to expose, but I agree there is no clear cut easy solution that's worth pushing for in this PR. Potentially we should just remove translog stats from common stats, I'm not sure how much value the aggregation per index/node/cluster adds here.

bleskes · 2017-02-17T07:20:59Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

-        final long maxSeqNo = Translog.INITIAL_MAX_SEQ_NO;
-        return new Checkpoint(in.readLong(), in.readInt(), in.readLong(), minSeqNo, maxSeqNo, SequenceNumbersService.UNASSIGNED_SEQ_NO);
+    static Checkpoint readChecksummedV1(final DataInput in) throws IOException {
+        final long minSeqNo = SequenceNumbersService.NO_OPS_PERFORMED;


these should be UNASSIGNED_SEQ_NO no? (unless maybe if total count is 0, but I'm not sure it's worth the code song and dance it will require)

bleskes · 2017-02-17T07:22:02Z

core/src/test/java/org/elasticsearch/index/translog/TranslogTests.java

+                writer.add(new BytesArray(bytes), randomNonNegativeLong());
+            }
+            writer.sync();
+            try (TranslogReader reader = writer.closeIntoReader()) {


I think we need to check the transfer of the checkpoint as well, no?

I pushed 48f02c2bedbc08e10099605030b92b8ca1bd34cd.

bleskes · 2017-02-17T07:28:47Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

    protected final long length;
+    private final int totalOperations;


I'm not sure it's worth having these extra fields now that we have a checkpoint? code that needs it can either call the getter methods or access the checkpoint directly?

Yeah, I thought about removing them and decided the code read easier with them than with checkpoint.numOps and checkpoint.offset everywhere so reverted it. I can remove them if you feel strongly.

I'd like to remove the fields but I'm fine with doing it in a dead simple followup. I like not doing it in this PR to keep it smaller.

Okay, I'm fine with it being a follow-up.

bleskes · 2017-02-17T07:38:00Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

-        return new Checkpoint(in.readLong(), in.readInt(), in.readLong(), SequenceNumbersService.UNASSIGNED_SEQ_NO);
+    // reads a checksummed checkpoint introduced in ES 5.0.0
+    static Checkpoint readChecksummedV1(final DataInput in) throws IOException {
+        final long minSeqNo = SequenceNumbersService.NO_OPS_PERFORMED;


copying the commit comment as the code moved around:

these should be UNASSIGNED_SEQ_NO no? (unless maybe if total count is 0, but I'm not sure it's worth the code song and dance it will require)

nik9000 · 2017-02-17T18:37:15Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

+        return getLastSyncedCheckpoint();
+    }
+
+    @Override
    public long sizeInBytes() {
        return totalOffset;
    }

    /**
     * closes this writer and transfers it's underlying file channel to a new immutable reader


I pushed ca0a0adf1820c68f096d9167b9f6f1efdac2c51d.

nik9000 · 2017-02-17T18:46:06Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -80,6 +85,8 @@ public TranslogWriter(
        this.outputStream = new BufferedChannelOutputStream(java.nio.channels.Channels.newOutputStream(channel), bufferSize.bytesAsInt());
        this.lastSyncedCheckpoint = initialCheckpoint;
        this.totalOffset = initialCheckpoint.offset;
+        this.minSeqNo = initialCheckpoint.minSeqNo;


I think this is right but it is confusing me some. Maybe you can explain it or maybe add a comment?

So my problem is that on figure reading it looks like you are keeping the minimum sequence number as made by some previous writer. So if you make more than one of these writers then this minimum value will stay the minimum value from the first writer. Would it make more sense to use the NO_OPS_PERFORMED constant here? Or maybe add a comment about what it when why we want this "dragging backwards" behavior sometimes.

This is only ever called with a new empty checkpoint. We can assert, would that make it better?

Yes please. That way I don't start thinking about crazy stuff that we're not doing.

I pushed 52f68e1979b80f9ae4a3c7e462546bd779110963.

jasontedor · 2017-02-17T23:25:53Z

retest this please

bleskes · 2017-02-20T09:30:20Z

test this please

s1monw

I left some comments nothing major

s1monw · 2017-02-20T14:33:43Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

-    // reads a checksummed checkpoint introduced in ES 5.0.0
-    static Checkpoint readChecksummedV1(DataInput in) throws IOException {
-        return new Checkpoint(in.readLong(), in.readInt(), in.readLong(), SequenceNumbersService.UNASSIGNED_SEQ_NO);
+    static Checkpoint readChecksummedV2(final DataInput in) throws IOException {


can we add a comment when this version was introduced. Maybe instead of V1 and V2 we use the version in the method name like Pre6_0_0?

I pushed a commit that does this.

s1monw · 2017-02-20T14:34:30Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

+                CodecUtil.checksumEntireFile(indexInput);
+                final int fileVersion = CodecUtil.checkHeader(indexInput, CHECKPOINT_CODEC, INITIAL_VERSION, CURRENT_VERSION);
+                if (fileVersion == INITIAL_VERSION) {
+                    assert indexInput.length() == V1_FILE_SIZE;


can we have the actual length in the message? this would be aweful if we didn't have it. same below

I pushed a commit that does this.

s1monw · 2017-02-20T14:34:45Z

core/src/main/java/org/elasticsearch/index/translog/Checkpoint.java

-                        assert indexInput.length() == FILE_SIZE;
-                        return Checkpoint.readChecksummedV2(indexInput);
-                    }
+                    assert fileVersion == CURRENT_VERSION;


can we get the version in the message

s1monw · 2017-02-20T14:35:14Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

     */
-    public static TranslogReader open(FileChannel channel, Path path, Checkpoint checkpoint, String translogUUID) throws IOException {
+    public static TranslogReader open(
+        final FileChannel channel,


ugh can we have one line?

I pushed a commit just for you and only for you that does this.

s1monw · 2017-02-20T14:35:39Z

core/src/main/java/org/elasticsearch/index/translog/TranslogReader.java

@@ -138,6 +158,11 @@ public int totalOperations() {
        return totalOperations;
    }

+    @Override
+    Checkpoint getCheckpoint() {


can this method be final?

Yes, I pushed a commit that does this.

s1monw · 2017-02-20T14:35:53Z

core/src/main/java/org/elasticsearch/index/translog/TranslogSnapshot.java

@@ -56,6 +55,11 @@ public int totalOperations() {
    }

    @Override
+    Checkpoint getCheckpoint() {


can this method be final?

A final modifier here would be redundant because the class is final.

s1monw · 2017-02-20T14:38:52Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

@@ -80,6 +85,10 @@ public TranslogWriter(
        this.outputStream = new BufferedChannelOutputStream(java.nio.channels.Channels.newOutputStream(channel), bufferSize.bytesAsInt());
        this.lastSyncedCheckpoint = initialCheckpoint;
        this.totalOffset = initialCheckpoint.offset;
+        assert initialCheckpoint.minSeqNo == SequenceNumbersService.NO_OPS_PERFORMED;


can we get the actual value in the message?

I agree. I pushed a commit that does this.

s1monw · 2017-02-20T14:40:06Z

core/src/main/java/org/elasticsearch/index/translog/TranslogWriter.java

                ex.addSuppressed(inner);
            }
            throw ex;
        }
        totalOffset += data.length();
+
+        if (minSeqNo == SequenceNumbersService.NO_OPS_PERFORMED) {


any chance we can get this logic as SequenceNumbersService#max and SequenceNumbersService#minwhere we can document and test it separately?

I agree, I pushed a commit that does this (and added tests).

Today, the relationship between Lucene and the translog is rather simple: every document not in Lucene is guaranteed to be in the translog. We need a stronger guarantee from the translog though, namely that it can replay all operations after a certain sequence number. For this to be possible, the translog has to made sequence-number aware. As a first step, we introduce the min and max sequence numbers into the translog so that each generation knows the possible range of operations contained in the generation. This will enable future work to keep around all generations containing operations after a certain sequence number (e.g., the global checkpoint).

jasontedor · 2017-02-20T19:10:04Z

I left some comments nothing major

Thanks @s1monw. I pushed 772a513.

s1monw

LGTM

s1monw · 2017-02-20T19:17:45Z

@jasontedor looks awesome

* master: Mark IP range aggregator test as awaits fix Add note and link to 'tune for disk usage' (elastic#23252)

jasontedor · 2017-02-20T20:05:45Z

Thanks @nik9000, @bleskes, and @s1monw.

jasontedor added :Sequence IDs >enhancement v6.0.0-alpha1 labels Jan 26, 2017

jasontedor requested review from nik9000 and bleskes January 26, 2017 19:13

nik9000 reviewed Jan 26, 2017

View reviewed changes

s1monw suggested changes Jan 27, 2017

View reviewed changes

bleskes suggested changes Jan 27, 2017

View reviewed changes

bleskes mentioned this pull request Feb 1, 2017

Add Sequence Numbers to write operations #10708

Closed

64 tasks

jasontedor force-pushed the introduce-sequence-number-aware-translog branch 2 times, most recently from fd7b4a6 to 4dfa75d Compare February 7, 2017 21:30

nik9000 reviewed Feb 8, 2017

View reviewed changes

bleskes suggested changes Feb 12, 2017

View reviewed changes

jasontedor force-pushed the introduce-sequence-number-aware-translog branch from 10cc2a8 to 7c50088 Compare February 16, 2017 15:12

bleskes approved these changes Feb 17, 2017

View reviewed changes

bleskes reviewed Feb 17, 2017

View reviewed changes

bleskes requested a review from s1monw February 17, 2017 16:28

nik9000 reviewed Feb 17, 2017

View reviewed changes

nik9000 approved these changes Feb 17, 2017

View reviewed changes

s1monw suggested changes Feb 20, 2017

View reviewed changes

jasontedor added 8 commits February 20, 2017 12:47

Add another usage of empty checkpoint helper

13a8030

Replace getMinSeqNo/getMaxSeqNo with getCheckpoint

5b60b52

Handling of min/max in TranslogWriter

5a55802

Reintroduce translog close into reader test

835c75e

Remove dead Javadoc

76ee8fb

Assert intial min/max sequence number

bed78d3

Test checkpoint transfer

d8a846e

Feedback

772a513

jasontedor force-pushed the introduce-sequence-number-aware-translog branch from 48f02c2 to 772a513 Compare February 20, 2017 19:08

s1monw approved these changes Feb 20, 2017

View reviewed changes

jasontedor added 2 commits February 20, 2017 14:44

Merge branch 'master' into introduce-sequence-number-aware-translog

36a2c85

* master: Mark IP range aggregator test as awaits fix Add note and link to 'tune for disk usage' (elastic#23252)

Add missing newline

ed3973c

jasontedor merged commit 4c2bd5f into elastic:master Feb 20, 2017

jasontedor deleted the introduce-sequence-number-aware-translog branch February 20, 2017 20:05

jasontedor mentioned this pull request Sep 29, 2017

Remove legacy support for translog checkpoint #21499

Closed

		@@ -778,6 +777,8 @@ public static Type fromId(byte id) {

		Source getSource();

		long seqNo();

		protected final long length;
		private final int totalOperations;

Introduce sequence-number-aware translog #22822

Introduce sequence-number-aware translog #22822

Conversation

jasontedor commented Jan 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes commented Jan 26, 2017

s1monw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasontedor commented Feb 7, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleskes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment