Merge tag 'v0.10.0-rc.1' into krombel_websockets

[Full Changelog](matrix-org/matrix-js-sdk@v0.9.2...v0.10.0-rc.1)
* Fix duplicated state events in timeline from peek
[\matrix-org#630](matrix-org#630)
* Create indexeddb worker when starting the store
[\matrix-org#627](matrix-org#627)
* Fix indexeddb logging
[\matrix-org#626](matrix-org#626)
* Don't do /keys/changes on incremental sync
[\matrix-org#625](matrix-org#625)
* Don't mark devicelist dirty unnecessarily
[\matrix-org#623](matrix-org#623)
* Cache the joined member count for a room state
[\matrix-org#619](matrix-org#619)
* Fix JS doc
[\matrix-org#618](matrix-org#618)
* Precompute push actions for state events
[\matrix-org#617](matrix-org#617)
* Fix bug where global "Never send to unverified..." is ignored
[\matrix-org#616](matrix-org#616)
* Intern legacy top-level 'membership' field
[\matrix-org#615](matrix-org#615)
* Don't synthesize RR for m.room.redaction as causes the RR to go missing.
[\matrix-org#598](matrix-org#598)
* Make Events create Dates on demand
[\matrix-org#613](matrix-org#613)
* Stop cloning events when adding to state
[\matrix-org#612](matrix-org#612)
* De-dup code: use the initialiseState function
[\matrix-org#611](matrix-org#611)
* Create sentinel members on-demand
[\matrix-org#610](matrix-org#610)
* Some more doc on how sentinels work
[\matrix-org#609](matrix-org#609)
* Migrate room encryption store to crypto store
[\matrix-org#597](matrix-org#597)
* add parameter to getIdentityServerUrl to strip the protocol for invites
[\matrix-org#600](matrix-org#600)
* Move Device Tracking Data to Crypto Store
[\matrix-org#594](matrix-org#594)
* Optimise pushprocessor
[\matrix-org#591](matrix-org#591)
* Set event error before emitting
[\matrix-org#592](matrix-org#592)
* Add event type for stickers [WIP]
[\matrix-org#590](matrix-org#590)
* Migrate inbound sessions to cryptostore
[\matrix-org#587](matrix-org#587)
* Disambiguate names if they contain an mxid
[\matrix-org#588](matrix-org#588)
* Check for sessions in indexeddb before migrating
[\matrix-org#585](matrix-org#585)
* Emit an event for crypto store migration
[\matrix-org#586](matrix-org#586)
* Supporting fixes For making UnknownDeviceDialog not pop up automatically
[\matrix-org#575](matrix-org#575)
* Move sessions to the crypto store
[\matrix-org#584](matrix-org#584)
* Change crypto store transaction API
[\matrix-org#582](matrix-org#582)
* Add some missed copyright notices
[\matrix-org#581](matrix-org#581)
* Move Olm account to IndexedDB
[\matrix-org#579](matrix-org#579)
* Fix logging of DecryptionErrors to be more useful
[\matrix-org#580](matrix-org#580)
* [BREAKING] Change the behaviour of the unverfied devices blacklist flag
[\matrix-org#568](matrix-org#568)
* Support set_presence=offline for syncing
[\matrix-org#557](matrix-org#557)
* Consider cases where the sender may not redact their own event
[\matrix-org#556](matrix-org#556)

CHANGELOG.md

@@ -1,3 +1,78 @@
+Changes in [0.10.0-rc.1](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.10.0-rc.1) (2018-03-19)
+============================================================================================================
+[Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.9.2...v0.10.0-rc.1)
+
+ * Fix duplicated state events in timeline from peek
+   [\#630](https://github.com/matrix-org/matrix-js-sdk/pull/630)
+ * Create indexeddb worker when starting the store
+   [\#627](https://github.com/matrix-org/matrix-js-sdk/pull/627)
+ * Fix indexeddb logging
+   [\#626](https://github.com/matrix-org/matrix-js-sdk/pull/626)
+ * Don't do /keys/changes on incremental sync
+   [\#625](https://github.com/matrix-org/matrix-js-sdk/pull/625)
+ * Don't mark devicelist dirty unnecessarily
+   [\#623](https://github.com/matrix-org/matrix-js-sdk/pull/623)
+ * Cache the joined member count for a room state
+   [\#619](https://github.com/matrix-org/matrix-js-sdk/pull/619)
+ * Fix JS doc
+   [\#618](https://github.com/matrix-org/matrix-js-sdk/pull/618)
+ * Precompute push actions for state events
+   [\#617](https://github.com/matrix-org/matrix-js-sdk/pull/617)
+ * Fix bug where global "Never send to unverified..." is ignored
+   [\#616](https://github.com/matrix-org/matrix-js-sdk/pull/616)
+ * Intern legacy top-level 'membership' field
+   [\#615](https://github.com/matrix-org/matrix-js-sdk/pull/615)
+ * Don't synthesize RR for m.room.redaction as causes the RR to go missing.
+   [\#598](https://github.com/matrix-org/matrix-js-sdk/pull/598)
+ * Make Events create Dates on demand
+   [\#613](https://github.com/matrix-org/matrix-js-sdk/pull/613)
+ * Stop cloning events when adding to state
+   [\#612](https://github.com/matrix-org/matrix-js-sdk/pull/612)
+ * De-dup code: use the initialiseState function
+   [\#611](https://github.com/matrix-org/matrix-js-sdk/pull/611)
+ * Create sentinel members on-demand
+   [\#610](https://github.com/matrix-org/matrix-js-sdk/pull/610)
+ * Some more doc on how sentinels work
+   [\#609](https://github.com/matrix-org/matrix-js-sdk/pull/609)
+ * Migrate room encryption store to crypto store
+   [\#597](https://github.com/matrix-org/matrix-js-sdk/pull/597)
+ * add parameter to getIdentityServerUrl to strip the protocol for invites
+   [\#600](https://github.com/matrix-org/matrix-js-sdk/pull/600)
+ * Move Device Tracking Data to Crypto Store
+   [\#594](https://github.com/matrix-org/matrix-js-sdk/pull/594)
+ * Optimise pushprocessor
+   [\#591](https://github.com/matrix-org/matrix-js-sdk/pull/591)
+ * Set event error before emitting
+   [\#592](https://github.com/matrix-org/matrix-js-sdk/pull/592)
+ * Add event type for stickers [WIP]
+   [\#590](https://github.com/matrix-org/matrix-js-sdk/pull/590)
+ * Migrate inbound sessions to cryptostore
+   [\#587](https://github.com/matrix-org/matrix-js-sdk/pull/587)
+ * Disambiguate names if they contain an mxid
+   [\#588](https://github.com/matrix-org/matrix-js-sdk/pull/588)
+ * Check for sessions in indexeddb before migrating
+   [\#585](https://github.com/matrix-org/matrix-js-sdk/pull/585)
+ * Emit an event for crypto store migration
+   [\#586](https://github.com/matrix-org/matrix-js-sdk/pull/586)
+ * Supporting fixes For making UnknownDeviceDialog not pop up automatically
+   [\#575](https://github.com/matrix-org/matrix-js-sdk/pull/575)
+ * Move sessions to the crypto store
+   [\#584](https://github.com/matrix-org/matrix-js-sdk/pull/584)
+ * Change crypto store transaction API
+   [\#582](https://github.com/matrix-org/matrix-js-sdk/pull/582)
+ * Add some missed copyright notices
+   [\#581](https://github.com/matrix-org/matrix-js-sdk/pull/581)
+ * Move Olm account to IndexedDB
+   [\#579](https://github.com/matrix-org/matrix-js-sdk/pull/579)
+ * Fix logging of DecryptionErrors to be more useful
+   [\#580](https://github.com/matrix-org/matrix-js-sdk/pull/580)
+ * [BREAKING] Change the behaviour of the unverfied devices blacklist flag
+   [\#568](https://github.com/matrix-org/matrix-js-sdk/pull/568)
+ * Support set_presence=offline for syncing
+   [\#557](https://github.com/matrix-org/matrix-js-sdk/pull/557)
+ * Consider cases where the sender may not redact their own event
+   [\#556](https://github.com/matrix-org/matrix-js-sdk/pull/556)
+
 Changes in [0.9.2](https://github.com/matrix-org/matrix-js-sdk/releases/tag/v0.9.2) (2017-12-04)
 ================================================================================================
 [Full Changelog](https://github.com/matrix-org/matrix-js-sdk/compare/v0.9.1...v0.9.2)

package.json

@@ -1,6 +1,6 @@
 {
   "name": "matrix-js-sdk",
-  "version": "0.9.2",
+  "version": "0.10.0-rc.1",
   "description": "Matrix Client-Server SDK for Javascript",
   "main": "index.js",
   "scripts": {

spec/integ/matrix-client-syncing.spec.js

@@ -437,7 +437,11 @@ describe("MatrixClient syncing", function() {
             });
         });
 
-        it("should correctly interpret state in incremental sync.", function() {
+        // XXX: This test asserts that the js-sdk obeys the spec and treats state
+        // events that arrive in the incremental sync as if they preceeded the
+        // timeline events, however this breaks peeking, so it's disabled
+        // (see sync.js)
+        xit("should correctly interpret state in incremental sync.", function() {
             httpBackend.when("GET", "/sync").respond(200, syncData);
             httpBackend.when("GET", "/sync").respond(200, nextSyncData);
 

spec/unit/matrix-client.spec.js

@@ -132,11 +132,12 @@ describe("MatrixClient", function() {
         ].reduce((r, k) => { r[k] = expect.createSpy(); return r; }, {});
         store = [
             "getRoom", "getRooms", "getUser", "getSyncToken", "scrollback",
-            "save", "setSyncToken", "storeEvents", "storeRoom", "storeUser",
+            "save", "wantsSave", "setSyncToken", "storeEvents", "storeRoom", "storeUser",
             "getFilterIdByName", "setFilterIdByName", "getFilter", "storeFilter",
             "getSyncAccumulator", "startup", "deleteAllData",
         ].reduce((r, k) => { r[k] = expect.createSpy(); return r; }, {});
         store.getSavedSync = expect.createSpy().andReturn(Promise.resolve(null));
+        store.getSavedSyncToken = expect.createSpy().andReturn(Promise.resolve(null));
         store.setSyncData = expect.createSpy().andReturn(Promise.resolve(null));
         client = new MatrixClient({
             baseUrl: "https://my.home.server",

src/client.js

@@ -196,6 +196,9 @@ function MatrixClient(opts) {
     // we still want to know which rooms are encrypted even if crypto is disabled:
     // we don't want to start sending unencrypted events to them.
     this._roomList = new RoomList(this._cryptoStore, this._sessionStore);
+
+    // The pushprocessor caches useful things, so keep one and re-use it
+    this._pushProcessor = new PushProcessor(this);
 }
 utils.inherits(MatrixClient, EventEmitter);
 utils.extend(MatrixClient.prototype, MatrixBaseApis.prototype);
@@ -635,6 +638,16 @@ MatrixClient.prototype.isEventSenderVerified = async function(event) {
     return device.isVerified();
 };
 
+/**
+ * Cancel a room key request for this event if one is ongoing and resend the
+ * request.
+ * @param  {MatrxEvent} event event of which to cancel and resend the room
+ *                            key request.
+ */
+MatrixClient.prototype.cancelAndResendEventRoomKeyRequest = function(event) {
+    event.cancelAndResendKeyRequest(this._crypto);
+};
+
 /**
  * Enable end-to-end encryption for a room.
  * @param {string} roomId The room ID to enable encryption in.
@@ -1773,8 +1786,7 @@ function _membershipChange(client, roomId, userId, membership, reason, callback)
  */
 MatrixClient.prototype.getPushActionsForEvent = function(event) {
     if (!event.getPushActions()) {
-        const pushProcessor = new PushProcessor(this);
-        event.setPushActions(pushProcessor.actionsForEvent(event));
+        event.setPushActions(this._pushProcessor.actionsForEvent(event));
     }
     return event.getPushActions();
 };

src/crypto/DeviceList.js

@@ -92,6 +92,12 @@ export default class DeviceList {
 
         // Promise resolved when device data is saved
         this._savePromise = null;
+        // Function that resolves the save promise
+        this._resolveSavePromise = null;
+        // The time the save is scheduled for
+        this._savePromiseTime = null;
+        // The timer used to delay the save
+        this._saveTimer = null;
     }
 
     /**
@@ -146,38 +152,68 @@ export default class DeviceList {
      * The actual save will be delayed by a short amount of time to
      * aggregate multiple writes to the database.
      *
+     * @param {integer} delay Time in ms before which the save actually happens.
+     *     By default, the save is delayed for a short period in order to batch
+     *     multiple writes, but this behaviour can be disabled by passing 0.
+     *
      * @return {Promise<bool>} true if the data was saved, false if
      *     it was not (eg. because no changes were pending). The promise
      *     will only resolve once the data is saved, so may take some time
      *     to resolve.
      */
-    async saveIfDirty() {
+    async saveIfDirty(delay) {
         if (!this._dirty) return Promise.resolve(false);
+        // Delay saves for a bit so we can aggregate multiple saves that happen
+        // in quick succession (eg. when a whole room's devices are marked as known)
+        if (delay === undefined) delay = 500;
+
+        const targetTime = Date.now + delay;
+        if (this._savePromiseTime && targetTime < this._savePromiseTime) {
+            // There's a save scheduled but for after we would like: cancel
+            // it & schedule one for the time we want
+            clearTimeout(this._saveTimer);
+            this._saveTimer = null;
+            this._savePromiseTime = null;
+            // (but keep the save promise since whatever called save before
+            // will still want to know when the save is done)
+        }
+
+        let savePromise = this._savePromise;
+        if (savePromise === null) {
+            savePromise = new Promise((resolve, reject) => {
+                this._resolveSavePromise = resolve;
+            });
+            this._savePromise = savePromise;
+        }
 
-        if (this._savePromise === null) {
-            // Delay saves for a bit so we can aggregate multiple saves that happen
-            // in quick succession (eg. when a whole room's devices are marked as known)
-            this._savePromise = Promise.delay(500).then(() => {
+        if (this._saveTimer === null) {
+            const resolveSavePromise = this._resolveSavePromise;
+            this._savePromiseTime = targetTime;
+            this._saveTimer = setTimeout(() => {
                 console.log('Saving device tracking data at token ' + this._syncToken);
                 // null out savePromise now (after the delay but before the write),
                 // otherwise we could return the existing promise when the save has
                 // actually already happened. Likewise for the dirty flag.
+                this._savePromiseTime = null;
+                this._saveTimer = null;
                 this._savePromise = null;
+                this._resolveSavePromise = null;
+
                 this._dirty = false;
-                return this._cryptoStore.doTxn(
+                this._cryptoStore.doTxn(
                     'readwrite', [IndexedDBCryptoStore.STORE_DEVICE_DATA], (txn) => {
                         this._cryptoStore.storeEndToEndDeviceData({
                             devices: this._devices,
                             trackingStatus: this._deviceTrackingStatus,
                             syncToken: this._syncToken,
                         }, txn);
                     },
-                );
-            }).then(() => {
-                return true;
-            });
+                ).then(() => {
+                    resolveSavePromise();
+                });
+            }, delay);
         }
-        return this._savePromise;
+        return savePromise;
     }
 
     /**
@@ -418,11 +454,11 @@ export default class DeviceList {
         if (this._deviceTrackingStatus[userId]) {
             console.log('No longer tracking device list for ' + userId);
             this._deviceTrackingStatus[userId] = TRACKING_STATUS_NOT_TRACKED;
-        }
-        // we don't yet persist the tracking status, since there may be a lot
-        // of calls; we save all data together once the sync is done
 
-        this._dirty = true;
+            // we don't yet persist the tracking status, since there may be a lot
+            // of calls; we save all data together once the sync is done
+            this._dirty = true;
+        }
     }
 
     /**
@@ -453,11 +489,11 @@ export default class DeviceList {
         if (this._deviceTrackingStatus[userId]) {
             console.log("Marking device list outdated for", userId);
             this._deviceTrackingStatus[userId] = TRACKING_STATUS_PENDING_DOWNLOAD;
-        }
-        // we don't yet persist the tracking status, since there may be a lot
-        // of calls; we save all data together once the sync is done
 
-        this._dirty = true;
+            // we don't yet persist the tracking status, since there may be a lot
+            // of calls; we save all data together once the sync is done
+            this._dirty = true;
+        }
     }
 
     /**