Skip to content

Commit

Permalink
loadgen touches
Browse files Browse the repository at this point in the history
  • Loading branch information
frankmcsherry committed Jul 12, 2024
1 parent 426ddf6 commit 7a981b0
Showing 1 changed file with 16 additions and 6 deletions.
22 changes: 16 additions & 6 deletions posts/2024-05-19.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Others have users of their own, and know that sluggish, stale, inconsistent resu
Many users won't believe a thing until the data looks like their data, with the same schemas and data distributions, and the same business logic.
These are all legitimate concerns, and to me they speak to the inherent *heterogeneity* involved in demonstrating something.

I want to be able to demonstrate Materialize more effectively, which is some amount tied up in demonstrating it more flexibly.
I want to be able to demonstrate Materialize more **effectively**, which is some amount tied up in demonstrating it more **flexibly**.

As a personal first, I'm going to try telling the story in reverse order, Memento-style.
We'll start with the outcomes, which I hope will make sense, and then figure out how we got there, and eventually arrive at the wall of SQL that makes it happen.
Expand Down Expand Up @@ -375,8 +375,9 @@ This guarantee is called "strict serializability", that the system behaves as if

That's it!
We've completed the introduction to Materialize, and used auction data to show off responsiveness, freshness, and consistency.
There's a lot more to show off, of course, but it's not exactly the point of this post.
Instead, we want to unpack how we got all of that `auctions` and `bids` data in the first place!
There's a lot more to show off, of course, and if any of this sounded fascinating you should swing by https://materialize.com/register/ to spin up a trial environment.

However, in this post we will continue to unpack how we got all of that `auctions` and `bids` data in the first place!

### Auction Data from Changing Moments

Expand All @@ -394,14 +395,15 @@ SELECT generate_series(
```

Unpacking this, `moments` contains rows with a single column containing a timestamp.
Whenever we look at it, the view contains those timestamps less than or equal to `now()`, but at most one day less.
Whenever we look at it, the view contains those timestamps at most one day less than `now()`.
It should have at any moment exactly 86,400 records present, as many as `auctions` up above.

Importantly, this view definition will not actually work for us.
You are welcome to try it out, but you'll find out that while it can be *inspected*, it cannot be *maintained*.
We'll fix that by the end of the post.
We'll fix that by the end of the post, but it will need to wait until the next section.
For the moment, let's assume we have this view and the magical ability to keep it up to date.

These moments are not auction data, though.
These "moments" are not auction data, though.
How do we get from moments to auctions and bids?

The `auctions` and `bids` collections look roughly like so:
Expand Down Expand Up @@ -552,6 +554,10 @@ For example, random bids means that auctions stop changing as they go on, becaus
You could instead have the bids trend up with time, to keep the data interesting.
But, the changes are pretty easy to roll out, and just amount to editing the SQL that defines them.

Let's pause for now on noodling on ways we could make the data even more realistic.
Up next we have to unpack how we got that `moments` view in the first place.
Once we've done that, you are welcome to go back to playing around with load generator novelties and variations!

### Operational Data from Thin Air

Our `auctions` and `bids` data was based on a view `moments` that showed us all timestamps within the past three hours.
Expand Down Expand Up @@ -736,6 +742,10 @@ materialize=>
All rows with a second column of `t` are "progress" statements rather than data updates.
The second row, the only one with a `f`, confirms a single record (`1`) with a value of `86400`.

Yeah, that's it! The only thing left is to read a wall of text containing all the SQL.
Actually, I recommend bouncing up to the start of the post again, and confirming that the pieces fit together for you.
It's also a fine time to [try out Materialize](https://materialize.com/register/), the only system that can run all of these views.

### Appendix: All the SQL

```sql
Expand Down

0 comments on commit 7a981b0

Please sign in to comment.