Merge pull request #110 from upserve/fix_docs

Add some diagrams to the readme and fix docs

README.md

@@ -5,9 +5,15 @@ Uppend: an append-only, key-multivalue store
 [![Test Coverage](https://img.shields.io/codecov/c/github/upserve/uppend/master.svg?style=flat-square)](https://codecov.io/github/upserve/uppend?branch=master)
 [![Release](https://jitpack.io/v/upserve/uppend.svg)](https://jitpack.io/#upserve/Uppend)
 
-Uppend is an append-only, key-multivalue store which is suitable for streaming
-event aggregation. It assumes a single writer process, and appended values are
-immutable.
+Uppend is an append-only, key-multivalue store which is suitable for streaming aggregation of immutable event data. 
+It is designed to handle both wide (large number of unique keys) and deep (large number of elements in a particular 
+key) data. Uppend is a multithreaded embedded java database. Uppend requires a single writer process but allows 
+multiple reader processes sharing the same file system. Uppend guarantees you will read your writes from the writer 
+process. New keys are not durably written immediately, they are added to a temporary cache. The write cache is 
+periodically flushed to disk. Readonly processes are configured to re-read the key lookup data periodically. Uppend
+heavily utilizes the mmap, and the linux page cache. On the i3.metal AWS instance class real Upserve applications
+sustain millions of read and append operations per minute with median latency below 100 micro seconds on tables with
+more than 1B keys.
 
 Use
 ---
@@ -18,13 +24,13 @@ Maven:
 <dependency>
     <groupId>com.upserve</groupId>
     <artifactId>uppend</artifactId>
-    <version>0.2.1</version>
+    <version>0.3.0</version>
 </dependency>
 ```
 
 Gradle:
 ```gradle
-compile 'com.upserve:uppend:0.2.1'
+compile 'com.upserve:uppend:0.3.0'
 ```
 
 Hello world:
@@ -41,6 +47,48 @@ String values = db.readSequential("my-appendStorePartition", "my-key")
 // value-1, value-2
 ```
 
+Diagrams
+--------
+
+AppendStore & CounterStore
+
+![interface](images/uppend_interface.png)
+
+
+Data Model
+----------
+
+Uppend stores four different kinds of data stored in four distinct files
+* Arbitrary blobs of bytes are appended immediately in the order they are received. The starting position is all that is required 
+  to read the blob again.
+* Blocks of blob positions are allocated for each key and blob position are added to the block immediately. The starting position 
+  of the first block for a particular key is all that is required to read the array of blob positions from a linked 
+  list of blocks.
+* The key and the starting position of the block appended to a blob store. These are held in a write cache until they are flushed.
+* The starting position of each key is stored as an array of integers in lexical key sort order written during a flush.
+
+![DataModel](https://docs.google.com/drawings/d/e/2PACX-1vTNVV_GxkaU_mvig5xLnIEtC2a1Aeheq5VokAp0aS_3OQYsNvsDMUiFPtlB2Vgyz3g9RE4SZWcYjOVF/pub?w=1182&h=374)
+
+This is the contents of an AppendOnlyStore directory with 128 partitions
+
+```
+$ ls * partitions/0000
+readLock	writeLock
+
+partitions:
+0000	0008	0016	0024	0032	0040	0048	0056	0064	0072	0080	0088	0096	0104	0112	0120
+0001	0009	0017	0025	0033	0041	0049	0057	0065	0073	0081	0089	0097	0105	0113	0121
+0002	0010	0018	0026	0034	0042	0050	0058	0066	0074	0082	0090	0098	0106	0114	0122
+0003	0011	0019	0027	0035	0043	0051	0059	0067	0075	0083	0091	0099	0107	0115	0123
+0004	0012	0020	0028	0036	0044	0052	0060	0068	0076	0084	0092	0100	0108	0116	0124
+0005	0013	0021	0029	0037	0045	0053	0061	0069	0077	0085	0093	0101	0109	0117	0125
+0006	0014	0022	0030	0038	0046	0054	0062	0070	0078	0086	0094	0102	0110	0118	0126
+0007	0015	0023	0031	0039	0047	0055	0063	0071	0079	0087	0095	0103	0111	0119	0127
+
+partitions/0000:
+blobStore	blockedLongs	keyMetadata	keys
+```
+
 Development
 -----------
 

src/main/java/com/upserve/uppend/blobs/VirtualPageFile.java

@@ -14,9 +14,9 @@
 import java.util.stream.IntStream;
 
 /**
- * Simulates contiguous pages for virtual files in a single physical file
- * Pages are double linked with head and tail pointers for each virtual file
- * The header maintains a table of pages for each virtual file.
+ * The VirtualPageFile simulates contiguous pages for virtual files contained in a single physical file.
+ * The header and page table contain metadata similar to inodes in a file system.
+ * The number of virtual files and the page size is fixed when the object is created.
  * <p>
  * Self Describing Header: virtualFiles (int), pageSize (int)
  * <p>