Skip to content

Commit

Permalink
Remove unicode optimization in Lua cjson library (#1785)
Browse files Browse the repository at this point in the history
The Lua cjson library implements an optimization that pre-allocates a
string buffer by estimating the maximum memory used if all characters in
a string require to be represented as unicode escapes, which may take 6
bytes each. Therefore, if a string has `len` bytes, the pre-allocated
buffer will have `len * 6` bytes.

This optimization can easily cause OOM errors, because if someone uses a
string with a few gigabytes of size, the pre-allocator will require 6
times the size of that string, and when running the Lua script in a
small instance with low memory, it will make the valkey-server process
to abort.

I ran the following Lua script to check if there is a significant
performance regression:

```
local s = string.rep("a", 1024 * 1024 * 1024)
return #cjson.encode(s..s..s)
```

The execution duration, in seconds, for 3 runs before and after this
commit is the following:

Before: 46.309; 42.443; 42.242

After: 46.729; 42.969; 42.774

---------

Signed-off-by: Ricardo Dias <ricardo.dias@percona.com>
Signed-off-by: Ricardo Dias <rjd15372@gmail.com>
Co-authored-by: Viktor Söderqvist <viktor.soderqvist@est.tech>
  • Loading branch information
rjd15372 and zuiderkwast authored Feb 26, 2025
1 parent 7db01d8 commit 1ad0c4f
Showing 1 changed file with 4 additions and 8 deletions.
12 changes: 4 additions & 8 deletions deps/lua/src/lua_cjson.c
Original file line number Diff line number Diff line change
Expand Up @@ -469,23 +469,19 @@ static void json_append_string(lua_State *l, strbuf_t *json, int lindex)

str = lua_tolstring(l, lindex, &len);

/* Worst case is len * 6 (all unicode escapes).
* This buffer is reused constantly for small strings
* If there are any excess pages, they won't be hit anyway.
* This gains ~5% speedup. */
if (len > SIZE_MAX / 6 - 3)
if (len > SIZE_MAX - 3)
abort(); /* Overflow check */
strbuf_ensure_empty_length(json, len * 6 + 2);
strbuf_ensure_empty_length(json, len + 2);

strbuf_append_char_unsafe(json, '\"');
for (i = 0; i < len; i++) {
escstr = char2escape[(unsigned char)str[i]];
if (escstr)
strbuf_append_string(json, escstr);
else
strbuf_append_char_unsafe(json, str[i]);
strbuf_append_char(json, str[i]);
}
strbuf_append_char_unsafe(json, '\"');
strbuf_append_char(json, '\"');
}

/* Find the size of the array on the top of the Lua stack
Expand Down

0 comments on commit 1ad0c4f

Please sign in to comment.