HPCC-33601 Document lz4s and lz4shc index compression and options #19605

JamesDeFabia · 2025-03-11T01:17:26Z

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Smoketest:

Send notifications about my Pull Request position in Smoketest queue.
Test my draft Pull Request.

Testing:

Unit Test:
https://github.com/JamesDeFabia/github-action-dev-build/actions/runs/13777976447

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

github-actions · 2025-03-11T01:17:44Z

Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-33601

Jirabot Action Result:
Workflow Transition To: Merge Pending
Updated PR

g-pan

few comments inline

g-pan · 2025-03-11T11:58:54Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

+            role="bold">'inplace:lz4shc'</emphasis> </emphasis></entry>
+
+            <entry>The default compression in versions after versions 9.6.90,
+            9.8.66 ,and 9.10.12. Causes inplace compression on the key fields


spacing s/b: " 9.8.66, and "

g-pan · 2025-03-11T11:59:29Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

-            Lempel-Ziv-Welch algorithm. It remains the default for backward
-            compatibility.</entry>
+            <entry>A variant of the Lempel-Ziv-Welch algorithm. This was the
+            the default compression prior to versions 9.6.90, 9.8.66 ,and


spacing s/b: " 9.8.66, and "

g-pan · 2025-03-11T12:13:01Z

docs/EN_US/ECLLanguageReference/ECLR_mods/Recrd-Index.xml

-            Lempel-Ziv-Welch algorithm. It remains the default for backward
-            compatibility.</entry>
+            <entry>A variant of the Lempel-Ziv-Welch algorithm. This was the
+            the default compression prior to versions 9.6.90, 9.8.66 ,and


spacing s/b: " 9.8.66, and "

g-pan · 2025-03-11T12:16:28Z

docs/EN_US/ECLLanguageReference/ECLR_mods/Recrd-Index.xml

@@ -412,18 +433,82 @@ BUILD(VehicleKey3);
    without decompression. The original index compression implementation
    decompresses the rows when they are read from disk.</para>

+    <para>The inplace index compression format (introduced in versions 9.6.90,
+    9.8.66 ,and 9.10.12 9.2.0 or later) improves compression and reduces build


spacing, comma s/b: " 9.8.66, 9.10.12, and 9.2.0 or "

g-pan · 2025-03-11T12:16:45Z

docs/EN_US/ECLLanguageReference/ECLR_mods/Recrd-Index.xml

+    9.8.66 ,and 9.10.12 9.2.0 or later) improves compression and reduces build
+    time. These formats require an engine that supports it. In other words,
+    <emphasis role="bold">if you build an index using the lz4s or lz4shc
+    formats, you must use a platform later than 9.6.90, 9.8.66 ,and 9.10.12 to


spacing s/b: " 9.8.66, and "

g-pan · 2025-03-11T12:17:44Z

docs/EN_US/ECLLanguageReference/ECLR_mods/Recrd-Index.xml

+    read those indexes. </emphasis></para>
+
+    <para>If you attempt to read an index with the inplace compression format
+    on a system that does not support them, you will receive an error


awkward grammar: an index -- support them

g-pan

looks good now

ghalliday

Thanks. A few comments.

ghalliday · 2025-03-11T13:36:02Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

@@ -441,8 +441,8 @@

              <entry><para>Optional. Specifies the index should be compressed
              using the type of compression specified. If omitted, the default
-              is <emphasis role="bold">LZW</emphasis>, a variant of the
-              Lempel-Ziv-Welch algorithm. </para></entry>
+              is <emphasis role="bold">'inplace:lz4shc'</emphasis>.


This has not changed yet - the default is still LZW

ghalliday · 2025-03-11T13:37:25Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

+            <entry><emphasis role="bold"><emphasis
+            role="bold">'inplace:lz4shc'</emphasis> </emphasis></entry>
+
+            <entry>The default compression in versions after versions 9.6.90,


The default compression for inplace indexes.
Both this and lz4s are only supported in those versions and later.

ghalliday · 2025-03-11T13:40:17Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

+            role="bold">'inplace:lz4s'</emphasis> </emphasis></entry>
+
+            <entry>Causes inplace compression on the key fields and lz4s
+            compression on the payload. This uses the streaming API to build


This uses the stream LZ4 API to avoid recompressing the data and reduce the index build times.

Similar for lz4shc below.

ghalliday · 2025-03-11T13:41:44Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

@@ -903,25 +925,86 @@ BUILD(FilterDsLib1);
    without decompression. The original index compression implementation
    decompresses the rows when they are read from disk.</para>

+    <para>The inplace index compression format (introduced in versions 9.6.90,


It isn't the inplace index compression that was introduced it was already supported. I am not sure that either of these paragraphs are needed.

ghalliday · 2025-03-11T13:42:58Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml


-    <para>If you attempt to read an index with the inplace compression format


This paragraph is still true. Possibly should be extended with "or an inplace compression format"

ghalliday · 2025-03-11T13:43:55Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

+          <row>
+            <entry><emphasis role="bold">hclevel</emphasis></entry>
+
+            <entry>An integer between 0 and 9 to specify the level of


Range is 2 to 12.

ghalliday · 2025-03-11T13:46:04Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

+
+            <entry>An integer between 0 and 9 to specify the level of
+            compression. The default is 3. Higher levels increase compression
+            times, but may be cost-effective.</entry>


Higher levels increase the compression, but also increase the compression times.

I'm not sure if the cost comment is worthwhile, but if it is it needs to be clear.

This may be cost effective depending on the length of time the data is stored, and the storage costs compared to the compute costs to build the index.

ghalliday · 2025-03-11T13:46:53Z

docs/EN_US/ECLLanguageReference/ECLR_mods/BltInFunc-BUILD.xml

+            <entry><emphasis role="bold">maxrecompress</emphasis></entry>
+
+            <entry>Specifies the number of times the entire input dataset
+            should be compressed to free up space. Increasing the number


compressed -> recompressed

ghalliday · 2025-03-11T13:48:05Z

docs/EN_US/ECLLanguageReference/ECLR_mods/Recrd-Index.xml

+
+          <row>
+            <entry><emphasis role="bold"><emphasis
+            role="bold">'inplace:lz4s'</emphasis> </emphasis></entry>


Is there any way to common up this text - otherwise you will have to apply the same edits to both.

…ions Signed-off-by: Jim DeFabia <jamesdefabia@lexisnexis.com>

JamesDeFabia requested review from ghalliday, g-pan and Copilot March 11, 2025 01:17

Copilot AI reviewed Mar 11, 2025

View reviewed changes

g-pan reviewed Mar 11, 2025

View reviewed changes

JamesDeFabia force-pushed the HPCC-33601lz4shc branch from 1862d88 to 762c1f6 Compare March 11, 2025 12:33

JamesDeFabia requested a review from g-pan March 11, 2025 12:35

JamesDeFabia force-pushed the HPCC-33601lz4shc branch from 762c1f6 to dfa3307 Compare March 11, 2025 12:48

g-pan approved these changes Mar 11, 2025

View reviewed changes

ghalliday requested changes Mar 11, 2025

View reviewed changes

HPCC-33601 Document the new lz4s and lz4shc index compression and opt…

02b0cd8

…ions Signed-off-by: Jim DeFabia <jamesdefabia@lexisnexis.com>

JamesDeFabia force-pushed the HPCC-33601lz4shc branch from dfa3307 to 02b0cd8 Compare March 12, 2025 00:41

JamesDeFabia requested a review from ghalliday March 12, 2025 00:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-33601 Document lz4s and lz4shc index compression and options #19605

HPCC-33601 Document lz4s and lz4shc index compression and options #19605

JamesDeFabia commented Mar 11, 2025 •

edited

Loading

github-actions bot commented Mar 11, 2025

g-pan left a comment

g-pan Mar 11, 2025

g-pan Mar 11, 2025

g-pan Mar 11, 2025

g-pan Mar 11, 2025

g-pan Mar 11, 2025

g-pan Mar 11, 2025

g-pan left a comment

ghalliday left a comment

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025

ghalliday Mar 11, 2025


		<para>If you attempt to read an index with the inplace compression format

HPCC-33601 Document lz4s and lz4shc index compression and options #19605

Are you sure you want to change the base?

HPCC-33601 Document lz4s and lz4shc index compression and options #19605

Conversation

JamesDeFabia commented Mar 11, 2025 • edited Loading

Type of change:

Checklist:

Smoketest:

Testing:

Choose a reason for hiding this comment

github-actions bot commented Mar 11, 2025

g-pan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

g-pan left a comment

Choose a reason for hiding this comment

ghalliday left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JamesDeFabia commented Mar 11, 2025 •

edited

Loading