Clean up linear_int8_dynamic_activation_intx_weight_subclass #1553

metascroy · 2025-01-11T04:00:22Z

Summary:
Cleans up layout and quantization API:

int8_dynamic_activation_intx_weight(
    weight_dtype=torch.int4,
    granularity=PerGroup(128),
    has_weight_zeros: bool = False,
    weight_mapping_type=MappingType.ASYMMETRIC,
    act_mapping_type=MappingType.ASYMMETRIC,
    layout=PackedLinearInt8DynamicActivationIntxWeightLayout(),
)

int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower).

The fallback option is removed and instead relies on using PlainLayout().

Reviewed By: jerryzh168

Differential Revision: D67821939

pytorch-bot · 2025-01-11T04:00:27Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1553

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f04f3b2 with merge base de5c6e1 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-01-11T04:00:57Z

This pull request was exported from Phabricator. Differential Revision: D67821939

jerryzh168 · 2025-01-11T05:37:21Z

torchao/experimental/docs/readme.md

@@ -60,15 +60,15 @@ from torchao.quantization.quant_api import quantize_
 quantize_(
    my_model,
    int8_dynamic_activation_intx_weight(
+        bit_width=4,


this could be updated to using real dtype now if you can use torch nightly btw, both torch.uintx and torch.intx are available in nightly (2.6 and later I think)

Changed bit_width to weight_dtype

please change README as well

Updated readme

facebook-github-bot · 2025-01-13T05:00:28Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T05:06:47Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T18:57:32Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T19:10:38Z

This pull request was exported from Phabricator. Differential Revision: D67821939

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

jerryzh168 · 2025-01-13T21:21:39Z

torchao/experimental/quant_api.py

+            + "  Alternatively, use layout=PlainLayout() with int8_dynamic_activation_intx_weight, but note that doing so will result in much slower performance."
+        )
+
+    dtype_to_bit_width = {


why not pass this around to layout as well?

I think for layout, bit_width is more convenient a more to use because I can then do something like this to call the kernel:

getattr(torch.ops.torchao, f"_pack_8bit_act_{layout.bit_width}bit{wzp_suffix}_weight")(*args)

I can change, though, if you think it's better.

I see, depending on whether there could be cases of uintx as well I think, if not, then it's fine since it's not user facing

Let's leave it as bitwidth then. The quantizer is specifically for intx, not uintx, and the layout is not user facing.

sg, also we have a util here:

ao/torchao/quantization/quant_primitives.py

Line 179 in b3deb16

_DTYPE_TO_BIT_WIDTH.update(

…#1553) Summary: Pull Request resolved: pytorch#1553 Cleans up layout and quantization API: ``` int8_dynamic_activation_intx_weight( group_size: int = 128, bit_width: int = 4, has_weight_zeros: bool = False, weight_mapping_type=MappingType.ASYMMETRIC, act_mapping_type=MappingType.ASYMMETRIC, layout=PackedLinearInt8DynamicActivationIntxWeightLayout(), ) ``` int8_dynamic_activation_intx_weight is now very similar to int8_dynamic_activation_int4_weight. By passing bit_width=4, has_weight_zeros=false, and layout=PlainLayout(), it should be numerically identical (but slower). The fallback option is removed and instead relies on using PlainLayout(). Reviewed By: jerryzh168 Differential Revision: D67821939

facebook-github-bot · 2025-01-13T22:09:06Z

This pull request was exported from Phabricator. Differential Revision: D67821939

jerryzh168 · 2025-01-14T20:06:04Z

torchao/experimental/quant_api.py

+        quant_min = -(1 << (bit_width - 1))
+        quant_max = (1 << (bit_width - 1)) - 1


we also have utils here:

ao/torchao/quantization/quant_primitives.py

Line 168 in b3deb16

_SUB_BYTE_INT_BOUNDS.update(

jerryzh168

LGTM, thanks

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 11, 2025

facebook-github-bot added the fb-exported label Jan 11, 2025

metascroy requested a review from jerryzh168 January 11, 2025 04:01

metascroy mentioned this pull request Jan 11, 2025

[Feat]: Add support for kleidiai quantization schemes #1447

Open

jerryzh168 reviewed Jan 11, 2025

View reviewed changes

metascroy force-pushed the export-D67821939 branch from 8752c4c to 23688a9 Compare January 13, 2025 05:00

metascroy added the topic: improvement Use this tag if this PR is an improvement (doesn't fit into any of the other categories) label Jan 13, 2025

metascroy force-pushed the export-D67821939 branch from 23688a9 to 8d47b97 Compare January 13, 2025 05:06

metascroy force-pushed the export-D67821939 branch from 8d47b97 to 0552dcf Compare January 13, 2025 18:57

metascroy force-pushed the export-D67821939 branch from 0552dcf to dc32105 Compare January 13, 2025 19:10

metascroy requested a review from jerryzh168 January 13, 2025 21:07

jerryzh168 reviewed Jan 13, 2025

View reviewed changes

metascroy force-pushed the export-D67821939 branch from dc32105 to f04f3b2 Compare January 13, 2025 22:09

metascroy requested a review from jerryzh168 January 14, 2025 17:36

jerryzh168 reviewed Jan 14, 2025

View reviewed changes

jerryzh168 approved these changes Jan 14, 2025

View reviewed changes

facebook-github-bot merged commit 0bc5b00 into pytorch:main Jan 14, 2025
19 of 20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up linear_int8_dynamic_activation_intx_weight_subclass #1553

Clean up linear_int8_dynamic_activation_intx_weight_subclass #1553

metascroy commented Jan 11, 2025 •

edited

Loading

pytorch-bot bot commented Jan 11, 2025 •

edited

Loading

facebook-github-bot commented Jan 11, 2025

jerryzh168 Jan 11, 2025

metascroy Jan 13, 2025

jerryzh168 Jan 13, 2025

metascroy Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

jerryzh168 Jan 13, 2025

metascroy Jan 13, 2025

jerryzh168 Jan 13, 2025

metascroy Jan 13, 2025

jerryzh168 Jan 14, 2025 •

edited

Loading

facebook-github-bot commented Jan 13, 2025

jerryzh168 Jan 14, 2025

jerryzh168 left a comment

		quant_min = -(1 << (bit_width - 1))
		quant_max = (1 << (bit_width - 1)) - 1

Clean up linear_int8_dynamic_activation_intx_weight_subclass #1553

Clean up linear_int8_dynamic_activation_intx_weight_subclass #1553

Conversation

metascroy commented Jan 11, 2025 • edited Loading

pytorch-bot bot commented Jan 11, 2025 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/1553

✅ No Failures

facebook-github-bot commented Jan 11, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

facebook-github-bot commented Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

facebook-github-bot commented Jan 13, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jerryzh168 Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

facebook-github-bot commented Jan 13, 2025

Choose a reason for hiding this comment

jerryzh168 left a comment

Choose a reason for hiding this comment

metascroy commented Jan 11, 2025 •

edited

Loading

pytorch-bot bot commented Jan 11, 2025 •

edited

Loading

jerryzh168 Jan 14, 2025 •

edited

Loading