Voodoo Quant

Quantization Survey: AuxDecQ MLP32/128 Recovery

Base model: Qwen/Qwen3.5-0.8B-Base evaluated on the FineWeb validation set (50-step eval).

BF16 base: PPL=37.2, NLL=3.6162. Each AuxDecQ variant is a per-layer MLP AuxDecoder trained for 200 steps.

Quant	Size ↑%	Baseline PPL	AuxDecQ PPL	PPL ↓%	Baseline KL	AuxDecQ KL	KL ↓%
Qwen3.5-0.8B-IQ1_M	3.7%	599.59	111.61	81.4%	2.6706	1.0962	59.0%
Qwen3.5-0.8B-IQ1_S	3.8%	1756.8	126.07	92.8%	3.7481	1.2345	67.1%
Qwen3.5-0.8B-IQ4_XS	2.6%	47.7	44.09	7.6%	0.2123	0.2019	4.9%
Qwen3.5-0.8B-Q4_0	2.5%	48.56	43.52	10.4%	0.2328	0.1894	18.6%
Qwen3.5-0.8B-Q4_K_M	2.4%	45.96	44.48	3.2%	0.1921	0.1792	6.7%
Qwen3.5-0.8B-Q4_K_S	2.5%	46.76	43.0	8.0%	0.2086	0.1853	11.2%
Qwen3.5-0.8B-UD-IQ2_M	3.4%	75.57	50.88	32.7%	0.6516	0.402	38.3%
Qwen3.5-0.8B-UD-IQ2_XXS	3.7%	170.82	72.24	57.7%	1.3744	0.6473	52.9%
Qwen3.5-0.8B-UD-IQ3_XXS	3.2%	60.3	49.67	17.6%	0.4313	0.3166	26.6%
Qwen3.5-0.8B-UD-Q2_K_XL	3.0%	65.41	54.16	17.2%	0.5339	0.3739	30.0%
Qwen3.5-0.8B-UD-Q3_K_XL	2.6%	49.77	45.88	7.8%	0.2646	0.215	18.7%
Qwen3.5-0.8B-UD-Q4_K_XL	2.3%	46.0	44.51	3.2%	0.1866	0.1794	3.9%
Qwen3.5-0.8B-UD-Q5_K_XL	2.1%	45.9	44.16	3.8%	0.1744	0.169	3.1%
Qwen3.5-0.8B-UD-Q6_K_XL	1.6%	45.73	42.79	6.4%	0.1649	0.1593	3.4%

Base model: unsloth/Qwen3.5-4B-MTP-GGUF evaluated on the FineWeb validation set (50-step eval).

True 4B BF16 base: PPL ~17.69, NLL ~2.87. Each AuxDecQ variant is a per-layer bottleneck-MLP AuxDecoder trained for 200 steps.

Quant	Size ↑%	Baseline PPL	AuxDecQ PPL	PPL ↓%	Baseline KL	AuxDecQ KL	KL ↓%
Qwen3.5-4B-IQ1_M	3.1%	94.1	40.27	57.2%	1.7343	0.9337	46.2%
Qwen3.5-4B-IQ1_S	3.2%	190.92	47.54	75.1%	2.3896	1.1095	53.6%
Qwen3.5-4B-IQ4_XS	1.8%	19.0	17.73	6.7%	0.1245	0.1011	18.8%
Qwen3.5-4B-Q4_0	1.7%	18.43	16.88	8.4%	0.1266	0.0955	24.6%
Qwen3.5-4B-Q4_K_M	1.6%	18.32	17.09	6.7%	0.1072	0.0863	19.5%
Qwen3.5-4B-Q4_K_S	1.7%	18.47	17.27	6.5%	0.1117	0.0871	22.0%
Qwen3.5-4B-UD-IQ2_M	2.4%	24.11	20.19	16.3%	0.3786	0.2674	29.4%
Qwen3.5-4B-UD-IQ3_XXS	2.2%	21.81	19.18	12.1%	0.2764	0.2015	27.1%
Qwen3.5-4B-UD-Q2_K_XL	2.2%	24.18	20.42	15.6%	0.3623	0.259	28.5%
Qwen3.5-4B-UD-Q3_K_XL	1.8%	19.02	17.39	8.6%	0.1517	0.1212	20.1%
Qwen3.5-4B-UD-Q4_K_XL	1.5%	18.36	16.96	7.6%	0.1063	0.0843	20.8%
Qwen3.5-4B-UD-Q5_K_XL	1.4%	18.02	16.72	7.2%	0.0971	0.0736	24.3%
Qwen3.5-4B-UD-Q6_K_XL	1.1%	17.64	16.56	6.1%	0.0914	0.0705	22.8%

lm-eval, 0-shot.

Model	piqa	arc_easy	arc_challenge	hellaswag	winogrande
BF16 base	0.695	0.484	0.358	0.495	0.587
IQ4_XS stock	0.694	0.481	0.34	0.485	0.58
IQ4_XS + ADQ	0.704	0.486	0.356	0.504	0.599