Fix CMYK blending order: blend in CMYK space then convert to RGB

[hjanuschka]

Summary

Fix multi-layer CMYK image blending by deferring CMS conversion until after blending operations complete.

Problem: For CMYK images with alpha blending across multiple layers, blending in RGB space (after CMS conversion) produces incorrect results. The correct approach is to blend in CMYK space first, then convert to RGB.

Solution: Defer CMS conversion for CMYK images when:

1. The frame needs blending, OR
2. The frame can be referenced by later frames (save_before_ct=false)

This ensures reference frames are saved in CMYK space so subsequent frames blend correctly before final CMS conversion.

Changes

• jxl/src/frame/render.rs: Restructured CMS stage application to support deferred execution for CMYK blending

Credit

Based on the fix from @lilith in lilith/jxl-rs.

[github-actions]

Benchmark @ 6bc7538

MULTI-FILE BENCHMARK RESULTS (4 files)
  CPU architecture: x86_64
  WARNING: System appears noisy: high system load (2.03). Results may be unreliable.
Statistics:
  Confidence:               99.0%
  Max relative error:        3.0%

Comparing: 90a2bb2a (Base) vs fbe56cd9 (PR)

File	Base (MP/s)	PR (MP/s)	Δ%
bike.jxl	25.124	24.817	-1.22% ±1.9%
green_queen_modular_e3.jxl	7.880	7.859	-0.27% ±0.4%
green_queen_vardct_e3.jxl	23.168	23.192	+0.11% ±1.7%
sunset_logo.jxl	2.245	2.269	+1.09% ±3.0%

[jonsneyers]

For multi-layer images, not only in the case of CMYK, the frame blending has to be done in the image space, not in the final decoded image space. This also makes a difference in the RGB case, e.g. if the image space is linear Rec2020 (i.e. that's what the imageheader says), and we're decoding to sRGB, then frames have to be kept in or converted to linear Rec2020 for frame blending and only after blending the result should be converted to the final decode space.

Are we doing this correctly?
It's a subtle point but it does make a difference.

[hjanuschka]

Tried generalizing this to all blending cases but it breaks conformance.
(animation_icos4d_5 goes from ~0.0005 to ~0.2 RMSE)

So it seems libjxl blends after XybStage but before CMS for non-CMYK?

[jonsneyers]

Tried generalizing this to all blending cases but it breaks conformance. (animation_icos4d_5 goes from ~0.0005 to ~0.2 RMSE)

So it seems libjxl blends after XybStage but before CMS for non-CMYK?

Yes, blending should be done after converting XYB to RGB, in particular whatever RGB space the image is tagged to be in.
So in case a different RGB space is requested as output space than the image space, then the blending still has to be done in the image space, not in the output space.

For example if an image is tagged as sRGB, encoded with XYB, and we're decoding to Linear P3 (just for the sake of the example), then the correct flow should be this:

• decode frame in XYB (and apply patches and splines and whatnot all in XYB)
• convert to sRGB
• save frame as reference frame in sRGB space
• if duration>0: convert to linear P3 and return that as an animation frame
• decode next frame in XYB
• convert it to sRGB too
• blend it in sRGB space with the previous frame
• save frame as reference frame in sRGB space
• if duration>0 or last frame: convert to linear P3 and return that

This is somewhat annoying because of the double conversions but we cannot define blending to be done in whatever decode space you like (since that would give different results) and we also cannot define it has to be done in the encoded space (XYB in this example) because then it would also look different between a lossy encoding and a lossless one that would use sRGB as the encoding space in this example.

In particular existing GIF or APNG files cannot be losslessly represented (neither accurately in a lossy way) if we don't define blending to have to happen in the image space. Same with existing layered images like XCF or PSD files, which also have the convention that layer blending is done in image space.

[hjanuschka]

generalized the CMS deferral to all blending cases now, not just CMYK.The key insight for making it work was two-fold:So the pipeline order is now:
XybStage -> FromLinearStage(image TF) -> Blend -> Save ref frame -> CMS(image space -> output space)All tests pass including animation_icos4d_5