Enhanced decoding of YJK images

Prophet (3424)

26-12-2012, 00:46

Many of you probably have noticed that the YJK color system used by the V9958 screens 10-12 have strong similarities with YUV. Also we now have a lot more CPU power than we had back in the 80's.

Combining those two premisses, I decided to try some of the techniques used to decode YUV/JPG images on the YJK system, and the results were very promising. The color artifacts were almost completely gone, and the number of simultaneous colors increased a lot.

I'm sharing this with you as a kind of Christmas gift, so the following kind of softwares can benefit:

1) MSX image readers/converters: Those tools import MSX images to a PC graphic editor, or convert the MSX image to a format suitable for editing on a PC. By adding this algorithm as an option, you can work with much higher quality images.
2) Emulators. Many emulators make use of some image enhancing algorithms to improve the image quality so it's more acceptable on modern displays. This algorithm may be used by them (and even combined with the other algorithms) to shown a much richer image.
3) FPGA implementations of the V9958, like the OneChipMSX: By adding this enhanced decoder they will be able to show the YJK images using a much more advanced decoder that results in a higher color count.

The algorithm is simple. YJK is very similar to YUV4:1:1. Just as with YUV conversion, you need to add an extra step to compensate for the color subsampling.

1) Read the YJK values as usual, for the group of 4 pixels.
2) Extra step: Apply Lanczos3 interpolation to the J and K values. This will give you the extra chroma information for each pixel, instead of just flat copying the J/K values for the 4 pixels of the group. For real time processing without too much power, you can replace Lanczos3 with linear or cubic interpolation, but the results will be also simpler.
3) Convert the Y and the new *per pixel* JK values to RGB values. You can even use RGB truecolor for this.

Just like YUV, this interpolation doesn't result in lack of perceived sharpness, since it only affects the chroma information. For more information on this topic, read this article. The algorithm can be used for both YJK and YAE (mixed YJK).

I simulated the process on GIMP, and the results shown bellow pretty much speak for themselves. They're zoomed to 2x or 3x to allow you to notice the results better.

First, the Gulliver pin-up by Teruchan. The .SCC file was read using the traditional flat copy of the JK values to the set of 4 pixels. You can notice a lot of color bleeding/blocking, resulting in a horrible jigsaw effects everywhere. Color count= 3830.

Then, the same SCC file being read when decoding the JK colors with Lanczos3 interpolation. All the blocking and jigsaw effects are gone, and the color bleeding is barely noticeable exactly as it happens on YUV and JPG images. Color count= 32847.

For comparison, those are the results when reading the same image by using Cubic and linear interpolation. You can notice that they also achieve much better results than the usual flat color copy, but are also inferior to the awesome results achieved by the Lanczos3 interpolation. This means one must use the best interpolation it CPU affords.

Cubic Interpolation. Color count=29957

Linear Interpolation. Color count=29244

Since the Lanczos3 are incomparably better, the next images will only show the comparison between the flat color copy and the Lanczos3 interpolation.

Haruko pin-up, with chroma information decoded with the traditional flat color copy. Blocks are formed everywhere. Color count=2667

Haruko pin-up, with chroma information decoded with Lanczos3. Smooth backgrounds, blocking is gone. Color count=27803

The classic Lena image, with chroma information decoded with the traditional flat color copy. Notice the horizontal color gradients have very low resolution. There's jigsaw on all curves. Color count=1280.

The classic Lena image, now with chroma information decoded with Lanczos3. Smooth horizontal gradients, blocking is gone. There's no jigsaw on the curves anymore. Color count=18486.

The screen12 opening screen of the game Nyancle Racing. All diagonal lines show jigsawed colors. Notice the cat ears. Color count=2293

The same opening screen, now decoded with Lanczos3. Jigsaws are gone, borders are perfect. Color count=22221.

One of the Quinpl intro screens, with chroma information decoded with the traditional flat color copy. Notice the jigsaw on all red elements. Color count=2836

The same intro screen, now decoded with Lanczos3. Jigsaws are gone, borders are perfect. Color count=28172.

Por sd_snatcher

Prophet (3424)

26-12-2012, 01:41

Here's the arm of the Golvellius intro image, animated to make the comparison easier. Notice the enhanced gradients:

Por ARTRAG

Enlighted (6467)

26-12-2012, 09:44

Great finding! It could be a case where emulators render better images than the original

Por Edwin

Paragon (1182)

26-12-2012, 10:50

While this is all true of course, I don't really see much point in this. Naturally you can improve quality with better modern stuff that provides more colors and nearly limitless processing power. But in this case, you may as well get the original image and have even better quality.

A more interesting thing would be to take what you've learnt and have a look at the encoding side of things. I've always had a feeling that the standard algorithm that is presented to encode four RGB values to 4x Y, J and K was suboptimal. And looking at the first image with the horrible edge along the hat, I still believe that there must be a better solution within the standard v9958 YJK encoding. Maybe some sort of J/K selection based on perceptual encoding. If you could improve on that, then you could make sc11/sc12 much better on a real MSX as well.

Por MsxKun

Paladin (959)

26-12-2012, 11:42

Whatever the purpose is for, i love the image selected for first example

Por supmsx

Master (158)

26-12-2012, 12:26

Great job,very useful

Por sd_snatcher

Prophet (3424)

26-12-2012, 14:05

@Edwin

The algorithm shown here focuses only on the decoding side, but you're right: It's also possible enhance the way the images are generated, to achieve a better result even on the standard YJK decoder. (Of course the enhanced YJK decoder will achieve even better results).

That said, the same principle applies for the encoding of YJK images: the algorithms used for YUV chroma subsampling may be used to produce better YJK images than the traditional "median color" used by the old YJK encoders. The "median color" algorithm produce the sharp sawtooth effect you saw on the first images.

Here's an excellent article about producing better results on chroma subsampling that certainly could be used for a better YJK encoder. For YAE it could even be combined with using a specified maximum number of RGB colors to "patch" the worst color bleeds

Por sd_snatcher

Prophet (3424)

11-03-2013, 05:04

Additional tests on real hardware revealed some more interesting aspects:

MSX connected to a CRT TV using S-Video

MSX connected to a LCD Monitor using RGB

MSX connected to a LCD Monitor using RGB (detail)

As you can notice, the MSX connected via S-Video results in a image very similar to the Cubic interpolation decoder, while when connected via RGB it is clearly like the flat-copy decoder that results in jagged edges (note: I incorrectly termed them "jigsaws" on the 1st post).

It seems that the designers of the YJK system took advantage of the NTSC decoder circuit (built in on any TV) to do the needed chroma interpolation for them. This means that screen12 images are one of the rare cases where the perceived image quality will be worse on RGB than on a S-Video connection: the perceived image will have less colors and jagged borders. Around 90% of the perceived colors are lost on an RGB connection due to the lack of the chroma interpolation step.

For emulators, it would suffice to implement any of the enhanced YJK decoders mentioned on the beginning of this thread. I would like to know the opinion of the openMSX team on this matter. What do you think?

In a nutshell, the YJK system seems to have been designed to have an "analog chroma interpolation step" that isn't available on any current emulator.

Of course when the V9958 is directly connected to a RGB monitor this step also isn't present. But it seems that the VDP was designed based to cover the 99% of the cases of how an MSX would be connected to a monitor: Using an NTSC decoder instead of direct RGB. This means that also the ideal way to connect the V9958 to a monitor would be via component-video, with a filter circuit (that could be enabled by software when needed) on the Pb and Pr signals. This filter would interpolate those chroma signals to generate the missing colors and eliminate the jagged borders.

For anyone wanting to reproduce the test on their real hardware, the GULLIVER.SCC image is available here.

Por hit9918

Prophet (2899)

11-03-2013, 07:58

@sd_snatcher, on the red hat, one can see the jaggies having a problem with the BRIGHTNESS of the pixels.
That is unlogic. Because one thought all problems are in the color info.

It feels like one could fix the red jaggies just by tuning the Y bytes, right?

I played with vpoke (on emu), and if I remember right, things are like this:
Problem 1: with very different colorvalues, same Y value means NOT same brightness.
Problem 2: maximum Y value does NOT mean white.
e.g. with a very saturated color, there is a LIMITED brightness that can be reached in that block.

So when some pixel needs to be above brightness limit of the block,
the converter needs to make a false color/saturation in that block to reach the brightness request.
Because the eye is picky on Y, the false color will be less worse than brightness jaggies.

But, in the magnified picture, I wonder about another thing:
Look at that top left edge of the red hat.
Next to the black pixels, red pixels got some funny spikes that are thiner than a pixel.
Also those jaggies look like converter bug, should be possible to go more soft from black to red.
Would be interesing to see same picture on the emu, to get some hints from "infinite clear" output.

Por WORP3

Paladin (813)

11-03-2013, 09:55

Great work, you made screen 10,11 & 12 usable again

Por wouter_

Champion (450)

11-03-2013, 10:12

sd_snatcher wrote:

[I'm just one of the openMSX developers, other developers may have a different opinion.]
I do like these enhanced graphics. Though because screen 11/12 is not widely used in MSX software, working on this stuff is not very high on my priority list.

Today you say that a similar effect can be achieved by emulating a generic NTSC decoding effect (so not specifically for screen 11/12). Because this is more general applicable, this approach would have my personal preference. There is existing code for such NTSC effects. For example meisei uses (one of the variants of) this effect.

Though, I still don't plan to work on this myself anytime soon. But of course I'd be very happy to help anyone who wants to work on integrating this effect in openMSX. After all openMSX is open source.