Nandemo PiGa the binary <-> wav converter for tapes

Page 4/6
1 | 2 | 3 | | 5 | 6

By Manuel

Ascended (15951)

Manuel's picture

15-12-2019, 22:35

enribar wrote:

Me and Takamichi have sampled in wave a couple of rare tapes, but expecially one of them doesn't load on openMSX.

Can you help us to investigate by providing these digital recordings?

By pgimeno

Resident (46)

pgimeno's picture

15-12-2019, 23:28

mcolom wrote:

Some time ago I analyzed the circuit of the Sony HB-101P/201P (http://www.msxarchive.nl/pub/msx/mirrors/hanso/service_manua...) which is similar and also with the 311 comparator.

That one is exactly like the other Sony I checked. The diodes are obviously to limit the input voltage to ±0.7v, and the feedback loop of the comparator is also similar to the one in the Philips and the Canon, but more convoluted.

mcolom wrote:

It's exactly the "open collector output" in page 3 of the datasheet: pins 4 and 1 to ground, 8 to Vcc, and 7 connected to 8 via R7 (RL in the datasheet).

I was told on IRC that the MSX had a Schmitt trigger, and indeed the configuration in the Philips circuit of R8 and R9 seem like the inverting Schmitt trigger. So I presume it's used for hysteresis and to saturate the signal. The emulator doesn't have hysteresis, because according to the developer, it didn't improve anything.

Today I've set up a test with a VG-8020/40. The real MSX wasn't able to load directly from the computer the problematic WAVs, therefore the emulator doesn't seem to be at fault for not accepting them. Hysteresis doesn't seem to be a factor; however, interpolation probably is, because resampling the WAVs at a higher frequency made the emulator suddenly accept them, even better than the real MSX (they loaded fully in the emulator, while in the real machine they load but hang at some point). That convinced wouterv of implementing interpolation into the emulator, so that's hopefully going to solve quite a few problems.

By Manuel

Ascended (15951)

Manuel's picture

16-12-2019, 23:26

It solved Ingrid's Back so far Smile

By mcolom

Resident (57)

mcolom's picture

17-12-2019, 00:28

The circuit doesn't seem to have hysteresis. And actually, when you analyze the circuit and you compare to what openMSX is doing, you see that there isn't much difference. Well, perhaps the analog filters cut-offs, that could be different at each MSX.
The fact that resampling the signal to a higher resolution makes the emulator load the signal is surprising. I don't see why it should help, but it's good news! Smile To figure out why, I think the best would be to implement the DC filter in openMSX and to compare the signals. I resampled with Audacity and high-pass filtered with a 800 Hz threshold and actually the zero-crossings looked the same (well, both versions look the same, only that one has a better resolution). But perhaps with openMSX's filter they're different.
On the other hand, I'm doing some experiments with a Laplacian filter (a LoG) to detect edges without changing much the shape of the signal and being robust to offset changes. We'll see if that's useful. I'm on it... no promises!

By pgimeno

Resident (46)

pgimeno's picture

17-12-2019, 01:22

mcolom wrote:

The fact that resampling the signal to a higher resolution makes the emulator load the signal is surprising. I don't see why it should help, but it's good news! Smile

If one sample can make a difference, having sub-sample resolution helps. At least that's my explanation. Note that the MSX samples the input at a rate faster than the 44.1KHz of the WAV.

By mcolom

Resident (57)

mcolom's picture

17-12-2019, 18:09

pgimeno wrote:

Note that the MSX samples the input at a rate faster than the 44.1KHz of the WAV.

I'm starting to see your point. The fact is that, in a real machine, the filtered signal is analog and therefore sampling it at a rate more than 44.1 KHz is more than enough (actually, it's an oversampling way over the Nyquist frequency, which is OK).
But in the emulator it's a digital signal which is digitally filtered. So, indeed it's not the same.
It could be that the simple digital high-pass filter that is used is not numerically accurate and the fact that the input is already digital doesn't help. That would explain why a larger sampling rate in the inputs makes a difference.
If we can confirm this, we could think of a better filter.

By pgimeno

Resident (46)

pgimeno's picture

19-12-2019, 20:51

mcolom wrote:

It could be that the simple digital high-pass filter that is used is not numerically accurate and the fact that the input is already digital doesn't help.

I don't think numerical accuracy of the filter is a problem here. The fact that it is filtered with a HP filter, however, can be a factor.

I don't think that there's one be-all end-all filter that solves all, not even most, problems with problematic digital recordings of tape programs. At least not a filter that can be implemented in openMSX. The possible sources of distortion during recording introduce modifications in the signal that need different strategies, depending on the specific source. For example, look at how a fragment of the Gnome Ranger tape recording looks like, before and after a single-pole 800 Hz HP filter like the one internally performed by the emulator (so I presume the signal is basically the same as the emulated machine sees):

As you can see in the last image, this has negatively affected the gaps between some of the pulses, most notably at the changes between high-frequency pulses and low-frequency ones. This gap is crucial for the BIOS algorithm.

(In case you wonder, no, it doesn't work without the filter either, I still don't know the reason as the signal is not too bad).

Problem is, if you have a single sine wave of a certain frequency and take its FFT, you will see a clean peak at a certain frequency; if it's the double of that frequency, you'll see a clean peak too. But when you intercalate pulses of both frequnecies in the way MSX does, you will have MANY more frequencies involved, all of which contribute to the changes between the high-frequency pulse and the low-frequency pulse. Having any kind of frequency filters is therefore not going to help a great deal, as they will remove frequencies that affect these gaps.

Now, Gnome Ranger was affected by both HP and LP filtering (the presence of the latter is visible because otherwise the corners of the square wave would not be so round). To undo them, the correct thing to do is not to apply more frequency filters, but some kind of deconvolution. And it must be specific to the kind of distortion present, which is not always the same. In some cases it will be the tape head being unadjusted, which lowers the frequencies, so it will need a LP deconvolution. In some other cases it will be asymmetry in the signal (as in the case of the CD Sequential wavs), which isn't so easily fixable (it's not just DC offset).

To sum up, I don't think that applying filters will solve anything in the general case.

By mcolom

Resident (57)

mcolom's picture

19-12-2019, 22:26

pgimeno wrote:

I don't think numerical accuracy of the filter is a problem here. The fact that it is filtered with a HP filter, however, can be a factor.

To me the problem is that sampling a continuous signal (the tape signal after the analog filter) and applying a digital filter to an already digital signal is completely different. When you increase the sampling rate of the input signal you're not adding new information (actually, it's just an oversampling), but allows to load the game with the emulator. It means that the filter is not stable enough. And it explains why the a real MSX would load that WAV without much problem. I don't know if the WAV shoud show is the one that loaded after increasing the sampling rate, but I'm talking about that one.

pgimeno wrote:

As you can see in the last image, this has negatively affected the gaps between some of the pulses, most notably at the changes between high-frequency pulses and low-frequency ones. This gap is crucial for the BIOS algorithm.
(In case you wonder, no, it doesn't work without the filter either, I still don't know the reason as the signal is not too bad).

The example you show is a case where the BIOS should be able to perfectly to load data.
The BIOS takes care of computing the width of the short pulses (and it assumes that the long ones take twice). It needs to be robust to cassette players with different speeds and jitters. A case when it could fail is when it computes a pulse width in the header and then it's really different later. Or if there's a slow width increase that makes the BIOS fail to recognize a short or long width before a header re-synchronizes again.

In the example you show there isn't a significant change in the width of the pulses. So the reason must be other. Different from the pulses of the header, perhaps? As you say, it seems really clean.

pgimeno wrote:

Problem is, if you have a single sine wave of a certain frequency and take its FFT, you will see a clean peak at a certain frequency; if it's the double of that frequency, you'll see a clean peak too. But when you intercalate pulses of both frequnecies in the way MSX does, you will have MANY more frequencies involved, all of which contribute to the changes between the high-frequency pulse and the low-frequency pulse. Having any kind of frequency filters is therefore not going to help a great deal, as they will remove frequencies that affect these gaps.

Sure, the Fourier transform of a step function is sinc(pi f), so as you say, there's a lot (infinite, or as many frequencies as allowed in the case of a digital signal) of frequencies involved. A low-pass filter is needed before digitizing the signal (say, when a real MSX saves it) to avoid the aliasing effect. Then, when reading it it applies the same filter (because any signal in the removed band is clearly noise or a parasite, given that the signal was previously filtered).

And about the high-pass filter, it doesn't seem to have a significant effect, at least in the example you show. Sure, it changes the shape of the signal (it's normal, it can no longer be square because you removed frequencies and you needed a sinc function with all of them for a perfect reconstruction), but it seems OK. And it removed parasite frequencies.

pgimeno wrote:

Now, Gnome Ranger was affected by both HP and LP filtering (the presence of the latter is visible because otherwise the corners of the square wave would not be so round). To undo them, the correct thing to do is not to apply more frequency filters, but some kind of deconvolution. And it must be specific to the kind of distortion present, which is not always the same.

Even if we estimated the convolution kernel (that we don't have), it won't be invertible. Most of the distortion of the signal is because some frequencies have been attenuated by a filter. Thus, the inverse filter would simply multiply the noise by very large values. In short, it would be quite unstable.
However, indeed you can estimate a convolution kernel in the range of frequencies of the filtered image, and yes... you would correct some (little) distortions. A deconvolution (say, Wiener) is fast, but I don't think it would help much with this particular problem. I don't know, I have that impression.

pgimeno wrote:

To sum up, I don't think that applying filters will solve anything in the general case.

And I think that in fact it's the digital filter in openMSX what makes some tapes not load but in a real machine. I think the digital high-pass filter it could be unstable. And to me the key experiment is the one you did: increasing the sampling rate made the emulator load the game. But that doesn't add new information. So, it's a problem in the digital filter.

Well, that's my guess, but actually I must say I don't know the reason for sure. I have to look at it in more detail.
It's fun to figure out what is going on! :)

By Manuel

Ascended (15951)

Manuel's picture

19-12-2019, 23:07

Keep in mind: the real MSX couldn't load that signal either. And only one game (Gnome Ranger) was able to be loaded in openMSX by increasing the sample rate with an interpolation.

By mcolom

Resident (57)

mcolom's picture

19-12-2019, 23:26

Yes, they're different cases.
The one that improves after resampling indicates that a better filter is needed.
For the others, who knows. They need of a more complex processing, assuming that their distortion can be fixed. If that it's possible, one could think of adding it to openMSX (but I'm not there yet, of course). The think is right now openMSX implements a simple filter that is close to what the real MSXs do. Changing that could improve the load of some tapes, but perhaps it'd go against the "concept of emulation". We could end up with tapes that would load in the emulator whereas no real MSX can read them. But we'll see... we are (or I am) far from that yet.

Page 4/6
1 | 2 | 3 | | 5 | 6