Idea/Proposal: ROM Headers for emulation

Prophet (2409)

11-01-2021, 18:05

This topic is intended to start a discussion - not to piss puritans off!

One of the biggest issues these days is to get all ROMs running in an emulator, despite the auto detection mechanism being very good at detecting ROM types it's not always 100% perfect.

Since we have to face reality and understand that MSX software is being bought up by hoarders and collectors who will hang on to it so they can sell them for extremely high prices ("manufactured" scarcity) or store them in the attic in boxes and when they "pass on" everything goes to the garbage dump because it's all old junk in the eyes of most people anyway.

With the rise of FPGA based MSX-es (like the MiSTer) we still don't have a header in our ROM dumps. Now I can hear the puritans scream 'IT'S NOT PART OF THE SOFTWARE!!!' correct - it's part of the hardware!

I've been working for close to 17 years on the openMSX ROM database and the thing is getting HUGE since my life isn't infinite and my goal always has been the preservation of software/hardware on the platform (hence why I have been assisting the openMSX team for close to 18 years now)

Anyway - what I am getting at is the following - and I'm probably going to get yelled at by puritans.

NES ROMs have a header and these headers identify the mapper types. There are various mapper types for the MSX. Konami/KonamiSCC/ASCII8/ASCII16 being the most common ones for megaroms. MSX ROMs do not have headers and the emulator/sofarun has the guess what mapper a ROM has unless it uses a ROM Database like blue and openMSX.

For the FPGA based computers this isn't an option since there is no real way for FPGA to go through a big blob of XML data to pick out the right mapper. And most people don't feel like picking their own mappers.

I was going to propose creating a separate ROM type to indicate if a header is present. I have the information needed to add a header and some other information to a ROM in the database I maintain/update.

Here is what I propose: having a header of 256bytes which contains:
- Indication a header is present (including mapper type version)
- mapper type
- year of release
- name
- publisher

(name/year/publisher can be optional or just 1 blob of text I guess - to keep it small)

To prevent confusion with the clean dumped ROMs we could have a separate extension for those ROMS like MSH or MSX even. I know I'm proposing to create a new pile of ROMs dumps with headers that won't work on real MSX computers with programs like SofaRun or older emulators that don't support headers (that would be most of the inactive emus like blueMSX)

In the long run emulators could benefit from this as it will be clear what mapper type is used.

But in my opinion the future of the MSX is FPGA and Emulation.

Alternatively I could also create a extension to indicate what mapper is used, Like

MS1 trought MSZ the last letter in the extension being a mapper type (so we have 37 different mapper types to indicate). That way we can avoid an added headers inside a ROM.

Maybe some FPGA wizz can step in and propose something else entirely.

The goal is to have a standard with ROM headers

https://wiki.nesdev.com/w/index.php/NES_2.0 << NES 2.0 Header implementation

By Kitrinx

Supporter (6)

11-01-2021, 19:14

While I am not heavily invested in the MSX scene, I have dealt quite a bit with the NES and FPGA scenes, and I have a few technical thoughts on how such a header could be implemented on a technical level. The NES header dates back quite a long ways and some compromises were made for compatibility. With the advantage of starting fresh, I would suggest the following:

-Use a unique file extension for headered roms
-Begin with a four CC (four bytes like "MSX " to easily identify that the rom has a valid header
-Follow with a fixed four bytes of "header version" to make future additions to the concept easier to maintain
-Advance with pairs as follows
-Conclude with an "End of Header" Chunk ID with a size of 0, or a data tag with the size of the data chunk.

The reason I suggest this format is the following:
-It is easy to deal with in low level implementations like flash carts and fpgas
-It is also easy to deal with in high level implementations like emulators
-It can be extended as needed if new information needs to be in the header, or longer names, or perhaps even if someone chooses to add JIS or unicode or any other types of fields in the future. It can even contain patch data if necessary, allowing a rom's data to be kept pristine, while still having whatever advantages the patch provides optional. Any unrecognized fields can be safely skipped. Allowing for a reasonable level of backwards compatibility as new concepts are added.

As Vampier mentioned, the hardware configuration of a cart is itself information. Information that should be available by any situation seeking to recreate the hardware without having to lean on heuristics or outside databases. To separate this data from a rom's software image is to risk losing it forever.

-Kitrinx

By tfh

Prophet (3318)

11-01-2021, 18:47

Sounds like a good idea. You can also consider adding flags for:

* MSX Generation
* Sound Chip (PSG/MSX-Music/MSX-Audio/SCC/etc...)
* 50/60 Hz
* Region Lock
* Kanji / Hangul / Arabic charset

etc... so the emulator/fpgaMSX can set all the right settings.
You should aslo think about a naming convention. And maybe even go extreme: Think about optional things like screenshot (title screen, in-game, cover), etc...

By Grauw

Ascended (10713)

11-01-2021, 19:16

This is very similar to this recent topic:

https://www.msx.org/forum/msx-talk/general-discussion/new-ms...

I think it’s a great idea to have a decentralised metadata format, so it’s really nice that people are thinking about this!

But I would suggest a more extensible format than a 256 byte header. There is much more potentially useful information that could be added than would fit in a fixed 256 bytes schema.

Also, in principle it’s applicable to more than just ROM; also DSK, DMK, CAS, etc. So there are benefits to having a separate manifest file. Use a text-based format for easy editing without special tools. Zip it together, throw in some screenshots if you like, and you got nice ROM packages.

This remains usable for tools that don’t support the format yet, just unzip to get the ROM file and look in the manifest to see what mapper type to use.

So I like the idea but I also think the approach described in the other topic has benefits.

By Kitrinx

Supporter (6)

11-01-2021, 19:17

Grauw wrote:

Also, in principle it’s applicable to more than just ROM, also DSK, DMK, CAS, etc. So there are benefits to having a separate manifest file. Use a text-based format for easy editing without special tools. Zip it together, throw in some screenshots if you like, and you got nice ROM packages.

I would urge not to use plain text or zips, as it makes hardware level implementations difficult.

By NYYRIKKI

Enlighted (6033)

11-01-2021, 19:22

Indeed... I find it very bad that software and hardware has been separated this way as the hardware is useless without software and the software is useless without hardware. I would not mind if things like ROM mirroring information would be stored as well.

Best idea that I've come up so far is that the game would be a zip-file that includes the "GAMENAME.ROM" file as well as some description file... Like "GAMENAME.000". The description file could include the things that you mentioned in a standard, easy to interpret format. It could also describe other files ".001", ".002"... that could include things like screenshots, cover art, compatibility patches, protection removal patches, cheats etc.

The beauty would be that these zip-files would be compatible with what we already have, but emulators and loaders could read these description files and enhance functionality. Since all files have same start of filename these games could be extracted to same directory just as well and also old loaders could be used.

The challenge would be defining the format in a way that it does not become some OOXML type of massive monster for one dedicated environment, but serves real MSX users as well as FPGA implementations and emulator users.

By Grauw

Ascended (10713)

11-01-2021, 19:23

Kitrinx wrote:

I would urge not to use plain text or zips, as it makes hardware level implementations difficult.

I really don’t see why that would be the case. What kind of hardware do you have in mind? SD card access also can’t be done “in hardware”, there must always be a microprocessor involved…

Thinking more specifically about the OCM, you need to load it into memory first, and that loader can configure the mapper based on the manifest.

Also, the zipping is optional, you can also match based on filename.

By Vampier

Prophet (2409)

11-01-2021, 19:23

FPGA for example

By Grauw

Ascended (10713)

11-01-2021, 19:28

Vampier wrote:

FPGA for example

I really don’t see why that would be the case. What kind of hardware do you have in mind? SD card access also can’t be done “in hardware”, there must always be a microprocessor involved…

That Kitrinx was referring to FPGA was implied. But FPGA by itself can’t process ROM files. So it has no benefit from a binary header format.

By ren

Paragon (1932)

11-01-2021, 19:27

Yup, remarkable coincide (in title as well), or ... ??

(Of course I like my idea better , which entails more relevant information on a release, catering to multi-disk games, a nifty icon, SofaRun support (maybe ) and such.)

(And ah, @tfh does react in this topic.. Striking as well?)

By Vampier

Prophet (2409)

11-01-2021, 19:28

FPGA can't do pre-processing as it doesn't work like an emulator where you load a blob of data - dissect it and have nice small chunks. FPGA is byte by byte processing and making decisions tight there.