CHD Support

[Ad_Enuff]

How does RC deal with CHD's?

Currently, I have several CHD's that have been added in various new versions of MAME, but on using RC they still appear and that the CHD's themselves are described as "Unkown ROM".

Why is this?

[RomCenter]

CHD are not specifically handled by romcenter. It could be handled with other roms if they appear in the datafile, which is not the case actually with datutil. Note that if you add then to the datafile, you need to know their crc, the SHA is not supported.

[Ad_Enuff]

So it begs the question when will this be added to RC? and that Logiqx's DATUTIL needs to be updated to accomodate SHA MD5 and CRC for the verification of CHD's.

This has now become a quite crucial point now. With CLR becoming more powerful all be it very unuser friendly in look, feel and use, I see more people converting to it if the CHD issue isnt resolved.

At the moment they stick out like sore thumbs saying they are incomplete which is rather irritating to say the least.

The way I treat my CHD's is keeping the CHD unzipped in the roms folder under the rom name

eg C:\MAME\roms\kinst

The roms will still remainig the root as follows:-

ie C:\MAME\roms\kinst.zip

[Logiqx]

So it begs the question when will this be added to RC? and that Logiqx's DATUTIL needs to be updated to accomodate SHA MD5 and CRC for the verification of CHD's.

It's not quite as straightforward as that. DatUtil knows the SHA and MD5 of CHDs and outputs the CHDs in CMPro data files (with SHA and MD5). CRCs are not available for CHDs though and thus DatUtil can't output them in the RomCenter data file.

There are obviously manual options available but they aren't very appealing!

Logiqx

[Iron Man]

Maybe its time for RC to move on sometime in the future to support SHA1/MD5 checksums? (with even more CHD's on the way and CD-CHD support on the other hand).

As i am not a coder myself i can only imagine how hard to implement such a feature is... And time is sure another main reason i guess..

Any comments?

[Ad_Enuff]

DatUtil knows the SHA and MD5 of CHDs and outputs the CHDs in CMPro data files (with SHA and MD5).

If that is the case with CLR why cant it be implemented with RC?

RC for me a well designed and easy to use as a rom utility goes. Although I have to admit CLR is far more powerful, its GUI is far from intuitive and easy to understand.

As Iron Man has pointed out, that more CHD's are on teh way not to mention CD based arcade boards that will also use the CHD format.

We simply cant get away from the fact that it needs implementing in the not to distant future.

[Logiqx]

There is no reason why true CHD support can't be added but it is something that requires an extension to the RomCenter data file format and significant development to RomCenter too. Eric is a busy chap in real life so I am not going to request or demand these changes because he simply might not have the time right now.

I think Eric was suggesting that if the CRC of the entire CHD was added into data files (as a normal ROM) then essentially RomCenter could scan it as an an uncompressed ROM. This might work but this is not the same as SHA/MD5 checks of a CHD because SHA/MD5 are for the internal disk image and not the CHD file itself (as the CRC would be). Everyone should think of CHD as an alternative to ZIP but for a specific purpose and as such, the entire file does not need to be read by a ROM manager (just the header record that states the SHA/MD5). Creating checksums for an overal ZIP file is not a valid activity and as such, neither is it for a CHD (remember the SmallCHD tool a while back that demonstrated how the overall CHD could change but uncompressed disk image within it is the same?).

Obviously, supporting a new compressor (i.e. CHD) is not a trivial amount of work for Eric when everything is considered. Finally, SHA and MD5 support could also be added but again they require extensions to the RomCenter format and a fairly significant amount of work on RomCenter itself. Both activities are fairly straightforward and easily within the reach of RomCenter but I think that time is the issue and Eric has a busy life away from emulation.

Logiqx

[RomCenter]

CHD is for sure something to implement. The main problem is how to include it in the gui.

This should be the situation to get after a romcenter fix.
- A chd is not zipped
- It must be in a folder named with the game name
- It must be stored in the same folder as the corresponding rom

Now, what rc must deal with:
- The chd is zipped (unlikely, but possible)
- The chd is not in the rom folder
- The chd is not in a folder
- The chd is missing

The gui:
Databse node:
I think the best solution is to include it with the other roms, maybe with a special icon (red / yellow / green hd icon to design)
Folder node:
green folder : folder name ok, chd ok
Yellow folder : bad folder name and/or bad chd name
red folder : folder missing and zip available.
(RomCenter does not know yet how to handle missing folders.)

The datafile:
I must find a way to tell romcenter that the chd is not a simple rom. I'm not happy with the datafile structure because it is not easy to add informations, but I made it like that for efficiency when loading it. It is less an issue now that everybody have powerfull computer
So, a xml datafile will be my choice, but I still have to find a fast xml parser. Anyway, this is for future. For now, I think I will detect chd from the extension, so, no need to modify the datafile structure.

The sha1:
The chd sha1 can be stored in the datafile. The arcade plugin must be modified to add a sha1 calculator engine. This is trivial. But how to tell the plugin which signature it must calculate? crc or sha1? The best solution is to modify plugin interface and add a 'signature type' parameters to tell which signature I want. Then, we can get one datafile with multiple checksum. Right, this is what we need for chd: crc for common roms, and sha1 for chd.
I keep crc for common rom because crc is extracted from the zip archive, which is extremly fast. The use of md5 or sha1 for all roms means unzipping and running the routine for every rom. Anyway, if md5 and sha1 is implemented in the plugin, this is still possible to build a full sha1 or md5 datafile.
I still need to tell in the datafile which checksum to use with a rom. Once again, the best thing is to modify the datafile and add a signature type column to tell whether the signature is a crc, a sha1, a md5, or whatever you want as long as it match the signature type available in the plugin.
In a first time, I will leave the datafile like it is, and attach sha1 to chd extension (or sha1 to signature longer than 8 char ?)
The datafile modification will be trivial if needed.

So, lot of work, but especially lot of thought before doing it.

I don't know cd-chd. Can I handle them the same way as chd? Some games already use that ?

[RomCenter]

Thanks Logiqx for the info on chd. So I only need to scan the file header and grab the sha1 or md5...

Well, I don't need a sha1 or md5 algorithm then... But if I handle the chd this way, it is very easy to create a fake chd by changing the header. Easier than creating a fake rom with good crc...

Is it the way mame checks the chd validity ?

[Logiqx]

There are two levels of checks:

1) Check that the CHD claims to be the correct image (internal SHA). Fast.
2) Check that the CHD passes the 'chdman -verify' algorithm. Slow.

I would do the first check after every MAME release but only do the second check after acquiring a CHD (then not done again once it is known to be good).

[RomCenter]

Ok. That's clear.
Before I dig into mame code, do you know if a document exist explaining the chd header format ?

[Logiqx]

Code: Select all

/***************************************************************************

	Compressed Hunks of Data header format. All numbers are stored in
	Motorola (big-endian) byte ordering. The header is 76 (V1) or 80 (V2) 
	bytes long.

	V1 header:

	[  0] char   tag[8];		// 'MComprHD'
	[  8] UINT32 length;		// length of header (including tag and length fields)
	[ 12] UINT32 version;		// drive format version
	[ 16] UINT32 flags;			// flags (see below)
	[ 20] UINT32 compression;	// compression type
	[ 24] UINT32 hunksize;		// 512-byte sectors per hunk
	[ 28] UINT32 totalhunks;	// total # of hunks represented
	[ 32] UINT32 cylinders;		// number of cylinders on hard disk
	[ 36] UINT32 heads;			// number of heads on hard disk
	[ 40] UINT32 sectors;		// number of sectors on hard disk
	[ 44] UINT8  md5[16];		// MD5 checksum of raw data
	[ 60] UINT8  parentmd5[16];	// MD5 checksum of parent file
	[ 76] (V1 header length)

	V2 header:

	[  0] char   tag[8];		// 'MComprHD'
	[  8] UINT32 length;		// length of header (including tag and length fields)
	[ 12] UINT32 version;		// drive format version
	[ 16] UINT32 flags;			// flags (see below)
	[ 20] UINT32 compression;	// compression type
	[ 24] UINT32 hunksize;		// seclen-byte sectors per hunk
	[ 28] UINT32 totalhunks;	// total # of hunks represented
	[ 32] UINT32 cylinders;		// number of cylinders on hard disk
	[ 36] UINT32 heads;			// number of heads on hard disk
	[ 40] UINT32 sectors;		// number of sectors on hard disk
	[ 44] UINT8  md5[16];		// MD5 checksum of raw data
	[ 60] UINT8  parentmd5[16];	// MD5 checksum of parent file
	[ 76] UINT32 seclen;		// number of bytes per sector
	[ 80] (V2 header length)

	V3 header:

	[  0] char   tag[8];		// 'MComprHD'
	[  8] UINT32 length;		// length of header (including tag and length fields)
	[ 12] UINT32 version;		// drive format version
	[ 16] UINT32 flags;			// flags (see below)
	[ 20] UINT32 compression;	// compression type
	[ 24] UINT32 totalhunks;	// total # of hunks represented
	[ 28] UINT64 logicalbytes;	// logical size of the data (in bytes)
	[ 36] UINT64 metaoffset;	// offset to the first blob of metadata
	[ 44] UINT8  md5[16];		// MD5 checksum of raw data
	[ 60] UINT8  parentmd5[16];	// MD5 checksum of parent file
	[ 76] UINT32 hunkbytes;		// number of bytes per hunk
	[ 80] UINT8  sha1[20];		// SHA1 checksum of raw data
	[100] UINT8  parentsha1[20];// SHA1 checksum of parent file
	[120] (V3 header length)

	Flags:
		0x00000001 - set if this drive has a parent
		0x00000002 - set if this drive allows writes

***************************************************************************/