ROMCENTER OPTION REQUEST: DATUTIL updater

[Ad_Enuff]

Eric,

I have noticed that Logiqx frequently updates the tools on his site fairly frequently.

The problem lies when MAME changes something internally and Logiqx adjusts the tool to comply with the new code.

We then have a problem where by RC doesnt create a compatible MAME dat file to verify any roms.

Wanderer noticed this on MAME 0.102 onwards, with Logiqx's new DATUTIL it fixed this.

The problem then lies when RC hasnt been updated with a new build to incorporate this new EXE.

Therefore, may I suggest we add a piece of code to check Logiqx's website for a new DATUTIL version each time it's run? Allowing the user to be made aware of this and download it?

In version 3.x onwards I hope we can incorporate it better, but for now we just need RC to report back that an upgrade is needed to ensure dat comaptibility.

Thanks.

[Hakkk]

There are more troubles with RomCenter. Some new formats of codes cannot be recognized correctly by RomCenter. ClrMame may also have these new format troubles but it is updated more frequently than RomCenter.

For example, there may be some codes like this:
gamename A,
romname D;
gamename B, clone of A
romname E as same crc32 as D
gamename C, clone of A
romname E

In this instance, two clone sets contain roms with same name but different crc32 (means different contents), one is as same as in main set (with different name, but can be merged), another one should be put independently.

This instance has already happend in MAME0.103, or maybe earlier. In non-merge or split mode it works all OK, but in full merge mode a warning will be shown in RomCenter and it tells you that there are two rom "E" with different crc32s and only the former one will be stored. I think the right way is always to store the one who has different crc32 from the one in the main set, no matter it is the former or the latter one in data file.

Forgive my poor English. If still not clearly, please mention the game "dynduke" and its clone sets. RomCenter performs bad on these games in full merge mode.

[Ad_Enuff]

There are more troubles with RomCenter. Some new formats of codes cannot be recognized correctly by RomCenter. ClrMame may also have these new format troubles but it is updated more frequently than RomCenter.

For example, there may be some codes like this:
gamename A,
romname D;
gamename B, clone of A
romname E as same crc32 as D
gamename C, clone of A
romname E

Is this still true when using Logiqx's latest DATULTIL version 2.25?

[Hakkk]

There are more troubles with RomCenter. Some new formats of codes cannot be recognized correctly by RomCenter. ClrMame may also have these new format troubles but it is updated more frequently than RomCenter.

For example, there may be some codes like this:
gamename A,
romname D;
gamename B, clone of A
romname E as same crc32 as D
gamename C, clone of A
romname E

Is this still true when using Logiqx's latest DATULTIL version 2.25?

Yes, still. And the data released on Logiqx's site also has this problem.
I guess it is caused by some defects of RC data's format, or defects of the way the RC understands the data.

[Hakkk]

For a reference:


A few extracts from MAME0.103's listxml file:

<game name="dynduke" sourcefile="dynduke.c">
<rom name="dd9.jk5" size="16384" crc="f2bc9af4" sha1="9092ebe9dced243c3a7f63198a1180143dd42cea" region="gfx1" dispose="yes" offset="0"/>

<game name="dyndukj" sourcefile="dynduke.c" cloneof="dynduke" romof="dynduke">
<rom name="9.5k" merge="dd9.jk5" size="16384" crc="f2bc9af4" sha1="9092ebe9dced243c3a7f63198a1180143dd42cea" region="gfx1" dispose="yes" offset="0"/>

<game name="dbldyn" sourcefile="dynduke.c" cloneof="dynduke" romof="dynduke">
<rom name="9.5k" size="65536" crc="16bec703" sha1="c3b5311dee0e4750bc4dbce10deb46669179ac6f" region="gfx1" dispose="yes" offset="4000"/>

RC seems cannot understand this kind of merge, or RCdata format cannot describe it, I don't know.

[Ad_Enuff]

For a reference:


A few extracts from MAME0.103's listxml file:

<game name="dynduke" sourcefile="dynduke.c">
<rom name="dd9.jk5" size="16384" crc="f2bc9af4" sha1="9092ebe9dced243c3a7f63198a1180143dd42cea" region="gfx1" dispose="yes" offset="0"/>

<game name="dyndukj" sourcefile="dynduke.c" cloneof="dynduke" romof="dynduke">
<rom name="9.5k" merge="dd9.jk5" size="16384" crc="f2bc9af4" sha1="9092ebe9dced243c3a7f63198a1180143dd42cea" region="gfx1" dispose="yes" offset="0"/>

<game name="dbldyn" sourcefile="dynduke.c" cloneof="dynduke" romof="dynduke">
<rom name="9.5k" size="65536" crc="16bec703" sha1="c3b5311dee0e4750bc4dbce10deb46669179ac6f" region="gfx1" dispose="yes" offset="4000"/>

RC seems cannot understand this kind of merge, or RCdata format cannot describe it, I don't know.

To me this sounds and looks like an Internal MAME CRC SHA1 problem which is causing rom verifying checks a bit of a headache.

clr or rc cannot rectify this, even Logiqx will have his work cut out to fix something like this.

But my main point still remains, we need Romcenter to check for a NEW version of DATUTIL. If this type of error can be dealt with by a new DATUTIL then this is essential for RC to essentially work.

[RomCenter]

Hi

These cases are not new, and romcenter already handle them thru the warning message.
A choice has been made to keep the first file. RomCenter doesn't inspect further to see if another rom with the same crc already exist in the set.
I will add this in the new version.

But keep in mind that this state is an error in mame. When you run dynduke in full merge mode, mame will check the rom 9.5k with one crc. Which one ? dyndukj ? dbldyn ?

Ad_Enuff, I know this is a problem, and I think the correct solution is yours. I still have to figure a simple way to look after datutil release.

I'm still working on RomCenter 3 on my spare time, and I will asap publish a pre-alpha for a first test. But this will be far from complete, and won't be usable for real rom sorting.

[Hakkk]

Thanks for you explanation and announcement of progress of 3.x

[Logiqx]

Eric, I could generate a delimited file for the web that lists my tools, version numbers and other useful info.

e.g. DatUtil|v2.25|Updated|06/12/2005|dutil225.zip|293|

All of this information is currently in XML that generates the web page.

I could also invent another value that specifies the minimum version number required for RomCenter (to save everyone downloading some of the versions unneccesarily - maybe only 1 version in 10 is needed by RC).

Logiqx

[Ad_Enuff]

Eric, I could generate a delimited file for the web that lists my tools, version numbers and other useful info.

e.g. DatUtil|v2.25|Updated|06/12/2005|dutil225.zip|293|

All of this information is currently in XML that generates the web page.

I could also invent another value that specifies the minimum version number required for RomCenter (to save everyone downloading some of the versions unneccesarily - maybe only 1 version in 10 is needed by RC).

Logiqx,

This woudl be a great help! One of the problems RC faces is the DATUTIL updating process. If this can be acheived then this is great news for RC and also will remove the need to update RC everytime DATUTIL gets updated.

[Wanderer]

Well, this is an issue which would be great if it was solved.

A few suggestions

1. Maybe it's not my business but IMHO it would be better if an XML file was used instead of a delimeted one. XML is a standard so anyone would be able to use such a file.

2. From what i've understood, RC would check online for a new version of datutil and automatically download and "install" it. If this is the case then there are a few things to keep in mind. If i.e. a user is checking mame v101 and already has installed the required version of datutil for this mame version, he may not wish to wait for RC to download any newer version of datutil which may have been released for a later version of mame. For that reason, perhaps the delimeted / XML file should also contain the "minimum mame version that requires the new datutil".

[Logiqx]

I considered XML but wasn't sure if Eric would prefer parsing XML or delimited data. If Eric is happy to receive XML then that is fine by me but if he hasn't loaded XML data before, it will be more work for him.

The second option is not worthwhile in my opinion. The number of people who only maintain old MAME ROM collections and never intend to use a newer version must tiny. They can suffer the 300KB download when a DatUtil update is required by RomCenter for the newest MAME.

Don't forget that historically, I have only released about 1 or 2 updates a year that have been required by RomCenter. Maintaining a list of MAME version vs. DatUtil versions is work for me that I don't think is worthwhile. All of my tools are backwards compatible with every version of MAME since I started out so I don't see any need to retain a history of version compatibilities.

[Logiqx]

Eric: Let me know your preference (delimited text or XML) and I will generate the file for you.

[Wanderer]

I don't see any need to retain a history of version compatibilities.

My main concern for making this suggestion is RC's behavior in case your site is down or if an internet connection is not available. I wouldn't like to see a delay when RC starts up or when loading a mame data file, just because everytime it will try to connect to the internet to check for a new version of datutil.

No, no, i wasn't talking about "a history of version compatibilities". If i understood correctly, there will be one delimeted file only, containing info for the latest version of datutil. When mame v.102 was released, you had to release a new datutil to support it. The delimeted file would contain something like "v0.102" denoting that the current version of datutil is "mandatory" for mame v0.102 or later. It would then be up to Eric to decide how to handle this info (display a message in RC saying "Please download the new datutil, its mandatory for this version of mame" or perhaps in a later version, download it automatically).

My 2 c.

[Logiqx]

The internet connectivity issue is something that Eric would need to address but as you say, lack of internet connectivity should not stop RomCenter from carrying on as normal.

I understand what you were trying to say about MAME versions now. That is a good idea and can also be extended to other emulators. If Eric ever wants to use DatUtil for emulators like M1 (once it's XML based configuration goes public), then I can include information about those too if my file is in XML format.

[RomCenter]

The best would be to have the mame version, like wanderer said.

I could get the mame version in the xml info, and compare it to the previous datutil mame version (keeped from the last download). If it is different, I will ask for an update.

This way, datutil could have several releases not related to romcenter without triggering a update in romcenter.

In fact, we can see this datutil mame version like a 'datutil romcenter version'.

On the other hand, Datutil has not a lot of updates in a year, and I could simply stick to the datutil version, ask user to update, add an option to disable the datutil update and a manual update action.

I would prefer the second solution, as it involves less work to maintain.

Logiqx, a xml file is ok.

I will add this feature in romcenter 3, as I do not want to spend more time on the rc 2.xx line instead of working on rc 3.0.
If it gets really annoying, I will update the rc2.70 package with the new datutil.

[RomCenter]

The internet connectivity issue is something that Eric would need to address but as you say, lack of internet connectivity should not stop RomCenter from carrying on as normal.

I understand what you were trying to say about MAME versions now. That is a good idea and can also be extended to other emulators. If Eric ever wants to use DatUtil for emulators like M1 (once it's XML based configuration goes public), then I can include information about those too if my file is in XML format.

Oh, if it makes sense to add news tags, it's ok for me. And yes, other emulators datafile generators is possible (If datutil supports it ).

[Logiqx]

Ok, that is decided then. I will create an XML file when I find a moment.

Any emulators that DatUtil fully supports, we can look forward to in RomCenter 3.

[RomCenter]

eeeeeeeeeehhhhhhhh

yes, eventually.