-------------------------------------------------------------------------------
                         DatUtil v2.1 - 04/07/2004

                  Written by Logiqx (http://www.logiqx.com)
-------------------------------------------------------------------------------

Introduction
------------

DatUtil was created to aid in the creation of dat files for Rom Managers such
as ClrMamePro and RomCenter (Roman Scherzer / Eric Bole-Feysot). It can convert
between formats, extract individual games (and optionally their clones) and
cleanse dats. It supports a variety of formats...

Load and Save:

  - MAME ListInfo (ClrMamePro, MAME, MESS, RAINE, Shark)
  - RomCenter 2.50

Load only:

  - RomCenter 1.90
  - RomCenter 2.00
  - MAME ListXML (xml2info.exe must be in your path for this feature to work)
  - Nebula Drivers
  - ClrMame Log
  - Folder Scan (subdirectories and zips)

Save only:

  - Title List (for listing game titles - used by CPS-2 readme and RAINE docs)

Adding support for additional formats is really easy too. If you do add support
for a new format please let me know and send me a copy of the code!


Distribution
------------

The home page for DatUtil is http://www.logiqx.com/

Links can also be found at the following two sites:

ClrMamePro Home Page                    RomCenter Home Page        
http://www.mameworld.net/clrmame        http://www.romcenter.com/


Converting dats
---------------

It is possible to convert dats between ClrMamePro and RomCenter with ease. The
most simple way to convert from one format to the other is like this:

  datutil <datfile>

This will convert a ClrMamePro dat (listinfo) to a RomCenter datafile or
a RomCenter datafile to a ClrMamePro dat. The new dat is called 'datutil.dat'.
If you want to specify a different output filename, use the '-o' option.

  e.g. datutil -o clrmame.dat romcenter.dat

This would convert romcenter.dat, saving the new file as clrmame.dat.
There is also a '-a' option to append to an existing file. This is especially
useful when converting/extracting individual games (explained later).

If you wish to specify the output format directly (for example, if you want
DatUtil to clean up the dat and resave in the same format) you can use the '-f'
option. Valid formats are 'listinfo', 'romcenter2' or 'titlelist'.

  e.g. datutil -f titlelist clrmame.dat

If you just wish to convert/extract an individual game of the original dat
you can use the -g option (and optionally -c for its clones as well).

  e.g. datutil -g sf2 -c romcenter.dat

This would extract sf2 and all of its clones from romcenter.dat, saving them
under the name datutil.dat (of course, the '-o', '-a' and '-f' options could
be used in addition). This feature is useful for creating dats for an emulator
that uses ROM sets like those in MAME (extract from the output of listinfo).
When using -g and -c together you can specify a system such as neogeo or cpzn1
to extract all games on that system.

If you wish to exclude clones from the newly saved dat, you can use the '-r'
(remove clones) option.

People converting dats for home computers will often wonder what the 'funny
size' warnings are. This is intended for arcade dat creators so you can switch
off the size checking using the '-z' option.

To convert all game names, ROM names and ROM CRCs to lower case, use the '-l'
option.

By default, the 'improve merging' feature will ensure that CRCs match in
the parent and its clones where they were previously a mixture of 00000000s and
genuine CRCs. After the CRCs have been unified it will then specify that the
ROMs can be merged and flag the 00000000 to xxxxxxxx conversions as bad dumps.
This option will also remove bad parent references for cloneof/romof/sampleof
and ensure that there are no clones of clones in the data file. Duplicate ROMs,
disks and samples are also removed by default. In this instance, the
definition of 'duplicate' is where there are two or more ROMs/disks/samples of
the same name in the same game.

The final option is '-s' for sorting the games before saving. This actually
sorts games by their parent name first (n.b. name, not description) and then
by the game name (n.b. name, not description).


Additional notes on conversion
------------------------------

The RomCenter 2 format doesn't support all of the potential information that is
in the MAME listinfo output. DatUtil will warn you if any information is lost
during conversion process though (e.g. SHA1 checksums, disks and samples).

When editing dats I find it easiest to work on a ClrMame dat (it's easier to
read) then convert it to RomCenter afterwards. DatUtil will create a RomCenter
dat just as good as you could have done by hand but less prone to errors.  ;-)

You will have noticed that ROMCenter dats and CMPro dats have a header section.
To specify values for the fields in the header, use the '-A', '-V', '-C', '-R'
and '-F' options (author, version, comment, refname, fullname/emu version).
'-M' and '-Z' are used to specify the type of merging and zipping required.
Note that these options are all upper case (you must use the correct case).

The log file generated is by default a summary. If you want detailed warning
information and exact details of the cleansing performed, use the '-v' option
for a verbose log file.


Quick reference
---------------

Defaults (behavior within DatLib that can not be switched off):

  -   fix merging (used to be the -m option)    * Improved in DatUtil v2.0 *
  -   remove duplicate ROMs/disks/samples       * New in DatUtil v2.0 *

Cleansing:

  -l  lower case game names and ROM names
  -s  sort games by parent (sorted by parent name then game name)

Game selection:

  -g  game
  -c  game and clones (for use with the -g option)
  -r  remove clones

Header text:

  -A  author
  -V  version
  -C  comment/category
  -R  ref name
  -F  full name (i.e. description)
  -M  merging (none, split or full)
  -Z  zipping (zip or unzip)

Saving:

  -f  output format
  -a  append to file
  -o  output to file

Information:

  -z  Do not show 'funny size' messages
  -v  Verbose logging                      * New in DatUtil v2.0 *
  -d  Show debug messages                  * New in DatUtil v2.0 *


Dat features supported
----------------------

DatUtil has its own internal representation for datfiles but in MAME Listinfo
terms it supports:

  Games      game, resource, machine, name, description, year, manufacturer,
             rebuildto, cloneof, romof, sampleof
  ROMs       rom, name, merge, size, crc, md5, sha1, region, nodump, baddump
  Disks      disk, name, md5, sha1, region
  Samples    sample, name

Note: DatUtil does not support the 'archive' information in Raine's gameinfo
or Shark's listinfo files as no ROM manager supports this property.


Adding support for new dat formats
----------------------------------

To add support for new formats, take a look at the documentation for DatLib.
It is really easy to add support for a new format but if you can't code, drop
me an e-mail and I might do it for you (if I think it is a useful addition).


Other uses?
-----------

DatUtil can scan a directory and create a datfile for the ZIPs/subdirectories
within it. Simply specify the directory name to scan (instead of a datfile) and
DatUtil will generate the dat for you. Use the -x option to calculate SHA1 and
MD5 values for zipped ROMs.

For comparing two data files of any format supported by DatUtil, use MAMEDiff.
You can find the latest version of MAMEDiff on my website.


Thanks
------

Thanks go to Eric Bole-Feysot and Pi for beta testing DatUtil.


Source
------

As with all of my tools, I have included the source code so you can make
your own modifications but please do not distribute those versions. If
you make any useful changes then please send them to me so that I can
include them in future releases.

To compile this tool, you will need to get DatLib from http://www.logiqx.com/


Bug Reports
-----------

If you find any bugs please let me know about them and provide details such as
OS version, command you are using, dat you are running it against etc.

Feel free to send me any suggestions that you have also.


History
-------

04/07/2004  *** v2.1 ***
            - Added a message to say that the -m option is not required now.
            - Uses DatLib v1.1:
            - ClrMamePro/RomCenter header is now ignored when using OPTION_GAME.
            - Removed a warning when you try using 'datutil -g <resource> -c'.
            - Added debug information to identify spurious warnings like above.
            - Added the -x option to calculate SHA1/MD5 when scanning ZIPs.
            - n.b. SHA1/MD5 used to be calculated by default in DatUtil v2.0.

02/07/2004  *** v2.0 ***
            Wow, I never thought I'd get around to this rewrite but here it is!

-------------------------------------------------------------------------------

Uses DatLib - The new library for handling dats of many formats
===========

- A generic dat structure with various loading and cleansing routines.

- Far better memory handling. No longer uses huge blocks of fixed size memory.

  Info: DatUtil used to allocate almost 45MB for a dat when it was loaded.
        This was without MD5/SHA1 support and for a limited number of games!
        DatLib now allocates only what it really requires (which is usually
        about twice the size of the dat file).

- Multi-phase processing (pre-parse, parse, cleanse, save):

  - Pre-parser translates dat into an intermediate dat format.
  - Parser interprets the intermediate format to populate the dat structure.
  - Cleanser does things like 'lowercase', 'remove dups', 'fix merging', etc.
  - Saving outputs the cleansed dat structure in the required output format.

  Note: The main difference is the addition of the pre-parser (which removes
        common operations from the different dat parsers) and the separation
        of the cleansing operations from the loader (they used to be embedded
        in the various parsers, particularly painful for RomCenter format!).

- No limit on the number of games, ROMs, disks, samples etc.

- No limit on the length of file names, game titles, manufacturer names etc.

- Improved size and CRC handling (now numeric, not text based).

- Supports 'rebuildto' information for games.

- Supports alternative checksums (MD5 and SHA1).
  N.B. The directory scanner also generates MD5 and SHA1 checksums for files.

- Supports 'region' information of ROMs.

- Improved driver system for developers adding new dat formats.

- Fully supports the latest RomCenter dat format (2.50).

- Improved RomCenter parser is not case sensitive like the old one was.

- All cleansing features can now be used with all supported formats.

- Improved 'fix merging'. Now operates across clones, not just parent/clone.
  Also fixes bad parent references and prevents clones of clones.
  The feature is run by default when a data file is loaded into memory.
  A summary of fixes is shown in datutil.log, use -v option for full details.

- Duplicate ROMs are now removed when the data file is loaded into memory.

- Added debugging information (to help me track down problems).

- Removed alternate ROM support. It was only used by Shark and made data file
  comparisons unnecessarily complicated!

- Removed code that fixed the bad parent 'pc' in MESS dats (the default
  'fix merging' code does the same job now but without being hard coded).

- ClrMamePro log parser now supports disk images and alternative checksums.

- Nebula parser is now much better. With the default 'fix merging' behaviour,
  the data file that is generated should be almost usable now!

- Improved baddump and nodump handling.
  Also supports the syntax of MAME v0.68 (where the word 'flags' was missing).

- Improved logging. Added a verbose (-v) option for even more detail!

- You are given a warning if you try to convert a RomCenter data file where
  the plugin is not 'arcade.dll'. It is likely that such a conversion will
  not work correctly in ClrMamePro.

- The -g and -c options can now be used on resources. You can therefore
  extract games for neogeo, zn1, etc. (e.g. 'datutil -g cpzn1 -c mame.dat').

- Removed the dat comparison facility. Use MAMEDiff instead!

- Added load support for MAME ListXML format (uses the xml2info tool).

- Added a warning summary to datutil.log.
  Warnings are for missing details, funny ROM sizes and CRC conflicts.
  For full details of the warnings, use the DatUtil -v (verbose) option.

- Added details of all information that is lost when saving in a different
  format (e.g. Years and Manufacturers are lost when using RomCenter format).


Unsupported formats - Not included in the rewrite as they are of little use!
===================

Load and Save:

  - Calice DRV (drivers)

Load only:

  - Callus gamelist
  - CPSE data (drivers)
  - Mimic HWC (drivers)
  - System16 GCS (drivers)
  - Sparcade RAT (drivers)
  - Retrocade Romlist
  - JAS romlist
  - JFF text

-------------------------------------------------------------------------------

v1.x History
------------

unreleased  *** v1.33 ***
            - Added support for comments in Nebula drivers.

26/02/2004  *** v1.32 ***
            - Finally added disk support for MAME!

29/01/2004  *** v1.31 ***
            - When saving data files in listinfo format, 'crc 00000000' is now
              written out as 'flags nodump'.

24/01/2004  *** v1.30 ***
            - Increased the maximum number of games from 5000 to 6000.
            - Removed support for the old RomCenter 1 format...
              The old format is inadequate for modern day MAME releases.
            - Removed knowledge of the MAME resource names. RomCenter 2 data
              files must therefore have a [RESOURCES] section from now on.

20/01/2004  *** v1.29 ***
            - Minor adjustment to the routine that decides if a ROM lives in a
              resource. This fixes the handling of shootgal/sg-01-0.

31/12/2003  *** v1.28 ***
            - Added knowledge of the atpsx resource.

28/12/2003  *** v1.27 ***
            - Added knowledge of the two new resources added in MAME v0.78.

14/09/2003  *** v1.26 ***
            - Changed names of Neo-Geo BIOS ROMs in the Nebula driver reader.

17/08/2003  *** v1.25 ***
            - Added knowledge of the seven new resources added in MAME v0.72.

07/06/2003  *** v1.24 ***
            - Added support for '[RESOURCES]' in RomCenter2 data files.

19/05/2003  *** v1.23 ***
            - Added a fix for the 'nodump' syntax of MAME v0.68.

06/04/2003  *** v1.22 ***
            - Fixed CMPro log scanner to support the syntax 'bad dump'.
            - Added knowledge of konamigx resource. 

18/01/2003  *** v1.21 ***
            - Removed 'cloneof neogeo' from 'neocd' in Nebula driver.
            - Added knowledge of skns and stvbios resources.

20/05/2002  *** v1.20 ***
            - New resource ROMs in MAME listinfo are automatically detected.
            - ClrMamePro log and Nebula driver loaders now use the STORE_ROM
              macro (this makes -g work correctly, amongst other things).
            - Improved -m (fix merging) behaviour for resource ROMs.
            - Nebula drivers now output the correct 'romof' for Neo-Geo clones.

22/04/2002  *** v1.19 ***
            - Added support for the PGM resource.

20/04/2002  *** v1.18 ***
            - Increased maximum number of samples to 10,000 (for PinMAME).
            - Exceeding the sample limit now results in a graceful exit.

04/04/2002  *** v1.17 ***
            - Made 'fix-merging' option check the CRC as well as the name.
            - Fix merging option warns of CRC conflicts it finds. Useful test!
            - Strips trailing spaces from Nebula ROM names (for convenience).
            - Executable is called 'datutil.exe' again.

28/03/2002  *** v1.16 ***
            - Added Nebula driver support (useful for data file validation).
            - Supports the new '0x' syntax of CMPro logs.
            - ZIP scanner now removes '.Zip' from filenames (case sensitive).
            - Increased maximum line length to 16384 bytes (for MAME history).
            - Increased maximum number of ROMs to 75000.
            - Exceeding the game or ROM limits now results in a graceful exit.

10/02/2002  *** v1.15 ***
            - Enhanced Calice support to cope with Neo-Geo sound ROMs.
            - Modified source to compile easily with CygWin (and FreeBSD?).

11/11/2001  *** v1.14 ***
            - Added knowledge of ng-lo.rom.

27/09/2001  *** v1.13 ***
            - Supports CPSE driver file.
            - Added 'ignore funny size' option (-z) for non-arcade dats.
            - Added 'make lower case' option (-l). Affects names and CRCs.
            - Puts "s around empty CMPro header items.
            - Fixed directory/zip scanner for ROM names containing spaces.
            - Fixed zip scanner to strip uppercase '.ZIP' from game names.
            - Improved the 'funny size' warning relating to RomCenter 2 dats.

28/08/2001  *** v1.12 ***
            - Supports new ROM loading introduced in Calice v0.3.8 to v0.4.4

28/06/2001  *** v1.11 ***
            - Added some warnings for bad quality RomCenter dats (wrong version
              number in header and missing ROM sizes)

23/06/2001  *** v1.10 ***
            - Fixed detection of RAINE32 dat format.
            - Improved ZIP scanner (not case sensitive when searching for zips).

08/06/2001  *** v1.9 ***
            - Added directory scanning feature (reads ZIPs and subdirectories).
            - Calice loader now uses the main driver code, not 'verifyroms'.
            - RomCenter 2 loader fixes missing 'romof' details automatically.

18/05/2001  *** v1.8 ***
            - Fixed a minor bug in the '-m' feature (bug introduced in v1.7).
            - Now compiled with MinGW (no long filename problems on NT now).

26/04/2001  *** v1.7 ***
            - Added handling of decocass resource in MAME dats.
            - Added load/save support for CMPro element 'clrmamepro' and
              ROMCenter headers.
            - Added preliminary support for Calice Driver files.
            - Added load support for ClrMamePro log files (i.e. to convert into
              a dat format).
            - Further improved the '-m' feature ('fix merging' option).
            - RomCenter2 save routine strips redundant 'merge' information from
              non-clones.
            - DJGPP compilation now uses -Wall option.
            - Can now be compiled with Microsoft Visual C++.

21/02/2001  *** v1.6 ***
            - Added option to improve MAME's merging information ('-m').

06/02/2001  *** v1.5 ***
            - Some handling of the playch10 and cvs resources in MAME dats.
            - RomCenter 2 is now the default save format when converting CMPro
              dats.

18/12/2000  *** v1.4 ***
            - Mimic HWC support.
            - New format to list game titles ('-f titlelist').
            - Year/Manufacturer support for formats that can include them.
            - Can specify values for ROMCenter/ROMCenter2 headers.
            - Remove clones factility ('-r').
            - GWC loader now understands '.speech' tags (System16 Beta).
            - Callus95 Patch renamed 'directory' to 'parent'. Now supported.
            - Auto sorts when converting to RomCenter format or a Title List.
            - Improved sorting (ensures each parent precedes its clones).
            - RAINE and MESS files are converted to CMPro format by default.
            - Fixed a problem with the '-g' option and RAINE gameinfo files.
            - Increased maximum ROM size to 9 characters (~1GB) for U64Emu.
            - Increased number of lines scanned when identifying CMPro dats.
            - Source: Added a makefile and included a SAVE_ONLY_FORMAT.
            - Source: Changed parameter passing (uses a structure now).
            - Preliminary: 'Verify Merging' facility (currently disabled).

19/05/2000  *** v1.3 ***
            - RomCenter 2 support ('-f romcenter2').
            - New method of sorting (useful for RomCenter).
            - Two EXEs. This is because the long ROM name version eats memory
              and is slow on machines without much physical memory.
            - Improved handling of invalid parents.
            - Warns of funny ROM sizes.
            - Generates log of warnings during conversion.

28/04/2000  *** v1.2 ***
            - Increased game/rom name length to 80 characters.
              (DatUtil now requires even more memory for the dats!)
            - Minor string handling improvements. yawn...

08/04/2000  *** v1.1 ***
            - Added support for JFF.TXT, Sparcade RAT and JAS Romlist.
            - Increased maximum number of games from 4096 to 5000.
            - Increased game/rom name length to 50 characters.
              (DatUtil now requires a HUGE block of memory to hold the data)
            - Improved listinfo parser so that it can handle odd formatting.
            - Other minor modifications.

31/03/2000  *** v1.0 ***
