Podd 1.5 to 1.6 Database Conversion¶
In Analyzer 1.5 and earlier, many (but not all) detector classes employed fixed-format database files. These are problematic for a number of reasons: they are prone to errors, hard to extend, inefficient with time stamps, not readily transferable to non file-based database systems, etc. To address this problem, all modules (detectors, spectrometers, physics modules etc.) in Analyzer version 1.6 use the LoadDB database API. LoadDB expects database files in free-form key/value format.
To help users convert their existing databases to the new format, which could be a non-trivial and potentially very time-consuming task, we have written an automatic conversion tool called dbconvert. To build this tool, proceed as follows
- Ensure that the analyzer has been built properly
- Point the environment variable ANALYZER to the top-level installation directory. For example, if you have installed the distribution in ~/analyzer-1.6.0, do
% export ANALYZER=~/analyzer-1.6.0
or for tcsh users (JLab CUE)
% setenv ANALYZER ~/analyzer-1.6.0
- Now build dbconvert
% cd $ANALYZER/utils % make % cd ..
Once built, the tool allows you to convert an entire directory tree of database files to a new directory tree of free-format files. Before starting, make sure you have a backup of your old databases. Assuming the old databases are stored under $DB_DIR, back them up and write-protect them:
% cp -av $DB_DIR DB_backup % chmod -R -w DB_backup
Now you can convert the files in $DB_DIR simply like this
./utils/dbconvert $DB_DIR newDB
where newDB is the name of the output directory. That directory will be created if it does not exist. Treat it like scratch space; the name can always be changed later. If the target directory does exists, all database files in it will be erased after asking for confirmation.
dbconvert has to make assumptions about what detector each database files belongs to. To do so, it uses a mapping from typical detector names to detector types. For example, databases with detector name "R.s1" are interpreted as belonging to THaScintillator or derived classes. You can inspect the full name mapping in the source code of dbconvert.cxx in function DefaultMap(). To change this map, you currently have to edit the source and recompile; we are planning to make the mapping user-configurable in a later version. This is important to know in case you are using custom names for some detectors in your experiment. dbconvert is unlikely to know about them. To have these files converted automatically, you will need to edit the list.
Of course, if you have written an entirely new custom detector with its own database format, dbconvert will not have a conversion function. In that case, no worries. In general, you do not need to convert databases for your own custom-written modules. Even with Analyzer 1.6, you can do whatever you want in the ReadDatabase functions of your own modules, and things will continue to work as before. Using the new database system is strongly encouraged, however, for all new development. Eventually, we might use different database backends, and your old-style code would not be able to take advantage of those new features.
Caution: dbconvert is still under development at this time (Feburary 2018). Bugs may cause incorrect conversions. Additional features may be added that improve the conversion results (formatting, version indicators, etc.) It goes without saying that you must keep backup copies of your original database files so you will be able to re-run the conversion at a later time. You should also use version control with your converted databases so you have a record of any edits you made. Do spot-checks of your converted databases for apparent errors and file a bug report if necessary.
Below is a summary of options currently understood by dbconvert:
Usage: dbconvert [options] SRC_DIR DEST_DIR Convert Podd 1.5 and earlier database files under SRC_DIR to Podd 1.6 and later format, written to DEST_DIR Options: -h, --help show this help message -v, --verbose increase verbosity -d, --debug print extensive debug info -p, --preserve-subdirs preserve subdirectory structure --no-preserve-subdirs don't preserve subdirectory structure (negates -p) --no-clean don't erase all existing files from DEST_DIR if it exists --no-verify don't ask for confirmation before deleting any files -s, --subdirs ARG split output files into new set of sudirectories given in ARG. ARG is a comma-separated list of directory names which must be in YYYYMMDD format. Overrides -p. -z, --input-timezone ARG interpret all validity timestamps in the input that don't have a timezone indicator to be in timezone ARG. ARG may be the name of a system timezone file (e.g. 'US/Eastern'), which supports DST, or a fixed offset specified as [+|-]nnnn (e.g. +0100 for central European standard time). The default is to use the current timezone at run time. --output-timezone ARG write all output timestamps relative to timezone ARG, specified in the same way as --input-timezone. The default is to use the current timezone at run time.
As mentioned before, additional features are expected to be added over the next several months as we test and improve the program.
Updated by Ole Hansen about 5 years ago · 1 revisions