[ehellbar]

Related JIRA ticket: https://its.cern.ch/jira/browse/O2-6543

This will use the script GRID/utils/getCCDBTimeMachineTimestamp.py in anchorMC.sh to obtain the CCDB time machine timestamp from the CCDB (or estimate it from the given anchored apass output files). This is now the default behavior. If the timestamp cannot be obtained, anchorMC.sh exits with an error. If set, the JDL vairable ALIEN_JDL_CCDB_CONDITION_NOT_AFTER can still be used to set the --condition-not-after timestamp by hand instead of obtaining it from CCDB.

The default object path for the time machine timestamp is Users/a/aliprod/CCDBTimeMachineTS. All objects are uploaded with validity 1:2, and the metadata contains the pass, period, NotAfter and LPMProductionTag (to track the productions which uploaded the object). So there should be one object per reconstruction pass per period, valid for all runs of the given pass and period. Some test examples are already uploaded in http://ccdb-test.cern.ch:8080/browse/Users/e/ehellbar/CCDBTimeMachineTS.

This PR contains:

1. in getCCDBTimeMachineTimestamp.py, add a second CCDB query to check for existing objects right before the upload of a new object, to check if between the last empty query and now (after the estimation of the not-after timestamp from the reco output) another job was faster and already uploaded the object. This is an attempt to avoid duplicate objects for the same period/pass uploaded by a single LPMProductionTag.
2. use getCCDBTimeMachineTimestamp.py in anchorMC.sh to obtain the CCDB time machine timestamp by default, keeping the current ALIEN_JDL_CCDB_CONDITION_NOT_AFTER option to enforce it by hand
3. in o2dpg_sim_workflow.py, use the script's argument for condition-not-after instead of the env var when running o2-create-aligned-geometry-workflow
4. refactor of log messages
5. CCDB cross check

• do it for final not-after (from CCDB or estimated) instead of only when estimating it
• simplify logic
• make it non-fatal

6. catch errors during CCDB query and upload
7. set aliprod as default user for the CCDB object path
8. from anchorMC.sh, only upload new objects when running as aliprod in GRID jobs

I mark this as WIP since the following is still missing:

• open points from the feedback I received privately by @sawenzel
  • whether to query the object by run number via the validity or by period, pass via the metadata
  • treatment of MC specific CCDB objects: There is a handful of MC CCDB paths (TPC corrections, GLO/Param/MatLUT) for which the objects are uploaded independently of the data calibration objects, so the NotAfter as it is obtained here might invalidate the correct object to use.
  • cross-check whether suggested changes in the framework, i.e. to add the time machine check when retrieving blobs from local cache, are really necessary
• I still need to run a small test on the grid to finally validate the implementation.
• Where / when to actually upload new objects
  • on-the-fly in MC jobs (that's how it's currently done in this PR)
  • run manually once for existing passes?
  • run at the end of a real data reconstruction pass / job

[github-actions]

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

[ehellbar]

I pushed a first round of new commits which take care of some points of the first review I received by @sawenzel . I updated the description and I also added the remaining open points from the first review there. I will continue with them next.