2026-04-20T09:48:23Z Job state change UNDEFINED -> ACCEPTED   Reason: (Re)Accepting new job
2026-04-20T09:48:23Z Job state change ACCEPTED -> PREPARING   Reason: Starting job processing
[2026-04-20 11:48:24] [INFO] [378758/6] DTR 030d...2439: Scheduler received new DTR 030da5bd-1bcd-466a-9ab3-05184e8c2439 with source: gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif, destination: file:/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/LDMX_pro-4.3.3.sif, assigned to transfer share _default-download with priority 25
[2026-04-20 11:48:24] [VERBOSE] [378758/6] DTR 030d...2439: File is cacheable, will check cache
[2026-04-20 11:48:24] [VERBOSE] [378758/6] DTR 030d...2439: NEW->CHECK_CACHE
[2026-04-20 11:48:24] [VERBOSE] [378758/6] DTR 030d...2439: CHECK_CACHE->CHECKING_CACHE
[2026-04-20 11:48:24] [INFO] [378758/35] DTR 030d...2439: File gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif is cached (/projects/hep/fs9/scratch/ldmx/ldcs/arccache/data/62/054a1e8256c2f39a2784f5779cfe7ecc014f38) - checking permissions
[2026-04-20 11:48:24] [VERBOSE] [378758/35] DTR 030d...2439: DN /DC=org/DC=nordugrid/DC=ARC/O=LDMX/CN=Simulation Agent is cached and is valid until 2026-04-23 10:43:13 for URL gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif
[2026-04-20 11:48:24] [INFO] [378758/35] DTR 030d...2439: Permission checking passed
[2026-04-20 11:48:24] [VERBOSE] [378758/35] DTR 030d...2439: Cached copy is still valid
[2026-04-20 11:48:24] [VERBOSE] [378758/35] DTR 030d...2439: CHECKING_CACHE->CACHE_CHECKED
[2026-04-20 11:48:24] [VERBOSE] [378758/6] DTR 030d...2439: Destination file is in cache
[2026-04-20 11:48:24] [VERBOSE] [378758/6] DTR 030d...2439: CACHE_CHECKED->PROCESS_CACHE
[2026-04-20 11:48:24] [VERBOSE] [378758/6] DTR 030d...2439: PROCESS_CACHE->PROCESSING_CACHE
[2026-04-20 11:48:24] [INFO] [378758/36] DTR 030d...2439: Linking/copying cached file to /projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/LDMX_pro-4.3.3.sif
[2026-04-20 11:48:32] [VERBOSE] [378758/36] DTR 030d...2439: PROCESSING_CACHE->CACHE_PROCESSED
[2026-04-20 11:48:32] [INFO] [378758/6] DTR 030d...2439: Finished successfully
[2026-04-20 11:48:32] [VERBOSE] [378758/6] DTR 030d...2439: CACHE_PROCESSED->DONE
[2026-04-20 11:48:32] [INFO] [378758/6] DTR 030d...2439: Returning to generator
[2026-04-20 11:48:33] [INFO] [378758/7] DTR 030d...2439: f666ed2af92e: All downloads finished successfully
2026-04-20T09:48:34Z Job state change PREPARING -> SUBMIT   Reason: Pre-staging finished, passing job to LRMS
----- starting submit_slurm_job -----
SLURM jobname: LDMX_Prod_Simul
SLURM job script built
SLURM script follows:
-------------------------------------------------------------------
#!/bin/bash -l
# SLURM batch job script built by arex
#SBATCH --no-requeue
#SBATCH --export=NONE
#SBATCH -e /projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e.comment
#SBATCH -o /projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e.comment

#SBATCH -p lu48
#SBATCH --nice=50
#SBATCH -J 'LDMX_Prod_Simul'
#SBATCH --get-user-env
#SBATCH -n 1
#SBATCH 
#SBATCH -t 1200:0
#SBATCH -t 1200:0
#SBATCH --mem-per-cpu=2000
#SBATCH -A lu2026-2-27

# Overide umask of execution node (sometime values are really strange)
umask 077
 
# source with arguments for DASH shells
sourcewithargs() {
script=$1
shift
. $script
}
# Record job start timestamp
ACCOUNTING_STARTTIME=`date +"%s"`
# Select accounting method
echo "Detecting resource accounting method available for the job." 1>&2
JOB_ACCOUNTING=""
# Try to use cgroups first
if command -v arc-job-cgroup >/dev/null 2>&1; then
    echo "Found arc-job-cgroup tool: trying to initialize accounting cgroups for the job." 1>&2
    while true; do
        # memory cgroup
        memory_cgroup="$( arc-job-cgroup -m -n f666ed2af92e )"
        if [ $? -ne 0 -o -z "$memory_cgroup" ]; then
            echo "Failed to initialize memory cgroup for accounting." 1>&2
            break;
        fi
        # cpuacct cgroup
        cpuacct_cgroup="$( arc-job-cgroup -c -n f666ed2af92e )"
        if [ $? -ne 0 -o -z "$cpuacct_cgroup" ]; then
            echo "Failed to initialize cpuacct cgroup for accounting." 1>&2
            break;
        fi
        echo "Using cgroups method for job accounting" 1>&2
        JOB_ACCOUNTING="cgroup"
        break;
    done
fi

# Fallback to GNU_TIME if cgroups are not working
if [ -z "$JOB_ACCOUNTING" ]; then
    GNU_TIME='/usr/bin/time'
    echo "Looking for $GNU_TIME tool for accounting measurements" 1>&2
    if [ ! -z "$GNU_TIME" ] && ! "$GNU_TIME" --version >/dev/null 2>&1; then
        echo "GNU time is not found at: $GNU_TIME" 1>&2
    else
        echo "GNU time found and will be used for job accounting." 1>&2
        JOB_ACCOUNTING="gnutime"
    fi
fi

# Nothing works: rely on LRMS only
if [ -z "$JOB_ACCOUNTING" ]; then
    echo "Failed to use both cgroups and GNU time for resource usage accounting. Accounting relies on LRMS information only." 1>&2
fi

# Setting environment variables as specified by user
export 'GRID_GLOBAL_JOBID=f666ed2af92e'
export 'GRID_GLOBAL_JOBURL=https://l-pilot2.lunarc.lu.se:443/arex/f666ed2af92e'
export 'GRID_GLOBAL_JOBINTERFACE=org.nordugrid.arcrest'
export 'GRID_GLOBAL_JOBHOST=l-pilot2.lunarc.lu.se'

RUNTIME_JOB_DIR=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e
RUNTIME_JOB_STDIN=/dev/null
RUNTIME_JOB_STDOUT=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/stdout
RUNTIME_JOB_STDERR=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/stdout
RUNTIME_JOB_DIAG=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e.diag
if [ ! -z "$RUNTIME_GRIDAREA_DIR" ] ; then
  RUNTIME_JOB_DIR=$RUNTIME_GRIDAREA_DIR/`basename $RUNTIME_JOB_DIR`
  RUNTIME_JOB_STDIN=`echo "$RUNTIME_JOB_STDIN" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"`
  RUNTIME_JOB_STDOUT=`echo "$RUNTIME_JOB_STDOUT" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"`
  RUNTIME_JOB_STDERR=`echo "$RUNTIME_JOB_STDERR" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"`
  RUNTIME_JOB_DIAG=`echo "$RUNTIME_JOB_DIAG" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"`
  RUNTIME_CONTROL_DIR=`echo "$RUNTIME_CONTROL_DIR" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"`
fi
RUNTIME_LOCAL_SCRATCH_DIR=${RUNTIME_LOCAL_SCRATCH_DIR:-}
RUNTIME_LOCAL_SCRATCH_MOVE_TOOL=${RUNTIME_LOCAL_SCRATCH_MOVE_TOOL:-mv}
RUNTIME_FRONTEND_SEES_NODE=${RUNTIME_FRONTEND_SEES_NODE:-}
RUNTIME_NODE_SEES_FRONTEND=${RUNTIME_NODE_SEES_FRONTEND:-yes}
  if [ ! -z "$RUNTIME_LOCAL_SCRATCH_DIR" ] && [ ! -z "$RUNTIME_NODE_SEES_FRONTEND" ]; then
    RUNTIME_NODE_JOB_DIR="$RUNTIME_LOCAL_SCRATCH_DIR"/`basename "$RUNTIME_JOB_DIR"`
    rm -rf "$RUNTIME_NODE_JOB_DIR"
    mkdir -p "$RUNTIME_NODE_JOB_DIR"
    # move directory contents
    for f in "$RUNTIME_JOB_DIR"/.* "$RUNTIME_JOB_DIR"/*; do 
      [ "$f" = "$RUNTIME_JOB_DIR/*" ] && continue # glob failed, no files
      [ "$f" = "$RUNTIME_JOB_DIR/." ] && continue
      [ "$f" = "$RUNTIME_JOB_DIR/.." ] && continue
      [ "$f" = "$RUNTIME_JOB_DIR/.diag" ] && continue
      [ "$f" = "$RUNTIME_JOB_DIR/.comment" ] && continue
      [ -f "$f" ] || continue
      if ! $RUNTIME_LOCAL_SCRATCH_MOVE_TOOL "$f" "$RUNTIME_NODE_JOB_DIR"; then
        echo "Failed to '$RUNTIME_LOCAL_SCRATCH_MOVE_TOOL' '$f' to '$RUNTIME_NODE_JOB_DIR'" 1>&2
        exit 1
      fi
    done
    if [ ! -z "$RUNTIME_FRONTEND_SEES_NODE" ] ; then
      # creating link for whole directory
       ln -s "$RUNTIME_FRONTEND_SEES_NODE"/`basename "$RUNTIME_JOB_DIR"` "$RUNTIME_JOB_DIR"
    else
      # keep stdout, stderr and control directory on frontend
      # recreate job directory
      mkdir -p "$RUNTIME_JOB_DIR"
      # make those files
      mkdir -p `dirname "$RUNTIME_JOB_STDOUT"`
      mkdir -p `dirname "$RUNTIME_JOB_STDERR"`
      touch "$RUNTIME_JOB_STDOUT"
      touch "$RUNTIME_JOB_STDERR"
      RUNTIME_JOB_STDOUT__=`echo "$RUNTIME_JOB_STDOUT" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"`
      RUNTIME_JOB_STDERR__=`echo "$RUNTIME_JOB_STDERR" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"`
      rm "$RUNTIME_JOB_STDOUT__" 2>/dev/null
      rm "$RUNTIME_JOB_STDERR__" 2>/dev/null
      if [ ! -z "$RUNTIME_JOB_STDOUT__" ] && [ "$RUNTIME_JOB_STDOUT" != "$RUNTIME_JOB_STDOUT__" ]; then
        ln -s "$RUNTIME_JOB_STDOUT" "$RUNTIME_JOB_STDOUT__"
      fi
      if [ "$RUNTIME_JOB_STDOUT__" != "$RUNTIME_JOB_STDERR__" ] ; then
        if [ ! -z "$RUNTIME_JOB_STDERR__" ] && [ "$RUNTIME_JOB_STDERR" != "$RUNTIME_JOB_STDERR__" ]; then
          ln -s "$RUNTIME_JOB_STDERR" "$RUNTIME_JOB_STDERR__"
        fi
      fi
      if [ ! -z "$RUNTIME_CONTROL_DIR" ] ; then
        # move control directory back to frontend
        RUNTIME_CONTROL_DIR__=`echo "$RUNTIME_CONTROL_DIR" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"`
        mv "$RUNTIME_CONTROL_DIR__" "$RUNTIME_CONTROL_DIR"
      fi
    fi
    # adjust stdin,stdout & stderr pointers
    RUNTIME_JOB_STDIN=`echo "$RUNTIME_JOB_STDIN" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"`
    RUNTIME_JOB_STDOUT=`echo "$RUNTIME_JOB_STDOUT" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"`
    RUNTIME_JOB_STDERR=`echo "$RUNTIME_JOB_STDERR" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"`
    RUNTIME_FRONTEND_JOB_DIR="$RUNTIME_JOB_DIR"
    RUNTIME_JOB_DIR="$RUNTIME_NODE_JOB_DIR"
  fi
  if [ -z "$RUNTIME_NODE_SEES_FRONTEND" ] ; then
    mkdir -p "$RUNTIME_JOB_DIR"
  fi

RESULT=0

if [ "$RESULT" = '0' ] ; then
# RunTimeEnvironment function for APPS/LDMX-SIMPROD-3.0:
RTE_function_0 () {
LDMX_STORAGE_BASE="/projects/hep/fs9/shared/ldmx/ldcs/output/"
KEEP_LOCAL_COPY=""
# description: LDMX Production Simulation wrapper for aCT/Rucio
# param:LDMX_STORAGE_BASE:string::local path to persistent storage base for LDMX data
# param:KEEP_LOCAL_COPY:string::set to non-zero (suggest: 1) to keep a local copy of LDMX data (in local persistent storage)

if [ "x$1" = "x0" ]; then
  if [ -z "$LDMX_STORAGE_BASE" ]; then
    echo "ERROR: ARC CE admin should define LDMX_STORAGE_BASE with arcctl rte params-set" >&2
    exit 1
  fi
elif [ "x$1" = "x1" ]; then
  export LDMX_STORAGE_BASE
  export KEEP_LOCAL_COPY
fi

}
# Running RTE scripts (stage 1)
runtimeenvironments=
runtimeenvironments="${runtimeenvironments}APPS/LDMX-SIMPROD-3.0;"
# Calling APPS/LDMX-SIMPROD-3.0 function: 
RTE_function_0 1 
if [ $? -ne 0 ]; then
    echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 1 execution failed." 1>&2
    echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 1 execution failed." 1>>"${RUNTIME_JOB_STDERR}"
    exit 1
fi

echo "runtimeenvironments=$runtimeenvironments" >> "$RUNTIME_JOB_DIAG"
if [ ! "X$SLURM_NODEFILE" = 'X' ] ; then
  if [ -r "$SLURM_NODEFILE" ] ; then
    cat "$SLURM_NODEFILE" | sed 's/\(.*\)/nodename=\1/' >> "$RUNTIME_JOB_DIAG"
    NODENAME_WRITTEN="1"
  else
    SLURM_NODEFILE=
  fi
fi
# Detecting WN operating system for accounting purposes
if [ -f "/etc/os-release" ]; then
  SYSTEM_SOFTWARE="$( eval $( cat /etc/os-release ); echo "${NAME} ${VERSION}" )"
elif [ -f "/etc/system-release" ]; then
  SYSTEM_SOFTWARE="$( cat /etc/system-release )"
elif command -v lsb_release >/dev/null 2>&1; then
  SYSTEM_SOFTWARE=$(lsb_release -ds)
elif command -v hostnamectl >/dev/null 2>&1; then
  SYSTEM_SOFTWARE="$( hostnamectl 2>/dev/null | sed -n '/Operating System/s/^\s*Operating System:\s*//p' )"
elif command -v uname >/dev/null 2>&1; then
  SYSTEM_SOFTWARE="Linux $( uname -r)"
fi
[ -n "$SYSTEM_SOFTWARE" ] && echo "systemsoftware=${SYSTEM_SOFTWARE}" >> "$RUNTIME_JOB_DIAG"

if [ "$RESULT" = '0' ] ; then
  # Changing to session directory
  HOME=$RUNTIME_JOB_DIR
  export HOME
  if ! cd "$RUNTIME_JOB_DIR"; then
    echo "Failed to switch to '$RUNTIME_JOB_DIR'" 1>&2
    RESULT=1
  fi
  if [ ! -z "$RESULT" ] && [ "$RESULT" != 0 ]; then
    exit $RESULT
  fi
# Write nodename if not already written in LRMS-specific way
if [ -z "$NODENAME_WRITTEN" ] ; then
  nodename=`/bin/hostname -f`
  echo "nodename=$nodename" >> "$RUNTIME_JOB_DIAG"
fi
echo "Processors=1" >> "$RUNTIME_JOB_DIAG"
echo "Benchmark=HEPSPEC:1.0" >> "$RUNTIME_JOB_DIAG"
executable='./ldmxsim.sh'
# Check if executable exists
if [ ! -f "$executable" ]; 
then 
  echo "Path \"$executable\" does not seem to exist" 1>$RUNTIME_JOB_STDOUT 2>$RUNTIME_JOB_STDERR 1>&2
  exit 1
fi
# See if executable is a script, and extract the name of the interpreter
line1=$(dd if="$executable" count=1 2>/dev/null | head -n 1 | tr -d '\0')
shebang=`echo $line1 | sed -n 's/^#! *//p'`
interpreter=`echo $shebang | awk '{print $1}'`
if [ "$interpreter" = /usr/bin/env ]; then interpreter=`echo $shebang | awk '{print $2}'`; fi
# If it's a script and the interpreter is not found ...
[ "x$interpreter" = x ] || type "$interpreter" > /dev/null 2>&1 || {

  echo "Cannot run $executable: $interpreter: not found" 1>$RUNTIME_JOB_STDOUT 2>$RUNTIME_JOB_STDERR 1>&2
  exit 1; }
if [ "x$JOB_ACCOUNTING" = "xgnutime" ]; then
  $GNU_TIME -o "$RUNTIME_JOB_DIAG" -a -f 'WallTime=%es\nKernelTime=%Ss\nUserTime=%Us\nCPUUsage=%P\nMaxResidentMemory=%MkB\nAverageResidentMemory=%tkB\nAverageTotalMemory=%KkB\nAverageUnsharedMemory=%DkB\nAverageUnsharedStack=%pkB\nAverageSharedMemory=%XkB\nPageSize=%ZB\nMajorPageFaults=%F\nMinorPageFaults=%R\nSwaps=%W\nForcedSwitches=%c\nWaitSwitches=%w\nInputs=%I\nOutputs=%O\nSocketReceived=%r\nSocketSent=%s\nSignals=%k\n'  "./ldmxsim.sh" <$RUNTIME_JOB_STDIN 1>$RUNTIME_JOB_STDOUT 2>&1
else
   "./ldmxsim.sh" <$RUNTIME_JOB_STDIN 1>$RUNTIME_JOB_STDOUT 2>&1
fi
RESULT=$?

fi
fi
# Running RTE scripts (stage 2)
runtimeenvironments=
runtimeenvironments="${runtimeenvironments}APPS/LDMX-SIMPROD-3.0;"
# Calling APPS/LDMX-SIMPROD-3.0 function: 
RTE_function_0 2 
if [ $? -ne 0 ]; then
    echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 2 execution failed." 1>&2
    echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 2 execution failed." 1>>"${RUNTIME_JOB_STDERR}"
    exit 1
fi

# Measuring used scratch space
echo "usedscratch=$( du -sb "$RUNTIME_JOB_DIR" | sed "s/\s.*$//" )" >> "$RUNTIME_JOB_DIAG"
# Cleaning up extra files in the local scratch
if [ ! -z  "$RUNTIME_LOCAL_SCRATCH_DIR" ] ; then
  find ./ -type l -exec rm -f "{}" ";"
  chmod -R u+w "./"
  chmod -R u-w "$RUNTIME_JOB_DIR"/'rucio.metadata' 2>/dev/null
  dynlist='output.files'
  chmod -R u-w "./$dynlist" 2>/dev/null
  cat "./$dynlist" | while read name rest; do
    chmod -R u-w "./$name" 2>/dev/null
  done
  chmod -R u-w "$RUNTIME_JOB_DIR"/'stdout' 2>/dev/null
  chmod -R u-w "$RUNTIME_JOB_DIR"/'stdout' 2>/dev/null
  chmod -R u-w "$RUNTIME_JOB_DIR"/'gmlog/' 2>/dev/null
  find ./ -type f -perm /200 -exec rm -f "{}" ";"
  chmod -R u+w "./"
fi

  if [ ! -z "$RUNTIME_LOCAL_SCRATCH_DIR" ] && [ ! -z "$RUNTIME_NODE_SEES_FRONTEND" ]; then 
    if [ ! -z "$RUNTIME_FRONTEND_SEES_NODE" ] ; then
      # just move it
      rm -rf "$RUNTIME_FRONTEND_JOB_DIR"
      destdir=`dirname "$RUNTIME_FRONTEND_JOB_DIR"`
      if ! mv "$RUNTIME_NODE_JOB_DIR" "$destdir"; then
        echo "Failed to move '$RUNTIME_NODE_JOB_DIR' to '$destdir'" 1>&2
        RESULT=1
      fi
    else
      # remove links
      rm -f "$RUNTIME_JOB_STDOUT" 2>/dev/null
      rm -f "$RUNTIME_JOB_STDERR" 2>/dev/null
      # move directory contents
      for f in "$RUNTIME_NODE_JOB_DIR"/.* "$RUNTIME_NODE_JOB_DIR"/*; do 
        [ "$f" = "$RUNTIME_NODE_JOB_DIR/*" ] && continue # glob failed, no files
        [ "$f" = "$RUNTIME_NODE_JOB_DIR/." ] && continue
        [ "$f" = "$RUNTIME_NODE_JOB_DIR/.." ] && continue
        [ "$f" = "$RUNTIME_NODE_JOB_DIR/.diag" ] && continue
        [ "$f" = "$RUNTIME_NODE_JOB_DIR/.comment" ] && continue
        [ -f "$f" ] || continue
        if ! mv "$f" "$RUNTIME_FRONTEND_JOB_DIR"; then
          echo "Failed to move '$f' to '$RUNTIME_FRONTEND_JOB_DIR'" 1>&2
          RESULT=1
        fi
      done
      rm -rf "$RUNTIME_NODE_JOB_DIR"
    fi
  fi
# Handle cgroup measurements
if [ "x$JOB_ACCOUNTING" = "xcgroup" ]; then
    # Max memory used (total)
    maxmemory=$( cat "${memory_cgroup}/memory.memsw.max_usage_in_bytes" )
    maxmemory=$(( (maxmemory + 1023) / 1024 ))
    echo "maxtotalmemory=${maxmemory}kB" >> "$RUNTIME_JOB_DIAG"

    # Max memory used (RAM)
    maxram=$( cat "${memory_cgroup}/memory.max_usage_in_bytes" )
    maxram=$(( (maxram + 1023) / 1024 ))
    echo "maxresidentmemory=${maxram}kB" >> "$RUNTIME_JOB_DIAG"
    # TODO: this is for compatibilty with current A-REX accounting code. Remove when A-REX will use max value instead.
    echo "averageresidentmemory=${maxram}kB" >> "$RUNTIME_JOB_DIAG"

    # User CPU time
    if [ -f "${cpuacct_cgroup}/cpuacct.usage_user" ]; then
        # cgroup values are in nanoseconds
        user_cputime=$( cat "${cpuacct_cgroup}/cpuacct.usage_user" )
        user_cputime=$(( user_cputime / 1000000 ))
    elif [ -f "${cpuacct_cgroup}/cpuacct.stat" ]; then
        # older kernels have only cpuacct.stat that use USER_HZ units
        user_cputime=$( cat "${cpuacct_cgroup}/cpuacct.stat" | sed -n '/^user/s/user //p' )
        user_hz=$( getconf CLK_TCK )
        user_cputime=$(( user_cputime / user_hz ))
    fi
    [ -n "$user_cputime" ] && echo "usertime=${user_cputime}" >> "$RUNTIME_JOB_DIAG"

    # Kernel CPU time
    if [ -f "${cpuacct_cgroup}/cpuacct.usage_sys" ]; then
        # cgroup values are in nanoseconds
        kernel_cputime=$( cat "${cpuacct_cgroup}/cpuacct.usage_sys" )
        kernel_cputime=$(( kernel_cputime / 1000000 ))
    elif [ -f "${cpuacct_cgroup}/cpuacct.stat" ]; then
        # older kernels have only cpuacct.stat that use USER_HZ units
        kernel_cputime=$( cat "${cpuacct_cgroup}/cpuacct.stat" | sed -n '/^system/s/system //p' )
        [ -z "$user_hz" ] && user_hz=$( getconf CLK_TCK )
        kernel_cputime=$(( kernel_cputime / user_hz ))
    fi
    [ -n "$kernel_cputime" ] && echo "kerneltime=${kernel_cputime}" >> "$RUNTIME_JOB_DIAG"

    # Remove nested job accouting cgroups
    arc-job-cgroup -m -d
    arc-job-cgroup -c -d
fi

# Record CPU benchmarking values for WN user by the job
[ -n "${ACCOUNTING_BENCHMARK}" ] && echo "benchmark=${ACCOUNTING_BENCHMARK}" >> "$RUNTIME_JOB_DIAG"
# Record WN instance tag if defined
[ -n "${ACCOUNTING_WN_INSTANCE}" ] && echo "wninstance=${ACCOUNTING_WN_INSTANCE}" >> "$RUNTIME_JOB_DIAG"

# Record execution clock times
ACCOUNTING_ENDTIME=`date +"%s"`
# Mds date format (YYYYMMDDHHMMSSZ)
echo "LRMSStartTime=`date -d "1970-01-01 UTC ${ACCOUNTING_STARTTIME} seconds" +"%Y%m%d%H%M%SZ"`" >> "$RUNTIME_JOB_DIAG"
echo "LRMSEndTime=`date -d "1970-01-01 UTC ${ACCOUNTING_ENDTIME} seconds" +"%Y%m%d%H%M%SZ"`" >> "$RUNTIME_JOB_DIAG"
echo "walltime=$(( ACCOUNTING_ENDTIME - ACCOUNTING_STARTTIME ))" >> "$RUNTIME_JOB_DIAG"

# Add exit code to the accounting information and exit the job script
echo "exitcode=$RESULT" >> "$RUNTIME_JOB_DIAG"
exit $RESULT
-------------------------------------------------------------------

job submitted successfully!
local job id: 2914224
----- exiting submit_slurm_job -----

2026-04-20T09:48:36Z Job state change SUBMIT -> INLRMS   Reason: Job is passed to LRMS
------- Contents of output stream forwarded by the LRMS ---------
Detecting resource accounting method available for the job.
Looking for /usr/bin/time tool for accounting measurements
GNU time found and will be used for job accounting.
------------------------- End of output -------------------------
2026-04-20T09:50:09Z Job state change INLRMS -> FINISHING   Reason: Job finished executing in LRMS
[2026-04-20 11:50:14] [INFO] [378758/6] DTR 3208...3a2d: Scheduler received new DTR 32088d9e-e843-413a-8f75-357a39e63a2d with source: file:/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/simoutput.root, destination: gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root, assigned to transfer share _default-upload with priority 25
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: File is not cacheable, was requested not to be cached or no cache available, skipping cache check
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: NEW->CACHE_CHECKED
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: Neither source nor destination are index services, will skip resolving replicas
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: CACHE_CHECKED->RESOLVED
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: Checking source file is present
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: RESOLVED->QUERY_REPLICA
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: QUERY_REPLICA->QUERYING_REPLICA
[2026-04-20 11:50:14] [INFO] [378758/37] DTR 3208...3a2d: Checking file:/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/simoutput.root
[2026-04-20 11:50:14] [VERBOSE] [378758/37] DTR 3208...3a2d: QUERYING_REPLICA->REPLICA_QUERIED
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: REPLICA_QUERIED->PRE_CLEAN
[2026-04-20 11:50:14] [VERBOSE] [378758/6] DTR 3208...3a2d: PRE_CLEAN->PRE_CLEANING
[2026-04-20 11:50:14] [DEBUG] [378758/38] DTR 3208...3a2d: Starting helper process:  /usr/lib64/arc/arc-dmcgridftp -F 2 -V DEBUG -s 0 -p 0 -o 0 stat gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root 0
[2026-04-20 11:50:14] [DEBUG] [378758/38] DTR 3208...3a2d: Starting helper process:  /usr/lib64/arc/arc-dmcgridftp -F 2 -V DEBUG -s 0 -p 0 -o 0 stat gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root 0
[2026-04-20 11:50:15] [WARNING] [378758/38] DTR 3208...3a2d: Can not find voms service configuration file (vomses) in default locations: ~/.arc/vomses, ~/.voms/vomses, $ARC_LOCATION/etc/vomses, $ARC_LOCATION/etc/grid-security/vomses, $PWD/vomses, /etc/vomses, /etc/grid-security/vomses
[2026-04-20 11:50:15] [VERBOSE] [378758/38] DTR 3208...3a2d: PRE_CLEANING->PRE_CLEANED
[2026-04-20 11:50:15] [VERBOSE] [378758/6] DTR 3208...3a2d: No need to stage source or destination, skipping staging
[2026-04-20 11:50:15] [VERBOSE] [378758/6] DTR 3208...3a2d: PRE_CLEANED->STAGED_PREPARED
[2026-04-20 11:50:15] [VERBOSE] [378758/6] DTR 3208...3a2d: DTR is ready for transfer, moving to delivery queue
[2026-04-20 11:50:15] [VERBOSE] [378758/6] DTR 3208...3a2d: STAGED_PREPARED->TRANSFER
[2026-04-20 11:50:15] [INFO] [378758/6] DTR 3208...3a2d: Delivery received new DTR 32088d9e-e843-413a-8f75-357a39e63a2d with source: file:/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/simoutput.root, destination: gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root
[2026-04-20 11:50:15] [VERBOSE] [378758/6] DTR 3208...3a2d: TRANSFER->TRANSFERRING
[2026-04-20 11:50:15] [DEBUG] [378758/39] DTR 3208...3a2d: Running command: /usr/lib64/arc/DataStagingDelivery --surl file://;httpgetpartial=no/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/f666ed2af92e/simoutput.root --durl gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root --sopt ca=/etc/grid-security/certificates --dopt ca=/etc/grid-security/certificates --sopt casystem=0 --dopt casystem=0 --sopt cagrid=1 --dopt cagrid=1 --topt minspeed=0 --topt minspeedtime=300 --topt minavgspeed=0 --topt maxinacttime=300 --size 1549463 --cstype adler32 
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: Will calculate adler32 checksum
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: Can not find voms service configuration file (vomses) in default locations: ~/.arc/vomses, ~/.voms/vomses, $ARC_LOCATION/etc/vomses, $ARC_LOCATION/etc/grid-security/vomses, $PWD/vomses, /etc/vomses, /etc/grid-security/vomses
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:     1 s:        0.0 kB       0.0 kB/s       0.0 kB/s    . . .       
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: write_thread: get and pass buffers
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Can not access proxy file: /tmp/x509up_u6331
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using certificate file: /etc/grid-security/hostcert.pem
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using key file: /etc/grid-security/hostkey.pem
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using CA certificate directory: /etc/grid-security/certificates
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using insecure data transfer
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: mkdir
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] mkdir_ftp: making gsiftp://hep-fs.lunarc.lu.se:2811/ldcs
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] mkdir_ftp: making gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] mkdir_ftp: making gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] mkdir_ftp: making gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] mkdir_ftp: making gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: put
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for data tag
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for data chunk
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: data chunk: 0 1048576
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for data tag
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for data chunk
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: write_thread: for_write eof
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: write_thread: exiting
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: data chunk: 1048576 500887
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for some buffers sent
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for data tag
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: waiting for data chunk
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_writing_ftp: data chunk: 1549463 0
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] start_wr
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: iting_ftp: waiting for some buffers sent
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] ftp_write_thread: waiting for transfer complete
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] ftp_write_thread: waiting for buffers released
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] ftp_write_thread: exiting
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: StopWriting: Calculated checksum adler32:59654d2d
[2026-04-20 11:50:16] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: Can not find voms service configuration file (vomses) in default locations: ~/.arc/vomses, ~/.voms/vomses, $ARC_LOCATION/etc/vomses, $ARC_LOCATION/etc/grid-security/vomses, $PWD/vomses, /etc/vomses, /etc/grid-security/vomses
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Can not access proxy file: /tmp/x509up_u6331
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using certificate file: /etc/grid-security/hostcert.pem
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using key file: /etc/grid-security/hostkey.pem
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using CA certificate directory: /etc/grid-security/certificates
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Using insecure data transfer
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Command: DCAU N
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Response: 200 Authentication type accepted
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Command: MLST /ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] Response: Information follows
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  type=file;size=1549463;modify=20260420095016;perm=dr; /ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: 250 Information finished
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] list record: type=file;size=1549463;modify=20260420095016;perm=dr; /ldcs/validation/v15-8gev/8.0GeV/v4.3.3-1e-test/mc_v15-8gev-8.0GeV-1e-inclusive_run1_t1776678603.root
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] ftp_complete_callback: error: globus_ftp_client: the server responded with an error/500 Do 
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: not understand
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery:  [external] list_files_ftp: no checksum information returned
[2026-04-20 11:50:17] [INFO] [378758/16] DTR 3208...3a2d: DataDelivery: No checksum information possible
[2026-04-20 11:50:17] [INFO] [378758/4] DTR 3208...3a2d: Transfer finished: 1549463 bytes transferred : checksum adler32:59654d2d
[2026-04-20 11:50:17] [VERBOSE] [378758/4] DTR 3208...3a2d: TRANSFERRING->TRANSFERRED
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: Neither source nor destination were staged, skipping releasing requests
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: TRANSFERRED->REQUEST_RELEASED
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: REQUEST_RELEASED->REPLICA_FINALISED
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: Destination is not index service, skipping replica registration
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: REPLICA_FINALISED->REPLICA_REGISTERED
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: File is not cacheable, skipping cache processing
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: REPLICA_REGISTERED->CACHE_PROCESSED
[2026-04-20 11:50:17] [INFO] [378758/6] DTR 3208...3a2d: Finished successfully
[2026-04-20 11:50:17] [VERBOSE] [378758/6] DTR 3208...3a2d: CACHE_PROCESSED->DONE
[2026-04-20 11:50:17] [INFO] [378758/6] DTR 3208...3a2d: Returning to generator
[2026-04-20 11:50:19] [INFO] [378758/7] DTR 3208...3a2d: f666ed2af92e: All uploads finished successfully
2026-04-20T09:50:19Z Job state change FINISHING -> FINISHED   Reason: Stage-out finished.