2026-04-20T08:43:14Z Job state change UNDEFINED -> ACCEPTED Reason: (Re)Accepting new job 2026-04-20T08:43:14Z Job state change ACCEPTED -> PREPARING Reason: Starting job processing [2026-04-20 10:43:16] [INFO] [378758/6] DTR 4f6d...bc6e: Scheduler received new DTR 4f6d1de1-d600-4eb2-83c7-673cdf49bc6e with source: gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif, destination: file:/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29/LDMX_pro-4.3.3.sif, assigned to transfer share _default-download with priority 25 [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: File is cacheable, will check cache [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: NEW->CHECK_CACHE [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: CHECK_CACHE->CHECKING_CACHE [2026-04-20 10:43:16] [VERBOSE] [378758/19] DTR 4f6d...bc6e: Will download to cache file /projects/hep/fs9/scratch/ldmx/ldcs/arccache/data/62/054a1e8256c2f39a2784f5779cfe7ecc014f38 [2026-04-20 10:43:16] [VERBOSE] [378758/19] DTR 4f6d...bc6e: CHECKING_CACHE->CACHE_CHECKED [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: Neither source nor destination are index services, will skip resolving replicas [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: CACHE_CHECKED->RESOLVED [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: Checking source file is present [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: RESOLVED->QUERY_REPLICA [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: QUERY_REPLICA->QUERYING_REPLICA [2026-04-20 10:43:16] [INFO] [378758/20] DTR 4f6d...bc6e: Checking gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif [2026-04-20 10:43:16] [DEBUG] [378758/20] DTR 4f6d...bc6e: Starting helper process: /usr/lib64/arc/arc-dmcgridftp -F 2 -V DEBUG -s 0 -p 0 -o 0 stat gsiftp://hep-fs.lunarc.lu.se:2811;cache=copy;httpgetpartial=no/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif 8 [2026-04-20 10:43:16] [DEBUG] [378758/20] DTR 4f6d...bc6e: Starting helper process: /usr/lib64/arc/arc-dmcgridftp -F 2 -V DEBUG -s 0 -p 0 -o 0 stat gsiftp://hep-fs.lunarc.lu.se:2811;cache=copy;httpgetpartial=no/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif 8 [2026-04-20 10:43:16] [WARNING] [378758/20] DTR 4f6d...bc6e: Can not find voms service configuration file (vomses) in default locations: ~/.arc/vomses, ~/.voms/vomses, $ARC_LOCATION/etc/vomses, $ARC_LOCATION/etc/grid-security/vomses, $PWD/vomses, /etc/vomses, /etc/grid-security/vomses [2026-04-20 10:43:16] [VERBOSE] [378758/20] DTR 4f6d...bc6e: QUERYING_REPLICA->REPLICA_QUERIED [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: REPLICA_QUERIED->PRE_CLEAN [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: PRE_CLEAN->PRE_CLEANING [2026-04-20 10:43:16] [VERBOSE] [378758/21] DTR 4f6d...bc6e: PRE_CLEANING->PRE_CLEANED [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: No need to stage source or destination, skipping staging [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: PRE_CLEANED->STAGED_PREPARED [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: DTR is ready for transfer, moving to delivery queue [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: STAGED_PREPARED->TRANSFER [2026-04-20 10:43:16] [INFO] [378758/6] DTR 4f6d...bc6e: Delivery received new DTR 4f6d1de1-d600-4eb2-83c7-673cdf49bc6e with source: gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif, destination: file:/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29/LDMX_pro-4.3.3.sif [2026-04-20 10:43:16] [VERBOSE] [378758/6] DTR 4f6d...bc6e: TRANSFER->TRANSFERRING [2026-04-20 10:43:16] [DEBUG] [378758/22] DTR 4f6d...bc6e: Running command: /usr/lib64/arc/DataStagingDelivery --surl gsiftp://hep-fs.lunarc.lu.se:2811;cache=copy;httpgetpartial=no/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif --durl /projects/hep/fs9/scratch/ldmx/ldcs/arccache/data/62/054a1e8256c2f39a2784f5779cfe7ecc014f38 --sopt ca=/etc/grid-security/certificates --dopt ca=/etc/grid-security/certificates --sopt casystem=0 --dopt casystem=0 --sopt cagrid=1 --dopt cagrid=1 --topt minspeed=0 --topt minspeedtime=300 --topt minavgspeed=0 --topt maxinacttime=300 [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: start_reading [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: Can not find voms service configuration file (vomses) in default locations: ~/.arc/vomses, ~/.voms/vomses, $ARC_LOCATION/etc/vomses, $ARC_LOCATION/etc/grid-security/vomses, $PWD/vomses, /etc/vomses, /etc/grid-security/vomses [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: read_thread: get and register buffers [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] Can not access proxy file: /tmp/x509up_u0 [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] Using certificate file: /etc/grid-security/hostcert.pem [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] Using key file: /etc/grid-security/hostkey.pem [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] Using CA certificate directory: /etc/grid-security/certificates [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] Using insecure data transfer [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] start_reading_ftp [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] start_reading_ftp: globus_ftp_client_get [2026-04-20 10:43:17] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_thread: waiting for eof [2026-04-20 10:43:18] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 1 s: 0.0 kB 0.0 kB/s 0.0 kB/s . . . [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 2 s: 176128.0 kB 86596.3 kB/s 88064.0 kB/s . . . [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 215220224 != 215154688 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 215220224 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 310837248 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 310902784 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 310968320 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311033856 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311099392 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311164928 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311230464 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311296000 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311361536 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311427072 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311492608 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311558144 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_ [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: read_callback: unexpected data out of order: 311623680 != 310771712 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 310837248 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 310902784 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 310968320 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311033856 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311099392 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311164928 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311230464 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311296000 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311361536 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311427072 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311492608 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311558144 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311623680 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311820288 != 311754752 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311885824 != 311754752 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 311951360 != 311754752 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 312016896 != 311754752 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 312082432 != 311754752 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311820288 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311885824 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 311951360 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 312016896 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 312082432 65536 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 312279040 != 312213504 [2026-04-20 10:43:19] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 312279040 65536 [2026-04-20 10:43:20] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 3 s: 351424.0 kB 114226.8 kB/s 117141.3 kB/s . . . [2026-04-20 10:43:21] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 4 s: 605504.0 kB 146703.6 kB/s 151376.0 kB/s . . . [2026-04-20 10:43:22] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 5 s: 838208.0 kB 161171.9 kB/s 167641.6 kB/s . . . [2026-04-20 10:43:23] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 6 s: 1087103.4 kB 172862.7 kB/s 181183.9 kB/s . . . [2026-04-20 10:43:24] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 7 s: 1303164.1 kB 176049.9 kB/s 186166.3 kB/s . . . [2026-04-20 10:43:25] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 8 s: 1521276.2 kB 178285.9 kB/s 190159.5 kB/s . . . [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 9 s: 1737596.8 kB 179470.1 kB/s 193066.3 kB/s . . . [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1781989376 != 1781923840 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1781989376 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782120448 != 1782054912 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782120448 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782251520 != 1782185984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782317056 != 1782185984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782382592 != 1782185984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782251520 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782317056 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782382592 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782513664 != 1782448128 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782513664 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: unexpected data out of order: 1782644736 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782710272 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782775808 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782841344 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782906880 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1782972416 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783037952 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783103488 != 1782579200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782644736 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782710272 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782775808 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782841344 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1782906880 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callba [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: ck: delayed data chunk: 1782972416 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783037952 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783103488 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783365632 != 1783300096 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783365632 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783496704 != 1783431168 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783562240 != 1783431168 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783496704 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783562240 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783693312 != 1783627776 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783693312 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783824384 != 1783758848 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1783889920 != 1783758848 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: : 1783955456 != 1783758848 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784020992 != 1783758848 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783824384 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783889920 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1783955456 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784020992 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784152064 != 1784086528 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784152064 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784283136 != 1784217600 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784348672 != 1784217600 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784414208 != 1784217600 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784479744 != 1784217600 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784545280 != 1784217600 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 283136 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784348672 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784414208 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784479744 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784545280 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784676352 != 1784610816 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784741888 != 1784610816 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784676352 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784741888 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784872960 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1784938496 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785004032 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785069568 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785135104 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785200640 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785266176 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785331712 != 1784807424 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784872960 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1784938496 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785004032 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785069568 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785135104 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785200640 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785266176 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785331712 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785462784 != 1785397248 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785462784 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 17855938 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 56 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785659392 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785724928 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785790464 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785856000 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785921536 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1785987072 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786052608 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786118144 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786183680 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786249216 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786314752 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 1786380288 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786445824 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786511360 != 1785528320 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785593856 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785659392 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785724928 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785790464 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785856000 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785921536 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1785987072 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786052608 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786118144 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786183680 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786249216 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786314752 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: ftp_read_callback: delayed data chunk: 1786380288 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786445824 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786511360 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786839040 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786904576 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1786970112 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787035648 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787101184 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787166720 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787232256 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787297792 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787363328 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of ord [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: er: 1787428864 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787494400 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787559936 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787625472 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787691008 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787756544 != 1786773504 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786839040 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786904576 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1786970112 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787035648 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787101184 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787166720 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787232256 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787297792 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_ [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: callback: delayed data chunk: 1787363328 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787428864 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787494400 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787559936 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787625472 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787691008 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787756544 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1787887616 != 1787822080 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1787887616 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788018688 != 1787953152 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788018688 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788149760 != 1788084224 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788149760 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788280832 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_ [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: callback: unexpected data out of order: 1788346368 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788411904 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788477440 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788542976 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788608512 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788674048 != 1788215296 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788280832 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788346368 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788411904 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788477440 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788542976 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788608512 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788674048 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: order: 1788870656 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1788936192 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789001728 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789067264 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789132800 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789198336 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789263872 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789329408 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789394944 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789460480 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789526016 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789591552 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected dat [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: a out of order: 1789657088 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789722624 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789788160 != 1788805120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788870656 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1788936192 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789001728 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789067264 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789132800 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789198336 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789263872 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789329408 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789394944 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789460480 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789526016 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789591552 6 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 5536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789657088 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789722624 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789788160 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1789984768 != 1789919232 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790050304 != 1789919232 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790115840 != 1789919232 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1789984768 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790050304 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790115840 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790246912 != 1790181376 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790312448 != 1790181376 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790246912 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790312448 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected dat [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: a out of order: 1790443520 != 1790377984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790509056 != 1790377984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790574592 != 1790377984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790640128 != 1790377984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790705664 != 1790377984 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790443520 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790509056 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790574592 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790640128 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790705664 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1790836736 != 1790771200 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1790836736 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791361024 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: : 1791426560 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791492096 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791557632 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791623168 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791688704 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791754240 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791819776 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791885312 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1791950848 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792016384 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792081920 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792147456 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: of order: 1792212992 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792278528 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792344064 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792409600 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792475136 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792540672 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792606208 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792671744 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792737280 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792802816 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792868352 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1792933888 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected d [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: ata out of order: 1792999424 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793064960 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793130496 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793196032 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793261568 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793327104 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793392640 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793458176 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793523712 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793589248 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793654784 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793720320 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unex [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: pected data out of order: 1793785856 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793851392 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793916928 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1793982464 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794048000 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794113536 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794179072 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794244608 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794310144 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794375680 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794441216 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794506752 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callba [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: ck: unexpected data out of order: 1794572288 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794637824 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794703360 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794768896 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794834432 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794899968 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1794965504 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795031040 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795096576 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795162112 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795227648 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795293184 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_rea [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: d_callback: unexpected data out of order: 1795358720 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795424256 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795489792 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795555328 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795620864 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795686400 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795751936 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795817472 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795883008 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1795948544 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796014080 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796079616 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: ftp_read_callback: unexpected data out of order: 1796145152 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796210688 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796276224 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796341760 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796407296 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796472832 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796538368 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796603904 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796669440 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796734976 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796800512 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1796866048 != 1791295488 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [e [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: xternal] ftp_read_callback: delayed data chunk: 1791361024 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791426560 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791492096 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791557632 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791623168 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791688704 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791754240 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791819776 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791885312 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1791950848 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792016384 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792081920 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792147456 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792212992 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792278528 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [exter [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: nal] ftp_read_callback: delayed data chunk: 1792344064 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792409600 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792475136 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792540672 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792606208 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792671744 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792737280 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792802816 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792868352 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792933888 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1792999424 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793064960 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793130496 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793196032 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793261568 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: ftp_read_callback: delayed data chunk: 1793327104 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793392640 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793458176 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793523712 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793589248 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793654784 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793720320 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793785856 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793851392 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793916928 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1793982464 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794048000 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794113536 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794179072 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794244608 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: _read_callback: delayed data chunk: 1794310144 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794375680 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794441216 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794506752 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794572288 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794637824 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794703360 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794768896 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794834432 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794899968 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1794965504 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795031040 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795096576 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795162112 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795227648 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_rea [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: d_callback: delayed data chunk: 1795293184 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795358720 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795424256 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795489792 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795555328 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795620864 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795686400 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795751936 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795817472 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795883008 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1795948544 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796014080 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796079616 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796145152 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796210688 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_ca [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: llback: delayed data chunk: 1796276224 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796341760 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796407296 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796472832 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796538368 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796603904 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796669440 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796734976 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796800512 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1796866048 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797062656 != 1796997120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797128192 != 1796997120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797193728 != 1796997120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797259264 != 1796997120 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_ca [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: llback: delayed data chunk: 1797062656 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797128192 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797193728 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797259264 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797521408 != 1797455872 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797521408 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797849088 != 1797783552 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797914624 != 1797783552 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1797980160 != 1797783552 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797849088 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797914624 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1797980160 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1798176768 != 1798111232 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1798242304 ! [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: = 1798111232 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1798307840 != 1798111232 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1798176768 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1798242304 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1798307840 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1798438912 != 1798373376 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1798438912 65536 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 1799946240 != 1799880704 [2026-04-20 10:43:26] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 1799946240 65536 [2026-04-20 10:43:27] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 10 s: 1827516.8 kB 167673.2 kB/s 182751.7 kB/s . . . [2026-04-20 10:43:28] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 11 s: 1935103.3 kB 159507.2 kB/s 175918.5 kB/s . . . [2026-04-20 10:43:29] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 12 s: 2062972.2 kB 154256.2 kB/s 171914.3 kB/s . . . [2026-04-20 10:43:30] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 13 s: 2252860.0 kB 154380.5 kB/s 173296.9 kB/s . . . [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 14 s: 2483264.0 kB 157147.3 kB/s 177376.0 kB/s . . . [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620260352 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620325888 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620391424 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620456960 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620522496 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620588032 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620653568 != 2620194816 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620260352 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620325888 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620391424 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620456960 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620522496 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620588032 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ft [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: p_read_callback: delayed data chunk: 2620653568 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620784640 != 2620719104 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620850176 != 2620719104 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2620915712 != 2620719104 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620784640 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620850176 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2620915712 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621243392 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621308928 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621374464 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621440000 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621505536 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621571072 != 262 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 1177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621636608 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621702144 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621767680 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621833216 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621898752 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2621964288 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622029824 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622095360 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622160896 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622226432 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622291968 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 262235750 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 4 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622423040 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622488576 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622554112 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622619648 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622685184 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622750720 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: unexpected data out of order: 2622816256 != 2621177856 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621243392 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621308928 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621374464 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621440000 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621505536 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621571072 6553 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 6 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621636608 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621702144 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621767680 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621833216 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621898752 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2621964288 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622029824 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622095360 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622160896 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622226432 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622291968 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622357504 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622423040 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622488576 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622554112 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [ [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: external] ftp_read_callback: delayed data chunk: 2622619648 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622685184 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622750720 65536 [2026-04-20 10:43:31] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_callback: delayed data chunk: 2622816256 65536 [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: 15 s: 2714238.0 kB 159367.9 kB/s 180949.2 kB/s . . . [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_thread: waiting for buffers released [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: [external] ftp_read_thread: exiting [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: read_thread: exiting [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: stop_reading: waiting for transfer to finish [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: stop_reading: exiting: gsiftp://hep-fs.lunarc.lu.se:2811/ldcs/prod/prod.image/v4.3.3_image/LDMX_pro-4.3.3.sif [2026-04-20 10:43:32] [INFO] [378758/16] DTR 4f6d...bc6e: DataDelivery: Checksum not computed [2026-04-20 10:43:32] [INFO] [378758/4] DTR 4f6d...bc6e: Transfer finished: 2790465536 bytes transferred [2026-04-20 10:43:32] [VERBOSE] [378758/4] DTR 4f6d...bc6e: TRANSFERRING->TRANSFERRED [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: Neither source nor destination were staged, skipping releasing requests [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: TRANSFERRED->REQUEST_RELEASED [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: REQUEST_RELEASED->REPLICA_FINALISED [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: Destination is not index service, skipping replica registration [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: REPLICA_FINALISED->REPLICA_REGISTERED [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: Will process cache [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: REPLICA_REGISTERED->PROCESS_CACHE [2026-04-20 10:43:32] [VERBOSE] [378758/6] DTR 4f6d...bc6e: PROCESS_CACHE->PROCESSING_CACHE [2026-04-20 10:43:32] [INFO] [378758/24] DTR 4f6d...bc6e: Linking/copying cached file to /projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29/LDMX_pro-4.3.3.sif [2026-04-20 10:43:40] [VERBOSE] [378758/24] DTR 4f6d...bc6e: PROCESSING_CACHE->CACHE_PROCESSED [2026-04-20 10:43:40] [INFO] [378758/6] DTR 4f6d...bc6e: Finished successfully [2026-04-20 10:43:40] [VERBOSE] [378758/6] DTR 4f6d...bc6e: CACHE_PROCESSED->DONE [2026-04-20 10:43:40] [INFO] [378758/6] DTR 4f6d...bc6e: Returning to generator [2026-04-20 10:43:41] [INFO] [378758/7] DTR 4f6d...bc6e: e721710e1f29: All downloads finished successfully 2026-04-20T08:43:42Z Job state change PREPARING -> SUBMIT Reason: Pre-staging finished, passing job to LRMS ----- starting submit_slurm_job ----- SLURM jobname: LDMX_Prod_Simul SLURM job script built SLURM script follows: ------------------------------------------------------------------- #!/bin/bash -l # SLURM batch job script built by arex #SBATCH --no-requeue #SBATCH --export=NONE #SBATCH -e /projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29.comment #SBATCH -o /projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29.comment #SBATCH -p lu48 #SBATCH --nice=50 #SBATCH -J 'LDMX_Prod_Simul' #SBATCH --get-user-env #SBATCH -n 1 #SBATCH #SBATCH -t 1200:0 #SBATCH -t 1200:0 #SBATCH --mem-per-cpu=2000 #SBATCH -A lu2026-2-27 # Overide umask of execution node (sometime values are really strange) umask 077 # source with arguments for DASH shells sourcewithargs() { script=$1 shift . $script } # Record job start timestamp ACCOUNTING_STARTTIME=`date +"%s"` # Select accounting method echo "Detecting resource accounting method available for the job." 1>&2 JOB_ACCOUNTING="" # Try to use cgroups first if command -v arc-job-cgroup >/dev/null 2>&1; then echo "Found arc-job-cgroup tool: trying to initialize accounting cgroups for the job." 1>&2 while true; do # memory cgroup memory_cgroup="$( arc-job-cgroup -m -n e721710e1f29 )" if [ $? -ne 0 -o -z "$memory_cgroup" ]; then echo "Failed to initialize memory cgroup for accounting." 1>&2 break; fi # cpuacct cgroup cpuacct_cgroup="$( arc-job-cgroup -c -n e721710e1f29 )" if [ $? -ne 0 -o -z "$cpuacct_cgroup" ]; then echo "Failed to initialize cpuacct cgroup for accounting." 1>&2 break; fi echo "Using cgroups method for job accounting" 1>&2 JOB_ACCOUNTING="cgroup" break; done fi # Fallback to GNU_TIME if cgroups are not working if [ -z "$JOB_ACCOUNTING" ]; then GNU_TIME='/usr/bin/time' echo "Looking for $GNU_TIME tool for accounting measurements" 1>&2 if [ ! -z "$GNU_TIME" ] && ! "$GNU_TIME" --version >/dev/null 2>&1; then echo "GNU time is not found at: $GNU_TIME" 1>&2 else echo "GNU time found and will be used for job accounting." 1>&2 JOB_ACCOUNTING="gnutime" fi fi # Nothing works: rely on LRMS only if [ -z "$JOB_ACCOUNTING" ]; then echo "Failed to use both cgroups and GNU time for resource usage accounting. Accounting relies on LRMS information only." 1>&2 fi # Setting environment variables as specified by user export 'GRID_GLOBAL_JOBID=e721710e1f29' export 'GRID_GLOBAL_JOBURL=https://l-pilot2.lunarc.lu.se:443/arex/e721710e1f29' export 'GRID_GLOBAL_JOBINTERFACE=org.nordugrid.arcrest' export 'GRID_GLOBAL_JOBHOST=l-pilot2.lunarc.lu.se' RUNTIME_JOB_DIR=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29 RUNTIME_JOB_STDIN=/dev/null RUNTIME_JOB_STDOUT=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29/stdout RUNTIME_JOB_STDERR=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29/stdout RUNTIME_JOB_DIAG=/projects/hep/fs9/scratch/ldmx/ldcs/arcsession/e721710e1f29.diag if [ ! -z "$RUNTIME_GRIDAREA_DIR" ] ; then RUNTIME_JOB_DIR=$RUNTIME_GRIDAREA_DIR/`basename $RUNTIME_JOB_DIR` RUNTIME_JOB_STDIN=`echo "$RUNTIME_JOB_STDIN" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"` RUNTIME_JOB_STDOUT=`echo "$RUNTIME_JOB_STDOUT" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"` RUNTIME_JOB_STDERR=`echo "$RUNTIME_JOB_STDERR" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"` RUNTIME_JOB_DIAG=`echo "$RUNTIME_JOB_DIAG" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"` RUNTIME_CONTROL_DIR=`echo "$RUNTIME_CONTROL_DIR" | sed "s#^$RUNTIME_JOB_DIR#$RUNTIME_GRIDAREA_DIR#"` fi RUNTIME_LOCAL_SCRATCH_DIR=${RUNTIME_LOCAL_SCRATCH_DIR:-} RUNTIME_LOCAL_SCRATCH_MOVE_TOOL=${RUNTIME_LOCAL_SCRATCH_MOVE_TOOL:-mv} RUNTIME_FRONTEND_SEES_NODE=${RUNTIME_FRONTEND_SEES_NODE:-} RUNTIME_NODE_SEES_FRONTEND=${RUNTIME_NODE_SEES_FRONTEND:-yes} if [ ! -z "$RUNTIME_LOCAL_SCRATCH_DIR" ] && [ ! -z "$RUNTIME_NODE_SEES_FRONTEND" ]; then RUNTIME_NODE_JOB_DIR="$RUNTIME_LOCAL_SCRATCH_DIR"/`basename "$RUNTIME_JOB_DIR"` rm -rf "$RUNTIME_NODE_JOB_DIR" mkdir -p "$RUNTIME_NODE_JOB_DIR" # move directory contents for f in "$RUNTIME_JOB_DIR"/.* "$RUNTIME_JOB_DIR"/*; do [ "$f" = "$RUNTIME_JOB_DIR/*" ] && continue # glob failed, no files [ "$f" = "$RUNTIME_JOB_DIR/." ] && continue [ "$f" = "$RUNTIME_JOB_DIR/.." ] && continue [ "$f" = "$RUNTIME_JOB_DIR/.diag" ] && continue [ "$f" = "$RUNTIME_JOB_DIR/.comment" ] && continue [ -f "$f" ] || continue if ! $RUNTIME_LOCAL_SCRATCH_MOVE_TOOL "$f" "$RUNTIME_NODE_JOB_DIR"; then echo "Failed to '$RUNTIME_LOCAL_SCRATCH_MOVE_TOOL' '$f' to '$RUNTIME_NODE_JOB_DIR'" 1>&2 exit 1 fi done if [ ! -z "$RUNTIME_FRONTEND_SEES_NODE" ] ; then # creating link for whole directory ln -s "$RUNTIME_FRONTEND_SEES_NODE"/`basename "$RUNTIME_JOB_DIR"` "$RUNTIME_JOB_DIR" else # keep stdout, stderr and control directory on frontend # recreate job directory mkdir -p "$RUNTIME_JOB_DIR" # make those files mkdir -p `dirname "$RUNTIME_JOB_STDOUT"` mkdir -p `dirname "$RUNTIME_JOB_STDERR"` touch "$RUNTIME_JOB_STDOUT" touch "$RUNTIME_JOB_STDERR" RUNTIME_JOB_STDOUT__=`echo "$RUNTIME_JOB_STDOUT" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"` RUNTIME_JOB_STDERR__=`echo "$RUNTIME_JOB_STDERR" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"` rm "$RUNTIME_JOB_STDOUT__" 2>/dev/null rm "$RUNTIME_JOB_STDERR__" 2>/dev/null if [ ! -z "$RUNTIME_JOB_STDOUT__" ] && [ "$RUNTIME_JOB_STDOUT" != "$RUNTIME_JOB_STDOUT__" ]; then ln -s "$RUNTIME_JOB_STDOUT" "$RUNTIME_JOB_STDOUT__" fi if [ "$RUNTIME_JOB_STDOUT__" != "$RUNTIME_JOB_STDERR__" ] ; then if [ ! -z "$RUNTIME_JOB_STDERR__" ] && [ "$RUNTIME_JOB_STDERR" != "$RUNTIME_JOB_STDERR__" ]; then ln -s "$RUNTIME_JOB_STDERR" "$RUNTIME_JOB_STDERR__" fi fi if [ ! -z "$RUNTIME_CONTROL_DIR" ] ; then # move control directory back to frontend RUNTIME_CONTROL_DIR__=`echo "$RUNTIME_CONTROL_DIR" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"` mv "$RUNTIME_CONTROL_DIR__" "$RUNTIME_CONTROL_DIR" fi fi # adjust stdin,stdout & stderr pointers RUNTIME_JOB_STDIN=`echo "$RUNTIME_JOB_STDIN" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"` RUNTIME_JOB_STDOUT=`echo "$RUNTIME_JOB_STDOUT" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"` RUNTIME_JOB_STDERR=`echo "$RUNTIME_JOB_STDERR" | sed "s#^${RUNTIME_JOB_DIR}#${RUNTIME_NODE_JOB_DIR}#"` RUNTIME_FRONTEND_JOB_DIR="$RUNTIME_JOB_DIR" RUNTIME_JOB_DIR="$RUNTIME_NODE_JOB_DIR" fi if [ -z "$RUNTIME_NODE_SEES_FRONTEND" ] ; then mkdir -p "$RUNTIME_JOB_DIR" fi RESULT=0 if [ "$RESULT" = '0' ] ; then # RunTimeEnvironment function for APPS/LDMX-SIMPROD-3.0: RTE_function_0 () { LDMX_STORAGE_BASE="/projects/hep/fs9/shared/ldmx/ldcs/output/" KEEP_LOCAL_COPY="" # description: LDMX Production Simulation wrapper for aCT/Rucio # param:LDMX_STORAGE_BASE:string::local path to persistent storage base for LDMX data # param:KEEP_LOCAL_COPY:string::set to non-zero (suggest: 1) to keep a local copy of LDMX data (in local persistent storage) if [ "x$1" = "x0" ]; then if [ -z "$LDMX_STORAGE_BASE" ]; then echo "ERROR: ARC CE admin should define LDMX_STORAGE_BASE with arcctl rte params-set" >&2 exit 1 fi elif [ "x$1" = "x1" ]; then export LDMX_STORAGE_BASE export KEEP_LOCAL_COPY fi } # Running RTE scripts (stage 1) runtimeenvironments= runtimeenvironments="${runtimeenvironments}APPS/LDMX-SIMPROD-3.0;" # Calling APPS/LDMX-SIMPROD-3.0 function: RTE_function_0 1 if [ $? -ne 0 ]; then echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 1 execution failed." 1>&2 echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 1 execution failed." 1>>"${RUNTIME_JOB_STDERR}" exit 1 fi echo "runtimeenvironments=$runtimeenvironments" >> "$RUNTIME_JOB_DIAG" if [ ! "X$SLURM_NODEFILE" = 'X' ] ; then if [ -r "$SLURM_NODEFILE" ] ; then cat "$SLURM_NODEFILE" | sed 's/\(.*\)/nodename=\1/' >> "$RUNTIME_JOB_DIAG" NODENAME_WRITTEN="1" else SLURM_NODEFILE= fi fi # Detecting WN operating system for accounting purposes if [ -f "/etc/os-release" ]; then SYSTEM_SOFTWARE="$( eval $( cat /etc/os-release ); echo "${NAME} ${VERSION}" )" elif [ -f "/etc/system-release" ]; then SYSTEM_SOFTWARE="$( cat /etc/system-release )" elif command -v lsb_release >/dev/null 2>&1; then SYSTEM_SOFTWARE=$(lsb_release -ds) elif command -v hostnamectl >/dev/null 2>&1; then SYSTEM_SOFTWARE="$( hostnamectl 2>/dev/null | sed -n '/Operating System/s/^\s*Operating System:\s*//p' )" elif command -v uname >/dev/null 2>&1; then SYSTEM_SOFTWARE="Linux $( uname -r)" fi [ -n "$SYSTEM_SOFTWARE" ] && echo "systemsoftware=${SYSTEM_SOFTWARE}" >> "$RUNTIME_JOB_DIAG" if [ "$RESULT" = '0' ] ; then # Changing to session directory HOME=$RUNTIME_JOB_DIR export HOME if ! cd "$RUNTIME_JOB_DIR"; then echo "Failed to switch to '$RUNTIME_JOB_DIR'" 1>&2 RESULT=1 fi if [ ! -z "$RESULT" ] && [ "$RESULT" != 0 ]; then exit $RESULT fi # Write nodename if not already written in LRMS-specific way if [ -z "$NODENAME_WRITTEN" ] ; then nodename=`/bin/hostname -f` echo "nodename=$nodename" >> "$RUNTIME_JOB_DIAG" fi echo "Processors=1" >> "$RUNTIME_JOB_DIAG" echo "Benchmark=HEPSPEC:1.0" >> "$RUNTIME_JOB_DIAG" executable='./ldmxsim.sh' # Check if executable exists if [ ! -f "$executable" ]; then echo "Path \"$executable\" does not seem to exist" 1>$RUNTIME_JOB_STDOUT 2>$RUNTIME_JOB_STDERR 1>&2 exit 1 fi # See if executable is a script, and extract the name of the interpreter line1=$(dd if="$executable" count=1 2>/dev/null | head -n 1 | tr -d '\0') shebang=`echo $line1 | sed -n 's/^#! *//p'` interpreter=`echo $shebang | awk '{print $1}'` if [ "$interpreter" = /usr/bin/env ]; then interpreter=`echo $shebang | awk '{print $2}'`; fi # If it's a script and the interpreter is not found ... [ "x$interpreter" = x ] || type "$interpreter" > /dev/null 2>&1 || { echo "Cannot run $executable: $interpreter: not found" 1>$RUNTIME_JOB_STDOUT 2>$RUNTIME_JOB_STDERR 1>&2 exit 1; } if [ "x$JOB_ACCOUNTING" = "xgnutime" ]; then $GNU_TIME -o "$RUNTIME_JOB_DIAG" -a -f 'WallTime=%es\nKernelTime=%Ss\nUserTime=%Us\nCPUUsage=%P\nMaxResidentMemory=%MkB\nAverageResidentMemory=%tkB\nAverageTotalMemory=%KkB\nAverageUnsharedMemory=%DkB\nAverageUnsharedStack=%pkB\nAverageSharedMemory=%XkB\nPageSize=%ZB\nMajorPageFaults=%F\nMinorPageFaults=%R\nSwaps=%W\nForcedSwitches=%c\nWaitSwitches=%w\nInputs=%I\nOutputs=%O\nSocketReceived=%r\nSocketSent=%s\nSignals=%k\n' "./ldmxsim.sh" <$RUNTIME_JOB_STDIN 1>$RUNTIME_JOB_STDOUT 2>&1 else "./ldmxsim.sh" <$RUNTIME_JOB_STDIN 1>$RUNTIME_JOB_STDOUT 2>&1 fi RESULT=$? fi fi # Running RTE scripts (stage 2) runtimeenvironments= runtimeenvironments="${runtimeenvironments}APPS/LDMX-SIMPROD-3.0;" # Calling APPS/LDMX-SIMPROD-3.0 function: RTE_function_0 2 if [ $? -ne 0 ]; then echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 2 execution failed." 1>&2 echo "Runtime APPS/LDMX-SIMPROD-3.0 stage 2 execution failed." 1>>"${RUNTIME_JOB_STDERR}" exit 1 fi # Measuring used scratch space echo "usedscratch=$( du -sb "$RUNTIME_JOB_DIR" | sed "s/\s.*$//" )" >> "$RUNTIME_JOB_DIAG" # Cleaning up extra files in the local scratch if [ ! -z "$RUNTIME_LOCAL_SCRATCH_DIR" ] ; then find ./ -type l -exec rm -f "{}" ";" chmod -R u+w "./" chmod -R u-w "$RUNTIME_JOB_DIR"/'rucio.metadata' 2>/dev/null dynlist='output.files' chmod -R u-w "./$dynlist" 2>/dev/null cat "./$dynlist" | while read name rest; do chmod -R u-w "./$name" 2>/dev/null done chmod -R u-w "$RUNTIME_JOB_DIR"/'stdout' 2>/dev/null chmod -R u-w "$RUNTIME_JOB_DIR"/'stdout' 2>/dev/null chmod -R u-w "$RUNTIME_JOB_DIR"/'gmlog/' 2>/dev/null find ./ -type f -perm /200 -exec rm -f "{}" ";" chmod -R u+w "./" fi if [ ! -z "$RUNTIME_LOCAL_SCRATCH_DIR" ] && [ ! -z "$RUNTIME_NODE_SEES_FRONTEND" ]; then if [ ! -z "$RUNTIME_FRONTEND_SEES_NODE" ] ; then # just move it rm -rf "$RUNTIME_FRONTEND_JOB_DIR" destdir=`dirname "$RUNTIME_FRONTEND_JOB_DIR"` if ! mv "$RUNTIME_NODE_JOB_DIR" "$destdir"; then echo "Failed to move '$RUNTIME_NODE_JOB_DIR' to '$destdir'" 1>&2 RESULT=1 fi else # remove links rm -f "$RUNTIME_JOB_STDOUT" 2>/dev/null rm -f "$RUNTIME_JOB_STDERR" 2>/dev/null # move directory contents for f in "$RUNTIME_NODE_JOB_DIR"/.* "$RUNTIME_NODE_JOB_DIR"/*; do [ "$f" = "$RUNTIME_NODE_JOB_DIR/*" ] && continue # glob failed, no files [ "$f" = "$RUNTIME_NODE_JOB_DIR/." ] && continue [ "$f" = "$RUNTIME_NODE_JOB_DIR/.." ] && continue [ "$f" = "$RUNTIME_NODE_JOB_DIR/.diag" ] && continue [ "$f" = "$RUNTIME_NODE_JOB_DIR/.comment" ] && continue [ -f "$f" ] || continue if ! mv "$f" "$RUNTIME_FRONTEND_JOB_DIR"; then echo "Failed to move '$f' to '$RUNTIME_FRONTEND_JOB_DIR'" 1>&2 RESULT=1 fi done rm -rf "$RUNTIME_NODE_JOB_DIR" fi fi # Handle cgroup measurements if [ "x$JOB_ACCOUNTING" = "xcgroup" ]; then # Max memory used (total) maxmemory=$( cat "${memory_cgroup}/memory.memsw.max_usage_in_bytes" ) maxmemory=$(( (maxmemory + 1023) / 1024 )) echo "maxtotalmemory=${maxmemory}kB" >> "$RUNTIME_JOB_DIAG" # Max memory used (RAM) maxram=$( cat "${memory_cgroup}/memory.max_usage_in_bytes" ) maxram=$(( (maxram + 1023) / 1024 )) echo "maxresidentmemory=${maxram}kB" >> "$RUNTIME_JOB_DIAG" # TODO: this is for compatibilty with current A-REX accounting code. Remove when A-REX will use max value instead. echo "averageresidentmemory=${maxram}kB" >> "$RUNTIME_JOB_DIAG" # User CPU time if [ -f "${cpuacct_cgroup}/cpuacct.usage_user" ]; then # cgroup values are in nanoseconds user_cputime=$( cat "${cpuacct_cgroup}/cpuacct.usage_user" ) user_cputime=$(( user_cputime / 1000000 )) elif [ -f "${cpuacct_cgroup}/cpuacct.stat" ]; then # older kernels have only cpuacct.stat that use USER_HZ units user_cputime=$( cat "${cpuacct_cgroup}/cpuacct.stat" | sed -n '/^user/s/user //p' ) user_hz=$( getconf CLK_TCK ) user_cputime=$(( user_cputime / user_hz )) fi [ -n "$user_cputime" ] && echo "usertime=${user_cputime}" >> "$RUNTIME_JOB_DIAG" # Kernel CPU time if [ -f "${cpuacct_cgroup}/cpuacct.usage_sys" ]; then # cgroup values are in nanoseconds kernel_cputime=$( cat "${cpuacct_cgroup}/cpuacct.usage_sys" ) kernel_cputime=$(( kernel_cputime / 1000000 )) elif [ -f "${cpuacct_cgroup}/cpuacct.stat" ]; then # older kernels have only cpuacct.stat that use USER_HZ units kernel_cputime=$( cat "${cpuacct_cgroup}/cpuacct.stat" | sed -n '/^system/s/system //p' ) [ -z "$user_hz" ] && user_hz=$( getconf CLK_TCK ) kernel_cputime=$(( kernel_cputime / user_hz )) fi [ -n "$kernel_cputime" ] && echo "kerneltime=${kernel_cputime}" >> "$RUNTIME_JOB_DIAG" # Remove nested job accouting cgroups arc-job-cgroup -m -d arc-job-cgroup -c -d fi # Record CPU benchmarking values for WN user by the job [ -n "${ACCOUNTING_BENCHMARK}" ] && echo "benchmark=${ACCOUNTING_BENCHMARK}" >> "$RUNTIME_JOB_DIAG" # Record WN instance tag if defined [ -n "${ACCOUNTING_WN_INSTANCE}" ] && echo "wninstance=${ACCOUNTING_WN_INSTANCE}" >> "$RUNTIME_JOB_DIAG" # Record execution clock times ACCOUNTING_ENDTIME=`date +"%s"` # Mds date format (YYYYMMDDHHMMSSZ) echo "LRMSStartTime=`date -d "1970-01-01 UTC ${ACCOUNTING_STARTTIME} seconds" +"%Y%m%d%H%M%SZ"`" >> "$RUNTIME_JOB_DIAG" echo "LRMSEndTime=`date -d "1970-01-01 UTC ${ACCOUNTING_ENDTIME} seconds" +"%Y%m%d%H%M%SZ"`" >> "$RUNTIME_JOB_DIAG" echo "walltime=$(( ACCOUNTING_ENDTIME - ACCOUNTING_STARTTIME ))" >> "$RUNTIME_JOB_DIAG" # Add exit code to the accounting information and exit the job script echo "exitcode=$RESULT" >> "$RUNTIME_JOB_DIAG" exit $RESULT ------------------------------------------------------------------- job submitted successfully! local job id: 2913281 ----- exiting submit_slurm_job ----- 2026-04-20T08:43:44Z Job state change SUBMIT -> INLRMS Reason: Job is passed to LRMS ------- Contents of output stream forwarded by the LRMS --------- Detecting resource accounting method available for the job. Looking for /usr/bin/time tool for accounting measurements GNU time found and will be used for job accounting. ------------------------- End of output ------------------------- 2026-04-20T08:44:58Z Job state change INLRMS -> CANCELING Reason: Job failure detected 2026-04-20T08:44:58Z Job state change CANCELING -> FINISHING Reason: Job cancellation succeeded 2026-04-20T08:44:58Z Job state change FINISHING -> FINISHED Reason: Job failure detected