forked from DIRACGrid/DIRAC
-
Notifications
You must be signed in to change notification settings - Fork 0
/
release.notes
6850 lines (4994 loc) · 235 KB
/
release.notes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
[v6r20-pre23]
*Core
NEW: MJF utility added, providing a general interface to Machine/Job Features values.
NEW: DEncode - added unit tests
NEW: JEncode for json based serialization
NEW: Add conditional printout of the traceback when serializing/deserializing non json compatible
object in DEncode (enabled with DIRAC_DEBUG_DENCODE_CALLSTACK environment variable)
NEW: File.py - utility to convert file sizes between different unit
NEW: new flag in dirac-install script to install DIRAC-OS on demand
CHANGE: Removed deprecated option "ExtraModules" (dirac-configure, dirac-install scripts)
CHANGE: dirac-deploy-scripts, dirac-install - allow command modules with underscores in
their names in order for better support for the code checking tools
CHANGE: dirac-distribution and related scripts - compile web code while release
generation
CHANGE: dirac-external-requirements - reimplemented to use preinstalled pip command rather than
the pip python API
FIX: dirac-distribution - fixed wrong indentation
NEW: new command name for voms proxy
FIX: dirac-install default behaviour preserved even with diracos options
New: Add additional check in MJF utility to look for a shutdown file located at '/var/run/shutdown_time'
FIX: The hardcoded rule was not taken into account when the query was coming from the web server
CHANGE: VOMSService - reimplemented using VOMS REST interface
FIX: MJF utility won't theow exceptions when MJF is not fully deployed at a site
*Framework
NEW: WebAppCompiler methods is implemented, which is used to compile the web framework
NEW: add JsonFormatter for logs
NEW: add default configuration to CS: only TrustedHost can upload file
CHANGE: ComponentInstaller - remove the old web portal configuration data
used during the installation
CHANGE: MessageQueue log handler uses JsonFormatter
*Monitoring
CHANGE: fixes for testing in Jenkins with locally-deployed ElasticSearch
FIX: fixes in the query results interpretation
*Configuration
FIX: ConfigurationHandler, PilotCStoJSONSynchronizer - fixes for enabling pilotCStoJSONSynchronizer, and doc
NEW: dirac-admin-voms-sync - command line for VOMS to CS synchronization
NEW: VOMS2CSSynchronizer - new class encapsulating VOMS to CS synchronization
CHANGE: VOMS2CSAgent - reimplemented to use VOMS2CSSynchronizer
*WorkloadManagementSystem
NEW: StopSigRegex, StopSigStartSeconds, StopSigFinishSeconds, StopSigNumber added to JDL, which cause Watchdog to send a signal StopSigNumber to payload processes matching StopSigRegex when there are less than StopSigFinishSeconds of wall clock remaining according to MJF.
NEW: PilotLoggingDB, Service and Client for handling extended pilot logging
NEW: added a new synchronizer for Pilot3: sync of subset of CS info to JSON file,
and sync of pilot3 files
NEW: dirac-admin-get-pilotslogging script for viewing PilotsLogging
NEW: Bulk job submission with protection of the operation transaction
NEW: WMSHistoryCorrector and MonitoringHistoryCorrector classes inheriting from a common BaseHistoryCorrector class
CHANGE: SiteDirector - refactored Site Director for better extensibility
CHANGE: dirac-wms-cpu-normalization uses the abstracted DB12 benchmark script used by the HEPiX Benchmarking Working Group, and the new MJF utility to obtain values from the system and to save them into the DIRAC LocalSite configuration.
CHANGE: Removed TaskQueueDirector and the other old style (WMS) *PilotDirector
CHANGE: TaskQueueDB - removed PilotsRequirements table
CHANGE: TaskQueueDB - added FOREIGN KEYS
CHANGE: Removed gLite pilot related WMS code
FIX: always initialize gPilotAgentsDB object
FIX: JobManager - Added some debug message when deleting jobs
FIX: Job.py - fixing finding XML file
NEW: SiteDirector - added flag for sending pilot3 files
CHANGE: SiteDirector - changed the way we create the pilotWrapper (better extensibility)
NEW: SiteDirector - added possibility for deploying environment variables in the pilot wrapper
*Workflow
CHANGE: Script.py: created _exitWithError method for extension possibilities
*TS
FIX: TranformationCleaningAgent - just few simplifications
*DMS
NEW: FTS3Agent working only with the FTS3 service to replace the existing one
NEW: FTS3Utilities - use correct FTS Server Selection Policy
NEW: StorageElement service - getFreeDiskSpace() and getTotalDiskSpace() take into account
MAX_STORAGE_SIZE parameter value
CHANGE: Adding vo name argument for StorageElement
CHANGE: Fixing rss to fetch fts3 server status
NEW: Add a feature to the DFC LHCbManager to dump the content of an SE as a CSV file
FIX: FTS3DB: sqlalchemy filter statements with "is None" do not work and result in no lines being selected
NEW: FTS3Agent and FTS3DB: add functionality to kick stuck requests and delete old requests
*RMS
FIX: Really exit the RequestExecutingAgent when the result queue is buggy
*RSS
CHANGE: Using StorageElement.getOccupancy()
FIX: Initialize RPC to WMSAdministrator only once
FIX: Using MB as default for the size
FIX: flagged some commands that for the moment are unusable
FIX: fixed documentation of how to develop commands
*Resources
NEW: New SingularityComputingElement to submit jobs to a Singularity container
NEW: Added StorageElement.getOccupancy() method for DIP and GFAL2_SMR2 SE types
CHANGE: enable Stomp logging only if DIRAC_DEBUG_STOMP environment variable is set to any value
*Interfaces
CHANGE: Dirac.py - saving output of jobs run with 'runLocal' when they fail (for DEBUG purposes)
*Docs
CHANGE: WebApp release procedure
FIX: Drop some old stuff, add link for FTS3 page
*Tests
FIX: add MonitoringDB to the configuration
FIX: Installing elasticSeach locally in Jenkins, with ComponentInstaller support.
[v6r19p21]
*Configuration
FIX: Bdii2CSAgent - make the GLUE2 information gathering less verbose; Silently ignore StorageShares
*Test
CHANGE: backported some of the CI tools from the integration branch
[v6r19p20]
*StorageManagement
FIX: StorageManagementDB - fixed buggy group by with MySQL 5.7
[v6r19p19]
*Configuration
NEW: BDII2CSAgent - new options: GLUE2URLs, if set this is queried in addition to the other BDII;
GLUE2Only to turn off looking on the old schema, if true only the main BDII URL is queried;
Host to set the BDII host to search
NEW: dirac-admin-add-resources new option G/glue2 , enable looking at GLUE2 Schema,
H/host to set the host URL to something else
[v6r19p18]
*Configuration
CHANGE: Better logging of the Configuration file write exception
*RSS
FIX: SummarizeLogsAgent - fix the case when no previous history
[v6r19p17]
*Framework
FIX: ProxyManager - if an extension has a ProxyDB, use it
*RSS
FIX: CSHelpers.py minor fixes
[v6r19p16]
*WMS
FIX: pilotCommands - cast maxNumOfProcs to an int.
CHANGE: pilotTools - change maxNumOfProcs short option from -P to -m.
[v6r19p15]
*Framework
NEW: ProxyDB - allow FROM address to be set for proxy expiry e-mails
*DMS
CHANGE: FTSJob - FailedSize is now BIGINT in FTSJob
CHANGE: FTSJob - increase the bringonline time
*WMS
FIX: SiteDirector won't set CPUTime of the pilot
FIX: convert MaxRAM inside the pilots to int
*RSS
FIX: SummarizeLogsAgent: comparison bug fix
FIX: Fixed sites synchronizer
[v6r19p14]
*WMS
NEW: pilotCommands/Tools - added possibility to specify a maxNumberOfProcessors parameter for pilots
CHANGE: MultiProcessorSiteDirector - allow kwargs to SiteDirector getExecutable & _getPilotOptions functions
*RMS
FIX: Fix a bug in ReplicateAndRegister Operation preventing files having failed once to be retried
*DMS
FIX: FileCatalogWithFkAndPsDB.sql - Fixes for the DFC to be compatible with strict group by mode
(https://dev.mysql.com/doc/refman/5.7/en/sql-mode.html#sqlmode_only_full_group_by)
*docs
CHANGE: added little documentation for lcgBundles
[v6r19p13]
*WMS
FIX: JobWrapper - added a debug message
FIX: Allow non-processor related tags to match TQ in MultiProcessorSiteDirector.
*Test
CHANGE: improve Gfal2 integration tests by checking the metadata
[v6r19p12]
*Core
CHANGE: QualityMapGraph - change the color map of the Quality plots
*Framework
FIX: Logging - remove the space after log messages if no variable message is printed, fixes #3587
*MonitoringSystem
CHANGE: ElasticSearch 6 does not support multiple types, only one type is created instead.
*RSS
FIX: GOCDBClient - encode in utf-8, update goc db web api URL
FIX: fixed bug in creation of history of status (avoid repetition of entries)
*DMS
FIX: fixed bug in FTSAgent initialization
*WMS
FIX: fix bug in dirac-wms-job-select: treating the case of jobGroup(s) not requested
[v6r19p11]
*Framework:
CHANGE: moved column "Instance" of InstalledComponentsDB.InstalledComponent
table from 64 to 32 characters
*WMS
FIX: JobWrapperTemplate - fix exception handling
CHANGE: dirac-wms-select-jobs - new option to limit the number of selected jobs
CHANGE: returning an error when sandboxes can't be unassigned from jobs (JobCleaningAgent)
*RMS
FIX: RequestDB - add missing JOIN in the web summary query
NEW: dirac-rms-request - add option to allow resetting the NotBefore member even
for non-failed requests
*DMS
FIX: FTSAgent - change data member names from uppercase to lower case
*Interfaces
CHANGE: autopep8 on the API/Dirac module
*docs:
NEW: added some doc about shifterProxy
[v6r19p10]
*Core
FIX: MySQL - catch exception when closing closed connection
*TS
CHANGE: add possibility to get extension-specific tasks and files statuses in TransformationMonitor web application
*RMS
NEW: dirac-rms-request - add option --ListJobs to list the jobs for a set of requests
*Resources
FIX: Use parameters given at construction for SRM2 protocols List
*StorageManagement
FIX: use StorageElement object to get disk cache size
*DMS
FIX: DMSHelpers - fix case when no site is found for an SE
FIX: ReplicateAndRegister - don't try and get SE metadata is replica is inactive
[v6r19p9]
*WMS
CHANGE: DownloadInputData was instantiating all local SEs which is not necessary... Only instantiate those that are needed
CHANGE: JobWrapper - use resolveSEGroup in order to allow defining SE groups including other SE groups
FIX: JobDB - fixed typo in getSiteMaskStatus() method
FIX: Fix getSiteMaskStatus in SiteDirector and MultiProcessSiteDirector
CHANGE: WatchdogLinux - using python modules in instead of shell calls
*DMS
FIX: in DMSHelpers don't complain if an SE is at 0 sites
*Interfaces
CHANGE: Job.py - using the deprecated decorator for 2 deprecated methods
*RSS
FIX: EmailAction considers also CEs, not only SEs
*Resources
FIX: removed a useless/broken method in Resources helper
FIX: marked as obsoleted two methods in Resources helper (FTS2 related)
[v6r19p8]
*Configuration
FIX; Resources - don't overwrite queue tags if requiredtags are set.
*Framework
CHANGE: dirac-proxy-init - increase dirac-proxy-init CRL update frequency
*Accounting
CHANGE: AccountingDB - if the bucket length is part of the selected conditions,
add to the grouping
*WorkloadManagement
FIX: ConfigTemplate.cfg - allow user access to getSiteMaskStatus
*DataManagementSystem
FIX: DMSHelpers - recursive resolution of SEGroup was keeping the SEGroup in the list
*RSS
FIX: CSHelper - getting FTS from the correct location
CHANGE: use the SiteStatus object wherever possible
*Resources
FIX: CREAMComputingElement - added CS option for extra JDL parameters
*Documentation
CHANGE: point README to master and add badges for integration
[v6r19p7]
*WorkloadManagement
FIX: SiteDirector - correct escaping in pilot template
FIX: dirac-wms-get-wn-parameters - added some printouts to dirac-wms-get-wn-parameters
[v6r19p6]
*Core
FIX: SocketInfo - log proper message on CA's init failure.
*Accounting
CHANGE: NetworkAgent - remove support of perfSONAR summaries and add support of raw metrics.
*WMS
FIX: JobDB - don't trigger exception in webSummary if a site with a single dot is in the system
CHANGE: SiteDirector - added logging format and UTC timestamp to pilot wrapper
FIX: JobMonitoring - fix in getJobPageSummaryWeb() for showing correct sign of life for stalled jobs
*TS
FIX: TransformationManager - fix for wrong method called by the Manager
*RSS
NEW: SiteStatus object uses the RSS Cache
FIX: expiration time is a date (dirac-rss-query-db)
[v6r19p5]
*WMS
CHANGE: ParametricJob - added getParameterVectorLength() to replace getNumberOfParameters with a more detailed check of the job JDL validity
FIX: JobManagerHandler - restored the use of MaxParametricJobs configuration option
*Interfaces
FIX: Always use a list of LFNs for input data resolution (local run, mostly)
*tests
FIX: use rootPath instead of environment variable
[v6r19p4]
NEW: Added dummy setup.py in anticipation for standard installation procedure
*Core
CHANGE: SocketInfoFactory - version check of GSI at run time is removed
*Configuration
FIX: Resources - fix RequiredTags in getQueue() function
*Interfaces
FIX: fix exception when using Dirac.Job.getJobJDL
*WMS
FIX: SiteDirector - fix proxy validity check in updatePilotStatus, a new proxy was
never created because isProxyValid returns non-empty dictionary
FIX: JobMonitoring - web table was not considering correctly Failed jobs because
stalled for setting the LastSignOfLife
*DMS
FIX: StorageFactory - avoid complaining if Access option is not in SE section
CHANGE: dirac-dms-user-lfns - the wildcard flag will always assume leading "*" to match files,
unless the full path was specified in the wildcard no files were previously matched
*RSS
FIX: CacheFeederAgent resilient to command exceptions
*Resources
FIX: ARCComputingElement - the proxy environment variable was assumed before the
return value of the prepareProxy function was checked, which could lead to exceptions
[v6r19p3]
CHANGE: .pylintrc - disable redefined-variable-type
CHANGE: .pylintrc - max-nested-blocks=10 due to the many tests of result['OK']
CHANGE: use autopep8 for auto-formatting with following exceptions:
tabs = 2 spaces and not 4
line length check disabled (i.e. 120 characters instead of 80)
Option for autopep8 are: --ignore E111,E101,E501
*Configuration
FIX: retrigger the initialization of the logger and the ObjectLoader after
all the CS has been loaded
*WMS
FIX: pilot commands will add /DIRAC/Extensions=extensions if requested
FIX: SiteDirector, pilotCommands - fix support for multiple values in the
RequiredTag CE parameter
FIX: MultiProcessorSiteDirector - fix dictionary changed size exception
*Workflow
FIX: application log name can also come from step_commons.get['logFile']
*Resources
CHANGE: Condor, SLURM, SSHComputingElement - added parameters to force allocation
of multi-core job slots
[v6r19p2]
*DMS
FIX: dirac-admin-allow-se: fix crash because of usage of old RSS function
*RSS
FIX: ResourceStatusDB - microseconds should always be 0
FIX: Multiple fixes for the RSS tests
[v6r19p1]
*Core
FIX: ElasticSearchDB - certifi package was miscalled
FIX: ElasticSearchDB - added debug messages for DB connection
*Framework
FIX: ComponentInstaller - handling correctly extensions of DBs found in sql files
*WMS
FIX: SudoComputingElement - prevent message overwriting application errors
FIX: JobDB.getInputData now returns list of cleaned LFNs strings, possible "LFN:"
prefix is removed
*Interfaces
FIX: Dirac.py - bring back treatment of files in working local submission directory
[v6r19]
FIX: In multiple places - use systemCall() rather than shellCall() to avoid
potential shell injection problems
FIX: All Databases are granting also REFERENCES grant to Dirac user to comply with
more strict policies of MySQL version >= 5.7
*Accounting
NEW: new functionality to plot the data gathered by perfSONARs. It allows to
present jitter, one-way delay, packet-loss rate and some derived functions.
FIX: compatibility of AccountingDB with MySQL 5.7
*ConfigurationSystem
NEW: Allow to define FailoverURLs and to reference MainServers in the URLs
*FrameworkSystem
NEW: gLogger is replaced by the new logging system based on the python logging module
NEW: Added ElasticSearch backend for the logging
NEW: Central Backends configuration to customize their use by multiple components
NEW: BundleDelivery - serves also CA's and CRL's all-in-one files
NEW: added shell scripts for generating CAs and CRLs with the possibility to specify the Input and/or output directories
CHANGE: can now send mails to multiple recipients using the NotificationClient
CHANGE: Make the new logging system thread-safe
FIX: Adapting query to MySLQ 5.7 "GROUP BY" clause
FIX: TopErrorMessagesReporter - more precise selection to please stricter versions of MySQL
CHANGE: ProxyGeneration - make RFC proxies by default, added -L/--legacy flag to dirac-proxy-init
to force generation of no-RFC proxies
*Core
FIX: dirac-install - allow to use local md5 files
CHANGE: X509Chain - fixes to allow robot proxies with embedded DIRAC group extension
( allow DIRAC group extension not in the first certificate chain step )
CHANGE: BaseClient - recheck the useServerCertificate while establishing connection
and take it into account even if it has changed after the client object creation
FIX: PlainTransport - fixed socket creation in initAsClient()
NEW: Technology preview of new logging system, based on standard python logging module
CHANGE: Added graphviz extension to sphinx builds
FIX: Added documentation of low level RPC/DISET classes
FIX: Gateway service - multiple fixes to resurrect the service and to correctly instantiate it
NEW: dirac-install will change the shebang of the python scripts to use the environment
python instead of the system one
NEW: Security.Utilities - methods to generate all-in-one CA certificates and CRLs files
NEW: ElasticSearchDB - gets CA's all-in-one file from the BundleDelivery service if needed
NEW: genAllCAs.sh, genRevokedCerts.sh - DIRAC-free commands to generate all-in-one CA
certificates and CRLs files
CHANGE: dirac-create-distribution-tarball - removing docs and tests directories when
creating release tarballs
*DMS
CHANGE: FTSJob - use Request wrapper for the fts3 REST interface instead of pycurl based
client
CHANGE: FTSHistoryView - drop FTSServer field from the view description
CHANGE: FTSFile DB table: increased length of fields LFN(955), SourceSURL(1024), TargetSURL(1024)
CHANGE: Uniform length of LFN to 255 across DIRAC dbs
FIX: FTSJob - fix the serialization of 0 values
FIX: FTSFile, FTSJob - fix SQL statement generation for stricter versions of MySQL
*Resources
NEW: New method in the StorageElement to generate pair of URLs for third party copy.
Implement the logic to generate pair of URLs to do third party copy.
This will be used mostly by FTS, but is not enabled as of now
FIX: StorageElement - fix different weird behaviors in Storage Element, in particular,
the inheritance of the protocol sections
FIX: GFAL2 storage element: update for compatibility with GFAL2 2.13.3 APIs
NEW: Introduced Resources/StorageElementBases configuration section for definitions
of abstract SEs to be used in real SEs definition by inheritance
*RMS
NEW: dirac-rms-request - command including functionality of several other commands:
dirac-rms-cancel|reset|show-request which are dropped. The required functionality
is selected by the appropriate switches
*RSS
NEW: Put Sites, ComputingElements, FTS and Catalogs under the status control of the
RSS system
NEW: Rewrote RsourceStatus/ResourceManagementDB tables with sqlAlchemy (RM DB with declarative base style)
NEW: SiteStatus client to interrogate site status with respect to RSS
CHANGE: introduced backward compatibility of RSS services with DIRAC v6r17 clients
CHANGE: moved some integration tests from pytest to unittest
CHANGE: Moved ResourceStatusDB to sqlAlchemy declarative_base
FIX: Automated setting of lastCheckTime and Dateffective in ResourceStatusDB and ResourceManagementDB
FIX: fixes for tables inheritance and extensions
FIX: fixes for Web return structure ("meta" column)
FIX: ResourceStatus, RSSCacheNoThread - fixed RSS cache generation
FIX: ResourceStatus - fixes for getting status from the CS information
FIX: ResourceManagement/StatusDB - fixed bugs in meta parameter check
FIX: fixed incompatibility between Active/InActive RSS clients return format
FIX: SiteStatus - bug fixed in getSites() method - siteState argument not propagated to
the service call
FIX: ResourceStatus - return the same structure for status lookup in both RSS and CS cases
FIX: Bug fixes in scripts getting data out of DB
*Monitoring
CHANGE: DBUtils - change the bucket sizes for the monitoring plots as function of the time span
*WMS
NEW: SiteDirector - checks the status of CEs and Sites with respect to RSS
NEW: pilotCommands - new ReplaceDIRACCode command mostly for testing purposes
NEW: JobAgent, JobWrapper - several fixes to allow the work with PoolComputingElement
to support multiprocessor jobs
NEW: JobScheduling - interpret WholeNode and NumberOfProcessors job JDL parameters and
convert then to corresponding tags
NEW: SiteDirector - CEs can define QueryCEFlag in the Configuration Service which can be
used to disallow querying the CE status and use information from PiltAgentsDB instead
NEW: The application error codes, when returned, are passed to the JobWrapper, and maybe interpreted.
NEW: The JobWrapperTemplate can reschedule a job if the payload exits with status DErrno.EWMSRESC & 255 (222)
FIX: SiteDirector - unlink is also to be skipped for Local Condor batch system
FIX: JobDB - fixes necessary to suite MySQL 5.7
FIX: dirac-pilot, pilotTools - PYTHONPATH is cleared on pilot start, pilot option keepPP
can override this
FIX: WMSAdministratorHandler - make methods static appropriately
FIX: Bug fix for correctly excluding WebApp extensions
CHANGE: JobScheduling - more precise site name while the job is Waiting, using the set of
sites at which the input files are online rather than checking Tier1s in eligible sites
FIX: SiteDirector - aggregate tags for the general job availability test
FIX: JobScheduling - bug fix in __sendToTQ()
FIX: pilotTools,pilotCommands - pick up all the necessary settings from the site/queue configuration
related to Tags and multi-processor
NEW: SiteDirector - added option to force lcgBundle version in the pilot
FIX: SiteDirector - if MaxWaitingJobs or MaxTotalJobs not defined for a queue, assume a default value of 10
FIX: MatcherHandler - preprocess resource description in getMatchingTaskQueues()
FIX: JobDB - set CPUTime to a default value if not defined when rescheduling jobs
*TS
FIX: TransformationClient - fix issue #3446 for wrong file error counting in TS
FIX: TransformationDB - set ExternalID before ExternalStatus in tasks
BUGFIX: TransformationClient - fix a bug in the TS files state machine (comparing old status.lower()
with new status)
*Interfaces
CHANGE: Dirac API - expose the protocol parameter of getAccessURL()
CHANGE: Dirac API - added runLocal as an API method
*Docs
NEW: Documentation for developing with a container (includes Dockerfile)
NEW: Add script to collate release notes from Pull Request comments
NEW: Chapter on scaling and limitations
CHANGE: Added documentation about runsv installation outside of DIRAC
*tests
NEW: Added client (scripts) system test
CHANGE: Add to the TS system test, the test for transformations with meta-filters
FIX: Minor fixes in the TS system test
FIX: correctly update the DFC DB configuration in jenkins' tests
[v6r17p35]
*Core
FIX: GOCDBClient - add EXTENSIONS & SCOPE tag support to GOCDB service queries.
[v6r17p34]
*SMS
FIX: StorageManagerClient - fix logic for JobScheduling executor when CheckOnlyTapeSEs is
its default true and the lfn is only on a tapeSE
[v6r17p33]
*WMS
FIX: StalledJobAgent - if no PilotReference found in jobs parameters, do as if there would be
no pilot information, i.e. set Stalled job Failed immediately
CHANGE: DownloadInputData - job parameters report not only successful downloads but also failed ones
FIX: JobDB - back port - set CPUTime to 0 if not defined at all for the given job
FIX: JobDB - back port - use default CPUTime in the job description when rescheduling jobs
*Resources
FIX: ARCComputingElement - fix job submission issue due to timeout for newer lcg-bundles
[v6r17p32]
Resources:
CHANGE: /Computing/BatchSystems/Condor.py: do not copy SiteDirector's shell environment variables into the job environment
*WMS
CHANGE: Add option to clear PYTHONPATH on pilot start
[v6r17p31]
*RMS
FIX: ReqClient - avoid INFO message in client
*WMS
CHANGE: JobWrapper - allow SE-USER to be defined as another SE group (e.g. Tier1-USER)
*DMS
CHANGE: DMSHelpers - make resolveSEGroup recursive in order to be able to define SE groups in terms of SE groups
[v6r17p30]
*DMS
CHANGE: StorageElement - added status(), storageElementName(), checksumType() methods returning
values directly without the S_OK structure. Remove the checks of OK everywhere
NEW: dirac-dms-add-file, DataManager - added option (-f) to force an overwrite of an existing file
*TS:
FIX: TransformationDB.py - set the ExternalID before the ExternalStatus in order to avoid inconsistent
tasks if setting the ExternalID fails
*StorageManagementSystem
FIX: StorageManagementClient.py - return the full list of onlineSites while it was previously happy
with only one
*Resources
FIX: HTCondorCEComputingElement.py - transfer output files(only log and err) for remote scheduler
[v6r17p29]
*WMS
CHANGE: split time left margins in cpuMargin and wallClockMargin. Also simplified check.
[v6r17p28]
*WMS
BUGFIX: JobScheduling - fix a bug introduced in 6r17p27 changes
*Monitoring
BUGFIX: MonitoringReporter - do not try to close the MQ connection if MD is not used
[v6r17p27]
*Configuration
FIX: ConfigurationClient - allow default value to be a tuple, a dict or a set
*Monitoring
CHANGE: DBUtils - change bucket sizes and simplify settings
*DMS
FIX: DMSRequestOperationsBase, RemoveFile - allow request to not fail if an SE is temporarily banned
FIX: dirac-admin-allow-se - first call of gLogger after its import
*RMS
CHANGE: remove scripts dirac-rms-show-request, dirac-rms-cancel-request and dirac-rms-reset-request
and replace with a single script dirac-rms-request with option (default is "show")
CHANGE: allow script to finalize a request if needed and set the job status appropriately
*Resources
FIX: LocalComputingElement - pilot jobIDs start with ssh to be compatible with pilotCommands.
Still original jobIDs are passed to getJobStatus. To be reviewed
*WMS
CHANGE: JobScheduling - assign a job to Group.<site>.<country>, if input files are at <site>.<country>.
If several input replicas, assign Waiting to "MultipleInput"
[v6r17p26]
*Core
FIX: dirac-install.py to fail when installation of lcgBundle has failed
FIX: ClassAdLight - getAttributeInt() and getAttributeFloat() return None
if the corresponding JDL attribute is not defined
*MonitoringSystem
CHANGE: The Consumer and Producer use separate connections to the MQ;
If the db is not accessible, the messaged will not be consumed.
*WMS
FIX: JobDB - fix the case where parametric job placeholder %j is used in the JobName attribute
FIX: JobDB - take into account that ClassAdLight methods return None if numerical attribute is not defined
FIX: ParametricJob utility - fixed bug in evaluation of the ParameterStart|Step|Factor.X job numerical attribute
[v6r17p25]
*Monitoring
NEW: Implemented the support of monthly indexes and the unit tests are fixed
*RMS
FIX: RequestExecutingAgent - fix infinite loop for duplicate requests
*WMS
NEW: ARCComputingElement - add support for multiprocessor jobs
[v6r17p24]
*WMS
FIX: SiteDirector - unlink is also to be skipped for Local Condor batch system
[v6r17p23]
*WMS
FIX: get job output for remote scheduler in the case of HTCondorCE
[v6r17p22]
*Framework
FIX: NotificationClient - added avoidSpam flag to sendMail() method which is propagated to
the corresponding service call
*Integration
FIX: several fixes in integration testing scripts
[v6r17p21]
*Core
NEW: Mail.py - added mechanism to compare mail objects
FIX: Grid.py - take into account the case sometimes happening to ARC CEs
where ARC-CE BDII definitions have SubClusters where the name isn't set to
the hostname of the machine
*Framework
FIX: Notification service - avoid duplicate emails mechanism
[v6r17p20]
*Core
NEW: API.py - added __getstate__, __setstate__ to allow pickling objects inheriting
API class by special treatment of internal Logger objects, fixes #3334
*Framework
FIX: SystemAdministrator - sort software version directories by explicit versions in the
old software cleaning logic
FIX: MonitoringUtilities - sets a suitable "unknown" username when installing DIRAC from scratch,
and the CS isn't initialized fully when running dirac-setup-site
CHANGE: Logger - added getter methods to access internal protected variables, use these methods
in various places instead of access Logger protected variables
*WMS
CHANGE: JobDB - removed unused CPUTime field in the Jobs table
CHANGE: JobScheduling - make check for requested Platform among otherwise eligible sites
for a given job, fail jobs if no site with requested Platform are available
*RSS
FIX: Commands - improved logging messages
*SMS
FIX: StorageManagerClient - instantiate StorageElement object with an explicit vo argument,
fixes #3335
*Interfaces
NEW: dirac-framework-self-ping command for a server to self ping using it's own certificate
[v6r17p19]
*Core
FIX: Adler - fix checksum with less than 8 characters to be 8 chars long
*Configuration
FIX: VOMS2CSAgent - fix to accomodate some weird new user DNs (containing only CN field)
*DMS
FIX: FileCatalog - fix for the doc strings usage in file catalog CLI, fixes #3306
FIX: FileCatalog - modified recursive file parameter setting to enable usage of the index
*SMS
CHANGE: StorageManagerClient - try to get sites with data online if possible in getFilesToStage
*RMS
FIX: RequestExecutingAgent - tuning of the request caching while execution
*WMS
FIX: DownloadInputData - do not mistakenly use other metadata from the replica info than SEs
FIX: JobScheduling - put sites holding data before others in the list of available sites
FIX: JobScheduling - try and select replicas for staging at the same site as online files
FIX: SiteDirector - keep the old pilot status if the new one can not be obtained in updatePilotStatus()
*Resources
FIX: CREAMComputingElement - return error when pilot output is missing in getJobOutput()
*Monitoring
FIX: DBUtils - change the buckets in order to support queries which require more than one year
data. The maximum buckets size is 7 weeks
[v6r17p18]
*Framework
NEW: SystemAdministrator - added possibility to remove old software installations keeping
only a predefined number of the most recent ones.
*DMS
FIX: RemoveReplica - removing replica of a non-existing file is considered successful
*SMS
CHANGE: StorageManagerClient - restrict usage of executeWithUserProxy decorator
to calling the SE.getFileMetadata only; added flag to check only replicas
at tape SEs
*WMS
FIX: JobScheduling - added CS option to flag checking only replicas at tape SEs;
fail jobs with input data not available in the File Catalog
[v6r17p17]
*DMS
NEW: FTSAgent has a new CS parameter ProcessJobRequests to be able to process job
requests only. This allows to run 2 FTS agents in parallel
*Resources
FIX: GFAL2_StorageBase - only set the space token if there is one to avoid problems
with some SEs
[v6r17p16]
*Configuration
FIX: VOMS2CSAgent - create user home directory in the catalog without
recursion in the chown command
*RMS
FIX: RequestExecutingAgent - catch error of the cacheRequest() call
FIX: ReqClient - enhanced log error message
*SMS
FIX: StorageManagerClient - treat the case of absent and offline files on an SE
while staging
*TS
FIX: TaskManagerBase - process tasks in chunks of 100 in order to
update faster the TS (tasks and files)
*WMS
FIX: JobScheduling - do not assume that all non-online files required staging
[v6r17p15]
*WMS
CHANGE: StalledJobAgent - ignore or prolong the Stalled state period for jobs
at particular sites which can be suspended, e.g. Boinc sites
[v6r17p14]
*Core
FIX: PrettyPrint.printTable utility enhanced to allow multi-row fields and
justification specification for each field value
*Accounting
NEW: DataStore - allow to run several instances of the service with only one which
is enabled to do the bucketing
*RMS
NEW: new dirac-rms-list-req-cache command to list the requests in the ReqProxies services
*Interfaces
CHANGE: Dirac API - make several private methods visible to derived class
[v6r17p13]
*Core
NEW: Proxy - added executeWithoutServerCertificate() decorator function
*Resources
FIX: CREAMComputingElement - split CREAM proxy renewal operation into smaller chunks for
improved reliability
[v6r17p12]
*Framework
FIX: SecurityFileLog - when the security logs are rotated, the buffer size is reduced
to 1 MB to avoid gzip failures ( was 2 GBs )
*WMS
FIX: pilotCommands - fix for interpreting DNs when saving the installation environment
FIX: SandboxStoreClient - do not check/make destination directory if requested sandbox
is returned InMemory
*TS
FIX: TransformationAgent CS option MaxFiles split in MaxFilesToProcess and MaxFilesPerTask,
MaxFiles option is interpreted as MaxFilesPerTask for backward compatibility
*Resources
NEW: Added plug-ins for GSIFTP and HTTPS Storage protocols
[v6r17p11]
*Core
FIX: ElasticSearchDB - set a very high number (10K) for the size of the ElasticSearch result
*Monitoring
FIX: MonitoringDB - et a very high number (10K) for the size of the ElasticSearch result
*WMS
FIX: pilotCommands - get the pilot environment from the contents of the bashrc script
*DMS
FIX: RemoveReplica - fix for the problem that if an error was set it was never reset
FIX: SE metadata usage in several components: ConsistencyInspector, DataIntwgrityClient,
FTSRequest, dirac-dms-replica-metadata, StageMonitorAgent, StageRequestAgent,
StorageManagerClient, DownloadInputData, InputDataByProtocol
[v6r17p10]
*Core
NEW: Logger - printing methods return True/False if the message was printed or not
FIX: ElastocSearchDB - error messages demoted to warnings
*Monitoring
FIX: MonitoringReporter - create producers if the CS definitions are properly in place
*TS
CHANGE: TaskManagerPlugin - allow to redefine the AutoAddedSites for each job type
[v6r17p9]
*WMS
BUGFIX: JobScheduling - bug fixed introduced in the previous patch
NEW: pilotTools - introduced -o swicth for a generic CS option
*SMS
FIX: StorageManagerClient - fixes in the unit test
*DMS
FIX: FileManagerPs - in _getFileLFNs() - break a long list of LFNs into smaller chunks
[v6r17p8]
*Core
NEW: DErrno.ENOGROUP error to denote proxies without DIRAC group extension embedded
CHANGE: X509Chain - use DErrno.ENOGROUP error
FIX: dirac-install, dirac-deploy-scripts - fixes to allow DIRAC client installation on
recent MacOS versions with System Integrity Protection feature
CHANGE: Proxy - added executionLock optional argument to executeWithUserProxy() decorator
to lock while executing the function with user proxy
FIX: Proxy - fix indentation in getProxy() preventing looping on the DNs
*Framework
FIX: ProxyDB - fix of error message check in completeDelegation()
*WMS
FIX: TaskQueueDB - when an empty TaskQueue is marked for deletion, it can still get matches
which result in no selected jobs that produced unnecessary error messages
FIX: JobScheduling executor - calls getFilesToStage() with a flag to lock while file lookup
with user proxy; same for InputData executor for calling _resolveInputData()
*TS
FIX: FileReport - fix in setFileStatus() for setting status for multiple LFNs at once
*SMS
FIX: StorageManagerClient - in getFilesToStage() avoid using proxy if no files to check
on a storage element
*Resources
FIX: GFAL2_XROOTStorage - fix to allow interactive use of xroot plugin
FIX: GFAL2_StorageBase - enable IPV6 for gsiftp
[v6r17p7]
*DMS
FIX: dirac-dms-user-lfns - do not print out empty directories
*WMS
FIX: InputData Executor, JobWrapper - use DataManager.getReplicasForJobs() for
getting input data replicas
*TS
FIX: TransformationAgent - use DataManager.getReplicasForJobs() for transformations
creating jobs
[v6r17p6]
*DMS
NEW: DataManager - add key argument forJobs (default False) in getReplicas() in order
to get only replicas that can be used for jobs (as defined in the CS); added
getReplicasForJobs(), also used in the Dirac API
*SMS
FIX: Stager agents - monitor files even when there is no requestID, e.g. dCache returns None
when staging a file that is already staged
*Resources
FIX: StorageFactory - bug fixes when interpreting SEs inheriting other SE parameters
NEW: Test_StorageFactory unit test and corresponding docs
FIX: Torque - some sites put advertising in the command answer that can not be parsed:
redirect stderr to /dev/null
[v6r17p5]
*Resources
FIX: LcgFileCatalogClient - do not evaluate GUID if it is not a string