-
Notifications
You must be signed in to change notification settings - Fork 29
Update ACS BLEVCORR routines to accommodate new supported subarrays #312
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update ACS BLEVCORR routines to accommodate new supported subarrays #312
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some comments provided. I cannot approve or otherwise, as I don't think I am in the position to do so. As long as regression tests pass, should be okay? You might also need to add new test cases.
* | ||
* 16-May-2018 M.D. De La Pena: Generalized bias_shift_corr() to handle subarray data. | ||
*/ | ||
int bias_shift_corr(ACSInfo *acs, int nGroups, ...) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, this is magical! 🔮
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...So much easier and more obvious with an OO language.
/* For full frame data, there are four amps in use. For supported subarray | ||
* data, there is only one amp in use. The actual amp in use must be determined. | ||
*/ | ||
int numAmpsInUse = (acs->subarray == YES) ? 1 : NAMPS; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is numAmpsInUse = NAMPS
wrong for subarray cases? What about subarrays with 2 amps? For example, AD
or BC
. Will those never get to this point in the pipeline?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NAMPS is hard-coded to 4 which nicely accommodates full frame data. However, the new, "supported" WFC subarrays are single amp (512, 1K, or 2K). This may have to be updated once the POL and RAMP data are incorporated into this new algorithm. All other WFC subarrays will use the old blev algorithm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This may have been worth hiding within a func, in case the logic becomes more complicated in the future.
for (k = 13; k <= 22; k++) { | ||
sum += ampdata[arr_cols*j + k]; | ||
num++; | ||
for (j = 2057; j <= 2066; j++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even prior to this PR, I was uncomfortable with these magic numbers. Perhaps it is time to make them constant variables with meaningful names?
This comment also applies to other occurrences of magic numbers in this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is already in progress. An updated overscan table has been created with primary header keywords and a new binary table in the second extension to accommodate amp-dependent values.
pkg/acs/calacs/acsccd/doccd.c
Outdated
@@ -32,6 +35,8 @@ static void dqiMsg (ACSInfo *, int); | |||
static void BiasMsg (ACSInfo *, int); | |||
static void BlevMsg (ACSInfo *, int); | |||
static void SinkMsg (ACSInfo *, int); | |||
/* These are the subarray supported apertures for the bias shift correction */ | |||
static const char *subApertures[] = {"WFC1A-2K", "WFC1B-2K", "WFC2C-2K", "WFC2D-2K"}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you ever want to add polarizers, keep in mind that the aperture name never changed after flight software update, so same POL aperture is "old subarray" before that update, and "new subarray" after. Becareful with those.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Understood. The POL and RAMP data have to be checked for actual size and also possibly date as the discriminants.
pkg/acs/calacs/acsccd/doccd.c
Outdated
return (status); | ||
***FIX MDD: What about polarizer and ramp apertures? | ||
*/ | ||
Bool supportedSubAperture = NO; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel that this should be inside the logic for WFC only, because this same code is also used to process HRC.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI: I've been trying to move stuff over to using stdbool.h::bool
instead.
pkg/acs/calacs/acsccd/doccd.c
Outdated
|
||
} /* End of only WFC detector */ | ||
|
||
/* All HRC and SBC data - use the original blev correction */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SBC data don't get processed here. This is only for CCD, not MAMA. (Please correct me if I misremember.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ooops. I got a bit too inclusive in my comments - here there be no SBC.
b401ee9
to
0fd4ffe
Compare
Polarized and ramp data are now handled by this upgrade. The new WFC 512 and 1K subarrays are not yet accommodated by this upgrade as they do not have virtual overscan, so there is no "magic square" data to use. The team needs to determine an alternative to the "magic square". The additional work to handle the WFC 512 and 1K subarrays will be handled by a new issue. |
0fd4ffe
to
63cbd7d
Compare
Nate (ACS Team) sent on 07 August 2018: |
@pllim Can you please re-review? Basically, I added a lot of code to (1) handle all the cases for the full frame and subarray data with regard to the bias shift correction, and (2) free memory. I will note the logic for application of the bias shift correction has been expanded at the request of the ACS Team lead. The code is now rather verbose and repetitive, but it is more approachable to an astronomer. |
63cbd7d
to
6d5dc0d
Compare
|
||
/* Initializing arguments to store all the values being passed to this routine via "..." */ | ||
va_start (arguments, nGroups); | ||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why the extra scope?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps you want to scope arguments
, which is a very good idea, in which case this needs to be moved up just above va_list arguments;
and the closing, just after va_end (arguments);
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If that was not your intention, I would scope the va_list
.
* the physical overscan and must be divided by 2. If this is a subarray, then "nx" represents | ||
* just the columns for a single amp plus physical overscan. | ||
*/ | ||
const int arr_rows = sg[0]->sci.data.ny; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sg[0]
is being accessed without first being checked. This can be done here, or better still when first assigned to sg
so as to shrink it to only valid ptrs (bookkeeping required of nGroups
etc).
/* sg[0] is chip2 and sg[1] is chip 1 for a full frame image */ | ||
{unsigned int i; | ||
for (i = 0; i < nGroups; i++) { | ||
sg[i]= va_arg (arguments, SingleGroup *); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You may want to check that they're not NULL
here, see comment below.
* | ||
* 16-May-2018 M.D. De La Pena: Generalized bias_shift_corr() to handle subarray data. | ||
*/ | ||
int bias_shift_corr(ACSInfo *acs, int nGroups, ...) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add assert(acs);
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also:
if (nGroups < 1)
return HSTCAL_OK; // or NOTHING_TO_DO, NO_GOOD_DATA, INTERNAL_ERROR
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, if going for INTERNAL_ERROR
I would instead assert
. This is a much better way to handle internal errors as it gives the src line in question.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tend to treat this kind of thing either as a NOP and would return HSTCAL_OK
or assert
, upto you to think about the function's semantics.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes.
* just the columns for a single amp plus physical overscan. | ||
*/ | ||
const int arr_rows = sg[0]->sci.data.ny; | ||
const int arr_cols = (acs->subarray == NO) ? sg[0]->sci.data.nx/2 : sg[0]->sci.data.nx; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rename arr_cols
more semantically specific, e.g. ampArrCols
etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI: (I know the mix already existed) but I don't like the mix of camel case and underscores.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keeping the arr_cols vs ampArrCols as the remaining functions in this file also use this variable name for a total of 87 name changes. I want to keep the functions consistent in their terminology.
* just the columns for a single amp plus physical overscan. | ||
*/ | ||
const int arr_rows = sg[0]->sci.data.ny; | ||
const int arr_cols = (acs->subarray == NO) ? sg[0]->sci.data.nx/2 : sg[0]->sci.data.nx; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding the logic: I'm only now getting the impression that the singleGroups
being passed in are actually amps. Is this correct? So what does this func act on amps or...? If amps, then nx
should already be correct, right? Then why the /2
?
/* arrays below are in order of amp A, B, C, D */ | ||
|
||
/* time constant of AC high pass filter in external pre-amp. */ | ||
/* time constant of AC high pass filter in external pre-amp in microseconds */ | ||
const double time_const[NAMPS] = {37.290251, 36.180001, 37.867770, 42.461249}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If in microsecs, reflect this in the param name, e.g. usecTimeConst
.
/* array for amp data and amp data + gap pixels */ | ||
double *ampdata = malloc(arr_rows * arr_cols * sizeof(double)); | ||
addPtr(&ptrReg, ampdata, &free); | ||
double *ampdata_gap = malloc(nquad_pix * sizeof(double)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All of the above malloc
s need to be NULL
checked handling OUT_OF_MEMORY
errors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed. Went to great trouble to add these checks in other routines, but I neglected to add the checks here.
|
||
/* allocate space for true DC bias levels array */ | ||
|
||
PtrRegister ptrReg; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
*/ | ||
int ampInUse = 0; | ||
{unsigned int i; | ||
for (i = 0; i < numAmpsInUse; i++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ampdata[arr_cols*j + k] = ampdata_gap[(arr_cols + ngap_pix)*j + k]; | ||
} | ||
} | ||
/* copy corrected data back to ampdata */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it not possible to combine this op within the loop above (correction comp) and assign the correction directly to ampdata
? This would be more efficient.
|
||
int SameInt (int, int); | ||
int SameString (char *, char *); | ||
|
||
int foundit = NO; | ||
*overscan = NO; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good job moving this to before the return.
return (status); | ||
return (status); | ||
{unsigned int i; | ||
for (i=0; i < strlen(acs->aperture); i++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
indentation
|
||
if (GetKeyStr (phdr, "APERTURE", USE_DEFAULT, "", acs->aperture, ACS_CBUF)) | ||
return (status); | ||
return (status); | ||
{unsigned int i; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be a func toUpper()
that excepts a string. E.g. use this utility in cfitsio Convert a character string to uppercase (operates in place). void fits_uppercase / ffupch (char *string)
.
return (status); | ||
{unsigned int i; | ||
for (i=0; i < strlen(acs->aperture); i++) { | ||
acs->aperture[i] = toupper (acs->aperture[i]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is only acceptable/valid if it is "illegal" for the char set to be [a-z], if not it must be removed. Even if it is I really don't think it is this function's responsibility to do so.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Imagine someone wanting to write a validity wrapper to GetACSKeys()
, they now can't because the functionality/semantics has been mangled.
|
||
int to_electrons(ACSInfo *, SingleGroup *); | ||
int doBias (ACSInfo *, SingleGroup *); | ||
int biasHistory (ACSInfo *, Hdr *); | ||
int doBlev (ACSInfo *, SingleGroup *, int, float *, int *, int *); | ||
int bias_shift_corr(ACSInfo *, SingleGroup *, SingleGroup *); | ||
int bias_shift_corr(ACSInfo *, int, ...); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be in a header. It should also have a comment describing the variadic arg.
/* set up an array of SingleGroups to hold all image extensions */ | ||
SingleGroup * x = NULL; /* used for both input and output */ | ||
x = (SingleGroup *) malloc(acs_info->nimsets * sizeof(SingleGroup)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would have been good to have taken the opportunity and removed the cast.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and also changed sizeof(SingleGroup)
-> sizeof(*x)
x = (SingleGroup *) malloc(acs_info->nimsets * sizeof(SingleGroup)); | ||
if (!x) | ||
return (status = OUT_OF_MEMORY); | ||
addPtr(&ptrReg, x, &free); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is ok as is because the freeing happens in reverse order of that added. However, I'm wondering if this is "ok". Hmmm...
return (status); | ||
} | ||
|
||
/* Read in keywords from primary header... */ | ||
|
||
if (GetKeyBool(x[i].globalhdr, "SUBARRAY", NO_DEFAULT, 0, &subarray)) | ||
if (GetKeyBool(x[i].globalhdr, "SUBARRAY", NO_DEFAULT, 0, &subarray)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if ((status = GetKeyBool(x[i].globalhdr, "SUBARRAY", NO_DEFAULT, 0, &subarray)))
|
||
/* Get overscan region information from OSCNTAB */ | ||
int * overscan = NULL; | ||
int * virtOverscan = NULL; | ||
overscan = (int *) malloc(acs_info->nimsets * sizeof(int)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For maintenance reasons, it's better to remove the cast and also use sizeof(*overscan)
} | ||
addPtr (&ptrReg, overscan, &free); | ||
|
||
virtOverscan = (int *) malloc(acs_info->nimsets * sizeof(int)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Likewise
This code is verbose and potentially harder to maintain, but the logic is clearer | ||
for the scientist reading the code. | ||
|
||
NOTE: The variable "done" is overloaded. When done = overscan[i], this means |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't we un-overload and instead of any if (done)
use if (blevDone || overscan[i])
? Not 100% certain of ||
vs &&
as I haven't read any of the below logic.
int driftcorr = NO; /* true means bias level was corrected for drift */ | ||
Bool isSupportedSubAperture = NO; | ||
float meanblev = 0.0; /* mean value of overscan bias (for history) */ | ||
if (acs_info->blevcorr == PERFORM) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (!OmitStep(acs_info->blevcorr))
int driftcorr = NO; /* true means bias level was corrected for drift */ | ||
Bool isSupportedSubAperture = NO; | ||
float meanblev = 0.0; /* mean value of overscan bias (for history) */ | ||
if (acs_info->blevcorr == PERFORM) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would pull this entire block out into a sep func. This would help clear up the crowded logic. Same prob goes for all the other steps.
} | ||
|
||
/* Only Pre-SM4 WFC data - this is the original bias level subtraction */ | ||
if (acs_info->expstart < SM4MJD) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A this stage of me reading this huge logic block(s) I would reduce all of them to calls. This would make things infinitely clearer.
E.g.
if (acs_info->expstart < SM4MJD)
doSM4MJD(...);
else
doPostSM4MJD(...);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The nesting could also be described as a comment here at the root level. Maybe even inline e.g.
if (acs_info->expstart < SM4MJD)
doSM4MJD(...); // -> if (subaray)
// doSub();
else
doPostSM4MJD(...);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, perhaps it might be possible (and better?) to use a switch and order such that the logic exists as case stmts dropping through?
*/ | ||
if (virtOverscan[0]) { | ||
isSupportedSubAperture = YES; | ||
trlmessage("This subarray data is supported for the bias shift correction."); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you want a break
here?
|
||
/* The variable done is set to the results of FindOver indicating there is | ||
physical overscan. */ | ||
done = overscan[i]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Making this a note since you didn't edit doBlev()
. It's better to not overload params, i.e. separate those that are input and those that are output. In this case it's done
. This should either be past in (as a const) or a ptr so that it can be modified as output, but never both.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the time given, I was not able to compare the logic to make a remark on whether it does the same as it does before (for the parts that we want this to be true for (whichever those are?)). Or even, whether it now functionally satisfies the issue it addresses.
That being said I don't think there are currently any bugs though it has introduced scope for future dev work to easily produce some, e.g. lack of NULL
checking and variadic usage, etc.
Requesting changes, at least, on the grounds of the malloc
s needing to be addressed. And, ptrs being accessed before being checked.
Addressing the mallocs and checking the sg[] pointers before actual use. New IT #334 opened which refer to the remaining issues in this PR. |
…new bias level processing which previously only applied to full frame data.
b91615a
to
d2ef21f
Compare
/* This function has built in assumptions regarding the data being passed */ | ||
assert (acs != NULL); /* Make sure the ACSInfo structure is defined */ | ||
assert ((nGroups == 1) || (nGroups == 2)); /* Must be at least one and no more than two SingleGroups */ | ||
assert (((nGroups == 1) && (acs->subarray == YES)) || ((nGroups == 2) && (acs->subarray == NO))); /* Subarray has one and full frame has two SingleGroups */ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just so I've said it - this and the line above are ugly. The variadic arg should have been removed instead and these asserts handled by the compiler (at compile time).
for (i = 0; i < nGroups; i++) { | ||
sg[i] = NULL; | ||
sg[i] = va_arg (arguments, SingleGroup *); | ||
if (!sg[i]) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not what I had in mind. I meant that if va_arg (arguments, SingleGroup *) == NULL
it just wouldn't be added, i.e.
SingleGroup * tmpGroup = va_arg (arguments, SingleGroup *);
if (tmpGroup)
sg[i] = tmpGroup;
else
continue;
You could argue that in such a case nGroups
is actually incorrect, it is unclear of the use of the function params now that they are variadic. I.e. are they explicit slots, e.g. SingleGroup * amp2, SingleGroup * amp1
or is the order irrelevant and in the case that there is only one, it must always be the first, e.g. not passed as NULL, amp1
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the rest of the code assumes a specific ordering.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TBH I'd prefer not to approve this, however, the issues introduced will be addressed by #334 and there is significant pressure to merge this by today for the next release. I think the band-aids now applied should be temporarily sufficient. Oh well, here we go...
Addresses IT #168 (EDIT: Fixed linking)
The new supported subarrays (circa mid-2016) need to have the improved bias level processing applied during CALACS processing akin to what is done for the full frame data. In detail only the bias shift correction is applicable to subarray data. This is currently a work in progress in that only the new WFC 2K subarrays are handled. The WFC 512 and 1K subarrays do not have virtual overscan, so there is no "magic square" data. This issue will be addressed when an alternative to the "magic square" is defined. Also, the polarizer and ramp subarrays are not yet handled.
It should be noted the block of code handling the logic of which BLEV algorithm (original or new and improved) should be applied and the data appropriate for each algorithm has been re-written for clarity and at the request of the team lead. While it is more verbose, the logic is cleaner.