Incorrect dates returned with Monthly Time series #53
Comments
|
@scottRMA Can you please post the dss file? |
|
BugExample.zip |
|
The C structure for regular time series includes an attribute that represents the interval between data points in seconds. This value is constant and does not accommodate variations in intervals, such as those that change on a monthly or yearly basis. I will explore potential solutions to address this issue. |
|
This might be related to an issue I am facing, there is seemingly inconsistent behavior when reading monthly regular timeseries using wildcards in the D-part of the DSS path, vs when being explicit. Using The above results in the following (DLL printouts removed for clarity) Where the dataset shows as the following in DSS-Vue: Full print: |
|
I think DSSVue might be doing some special processing to handle the dates for 1MON intervals; or at the least the java layer that DSSVue uses. Technically within DSS any regular interval (e.g. not IR-...) is supposed to be a fix interval like what you're seeing but that's pretty silly for monthly and yearly intervals. I'll check the DSSVue and other Java code on Monday though and verify. If there's actually some specially logic I'll paste it here for use, if not just make a PR... Vaguely remember we discussed internally this or a similar issue (maybe it was weekly) but was focused on other projects at the time. |
|
The HEC-DSS API defines 1MON as 43,200 minutes, assuming a 30-day month. Therefore, we can consider the interval to be 1MON if the interval value is 43,200 minutes. My concern revolves around the acceptable date/times in a 1MON regular time series data. Are data entries such as 01JAN 0500 or 05JAN 2400 permissible, or is only the first day of the month with a timestamp of "0000" considered acceptable? Applying a similar rationale, we could interpret the interval as 1YEAR if the value is 525,600 minutes. It must be noted that data with intervals such as 2MON, 3YEAR, etc., cannot be sensibly handled and should be computed using the interval value literally. |
|
That might be roughly what the code is doing. I just found the section of code that reads the DSS times array and turns it into the java HecTime. There's a comment about "keeps tracks of months" but the logic itself is otherwise fairly opaque. protected void computeTimes(TimeSeriesContainer tsc)
{
int timeGranularitySeconds = tsc.timeGranularitySeconds;
if (timeGranularitySeconds == 0) timeGranularitySeconds = 60;
if ((tsc.interval > 0) && (tsc.times == null)) {
tsc.times = new int[tsc.numberValues];
// Using HecTime keeps track of months, etc.
// timeGranularitySeconds
if (timeGranularitySeconds == 1) {
for (int i=0; i<tsc.numberValues; i++) {
tsc.times[i] = tsc.startTime + (tsc.interval * i);
}
}
else {
//
int timeMultiplier = 86400 / timeGranularitySeconds;
int startJulianDate = tsc.startTime / timeMultiplier;
int startTimeSeconds = tsc.startTime - (startJulianDate * timeMultiplier);
startJulianDate += tsc.julianBaseDate;
int minutesSinceMidnight = startTimeSeconds/60;
HecTime htime = new HecTime();
htime.setJulian(startJulianDate, minutesSinceMidnight);
if (tsc.interval <= 1440) {
for (int i=0; i<tsc.numberValues; i++) {
tsc.times[i] = htime.value();
htime.addMinutes(tsc.interval);
}
}
else {
int julian[] = new int[1];
int minutes[] = new int[1];
int offset[] = new int[1];
julian[0] = htime.julian();
minutes[0] = htime.minutesSinceMidnight();
offset[0] = _recordData.timeOffset;
//Heclib.zofset (julian, minutes, tsc.interval, 2, offset);
int jul[] = new int[1];
int min[] = new int[1];
for (int i=0; i<tsc.numberValues; i++) {
HecTime.inctim (tsc.interval, i, julian[0], minutes[0], jul, min);
tsc.times[i] = (jul[0] * 1440) + min[0];
}
}
}
}Not sure if that helps or not; the "HecTime" object is documented as part of DssVue scripting. /**
* increments a Julian date and time a specified number of periods, based
* on a given time interval.
*
* @param interval
* (input) The time interval in minutes corresponding to the
* number of periods to increment the date and time by.
* @param numPeriods
* (input) The number of periods to increment the date and time
* by. May be a negative number to decrement the date and time.
* @param startJulian
* (input) The starting Julian date, in days since December 31,
* 1899.
* @param startTime
* (input) The starting time, in minutes past midnight.
* @param endJulian
* (output) The incremented Julian date, in days since December
* 31, 1899.
* @param endTime
* (output) The incremented time, in minutes past midnight.
*/
public static void inctim(int interval, int numPeriods, int startJulian,
int startTime, int[] endJulian, int[] endTime) {
//
if (endJulian == null || endJulian.length == 0) {
throw new RuntimeException("Parameter endJulian is null or zero-length");
}
if (endTime == null || endTime.length == 0) {
throw new RuntimeException("Parameter endTime is null or zero-length");
}
int julian;
int time;
int iymd[], jymd[];
iymd = new int[3];
jymd = new int[3];
//
int periodsPerYear = 0;
int daysPerPeriod = 0;
//
if (interval == 14400) {
// Tri-monthly
periodsPerYear = 36;
daysPerPeriod = 10;
} else if (interval == 21600) {
// Semi-monthly
periodsPerYear = 24;
daysPerPeriod = 15;
} else if (interval >= 40000 && interval < 45000) {
// Monthly
periodsPerYear = 12;
} else if (interval > 520000 && interval < 530000) {
// Yearly
periodsPerYear = 1;
}
//
if (periodsPerYear == 0) {
// This date is not an end-of-period date
if (interval % 1440 == 0) {
// Simple computation for daily intervals...
julian = startJulian + (interval / 1440) * numPeriods;
time = startTime;
} else {
// A bit more complex computation for non-daily interval...
int numDays = (numPeriods * interval) / 1440;
julian = startJulian + numDays;
time = startTime + (numPeriods * interval) - (numDays * 1440);
}
// Clean-up the time, if needed...
datcln(julian, time, endJulian, endTime);
time = endTime[0];
julian = endJulian[0];
} else {
//
//This date requires an end-of-period date
time = startTime;
//
jliymd(startJulian, iymd);
//
int jjul = startJulian + 1;
//
jliymd(jjul, jymd);
//
boolean isEndOfMonth;
int offset = 0;
//
if ( iymd[1] != jymd[1] ) {
isEndOfMonth = true;
} else {
isEndOfMonth = false;
// get the day offse, e.g., the 4th's offset would be 4
if (periodsPerYear >= 24) {
offset = iymd[2] % daysPerPeriod;
}
// If Feb 28, check for leap year
if ((jymd[1] == 2) && (jymd[2] == 29) && (interval >= 1440)) {
isEndOfMonth = true;
}
}
//
// Determine number of years to increment
//
jymd[0] = numPeriods / periodsPerYear;
iymd[0] = iymd[0] + jymd[0];
//
// Determine number of months to increment
//
int leftOver = 0;
//
if (periodsPerYear >= 12) {
jymd[1] = ((12 * numPeriods) / periodsPerYear)
- (jymd[0] * 12);
iymd[1] = iymd[1] + jymd[1];
if (periodsPerYear > 12) {
leftOver = numPeriods
- ((jymd[1] * periodsPerYear) / 12)
- (jymd[0] * periodsPerYear);
if (leftOver < 0) {
leftOver = leftOver + periodsPerYear / 12;
iymd[1]--;
}
}
}
//
// Clean up months
//
while( true ){
if (iymd[1] > 12) {
iymd[0]++;
iymd[1] = iymd[1] - 12;
}else if(iymd[1] < 1) {
iymd[0]--;
iymd[1] = iymd[1] + 12;
}else{
break;
}
}
//
if (periodsPerYear <= 12) {
//
// Determine day portion for monthly and yearly intervals
//
if (isEndOfMonth) {
iymd[1]++;
julian = iymdjl(iymd[0], iymd[1], 1) - 1;
} else {
julian = iymdjl(iymd[0], iymd[1], iymd[2]);
}
} else {
//
// Determine day portion for semi and tri monthly
//
if (leftOver == 0) {
if (isEndOfMonth) {
iymd[1]++;
julian = iymdjl(iymd[0], iymd[1], 1) - 1;
} else {
julian = iymdjl(iymd[0], iymd[1], iymd[2]);
}
} else {
for (int i = 1; i <= leftOver; i++) {
int numDayMon = iymdjl(iymd[0], iymd[1] + 1, 1)
- iymdjl(iymd[0], iymd[1], 1);
if (numDayMon > 30)
numDayMon = 30;
iymd[2] = iymd[2] + daysPerPeriod;
if (iymd[2] >= numDayMon) {
iymd[1]++;
julian = iymdjl(iymd[0], iymd[1], 1) - 1
+ offset;
if ((iymd[2] > 32) && (offset == 0)) {
julian = julian + daysPerPeriod;
}
jliymd(julian, iymd);
if ((iymd[2] >= numDayMon) && (i != leftOver)) {
iymd[2] = 0;
iymd[1]++;
}
}
}
julian = iymdjl(iymd[0], iymd[1], iymd[2]);
}
}
}
if (time == 0) {
time = 1440;
julian--;
}
endJulian[0] = julian;
endTime[0] = time;
}
|
|
Please provide your feedback regarding the outlined modifications: |
|
It looks reasonable, but I think I'm going to have to chew on it a while, perhaps even talk to Karl. |
|
I have implemented the logic as suggested earlier in version 2.3.2. It seems to be functioning correctly. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello, when I read in monthly time series, I am not getting my actual dates returned. Instead, I am getting dates that are exactly 30 days apart. Attached is the tabulated values of my monthly timeseries in DSSVue
I am reading in the values in python using the following calls
where my start_time is DateTime(2020, 1, 1, 0, 0, 0) and my end_time is DateTime(2020, 12, 31, 0, 0 ,0). When I use ts.pytimes, these are the values I get back
[datetime.datetime(2020, 1, 1, 0, 0), datetime.datetime(2020, 1, 31, 0, 0), datetime.datetime(2020, 3, 1, 0, 0), datetime.datetime(2020, 3, 31, 0, 0), datetime.datetime(2020, 4, 30, 0, 0), datetime.datetime(2020, 5, 30, 0, 0), datetime.datetime(2020, 6, 29, 0, 0), datetime.datetime(2020, 7, 29, 0, 0), datetime.datetime(2020, 8, 28, 0, 0), datetime.datetime(2020, 9, 27, 0, 0), datetime.datetime(2020, 10, 27, 0, 0), datetime.datetime(2020, 11, 26, 0, 0), datetime.datetime(2020, 12, 26, 0, 0)]I get similar results even when changing my start and end times, and changing the record to a different monthly record also gets me the same results. When I convert the values to daily, and then read them in, the correct values are at the correct times. This also happens reading from a DSS6 vs DSS7 file.
Additional info:
python version: 3.9.13
DSSVue version: 3.3.25
pydsstools version: 2.3.1
The text was updated successfully, but these errors were encountered: