Timestamp must be non-decreasing for series attribute

[lvxhnat]

Describe the bug

I have referred to issue #733 and #1327 but none has seemed to offer a viable solution so far for the same issue, even with the latest versions of neptune. Has there been an update on possible solutions on logging in multiprocessing/threaded processes? Suppressing error logs seems like a hacky way to go about doing this.

I am currently running neptune on a kedro environment via tensorflow keras, initialising the Neptune client on my Kedro node. But am still getting the error logs.

arning: string series 'monitoring/bb773e00/stdout' value was longer than 1000 characters and was truncated. This warning is printed only once per series.
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:28.341Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:28.514Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:28.686Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:28.857Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.031Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.031Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.203Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.376Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.547Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.719Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:29.890Z                                       
                    ERROR    Error occurred during asynchronous    async_operation_processor.py:272
                             operation processing: Timestamp must                                  
                             be non-decreasing for series                                          
                             attribute:                                                            
                             monitoring/bb773e00/stdout. Invalid                                   
                             point: 2023-08-28T12:22:30.062Z      

Code i used below

def train_and_evaluate_resnet_model(
    generators: List[MATGenerator],
    params,
):

    class LearningRateMonitor(tf.keras.callbacks.Callback):
        def on_train_begin(self, logs={}):
            self.lrates = list()

        def on_epoch_end(self, epoch, logs={}):
            # get and store the learning rate
            optimizer = self.model.optimizer
            lrate = float(tf.keras.backend.get_value(self.model.optimizer.lr))
            self.lrates.append(lrate)


    reduce_lr = tf.keras.callbacks.ReduceLROnPlateau(
        monitor="val_loss",
        factor=0.1,
        patience=1,
        verbose=1,
        mode="max",
        min_delta=0.0001,
        cooldown=0,
        min_lr=0,
    )

    early_stop = tf.keras.callbacks.EarlyStopping(
        monitor="val_loss",
        min_delta=0.001,
        patience=5,
        restore_best_weights=True,
        verbose=1,
    )

    run = neptune.init_run()
    neptune_callback = NeptuneCallback(run=run)

    n_classes, residual_blocks, block_filters = params["n_classes"], params["residual_blocks"], params["block_filters"]
    
    resnet_model = generate_resnet_model(
        n_classes=n_classes,
        residual_blocks=residual_blocks,
        block_filters=block_filters,
    )

    evaluations = []

    for train_generator, val_generator, test_generator in generators:

        resnet_model.fit(
            train_generator,
            validation_data=val_generator,
            epochs=30,
            batch_size=64,
            callbacks=[reduce_lr, early_stop, neptune_callback],
            verbose=1,
        )

        evaluations.append(resnet_model.evaluate(test_generator))

    return evaluations

[SiddhantSadangi]

Hello @lvxhnat 👋

Unless you have important metadata in our stdout stream, the above error should not be of much relevance. Since we log console output as a series, the above error prevents logging of streams with a timestamp <= the previously logged timestamp in the same field (monitoring/<hash>/stdout): https://docs.neptune.ai/help/error_step_must_be_increasing/

If you do want to fix this, could you please share a reproducible code sample that we can run end-to-end to see where we've multiple threads writing to the same field?
Please also share the output of pip list, and your system details (number of parallel processes, GPUs, etc).

It would also help us a lot if you could share the link to a run on the Neptune app where you encountered this error. If you are not comfortable sharing this on GitHub, you can send it over to us at [email protected], or through the chat at the bottom right of the Neptune app.

[lvxhnat]

@SiddhantSadangi Got it, I will turn off the warnings for now in this case. Thank you.