IndexError: list index out of range

[deepuak01]

I get the below error from great-expectations=0.14.10

Traceback (most recent call last):
  File "/mnt/tmp/spark-1b3d4863-c2e5-4665-baca-3f5b4514259e/EPI_SPARK_GE_POC.py", line 122, in <module>
    expectation_suite_name="bod_epi_expectation_suit"
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 1870, in get_validator
    batch_list=batch_list,
  File "/usr/local/lib/python3.7/site-packages/great_expectations/data_context/data_context.py", line 1880, in get_validator_using_batch_list
    batch_definition: BatchDefinition = batch_list[-1].batch_definition
IndexError: list index out of range

The above error is printed when control enters below code:

validator = ge_context.get_validator(
    batch_request=batch_request,
    expectation_suite_name="bod_epi_expectation_suit"   
)

Please find below my code. Note that i am running this code from EMR 6.3.0:

from pyspark.sql import SparkSession
from great_expectations.checkpoint import SimpleCheckpoint
from great_expectations.core.batch import BatchRequest
from great_expectations.data_context import BaseDataContext
from great_expectations.data_context.types.base import DataContextConfig, DatasourceConfig

spark = SparkSession.builder.appName('GE-POC').getOrCreate()

print("**********starting GE code************")

data_context_config = DataContextConfig(
    config_version=2,
    plugins_directory=None,
    config_variables_file_path=None,
    datasources={
        "dataframe_epi_source": DatasourceConfig(
            class_name="Datasource",
            execution_engine={
                "class_name": "SparkDFExecutionEngine"
            },
            data_connectors={
                "dataframe_epi_data_connector": {
                    "class_name": "InferredAssetS3DataConnector",                    
                    "bucket": "adj-raw",
                    "prefix": "dev/Source/",
                    "default_regex": {
                       "group_names": [
                            "epi",
                        ],
                       "pattern": r"epi(.*)\.csv",
                    },
                   
                }
            },
        )
    },
    stores={
        "expectations_S3_store": {
            "class_name": "ExpectationsStore",
            "store_backend": {
                "class_name": "TupleS3StoreBackend",
                "bucket": "adj-output",
                "prefix": "expectation",
            },
        },
        "validations_S3_store": {
            "class_name": "ValidationsStore",
            "store_backend": {
                "class_name": "TupleS3StoreBackend",
                "bucket": "adj-output",
                "prefix": "validation",
            },
        },
        "evaluation_parameter_store": {"class_name": "EvaluationParameterStore"},
    },
    expectations_store_name="expectations_S3_store",
    validations_store_name="validations_S3_store",
    evaluation_parameter_store_name="evaluation_parameter_store",
    data_docs_sites={
        "s3_site": {
            "class_name": "SiteBuilder",
            "store_backend": {
                "class_name": "TupleS3StoreBackend",
                "bucket":  "adj-output",
                "prefix":  "docs",
            },
            "site_index_builder": {
                "class_name": "DefaultSiteIndexBuilder",
                "show_cta_footer": True,
            },
        }
    },
    validation_operators={
        "action_list_operator": {
            "class_name": "ActionListValidationOperator",
            "action_list": [
                {
                    "name": "store_validation_result",
                    "action": {"class_name": "StoreValidationResultAction"},
                },
                {
                    "name": "store_evaluation_params",
                    "action": {"class_name": "StoreEvaluationParametersAction"},
                },
                {
                    "name": "update_data_docs",
                    "action": {"class_name": "UpdateDataDocsAction"},
                },
            ],
        }
    },
    anonymous_usage_statistics={
      "enabled": True
    }
)
ge_context = BaseDataContext(project_config=data_context_config)

print("****************************List data sources******************")
print(ge_context.list_datasources())
print("***************************************************************")

batch_request = BatchRequest(
    datasource_name="dataframe_epi_source",
    data_connector_name="dataframe_epi_data_connector",
    data_asset_name="epi",
)
print("****************************After BatchRequest******************")
validator = ge_context.get_validator(
    batch_request=batch_request,
    expectation_suite_name="bod_epi_expectation_suit"   
)

print("**********validator.head()************")
print(validator.head())
print("*******************************")

available_data_asset_names = ge_context.datasources[
    "dataframe_bod_epi_source"
].get_available_data_asset_names(
    data_connector_names="dataframe_bod_epi_data_connector"
)[
    "dataframe_bod_epi_data_connector"
]

print("**********asset count************")
count = len(available_data_asset_names)
print(count)
print("*********************************")

print("**********dataframe count of asset ************")
print(validator.active_batch.data.dataframe.count())
print("***********************************************")

expectation_suite_name = "bod_epi_expectation_suit"
checkpoint_config = {
    "class_name": "SimpleCheckpoint",
    "validations": [
        {
            "batch_request": batch_request,
            "expectation_suite_name": expectation_suite_name,
        }
    ],
}

checkpoint = SimpleCheckpoint(
    f"{validator.active_batch_definition.data_asset_name}_{expectation_suite_name}", ge_context, **checkpoint_config
)

checkpoint_result = checkpoint.run()

ge_context.build_data_docs()

[austiezr]

Hey @deepuak01 ! Thanks for reaching out.

This error typically results from a mis-configured datasource, as the error is essentially saying "there's no data here."

This documentation on testing your configs may be able to help you identify what is happening here, though working with EMR may prevent you from being able to use that workflow.

Have you had an opportunity to check out this documentation on EMR, and this documentation referenced by it?