FilterExpressionTextParser does not support 'null' values

[linarkou]

Bug description
FilterExpressionTextParser does not support 'null' values, so I can't write

SearchRequest searchRequest = SearchRequest.builder()
                        .query("Spring AI")
                        .similarityThresholdAll()
                        .topK(3)
                        .filterExpression("meta1 ==  null")
                        .build();

At the same time, FilterExpressionBuilder do support it.

var b = new FilterExpressionBuilder();
SearchRequest searchRequest = SearchRequest.builder()
                        .query("Spring AI")
                        .similarityThresholdAll()
                        .topK(3)
                        .filterExpression(b.eq("meta1", null).build())
                        .build();

Environment
Spring AI v1.0.0, Java 17, Vector Store - ClickHouse (but it doesn't depend on vector store type)

Steps to reproduce

SearchRequest searchRequest = SearchRequest.builder()
                        .query("Spring AI")
                        .similarityThresholdAll()
                        .topK(3)
                        .filterExpression("meta1 == null")
                        .build();

Expected behavior
I expect that Filter.Expression::right will return null when it called from filter expression converter (FilterExpressionConverter::convertExpression)

Minimal Complete Reproducible example
https://github.com/linarkou/spring-ai-clickhouse-store/blob/main/spring-ai-clickhouse-store/src/test/java/org/springframework/ai/vectorstore/clickhouse/ClickhouseVectorStoreIT.java

    private static List<Document> documents() {
        return List.of(
                new Document("1", getText("classpath:/test/data/spring.ai.txt"), Map.of("meta1", "meta1","intMeta", 456)),
                new Document("2", getText("classpath:/test/data/time.shelter.txt"), Map.of()),
                new Document("3", getText("classpath:/test/data/great.depression.txt"),
                        Map.of("meta2", "meta2", "intMeta", 123)));
    }

    private static String getText(String uri) {
        var resource = new DefaultResourceLoader().getResource(uri);
        try {
            return resource.getContentAsString(StandardCharsets.UTF_8);
        } catch (IOException e) {
            throw new RuntimeException(e);
        }
    }

    public static Stream<Arguments> filterExpressions() {
        var b = new FilterExpressionBuilder();
        return Stream.of(
                Arguments.of(
                        b.ne("meta1", null).build(),
                        "meta1 != null",
                        List.of("1")),
                Arguments.of(
                        b.eq("meta1", null).build(),
                        "meta1 == null",
                        List.of("2", "3"))
        );
    }

    @MethodSource("filterExpressions")
    @ParameterizedTest
    void testSimilaritySearchWithFilters(Filter.Expression filterExpression, String nativeFilterExpression,
                                         List<String> expectedIds) {
        this.contextRunner.run(context -> {
            VectorStore vectorStore = context.getBean(AbstractObservationVectorStore.class);

            List<Document> originalDocuments = documents();
            vectorStore.doAdd(originalDocuments);

            if (filterExpression != null) {
                SearchRequest searchRequest = SearchRequest.builder()
                        .query("Spring AI")
                        .similarityThresholdAll()
                        .topK(3)
                        .filterExpression(filterExpression)
                        .build();
                List<Document> documents = vectorStore.doSimilaritySearch(searchRequest);
                assertEquals(expectedIds, documents.stream().map(Document::getId).collect(Collectors.toList()));
            }

            if (nativeFilterExpression != null) {
                SearchRequest searchRequest = SearchRequest.builder()
                        .query("Spring AI")
                        .similarityThresholdAll()
                        .topK(3)
                        .filterExpression(nativeFilterExpression)
                        .build();
                List<Document> documents = vectorStore.doSimilaritySearch(searchRequest);
                assertEquals(expectedIds, documents.stream().map(Document::getId).collect(Collectors.toList()));
            }

            vectorStore.doDelete(originalDocuments.stream().map(Document::getId).toList());
            vectorStore.close();
        });
    }

[dev-jonghoonpark]

@linarkou
It seems likely that the code provided in Bug Description contains a mistake, as both pieces of code appear identical.
Based on the provided reproducible example, I understand what was intended. However, revising the code in Bug Description would make the distinction clearer.

[linarkou]

@dev-jonghoonpark thank you! Modified original post.

[dev-jonghoonpark]

Error Occurrence Point

FilterExpressionTextParser.java:line147

Filter.Operand operand = filterExpressionVisitor.visit(parser.where());

Error Message

Error: no viable alternative at input 'null'

Relevant Library

antlr

[dev-jonghoonpark]

@linarkou

I have submitted a PR to address this issue.
Please review it.

#3706

[ilayaperumalg]

@linarkou Thanks for reporting the issue. @dev-jonghoonpark Thanks for following it up with a PR!