Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Java Apache Beam, allow fake external Clients initialized in @Setup method of DoFn with Constructors variables #31258

Open
AerialSky93 opened this issue May 12, 2024 · 0 comments

Comments

@AerialSky93
Copy link

AerialSky93 commented May 12, 2024

Apache Beam recommends to use Fakes instead of Mocks, since Mocks cannot serialize over the pipeline.

I am writing unit tests for older code, where the class is using Apache Beam calling other external class services (which I have not written or cannot change). The classes aren't using interfaces.


public class DataAdapter extends DoFn<KV<String, Product>, Void>{
    public String processCategory;
    public Map<String, String> configs;
    public ServiceApi serviceApi; 
    
    public DataAdapter(String processCategory, Map<String, String> configs) {
        this.processCategory = processCategory;
        this.configs = configs;
    }
  
    @Setup
    public void init() {
        serviceApi = new ServiceApi(processCategory, configs);
    }

    @ProcessElement
    public void processData(ProcessContext context, @Element KV<String, Product> input) {
        try {
            String productId = input.getKey();
            serviceApi.getData();
            ....
     }
} 

Attempt:

I created the fake below. Calling the constructors with nulls will cause Argument Exception errors in the serviceApi constructor. So I tried a factory create pattern which is probably not ideal. (trying to find another solution)

However, the fake keeps getting overriden, since for each ProcessElement, it calls Setup, with reinitializes Parent Class. We had to initialize in each SetupBatch, since serviceApi was not serializable in running Apache Beam application.

Create Fake:


public class ServiceApiFake extends ServiceApi {

    public ServiceApiFake() {
        super(null, null);
    }

    public static ServiceApiFake create() {
        try {
            return new ServiceApiFake();
        } catch (Exception ignore) {
            return null;
        }
    }

    @Override
    public DataItem getData(String keyId) {
        return null;
    }
}

Test:


TestStream<KV<String, Product>> testStream = TestStream.create(KvCoder.of(StringUtf8Coder.of(), SerializableCoder.of(Product.class)))
            .addElements(KV.of("25", product)).advanceWatermarkToInfinity();

DataAdapter dataAdapter = new DataAdapter("productCategory", configs);
dataAdapter.serviceApi = new ServiceApiFake();
pipeline.apply(testStream).apply(ParDo.of(dataAdapter));
pipeline.run();

Error in test run: ArgumentExceptionError at Setup init, since it return to calling the regular service at Setup instead of fake.

I thought about going back to testing methods individually by calling "processData" Method, without calling setup, and using a mockito Mock on ProcessContext, ElementInput, and serviceApi when(serviceApi.getData()).thenReturn(..) . This strategy actually works, its not ideal, but allows me to test in some degree.

** I wish Apache Beam team had a better, robust alternative in using fakes, or allow mocks somehow. Or allow DoFnTester again (since its currently deprecated). Company policy does not allow deprecated usage anymore.

Resource: I looked at referring stackoverflow article. Their serviceAPI doesn't have constructor variables, or argument exception validators within the serviceAPI, so it works in their situation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant