Azure Data Factory trigger is not initiated when uploading a file using Java SDK

Symptoms: 

 

•   Uploading files using Java SDK class DataLakeFileClient fails to initiate the ADF trigger. 
•   Uploading files using Azure Portal initiates the trigger as expected. 
•   ADF trigger is configured to ignore empty blobs. 

 

Analysis: 

 

• Whenever a file is created using ADLS Gen2 REST API, there are 3 operations called at the backend i.e. CreateFile, AppendFile and FlushFile.
• Microsoft.Storage.BlobCreated event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
• As CreateFile API created a 0 byte blob, the event will not be triggered. 

 

Resolution: 

 

• You can initiate the trigger by adding FlushWithResponse method with close parameter as true.

Example: 

 

        boolean close = true;

File file = new File("C:\\Users\\mahmoudsamy\\route.txt");

        fileClient.uploadFromFile("C:\\Users\\mahmoudsamy\\route.txt");

        fileClient.flushWithResponse(file.length(), retainUncomittedData, close, httpHeaders, requestConditions, timeout, contect);

 

Note: Please make sure to pass the correct parameters when using FlushWithResponse to avoid unexpected results.

Continue to website...