Azure Data Factory trigger is not initiated when uploading a file using Java SDK

Symptoms:

Uploading files using Java SDK class DataLakeFileClient fails to initiate the ADF trigger.
Uploading files using Azure Portal initiates the trigger as expected.
ADF trigger is configured to ignore empty blobs.

Analysis:

Whenever a file is created using ADLS Gen2 REST API, there are 3 operations called at the backend i.e. CreateFile, AppendFile and FlushFile.
Microsoft.Storage.BlobCreated event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
As CreateFile API created a 0 byte blob, the event will not be triggered.

Resolution:

You can initiate the trigger by adding FlushWithResponse method with close parameter as true.

Example:

boolean close = true;

File file = new File("C:\\Users\\mahmoudsamy\\route.txt");

fileClient.uploadFromFile("C:\\Users\\mahmoudsamy\\route.txt");

fileClient.flushWithResponse(file.length(), retainUncomittedData, close, httpHeaders, requestConditions, timeout, contect);

Note: Please make sure to pass the correct parameters when using FlushWithResponse to avoid unexpected results.

Published on: January 12, 2023

Azure PaaS Blog articles

In this blog post is a way set up version control for Power BI semantic models (and reports) using the PBIP (Power BI Project) format, Azure D...

1 day ago

Run, invoke, and monitor AI agents locally or in Microsoft Foundry with the new azd AI agent extension commands. Plus GitHub Copilot-powered p...

2 days ago

Blog image

Learn more

Related posts