Azure Data Factory trigger is not initiated when uploading a file using Java SDK
Symptoms:
- Uploading files using Java SDK class DataLakeFileClient fails to initiate the ADF trigger.
- Uploading files using Azure Portal initiates the trigger as expected.
- ADF trigger is configured to ignore empty blobs.
Analysis:
- Whenever a file is created using ADLS Gen2 REST API, there are 3 operations called at the backend i.e. CreateFile, AppendFile and FlushFile.
- Microsoft.Storage.BlobCreated event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
- As CreateFile API created a 0 byte blob, the event will not be triggered.
Resolution:
- You can initiate the trigger by adding FlushWithResponse method with close parameter as true.
Example:
boolean close = true;
File file = new File("C:\\Users\\mahmoudsamy\\route.txt");
fileClient.uploadFromFile("C:\\Users\\mahmoudsamy\\route.txt");
fileClient.flushWithResponse(file.length(), retainUncomittedData, close, httpHeaders, requestConditions, timeout, contect);
Note: Please make sure to pass the correct parameters when using FlushWithResponse to avoid unexpected results.
Published on:
Learn moreRelated posts
Microsoft Purview: Data Lifecycle Management- Azure PST Import
Azure PST Import enables Microsoft 365 admins to import PST files from Azure Blob Storage into Exchange Online mailboxes using PowerShell. The...
April Patches for Azure DevOps Server
We are releasing patches for our self‑hosted product, Azure DevOps Server. We strongly recommend that all customers remain on the latest, most...
Integration Testing Azure Functions with Reqnroll and C#, Part 5 - Using Corvus.Testing.ReqnRoll in a build pipeline
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. In the final post in this serie...
Integration Testing Azure Functions with Reqnroll and C#, Part 4 - Controlling your functions with additional configuration
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. In the fourth of this series of...
Integration Testing Azure Functions with Reqnroll and C#, Part 3 - Using hooks to start Functions
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. In the third of a series of pos...
Integration Testing Azure Functions with Reqnroll and C#, Part 2 - Using step bindings to start Functions
If you use Azure Functions on a regular basis, you'll likely have grappled with the challenge of testing them. In the second of a series of po...