Azure Data Factory trigger is not initiated when uploading a file using Java SDK
Symptoms:
- Uploading files using Java SDK class DataLakeFileClient fails to initiate the ADF trigger.
- Uploading files using Azure Portal initiates the trigger as expected.
- ADF trigger is configured to ignore empty blobs.
Analysis:
- Whenever a file is created using ADLS Gen2 REST API, there are 3 operations called at the backend i.e. CreateFile, AppendFile and FlushFile.
- Microsoft.Storage.BlobCreated event is triggered when a blob is created or replaced. Specifically, this event is triggered when clients use the CreateFile and FlushWithClose operations that are available in the Azure Data Lake Storage Gen2 REST API.
- As CreateFile API created a 0 byte blob, the event will not be triggered.
Resolution:
- You can initiate the trigger by adding FlushWithResponse method with close parameter as true.
Example:
boolean close = true;
File file = new File("C:\\Users\\mahmoudsamy\\route.txt");
fileClient.uploadFromFile("C:\\Users\\mahmoudsamy\\route.txt");
fileClient.flushWithResponse(file.length(), retainUncomittedData, close, httpHeaders, requestConditions, timeout, contect);
Note: Please make sure to pass the correct parameters when using FlushWithResponse to avoid unexpected results.
Published on:
Learn moreRelated posts
Azure Storage - TLS 1.0 and 1.1 retirement
Overview TLS 1.0 and 1.1 retirement on Azure Storage was previously announced for Nov 1st, 2024, and it was postponed recently to 1 year later...
Efficient Management of Append and Page Blobs Using Azure Storage Actions
Overview In Azure Storage, Blob Lifecycle Management (BLM) allows you to automate the management of your data based on rules defined by...
[Azure AI Search] Internal Server Error when creating CMK encrypted objects
Scenario Customers follow the Microsoft doc to create CMK encrypted objects (data source, index etc.), but get the 500 Internal Serv...
Optimizing Azure Table Storage: Automated Data Cleanup using a PowerShell script with Azure Automate
Scenario This blog’s aim is to manage Table Storage data efficiently. Imagine you have a large Azure Table Storage that accumulates logs from ...
Optimizing Azure Table Storage: Automated Data Clean-up using a PowerShell script with Azure Automat
Scenario This blog’s aim is to manage Table Storage data efficiently. Imagine you have a large Azure Table Storage that accumulates logs from ...
Restoring Soft-Deleted Blobs with multithreading in Azure Storage Using C#
Blob soft delete is an essential feature that safeguards your data against accidental deletions or overwrites. By retaining deleted data for a...
Performing simple Azure Table Storage REST API operations using curl command.
The blog provides guidance to perform simple Table Storage REST API operations such as Create table, Delete Table, Insert entity, Delete entit...
Bulk delete all the old jobs from the batch account
Deleting a Job also deletes all Tasks that are part of that Job, and all Job statistics. This also overrides the retention period for Task dat...
Utilizing Azure Storage and Runbooks for scheduled automated backups of Azure SQL Databases
In this article, we are going to provide detailed steps to create a scheduled Azure SQL Database backup to storage account using automation. T...
[Azure Service Bus] JMS messages getting dead-lettered
The article discusses a problem where numerous messages end up in the dead letter queue (DLQ) when the JMS service bus consumer connects to th...