Delete blobs in Azure Data Factory by leveraging MSI

Recently, I used Azure Data Factory again to build a big data ingestion pipeline.  The input data came from different sources, but needed to end up eventually in Azure Blob Storage.  Everything went pretty smooth and I had my data in the desired format, inside the expected blob container.  One final task that remained: I needed to clean-up some temporary blobs within my container.  It was an unpleasant surprise to see that there’s no out-of-the-box support to delete files in blob storage.  It’s apparently a highly requested feature.

The workaround

On the web, I’ve found several alternatives, of which using Logic Apps seemed the most convenient workaround.  However, I was not satisfied, as this solution introduces an additional complexity and dependency.  Why not using the Azure Blob Storage REST API?  Authentication against this API can be typically hard, unless you can leverage Managed Service Identity.  As Data Factory supports MSI, I was curious if it could work…  Yes it did, otherwise I wouldn’t start a blog on it 🙂

  • Let’s add the Web activity and give it a meaning full name:

MSI1

  • Provide the URL of the blob, as the documentation states.  Select the DELETE method and include also the mandatory x-ms-date and x-ms-version headers:

MSI2

MSI3

  • Give the Managed Service Identity Storage Blob Data Contributor access rights on the storage account.  Remark that being an owner is not sufficient, in contradiction to what the documentation states.  This is probably because it’s still a preview feature.  Thanks Joonas Westlin, for your help on this one!

MSI4

  • If all if this is configured correctly, you can easily delete the blob from its storage account:

MSI5

Conclusion

Another password-less authentication, which we can only encourage.  Please remark that, at the time of writing, this AD integration is still a preview feature for Azure Storage.  A real pity that the Azure Data Factory documentation is not updated with this great MSI functionality!

Cheers
Toon

Advertisements

6 thoughts on “Delete blobs in Azure Data Factory by leveraging MSI

  1. Pingback: Microsoft Integration Weekly Update: December 10, 2018 | Hooking Stuffs Together

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s