[FIXED] How to upload AppendBlob/a file larger than the 4mb limit into Azure Storage/Blob in Java?

Issue

How to upload a large(>4mb) file as an AppendBlob using Azure Storage Blob client library for Java?

I've successfully implemented the BlockBlob uploading with large files and it seems that the library internally handles the 4mb(?) limitation for single request and chunks the file into multiple requests.

Yet it seems that the library is not capable of doing the same for AppendBlob, so how can this chunking be done manually? Basically I think this requires to chunk an InputStream into smaller batches...

Using Azure Java SDK 12.14.1

Solution

Inspired by below answer in SO (related on doing this in C#): c-sharp-azure-appendblob-appendblock-adding-a-file-larger-than-the-4mb-limit

... I ended up doing it like this in Java:

    AppendBlobRequestConditions appendBlobRequestConditions = new AppendBlobRequestConditions()
            .setLeaseId("myLeaseId");
    
    try (InputStream input = new BufferedInputStream(
            new FileInputStream(file));) {
        byte[] buf = new byte[AppendBlobClient.MAX_APPEND_BLOCK_BYTES];
        int bytesRead;
        while ((bytesRead = input.read(buf)) > 0) {
            if (bytesRead != buf.length) {
                byte[] smallerData = new byte[bytesRead];
                smallerData = Arrays.copyOf(buf, bytesRead);
                buf = smallerData;
            }
            try (InputStream byteStream = new ByteArrayInputStream(buf);) {
                appendBlobClient
                        .appendBlockWithResponse(byteStream, bytesRead,
                                null, appendBlobRequestConditions, null,
                                null);
            }
        }
    }

Of course you need to do bunch of stuff before this, like make sure the AppendBlob exists, and if not then create it before trying to append any data.

Answered By - Jokkeri

Answer Checked By - Mary Flores (PHPFixing Volunteer)