How can I upload multiple large files to Google drive API using PHP in chunks?

Question

0.00/5 (No votes)

See more:

I need to upload multiple large files in chunks using the following code:

The first file of the foreach upload works correctly, but I encounter issues starting before the second file upload.
The return type of $service->files->listFiles is GuzzleHttp\Psr7\Request, most likely because the media upload is deferred with the line $client->setDefer(true);.

I would like to find a solution where the loop waits for each upload to complete before moving on to the next file, similar to a thread. Alternatively, is it possible to send multiple uploads simultaneously (async/parallel)?

What I have tried:

PHP

$service = new Google\Service\Drive($client);

foreach ($files as $fileDatas) {

    //Check if a file with this name has already been uploaded
    $is_file_exists = $service->files->listFiles($Params);
    if (count($is_file_exists->files) > 0) {
       return false;
    }
    
    $file = new Google\Service\Drive\DriveFile();
    $file->name = $fileName;
    $chunkSizeBytes = 1 * 1024 * 1024;

    // Call the API with the media upload, defer so it doesn't immediately return.
    $client->setDefer(true);
    $request = $service->files->create($file);

    // Create a media file upload to represent our upload process.
    $media = new Google\Http\MediaFileUpload($client, $request, $type, null, true, $chunkSizeBytes);
    $media->setFileSize($size);

    // Upload the various chunks. $status will be false until the process is complete.
    $status = false;
    $handle = fopen($filePath, "rb");
    while (!$status && !feof($handle)) {
        // read until you get $chunkSizeBytes from TESTFILE
        // fread will never return more than 8192 bytes if the stream is read
        // buffered and it does not represent a plain file
        // An example of a read buffered file is when reading from a URL
        $chunk = readVideoChunk($handle, $chunkSizeBytes);
        $status = $media->nextChunk($chunk);
    }

    // The final value of $status will be the data from the API for the object
    // that has been uploaded.
    $result = false;
    if ($status != false) {
        ...
    }

    fclose($handle);
}

Posted 23-Jun-23 5:50am

Deltahun

Updated 24-Jun-23 6:38am

v3

Add a Solution

Comments

Patrice T 23-Jun-23 12:11pm

Did you tried 1 by 1 ?

Deltahun 24-Jun-23 11:33am

Yes. If the $files has only one element, it works. I also tried uploading each file separately and it worked.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Andre Oosthuizen · Answer 1 · 2023-06-24T00:49:00

Quote:
is it possible to send multiple uploads simultaneously (async/parallel)

- yes, it is but in my opinion, using parallel uploads in PHP with Google Drive API can be quite complex. I know that you can use PHP libraries or frameworks that support asynchronous operations, such as ReactPHP or Swoole or maybe use other languages like Node.js or Golang that has libraries for this.

To upload your files and wait for each to complete before starting the next in your loop, I will drop the defer option and make synchronous requests instead of asynchronous ones -

foreach ($files as $fileData) {
    $file = new Google\Service\Drive\DriveFile();
    $file->name = $fileName;
    $chunkSizeBytes = 1 * 1024 * 1024;

    // Create a media file upload
    $media = new Google\Http\MediaFileUpload($client, $file, $type, null, true, $chunkSizeBytes);
    $media->setFileSize($size);

    // Upload the file in chunks
    $handle = fopen($filePath, "rb");
    while (!$media->isFinished()) {
        // Read the next chunk from the file
        $chunk = readVideoChunk($handle, $chunkSizeBytes);
        $media->nextChunk($chunk);
    }

    // Get the result of the upload
    $result = $media->getResult();
    fclose($handle);

    // Do something with the result
    if ($result != false) {
        // Process the result...
    }
}

Deltahun · Answer 2 · 2023-06-24T06:38:00

Solution 2

I solved it by resetting "setDefer" to the client at the end of the loop:

PHP

fclose($handle);
$client->setDefer(false);

Posted 24-Jun-23 6:38am