Google Takeout Image Downloader

• Download source - 339.3 KB
• Download demo - 6.5 KB

Update

Please note that as of 13^th January 2020, the application described in the article DOES NOT WORK as Google has removed file links from the takeout JSON files.

You may want to refer to the source code if you want to develop any progress bar based application.

Introduction

Google provides a way for users to download data from Google photos through Google TakeOut. User can download a zip file containing all (all really?) photos. The problem is that Google selectively replaces a Json file for an image file inside the zip. This json file contains the link where the actual image is located.

The utility described in this article takes an extracted takeput folder as an input and downloads all the real images in the output folder.

Background

Some days ago, my wife's Google account happened to stop working. On investigating, I found out that she had used all the 15GB data. The main culprit was the image files. Her Android phone was syncing every image on the phone to Google Photos.

It left me with two options:

1. Opt for additional paid storage from Google.
2. Download the images and free up the Google storage to unlock the account.

I chose the second option.

I downloaded the Google photos zip through Google TakeOut under the impression that I would be done once I downloaded the zip. But on extracting the zip, what I found was, Google had zipped json files (which contains links to the actual image file) and not all image files.

To tackle this, I developed a utility which reads all these json files and downloads the corresponding images.

Using the Application

Prior to using this application, you will need to:

1. Download and extract the takeout zip from Google takeout.

   More information can be found here.

2. Once you extract the zip file, open the demo application.
3. Choose the extracted directory as an input directory.
4. Choose an Output directory as where you need to download your images.
5. Click on "Start Download".

Now sit back and relax. The application will download all the images to your output folder.

About the Source Code

There are only three classes in the demo application added:

1. DownloadManager: Which does the actual downloading task.
2. IDownloadProgressNotifier: An interface to provide download progress information back to the UI.
3. Form class: for the UI to be shown. It implements IDownloadProgressNotifier interface.

Download Manager

Download manager does the actual work of:

1. Reading the json file from the extracted folder for image link
2. Download the image file from the link in the json file

The fields of our interest in the json file are:

1. url: It is a link where the actual image is located.
2. title: Title of the image. We will save the image locally with this name.

Fragment of the json looks like this:

{
  "title": "IMG_20110711_192757772.jpg",
  "description": "",
  "url": "https://lh3.googleusercontent.com/-JHYN2OSYh21s/Vaxpt7adEp2I/
         AAAAAAAABCu0/hIlcgO9TzwwkEJm2eQ9PcBu2rL1kPOqZWqwCLABGAYYCw/
         s0-d/IMG_20110711_192757772.jpg",
  "imageViews": "0",

Using Download Manager

To use Download manager, you will need to create the object of download manager by providing the following parameters to its constructor.

1. Input directory
2. Output directory
3. Flag to tell whether to continue on error or not
4. Download progress notifier handler

DownloadManager downloader = new DownloadManager(srcPath, dstPath, true, false, this);

Now call StartDownload(). This will start the download progress. The download progress will be notified to the notifier handler which can be used to update the UI.

downloader.StartDownload();

Inside Download Manager

Download manager has only one public function.

public void StartDownload()

StartDownload() Function

This function will be a starting point for all processing. It does the following things:

1. It calls the PreDownload function to notify the UI that download process is about to start.
   C#

   if (Notifier != null)
       Notifier.PreDownload(TotalFilesCount);
   

2. Calls CreateDirectory to create output root directory.
   C#

   //
   //Create Folder structure in the output directory.
   //
   
   CreateDirectory(new DirectoryInfo(Path.GetDirectoryName(inputRootFolder)));
   

3. Calls TraverseFolderTree to go through all files in the inputFolder and its subfolder.
   C#

   TraverseFolderTree(inputRootFolder);
   

4. Function TraverseFolderTree returns when it completes downloading all files. On completion, the UI is being notified by calling the notifier function OnDownloadFinish().
   C#

   if (Notifier != null)
       Notifier.OnDownloadFinish();
   

TraverseFolderTree Function

This function goes through the input directory tree structure. It traverses the input directory structure recursively to check for all files with the extension "*.jpg.json". We are interested in these files only.

First, we need to create outputfolder path. The directory structure of the inputfolder and outputfolder will remain the same.

Example: If user selects the following directories as an input and output directory.

• Input directory: C:\Users\test\Desktop\TakeoutProject\Input
• Output directory: C:\Users\test\Desktop\TakeoutProject\Output

If the current input processing folder is C:\Users\test\Desktop\TakeoutProject\Input\2015-07-17, then the output directory name is created by replacing the input root directory prefix with the output root directory.

The current Output directory will be C:\Users\test\Desktop\TakeoutProject\Output\2015-07-17.

String sCurrentOutputFolder = sSeedFolder.Replace(inputRootFolder, outputRootFolder) ;

The output directory is created where the image files will be downloaded.

Directory.CreateDirectory(sCurrentOutputFolder);

Now enumerate through all the files with "*.jpg.json" extension in the currentInputDirectory. These files are our candidates for download.

//
// Now go through all files. Search for jpg link files only.
// "*.jpg.json" pattern will exclude other files like metadata.json
//
foreach (String strFile in Directory.EnumerateFiles
                (sSeedFolder, JSON_JPEG_FILE_FILTER))
{
        ....
        ....
        ....
}

Each file path is passed to the RetrieveImage function to read the file and download the actual image file.

RetrieveImage(strFile, sCurrentOutputFolder);

The UI is notified by calling the notifier function OnCurrentFileProgress. In case of the error, the UI is notified by calling the notifier function OnError.

The final loop looks like this:

//
// Now go through all files. Search for jpg link files only.
// "*.jpg.json" pattern will exclude other files like metadata.json
//
foreach (String strFile in Directory.EnumerateFiles
        (sSeedFolder, JSON_JPEG_FILE_FILTER))
{
    try
    {
        if (Notifier != null)
            Notifier.OnCurrentFileProgress
            (strFile, ProcessedFilesCount, TotalFilesCount);

        RetrieveImage(strFile, sCurrentOutputFolder);
        ProcessedFilesCount++;
    }
    catch (Exception exc)
    {
        if (Notifier != null)
            Notifier.OnError(strFile, exc.Message);

        if (stopOnError)
        {
            //
            // We are done as we are supposed to stop on error.
            //

            throw exc;
        }
    }
}

Now traverse through all the remaining directories/subdirectories recursively.

//
// Traverse through all directories recursively.
//
foreach (String strDirectory in Directory.EnumerateDirectories(sSeedFolder))
{
    TraverseFolderTree(strDirectory);
}

RetrieveImage Function

This function downloads the .jpeg.json file and reads it for actual image file location.

private void RetrieveImage(String sJsonFilePath, String sImageFilePathOutputFolder)
        {
            JsonValue fullJson = JsonValue.Parse(System.IO.File.ReadAllText(sJsonFilePath));
            String Title = fullJson["title"];
            String url = fullJson["url"];

            using (WebClient client = new WebClient())
            {
                client.DownloadFile(url, sImageFilePathOutputFolder + 
                                    Path.DirectorySeparatorChar + Title);
            }
        }

As already stated, we are interested only in "title" and "url" field.

The "url" represents the actual web location of the file. WebClient class is used to download the image file from this URL. The output filename will be the same as that of the "title".

using (WebClient client = new WebClient())
{
    client.DownloadFile(url, sImageFilePathOutputFolder + Path.DirectorySeparatorChar + Title);
}

IDownloadProgressNotifier Interface

This interface declares the following functions. Downloader manager uses this interface to notify the downloading status.

Interface Functions

OnCurrentFileProgress(String sCurrentFileName, int processedFilesCount, int totalFileCount)

This function notifies the handler about the current file being processed along with the total progress.

PreDownload(int TotalFilesCount)

This function will be called just before start of the download.

OnDownloadFinish();

This function will be called after download finishes.

OnError(String sFileName, String sErrorInformation);

This function will be called in case any error occurs. It passes the name of the file for which error occurred as well as the error information.

Form1 Class

This class is a simple C# form class which owns the UI. Along with the simple UI elements shown in the picture, it implements the IDownloadProgressNotifier interface. The reference to this class is sent to the Download Manager as a handler. The implementation of the IDownloadProgressNotifier interface functions is used to update the UI with progress status.

IDownloadProgressNotifier Interface Implementation

OnCurrentFileProgress(String sCurrentFileName, int processedFilesCount, int totalFileCount)

This function notifies the handler about the current file being processed along with the total progress. This callback function increments the progressbar by one. This makes progressbar to move forward by one unit.

public void OnCurrentFileProgress
   (string sCurrentFileName, int processedFilesCount, int totalFileCount)
      {
          pbTotalProgress.Increment(1);
      }

PreDownload(int TotalFilesCount)

This callback function is used to initialize the progress bar on the UI.

public void PreDownload(int TotalFilesCount)
       {
           pbTotalProgress.Minimum = 0;
           pbTotalProgress.Maximum = TotalFilesCount;
           pbTotalProgress.Step = 1;
       }

This function will be called just before start of the download.

OnDownloadFinish();

This function will be called after download finishes. This function is used to show the process completion message box.

public void OnDownloadFinish()
       {
           pbTotalProgress.Value = pbTotalProgress.Maximum;
           MessageBox.Show("Process finished.", "Success");
       }

OnError(String sFileName, String sErrorInformation);

This callback function is used to show the error occurred.

public void OnError(string sFileName, string sErrorInformation)
        {
            MessageBox.Show("Process interrupted with an error 
                             while processing file \n File Name :" + 
                             sFileName +"\n ErrorInformation" + sErrorInformation, "Error" );
        }

History

• First version(1)
• Second version: Added code walk through