Extending boost::filesystem for Windows and Linux: Part 2

• Part 1 article download of FSHelper_app_for_Windows_and_Linux.zip
• Part 1 article download of FSHelper-VS2012.zip
• Part 1 article download of FSHelper_Qt_project.zip

Introduction

In my previous article I posted my observations regarding one specific area which I haven't been able to find much examples for and that was the implementation of a file system watcher on Linux using C++. Now I will try to summarize my findings regarding my next topic - deleting to and restoring files from Trash on Linux with a brief overview of similar Windows functionality regarding Recycle Bin.

When I started working on extending the basic functionalities provided by the boost::filesystem namespace I was aware that it will be difficult to find sources for things like a .NET-like portable file system watcher which I discussed in part 1 but was really surprised that areas like working with Linux Trash are totally neglected as well and with almost no usable answers or documentation regarding C++ implementations. Even on Windows, although sources to follow can be found, I haven't been able to find code samples for managing Recycle Bin with C++ which worked without problems. I am planning to dedicate the bigger part of this article to the Linux implementation, but before that I will try to give some of my observations for the Windows implementation I have provided in the code you can download. Again, to compile this code, look for instructions available in the Part 1 article.

Addition: I added new project ready to compile with Visual Studio 2012, so now you have that option too. For additional instructions look at Part 1 article also.

Working with Windows Recycle Bin Using C++

I have to say that listing Recycle Bin contents and deleting files/folders to Recycle Bin isn't that hard once you get to know a couple of not-so obvious things, specially for people who don't have much previous experience with Windows Shell API like in my case. Without this knowledge, you can still create code samples that will work in some occasions but "mysteriously" fail in others. At least, that is what happened to me. For listing contents of Recycle Bin look for my implementation of the FileSystemController::ListRecycleBin() function. I have seen that a lot of people had problems when compiling code like I provided there using the MinGW compiler. To be specific, if you are using the CoTaskMemFree() API call, you'll likely get a linker error stating something like 'undefined reference to CoTaskMemFree@4'. After further research I found that the CoTaskMemFree() function implementation should be in ole32.dll and I tried to add it to libraries. After that, everything compiled without problems.

The second set of problems occurred while implementing deletion to Recycle Bin. My implementation looks like this:

OpResults moveToRecycleBin(const STRING_TYPE& source)
{
    OpResults opres;
    SHFILEOPSTRUCTW operation;
    operation.wFunc = FO_DELETE;
    unique_ptr<wchar_t[]> name(createDoubleNullTermString(source));
    operation.pFrom = (LPCWSTR)name.get();//source.c_str();
    operation.fFlags = FOF_ALLOWUNDO | FOF_SILENT | FOF_NOCONFIRMATION;

    //without values below defined operation will CRASH on WINDOWS XP!!!
    operation.hNameMappings = NULL;
    operation.lpszProgressTitle = NULL;
    operation.fAnyOperationsAborted = FALSE;
    operation.hwnd = NULL;
    operation.pTo = NULL;
    
    //return errors (values bigger that 0) are mostly defined in <winerror.h>
    int result = SHFileOperationW(&operation);
    if(result != 0)
        opres.AddResult(OperationResult(false, source, TO_STRING_TYPE("Return Error value:") + 
          Helpers::ToString<int, STR_SET>(result)));

    return opres;
}

Basically, we are providing a path to the file system item we want to be moved to Recycle Bin and return the operation outcome inside the specialized container class I implemented (OpResults). This is more or less something common to most functions I have provided inside the FileSystemController class which you can find inside the Qt Creator project for download. The first thing I wasn't aware of at first is that you need to provide an item path to SHFILEOPSTRUCTW in the form of a double null-terminated string. If you don't do that, you will get partial success with this operation, or to be more precise, sometimes it will work, sometimes it won't. To get a double null terminated string from the standard std:: wstring, you can use the function below:

wchar_t* createDoubleNullTermString(const wstring& ws)
{
    wchar_t* buff = new wchar_t[MAX_PATH + 2];
    wcscpy(buff, ws.c_str());
    memcpy(buff + ws.length() + 1, L"\0\0", 2);
    return buff;
} 

Please notice that this way we are creating a wchar_t* array on the heap which needs to be deleted later on. If you are using a unique_ptr smart pointer from the standard library like in my code, make sure to use "wchar_t[]" as the provided type of pointer to indicate that a specialized array delete should be used.

the next problem was that my implementation didn't work on Windows XP. After extensive research I found while browsing the internet, a piece of code from a totally unrelated project with a small comment pointing out that a SHFileOperation() call could crash without setting values to SHFILEOPSTRUCTW fields like hNameMappings, lpszProgressTitle, fAnyOperationsAborted, hwnd, and pTo. After assigning values like in the provided sample, the XP problem was gone.

Finally, before I start discussing the Linux Trash implementation, I have to point out that there seems to be no easy way to restore a file from Recycle Bin on Windows (at least I haven't been successful in finding one) using C++. I have tried a couple of different approaches including the ShellExecute() API call which needed a specific "verb" which I tried to find in the Registry but without success. This is the only functionality I wanted but failed to provide in the FileSystemController class.

Manipulating Trash on Linux using C++

As I mentioned before, when I started looking for available information on various internet forums, I ended up with almost no usable data to continue with regarding this topic. The only usable source I found is the freedesktop.org Trash specification. By saying usable I don't want to give you the wrong impression that it should be followed by the letter. You should test each and every feature described there to make sure that the actual behavior on the distribution you intend to use actually is implemented as described. For example, according to freedesktop.org Trash specification, the real home Trash location should be available over an environment variable as $XDG_DATA_HOME/Trash. This isn't the case with any major distributions I tested. I must say that I'm by no means a Linux expert so maybe I missed something in the process but it seems that we can expect two possible Trash locations on most popular distributions. The first is the so called home trash location which can usually be found in the /home/<user>/.local/share/Trash hidden folder and the behavior of this one is implemented mostly as described in the previously mentioned specification. The other(s) can be found on attachable medias like USB sticks and similar, and is located on the media's root as a .Trash-$uid hidden folder where $uid stands for current user id.

All these locations contain two sub-folders: files and info. The first one (files subdirectory) is used for storing actual data (files/folders) that are moved to trash. The other one (info subdirectory) has the corresponding info files with extension .trashinfo and contain URI encoded original item path which is used for restore operation, and a datetime value which indicates when this item was deleted. Although the corresponding files and info entries build up their names from the original item name which is moved to Trash, the specification states that these names should not be used as the source for a restore operation. In the Trash folder one additional file can occasionally be found. That would be the metadata file which holds the current Trash size. As I have found out, this file was/is a major source of the various unexpected behaviors when using Trash (this is from personal experience in my previous Open SUSE 12.1 OS). First of all, if you are experiencing behavior like the Trash limit is reached when you are certain that it actually isn't, you should look for the problem source in this place. It seems that the value stored in metadata is unsigned (which makes sense) and the actual correction provided by the OS sometimes happens to be wrong causing it to underflow with a negative value. This will cause a really big value to be written in the metadata file and thus the Trash appears as if the size limit has been reached. I also tried to create a mechanism for updating the trash size value inside the metadata file but after some time I gave up on this since I couldn't figure out why and how the system performs some calculations for a new size value, specially in cases when we are moving symbolic links to Trash. It doesn't seem to add neither the symlink size nor the actual item size that it points to. This doesn't seem to be the bug, just something I haven't been able to track in the documentation so I solved this the “easy” way - by deleting the metadata file each time some kind of change is made to the Trash content with the FileSystemController class. In this case the environment will rebuild the metadata file with the recalculated value inside, which just happens to be a nice convenience allowing us to pass this task away from us.

One more thing seems to be unclear regarding trash locations in Linux to many people – if you haven't still used Trash, chances are that the previously described actual Trash folders don't exist yet. They are usually created on first usage. If we intend to mimic Linux trash operations, we should implement the ability to create these paths if they are not present at the time we are accessing Trash.

To establish moving to and restoring from functionalities for Trash locations, I needed a number of different helper functions that would be responsible for each part of the job:

• Getting the user ID:
C++

inline UINT getUserID()
{
    return getuid();
}

• Getting the ID of the physical storage device:
C++

inline UINT getDeviceID(const string& source)
{
    struct stat _info;
    const char* orgPath = source.c_str();
    if(stat(orgPath, &_info) == 0)
    {
        return _info.st_dev;
    }
    else
        return 0; // Unsuccessfull stat() !!!
}

• Getting all the available mount points on the system and associating them with the ID of the actual device they are on:
C++

std::map<UINT, std::string> getMountPoints()
{
    std::map<UINT, std::string> mount_points;
    struct mntent* ent;
    FILE* aFile;
    aFile = setmntent("/proc/mounts", "r");
    if (aFile != NULL)
    {
        while (NULL != (ent = getmntent(aFile)))
        {
            mount_points.insert(std::make_pair(getDeviceID(ent->mnt_dir), ent->mnt_dir));
        }
    }
    endmntent(aFile);
    return mount_points;
}

• Functions for locating actual Trash path(s). As mentioned above, if we don't find the Trash location where it should be, we should create it. This is what the system would do for us in the background if we are using standard built-in ways to access Trash:
C++

//If user didn't move anything so far to Trash, chances are that directory
//doesn't exist yet
OperationResult createTrash(const std::string& trash_path)
{
    OperationResult opOK = createOKOpResult(trash_path);
    system::error_code ec;
    bfs::create_directory(trash_path, ec);
    if(ec.value() != 0)
        return OperationResult(false, trash_path, ec.message());
    //updateTrashMetadata(trash_path, 0, true); //create new file
    bfs::path trashInfoPath = trash_path;
    bfs::path trashfilesPath = trash_path;
    trashInfoPath /= "info";
    trashfilesPath /= "files";
    bfs::create_directory(trashInfoPath, ec);
    if(ec.value() != 0)
    {
        deleteItem(trash_path);
        return OperationResult(false, trashInfoPath.string(), ec.message());
    }
    bfs::create_directory(trashfilesPath, ec);
    if(ec.value() != 0)
    {
        deleteItem(trash_path);
        return OperationResult(false, trashfilesPath.string(), ec.message());
    }
    return opOK;
}
 //Creates home trash if doesn't exist
inline std::string getHomeTrashPath()
{
    bfs::path trash;
    std::string trashPath;
    if((trashPath = freedesktop_org_HomeDataPath()).empty())
    {
        trash = homePath();
        trash /= ".local/share/Trash";
    }
    else
    {
        trash = trashPath;
        trash /= "Trash";
    }
    if(!bfs::exists(trash))
    {
        OperationResult _result = createTrash(trash.string());
        if(!_result.result)
            trash = bfs::path(EMPTYSTR);
    }
    return trash.string();
}
 //MAX value for device_id means home trash
inline std::string getTrashPath(UINT device_id = UINT_MAX, bool fallBackToHomeTrash = true)
{
    if(device_id == UINT_MAX)
    {
        return getHomeTrashPath();
    }
    std::map<UINT, std::string> mount_points(getMountPoints());
    std::string trashOnMount(mount_points[device_id]);
    bfs::path trash;
    if(!trashOnMount.empty())
    {
        UINT usrID = getUserID();
        std::string trashName(".Trash-");
        trashName += Helpers::ToString(usrID);
        trash = trashOnMount;
        trash /= trashName;
        //freedesktop.org specification doesn't seem to be implemented to the letter on any
        //distro I tested. The only common behavior is existence of ./Thash-uid as local
        //trash implementation on medias like USB sticks
        if(!bfs::exists(trash))
        {
            std::string strTrash = trash.string();
            //On Fedora 17 x64 usb stick is mounted as "/run/media..." and on other systems I have tested as "/media..."
            if((strTrash.find("/media/") == 0 && strTrash.length() > 7) ||
                    (strTrash.find("/run/media/") == 0 && strTrash.length() > 10))
            {
                OperationResult _result = createTrash(trash.string());
                if(!_result.result)
                    trash = bfs::path(EMPTYSTR);
            }
            else if(fallBackToHomeTrash)
            {
                //Fallback to Home Trash
                trash = getHomeTrashPath();
            }
        }
    }
    return trash.string();
}
//Get All available trash paths from mount points(where they exist)
std::vector<std::string> getAvailableTrashPaths()
{
    std::map<UINT, std::string> mount_points(getMountPoints());
    vector<std::string> trashPaths;
    trashPaths.push_back(getHomeTrashPath());
    for_each(mount_points.begin(), mount_points.end(), [&trashPaths](const std::pair<UINT, std::string> p)
    {
        std::string possibleTrashPath(getTrashPath(p.first, false));
        if(bfs::exists(possibleTrashPath))
            trashPaths.push_back(possibleTrashPath);
    });
    return trashPaths;
}

For listing all trash contents we need to assemble all entries which can be found in each available Trash path on the system. During the process we also need to get the actual and restore paths for each item obtained in the previous search. To do that, reading the corresponding .trashinfo file in the info Trash subfolder is necessary. Finally, we would need to URI decode the obtained restore path. Here is my FileSystemController::ListTrash() function:

fInfosPtr FileSystemController::ListTrash(OpResults* resultOutcomes)
{
    //TODO:REWRITE !!!
    vector<std::string> availableTrashPaths(getAvailableTrashPaths());
    fInfosPtr returnEntries = new vector<F_INFO>;
    std::string homeTrash(getHomeTrashPath());
    for(auto& trashStr : availableTrashPaths)
    {
        unique_ptr<vector<F_INFO> > entries(getEntries(getTrash_Files(trashStr), true));
        unique_ptr<vector<F_INFO> > infoEntries(getEntries(getTrash_Info(trashStr), true));//, _SortInfo::dirFirst_asc));
        for_each(entries->begin(), entries->end(), [&infoEntries, &resultOutcomes](F_INFO& entry)
        {
            for(auto beg=infoEntries->begin(), end = infoEntries->end(); beg != end; ++beg)
            {
                const F_INFO& infoEntry = *beg;
                string::size_type pos = infoEntry.name.find(".trashinfo");
                if(pos != string::npos)
                {
                    string baseInfoName = infoEntry.name.substr(0, pos);
                    if(entry.name == baseInfoName)
                    {
                        try
                        {
                            entry.orgTrashPath = entry.path;
                            entry.path = getOriginalPathFromInfoTrashFile(infoEntry.path);
                            entry.name = boost::filesystem::path(entry.path).filename().string();
                            resultOutcomes->AddResult(createOKOpResult(entry.path));
                        }
                        catch(const string& errorMsg)
                        {
                            resultOutcomes->AddResult(OperationResult(false, infoEntry.path, errorMsg));
                        }
                        break;
                    }
                }
            };
        });
        Helpers::AddVectors(*returnEntries, *entries);
    }
    return returnEntries;
}

Each .trashinfo file has a structure which is well described (and followed to the letter on every distribution I have tested) in the freedesktop.org Trash specification:

[Trash Info]
Path=<URI encoded restore path>
DeletionDate=<deletion date and time>

The name of this file with the exception of the .trashinfo extension is the same as the one present in the files sub-folder in the Trash location belonging to the actual deleted content and this is the main criteria for their matching when file restoration needs to be done. To read the content that fills the .trashinfo file, you could use the function:

std::string getOriginalPathFromInfoTrashFile(const string& infoFilePath) throw(std::string)
{
    //The whole purpose of this prefix is to fill in
    // the missing parts of full restore path which is
    //the case when accessing parts of Trash 
    //on other places than home Trash...
    std::string prefix;
    //home Trash has different name than ".Trash-xyz"
    auto pref_pos = infoFilePath.find(".Trash-");
    if(pref_pos != std::string::npos)
        prefix = infoFilePath.substr(0, pref_pos);
    std::string text = Helpers::textFileRead(infoFilePath);
    std::string::size_type pos = text.find("Path=");
    std::string::size_type posEnd = text.find("\nDeletionDate=", pos);
    if(pos == std::string::npos || posEnd == std::string::npos)
        throw(std::string("Incorrect info file structure in ") + infoFilePath);
    pos += 5;
    std::string orgPath = text.substr(pos, posEnd - pos);
    return prefix + Helpers::URIDec(orgPath);
}

There is also a special case concerning names of corresponding files in the files and info sub-folders of Trash. When you have already deleted item in Trash which is named with a name that is identical to the item we want to move to trash, the system does the check and adds a suffix like "name 1", "name 2", "name 3"...etc. (depending on how many files with the same name you already have stored in trash; I believe that you could follow a different naming pattern than this as long as you make sure that each file stored in Trash has a unique name and that the names of entries in the files and info sub-folders match (with the exception of the .trashinfo extension). I personally followed the previously described behavior which I observed in my main development system -Open SUSE Linux. Once we want to manage Trash through our own code, we need to implement this ability to have unique names for each deleted item in Trash (bfs in the next function is an alias to boost::filesystem):

 //corrects first provided path and returns the corrected file name + extension for string concatenation
inline std::string correctTrashFileName(bfs::path& filePathToCorrect)
{
    typedef std::string::size_type POS_TYPE;
    string filePath;
    string parent_path = filePathToCorrect.parent_path().string();
    string ext = filePathToCorrect.extension().string();
    string fName = filePathToCorrect.filename().string();
    //fName still contains extension. This needs to be removed...
    fName = fName.substr(0, fName.find_last_of('.'));
    POS_TYPE pos = fName.find_last_of(' ');
    if(pos != string::npos)
    {
        string substr = fName.substr(++pos);
        int num = atoi(substr.c_str());
        if(num != 0)
        {
            ++num;
            substr = fName.substr(0, pos);
            fName = substr + Helpers::ToString(num) + ext;
            filePath = parent_path + "/" + fName;
            filePathToCorrect = bfs::path(filePath);
        }
        else
        {
            if(bfs::exists(filePathToCorrect))
            {
               fName += " ";
               fName += "1";
               fName += ext;
               filePath = parent_path + "/" + fName;
               filePathToCorrect = bfs::path(filePath);
            }
            else
                fName += ext;
        }
    }
    else
    {
        if(bfs::exists(filePathToCorrect))
        {
            fName += " ";
            fName += "1";
            fName += ext;
            filePath = parent_path + "/" + fName;
            filePathToCorrect = bfs::path(filePath);
        }
        else
            fName += ext;
    }
    if(bfs::exists(filePathToCorrect))
        fName = correctTrashFileName(filePathToCorrect);
    return fName;
}

Prior to calling this function we already have to have constructed the complete path to where we intend to move the current file/folder item (trash path) and check if it exists as such in Trash. The previously presented function then checks which suffix to add to the file name we already have to make it unique. It means checking if there already is a trashed file with the suffix, and if so, parsing that suffix as a number and incrementing it. The final name for Trash would then be something like "name 1" or "name x" where x stands for the smallest increment which isn't already taken by another trashed item.

For actual functions that provide Trash functionality you can look up the Qt Creator project in the download link. These functions are FileSystemController::ListTrash() already presented here, and the functions moveToTrash() located in FSC_Helpers_Linux.hpp and FileSystemController::RestoreFile(). The function moveToTrash() will do the actual "deleting" of the file system item to Trash:

OpResults moveToTrash(const string &source)
{
    OpResults results;
    bfs::path file(source);

    if(!bfs::exists(file))
    {
        results.AddResult(OperationResult(false, source, "source path does not exist"));
        return results;
    }

    string fName = file.filename().string();

    //this will create Home Trash location path if necessary (in case it still doesn't exist)
    std::string trashPath;
    if(goesInHomeTrash(source))
        trashPath = getHomeTrashPath();
    else //Creates or gets .Trash-$uid or falls back to home trash path
        trashPath = getTrashPath(getDeviceID(source));

    if(trashPath.empty())
    {
        results.AddResult(OperationResult(false, source, "Trash cannot be found!"));
        return results;
    }

    //Checking if moving should be made in "local" Trash rather than home Trash
    //"local" trash on attachable devices like USB sticks have a different folder
    //path
    bool isLocalTrash = trashPath.find(".Trash-") != std::string::npos ? true : false;

    std::string trashFiles(getTrash_Files(trashPath));
    std::string trashInfo(getTrash_Info(trashPath));

    bfs::path newFilePath(trashFiles);
    newFilePath /= fName;

    //If name already exists OS adds " 1" or other number
    //to the name to be unique - path provided is modified by ref and
    //file name + extension are in return value since it has to be used again to create new info file
    //It wouldn't make much sense to call this function twice.
    fName = correctTrashFileName(newFilePath);

    bfs::path trashInfoFilePath(trashInfo);
    trashInfoFilePath /= (fName + ".trashinfo");

    //write data to info folder
    //by freedesktop.org specification info file should be written first
    //If done otherwise, multiple files with the same name+extension (from different locations)
    //won't be recognized correctly in Trash folder by other file browsers like Dolphin
    std::string infoName;
    if(!isLocalTrash)
    {
        //Home Trash operates with ABSOLUTE PATHS only
        //according to freedesktop.org specification
        infoName = TrashRecoveryName(source);
    }
    else
    {
        //in case of local trash folder we should
        //put relative recovery path to Path field in info file
        infoName = LocalTrashRecoveryName(source);
    }
    std::string infoDate = getFileDeletionTime();

    try
    {
        //Write info file
        Helpers::FileWriter<> _writer(trashInfoFilePath.string());
        _writer << string("[Trash Info]");
        _writer << (string("Path=") + infoName);
        _writer << (string("DeletionDate=") + infoDate);
        //_writer will close stream upon D-tor execution
    }
    catch(const string& str)
    {
        results.AddResult(OperationResult(false, trashInfoFilePath.string(), str));
        return results;
    }
    catch(...)
    {
        results.AddResult(OperationResult(false, trashInfoFilePath.string(), 
          "Undefined error ocurred while trying to write info trash file"));
        return results;
    }

    //Actual movement to Trash location
    results = moveItem(source, newFilePath.string(), true);

    //check for errors and abort if error occured
    if(results.hasErrors())
    {
        if(!bfs::exists(source))
            results += moveItem(newFilePath.string(), source, true);
        return results;
    }

    if(bfs::exists(infoName))
        results += deleteItem(infoName);

    //check for errors and abort if error occured
    if(results.hasErrors())
    {
        if(!bfs::exists(source))
            results += moveItem(newFilePath.string(), source, true);
        return results;
    }

    /******************************************************************/
        /*Skipping this step... I will delete metadata file instead...*/
        //update size recorded in $Trash/metadata file

        /* updateTrashMetadata(trashPath, entrySize); */
    /******************************************************************/

    //By deleting this file KDE/Dolphin will create new one when needed and
    //update it with appropriate info
    std::string metadataPath(getTrashMetadataPath(trashPath));
    if(bfs::exists(metadataPath))
        deleteItem(metadataPath);

    return results;
}

The function moveToTrash() first needs to examine if the item path for deletion exists and get the appropriate Trash path where to move it. With the help of the previously described helper functions it should (re)build the Trash directory if it doesn't exist in the process. The next thing it should do is construct a .trashinfo file for the item we want to delete. This needs to be done before the actual movement of the selected item to the files subdirectory of Trash. Otherwise when using standard file managers like KDE Dolphin, we could experience that the Trash content isn't detected correctly. This happens mostly in cases when we need name "correction" for a new Trash item since there is already an item with the same name there. After that, the procedure is easy to understand - the function writes the .trashinfo file to the info subdirectory and moves the corresponding item to the files subdirectory and deletes the metadata file if existent.

Finally, the function FileSystemController::RestoreFile() can be used to do the actual restore from the Trash location. To do that, it needs to go through the contents of the files and info subdirectories of the appropriate Trash location and find the matching pair of entries in both directories, get the original restore path from the corresponding .trashinfo file, delete the .trashinfo file in the info subdirectory of Trash, and finally restore the Trash item by moving it from the Trash files subdirectory to its original location. Since we are making changes to the Trash size by doing this, the function deletes the metadata Trash file which holds the current Trash size, forcing the system to create a new one (if used) with the recalculated value.

Conclusion

I know that this article is quite lengthy, but I felt like this area has a lot of unknowns since working with Linux Trash proved to be similar to walking in the dark. To determine almost each step, I needed to observe (or assume) what the system is doing and to try to mimic that behavior through my code. I feel like my Trash implementation became usable to the point which makes standard managing tasks possible on major Linux distributions but it still has some flaws - not respecting the Trash size limit for one. This is because I wasn't able to find a way to get this limit information. There are probably other flaws as well but considering the amount of time I had for this task I don't feel like I could have done a much better job.

To summarize, if you intend to use my implementation in your code, feel free to do so but I suggest you test it first. The easiest way for this would be to download a compiled Linux "executable" of the FSHelper terminal application and try to use it for trash management (if you use x86 Linux, you're probably out of luck and you will need to recompile the provided Qt project). If it works OK, chances are that the underlying FileSystemController class implementation will work as well without problems.

For instructions about code available from download links, please refer to the Part 1 article under the section "Using the code".