Click here to Skip to main content
15,887,135 members
Search within:


How can I upload a tar.bz2 file to openstack swift object storage container using Python swift client?
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Python
Swift
I wrote a Python script which included the python-swiftclient module to connect to the OpenStack Object Storage and upload some files to the container after reading the python-swiftclient documentation: https://docs.openstack.org/python-swiftclient/latest/client-api.html

It works great if I upload a file that ends with the extension .gz however, I'm getting some errors when it comes to the compressed file that ends with the extension .tar.bz2.

I've included the Python script and the errors I get after running it. Please show me where I'm wrong, and I would like some assistance in solving this issue.

What I have tried:

Python
from keystoneauth1 import session
from keystoneauth1.identity import v3
from swiftclient.client import Connection
from swiftclient.client import ClientException
import gzip
import tarfile

# Create a password auth plugin
auth = v3.Password(auth_url='https://cloud.company.com:5000/v3/',
                   username='myaccount',
                   password='mypassword',
                   user_domain_name='Default',
                   project_name='myproject',
                   project_domain_name='Default')

# Create session
keystone_session = session.Session(auth=auth)

# Create swiftclient Connection
swift_conn = Connection(session=keystone_session)

# Create a new object with the contents of Netbox database backup
with gzip.open('/var/backup/netbox_backups/netbox_2024-03-16.psql.gz', 'rb') as file:
    swift_conn.put_object(
        container,
        'object_netbox_2024-03-16.psql.gz',
        contents=file,
        content_type='application/gzip'
    )

# Confirm the presence of the object holding the Netbox database backup
obj1 = 'object_netbox_2024-03-16.psql.gz'
container = 'netbox-backups'
try:
    resp_headers = swift_conn.head_object(container, obj1)
    print("The object " + obj1 + " was successfully created")
except ClientException as e:
    if e.http_status == '404':
        print("The object " + obj1 + " was not found!")
    else:
        print("An error occurred checking for the existence of the object " + obj1)

# Create a new object with the contents of the compressed Netbox media backup
with tarfile.open("/var/backup/netbox_backups/netbox_media_2024-03-16.tar.bz2", "r:bz2") as file_tar_bz2:
    swift_conn.put_object(
        container,
        'object_netbox_media_2024-03-16.tar.bz2',
        contents=file_tar_bz2,
        content_type='application/x-tar'
    )

# Confirm the presence of the object holding the compressed Netbox media backup
obj2 = 'object_netbox_media_2024-03-16.tar.bz2'
container = 'netbox-backups'
try:
    resp_headers = swift_conn.head_object(container, obj2)
    print("The object " + obj2 + " was successfully created")
except ClientException as e:
    if e.http_status == '404':
        print("The object " + obj2 + " was not found!")
    else:
        print("An error occurred checking for the existence of the object " + obj2)


Below are the errors I'm getting after running the script

Python
(venv) [root@scs-sandbox01 backups]# python netbox_backups_transfer.py 

The object object_netbox_2024-03-16.psql.gz was successfully created
Traceback (most recent call last):
  File "/opt/scripts/netbox_backups_transfer.py", line 52, in <module>
    swift_conn.put_object(
  File "/opt/netbox/venv/lib64/python3.9/site-packages/swiftclient/client.py", line 1963, in put_object
    return self._retry(reset_func, put_object, container, obj, contents,
  File "/opt/netbox/venv/lib64/python3.9/site-packages/swiftclient/client.py", line 1797, in _retry
    rv = func(self.url, self.token, *args,
  File "/opt/netbox/venv/lib64/python3.9/site-packages/swiftclient/client.py", line 1421, in put_object
    conn.request('PUT', path, contents, headers)
  File "/opt/netbox/venv/lib64/python3.9/site-packages/swiftclient/client.py", line 416, in request
    self.resp = self._request(method, url, headers=headers, data=data,
  File "/opt/netbox/venv/lib64/python3.9/site-packages/swiftclient/client.py", line 400, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/opt/netbox/venv/lib64/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/netbox/venv/lib64/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/opt/netbox/venv/lib64/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/netbox/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/opt/netbox/venv/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/opt/netbox/venv/lib64/python3.9/site-packages/urllib3/connection.py", line 399, in request
    for chunk in chunks:
  File "/opt/netbox/venv/lib64/python3.9/site-packages/swiftclient/utils.py", line 443, in iter_wrapper
    if len(chunk) == 0:
TypeError: object of type 'TarInfo' has no len()
Posted
Add a Solution

1 solution

Your issue lies within the message -
Quote:
TypeError: object of type 'TarInfo' has no len()

Your 'iter_wrapper' function in your 'swiftclient.utils' module is trying to read the length of a chunk of data, but it's encountering a 'TarInfo' object instead, which doesn't have a defined or set length as you are passing the 'tarfile.TarFile' object directly to 'swift_conn.put_object' as the contents parameter. This will not work because your 'put_object' method expects a file-like object with a 'read()' method.

You need to read the contents of the tar.bz2 file and then pass them as a file-like object to the 'put_object' - Python File read() Method[^]

Python
with tarfile.open("/var/backup/netbox_backups/netbox_media_2024-03-16.tar.bz2", "r:bz2") as file_tar_bz2:

    #Go over each file in the tar archive...
    for file_info in file_tar_bz2:

        #Read the contents...
        file_contents = file_tar_bz2.extractfile(file_info).read()

        #Create a file-like object from the contents...
        file_like_object = io.BytesIO(file_contents)

        #Upload the returned contents to Swift...
        swift_conn.put_object(
            container,
            file_info.name,  
        #Use the name of the file selected in the archive as your object name...
            contents=file_like_object,
            content_type='application/x-tar'  #Set the appropriate content type...
        )


[EDIT]
Your error 'AttributeError: 'NoneType' object' has no attribute 'read' occurs when you try to call the 'read()' method on a 'None' object, which is not allowed.This might happen when some files in the archive are not regular files, directories, symbolic links, or hard links. The tarfile module might not be able to handle certain types of files in the archive, and extractfile returns None for such files. You can modify your code to handle the case where 'extractfile' returns 'None' -

Python
with tarfile.open("/var/backup/netbox_backups/netbox_media_2024-03-24.tar.bz2", "r:bz2") as file_tar_bz2:
    #Go over each file in the tar archive...
    for file_info in file_tar_bz2:
        if file_info.isreg():
            #Read the contents...
            logger.info(f"Is regular file: {file_info.name}")
            extracted_file = file_tar_bz2.extractfile(file_info)
            if extracted_file:
                file_contents = extracted_file.read()
            else:
                logger.warning(f"Skipping {file_info.name}: Unable to extract file.")
                continue
        elif file_info.isdir():
            #Read the contents...
            logger.info(f"Is directory: {file_info.name}")
            extracted_file = file_tar_bz2.extractfile(file_info)
            if extracted_file:
                file_contents = extracted_file.read()
            else:
                logger.warning(f"Skipping {file_info.name}: Unable to extract file.")
                continue
        elif file_info.issym():
            #Read the contents...
            logger.info(f"Is symbolic link: {file_info.name}")
            extracted_file = file_tar_bz2.extractfile(file_info)
            if extracted_file:
                file_contents = extracted_file.read()
            else:
                logger.warning(f"Skipping {file_info.name}: Unable to extract file.")
                continue
        elif file_info.islnk():
            #Read the contents...
            logger.info(f"Is hard link: {file_info.name}")
            extracted_file = file_tar_bz2.extractfile(file_info)
            if extracted_file:
                file_contents = extracted_file.read()
            else:
                logger.warning(f"Skipping {file_info.name}: Unable to extract file.")
                continue
        else:
            logger.info(f"Is something else: {file_info.name}. Skip it")
            continue

        #Create a file-like object from the contents...
        file_like_object = io.BytesIO(file_contents)

        #Upload the returned contents to Swift...
        swift_conn.put_object(container, file_info.name,
                              contents=file_like_object,
                              content_type='application/x-tar')

[END EDIT]
 
Share this answer
 
Posted
Updated 28-Mar-24 4:09am
v3
Comments
Pete O'Hanlon 22-Mar-24 4:48am    
I like it. Have yourself a 5.
Hanginium2412 22-Mar-24 8:15am    
Thank you for your help Andre Oosthuizen and Peter O'Hanlon. I refactored my code as you suggested, but I got the error regarding the 'TarFile' object having no attribute 'read' after running my script.

# Create a new object with the contents of the compressed Netbox media backup
with tarfile.open("/var/backup/netbox_backups/netbox_media_2024-03-16.tar.bz2", "r:bz2") as file_tar_bz2:

# Read the contents of the compressed Netbox media backup file
file_contents = file_tar_bz2.read()

# Create a file-like object from the contents of the compressed Netbox media backup file
my_file_like_object = io.BytesIO(file_contents)

# Upload the returned contents to the OpenStack Object Storage container
swift_conn.put_object(
container,
'object_netbox_media_2024-03-16.tar.bz2',
contents=file_tar_bz2,
content_type='application/x-tar'
)


Below is the error I got after running the script. My apologies for these questions, I'm still a beginner in Python.

Python
Traceback (most recent call last):
File "/opt/scripts/netbox_backups_transfer.py", line 57, in <module>
file_contents = file_tar_bz2.read()
AttributeError: 'TarFile' object has no attribute 'read'
Andre Oosthuizen 22-Mar-24 12:56pm    
Thank you Pete!
Andre Oosthuizen 22-Mar-24 12:57pm    
Hanginium, I will look into this tomorrow when time allows.
Hanginium2412 23-Mar-24 13:01pm    
Good day Andre. Thank you for your response. I will gladly appreciate your help whenever you have time this weekend as this issue has been stressing me out for days and I would want it to be resolved asap.
Add a Solution

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
Top Experts
Last 24hrsThis month
Related Questions
How Can I Connect Swift Storage With Netdrive?
Objective-c or Swift
ASN 1 Decoder in Swift
core data swift xcode 6 question
Swift 5/iOS open weather API
how to handle special charecter in swift
speed up a bill printer using swift language
Internal storage video downloder
How to declare #pragma pack (1) in swift
How to save UIImage into byte using swift ?

Advertise
Privacy
Cookies
Terms of Use
Last Updated 28 Mar 2024
Layout: fixed | fluid
Copyright © CodeProject, 1999-2024
All Rights Reserved.

Web02 2.8:2024-04-02:1

CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900