S3 Attachment Storage

by ,
Odoo
v 14.0 Third Party 19 983
Download for v 14.0 Deploy on Odoo.sh
Included Dependencies
Lines of code 628
Technical Name ir_attachment_s3
Websitehttps://twitter.com/OdooFree
Also available in version v 10.0 v 13.0 v 11.0 v 9.0 v 12.0
You bought this module and need support? Click here!

S3 Attachment Storage

Upload attachments on Amazon S3

Features:

  • The module allows to upload the attachments in Amazon S3 automatically without storing them in Odoo database. It will allow to reduce the load on your server. Attachments will be uploaded on S3 depending on the condition you specified in Odoo settings. So you can choose and manage which type of attachments should be uploaded on S3.
  • It is useful in cases where your database was crashed, because you will be able to easily restore all attachments from external storage at any time.
  • The possibility to use one external storage for any number of databases.
It should be noted that the module works with any kind of attachments.

How it works

E.g., go to Sales → Products menu and open an item. Upload any images to the product.

At this moment in the Settings → Database Structure → Attachments menu the attachments will be created with the corresponding URL.

Then open Amazon S3 bucket and see uploaded attachments.

The attachments will be loaded to Odoo DB from S3 server by using URL and they will not be stored in your DB.

Configuration

Before using you need to make some configuration:

  • Install boto3 library and get credentials for it by using this quickstart instruction https://boto3.readthedocs.io/en/latest/guide/quickstart.html
  • Grant access to your S3 bucket using this instruction http://mikeferrier.com/2011/10/27/granting-access-to-a-single-s3-bucket-using-amazon-iam and set bucket as public
  • Enable technical features https://odoo-development.readthedocs.io/en/latest/odoo/usage/technical-features.html
  • Open menu Settings → Technical → Database Structure → S3 Settings and specify the following parameters:
    - s3.bucket: the name of your bucket (e.g. mybucket)
    - s3.condition: only the attachments that meet the condition will be sent to S3 (e.g. [('res_model', 'in', ['product.image'])]) - it is actually the way of specifying the models with fields.Binary fields that should be stored on S3 instead of local file storage or db. Don't specify anything if you want to store all your attachment data from fields.Binary and also ordinary attachments on S3.
    - s3.access_key_id: S3 access key ID
    - s3.secret_key: S3 secret access key
  • Click on Upload existing attachments if you want to upload attachments you had before installation the module

Free Support

You will get free support and assistance in case of any issues

Need our service?

Contact us by email or fill out request form

  • apps@it-projects.info
  • https://www.it-projects.info/page/website.contactus
  • https://m.me/itprojectsllc
  • skype@it-projects.info
Tested on Odoo
13.0 community
Tested on Odoo
13.0 enterprise

S3 Attachment Storage

Installation

  • Using this quickstart instruction install boto3 library and get credentials for it

  • Using this instruction grant access to your s3 bucket

  • Set your S3 bucket as public

  • Optionaly, add following parameter to prevent heavy logs from boto3 library:

    --log-handler=boto3.resources.action:WARNING

IAM

Minimal access policy for s3 credentials are as following:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
              "s3:PutObject",
              "s3:CreateBucket",
              "s3:GetBucketLocation",
              "s3:PutObjectAcl"
          ],
          "Resource": [
              "arn:aws:s3:::YOUBUCKETNAMEHERE",
              "arn:aws:s3:::YOUBUCKETNAMEHERE/*"
          ]
      }
  ]
}

You can also remove "s3:CreateBucket" if bucket already exists.

Configuration

  • Activate developer mode
  • Open menu Settings >> Technical >> Parameters >> System Parameters and specify the following parameters there
    • s3.bucket: the name of your bucket (e.g. mybucket)
    • s3.condition: only the attachments that meet the condition will be sent to s3 (e.g. [('res_model', 'in', ['product.image'])]) - it is actually the way of specifying the models with fields.Binary fields that should be stored on s3 instead of local file storage or db. Don't specify anything if you want to store all your attachment data from fields.Binary and also ordinary attachments on s3.
    • s3.access_key_id: S3 access key ID
    • s3.secret_key: S3 secret access key
    • s3.endpoint_url: optional parameter for s3-compatible storage. E.g. http://minio.example.com:9000
    • s3.obj_url: optional parameter for s3-compatible storage. E.g. http://minio.example.com:9000/bucketname/

Alternatively, you can set the parameters via environmental variables: S3_BUCKET, S3_CONDITION, S3_ACCESS_KEY_ID, S3_SECRET_KEY, S3_ENDPOINT_URL, S3_OBJ_URL

The settings are also available from the Settings >> Technical >> Database Structure >> S3 Settings.

S3

Minimal access policy for s3 credentials are as following:

{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": [
              "s3:PutObject",
              "s3:CreateBucket",
              "s3:GetBucketLocation",
              "s3:PutObjectAcl"
          ],
          "Resource": [
              "arn:aws:s3:::YOUBUCKETNAMEHERE",
              "arn:aws:s3:::YOUBUCKETNAMEHERE/*"
          ]
      }
  ]
}

You can also remove "s3:CreateBucket" if bucket already exists.

Usage

Depending on what you have in the s3.condition setting, some or all attachments will be uploaded on your s3. For example upload by editing product template from Sales >> Product menu some image for your product. By doing this you have uploaded image on your s3 storage.

Upload existing attachments

  • There are 2 ways to upload existing attachments: 1. Open odoo shell and run env["ir.attachment"].force_storage_s3() 2. Go to the Settings >> Technical >> Database Structure >> S3 Settings menu and click on the [Upload existing attachments] button there

    It is recommended to upload attachments via shell, because upload progress is visible in this method.

  • To add link of existing S3 bucket object to binary fields of existing odoo records:

    • Take Link urls from Amazon. If you open Overview of the object on Amazon you should see it at the bottom of the page
    • make sure that you have properly configured your odoo, see the Configuration section of this instruction once again in the To enable feature of linking... part
    • to link objects one-by-one from an odoo backend (this option is only available for images attachments):
      • In any place where you can upload images to odoo (i.e. from Sales >> Sales >> Products when you select a product and push [Edit] button there and hover your mouse pointer under the place on a form view where an image should be) along with standard pencil and trash bin buttons you can see the additional [@] button. Click on this button.
      • Copy-paste your url for image and click [Save] button
    • to link objects in batch you may use default import/export feature in odoo:
      • for example, export records of model product.template. Choose image field in the export dialog and save in file.
      • open the file with your favorite text editor and paste urls into image column there
      • import records from the edited CSV file
      • now when you open from Sales >> Sales >> Products your product form, you should see the image you specified by url in the file

Please log in to comment on this module

  • The author can leave a single reply to each comment.
  • This section is meant to ask simple questions or leave a rating. Every report of a problem experienced while using the module should be addressed to the author directly (refer to the following point).
  • If you want to start a discussion with the author, please use the developer contact information. They can usually be found in the description.
Please choose a rating from 1 to 5 for this module.
Great module, but images are public
by
Drew E
on 11/22/21, 12:40 PM

Hello,

Excellent module, works as expected. However, the bucket must be set to public which means anyone can access the uploaded files.

Considering the credentials are stored in the config, is there a way to use a private bucket with no public access?

Facing an issue : Odoo Server Error
by
sales@onedayonegroup.com
on 8/31/21, 6:11 AM

Error:

Odoo Server Error


Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/odoo/addons/base/models/ir_http.py", line 237, in _dispatch

    result = request.dispatch()

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 684, in dispatch

    result = self._call_function(**self.params)

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 360, in _call_function

    return checked_call(self.db, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/odoo/service/model.py", line 94, in wrapper

    return f(dbname, *args, **kwargs)

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 348, in checked_call

    result = self.endpoint(*a, **kw)

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 913, in __call__

    return self.method(*args, **kw)

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 532, in response_wrap

    response = f(*args, **kw)

  File "/usr/lib/python3/dist-packages/odoo/addons/web/controllers/main.py", line 1393, in call_button

    action = self._call_kw(model, method, args, kwargs)

  File "/usr/lib/python3/dist-packages/odoo/addons/web/controllers/main.py", line 1381, in _call_kw

    return call_kw(request.env[model], method, args, kwargs)

  File "/usr/lib/python3/dist-packages/odoo/api.py", line 396, in call_kw

    result = _call_kw_multi(method, model, args, kwargs)

  File "/usr/lib/python3/dist-packages/odoo/api.py", line 383, in _call_kw_multi

    result = method(recs, *args, **kwargs)

  File "/myaddons/ir_attachment_s3/models/res_config_settings.py", line 98, in s3_upload_existing

    self.env["ir.attachment"].force_storage_s3()

  File "/myaddons/ir_attachment_s3/models/ir_attachment.py", line 105, in force_storage_s3

    return self._force_storage_with_bucket(

  File "/myaddons/ir_attachment_url/models/ir_attachment.py", line 125, in _force_storage_with_bucket

    bin_data = base64.b64decode(data) if data else b""

  File "/usr/lib/python3.8/base64.py", line 87, in b64decode

    return binascii.a2b_base64(s)

Exception


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 640, in _handle_exception

    return super(JsonRequest, self)._handle_exception(exception)

  File "/usr/lib/python3/dist-packages/odoo/http.py", line 316, in _handle_exception

    raise exception.with_traceback(None) from new_cause

binascii.Error: Incorrect padding

Re: Facing an issue : Odoo Server Error
by
Elizaryev Ivan Nikolaevich
on 9/2/21, 7:22 AM Author

Hello, can you send to our e-mail the steps to reproduce this error and error itself?

Regards,

Denis

Facing an issue - This file could not be decoded as an image file. Please try with a different file.
by
Vinay Khosla
on 8/23/21, 10:44 AM

Hi there, when using the module we are getting an error "This file could not be decoded as an image file. Please try with a different file."

This error is seen when creating a new product & trying to save it. The image is getting uploaded in the S3 bucket but still the product isn't getting created/saved.

Could you guide me on what is the issue?

Re: Facing an issue - This file could not be decoded as an image file. Please try with a different file.
by
Elizaryev Ivan Nikolaevich
on 9/2/21, 7:21 AM Author

Hello,

we got your e-mail with the feedback and checking it. Thank you! 

CORS issue
by
Drew E
on 8/3/21, 1:18 PM

Hello,

This module looks very useful, however I can't get it working all the way through due to what appears to be a CORS issue.

I am able to upload an item to S3 using a binary field. However, when I try to download the item I get this error: Something happened while trying to contact the server, check that the server is online and that you still have a working network connection.

In the javascript console I see this:

Cross-origin redirection to https://<bucket>/<hash> denied by Cross-Origin Resource Sharing policy: Origin http://localhost:8069 is not allowed by Access-Control-Allow-Origin.

I've added this to the CORS config of the bucket, but I get the same error:

[

    {

        "AllowedHeaders": [

            "*"

        ],

        "AllowedMethods": [

            "GET",

            "PUT",

            "POST",

            "DELETE"

        ],

        "AllowedOrigins": [

            "*"

        ],

        "ExposeHeaders": []

    }

]


Any thoughts on why this is occurring and how to solve it?

Re: CORS issue
by
Elizaryev Ivan Nikolaevich
on 8/4/21, 10:03 AM Author

Hello!

Could you send steps to reproduce the problem to help@itpp.dev

Thank you


error with authentication
by
Fabio Tielen
on 6/3/21, 6:28 AM

Hello Dennis

I have installed your app, but when I try to move existing attachments in settings I get error as below

botocore.exceptions.ClientError: An error occurred (InvalidAccessKeyId) when calling the PutObject operation: The AWS Access Key Id you provided does not exist in our records.

But this is impossible, because it's the same key I also use for handling backups through rclone.org system and it works fine there. 
I tested up to 5x, and its 300% sure correct credentials, yet your app keeps alerting about non-existing key. 

I think there is something wrong with the part that is validating the key against S3 alternative providers and is trying to validate against amazon s3. 



S3 compatible providers
by
Fabio Tielen
on 6/1/21, 5:00 PM

Hello

Can you please kindly add also the field for custom endpoint (and region)? 
Then other providers can operate also with this like WASABI, Digital Ocean Spaces, Min.io, etc... 

These all use the same Amazon S3 API 1:1 compatible, there is only a requirement to set a different endpoint URL and region depending the chosen alternative provider. 

source examples: 

WASABI
https://wasabi-support.zendesk.com/hc/en-us/articles/115002579891-How-do-I-use-AWS-SDK-for-Python-boto3-with-Wasabi-




Re: S3 compatible providers
by
Elizaryev Ivan Nikolaevich
on 6/2/21, 6:43 AM Author

Hi Fabio,

we've already done this. Please, check Readme.

Best regards,

Denis

by
Robert Thompson
on 5/19/21, 6:58 AM

Hi,

I am trying to contact you regarding this app.  What is the best way to do so?

I have an issue where the mime type is as below and not image/jpeg as I suspect it should be?

Sorry to post this on here.

File
 image_256
https://bucket.s3.amazonaws.com/odoo/X
application/octet-stream
Re:
by
Elizaryev Ivan Nikolaevich
on 5/22/21, 5:20 AM Author

Hello Robert,

we got your e-mail message and will answer after some investigations.

Regards,

Denis

by
Robert Thompson
on 5/18/21, 6:22 AM 
Hi,
FYI this URL is dead -  http://mikeferrier.com/2011/10/27/granting-access-to-a-single-s3-bucket-using-amazon-iam
Re:
by
Elizaryev Ivan Nikolaevich
on 5/18/21, 7:45 AM Author

Hi Robert,

thank you for informing us, we'll fix it.

Regards,

Denis