Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix user exports to deal with s3 storage #3228

Merged
merged 51 commits into from
Apr 13, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
cbd0812
initial work on fixing user exports with s3
hughrun Jan 14, 2024
62cc6c2
oops
hughrun Jan 14, 2024
d4d2734
ignore exports dir
hughrun Jan 14, 2024
833f26f
Merge branch 'main' into user-export
hughrun Jan 18, 2024
4691729
cleanup and linting
hughrun Jan 18, 2024
26c37de
linting
hughrun Jan 19, 2024
2bb9a85
various fixes
hughrun Jan 28, 2024
0d619f7
Merge branch 'main' into user-export
hughrun Jan 28, 2024
582e97e
Merge branch 'image-serialize' into user-export
hughrun Jan 28, 2024
a3e0525
fix avatar import path
hughrun Jan 28, 2024
2c231ac
linting and tests
hughrun Jan 28, 2024
c106b2a
Subclass boto3.Session to use AWS_S3_ENDPOINT_URL
dato Jan 29, 2024
765fc1e
fix tests
hughrun Jan 29, 2024
adff3c4
allow user exports with s3
hughrun Jan 29, 2024
f96ddaa
Merge pull request #3 from dato/export_job_inject_aws_endpoint_setting
hughrun Jan 29, 2024
5f7be84
subclass boto3 session instead of adding new env value
hughrun Jan 29, 2024
3675a4c
disable user exports if using azure
hughrun Jan 29, 2024
518da3b
Merge from main into 'user-export'
dato Mar 18, 2024
a6dc5bd
Make `get_file_size` robust against typing errors
dato Mar 18, 2024
5b71e94
Merge branch 'main' into user-export
mouse-reeve Mar 23, 2024
dd27684
set signed s3 url expiry with env value
hughrun Mar 24, 2024
03587df
migrations
hughrun Mar 24, 2024
69f4644
Remove problematic migration
BartSchuurmans Mar 24, 2024
a770689
Merge branch 'main' into user-export
BartSchuurmans Mar 24, 2024
073f62d
Add exports_volume to docker-compose.yml
BartSchuurmans Mar 24, 2024
471233c
Use different export job fields for the different storage backends
BartSchuurmans Mar 24, 2024
ab7b089
User exports: handle files that no longer exist on file storage
BartSchuurmans Mar 24, 2024
5bd66cb
Only generate signed S3 link to user export when user clicks download
BartSchuurmans Mar 24, 2024
aee8dc1
Fix pylint warning
BartSchuurmans Mar 24, 2024
e0decbf
Fix urlescaped relative path to cover image in export
BartSchuurmans Mar 25, 2024
a514022
Refactor creation of user export archive
BartSchuurmans Mar 25, 2024
f721289
Simplify logic for rendering user exports
BartSchuurmans Mar 25, 2024
bd95bcd
Add test for special character in cover filename
BartSchuurmans Mar 25, 2024
d9bf848
Fix pylint warnings
BartSchuurmans Mar 25, 2024
6a67943
Merge branch 'main' into user-export
BartSchuurmans Mar 26, 2024
145c67d
Merge BookwyrmExportJob export_data field back into one with dynamic …
BartSchuurmans Mar 26, 2024
ef57c0b
Check last user export too in post handler
BartSchuurmans Mar 26, 2024
ed2e9e5
Merge migration
BartSchuurmans Mar 26, 2024
9685ae5
Consolidate BookwyrmExportJob into two tasks
BartSchuurmans Mar 26, 2024
9afd0eb
Update migrations
BartSchuurmans Mar 27, 2024
797d5cb
Update BookwyrmExportJob tests
BartSchuurmans Mar 27, 2024
c6ca547
Fix migration formatting
BartSchuurmans Mar 27, 2024
cdbc1d1
Fix double exports subdir in S3 user export
BartSchuurmans Mar 27, 2024
dabf7c6
User export testing fixes
BartSchuurmans Mar 28, 2024
bb5d815
Fix mypy error
BartSchuurmans Mar 28, 2024
2bbe3d4
Test user export archive contents
BartSchuurmans Mar 28, 2024
0ac9d12
Merge branch 'main' into user-export
BartSchuurmans Mar 29, 2024
5d597f1
Use new "with ()" style
BartSchuurmans Mar 29, 2024
501fb45
export avatars to own directory
hughrun Apr 13, 2024
d48d312
Merge branch 'main' into user-export
hughrun Apr 13, 2024
c3c4614
add merge migration
hughrun Apr 13, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ AWS_SECRET_ACCESS_KEY=
# AWS_S3_CUSTOM_DOMAIN=None # "example-bucket-name.s3.fr-par.scw.cloud"
# AWS_S3_REGION_NAME=None # "fr-par"
# AWS_S3_ENDPOINT_URL=None # "https://s3.fr-par.scw.cloud"
# S3_ENDPOINT_URL=None # same as AWS_S3_ENDPOINT_URL - needed for non-AWS for user exports
hughrun marked this conversation as resolved.
Show resolved Hide resolved

# Commented are example values if you use Azure Blob Storage
# USE_AZURE=true
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
# BookWyrm
.env
/images/
/exports/
bookwyrm/static/css/bookwyrm.css
bookwyrm/static/css/themes/
!bookwyrm/static/css/themes/bookwyrm-*.scss
Expand Down
4 changes: 2 additions & 2 deletions bookwyrm/activitypub/person.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
""" actor serializer """
from dataclasses import dataclass, field
from dataclasses import dataclass
from typing import Dict

from .base_activity import ActivityObject
Expand Down Expand Up @@ -35,7 +35,7 @@ class Person(ActivityObject):
endpoints: Dict = None
name: str = None
summary: str = None
icon: Image = field(default_factory=lambda: {})
icon: Image = None
bookwyrmUser: bool = False
manuallyApprovesFollowers: str = False
discoverable: str = False
Expand Down
92 changes: 92 additions & 0 deletions bookwyrm/migrations/0193_auto_20240128_0249.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Generated by Django 3.2.23 on 2024-01-28 02:49

import bookwyrm.storage_backends
import django.core.serializers.json
from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

dependencies = [
("bookwyrm", "0192_sitesettings_user_exports_enabled"),
]

operations = [
migrations.AddField(
model_name="bookwyrmexportjob",
name="export_json",
field=models.JSONField(
encoder=django.core.serializers.json.DjangoJSONEncoder, null=True
),
),
migrations.AddField(
model_name="bookwyrmexportjob",
name="json_completed",
field=models.BooleanField(default=False),
),
migrations.AlterField(
model_name="bookwyrmexportjob",
name="export_data",
field=models.FileField(
null=True,
storage=bookwyrm.storage_backends.ExportsFileStorage,
upload_to="",
),
),
migrations.CreateModel(
name="AddFileToTar",
fields=[
(
"childjob_ptr",
models.OneToOneField(
auto_created=True,
on_delete=django.db.models.deletion.CASCADE,
parent_link=True,
primary_key=True,
serialize=False,
to="bookwyrm.childjob",
),
),
(
"parent_export_job",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="child_edition_export_jobs",
to="bookwyrm.bookwyrmexportjob",
),
),
],
options={
"abstract": False,
},
bases=("bookwyrm.childjob",),
),
migrations.CreateModel(
name="AddBookToUserExportJob",
fields=[
(
"childjob_ptr",
models.OneToOneField(
auto_created=True,
on_delete=django.db.models.deletion.CASCADE,
parent_link=True,
primary_key=True,
serialize=False,
to="bookwyrm.childjob",
),
),
(
"edition",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
to="bookwyrm.edition",
),
),
],
options={
"abstract": False,
},
bases=("bookwyrm.childjob",),
),
]
Loading
Loading