Skip to content
This repository has been archived by the owner on Jan 22, 2025. It is now read-only.

Enhancement/projects file upload #62

Merged
merged 58 commits into from
Mar 12, 2024
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
12830d5
added query agent containing functions that can be used by multiple e…
AronBuzogany Mar 6, 2024
81e3d3c
loading env variables is only necessary in __main__
AronBuzogany Mar 6, 2024
658dfd3
removed unneeded load_dotenv
AronBuzogany Mar 6, 2024
7dd5682
completed functions that are ought to be used by multiple endpoints o…
AronBuzogany Mar 6, 2024
e42c57d
simplified endpoint functions by using query_agent functions
AronBuzogany Mar 6, 2024
3467bb2
fixed linting
AronBuzogany Mar 6, 2024
69df26e
fixed urljoin incorrectly joining url
AronBuzogany Mar 6, 2024
e836f50
lint: removed trailing whitepsace
AronBuzogany Mar 6, 2024
e26c015
completely replaced functionality with query_agent functions
AronBuzogany Mar 6, 2024
fd941b2
added functionality for patching an entry in the database
AronBuzogany Mar 6, 2024
073d6c5
fixed linting
AronBuzogany Mar 6, 2024
fd2ae83
filtered queries and forms to only contain entries that are valid in …
AronBuzogany Mar 6, 2024
d675fe6
created function that filters dict keys that are not in table
AronBuzogany Mar 6, 2024
2298d8c
made class serializable
AronBuzogany Mar 6, 2024
d0e9a10
url query is not a valid authentication method, filtered out option
AronBuzogany Mar 6, 2024
6b3e733
using query_agent functions to prevent code duplication
AronBuzogany Mar 6, 2024
a00cb99
split courses into multiple files to keep it organized
AronBuzogany Mar 6, 2024
f4aff02
fixed linting
AronBuzogany Mar 6, 2024
4e0945f
added new courses blueprint
AronBuzogany Mar 6, 2024
05038c9
removed trailing space
AronBuzogany Mar 6, 2024
a142a88
changed test to stay consistent with course admin relation also
AronBuzogany Mar 7, 2024
1f11956
added query agent functions to prevent code duplication
AronBuzogany Mar 7, 2024
7fad876
first version of file uploads
Gerwoud Mar 7, 2024
ca42936
formatting json for posting in the db
Gerwoud Mar 7, 2024
ed94408
Merge branch 'enhancement/endpoints-cleanup' into enhancement/project…
Gerwoud Mar 7, 2024
8d75ae6
working file upload system, reused parser
Gerwoud Mar 7, 2024
11377d1
fixed extracting of zip and uploading in project upload directory
Gerwoud Mar 11, 2024
bc6e4a9
fix
Gerwoud Mar 11, 2024
62632be
fixed tests
Gerwoud Mar 11, 2024
6f323ec
added test zip file
Gerwoud Mar 11, 2024
b5f8ef9
linter fixes
Gerwoud Mar 11, 2024
72a9b7d
removed some test files
Gerwoud Mar 11, 2024
e1f79c7
Merge branch 'development' into enhancement/projects-file-upload
Gerwoud Mar 11, 2024
12d0aa5
linter and test fixes
Gerwoud Mar 11, 2024
e1624b1
import depedency fix
Gerwoud Mar 11, 2024
123992c
fix import order
Gerwoud Mar 11, 2024
2c3a719
removed import getenv
Gerwoud Mar 11, 2024
33880d6
removed valid_project function
Gerwoud Mar 11, 2024
10b0c1a
fix fstring
Gerwoud Mar 11, 2024
0edbd46
fix: upload_directory
Gerwoud Mar 11, 2024
3438972
removed exist_ok=True
Gerwoud Mar 11, 2024
8892d98
use path.join
Gerwoud Mar 11, 2024
c5e3bc4
added url field
Gerwoud Mar 11, 2024
5c6a28d
fixed not checking for tuple type anymore
Gerwoud Mar 11, 2024
0be828f
fixed env var for tests
Gerwoud Mar 11, 2024
adbb14b
fixed env var for tests
Gerwoud Mar 11, 2024
f0be949
fixed with statements
Gerwoud Mar 11, 2024
220856f
using os.path.split instead of regular split
Gerwoud Mar 11, 2024
8c848d6
added exist_ok
Gerwoud Mar 11, 2024
cb7eac1
i forgot :skull: fix lmao yeet
Gerwoud Mar 11, 2024
6db6fad
i forgot :skull: fix lmao yeet
Gerwoud Mar 11, 2024
058f53e
goofy augh fstring
Gerwoud Mar 11, 2024
2314ff6
another small fix
Gerwoud Mar 11, 2024
251ff29
fixed the 'not fixed eh' problem
Gerwoud Mar 11, 2024
f01fed9
linting
Gerwoud Mar 11, 2024
d53eef1
fix handling fail
Gerwoud Mar 11, 2024
dad7015
added try block
Gerwoud Mar 12, 2024
f4fe9fa
linter
Gerwoud Mar 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 26 additions & 10 deletions backend/project/endpoints/projects/endpoint_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,34 @@
"""

from flask_restful import reqparse
import werkzeug

parser = reqparse.RequestParser()
parser.add_argument('title', type=str, help='Projects title')
parser.add_argument('descriptions', type=str, help='Projects description')
parser.add_argument('assignment_file', type=str, help='Projects assignment file')
parser.add_argument("deadline", type=str, help='Projects deadline')
parser.add_argument("course_id", type=str, help='Projects course_id')
parser.add_argument("visible_for_students", type=bool, help='Projects visibility for students')
parser.add_argument("archieved", type=bool, help='Projects')
parser.add_argument("test_path", type=str, help='Projects test path')
parser.add_argument("script_name", type=str, help='Projects test script path')
parser.add_argument("regex_expressions", type=str, help='Projects regex expressions')
parser.add_argument('title', type=str, help='Projects title', location="form")
parser.add_argument('descriptions', type=str, help='Projects description', location="form")
parser.add_argument(
'assignment_file',
type=werkzeug.datastructures.FileStorage,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only import from the dependency what you use.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

help='Projects assignment file',
location="form"
)
parser.add_argument("deadline", type=str, help='Projects deadline', location="form")
parser.add_argument("course_id", type=str, help='Projects course_id', location="form")
parser.add_argument(
"visible_for_students",
type=bool,
help='Projects visibility for students',
location="form"
)
parser.add_argument("archieved", type=bool, help='Projects', location="form")
parser.add_argument("test_path", type=str, help='Projects test path', location="form")
parser.add_argument("script_name", type=str, help='Projects test script path', location="form")
parser.add_argument(
"regex_expressions",
type=str,
help='Projects regex expressions',
location="form"
)


def parse_project_params():
Expand Down
3 changes: 2 additions & 1 deletion backend/project/endpoints/projects/project_detail.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,10 @@
from flask import request
from flask_restful import Resource

from project.models.project import Project
from project.utils.query_agent import query_by_id_from_model, delete_by_id_from_model, \
patch_by_id_from_model
from project.models.project import Project
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the point of this change? You moved this down for no obvious reason.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.




API_URL = getenv('API_HOST')
Expand Down
2 changes: 0 additions & 2 deletions backend/project/endpoints/projects/project_endpoint.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,11 @@
"""

from flask import Blueprint
from flask_restful import Api

from project.endpoints.projects.projects import ProjectsEndpoint
from project.endpoints.projects.project_detail import ProjectDetail

project_bp = Blueprint('project_endpoint', __name__)
project_endpoint = Api(project_bp)

project_bp.add_url_rule(
'/projects',
Expand Down
59 changes: 55 additions & 4 deletions backend/project/endpoints/projects/projects.py
Original file line number Diff line number Diff line change
@@ -1,16 +1,34 @@
"""
Module that implements the /projects endpoint of the API
"""
import os
from os import getenv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either import os, or everything you need from os, not both.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from urllib.parse import urljoin
import zipfile

from flask import request
from flask_restful import Resource

from project.models.project import Project
from project.utils.query_agent import query_selected_from_model, insert_into_model
from project.utils.query_agent import query_selected_from_model, create_model_instance

from project.endpoints.projects.endpoint_parser import parse_project_params

API_URL = getenv('API_HOST')
UPLOAD_FOLDER = getenv('UPLOAD_URL')
ALLOWED_EXTENSIONS = {'zip'}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't there multiple zip extensions we can make work without changing anything in the code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should indeed be possible.
The library for hanling zips will handle this for us u think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.






def allowed_file(filename: str):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to is_file_allowed to make it obvious that it returns a boolean.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is actually removable since it isn't used anywhere.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Than remove it 💀

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then* :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"""
check if file extension is allowed for upload
"""
return '.' in filename and \
filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS


class ProjectsEndpoint(Resource):
"""
Expand Down Expand Up @@ -40,7 +58,40 @@ def post(self):
using flask_restfull parse lib
"""

return insert_into_model(
Project,request.json,
file = request.files["assignment_file"]
project_json = parse_project_params()
filename = file.filename.split("/")[-1]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use the os dependency you already import. That is way safer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean by this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't split by "/"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


# save the file that is given with the request

new_project = create_model_instance(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't handle the case where this fails

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing that fails is me 😄

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Project,
project_json,
urljoin(API_URL, "/projects"),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to f"{API_URL}/"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"project_id")
required_fields=[
"title",
"descriptions",
"course_id",
"visible_for_students",
"archieved"]
)

project_upload_directory = f"{UPLOAD_FOLDER}{new_project.project_id}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You use path.join below, do that here also

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


file_location = "." + os.path.join(project_upload_directory)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the "." it shouldn't be relative, we need to allow storage outside codebase.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


if not os.path.exists(project_upload_directory):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just use, exist_ok=True without the if :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.makedirs(file_location, exist_ok=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you check if it exists if you set exist_ok to True anyway?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


file.save(file_location + "/" + filename)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use path.join

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try:
with zipfile.ZipFile(file_location + "/" + filename) as upload_zip:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not solved eh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

upload_zip.extractall(file_location)
except zipfile.BadZipfile:
return {"message": "Please provide a .zip file for uploading the instructions"}, 400
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing url field

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


return {
"message": "Project created succesfully",
"data": new_project,
"url": f"{API_URL}/projects/{new_project.project_id}"
}, 201
54 changes: 37 additions & 17 deletions backend/project/utils/query_agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,31 @@ def delete_by_id_from_model(
return {"error": "Something went wrong while deleting from the database.",
"url": base_url}, 500


def create_model_instance(model: DeclarativeMeta,
data: Dict[str, Union[str, int]],
response_url_base: str,
required_fields: List[str] = None):
"""
Create an instance of a model
"""
if required_fields is None:
required_fields = []
# Check if all non-nullable fields are present in the data
missing_fields = [field for field in required_fields if field not in data]

if missing_fields:
return {"error": f"Missing required fields: {', '.join(missing_fields)}",
"url": response_url_base}, 400

filtered_data = filter_model_fields(model, data)
new_instance: DeclarativeMeta = model(**filtered_data)
db.session.add(new_instance)
db.session.commit()

return new_instance


def insert_into_model(model: DeclarativeMeta,
data: Dict[str, Union[str, int]],
response_url_base: str,
Expand All @@ -69,26 +94,21 @@ def insert_into_model(model: DeclarativeMeta,
a message indicating that something went wrong while inserting into the database.
"""
try:
if required_fields is None:
required_fields = []
# Check if all non-nullable fields are present in the data
missing_fields = [field for field in required_fields if field not in data]

if missing_fields:
return {"error": f"Missing required fields: {', '.join(missing_fields)}",
"url": response_url_base}, 400

filtered_data = filter_model_fields(model, data)
new_instance: DeclarativeMeta = model(**filtered_data)
db.session.add(new_instance)
db.session.commit()
return jsonify({
new_instance = create_model_instance(model, data, response_url_base, required_fields)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use model_instance, status_code = create_model_instance instead

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# if its a tuple the model instance couldn't be created so it already
# is the right format of error message and we just need to return
if isinstance(new_instance, tuple):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make both cases return status code and check for the status code instead of instance, this isn't scalable

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return new_instance

return (jsonify({
"data": new_instance,
"message": "Object created succesfully.",
"url": urljoin(
f"{response_url_base}/",
str(getattr(new_instance, url_id_field)))}), 201
"url":
urljoin(response_url_base + "/",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use f-string

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

str(getattr(new_instance, url_id_field)))}),
201)
except SQLAlchemyError:
db.session.rollback()
return jsonify({"error": "Something went wrong while inserting into the database.",
"url": response_url_base}), 500

Expand Down
1 change: 1 addition & 0 deletions backend/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ services:
POSTGRES_PASSWORD: test_password
POSTGRES_DB: test_database
API_HOST: http://api_is_here
UPLOAD_URL: /project/endpoints/uploads/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For tests we try to come as close as possible to production. Putting the data storage in our codebase isn't something done in production. Change this to e.g. /data/assignments

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

volumes:
- .:/app
command: ["pytest"]
1 change: 0 additions & 1 deletion backend/tests/endpoints/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ def project(course):
title="Project",
descriptions="Test project",
course_id=course.course_id,
assignment_file="testfile",
deadline=date,
visible_for_students=True,
archieved=False,
Expand Down
16 changes: 12 additions & 4 deletions backend/tests/endpoints/project_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,16 @@ def test_post_project(db_session, client, course_ad, course_teacher_ad, project_
db_session.commit()

project_json["course_id"] = course_ad.course_id
# cant be done with 'with' because it autocloses then
# pylint: disable=R1732
project_json["assignment_file"] = open("testzip.zip", "rb")

# post the project
response = client.post("/projects", json=project_json)
response = client.post(
"/projects",
data=project_json,
content_type='multipart/form-data'
)
assert response.status_code == 201

# check if the project with the id is present
Expand All @@ -34,7 +41,6 @@ def test_post_project(db_session, client, course_ad, course_teacher_ad, project_

assert response.status_code == 200


def test_remove_project(db_session, client, course_ad, course_teacher_ad, project_json):
"""Test removing a project to the datab and fetching it, testing if it's not present anymore"""

Expand All @@ -46,8 +52,11 @@ def test_remove_project(db_session, client, course_ad, course_teacher_ad, projec

project_json["course_id"] = course_ad.course_id

# cant be done with 'with' because it autocloses then
# pylint: disable=R1732
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be done by putting client.post in the with also?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

project_json["assignment_file"] = open("testzip.zip", "rb")
# post the project
response = client.post("/projects", json=project_json)
response = client.post("/projects", data=project_json)

# check if the project with the id is present
project_id = response.json["data"]["project_id"]
Expand All @@ -59,7 +68,6 @@ def test_remove_project(db_session, client, course_ad, course_teacher_ad, projec
response = client.delete(f"/projects/{project_id}")
assert response.status_code == 404


def test_patch_project(db_session, client, course_ad, course_teacher_ad, project):
"""
Test functionality of the PUT method for projects
Expand Down
Binary file added backend/testzip.zip
Binary file not shown.