Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve documentation to describe how to mount a dedicated temp disk/partition/folder. #5496

Closed
JMarcosHP opened this issue Oct 20, 2024 · 13 comments
Labels
0. Needs triage Pending approval or rejection. This issue is pending approval. documentation Improvements or additions to documentation help wanted Extra attention is needed

Comments

@JMarcosHP
Copy link
Contributor

How to use GitHub

  • Please use the 👍 reaction to show that you are interested into the same feature.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.
Currently public file uploads are temporarily stored in /tmp and then moved to the actual location in the nextcloud data directory.

This can be a problem for users who do not have a dedicated partition with fixed size for /tmp or users who use Nextcloud-AIO in docker as it can completely fill up the partition or location where they have the docker installation files.

Describe the solution you'd like

It is known that for nextcloud 31 public file uploads will be migrated to vue in the application files and will support chunking, so why not take advantage and change the way they are stored temporarily, for example in the data directory as well as uploads are stored for registered users (data/user/uploads) can be /data/public/uploads in this case.

Additional context

(See)

@JMarcosHP JMarcosHP added 0. Needs triage Pending approval or rejection. This issue is pending approval. enhancement New feature or request labels Oct 20, 2024
@joshtrichards
Copy link
Member

I believe tempdirectory should already cover this scenario: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#tempdirectory

@JMarcosHP
Copy link
Contributor Author

JMarcosHP commented Oct 20, 2024

I believe tempdirectory should already cover this scenario: https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/config_sample_php_parameters.html#tempdirectory

That's a good point @joshtrichards, thank you.

@szaimen

Since Nextcloud-AIO is the official installation method, it can be a good idea let the users decide how to manage the tempdirectory for that usecase, the same way as the NEXTCLOUD_DATADIR

And suggest the users to have a large enough /tmp partition configured if they need support for big file uploads on NEXTCLOUD_UPLOAD_LIMIT as the documentation recommends.

Edit:
Let's wait until chunk size enhancement for Nextcloud 31 is finished.

@JMarcosHP JMarcosHP changed the title Change the folder where public upload chunks are stored. [Enhancement]: Change the folder where public upload chunks are stored. Oct 22, 2024
@joshtrichards joshtrichards changed the title [Enhancement]: Change the folder where public upload chunks are stored. Change the folder where public upload chunks are stored Oct 22, 2024
@susnux
Copy link

susnux commented Oct 24, 2024

Is there anything left? I think it is already handled by the tempdirectory variable mentioned by Josh

@JMarcosHP
Copy link
Contributor Author

JMarcosHP commented Oct 25, 2024

Is there anything left? I think it is already handled by the tempdirectory variable mentioned by Josh

In this case, on the Nextcloud-AIO (Docker official installation method) currently there is no way to mount a custom directory from the host to the nextcloud container tempdirectory.

If you try to upload a big file using a public link, the nextcloud container starts to grow until the upload file is moved to the data directory.

As the result, if you have your docker installation on a small drive, uploads can fail because you are running out of space (See nextcloud/server#5356)

The workaround for this is to move your entire docker installation to another drive with enough space or deploy a custom docker compose like linuxserver.io images and set a mountpoint for /tmp until this feature is approved on Nextcloud-AIO or Nextcloud server devs migrate public upload chunks to the data directory.

@joshtrichards
Copy link
Member

You should be able to point tempdirectory at any arbitrary location, such as a folder within your datadirectory or similar. For a time that's what AIO did automatically, see #2868

I don't see why that can't be done at your discretion still today if desired.

@susnux
Copy link

susnux commented Oct 28, 2024

In this case, on the Nextcloud-AIO (Docker official installation method) currently there is no way to mount a custom directory from the host to the nextcloud container tempdirectory.

Mount host directory to the Nextcloud container:
https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-allow-the-nextcloud-container-to-access-directories-on-the-host

Afterwards you should be able to set tempdirectory to that mount location.

@JMarcosHP
Copy link
Contributor Author

JMarcosHP commented Oct 29, 2024

In this case, on the Nextcloud-AIO (Docker official installation method) currently there is no way to mount a custom directory from the host to the nextcloud container tempdirectory.

Mount host directory to the Nextcloud container: https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-allow-the-nextcloud-container-to-access-directories-on-the-host

Afterwards you should be able to set tempdirectory to that mount location.

I tested this on a VM

But you need to modify some files inside the nextcloud container.

@joshtrichards
FIRST OF ALL, tempdirectory directive is ignored for file uploads as a known limitation from PHP, so you need to set this setting for nextcloud.ini php file and www.conf file for php-fpm, then you need to set the correct owner for the mounted directory (www-data).

@susnux
Steps to reproduce:

  1. Make your custom temp directory on a host disk (SSD recommended), then add the mountpoint on NEXTCLOUD_MOUNT:
#: mkdir /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp

docker-compose.yml
...
NEXTCLOUD_MOUNT= /srv # I want to mount the whole /srv directory to access other disks.
...
  1. Set your custom domain and download the preferred apps on the mastercontainer.
  2. Once all containers are ready, please don't login on nextcloud yet.
  3. Adjust nextcloud.ini and www.conf files:
#: docker exec -it nextcloud-aio-nextcloud bash

(container)#: vi /usr/local/etc/php/conf.d/nextcloud.ini

# Add this line
upload_tmp_dir= /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp

(container)#: vi /usr/local/etc/php-fpm.d/www.conf

# Uncomment this lines and add your mountpoint:
env[TMP] = /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp
env[TMPDIR] = /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp
env[TEMP] = /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp
  1. Save the config files and set the www-data ownership to your custom temp directory:
(container)#: chown -R www-data /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp
  1. Add the tempdirectory directive and restart the container:
(container)#: exit
#: docker exec -it -u www-data nextcloud-aio-nextcloud php occ config:system:set tempdirectory --value /srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp
#: docker restart nextcloud-aio-nextcloud
  1. Upload a file on a public folder link to test and watch your custom temp directory until the upload is finished:
#: watch -n 1 "/srv/dev-disk-by-uuid-73410a39-3979-4bd0-a3f9-a79d9a02533a/nctmp"

If you get an error when the upload is finished check the permissions and ownership of the temp folder. (Step 5)

@JMarcosHP
Copy link
Contributor Author

@szaimen
It could be a good idea to add this on the documentation of Nextcloud-AIO

@susnux susnux transferred this issue from nextcloud/server Oct 29, 2024
@susnux
Copy link

susnux commented Oct 29, 2024

I transferred this from server to aio as this is purely a configuration issue.
For PHP tempfiles are stored in upload_tmp_dir and for the temporary files you can use the Nextcloud tempdirectory setting.

So this is just configuration of AiO

@JMarcosHP JMarcosHP changed the title Change the folder where public upload chunks are stored Improve documentation to describe how to mount a dedicated temp disk/partition/folder. Oct 29, 2024
@ErikSteiner
Copy link

ErikSteiner commented Oct 30, 2024

@susnux Steps to reproduce:

1. Make your custom temp directory on a host disk (SSD recommended), then add the mountpoint on `NEXTCLOUD_MOUNT`:

...

Do I understand correctly that the adjustment of the temporary directory involves two parts that have nothing to do with each other? As far as I understand it, can the temporary directory of PHP be in a different location than the temporary directory of Nextcloud?

As far as I can see it, I have just an issue with the temporary files of PHP then, because those phpXXXX folders are filling up my container and the the docker host.

@szaimen szaimen added documentation Improvements or additions to documentation and removed enhancement New feature or request labels Oct 30, 2024
@JMarcosHP
Copy link
Contributor Author

@susnux Steps to reproduce:

1. Make your custom temp directory on a host disk (SSD recommended), then add the mountpoint on `NEXTCLOUD_MOUNT`:

...

Do I understand correctly that the adjustment of the temporary directory involves two parts that have nothing to do with each other? As far as I understand it, can the temporary directory of PHP be in a different location than the temporary directory of Nextcloud?

As far as I can see it, I have just an issue with the temporary files of PHP then, because those phpXXXX folders are filling up my container and the the docker host.

Yes sir, we can use the temp directory of nextcloud but this directive is ignored when you upload files, since PHP and PHP-FPM have their own ENV variables for the tmp directory is better to point all to a fixed mountpoint where Nextcloud and PHP stores all his files to avoid issues.

@ErikSteiner
Copy link

OK, but is there any known interaction between PHP-FPM and Nextcloud, that would require a path, that match for both?

@szaimen szaimen added the help wanted Extra attention is needed label Oct 30, 2024
@JMarcosHP
Copy link
Contributor Author

JMarcosHP commented Oct 31, 2024

I tested, and yes.
The Nextcloud docker image is based on PHP-FPM, some apps like memories save temporary files, for example when you download geocode database.

@nextcloud nextcloud locked and limited conversation to collaborators Nov 14, 2024
@szaimen szaimen converted this issue into discussion #5594 Nov 14, 2024

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
0. Needs triage Pending approval or rejection. This issue is pending approval. documentation Improvements or additions to documentation help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

5 participants