Objectstore multibucket #24760

rullzer · 2016-05-22T19:22:45Z

PR to allow multiple buckets being used when objectstore is used as homestorage.

A config section called 'objectstore_multibucket' is read in. Which has precedence over the normal objectstore config section.
This multibucket objectstore should for now only be used with new installations (same restriction for now as the single bucket implementation).
For now we only provide our own mapper from username => bucket. Which is MD5 the username and take the first 3 chars

Note that currently only the s3 homeobjectstore will use the mutlibuckets (we could extend swift but for now I would not do that as we have enough to tests with s3 already).

TODO:

Unit tests for ObjectHomeMountProvider

CC: @butonic @icewind1991 @PVince81 comments please.

mention-bot · 2016-05-22T19:22:46Z

By analyzing the blame information on this pull request, we identified @icewind1991 to be a potential reviewer

rullzer · 2016-05-23T09:03:20Z

Unit tests added

PVince81 · 2016-05-23T09:08:55Z

lib/private/Files/Mount/ObjectHomeMountProvider.php

@@ -52,9 +52,27 @@ public function __construct(IConfig $config) {
 	 * @return \OCP\Files\Mount\IMountPoint[]
 	 */
 	public function getHomeMountForUser(IUser $user, IStorageFactory $loader) {
+
+		$config = $this->multiBucketObjectStore($user);


getConfigForMultiBuckerObjectStore ? 😉
A method name that doesn't really say what the method does is a bit confusing

PVince81 · 2016-05-23T09:12:00Z

Please adjust "config.sample.php" for the new values.

I'm not 100% convinced about having a separate config section just for multi-bucket.
The way I understand it is that multi-bucket is basically the same as before but where the bucket name is dynamic. So extending the existing config would be simpler. @rullzer what do you think ?

PVince81 · 2016-05-23T09:14:26Z

Remember when we talked about it we said it would be good to be able to specify the Mapper class name. I see you haven't added the config option for that yet.

Maybe considering my comments above about the separate config, maybe we could replace the "bucket" argument in the settings with this:

"bucketMapper" => "OC\Files\ObjectStore\Mapper"

So if there is a "bucketMapper" arg, it has precedence over "bucket".

rullzer · 2016-05-23T09:17:20Z

I did not yet edit the config.php since this does not work for the swift home storage atm.

I created a separate config as a future proof thingy. Since if we reuse the config there would be no proper way for us to migrate from single to multibucket.

rullzer · 2016-05-23T09:18:02Z

Yes eventually we'd want the mapper to be specifiable. But for now I think 1 default is fine. We should first properly test this all before we do all the fancy stuff. Should be trivial to add btw.

PVince81 · 2016-05-23T09:23:09Z

Hmmm ok, then fine by me for now 👍

Would be good to know what @butonic @icewind1991 and @owncloud/qa think

rullzer · 2016-05-23T17:07:11Z

Ah forgot to copy the commit that saves the user mapping to the db... will follow later this evening.

rullzer · 2016-05-23T19:07:13Z

ok now we properly save the bucket in the preference table.
Unit tests updated and extended

PVince81 · 2016-05-23T19:24:09Z

.htaccess

+#### DO NOT CHANGE ANYTHING ABOVE THIS LINE ####
+
+ErrorDocument 403 /core/templates/403.php
+ErrorDocument 404 /core/templates/404.php


@rullzer 🙀

Ah my bad... fixed

PVince81 · 2016-05-23T19:25:06Z

lib/private/Files/Mount/ObjectHomeMountProvider.php

+			$mapper = new \OC\Files\ObjectStore\Mapper($user);
+			$config['arguments']['bucket'] .= $mapper->getBucket();
+
+			$this->config->setUserValue($user->getUID(), 'homeobjectstore', 'bucket', $config['arguments']['bucket']);


butonic · 2016-05-24T07:17:26Z

lib/private/Files/Mount/ObjectHomeMountProvider.php

+		}
+		$config['arguments']['user'] = $user;
+
+		$bucket = $this->config->getUserValue($user->getUID(), 'homeobjectstore', 'bucket', null);


For swift you would call this container. In ceph you might want to call it pool. We should stick to one terminology. I guess s3 is the most prominent, so bucket is fine. At least for our internal config.

Yeah we have to use something... I really don't care what ;)

PVince81 · 2016-05-24T10:17:38Z

@rullzer @DeepDiver1975 @butonic so should we defer the whole thing for 9.2 to have a more rounded solution ? Or is there a benefit of having this merged first for 9.1 where some people could already setup their new systems to use multi-bucket ?

CC @cmonteroluque

rullzer · 2016-05-24T10:19:29Z

I think moving it to 9.2 makes sense. To have things properly tested and reviewed.

MTRichards · 2016-05-24T11:34:28Z

I think moving it to 9.2 makes sense. To have things properly tested and reviewed.

In general, I agree, but we do have a few who are expecting this feature in 9.1.

As I understand it, multibucket is ok, but one can't migrate into it, it will only work for all new installations. Is that correct?

Is there a logical path to migration from there, or would new installs be isolated and stuck in the future?

rullzer · 2016-05-24T11:41:21Z

They can only be new installations. And then it should be the same.
And right now it only really works with S3..

PVince81 · 2016-05-24T13:08:55Z

Ok, then let's get this in in the current form.

Needs a second reviewer @butonic @DeepDiver1975 @icewind1991 @Xenopathic

icewind1991 · 2016-05-24T13:11:06Z

👍 I'm fine with this as an initial version

lock · 2019-08-05T17:02:42Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

rullzer added enhancement 2 - Developing app:objectstore labels May 22, 2016

rullzer added this to the 9.1-current milestone May 22, 2016

rullzer force-pushed the objectstore_multibucket branch from 6d043f9 to 492a026 Compare May 23, 2016 09:01

rullzer added 3 - To Review and removed 2 - Developing labels May 23, 2016

PVince81 reviewed May 23, 2016
View reviewed changes

rullzer force-pushed the objectstore_multibucket branch from 492a026 to f0391bf Compare May 23, 2016 09:19

rullzer added 2 - Developing and removed 3 - To Review labels May 23, 2016

rullzer added 4 commits May 23, 2016 20:42

Allow multibucket in objectstore

5e2316d

Add mapper unit tests

12b6325

Add unit tests for ObjectHomeMountProvider

7ef21b0

Fix Name

e03e492

rullzer force-pushed the objectstore_multibucket branch from f0391bf to 920d253 Compare May 23, 2016 19:05

rullzer added 3 - To Review and removed 2 - Developing labels May 23, 2016

PVince81 reviewed May 23, 2016
View reviewed changes

Store user bucket in preferences

abe338f

rullzer force-pushed the objectstore_multibucket branch from 920d253 to abe338f Compare May 23, 2016 20:00

butonic reviewed May 24, 2016
View reviewed changes

PVince81 added 4 - To release and removed 3 - To Review labels May 24, 2016

PVince81 merged commit e7110c7 into master May 24, 2016

PVince81 deleted the objectstore_multibucket branch May 24, 2016 13:16

lock bot locked as resolved and limited conversation to collaborators Aug 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Objectstore multibucket #24760

Objectstore multibucket #24760

rullzer commented May 22, 2016 •

edited

Loading

mention-bot commented May 22, 2016

rullzer commented May 23, 2016

PVince81 May 23, 2016

PVince81 commented May 23, 2016

PVince81 commented May 23, 2016

rullzer commented May 23, 2016

rullzer commented May 23, 2016

PVince81 commented May 23, 2016

rullzer commented May 23, 2016

rullzer commented May 23, 2016

PVince81 May 23, 2016

rullzer May 23, 2016

PVince81 May 23, 2016

butonic May 24, 2016

rullzer May 24, 2016

PVince81 commented May 24, 2016

rullzer commented May 24, 2016

MTRichards commented May 24, 2016

rullzer commented May 24, 2016

PVince81 commented May 24, 2016

icewind1991 commented May 24, 2016

lock bot commented Aug 5, 2019

Objectstore multibucket #24760

Objectstore multibucket #24760

Conversation

rullzer commented May 22, 2016 • edited Loading

mention-bot commented May 22, 2016

rullzer commented May 23, 2016

PVince81 May 23, 2016

Choose a reason for hiding this comment

PVince81 commented May 23, 2016

PVince81 commented May 23, 2016

rullzer commented May 23, 2016

rullzer commented May 23, 2016

PVince81 commented May 23, 2016

rullzer commented May 23, 2016

rullzer commented May 23, 2016

PVince81 May 23, 2016

Choose a reason for hiding this comment

rullzer May 23, 2016

Choose a reason for hiding this comment

PVince81 May 23, 2016

Choose a reason for hiding this comment

butonic May 24, 2016

Choose a reason for hiding this comment

rullzer May 24, 2016

Choose a reason for hiding this comment

PVince81 commented May 24, 2016

rullzer commented May 24, 2016

MTRichards commented May 24, 2016

rullzer commented May 24, 2016

PVince81 commented May 24, 2016

icewind1991 commented May 24, 2016

lock bot commented Aug 5, 2019

rullzer commented May 22, 2016 •

edited

Loading