UUID Support

[xfractured]

The upcoming release of 1.7.6 (and above) will be using UUIDs instead of player names since Mojang will be adding the ability to change player names soon.
It is very important that LWC implements this as soon as possible to prevent any issues from arising.

http://forums.bukkit.org/threads/psa-the-switch-to-uuids-potential-plugin-server-breakage.250915/

[dddeeefff]

I second this. This is incredibly important.

[Squawkers13]

Agreed.

[FourDown]

@Hidendra

Is this something you are working on?

[Hidendra]

I have been incredibly busy the last couple months with university so I've been slacking a fair bit w/regards LWC :(

I'm going to try my best to get this done tonight

[dddeeefff]

That is really ambitious! I will be genuinely massively impressed if you
can do it in one night.

Thanks a lot. I wish you the best with your university work.

[Hidendra]

One night is indeed a stretch but a start is better than nothing :) Given how the current LWC stores some things in the DB conversion is going to be a pain..

[dddeeefff]

Yes, I noticed that. I don't envy you.

[dddeeefff]

I thought I'd add this:
http://forums.bukkit.org/threads/player-name-uuid-fetcher.250926/

I've been using this to get UUIDs into a database, since bukkit doesn't
have a way to get the UUID of a player that has not connected since it
started storing them. It seems reliable and fast, but of course it may
return nothing for players who don't exist - for example if a server was
put into offline mode briefly as a test, or if someone connected with a
fake name whilst the spigot race bug was in effect

[Hidendra]

Indeed, that's (offline mode) probably the most annoying thing about it all..

The actual mass conversion shouldn't be so bad (slooowly in the background).

[TNTUP]

yay you're there :D Can't wait to see LWC updated

[Hidendra]

The initial work should be ready in builds 914 onwards.

At the moment, conversions are done in place as protections are seen. No full conversion is done automatically yet.

If you want to confirm if a protection has been converted, you may use /cinfo, e.g.:

Rights management will also resolve UUIDs automatically. Unknown players will still be added, and then resolved to the UUID when they actually exist at some point in the future.

[TNTUP]

yay :D Is it confirmed to work in 1.7.5? or its only for 1.7.9? I have a database of 10.2k of protections... they will be converted or not? (I guess yes later?)

[Hidendra]

• Is it confirmed to work in 1.7.5

Yes, seems fine (tested on b3042).

Actually, 1.6.4 seems to work perfectly fine too even though it does still convert to UUIDs

• they will be converted or not? (I guess yes later?)

If they're all cached at startup (large precache size), they'll all be converted on startup. If precache was (e.g.) 1000, 1000 protections would be converted, followed by any that are accessed by players in the world.

If you do try it, of course I'd recommend taking a backup first if you haven't already :-)

[TNTUP]

Haha Sweet! Will do tests!

[mibby]

@Hidendra

Just thought to chime in. Using dev build 914 on 1.7.9. Mysql seems to hang every single server start-up. Not sure if it is converting or not, but takes ~10-15 minutes to load past Mysql. After restarting the server, I have to go through the entire wait process again. Cache size of 75,000, precache -1.

Config: http://paste.ubuntu.com/7306183/

Using BungeeCord (online) and Spigot (offline). Bungee forwards the online UUID to Spigot, but I believe Spigot does the UUID conversions online regardless of setting.

[04:06:51] [Server thread/INFO]: [LWC] Enabling LWC v4.5.0-SNAPSHOT
[04:06:52] [Server thread/INFO]: [LWC] Connecting to MySQL
[04:16:48] [Server thread/WARN]: **** SERVER IS RUNNING IN OFFLINE/INSECURE MODE!
[04:16:48] [Server thread/WARN]: The server will make no attempt to authenticate usernames. Beware.
[04:16:48] [Server thread/WARN]: While this makes the game possible to play without internet access, it also opens up the ability for hackers to connect with any username they choose.
[04:16:48] [Server thread/WARN]: To change this, set "online-mode" to "true" in the server.properties file.

Edit: Ran /lwc info and punched a protected door, no UUID output. Didn't seem to convert old data. Also restarted with precache 75,000, still didn't convert old data. Just hangs for 10 minutes until it starts. :(

Newly protected chests seem to have UUID, but not formatted correctly nor does it have a name reference. http://i.imgur.com/AuTMOno.png

[Hidendra]

Thanks @mibby and just to confirm, protections are working fine still? I
kind of expected a long wait for large precaches so I'll make the change
tonight when I get home so it no longer does that.
On Apr 22, 2014 8:05 AM, "mibby" notifications@github.com wrote:

@Hidendra https://github.com/Hidendra

Just thought to chime in. Using dev build 914 on 1.7.9. Mysql seems to
hang every single server start-up. Not sure if it is converting or not, but
takes ~10 minutes to load past Mysql. After restarting the server, I have
to go through the entire wait process again. Precache size of 75,000.

Config: http://paste.ubuntu.com/7306183/

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/811#issuecomment-41027204
.

[mibby]

I'm not sure, only tested with myself online unfortunately, sorry. All I know is regardless of what settings I use, old protections aren't being converted automatically and it takes an extremely long time to load regardless of what is set. When a new protection is created or when a user uses his own protection, it seems to update with the UUID.

It would be nice to have all protections automatically convert and not have a long wait every single restart. :P

[dddeeefff]

I'm using the version before you disabled automatic conversion of the cache in favour of interaction (so that I an grab all the uuids then use the main version) but the update to UUIDs was imperfect:

mysql> select count(*) from lwc_protections where character_length(owner) < 17;
+----------+
| count(*) |
+----------+
|    79296 |
+----------+

mysql> select * from lwc_protections where character_length(owner) < 17 LIMIT 3;
+--------+------+---------+----------+-----------+----------+-------+------+-------+-------------------------+---------------+------+--------+
| id     | type | blockId | world    | owner     | password | x     | y    | z     | date                    | last_accessed | data | rights |
+--------+------+---------+----------+-----------+----------+-------+------+-------+-------------------------+---------------+------+--------+
| 320084 |    2 |      61 | w_novum  | [PLAYER1] |          |  -742 |   58 |  6496 | 2012-08-19 23:18:55.861 |    1345414735 | NULL | NULL   |
| 320072 |    2 |      68 | w_protos | [PLAYER2] |          |  2013 |   65 |  2861 | 2012-08-19 23:02:27.699 |    1345413747 | NULL | NULL   |
| 118542 |    2 |      68 | w_protos | [PLAYER3] |          | -2740 |   38 | -1011 | 2011-12-31 05:54:52.934 |    1325307292 | NULL | NULL   |
+--------+------+---------+----------+-----------+----------+-------+------+-------+-------------------------+---------------+------+--------+

These are players for whom I know there is is a UUID. The lack of conversion associated with a null data column:

mysql> select count(*) from lwc_protections where character_length(owner) < 17 AND data is null;
+----------+
| count(*) |
+----------+
|    79289 |
+----------+

That's a difference of 7. So I ran this query:

mysql> select * from lwc_protections where character_length(owner) < 17 AND data is not null;

All the players listed by the above query do not exist - they are effectively offline mode names. We're not an offline mode server but from time to time things have come up where offline names have been used e.g. when spigot had that login bug

In conclusion: UUIDs are not being updated where the data column is null.

I want to bypass this issue using the following query but I would like some sort of confirmation that is won't stuff up my database first.

update lwc_protections set data = {"flags":[],"rights":[]} where data is null

---

lwc_history does not convert to UUIDs either. I'm not sure if this is a problem:

mysql> select id,protectionId,player,type,status,metadata,from_unixtime(timestamp) from lwc_history order by id desc limit 2;
+--------+--------------+----------+------+--------+----------------------------------------------------------------+--------------------------+
| id     | protectionId | player   | type | status | metadata                                                       | from_unixtime(timestamp) |
+--------+--------------+----------+------+--------+----------------------------------------------------------------+--------------------------+
| 511485 |       529559 | [PLAYER4]|    0 |      0 | creator=[PLAYER4],                                             | 2014-04-26 08:57:29      |
| 511484 |       529558 | [PLAYER5]|    0 |      1 | creator=[PLAYER5],destroyer=[PLAYER5],destroyerTime=1398489515,| 2014-04-26 07:13:59      |
+--------+--------------+----------+------+--------+----------------------------------------------------------------+--------------------------+

---

lwc_menu_styles (I'm not even sure if this is used any more) doesn't update either:

mysql> select * from lwc_menu_styles limit 2;
+--------------+----------+
| player       | menu     |
+--------------+----------+
| [player6]    | basic    |
| [player7]    | basic    |
+--------------+----------+

---

lwc_actions, lwc_locks, lwc_modes, lwc_sessions and lwc_jobs are empty tables in my database so I can't confirm whether these are up to date or not.

---

Can I suggest an alternative arrangement for the storage of UUIDs in mysql?
Set up a table called lwc_players, with an auto-incrementing primary field, a name column (not unique) and a UUID column.

The advantages of this are: smaller database size (with quicker lookups?) and you don't have to rely on bukkit's methods to get an offline player by their UUID since player.dat files are occasionally corrupted when servers crash

[ralnivar]

I agree with @dddeeefff I would recommend to use a lwc_players table, By that making the storage table not dependent on player names / uuids.

[Hidendra]

good catch, although at the moment only lwc_protections is converted.

@dddeeefff I think that would certainly be more interesting to do (and straightforward). I assume you're also talking about converting all player fields in the DB to use integers instead of strings? I think in practice that would definitely be a lot faster, and is something I did as well in the LWC v5 rewrite (though far from done).

At the moment, you've likely already noticed that UUID conversion indeed isn't done on player names that don't exist. So it's reconverted everytime. A single conversion (to a players table) should resolve unneeded duplicate lookups on Mojang's servers, too.

The only problem is actually converting it in that case. Some LWC databases are massive (millions of protections). Simply creating a new set of tables seems like the easiest (and then slowly copying over) so that it can be ran in the background (while putting new protections in the new table).

If all of that sounds about right to you I'll go ahead and toy around with it on a second branch (any associated builds with be on a second Jenkins project.)

btw - lwc_actions, lwc_locks, lwc_modes, lwc_sessions, and lwc_jobs are REALLY old tables. They haven't been used for a very long time (and can be dropped if you don't want them there.)

edit: the other interesting thing to note is that doing it with a separate players table can usually make the implementation completely transparent to all of the ugly changes I had to make just to get UUIDs working okay, since it could transparently decode/encode player usernames/uuids to the int ID.

[dddeeefff]

I figured these were old tables. The server and its maps are really old now.
Would I be able to use this query to get LWC to convert over those ones that haven't converted?

update lwc_protections set data = {"flags":[],"rights":[]} where data is null

[Hidendra]

I'm not really seeing why it wouldn't convert if data is null. It checks the owner field to see if it's not a UUID, and then anything that could be inside data.

If you're on the latest build, then the only protections that are converted at the moment are ones that a player interacts with physically.

[dddeeefff]

I'm not I deliberately chose the second most recent because I wanted to get it all moved over. I was going to move to the newest build afterwards to prevent that loadup time

I thought maybe they might not be loaded into the cache to begin with, but a quick count(*) check on the table shows there are roughly the same amount of rows.
I'll dig around some more

[dddeeefff]

Okay I found one protection and /cinfoed it. It's now no longer an unconverted one. I don't fancy doing that with all 74000 unconverted ones though. Here's that protection:

mysql> select * from lwc_protections where id=320084;
+--------+------+---------+---------+--------------------------------------+----------+------+------+------+-------------------------+---------------+--------------------------+--------+
| id     | type | blockId | world   | owner                                | password | x    | y    | z    | date                    | last_accessed | data                     | rights |
+--------+------+---------+---------+--------------------------------------+----------+------+------+------+-------------------------+---------------+--------------------------+--------+
| 320084 |    2 |      61 | w_novum | 12345678-1234-1234-1234-123456789012 |          | -742 |   58 | 6496 | 2012-08-19 23:18:55.861 |    1345414735 | {"flags":[],"rights":[]} | NULL   |
+--------+------+---------+---------+--------------------------------------+----------+------+------+------+-------------------------+---------------+--------------------------+--------+

Perhaps there's something blocking the update of the protection because you're trying to match the flags and rights as well.

[Hidendra]

Full conversion will be done once I've settled on a definite solution although I'm thinking the separate players table is likely the better of them.

[TNTUP]

@Hidendra Planning to use Spigot 1.7.9, I wonder, is it safe to use LWC in dev version? I know its subject to bugs but Im unsure if the current stable LWC is working on 1.7.9

[Hidendra]

@TNTUP The latest builds should be fine; no conversion is done in them right now.

This should be mostly completed tomorrow .. I haven't pushed much to the feature/811-uuid-conversion branch yet as I want the actual conversion to be (mostly) working correctly incase someone borks their database by trying to run it. But right now it looks good, just needs to be looked at during the day.

The biggest problem I had was issues I'm having with MySQL on Windows and a very large LWC test db (10 GB) -- works ok in the MySQL prompt but when LWC's connected with JDBC it's extremely slow (even selects on an empty table). I'll likely have to try reimporting it...

[Hidendra]

This should now be working as it should on the 811-uuid-conversion branch / jobs. Note that they are NOT backwards compatible (at all) with any other builds.

Does anyone want to give this a shot on a test server (with a copy of your database)? I would appreciate it a lot, incase I've missed something even after using quite a few different DBs.

Sample output (when the server starts):

[17:47:48 INFO]: [LWC] Pre-loading distinct player names. This might take a minute or two.
[17:47:48 INFO]: [LWC] Once complete, everything else will run in the background
[17:47:48 INFO]: [LWC] Converting 4 names to normalized format (background task)
[17:47:48 INFO]: [LWC] [100.00% 4/4] Converted 0 players in 0ms
[17:47:48 INFO]: [LWC] Completed conversion for player names
[17:47:48 INFO]: [LWC] Moving to next stage
[17:47:48 INFO]: [LWC] Renaming old protections table
[17:47:48 INFO]: [LWC] Creating new protections table
[17:47:48 INFO]: [LWC] Restoring indexes
[17:47:48 INFO]: [LWC] Creating new history table
[17:47:48 INFO]: [LWC] Restoring indexes
[17:47:48 INFO]: [LWC] Beginning conversion of old protections
[17:47:50 INFO]: [LWC] [100.00% 114/114] Converted 0 rows in the table lwc_protections_old_converting in 0ms
[17:47:50 INFO]: [LWC] Completed conversion of lwc_protections
[17:47:50 INFO]: [LWC] Moving to next stage
[17:47:50 INFO]: [LWC] Converted all protections to new format
[17:47:50 INFO]: [LWC] Beginning conversion of old history
[17:47:53 INFO]: [LWC] [100.00% 211/211] Converted 0 rows in the table lwc_history_old_converting in 0ms
[17:47:53 INFO]: [LWC] Completed database conversion for UUIDs
[17:47:53 INFO]: [LWC] Converted all history to new format

Once completed, your database will then look something like this (tables wise):

sqlite> .tables
lwc_history          lwc_internal         lwc_protections    
lwc_history_old      lwc_players          lwc_protections_old

Sample contents of the new tables:

lwc_players

2|069a79f4-44e9-4726-a5be-fca90e38aaf5|Notch
3|86553713-3d20-4923-9fd6-587aa7ed7c16|Hidendra
4|7c6d3775-9dd7-4934-9f2c-32303a367226|test

id => (uuid, name)

One of uuid or name can be null. This is to allow smooth transitions (unknown players) and will be updated automatically (not done yet).

lwc_protections

1|4|2|-159|4|1234||{"flags":[],"rights":[]}|54|world||2014-01-01 20:14:54.626|1398038295
2|3|2|-148|4|1233||{"flags":[],"rights":[{"name":"5","rights":1,"type":1}]}|54|world||2014-04-20 19:17:18.588|1398043301
3|1|2|-148|4|1235||{"flags":[],"rights":[{"name":"5","rights":1,"type":1},{"name":"6","rights":1,"type":1},{"name":"2","rights":1,"type":1},{"name":"3","rights":1,"type":1}]}|54|world||2014-04-20 20:31:28.335|1398043919

Changes: owner is now INT, and rights.name is now an integer for the player (for type = 1 = PLAYER)

lwc_history

1|1|3|-159|4|1234|0|0|{"creator":"3"}|1399582262
2|2|3|-148|4|1233|0|0|{"creator":"3"}|1399582262
3|3|1|-148|4|1235|0|0|{"creator":"1"}|1399582262

Changes: player is now INT, data is now JSON, and creator/destroyer inside of the data field use integers for the player

[Hidendra]

and also: most commands (if not all) should be working as expected. The only "weird" thing you might see is that player output in some places will still show something such as PlayerInfo(id = 3, uuid = 86553713-3d20-4923-9fd6-587aa7ed7c16, name = Hidendra) for the time being.

[ralnivar]

@Hidendra Will run it on my test server, Will report back soon.

EDIT: On the first run (Start -> Stop, Change config -> start) (MYSQL DB)
http://pastebin.com/Qk0Juuc1
Got Null pointer error on the Converting * names to normalized format (background task),
Restarted the server, Converted, It looks like Everything is converted, Haven't tested it ingame, I do not have the world.

[Hidendra]

@raldo94 Thank you. It looks like that was because it converted SQLite -> MySQL but it then tried converting the SQLite database afterwards after it was closed. It should convert without error now.

Everything did convert correctly before except for the creator/destroyer in lwc_history's metadata which is largely irrelevent. It's only relevent if you want to see who created/destroyed a protection that no longer exists.

[mibby]

Spamming this error when trying to convert, latest UUID build. http://ci.griefcraft.com/job/LWC%20811-uuid-conversion/7/

[02:39:54] [Server thread/INFO]: [LWC] Beginning conversion of old history
[02:39:54] [Server thread/WARN]: [LWC] Task #783 for LWC v4.5.0-SNAPSHOT generated an exception
java.lang.NullPointerException
    at com.griefcraft.migration.uuid.HistoryRowHandler.handle(HistoryRowHandler.java:36) ~[?:?]
    at com.griefcraft.migration.SimpleTableWalker.run(SimpleTableWalker.java:88) ~[?:?]
    at org.bukkit.craftbukkit.v1_7_R3.scheduler.CraftTask.run(CraftTask.java:58) ~[spigot.jar:git-Spigot-1465]
    at org.bukkit.craftbukkit.v1_7_R3.scheduler.CraftScheduler.mainThreadHeartbeat(CraftScheduler.java:345) [spigot.jar:git-Spigot-1465]
    at net.minecraft.server.v1_7_R3.MinecraftServer.v(MinecraftServer.java:628) [spigot.jar:git-Spigot-1465]
    at net.minecraft.server.v1_7_R3.DedicatedServer.v(DedicatedServer.java:283) [spigot.jar:git-Spigot-1465]
    at net.minecraft.server.v1_7_R3.MinecraftServer.u(MinecraftServer.java:576) [spigot.jar:git-Spigot-1465]
    at net.minecraft.server.v1_7_R3.MinecraftServer.run(MinecraftServer.java:482) [spigot.jar:git-Spigot-1465]
    at net.minecraft.server.v1_7_R3.ThreadServerApplication.run(SourceFile:628) [spigot.jar:git-Spigot-1465]

[mibby]

@Hidendra

[TNTUP]

Any updates on this? 1.7.10 is out, and I guess 1.8 is coming soon, no date yet hopefully. This is the most last plugin waiting for the UUID update

[AiyionPrime]

Heyho?
As @TNTUP mentioned, time is running out :/
I dont like the idea of changing this whole database to another plugin -.-'
Please @Hidendra tell us, your'e doing well :S

[MrMaleficus]

@Hidendra some news ? :(

[TNTUP]

@MrMaleficus Latest dev build supports UUID, but current older databases aren't converted (or I should use the LWC-ConverterUUID then go back to LWC as normal. Newer protections registers UUID now.

It works on my end so it should work on your too. If anyone knows a command that will convert existent old database to UUID names, that will be appriciated!

[MrMaleficus]

The UUID system is working, yes.

But, the /climits ignore the protections for the UUID system...

[TNTUP]

@MrMaleficus /climits you mean like limited X protectable containers per player right? If so, oh.. well myself I have no limits because I don't see why but for preventing chest-lock abuse etc yeah