'Write' command to update tags hangs on NAS drive

[Dosman]

Hi

I have an beets database where I imported my music library (236 GB of flacs, 700 albums), which is contained externally on a NAS drive. When I now do a 'beet -v write' command to go in and fill in all the metadata the write command starts for a while but eventually hangs (have found it stops for up to 8 hours before I kill the process).

The verbose output shows the tag updates, then "Sending event: write Sending event: after_write Sending event: database_change". After a while of this though it gets to a "Sending event: write" and then stops.

I don't know how to work out whether this is an issue with beets or with the NAS drive. I've looked at the permissions of the files that worked and those that didn't, but don't see anything different and in this case the problem appears to occur half way through an album so it shouldn't be a permissions issue.

Do you have any advice for checks I can carry out, or other logfiles to check?
Thanks!

[sampsyo]

Thanks for getting in touch. Network filesystems can indeed be quite flakey, depending the driver and the NAS, so it's not too shocking that writes could hang occasionally.

Perhaps you can try narrowing it down to specific files? And if it really is random, maybe you can reproduce it with a simple shell script that just tries writing files until it hangs, so you have something you can report to your NAS vendor (or OS maintainer)?

[Dosman]

Thanks for the quick feedback Adrian. I can give it a go. Was mostly
checking to see if there was any other diagnostic output in beets that I
could try. Will see how I get on.
Steve

On 24 January 2015 at 21:06, Adrian Sampson notifications@github.com
wrote:

Thanks for getting in touch. Network filesystems can indeed be quite
flakey, depending the driver and the NAS, so it's not too shocking that
writes could hang occasionally.

Perhaps you can try narrowing it down to specific files? And if it really
is random, maybe you can reproduce it with a simple shell script that just
tries writing files until it hangs, so you have something you can report to
your NAS vendor (or OS maintainer)?

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[sampsyo]

Yeah, I wish there were more detail to offer here—the verbose mode you've been using is as detailed as we get currently. If you can fight your way through some Python, I recommend trying to add some print-logging in the write command function in beets.ui.commands—could help at least narrow down which file is at fault (if any).

[Dosman]

Ok . Will take a look.

btw: Thanks for putting the whole beets thing together. No one else was...

On 24 January 2015 at 21:24, Adrian Sampson notifications@github.com
wrote:

Yeah, I wish there were more detail to offer here—the verbose mode you've
been using is as detailed as we get currently. If you can fight your way
through some Python, I recommend trying to add some print-logging in the
write command function in beets.ui.commands—could help at least narrow
down which file is at fault.

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[Dosman]

Hiya

Am still trying to work out the 'write' issue. I've not delved into python
yet, but had a question about some of the odd behaviour:

When carrying out a 'write' it acts oddly in two ways:

Odd Behaviour #1 - Even though the metadata has already written out to the
music files, the write command tries to update the bitrate field on many of
the files; however although beets lists "Sending event: after_write" and
"Sending event: database_change", the bitrate change doesn't take. All the
other metadata gets updated but the bitrate field stays at zero kb/s
(confirmed via puddletag). This lack of update for bitrate appears to
affect all files that successfully 'write'.

Odd Behaviour #2 - When the 'write' command hangs, it always seems to do so
on the same files. If I carry out a specific 'write' for a hanging file
enough times it will eventually complete going to the after_write
statements. We then return to the situation at #1 above where some of the
metadata doesn't update, and eventually we hang again on the same file.

Have you had a situation before where only some of the metadata updates on
a 'write'?

Example hang:
Antônio Adolfo - Bar Rio: Classic & New Brazilian Flavours - Bola de vez
bitrate: 800kbps -> 800kbps
Sending event: write

Thanks for all the advice!

On 24 January 2015 at 23:51, Dosman admin@dosman.co.uk wrote:

Ok . Will take a look.

btw: Thanks for putting the whole beets thing together. No one else
was...

On 24 January 2015 at 21:24, Adrian Sampson notifications@github.com
wrote:

Yeah, I wish there were more detail to offer here—the verbose mode you've
been using is as detailed as we get currently. If you can fight your way
through some Python, I recommend trying to add some print-logging in the
write command function in beets.ui.commands—could help at least narrow
down which file is at fault.

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[sampsyo]

the bitrate change doesn't take

The bitrate field cannot change: it is not a tag, but metadata about the audio file. Some formats don't even have bitrate metadata, so beets calculates it based on the file size. If we're showing a modification for fields like that, then that's a UI bug.

When the 'write' command hangs, it always seems to do so
on the same files.

Can you supply samples so we can reproduce?

[Dosman]

Can do; however in the mean time I took some of the offending files off of
the NAS and onto the local hard drive, imported them into the beets db, and
then did a couple of write commands. No hang, and no issue with beets
attempting to write a new bitrate - just a clean import and clean write.

It looks like it's still a NAS issue, so it may not help to provide you
with samples. :/

On 26 January 2015 at 22:25, Adrian Sampson notifications@github.com
wrote:

the bitrate change doesn't take

The bitrate field cannot change: it is not a tag, but metadata about the
audio file. Some formats don't even have bitrate metadata, so beets
calculates it based on the file size. If we're showing a modification for
fields like that, then that's a UI bug.

When the 'write' command hangs, it always seems to do so
on the same files.

Can you supply samples so we can reproduce?

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[sampsyo]

OK, let's chalk this up to a network filesystem bug for now. Good luck!

[Dosman]

As a work around then; is it possible to form a beets query which will
ignore the bitrate field? That might allow my writes to complete at least.

On 27 January 2015 at 09:30, Adrian Sampson notifications@github.com
wrote:

Closed #1268 #1268.

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[sampsyo]

is it possible to form a beets query which will
ignore the bitrate field?

Can you explain what you mean by this? I don't know how a query would "ignore the bitrate field". Are you saying that the write command tries to write files even when the only "changed" field is bitrate? If so, that is definitely a bug—can you show example output where that happens?

[Dosman]

Hi Adrian

Are you saying that the write command tries to write files even when the
only "changed" field is bitrate?

Yep - when trying a global 'write' on the NAS beets tries to update the
bitrate field on ALL files (see attached example terminal output). I was
just wondering if there was a way to tell the database not to check the
bitrate field when it checks whether files need updating. No worries if
not, but I thought it might be a workaround.

As mentioned previously the bitrate updating doesn't occur when the files
are imported on a local drive and then a write command is carried out. on
that basis what I've decided to do is copy my 250GB library over to a USB
drive and then try the write on the USB drive version, as it might be
something to do with cifs/samba mounting of the NAS. That said I'm happy
to help out with bug-checking if you do think it is a bug though.
Steve

~$ beet -v write
user configuration: /home/steve/.config/beets/config.yaml
data directory: /home/steve/.config/beets
Sending event: pluginload
library database: /home/steve/Music/musiclibrary.blb
library directory: /mnt/raid/music/_reference
Sending event: library_opened
040 - The Best Ibiza Anthems... Ever! 2K - Ibiza Dreams
  bitrate: 856kbps -> 856kbps
Sending event: write
Sending event: after_write
Sending event: database_change
1 Giant Leap Feat. Robbie Williams & Maxi Jazz - The Karma Collection Sunrise - My Culture (Album Version)
  bitrate: 860kbps -> 860kbps
Sending event: write
Sending event: after_write
Sending event: database_change
2000 BC - The Best Ibiza Anthems... Ever! 2K - Everybody (club mix)
  bitrate: 880kbps -> 880kbps
Sending event: write
Sending event: after_write
Sending event: database_change
A Man Called Adam - GQ Ibiza Chillout - Yachts
  bitrate: 770kbps -> 770kbps
Sending event: write
Sending event: after_write
Sending event: database_change
A Man Called Adam - Ministry Presents: Ibiza Chillout - Easter Song
  bitrate: 950kbps -> 950kbps
Sending event: write
Sending event: after_write
Sending event: database_change
A Man Called Adam - The Karma Collection 2003 - Estelle
  bitrate: 915kbps -> 915kbps
Sending event: write
Sending event: after_write
Sending event: database_change
A:xus - The Karma Collection 2003 - Baghdad Cafe (Callin' U)
  bitrate: 952kbps -> 952kbps
Sending event: write
Sending event: after_write
Sending event: database_change
ABC - Q: Decade the Very Best of 1986-1996 - When Smokey Sings
  bitrate: 833kbps -> 833kbps
Sending event: write
Sending event: after_write
Sending event: database_change
AMbassador - Essential Millennium - Fade (Fade mix)
  bitrate: 901kbps -> 901kbps
Sending event: write
Sending event: after_write
Sending event: database_change
ATB - Ministry Presents: Class of '99: The Ministry Award Winners - 9 PM (Till I Come) (Sequential One 1999 remix)
  bitrate: 825kbps -> 825kbps
Sending event: write
Sending event: after_write
Sending event: database_change
ATB - Ministry of Sound: Dance Nation Six - 9 PM (Till I Come) (Sequential One 1999 remix)
  bitrate: 771kbps -> 771kbps
Sending event: write
Sending event: after_write
Sending event: database_change
ATFC Presents OnePhatDeeva Feat. Lisa Millett - The Annual 2000 - Bad Habit (ATFC Club Mix)
  bitrate: 1012kbps -> 1012kbps
Sending event: write
Sending event: after_write
Sending event: database_change
The Academy Is... - Festival 07 - We've Got a Big Mess on Our Hands
  bitrate: 1109kbps -> 1109kbps
Sending event: write
Sending event: after_write
Sending event: database_change
Ryan Adams - Q: Best of 2001 - The Rescue Blues
  bitrate: 909kbps -> 909kbps
Sending event: write
Sending event: after_write
Sending event: database_change
Adani & Wolf Feat. Ernest Ranglin - The Karma Collection Sunrise - Daylight
  bitrate: 919kbps -> 919kbps
Sending event: write

[sampsyo]

Thank you for clarifying! I just pushed a fix for write that does not try to write non-writable fields like bitrate.

[Dosman]

Cool! Have updated beets to the nightly and have started a global write.
It's looking promising as the 'write' has started at a point just after the
process has been hanging in the past. Will let you know how I get on.

Thanks Adrian!

~$ beet -v write
user configuration: /home/steve/.config/beets/config.yaml
data directory: /home/steve/.config/beets
Sending event: pluginload
library database: /home/steve/Music/musiclibrary.blb
library directory: /mnt/raid/music/_reference
Sending event: library_opened
Arcade Fire - The Suburbs - Month of May
original_day: 00 -> 03
Sending event: write
Sending event: after_write
Sending event: database_change
Arcade Fire - The Suburbs - Wasted Hours
original_day: 00 -> 03
Sending event: write
Sending event: after_write
Sending event: database_change
Arcade Fire - The Suburbs - Deep Blue
original_day: 00 -> 03
Sending event: write

On 27 January 2015 at 23:04, Adrian Sampson notifications@github.com
wrote:

Thank you for clarifying! I just pushed a fix for write that does not try
to write non-writable fields like bitrate.

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[Dosman]

Ok - the new beets version has definitely fixed the problem where it
attempted to perform a 'write' on every file when on a mounted NAS. It
still hangs as it goes, but at least this time when repeated 'writes'
succeed it can move on to further files rather than back to square one. I
suspect the hanging 'writes' will be down to the NAS mounting, so I assume
you can close this issue.

Many thanks for your help.

On 28 January 2015 at 07:45, Dosman admin@dosman.co.uk wrote:

Cool! Have updated beets to the nightly and have started a global write.
It's looking promising as the 'write' has started at a point just after the
process has been hanging in the past. Will let you know how I get on.

Thanks Adrian!

~$ beet -v write
user configuration: /home/steve/.config/beets/config.yaml
data directory: /home/steve/.config/beets
Sending event: pluginload
library database: /home/steve/Music/musiclibrary.blb
library directory: /mnt/raid/music/_reference
Sending event: library_opened
Arcade Fire - The Suburbs - Month of May
original_day: 00 -> 03
Sending event: write
Sending event: after_write
Sending event: database_change
Arcade Fire - The Suburbs - Wasted Hours
original_day: 00 -> 03
Sending event: write
Sending event: after_write
Sending event: database_change
Arcade Fire - The Suburbs - Deep Blue
original_day: 00 -> 03
Sending event: write

On 27 January 2015 at 23:04, Adrian Sampson notifications@github.com
wrote:

Thank you for clarifying! I just pushed a fix for write that does not
try to write non-writable fields like bitrate.

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[tlc]

With 1.3.10, I can't write MP3s on my NAS, but I can write FLAC.
I can write MP3s locally.

country: -> US
script: -> Latn
label: -> Def Jam Recordings
albumtype: -> album
Sending event: write

(Hang, file untouched)

[Dosman]

Also on NAS files I've been getting a lot of original_day tag writes that
get applied, but then get reapplied on a subsequent write. Things like the
following keep happening even though the music files have the 26th as the
date already:

Los Campesinos! - Romance Is Boring - I Warned You: Do Not Make an Enemy of
Me
original_day: 00 -> 26
Sending event: write
Sending event: after_write
Sending event: database_change
Los Campesinos! - Romance Is Boring - Heart Swells/100-1
original_day: 00 -> 26
Sending event: write
Sending event: after_write
Sending event: database_change
Los Campesinos! - Romance Is Boring - I Just Sighed. I Just Sighed, Just So
You Know
original_day: 00 -> 26
Sending event: write
Sending event: after_write
Sending event: database_change

On 4 February 2015 at 05:59, tlc notifications@github.com wrote:

With 1.3.10, I can't write MP3s on my NAS, but I can write FLAC.
I can write MP3s locally.

country: -> US
script: -> Latn
label: -> Def Jam Recordings
albumtype: -> album
Sending event: write

(Hang, file untouched)

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[tlc]

They're chasing the original_day issue in #1303.

Do you still have hangs? Did they correspond to MP3s, or all you all FLAC?

[Dosman]

Still getting hangs on particular files, but after a few attempts it will
generally get past a particular file. (Slow going!) Pretty sure it was
FLACs that are experiencing the problem (I'm ~95% flac with a few mp3s,
m4a, and aiffs hanging about). Will check tonight and confirm.

On 4 February 2015 at 12:55, tlc notifications@github.com wrote:

They're chasing the original_day issue in #1303
#1303.

Do you still have hangs? Did they correspond to MP3s, or all you all FLAC?

—
Reply to this email directly or view it on GitHub
#1268 (comment).

[Dosman]

Yep it is hanging on flacs. Haven't tried mp3s, but will do so.

On 4 February 2015 at 13:01, Dosman admin@dosman.co.uk wrote:

Still getting hangs on particular files, but after a few attempts it will
generally get past a particular file. (Slow going!) Pretty sure it was
FLACs that are experiencing the problem (I'm ~95% flac with a few mp3s,
m4a, and aiffs hanging about). Will check tonight and confirm.

On 4 February 2015 at 12:55, tlc notifications@github.com wrote:

They're chasing the original_day issue in #1303
#1303.

Do you still have hangs? Did they correspond to MP3s, or all you all FLAC?

—
Reply to this email directly or view it on GitHub
#1268 (comment).