Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Arm64 vs aarch64 in pull request 123 #124

Closed
gebau00a opened this issue Dec 23, 2022 · 27 comments
Closed

Arm64 vs aarch64 in pull request 123 #124

gebau00a opened this issue Dec 23, 2022 · 27 comments

Comments

@gebau00a
Copy link

Pull request 123 added an arch check for videophash.py
On arm64, it expects the arch type arm64, but distributions know arm64 as aarch64.
Running namer directly results in a Unsupport architecture error aarch64

Adding the dict link 'aarch64': 'arm64' runs, but videohashes-arm64-linux fails to produce output, thus an error is raised

Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
    self._bootstrap_inner()
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
> File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 137, in __processing_thread
    handle(command)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 54, in handle
    process_file(command)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/namer.py", line 176, in process_file
    phash = VideoPerceptualHash().get_stash_phash(command.target_movie_file) if command.config.search_phash else None
  File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 91, in get_stash_phash
   return self._get_stash_phash(file, stat.st_size, stat.st_mtime)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 96, in _get_stash_phash
    return self.__execute_stash_phash(file)
  File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 134, in __execute_stash_phash
    data = json.loads(stdout, object_hook=lambda d: SimpleNamespace(**d))
  File "/usr/lib/python3.11/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
 File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
   raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Running videohashes-arm64-linux manually with a link to a video file generates the error
ffmpeg/ffprobe executables not found

Scanning through the compiled code for videohashes-arm64-linux shows several matches for arm64, but none for aarch64.
I assume it's looking for an arm64 ffmpeg, and cannot find an aarch64 ffmpeg.

Not sure if this is an upstream error from https://github.com/peolic/videohashes or where the videohashes executable comes from
@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

Video hashes come from that repo, that they run locally is great (you do need to install ffmpeg though for it function). Adding aarch64 to this map: https://github.com/ThePornDatabase/namer/blob/main/namer/videophash.py#L38-L42. Adding it now.

@gebau00a
Copy link
Author

I have ffmpeg installed, namer was working flawlessly before the videophash function was updated.
Nor sure why it cannot find ffmpeg.

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022 •
edited
Loading

try latest release (just releaseed), should allow namer to call videohashes-arm64-linux. Verify ffmpeg is on the path, as videohashes-arm64-linux calls ffmpeg as well.

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

Just verified, nothing special in the ffmpeg/fprobe look up, just need to be on path: https://github.com/stashapp/stash/blob/develop/pkg/ffmpeg/downloader.go#L21-L39

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

@gebau00a please try with latest release and update here, I suspect it should work. Tempted to close this, but don't want to be perceived as rude, so please let me know what happens with the latest namer and ffmpeg/ffprobe verified on the path.

@gebau00a
Copy link
Author

Journal log, namer run as systems service:

Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | ffmpeg version 4.4.3 found
Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | ffprobe version 4.4.3 found
Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | Start porndb scene watcher.... watching: /mnt/downloads/toTag/newFiles
Dec 24 00:55:08 odroid python[113640]: 2022-12-24 00:55:08 | ℹ️ INFO     | Starting server: http://0.0.0.0:6980/
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | watchdog process called for xxx
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | Moving /mnt/downloads/toTag/newFiles/xxx to /mnt/downloads/toTag/tag_work/xxx for processing
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | Processing: xxx
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ℹ️ INFO     | Calculating phash for file "xxx"
Dec 24 00:55:25 odroid python[113640]: 2022-12-24 00:55:25 | ❌ ERROR    | An error has been caught in function '__processing_thread', process 'MainProcess' (113640), thread 'Thread-2 (__processing_thread)' (281472612823456):
Dec 24 00:55:25 odroid python[113640]: Traceback (most recent call last):
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/threading.py", line 995, in _bootstrap
Dec 24 00:55:25 odroid python[113640]:     self._bootstrap_inner()
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
Dec 24 00:55:25 odroid python[113640]:     self.run()
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/threading.py", line 975, in run
Dec 24 00:55:25 odroid python[113640]:     self._target(*self._args, **self._kwargs)
Dec 24 00:55:25 odroid python[113640]: > File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 137, in __processing_thread
Dec 24 00:55:25 odroid python[113640]:     handle(command)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/watchdog.py", line 54, in handle
Dec 24 00:55:25 odroid python[113640]:     process_file(command)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/namer.py", line 176, in process_file
Dec 24 00:55:25 odroid python[113640]:     phash = VideoPerceptualHash().get_stash_phash(command.target_movie_file) if command.config.search_phash else None
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 91, in get_stash_phash
Dec 24 00:55:25 odroid python[113640]:     return self._get_stash_phash(file, stat.st_size, stat.st_mtime)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 96, in _get_stash_phash
Dec 24 00:55:25 odroid python[113640]:     return self.__execute_stash_phash(file)
Dec 24 00:55:25 odroid python[113640]:   File "/home/andy/.local/lib/python3.11/site-packages/namer/videophash.py", line 134, in __execute_stash_phash
Dec 24 00:55:25 odroid python[113640]:     data = json.loads(stdout, object_hook=lambda d: SimpleNamespace(**d))
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/json/__init__.py", line 359, in loads
Dec 24 00:55:25 odroid python[113640]:     return cls(**kw).decode(s)
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
Dec 24 00:55:25 odroid python[113640]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Dec 24 00:55:25 odroid python[113640]:   File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
Dec 24 00:55:25 odroid python[113640]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Dec 24 00:55:25 odroid python[113640]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

@gebau00a
Copy link
Author

I think the problem is not in Namer itself, but in the videohash executable

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

Is xxx a real video file?

@gebau00a
Copy link
Author

gebau00a commented Dec 23, 2022 •
edited
Loading

Yes it is. I only replaced the name as the name has no influence on the bug.
None of the files is working

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

well crap. You can turn off phashes in the config as a work around. see the https://github.com/ThePornDatabase/namer/blob/main/namer/namer.cfg.default for the layout.

I have no way to debug this, so unless you have some spare time and or know a bit of go the above might be your only option . Hopefully someone with the hardware has time to look at the arm64 video hasher.

@DirtyRacer1337
Copy link
Contributor

Just need to print output and see that it says, errors not in json format even with -json flag

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

According to gebau00a:

Running videohashes-arm64-linux manually with a link to a video file generates the error
ffmpeg/ffprobe executables not found

which are on the path according to namer's log

Improving error handling is a good idea, but it'll lead us back here.

@DirtyRacer1337
Copy link
Contributor

Added error printing, #125

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

Releasing that as 1.10.5, maybe it'll provide more info.

@DirtyRacer1337
Copy link
Contributor

Well if videohashes can't find ffmpeg it's problem with stash code, https://github.com/stashapp/stash

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022

Yeah only down side is that seemed solid to me: https://github.com/stashapp/stash/blob/develop/pkg/ffmpeg/downloader.go#L21-L39

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 23, 2022 •
edited
Loading

@gebau00a give 1.10.5 a shot when you have a chance -- we can verify the error at least

@gebau00a
Copy link
Author

Yeah, cause is the videohash executable

Dec 24 08:36:19 odroid python[121854]: 2022-12-24 08:36:19 | ℹ️ INFO     | ffmpeg version 4.4.3 found
Dec 24 08:36:19 odroid python[121854]: 2022-12-24 08:36:19 | ℹ️ INFO     | ffprobe version 4.4.3 found
Dec 24 08:36:19 odroid python[121854]: 2022-12-24 08:36:19 | ℹ️ INFO     | Start porndb scene watcher.... watching: /mnt/downloads/toTag/newFiles
Dec 24 08:36:20 odroid python[121854]: 2022-12-24 08:36:20 | ℹ️ INFO     | Starting server: http://0.0.0.0:6980/
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | watchdog process called for xxx.mp4
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Moving /mnt/downloads/toTag/newFiles/xxx.mp4 to /mnt/downloads/toTag/tag_work/xxx.mp4 for processing
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Processing: /mnt/downloads/toTag/tag_work/xxx.mp4
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Calculating phash for file "/mnt/downloads/toTag/tag_work/xxx.mp4"
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ❌ ERROR    | ffmpeg/ffprobe executables not found
Dec 24 08:36:31 odroid python[121854]: 2022-12-24 08:36:31 | ℹ️ INFO     | Requesting GET "https://api.metadataapi.net/scenes?parse=xxx&limit=25"
Dec 24 08:36:32 odroid python[121854]: 2022-12-24 08:36:32 | ℹ️ INFO     | Requesting GET "https://api.metadataapi.net/scenes?parse=xxx&limit=25"

@4c0d3r 4c0d3r mentioned this issue Dec 24, 2022
Open
@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 24, 2022

Opened issue: peolic/videohashes#5 to track this.

@gebau00a
Copy link
Author

gebau00a commented Dec 24, 2022 •
edited
Loading

I put up some comments in the bug report in videohashes
Seems like the videohashes executable is only searching in the active working dir for the executable, which is ok for Windows but not standard practice for Unix systems.
I posted also a snippet how to correct this, using a different function.
Let's hope we see a result soon.

With your correction on the other hand to error handle the phash function, Namer is running already like before, despite
now only doing name checking, but good enough until a fix is up by upstream.
Thanks for thisA

A symlink of ffmpeg and ffprobe in the tools subdirectory works when running the videohashes program directly, but when namer is started as service, it's not finding them still, as videohashes is searching in the active directory during start of the service.

@gebau00a
Copy link
Author

I found a temporary solution for my problem, shifting the working dir of the namer service to /usr/bin

[Unit]
Description=Namer service
Wants=network-online.target
After=network-online.target nss-lookup.target

[Service]
WorkingDirectory=/usr/bin
Type=exec
Restart=always
RestartSec=1
User=andy
UMask=0000
ExecStart=python -m namer watchdog
#StandardOutput=null

[Install]
WantedBy=multi-user.target

For those interested in the watchdog service file.
The magic here is the WorkingDirectory shift, which is done before namer starts, so all current directory requests from Videohashes are pointing to /usr/bin/ before searching for ffmpeg and ffprobe executables.
It's working for now but I think in the end it's still an upstream bug as it will only work for watchdog now.

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 24, 2022

OMG.... you really trust this code.... I would copy ffmpeg to the work dir......

@gebau00a
Copy link
Author

This is my service script I wrote myself, so this one I trust.
Linking in the tools subdirectory only works when executing the file itself, else it runs with the dir from where namer was called, in / by default.
Better to let namer run in /usr/bin where the user has no write access, else I have to most likely create symlinks for ffmpeg in /, which I see as way worse.

@gebau00a
Copy link
Author

Poelic found the issue.
When ffmpeg is in the normal PATH, the configuration options are checked upfront.
webp option was missing from my ffmpeg.
Compiling it fresh with webp solved the issue.

I only suggested to change the error message as it's not consistent with what happens.

But I propose to close this bug as it's settled and only an upstream improvement left.

Thanks for all your help.

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 24, 2022

Perhaps Namer should follow stash's led here and download ffmpeg:

https://github.com/stashapp/stash/blob/develop/pkg/ffmpeg/downloader.go#L170-L190

@4c0d3r
Copy link
Collaborator

4c0d3r commented Dec 28, 2022

videohasher will get the ability to download ffmpeg: peolic/videohashes#6 which namer will leverage once it's merged.

@4c0d3r
Copy link
Collaborator

4c0d3r commented Jan 2, 2023

Fixed in namer in #130, now ffmpeg can be download as well. Though the running user needs to have permission to write to the pip install dir.

@4c0d3r 4c0d3r closed this as completed Jan 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@DirtyRacer1337 @gebau00a @4c0d3r