High load on Emby Server during sync with Ombi

[Jdiesel87]

Ombi build Version:

V 3.0.3295

Update Branch:

Open Beta

Media Sever:

Emby

Media Server Version:

3.4.0.4

Operating System:

Ubuntu 16.04

Ombi Applicable Logs (from /logs/ directory or the Admin page):

(Logs go here. Don't remove the ' tags for showing your logs correctly. Please make sure you remove any personal information from the logs)

Problem Description:

Ombi hammers Emby with requests during the scheduled sync causing Emby to become unresponsive at time. Can something be done to reduce the load on Emby durning the sync? @LukePulverenti mentioned he is willing to help with solution to the current way of syncing with Emby.

Reproduction Steps:

Navigate Emby webui when a sync is in progress. Webui is slow and sometimes unresponsive. View Emby log file to see request from Ombi.

[tidusjar]

Ok interesting.

The way we sync with Emby hasn't changed since I did it really. I'll update this with the steps we do later and maybe Luke can let me know if there is a better way to interact with Emby

[LukePulverenti]

What I would suggest is getting lists of items from folders rather than pulling down items individually. Not only will this reduce the number of round trips, but the list endpoints return a smaller set of information by default, and you have to opt-in to additional information by specifying the fields that you want. So not only can you reduce the number of requests but you can also reduce the amount of data that has to be pulled.

Second, and I don't know too much about the Ombi syncing process, but if at all possible I would let users configure when the sync takes place so that they can set it up for off hours, like in the middle of the night for example.

[tidusjar]

Hey @LukePulverenti Thanks for the input.

Do you have any further information on the actual API's you are referring to?

Ombi allows the user to customize the sync.

[LukePulverenti]

Well how are you getting all of the item Id's in the first place? I would imagine you're getting them from querying folders, right?

[tidusjar]

Sorry for the late reply, this is the process @LukePulverenti :

• Get all the movies
  • emby/users/{userId}/items?Recursive=true&IncludeItemTypes=Movie
• Foreach movie check if the type is a boxset
  • If it is then Get the collection: emby/users/{userId}/items?parentId={mediaId}
  • foreach of the items returneds above get the Movie Information: emby/users/{userId}/items/{mediaId}

Then

• Get all shows
  • emby/users/{userId}/items?Recursive=true&IncludeItemTypes=Series
• Get all the series information from the above emby/users/{userId}/items/{mediaId}

Once those processes are complete we then move onto episodes

• Get all episodes
  • emby/users/{userId}/items?Recursive=true&IncludeItemTypes=Episode
  • If it's LocationType is Virtual then skip it
• Get the ep info from the above: emby/users/{userId}/items/{mediaId}

If there are any improvements that you suggest i'd be happy to make changes

[anojht]

ping @LukePulverenti

[LukePulverenti]

Ok let's start with movies. Your initial query is fine:
emby/users/{userId}/items?Recursive=true&IncludeItemTypes=Movie

If it is then Get the collection: emby/users/{userId}/items?parentId={mediaId}

This will never happen. Instead, to get a user's boxsets, you can do this:
emby/users/{userId}/items?Recursive=true&IncludeItemTypes=BoxSet

foreach of the items returneds above get the Movie Information: emby/users/{userId}/items/{mediaId}

This is the part that you don't need, because you already have it when you query for the list of movies. You can use the Fields param to control what fields you get back.

Can you tell me what movie information you're using, and then I'll give you a sample query. Thanks.

[LukePulverenti]

As far as the list of shows, same comment & question as above. Thanks !

[LukePulverenti]

Episodes will also be the same, however you can avoid having to post-filter client-side by doing this:
emby/users/{userId}/items?Recursive=true&IncludeItemTypes=Episode&IsVirtualItem=false

[tidusjar]

Thanks for that information!

So the information I am after for Movies is:
ProviderIds (ImdbId, Tmdb),
That's why I call the movie individually

Same for TV and episodes but Tvdb aswell as the others.

[LukePulverenti]

So then for all of those queries just add &Fields=ProviderIds. If you need the Overview then it would be &Fields=ProviderIds,Overview

You can then get everything from those list queries without having to pull down items individually.

[tidusjar]

Awesome! That should improve things!

I'll get this done asap. Thanks for the help.

[tidusjar]

Improvements made in the develop branch