Turbinia not compatible with google-cloud-pubsub==0.29.0

[berggren]

pubsub.Client is now PublisherClient and SubscriberClient

[aarontp]

Took a bit of digging to figure out exactly what is going on, but here is the redesign:
googleapis/google-cloud-python#3859

The underlying code still supports it, but it sounds like they're trying to push everyone to asynchronous based polling. In the short term I'm just going to use the underlying .pull() method, and we can convert later if this thread indicates that it might not be supported in the long term.
googleapis/google-cloud-python#4338

[berggren]

SGTM

…

On Wed, Nov 22, 2017 at 8:18 PM Aaron Peterson ***@***.***> wrote: Took a bit of digging to figure out exactly what is going on, but here is the redesign: googleapis/google-cloud-python#3859 <googleapis/google-cloud-python#3859> The underlying code still supports it, but it sounds like they're trying to push everyone to asynchronous based polling. In the short term I'm just going to use the underlying .pull() method, and we can convert later if this thread indicates that it might not be supported in the long term. googleapis/google-cloud-python#4338 <googleapis/google-cloud-python#4338> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATTysLnZ-qOUJqawEc9fyhTxFsbFE5uks5s5HORgaJpZM4QXLKH> .

[aarontp]

Ok, so a few things:

I tried to just use the underlying implementation for .pull() by using the SubscriberClient directly from the gapic client in google.cloud.pubsub_v1.gapic.subscriber_client, but I couldn't get it to work correctly (it kept indefinitely hanging when waiting for messages rather than respecting the return_immediately flag). I ended up just using the asynchronous interface and just sticking things on a queue for processing as needed.

I have initial support for asynchronous pulls with the latest library version in the pubsub-fix branch here:
cd0cd14

Only problem is that it looks like PSQ doesn't actually support it yet either, :(. I filed this issue, but we might need to add that support ourselves:
GoogleCloudPlatform/psq#32

[aarontp]

We should be good to move forward to upgrading this. PSQ has updated to use the latest version as of psq==0.6.0. Hopefully no major refactors in the PSQ interface to go along with it, :)

[berggren]

Woop Woop!

…

On Thu, Dec 7, 2017, 22:03 Aaron Peterson ***@***.***> wrote: We should be good to move forward to upgrading this. PSQ has updated to use the latest version as of psq==0.6.0. Hopefully no major refactors in the PSQ interface to go along with it, :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AATTymAsByI32tVyhSUvKnJgqEnYcP8Vks5s-FKJgaJpZM4QXLKH> .

[aarontp]

Note to self: Unpin google-cloud-storage when this is fixed as well.

[beamcodeup]

@aarontp FYI

Per grpc/grpc#14892, gRPC doesn't support fork. Currently, we pin psq=0.5.0. The problem is this version fork() when psq.MultiprocessWorker(queue=...) is invoked and then segfaults.

I think we should explore upgrading psq to the latest version.

[aarontp]

@beamcodeup: Ok, thanks. It looks PSQ dropped the MultiprocessWorker altogether in GoogleCloudPlatform/psq#34, so specifying -S to the worker should be equivalent in this particular case. We do need to upgrade anyway though. I have a PR that is drafted, but completely untested, so I'll up the priority of finishing that.

…

On Mon, Apr 9, 2018 at 12:03 PM beamcodeup ***@***.***> wrote: @aarontp <https://github.com/aarontp> FYI Per grpc/grpc#14892 <grpc/grpc#14892>, gRPC doesn't support fork. Currently, we pin psq=0.5.0. The problem is this version fork() when psq.MultiprocessWorker(queue=...) is invoked and then segfaults. I think we should explore upgrading psq to the latest version. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACiyu-uxy9rNhrIVIbb9Pe826pmw8I3Cks5tm7BxgaJpZM4QXLKH> .

[aarontp]

Fixed in #198