Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'Interop+Sys+SocketEvent*' and 'System.Object'. #5298

Closed
Swoorup opened this issue Oct 6, 2021 · 5 comments

Comments

@Swoorup
Copy link

Swoorup commented Oct 6, 2021

Version Information
Version of Akka.NET? 1.4.26
Which Akka.NET Modules?

  • Akka
  • Akka.Cluster
  • Akka.Serialization.Hyperion

Describe the bug
Same issue as in #4936

Hocon

 akka : {
    loglevel : DEBUG
    loggers : ["Akka.Logger.Serilog.SerilogLogger, Akka.Logger.Serilog"]
    actor : {
      ask-timeout : 4.5s
      provider : cluster
      deployment : {
        /projectionStatus/broadcaster : {
          router : broadcast-group
          routees : {
            paths : [/user/projectionStatus]
          }
          cluster : {
            enabled : on
            allow-local-routees : on
            use-role :
          }
        }
        /worker-pool : {
          router : consistent-hashing-pool
          nr-of-instances : 1
          virtual-nodes-factor : 1
          cluster : {
            enabled : on
            allow-local-routees : on
            max-nr-of-instances-per-node : 1
          }
        }
      }
    }
    remote : {
      enabled-transports : [akka.remote.dot-netty.tcp]
      dot-netty : {
        tcp : {
          hostname : localhost
          port : 4054
          public-hostname : localhost
          public-port : 4054
        }
      }
    }
    cluster : {
      discovery : {
        provider : akka.cluster.discovery.cloudmap
        cloudmap : {
          REDACTED
        }
      }
      downing-provider-class : "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"
      active-strategy : keep-majority
      singleton : {
        singleton-name : work-manager
        role :
        hand-over-retry-interval : 1s
      }
      singleton-proxy : {
        singleton-name : work-manager
        role :
        singleton-identification-interval : 2s
        buffer-size : 100
      }
      seed-nodes : ["akka.tcp://test-service@localhost:4053","akka.tcp://test-service@localhost:4054"]
      roles :
    }
  }
Details
[10/06/2021 22:18:32 +11:00  ] Association with remote system akka.tcp://test-service@localhost:4053 has failed; address is now gated for 5000 ms. Reason is: [Akka.Remote.EndpointException: Failed to write message to the transport
 ---> Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'Interop+Sys+SocketEvent*' and 'System.Object'.
   at Hyperion.ValueSerializers.UnsupportedTypeSerializer.WriteManifest(Stream stream, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method599(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method603(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method615(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method629(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method725(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method799(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.SerializerFactories.EnumerableSerializerFactory.<>c__DisplayClass10_0.<BuildSerializer>b__1(Stream stream, Object o, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method803(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObjectWithManifest(Stream stream, Object value, SerializerSession session)
   at Hyperion.SerializerFactories.DelegateSerializerFactory.<>c__DisplayClass2_0.<BuildSerializer>b__1(Stream stream, Object value, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method434(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method434(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method434(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method739(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method753(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method793(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Serializer.Serialize(Object obj, Stream stream)
   at Akka.Serialization.HyperionSerializer.ToBinary(Object obj)
   at Akka.Remote.Serialization.DaemonMsgCreateSerializer.Serialize(Object obj)
   at Akka.Remote.Serialization.DaemonMsgCreateSerializer.PropsToProto(Props props)
   at Akka.Remote.Serialization.DaemonMsgCreateSerializer.ToBinary(Object obj)
   at Akka.Remote.MessageSerializer.Serialize(ExtendedActorSystem system, Address address, Object message)
   at Akka.Remote.EndpointWriter.WriteSend(Send send)
   --- End of inner exception stack trace ---
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level, Boolean needToThrow)
   at Akka.Remote.EndpointWriter.WriteSend(Send send)
   at Akka.Remote.EndpointWriter.<Writing>b__27_0(Send s)
   at lambda_method201(Closure , Object , Action`1 , Action`1 , Action`1 )
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass113_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)]
[10/06/2021 22:18:32 +11:00  ] Failed to write message to the transport
Akka.Remote.EndpointException: Failed to write message to the transport
 ---> Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'Interop+Sys+SocketEvent*' and 'System.Object'.
   at Hyperion.ValueSerializers.UnsupportedTypeSerializer.WriteManifest(Stream stream, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method599(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method603(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method615(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method629(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method725(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method799(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.SerializerFactories.EnumerableSerializerFactory.<>c__DisplayClass10_0.<BuildSerializer>b__1(Stream stream, Object o, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method803(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObjectWithManifest(Stream stream, Object value, SerializerSession session)
   at Hyperion.SerializerFactories.DelegateSerializerFactory.<>c__DisplayClass2_0.<BuildSerializer>b__1(Stream stream, Object value, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method434(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method434(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method434(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method739(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method753(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at lambda_method793(Closure , Stream , Object , SerializerSession )
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Extensions.StreamEx.WriteObject(Stream stream, Object value, Type valueType, ValueSerializer valueSerializer, Boolean preserveObjectReferences, SerializerSession session)
   at Hyperion.Serializer.Serialize(Object obj, Stream stream)
   at Akka.Serialization.HyperionSerializer.ToBinary(Object obj)
   at Akka.Remote.Serialization.DaemonMsgCreateSerializer.Serialize(Object obj)
   at Akka.Remote.Serialization.DaemonMsgCreateSerializer.PropsToProto(Props props)
   at Akka.Remote.Serialization.DaemonMsgCreateSerializer.ToBinary(Object obj)
   at Akka.Remote.MessageSerializer.Serialize(ExtendedActorSystem system, Address address, Object message)
   at Akka.Remote.EndpointWriter.WriteSend(Send send)
   --- End of inner exception stack trace ---
   at Akka.Remote.EndpointWriter.PublishAndThrow(Exception reason, LogLevel level, Boolean needToThrow)
   at Akka.Remote.EndpointWriter.WriteSend(Send send)
   at Akka.Remote.EndpointWriter.<Writing>b__27_0(Send s)
   at lambda_method201(Closure , Object , Action`1 , Action`1 , Action`1 )
   at Akka.Actor.ReceiveActor.<>c__DisplayClass11_0.<Become>b__0(Object m)
   at Akka.Actor.ActorCell.<>c__DisplayClass113_0.<Akka.Actor.IUntypedActorContext.Become>b__0(Object m)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[10/06/2021 22:18:34 +11:00  ] Shutdown finished, but flushing might not have been successful and some messages might have been dropped. Increase akka.remote.flush-wait-on-shutdown to a larger value to avoid this.
[10/06/2021 22:18:34 +11:00  ] Remoting shut down
[10/06/2021 22:18:34 +11:00  ] Remoting shut down.
[DEBUG][6/10/2021 11:18:34 am][Thread 0033][EventStream] Shutting down: StandardOutLogger started
[ERROR][6/10/2021 11:18:34 am][Thread 0065][akka.tcp://test-service@localhost:4053/remote/akka.tcp/test-service@localhost:4054/user/worker-pool/c2] swallowing exception during message send
Cause: Akka.Remote.RemoteTransportException: Attempted to send remote message but Remoting is not running.
   at Akka.Remote.Remoting.Send(Object message, IActorRef sender, RemoteActorRef recipient)
   at Akka.Remote.RemoteActorRef.SendSystemMessage(ISystemMessage message)
@Swoorup Swoorup changed the title Serialization issue when running locally Hyperion.ValueSerializers.UnsupportedTypeException: No coercion operator is defined between types 'Interop+Sys+SocketEvent*' and 'System.Object'. Oct 6, 2021
@Aaronontheweb Aaronontheweb added this to the 1.4.27 milestone Oct 6, 2021
@Aaronontheweb
Copy link
Member

Thanks for the detailed writeup @Swoorup - I'll ask @Arkatufus on our team to take a look at this.

Some questions though - the previous bug you alluded to was an error with Akka.IO and actor serialization being turned on. That's definitely fixed. In your case, it looks to me like you're trying to send a message over Akka.Remote that contains a socket inside of it? Am I interpreting that stack trace correctly?

@Swoorup
Copy link
Author

Swoorup commented Oct 6, 2021

@Aaronontheweb I don't believe I am sending any messages with socket or but purely F# immutable structures and during initial startup. This seems to occur whenever specifying seed nodes for instance A and B in a cluster setup:

  • Start A first
  • Delay start B after 20-30 seconds
  • Observe the error in B

@Aaronontheweb
Copy link
Member

@Swoorup looks like this operation fails when we're trying to remotely deploy an actor - do you know which actor it is that might be failing? We should probably update our error message here to make it clearer when the DaemonMsgCreate fails to serialize.

@Swoorup
Copy link
Author

Swoorup commented Oct 6, 2021

@Aaronontheweb
Atm only have one actor like follows:
In my startup once actor system is launched.

ActorStartup.fs

        let workerPoolRef = spawn system workerPoolActorName (Worker.create workFlow)
        // spawn the work manager
        spawn system workManagerActorName (WorkManager.create workRepository workerPoolRef) |> ignore

WorkManager.fs

namespace Domain.Location.Actor.Shared.Actor

open Akkling.ActorRefs
open Domain.Location.Actor.Shared
open Domain.Location.Importer
open Akkling

type WorkManagerMsg = StartWork of param: IWorkFlowCommand

[<RequireQualifiedAccess>]
module WorkManager =
  let name = "workManager"
  type State = Map<WorkId, IActorRef<WorkerMsgEnv>>
  
  let create (workRepository: IWorkRepository) (workerPool: IActorRef<WorkerMsgEnv>) =
    props (fun (mailbox: Actor<WorkManagerMsg>) ->
      let rec loop (state: State) =
        actor {
          let! msg = mailbox.Receive()

          match msg with
          | StartWork param ->
              let workHashKey = Core.getOrCreateWorkHashKey param
              let! workId = workRepository.Create(param)

              workerPool <! { WorkId = workId; HashKey = workHashKey; Msg = WorkerMsg.StartWork param }
              mailbox.Sender() <! workId // respond back with the workId

              let state = state.Add(workId, workerPool)
              return! loop state
        }

      loop Map.empty)

Worker.fs

namespace Domain.Location.Actor.Shared.Actor

open Akka.Routing
open Akkling
open Domain.Location.Actor.Shared
open Domain.Location.Actor.Shared.Core
open Domain.Location.Actor.Shared.Workflow
open Domain.Location.Importer

type WorkerMsg =
  | StartWork of param: IWorkFlowCommand
  | WorkCompleted

type WorkerMsgEnv =
  { Msg: WorkerMsg
    WorkId: WorkId
    HashKey: WorkHashKey }
  interface IConsistentHashable with
    member x.ConsistentHashKey = box x.HashKey

[<RequireQualifiedAccess>]
module Worker =
  type State =
    | ReadyForWork
    | Working of workId: WorkId

  let create (workflow: IWorkFlow) =
    let compute msgEnv (param: IWorkFlowCommand) =
      async {
        do! workflow.RunWork msgEnv.WorkId param
        return { msgEnv with Msg = WorkCompleted }
      }

    let props =
      props
        (fun (mailbox: Actor<WorkerMsgEnv>) ->
          let rec ready (state: State) =
            actor {
              let! msg = mailbox.Receive()

              match msg.Msg with
              | StartWork param ->
                let workId = msg.WorkId
                compute msg param |!> mailbox.Self
                return! busy (Working workId)
              | _ -> return! ready state
            }

          and busy (state: State) =
            actor {
              let! msg = mailbox.Receive()

              match msg.Msg with
              | StartWork _ ->
                mailbox.Stash()
                return! busy state
              | WorkCompleted ->
                mailbox.UnstashAll()
                return! ready ReadyForWork
            }

          ready ReadyForWork)

    { props with Router = Some(upcast FromConfig.Instance) }

@Aaronontheweb
Copy link
Member

Per our conversation in Gitter, I think we've resolved this - problem is caused by a remote deployment of an actor that takes a database connection as a dependency, which can't be serialized. Solution is to not remotely deploy an actor with these constructor requirements directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants