TFX taxi client utility run abnormally after serving started

[jinchihe]

After run TFX taxi pipeline, the tf-serving service is started, changed the service name to NodePort type, and then, try to run client script chicago_taxi_client.py to test the server. but hit the problem below (unquoted fields cannot have quotes/CRLFs inside).

Any work around solution? Thanks a lot!

$ python chicago_taxi_client.py   --num_examples 1  --examples_file data.csv  --schema_file schema.pbtxt   --server 9.30.213.64:30933 --model_name taxi-cab-classification-model-tfx-taxi-cab-classification-pipel
/Users/hejinchi/pipeline/taxi/lib/python2.7/site-packages/tensorflow_serving/apis/prediction_service_pb2.py:131: DeprecationWarning: beta_create_PredictionService_stub() method is deprecated. This method will be removed in near future versions of TF Serving. Please switch to GA gRPC API in prediction_service_pb2_grpc.
  'prediction_service_pb2_grpc.', DeprecationWarning)
Traceback (most recent call last):
  File "chicago_taxi_client.py", line 161, in <module>
    tf.app.run()
  File "/Users/hejinchi/pipeline/taxi/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "chicago_taxi_client.py", line 157, in main
    known_args.model_name)
  File "chicago_taxi_client.py", line 125, in _do_inference
    model_name=model_name)
  File "chicago_taxi_client.py", line 55, in _do_local_inference
    print(stub.Predict(request, _LOCAL_INFERENCE_TIMEOUT_SECONDS))
  File "/Users/hejinchi/pipeline/taxi/lib/python2.7/site-packages/grpc/_channel.py", line 547, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/hejinchi/pipeline/taxi/lib/python2.7/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "Unquoted fields cannot have quotes/CRLFs inside
	 [[Node: DecodeCSV = DecodeCSV[OUT_TYPE=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_STRING, DT_STRING, DT_FLOAT, DT_FLOAT], field_delim=",", na_value="", use_quote_delim=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_csv_input_placeholder_0_0, Const, Const, Const, Const, Const, Const_5, Const, Const, Const, Const, Const, Const_5, Const, Const_5, Const_5, Const, Const)]]"
	debug_error_string = "{"created":"@1548247299.274928000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"**Unquoted fields cannot have quotes/CRLFs inside**\n\t [[Node: DecodeCSV = DecodeCSV[OUT_TYPE=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_STRING, DT_STRING, DT_FLOAT, DT_FLOAT], field_delim=",", na_value="", use_quote_delim=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_csv_input_placeholder_0_0, Const, Const, Const, Const, Const, Const_5, Const, Const, Const, Const, Const, Const_5, Const, Const_5, Const_5, Const, Const)]]","grpc_status":3}"
>

[gaoning777]

Could you post your client script that tested the tf-serving model?
@jlewi any idea why this failure would happen to the tf-serving model?

[jinchihe]

@gaoning777 The client script is come from tensorflow/model-analysis in github. Thanks.
https://github.com/tensorflow/model-analysis/blob/master/examples/chicago_taxi/chicago_taxi_client.py

Is there any specific client for the pipeline sample? I haven't seen it. Thanks.

[jinchihe]

Had more investigation the problem, seems the original client script in model-analysis cannot be used in the pipeline directly, :-) , I created new client script for the pipeline and executed successfully.

Thank you, @gaoning777 . I'm going to close the ticket. Thanks.