SDK - Components - Hiding signature attribute from CloudPickle #2045
Conversation
Cloudpickle has some issues with pickling type annotations in python versions < 3.7, so they disabled it. cloudpipe/cloudpickle#196 `create component_from_airflow_op` spoofs the function signature by setting the `func.__signature__` attribute. cloudpickle then tries to pickle that attribute which leads to failures during unpickling. To prevent this we remove the `.__signature__` attribute before pickling.
|
/retest |
# Hack to prevent cloudpickle from trying to pickle generic types that might be present in the signature. See cloudpipe/cloudpickle#196 # Currently the __signature__ is only set by Airflow components as a means to spoof/pass the function signature to _func_to_component_spec
|
/retest |
2 similar comments
|
/retest |
|
/retest |
|
@numerology Can you please take a look? |
|
/lgtm |
|
I guess we might want to create a issue to track this hacking work-around. Once cloudpickle solves their problem we can switch to a more elegant way. WDYT? |
They tried several times and ultimately failed to handle it for python <3.7. So they disabled this for python <3.7. But it looks like they only disabled it for the actual function signature and not the normal attributes.
I'll think of a way to solve this. But generally it's just a way to make |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Ark-kun The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
The high-level user issue this solves is that some converted Airflow operators fail to work due to unpickling errors.
Cloudpickle has some issues with pickling type annotations in python versions < 3.7, so they disabled it. cloudpipe/cloudpickle#196
create component_from_airflow_opspoofs the function signature by setting thefunc.__signature__attribute. cloudpickle then tries to pickle that attribute which leads to failures during unpickling.To prevent this we remove the
.__signature__attribute before pickling.This change is