Skip to content

Commit

Permalink
Fix tutorial error and update API reference (#334)
Browse files Browse the repository at this point in the history
Minor fix and updates to 'Tutorials' and 'API reference'.
- 'API reference' claimed "wrappers" are a parameter of both classes
Pipeline() and Service(), but instead there are "before_handler" and
"after_handler", now updated.
- For the above reason, replaced term "wrapper" with "extra handler"
where appropriate within both tutorials and docs. (user guides required
no changes)
- Similarly and for the same reasons, references of "services" from
class Pipeline() changed to "components" both within docs and tutorials.
(user guides required no changes)
- In tutorial Script/Core:4.Transitions comments had wrong order of
transitions, now fixed.

# Checklist

- [x] I have performed a self-review of the changes

# To Consider

- In dff/stats/utils.py there is a function called get_wrapper_field(),
but wrappers were changed to extra handlers, I think. Is this intended?
- Currently, there are no explanations or links to API within the
tutorials as to what ExtraHandlerBuilder and ExtraHandlerFunction
objects are.
- Search for references to changed entities in the API
reference/tutorials/guides
  • Loading branch information
ZergLev authored Jun 19, 2024
1 parent 5f73d05 commit b5ed32e
Show file tree
Hide file tree
Showing 15 changed files with 73 additions and 52 deletions.
11 changes: 7 additions & 4 deletions dff/pipeline/pipeline/pipeline.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,9 @@ class Pipeline:
Class that automates service execution and creates service pipeline.
It accepts constructor parameters:
:param components: (required) A :py:data:`~.ServiceGroupBuilder` object,
that will be transformed to root service group. It should include :py:class:`~.Actor`,
but only once (raises exception otherwise). It will always be named pipeline.
:param script: (required) A :py:class:`~.Script` instance (object or dict).
:param start_label: (required) Actor start label.
:param fallback_label: Actor fallback label.
Expand All @@ -62,10 +65,10 @@ class Pipeline:
:param messenger_interface: An `AbsMessagingInterface` instance for this pipeline.
:param context_storage: An :py:class:`~.DBContextStorage` instance for this pipeline or
a dict to store dialog :py:class:`~.Context`.
:param services: (required) A :py:data:`~.ServiceGroupBuilder` object,
that will be transformed to root service group. It should include :py:class:`~.Actor`,
but only once (raises exception otherwise). It will always be named pipeline.
:param wrappers: List of wrappers to add to pipeline root service group.
:param before_handler: List of `ExtraHandlerBuilder` to add to the group.
:type before_handler: Optional[:py:data:`~.ExtraHandlerBuilder`]
:param after_handler: List of `ExtraHandlerBuilder` to add to the group.
:type after_handler: Optional[:py:data:`~.ExtraHandlerBuilder`]
:param timeout: Timeout to add to pipeline root service group.
:param optimization_warnings: Asynchronous pipeline optimization check request flag;
warnings will be sent to logs. Additionally it has some calculated fields:
Expand Down
4 changes: 2 additions & 2 deletions dff/pipeline/pipeline/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ def pretty_format_component_info_dict(
However, most preferable usage is via `pipeline.pretty_format`.
:param service: (required) Pipeline components info dictionary.
:param show_wrappers: (required) Whether to include Wrappers or not (could be many and/or generated).
:param show_extra_handlers: (required) Whether to include Extra Handlers or not (could be many and/or generated).
:param offset: Current level new line offset.
:param wrappers_key: Key that is mapped to Wrappers lists.
:param extra_handlers_key: Key that is mapped to Extra Handlers lists.
:param type_key: Key that is mapped to components type name.
:param name_key: Key that is mapped to components name.
:param indent: Current level new line offset (whitespace number).
Expand Down
5 changes: 2 additions & 3 deletions dff/pipeline/service/extra.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,10 +114,9 @@ async def _run_function(

async def _run(self, ctx: Context, pipeline: Pipeline, component_info: ServiceRuntimeInfo):
"""
Method for executing one of the wrapper functions (before or after).
Method for executing one of the extra handler functions (before or after).
If the function is not set, nothing happens.
:param stage: current `WrapperStage` (before or after).
:param ctx: current dialog context.
:param pipeline: the current pipeline.
:param component_info: associated component's info dictionary.
Expand Down Expand Up @@ -155,7 +154,7 @@ async def __call__(self, ctx: Context, pipeline: Pipeline, component_info: Servi
@property
def info_dict(self) -> dict:
"""
Property for retrieving info dictionary about this wrapper.
Property for retrieving info dictionary about this extra handler.
:return: Info dict, containing its fields as well as its type.
All not set fields there are replaced with `None`.
Expand Down
14 changes: 7 additions & 7 deletions dff/pipeline/service/group.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def __init__(
async def _run_services_group(self, ctx: Context, pipeline: Pipeline) -> None:
"""
Method for running this service group.
It doesn't include wrappers execution, start condition checking or error handling - pure execution only.
It doesn't include extra handlers execution, start condition checking or error handling - pure execution only.
Executes components inside the group based on its `asynchronous` property.
Collects information about their execution state - group is finished successfully
only if all components in it finished successfully.
Expand Down Expand Up @@ -137,7 +137,7 @@ async def _run(
) -> None:
"""
Method for handling this group execution.
Executes before and after execution wrappers, checks start condition and catches runtime exceptions.
Executes extra handlers before and after execution, checks start condition and catches runtime exceptions.
:param ctx: Current dialog context.
:param pipeline: The current pipeline.
Expand Down Expand Up @@ -198,13 +198,13 @@ def add_extra_handler(
condition: ExtraHandlerConditionFunction = lambda _: True,
):
"""
Method for adding a global wrapper to this group.
Adds wrapper to itself and propagates it to all inner components.
Uses a special condition function to determine whether to add wrapper to any particular inner component or not.
Method for adding a global extra handler to this group.
Adds extra handler to itself and propagates it to all inner components.
Uses a special condition function to determine whether to add extra handler to any particular inner component.
Condition checks components path to be in whitelist (if defined) and not to be in blacklist (if defined).
:param global_extra_handler_type: A type of wrapper to add.
:param extra_handler: A `WrapperFunction` to add as a wrapper.
:param global_extra_handler_type: A type of extra handler to add.
:param extra_handler: A `ExtraHandlerFunction` to add as an extra handler.
:type extra_handler: :py:data:`~.ExtraHandlerFunction`
:param condition: A condition function.
:return: `None`
Expand Down
7 changes: 5 additions & 2 deletions dff/pipeline/service/service.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,10 @@ class Service(PipelineComponent):
:param handler: A service function or an actor.
:type handler: :py:data:`~.ServiceBuilder`
:param wrappers: List of Wrappers to add to the service.
:param before_handler: List of `ExtraHandlerBuilder` to add to the group.
:type before_handler: Optional[:py:data:`~.ExtraHandlerBuilder`]
:param after_handler: List of `ExtraHandlerBuilder` to add to the group.
:type after_handler: Optional[:py:data:`~.ExtraHandlerBuilder`]
:param timeout: Timeout to add to the group.
:param asynchronous: Requested asynchronous property.
:param start_condition: StartConditionCheckerFunction that is invoked before each service execution;
Expand Down Expand Up @@ -159,7 +162,7 @@ async def _run_as_service(self, ctx: Context, pipeline: Pipeline) -> None:
async def _run(self, ctx: Context, pipeline: Pipeline) -> None:
"""
Method for handling this service execution.
Executes before and after execution wrappers, launches `_run_as_actor` or `_run_as_service` method.
Executes extra handlers before and after execution, launches `_run_as_actor` or `_run_as_service` method.
:param ctx: (required) Current dialog context.
:param pipeline: the current pipeline.
Expand Down
18 changes: 9 additions & 9 deletions dff/pipeline/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ class ComponentExecutionState(str, Enum):
@unique
class GlobalExtraHandlerType(str, Enum):
"""
Enum, representing types of global wrappers, that can be set applied for a pipeline.
Enum, representing types of global extra handlers, that can be set applied for a pipeline.
The following types are supported:
- BEFORE_ALL: function called before each pipeline call,
Expand All @@ -93,9 +93,9 @@ class ExtraHandlerType(str, Enum):
Enum, representing wrapper execution stage: before or after the wrapped function.
The following types are supported:
- UNDEFINED: wrapper function with undetermined execution stage,
- BEFORE: wrapper function called before component,
- AFTER: wrapper function called after component.
- UNDEFINED: extra handler function with undetermined execution stage,
- BEFORE: extra handler function called before component,
- AFTER: extra handler function called after component.
"""

UNDEFINED = "UNDEFINED"
Expand Down Expand Up @@ -126,10 +126,10 @@ class ExtraHandlerType(str, Enum):

ExtraHandlerConditionFunction: TypeAlias = Callable[[str], bool]
"""
A function type used during global wrappers initialization to determine
whether wrapper should be applied to component with given path or not.
A function type used during global extra handler initialization to determine
whether extra handler should be applied to component with given path or not.
Checks components path to be in whitelist (if defined) and not to be in blacklist (if defined).
Accepts str (component path), returns boolean (whether wrapper should be applied).
Accepts str (component path), returns boolean (whether extra handler should be applied).
"""


Expand All @@ -154,8 +154,8 @@ class ServiceRuntimeInfo(BaseModel):
Callable[[Context, "Pipeline", "ExtraHandlerRuntimeInfo"], Any],
]
"""
A function type for creating wrappers (before and after functions).
Can accept current dialog context, pipeline, and current wrapper info.
A function type for creating extra handler (before and after functions).
Can accept current dialog context, pipeline, and current extra handler info.
"""


Expand Down
2 changes: 1 addition & 1 deletion dff/stats/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@
from opentelemetry.sdk.trace.export import ConsoleSpanExporter
from opentelemetry.sdk._logs.export import InMemoryLogExporter, ConsoleLogExporter
from opentelemetry.sdk.metrics.export import InMemoryMetricReader, ConsoleMetricExporter
from .utils import get_wrapper_field, set_logger_destination, set_tracer_destination
from .utils import get_extra_handler_name, set_logger_destination, set_tracer_destination
from .instrumentor import OtelInstrumentor, OTLPMetricExporter, OTLPLogExporter, OTLPSpanExporter
6 changes: 3 additions & 3 deletions dff/stats/default_extractors.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

from dff.script import Context
from dff.pipeline import ExtraHandlerRuntimeInfo, Pipeline
from .utils import get_wrapper_field
from .utils import get_extra_handler_name


async def get_current_label(ctx: Context, pipeline: Pipeline, info: ExtraHandlerRuntimeInfo):
Expand Down Expand Up @@ -44,7 +44,7 @@ async def get_timing_before(ctx: Context, _, info: ExtraHandlerRuntimeInfo):
to the context storage.
"""
start_time = datetime.now()
ctx.framework_states[get_wrapper_field(info, "time")] = start_time
ctx.framework_states[get_extra_handler_name(info, "time")] = start_time


async def get_timing_after(ctx: Context, _, info: ExtraHandlerRuntimeInfo): # noqa: F811
Expand All @@ -56,7 +56,7 @@ async def get_timing_after(ctx: Context, _, info: ExtraHandlerRuntimeInfo): # n
As a result, the function output is cleared on every turn and does not get persisted
to the context storage.
"""
start_time = ctx.framework_states[get_wrapper_field(info, "time")]
start_time = ctx.framework_states[get_extra_handler_name(info, "time")]
data = {"execution_time": str(datetime.now() - start_time)}
return data

Expand Down
4 changes: 2 additions & 2 deletions dff/stats/instrumentor.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
from dff.script.core.context import get_last_index
from dff.stats.utils import (
resource,
get_wrapper_field,
get_extra_handler_name,
set_logger_destination,
set_meter_destination,
set_tracer_destination,
Expand Down Expand Up @@ -158,7 +158,7 @@ async def __call__(self, wrapped, _, args, kwargs):
:param kwargs: Keyword arguments of the decorated function.
"""
ctx, _, info = args
pipeline_component = get_wrapper_field(info)
pipeline_component = get_extra_handler_name(info)
attributes = {
"context_id": str(ctx.id),
"request_id": get_last_index(ctx.requests),
Expand Down
3 changes: 2 additions & 1 deletion dff/stats/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,15 @@ def set_tracer_destination(exporter: Optional[SpanExporter] = None):
get_tracer_provider().add_span_processor(BatchSpanProcessor(exporter))


def get_wrapper_field(info: ExtraHandlerRuntimeInfo, postfix: str = "") -> str:
def get_extra_handler_name(info: ExtraHandlerRuntimeInfo, postfix: str = "") -> str:
"""
This function can be used to obtain a key, under which the wrapper data will be stored
in the context.
:param info: Handler runtime info obtained from the pipeline.
:param postfix: Field-specific postfix that will be appended to the field name.
"""

path = info.component.path.replace(".", "-")
return f"{path}" + (f"-{postfix}" if postfix else "")

Expand Down
8 changes: 4 additions & 4 deletions tutorials/pipeline/3_pipeline_dict_with_services_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,16 @@
"""
When Pipeline is created using `from_dict` method,
pipeline should be defined as a dictionary.
It should contain `services` - a `ServiceGroupBuilder` object,
It should contain `components` - a `ServiceGroupBuilder` object,
basically a list of `ServiceBuilder` or `ServiceGroupBuilder` objects,
see tutorial 4.
On pipeline execution services from `services`
On pipeline execution services from `components`
list are run without difference between pre- and postprocessors.
Actor constant "ACTOR" is required to be passed as one of the services.
ServiceBuilder object can be defined either with callable
`ServiceBuilder` object can be defined either with callable
(see tutorial 2) or with dict / object.
It should contain `handler` - a ServiceBuilder object.
It should contain `handler` - a `ServiceBuilder` object.
Not only Pipeline can be run using `__call__` method,
for most cases `run` method should be used.
Expand Down
22 changes: 16 additions & 6 deletions tutorials/pipeline/3_pipeline_dict_with_services_full.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,28 +43,38 @@
is used to connect to channel and transfer IO to user.
* `context_storage` - Place to store dialog contexts
(dictionary or a `DBContextStorage` instance).
* `services` (required) - A `ServiceGroupBuilder` object,
* `components` (required) - A `ServiceGroupBuilder` object,
basically a list of `ServiceBuilder` or `ServiceGroupBuilder` objects,
see tutorial 4.
* `wrappers` - A list of pipeline wrappers, see tutorial 7.
* `before_handler` - a list of `ExtraHandlerFunction` objects,
`ExtraHandlerBuilder` objects and lists of them.
See tutorials 6 and 7.
* `after_handler` - a list of `ExtraHandlerFunction` objects,
`ExtraHandlerBuilder` objects and lists of them.
See tutorials 6 and 7.
* `timeout` - Pipeline timeout, see tutorial 5.
* `optimization_warnings` - Whether pipeline asynchronous structure
should be checked during initialization,
see tutorial 5.
On pipeline execution services from `services` list are run
On pipeline execution services from `components` list are run
without difference between pre- and postprocessors.
If "ACTOR" constant is not found among `services` pipeline creation fails.
If "ACTOR" constant is not found among `components` pipeline creation fails.
There can be only one "ACTOR" constant in the pipeline.
ServiceBuilder object can be defined either with callable (see tutorial 2) or
`ServiceBuilder` object can be defined either with callable (see tutorial 2) or
with dict of structure / object with following constructor arguments:
* `handler` (required) - ServiceBuilder,
if handler is an object or a dict itself,
it will be used instead of base ServiceBuilder.
NB! Fields of nested ServiceBuilder will be overridden
by defined fields of the base ServiceBuilder.
* `wrappers` - a list of service wrappers, see tutorial 7.
* `before_handler` - a list of `ExtraHandlerFunction` objects,
`ExtraHandlerBuilder` objects and lists of them.
See tutorials 6 and 7.
* `after_handler` - a list of `ExtraHandlerFunction` objects,
`ExtraHandlerBuilder` objects and lists of them.
See tutorials 6 and 7.
* `timeout` - service timeout, see tutorial 5.
* `asynchronous` - whether or not this service _should_ be asynchronous
(keep in mind that not all services _can_ be asynchronous),
Expand Down
4 changes: 2 additions & 2 deletions tutorials/pipeline/4_groups_and_conditions_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@
Pipeline can contain not only single services, but also service groups.
Service groups can be defined as `ServiceGroupBuilder` objects:
lists of `ServiceBuilders` and `ServiceGroupBuilders` or objects.
The objects should contain `services` -
a ServiceBuilder and ServiceGroupBuilder object list.
The objects should contain `components` -
a `ServiceBuilder` and `ServiceGroupBuilder` object list.
To receive serialized information about service,
service group or pipeline a property `info_dict` can be used,
Expand Down
11 changes: 8 additions & 3 deletions tutorials/pipeline/4_groups_and_conditions_full.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,14 @@
Alternatively, the groups can be defined as objects
with following constructor arguments:
* `components` (required) - A list of ServiceBuilder objects,
ServiceGroup objects and lists of them.
* `wrappers` - A list of pipeline wrappers, see tutorial 7.
* `components` (required) - A list of `ServiceBuilder` objects,
`ServiceGroupBuilder` objects and lists of them.
* `before_handler` - a list of `ExtraHandlerFunction` objects,
`ExtraHandlerBuilder` objects and lists of them.
See tutorials 6 and 7.
* `after_handler` - a list of `ExtraHandlerFunction` objects,
`ExtraHandlerBuilder` objects and lists of them.
See tutorials 6 and 7.
* `timeout` - Pipeline timeout, see tutorial 5.
* `asynchronous` - Whether or not this service group _should_ be asynchronous
(keep in mind that not all service groups _can_ be asynchronous),
Expand Down
6 changes: 3 additions & 3 deletions tutorials/script/core/4_transitions.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,10 +154,10 @@ def transition(_: Context, __: Pipeline) -> ConstLabel:
"node2": {
RESPONSE: Message("Good. What do you want to talk about?"),
TRANSITIONS: {
lbl.to_fallback(0.1): cnd.true(), # third check
lbl.to_fallback(0.1): cnd.true(), # fourth check
# lbl.to_fallback(0.1) is equivalent
# to ("global_flow", "fallback_node", 0.1)
lbl.forward(0.5): cnd.regexp(r"talk about"), # second check
lbl.forward(0.5): cnd.regexp(r"talk about"), # third check
# lbl.forward(0.5) is equivalent
# to ("greeting_flow", "node3", 0.5)
("music_flow", "node1"): cnd.regexp(
Expand All @@ -167,7 +167,7 @@ def transition(_: Context, __: Pipeline) -> ConstLabel:
# to ("music_flow", "node1", 1.0)
lbl.previous(): cnd.regexp(
r"previous", re.IGNORECASE
), # third check
), # second check
},
},
"node3": {
Expand Down

0 comments on commit b5ed32e

Please sign in to comment.