Implement new framework to observe and notify for trait changes

[kitchoi]

This is an issue for implementing a new, simpler and cleaner, parallel notification system to the current trait listeners, with the eventual intent of replacing the existing listeners in new code, and eventually deprecating for old code.

Targeting EEP 3.

Specific "end" goals for this issue:

• @observe decorator analogous to the @on_trait_change

More specific, smaller issues will be opened referencing this one.

Updated

Specific tasks:

• 1. Implement the ObserverGraph for representing a graph of observers (e.g. "a.b.c" is a graph from node "a" to node "b" to node "c"). See PR Implement ObserverGraph with NamedTraitObserver #976
• 2. Implement a notifier wrapper for user defined handler (See PR Add notifier for wrapping user handler for the observer framework #1000)
• 3. Implement a notifier wrapper for maintaining nested observers (See PR Add notifier for maintaining nested notifiers in the observer framework #1007)
• 4. User facing Expression for creating instances of ObserverGraph, depends on 1.
  See code in here. (issue User facing Expression for creating instances of ObserverGraph #1024, PR Add user-facing Expression for composing a graph of observers #1067)
• 5. Text parser for parsing traits domain specific language to an instance of Expression, depends on 4.
  See PR Add mini-language standalone parser for observe  #1066 for getting a syntax tree, and Implement parsing logic for converting text to Expression #1079 for converting it to Expression.
• 6. Logic for walking down the ObserverGraph and attach/detach notifiers (depends on 1 for using ObserverGraph in tests, but a stub replacement can be made easily)
  See PR Implement logic for adding/removing notifiers for a given ObserverGraph #1023
• 7. Implement the logic for attaching notifiers when a named trait changes (depends on 2, 3. Meaningful semi-integration tests will depend on 1 and 6 in addition)
  See PR Implement the logic for observing a named trait #1069
• 8. Implement the logic for attaching notifiers when the items of a list changes (depends on 2, 3. Meaningful semi-integration tests will depend on 1, 6 and 7 in addition.)
  See PR Implement ListItemObserver for observing mutations to a list #1071
• 9. Implement the logic for attaching notifiers when the items of a dict changes (depends on 2, 3. Meaningful semi-integration tests will depend on 1, 6 and 7 in addition.)
  See PR Implement DictItemObserver for observing mutations to a dict #1072
• 10. Implement the logic for attaching notifiers when the items of a set changes (depends on 2, 3. Meaningful semi-integration tests will depend on 1, 6 and 7 in addition.)
  See PR Implement SetItemObserver for observing mutations on a set  #1075
• 11. Implement the logic for handling trait_added (depends on 2, 3. Meaningful semi-integration tests will depend on 1, 6 and 7 in addition.)
  See PR Add trait_added support for observing a named trait #1077
• 11b. Add trait_added support to FilteredTraitObserver (Add trait_added support for observing a named trait #1077)
• 12. Implement the logic for observing traits satisfying a generic filter (depends on 2, 3. Meaningful semi-integration tests will depend on 1 and 6 in addition.).
  See PR Add an observer for observing traits using an arbitrary filter #1078
• 13. Implement the logic for observing traits whose metadata fulfils a given criterion (depends on 12). This will support the metadata method in Expression, and the use of "+" in the text parser.
  See Add MetadataFilter for observing traits using metadata #1095
• 14. Expose the observe decorator and HasTraits.observe instance method.
  See PR Add observe decorator and instance method #1086
• 15. Add warning regarding comparison_mode on List, Set, Dict
  See PR Add warning while observing container items with comparison mode set to equality #1117
• 16. Documentation! (Some of that can be done along the way) (Add user manual for observe #1060)
• 17. Implement the print method on Expression (bonus)
• 18. Rename subpackage observers -> observation.
• 19. Rename Expression -> ObserverExpression
  See PR Rename (observers) Expression to ObserverExpression #1118

[kitchoi]

Assuming the PoC in #942 has more or less get the framework right, I will try to break down components in the order of implementation:

1. ObserverPath (pending PR Implement ObserverGraph with NamedTraitObserver #976). (See also ListenerPath in the poc)
2. Notifier wrapper for the user defined change handler. (See TraitObserverNotifier in the poc)
3. Notifier wrapper for unhooking and rehooking downstream notifiers when an upstream object is changed. See ListenerChangeNotifier in the poc.
4. Implement observe function for a specific type of observer: e.g. A trait with a specific name. This should implement adding and removing notifiers, putting together the components above.
5. Implement a decorator for observer, to be used like on_trait_change.

After point 4, the following can happen independently of the rest. I would consider them orthogonal to this issue, although not necessarily excluded from the 6.1.0 release (to be confirmed):

• Observer for list items (see ListItemListener in the poc)
• Observer for dict items
• Observer for dict keys
• Observer for dict values
• Observer for set items
• Observer for filtering traits using metadata
• Observer for an arbitrary filter on traits
  ...

[kitchoi]

I have updated the main comment with more specific tasks and their dependencies. One can open separate issues when we come closer to being able to work on them.

[kitchoi]

Implement the print method on Expression (bonus)

I initially added this item in the list because I had implemented this feature in my test branch. The print method does something like this:

>>> parse("a.items.c").print()                                                                                                                                                                                                                                                                                                                                        
Observes a trait named 'a' with notifications,
    then optionally observes a trait named 'items' with notifications,
         then observes a trait named 'c' with notifications.
    then optionally observes items in a list with notifications,
         then observes a trait named 'c' with notifications.
    then optionally observes items in a dict with notifications,
         then observes a trait named 'c' with notifications.
    then optionally observes items in a set with notifications,
         then observes a trait named 'c' with notifications.

It was useful for inspection (i.e. for me while tackling recursion) and would be useful for users learning the mini-language, this is not needed for another purposes.

While self-documenting components are preferred to written documentation and the feature is valuable for educational purposes, it still requires code, and with code, maintenance.

For most use cases, I believe the mini-language expression is intuitive enough such that this feature for inspection may not be necessary. All-in-all, this is a nice-to-have, not a must have.

With that, I am planning to exclude this feature from the list and call this issue feature-complete.

[kitchoi]

Status update:

This is the last pending item

Documentation

API documentation is completed as part of the PRs introducing new code.
#1060 will expand the user manual.

[kitchoi]

Closing via #976, #1000, #1007, #1065, #1023, #1066, #1070, #1069, #1067, #1080, #1082, #1079, #1071, #1072, #1075, #1085, #1089, #1078, #1093, #1086, #1077, #1095, #1102, #1108, #1110, #1112, #1117, #1118, #1123, #1125, #1126, #1128, #1129, #1135, #1060