Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Android/iOS Accessible Live Regions #34735

Closed
aliossam opened this issue Sep 20, 2022 · 13 comments
Closed

Android/iOS Accessible Live Regions #34735

aliossam opened this issue Sep 20, 2022 · 13 comments
Assignees
Labels
Needs: Triage 🔍 Platform: Android Android applications. Platform: iOS iOS applications. Stale There has been a lack of activity on this issue and it may be closed soon.

Comments

@aliossam
Copy link

Description

The accessibilityLiveRegion prop is currently only supported on Android, and there are issues with it even on that platform. Ideally this prop would consistently work on both iOS and Android.

Expected Behavior
Whenever an element with accessibilityLiveRegion set, or any descendant of an element with accessibilityLiveRegion set has any content changes, they should be queued up to be announced by the screen reader. “Polite” announcements should not interrupt any ongoing announcements, while “assertive” ones should interrupt.

Android Details
On Android, the accessibilityLiveRegion prop works in the older Paper renderer, but not the newer Fabric renderer.

iOS Details

On iOS there is no concept of live regions at a system level, so to support this we’d need to detect when content changes on any element (or descendant) with the accessibilityLiveRegion prop set, and make a manual announcement. These announcements will have to be queued up with the UIAccessibilitySpeechAttributeQueueAnnouncement property set to either make them “polite” or “assertive”.

The behavior of exactly how the queue works (maximum length, what happens when new announcements are queued after that length, etc.) should match Android and web as closely as possible.

Version

0.66

Output of npx react-native info

see description

Steps to reproduce

see description

Snack, code example, screenshot, or link to a repository

see description

@react-native-bot react-native-bot added Platform: Android Android applications. Platform: iOS iOS applications. labels Sep 20, 2022
@fabOnReact fabOnReact self-assigned this Nov 18, 2022
@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 21, 2022

The functionality requires the creation of a queue for iOS announcements.

  • assertive may need to interrupt a system announcement (documentation)

Current Tasks:

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 21, 2022

Further investigate announcementFinished API

The functionality was implemented using ANNOUNCEMENT_DID_FINISH_EVENT as discussed on StackOverflow.

/**
* Add an event handler. Supported events:
*
* - `change`: Fires when the state of the screen reader changes. The argument
* to the event handler is a boolean. The boolean is `true` when a screen
* reader is enabled and `false` otherwise.
* - `announcementFinished`: iOS-only event. Fires when the screen reader has
* finished making an announcement. The argument to the event handler is a dictionary
* with these keys:
* - `announcement`: The string announced by the screen reader.
* - `success`: A boolean indicating whether the announcement was successfully made.
*/
addEventListener: function (
eventName: ChangeEventName,
handler: Function
): Object {
var listener;
if (eventName === 'change') {
listener = RCTDeviceEventEmitter.addListener(
VOICE_OVER_EVENT,
handler
);
} else if (eventName === 'announcementFinished') {
listener = RCTDeviceEventEmitter.addListener(
ANNOUNCEMENT_DID_FINISH_EVENT,
handler
);
}

@fabOnReact
Copy link
Contributor

Test UIAccessibilitySpeechAttributeQueueAnnouncement in react-native (StackOverflow)

https://reactnative.dev/docs/next/accessibilityinfo#announceforaccessibilitywithoptions

RCTAccessibilityManager

RCT_EXPORT_METHOD(announceForAccessibilityWithOptions
                  : (NSString *)announcement options
                  : (JS::NativeAccessibilityManager::SpecAnnounceForAccessibilityWithOptionsOptions &)options)
{
  if (@available(iOS 11.0, *)) {
    NSMutableDictionary<NSString *, NSNumber *> *attrsDictionary = [NSMutableDictionary new];
    if (options.queue()) {
      attrsDictionary[UIAccessibilitySpeechAttributeQueueAnnouncement] = @(*(options.queue()) ? YES : NO);
    }

    if (attrsDictionary.count > 0) {
      NSAttributedString *announcementWithAttrs = [[NSAttributedString alloc] initWithString:announcement
                                                                                  attributes:attrsDictionary];
      UIAccessibilityPostNotification(UIAccessibilityAnnouncementNotification, announcementWithAttrs);
    } else {
      UIAccessibilityPostNotification(UIAccessibilityAnnouncementNotification, announcement);
    }
  } else {
    UIAccessibilityPostNotification(UIAccessibilityAnnouncementNotification, announcement);
  }
}

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 21, 2022

Test existing react-native accessibilityLiveRegion functionality on Android Paper

  • Include Video recording of different scenarios (assertive, polite, none)
  • Write down expected behaviour for iOS
Testing Android accessibilityLiveRegion assertive (Paper)

        <RNTesterBlock title="LiveRegion">
          <TouchableWithoutFeedback onPress={this._addOne}>
            <View style={styles.embedded}>
              <Text>Click me</Text>
            </View>
          </TouchableWithoutFeedback>
          <View accessibilityLiveRegion="polite">
            <Text>
              Announced {this.state.count} times with accessibilityLiveRegion
              polite
            </Text>
          </View>
          <View accessibilityLiveRegion="assertive">
            <Text>
              Second announcement {this.state.count} times with
              accessibilityLiveRegion assertive
            </Text>
          </View>
        </RNTesterBlock>

https://www.icloud.com/iclouddrive/08b3HMFKoLEfpMCCyX9enjyIw#android_paper_assertive

Testing Android accessibilityLiveRegion polite (Paper)

        <RNTesterBlock title="LiveRegion">
          <TouchableWithoutFeedback onPress={this._addOne}>
            <View style={styles.embedded}>
              <Text>Click me</Text>
            </View>
          </TouchableWithoutFeedback>
          <View accessibilityLiveRegion="polite">
            <Text>
              Announced {this.state.count} times with accessibilityLiveRegion
              polite
            </Text>
          </View>
          <View accessibilityLiveRegion="polite">
            <Text>
              Second announcement {this.state.count} times with
              accessibilityLiveRegion polite
            </Text>
          </View>
        </RNTesterBlock>

https://www.icloud.com/iclouddrive/078YVxbmyt9XswScPrBtTuGrg#android_paper_polite

Takeaways on Android:

  1. Moving focus to another element (or an announcement triggered by Operating System) will stop the queue
  2. An assertive announcement stops the queue. Announcements that were interrupted on Android will not be repeated.

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 21, 2022

Implementation Design

  1. Different react components have the accessibilityLiveRegion=polite or assertive
<>
  <TouchableWithoutFeedback onPress={this._addOne}>
    <View style={styles.embedded}>
      <Text>Click me</Text>
    </View>
  </TouchableWithoutFeedback>
  <View accessibilityLiveRegion="polite">
    <Text>
      Announced {this.state.count} times with accessibilityLiveRegion
      polite
    </Text>
  </View>
  <View accessibilityLiveRegion="polite">
    <Text>
      Second announcement {this.state.count} times with
      accessibilityLiveRegion polite
    </Text>
  </View>
</>
  1. If their content changes, an announcement is triggered (link)
  2. Assertive or polite (API)

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 21, 2022

Detect accessibility content change

The accessibility content changes if one of the accessibility props (accessibilityLabel, hint, state or child props) changes. (example1, example2)

accessibilityLabel

Tasks:

Commit 972a40a

relevant notes

What are the conditions that may change the accessibility of content on Android

The parent component with accessible={true} should trigger new announcement (documentation and PR). The Diffing should be done on the parent component with accessible={true}

Detecting accessibility announcement change in JavaScript
  1. Parent accessible View (<View accessible={true}><Text /></View>) receives new props (accessiblityState, accessibilityHint or children).
  2. The props change the announcement and should trigger sendAccessibilityAnnouncementWithOptions.
  3. The diffing could be done in componentWillReceiveProps.

RCTRecursiveAccessibilityLabel already computes a new label by iterating over children's views. Logic should not be duplicated on JS side.

fabOnReact added a commit to fabOnReact/react-native that referenced this issue Nov 22, 2022
Read documentation, investigate different solutions, plan work (details)
--
Test existing react-native accessibilityLiveRegion functionality on Android Paper (details)Include Video recording of different scenarios (assertive, polite, none)Write down expected behaviour for iOS
Implementation Design (details)
Detect accessibility content change (details)

Read documentation, investigate different solutions, plan work ([details](facebook#34735 (comment)))
"Test existing react-native accessibilityLiveRegion functionality on Android Paper ([details](facebook#34735 (comment)))
Include Video recording of different scenarios (assertive, polite, none)
Write down expected behaviour for iOS"
Implementation Design ([details](facebook#34735 (comment)))
Detect accessibility content change ([details](facebook#34735 (comment)))
@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 22, 2022

Automatic content grouping

Solution proposted in PR #35432 does not work with automatic content grouping (for ex. child component Text change does not trigger a new announcement when focused on another component).

Sourcecode of the example

function AccessibilityExpandedExample(): React.Node {
  const [expand, setExpanded] = React.useState(false);
  const [pressed, setPressed] = React.useState(false);
  const expandAction = {name: 'expand'};
  const collapseAction = {name: 'collapse'};
  return (
    <>
      <RNTesterBlock title="Collapse/Expanded state change (Paper)">
        <Button
          onPress={() => {
            setExpanded(!expand);
          }}
          title="click me to change state"
        />
        <View
          accessibilityLiveRegion="polite"
          accessibilityRole="button"
          accessible={true}
          focusable={true}
          nativeID={'1'}
          style={{backgroundColor: 'red', height: 200, width: 400}}>
          <Text>{expand ? null : 'my text'}</Text>
        </View>
      </RNTesterBlock>
    </>
  );
}

The parent accessibility label is set to the value of Child attributedString.text. The value is changed when VoiceOver focuses on the component.

static NSString *RCTRecursiveAccessibilityLabel(UIView *view)
{
NSMutableString *result = [NSMutableString stringWithString:@""];
for (UIView *subview in view.subviews) {
NSString *label = subview.accessibilityLabel;
if (!label) {
label = RCTRecursiveAccessibilityLabel(subview);
}
if (label && label.length > 0) {
if (result.length > 0) {
[result appendString:@" "];
}
[result appendString:label];
}
}
return result;
}
- (NSString *)accessibilityLabel
{
NSString *label = super.accessibilityLabel;
if (label) {
return label;
}
return RCTRecursiveAccessibilityLabel(self);
}

- (NSString *)accessibilityLabel
{
return self.attributedText.string;
}

While the prop diffing is triggered when the new prop is passed to the child component, as there is no change in the accessibilityLabel prop, no announcement is triggered here.

if (!newViewProps.accessibilityLabel.empty() && oldViewProps.accessibilityLabel != newViewProps.accessibilityLabel && newViewProps.accessibilityLiveRegion != AccessibilityLiveRegion::None) {
if ([self.nativeId isEqualToString:@"1"]) {
if (@available(iOS 11.0, *)) {
NSMutableDictionary<NSString *, NSNumber *> *attrsDictionary = [NSMutableDictionary new];
attrsDictionary[UIAccessibilitySpeechAttributeQueueAnnouncement] = @(newViewProps.accessibilityLiveRegion == AccessibilityLiveRegion::Polite ? YES : NO);
NSAttributedString *announcementWithAttrs = [[NSAttributedString alloc] initWithString: RCTNSStringFromStringNilIfEmpty(newViewProps.accessibilityLabel)
attributes:attrsDictionary];
UIAccessibilityPostNotification(UIAccessibilityAnnouncementNotification, announcementWithAttrs);
}
}
}

Nested Text receives props update here, an option would be checking for a change in the attributedtext.string

- (void)updateProps:(Props::Shared const &)props oldProps:(Props::Shared const &)oldProps
{
auto const &oldParagraphProps = *std::static_pointer_cast<ParagraphProps const>(_props);
auto const &newParagraphProps = *std::static_pointer_cast<ParagraphProps const>(props);
_paragraphAttributes = newParagraphProps.paragraphAttributes;
if (newParagraphProps.isSelectable != oldParagraphProps.isSelectable) {
if (newParagraphProps.isSelectable) {
[self enableContextMenu];
} else {
[self disableContextMenu];
}
}
[super updateProps:props oldProps:oldProps];
}

The logic behind the Text accessibilityElements seems more complex than other component as they can be nested with accessibilityRole="link", but the idea is that every class can over-ride the accessibilityLabel and other getter methods to change the way labels are generated

- (NSArray *)accessibilityElements
{
auto const &paragraphProps = *std::static_pointer_cast<ParagraphProps const>(_props);
// If the component is not `accessible`, we return an empty array.
// We do this because logically all nested <Text> components represent the content of the <Paragraph> component;
// in other words, all nested <Text> components individually have no sense without the <Paragraph>.
if (!_state || !paragraphProps.accessible) {
return [NSArray new];
}
auto &data = _state->getData();
if (![_accessibilityProvider isUpToDate:data.attributedString]) {
auto textLayoutManager = data.layoutManager.lock();
if (textLayoutManager) {
RCTTextLayoutManager *nativeTextLayoutManager =
(RCTTextLayoutManager *)unwrapManagedObject(textLayoutManager->getNativeTextLayoutManager());
CGRect frame = RCTCGRectFromRect(_layoutMetrics.getContentFrame());
_accessibilityProvider =
[[RCTParagraphComponentAccessibilityProvider alloc] initWithString:data.attributedString
layoutManager:nativeTextLayoutManager
paragraphAttributes:data.paragraphAttributes
frame:frame
view:self];
}
}
return _accessibilityProvider.accessibilityElements;
}

Android - Behaviour when child text changes

https://www.icloud.com/iclouddrive/08aEHI1ivCO9iXBpIvTeFdFfA#android_live_region_announce_text_change

Draft solution fabOnReact@85b1c1f

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 23, 2022

The state update triggers onFocus (when VoiceOver screenreader focus moves to the text).

Mechanism behind ParagraphComponentView state update

/*
* Creates a `State` object (with `AttributedText` and
* `TextLayoutManager`) if needed.
*/
void updateStateIfNeeded(Content const &content);

void ParagraphShadowNode::updateStateIfNeeded(Content const &content) {
ensureUnsealed();
auto &state = getStateData();
react_native_assert(textLayoutManager_);
if (state.attributedString == content.attributedString) {
return;
}
setStateData(ParagraphState{
content.attributedString,
content.paragraphAttributes,
textLayoutManager_});
}

- (void)updateState:(State::Shared const &)state oldState:(State::Shared const &)oldState
{
_state = std::static_pointer_cast<ParagraphShadowNode::ConcreteState const>(state);
[self setNeedsDisplay];
}

- (NSAttributedString *_Nullable)attributedText
{
if (!_state) {
return nil;
}

- (NSString *)accessibilityLabel
{
return self.attributedText.string;
}

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 23, 2022

Investigate how RCTMountingManager triggers state updates in RCTParagraph

if (oldChildShadowView.state != newChildShadowView.state) {
[newChildComponentView updateState:newChildShadowView.state oldState:oldChildShadowView.state];
mask |= RNComponentViewUpdateMaskState;

- (void)updateState:(State::Shared const &)state oldState:(State::Shared const &)oldState
{
_state = std::static_pointer_cast<ParagraphShadowNode::ConcreteState const>(state);
[self setNeedsDisplay];
}

update state is called when Text.js children (the value of the text) changes

CLICK TO OPEN EXAMPLE SOURCECODE

function AccessibilityExpandedExample(): React.Node {
  const [expand, setExpanded] = React.useState(false);
  const [pressed, setPressed] = React.useState(false);
  const expandAction = {name: 'expand'};
  const collapseAction = {name: 'collapse'};
  return (
    <>
      <RNTesterBlock title="Collapse/Expanded state change (Paper)">
        <Button
          onPress={() => {
            setExpanded(!expand);
          }}
          title="click me to change state"
        />
        <View
          accessibilityLiveRegion="polite"
          accessibilityRole="button"
          accessible={true}
          focusable={true}
          nativeID={'1'}
          style={{backgroundColor: 'red', height: 200, width: 400}}>
          <Text>{expand ? null : 'my text'}</Text>
        </View>
      </RNTesterBlock>
    </>
  );
}

  1. The user clicks on the button change state
  2. The nearby component accessibility announcement changes
  3. The nearby component Text changes from my text to empty
  4. RCTMountingManager method RCTPerformMountInstructions receives the state update (the old state, the new state)
  5. RCTParagraphComponentView updateState changes the instance variable _state text field from "my text" to empty (old _state, new _state)

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 25, 2022

Android - testing announcement of accessibilityState selected with accessibilityLiveRegion

ENABLE AUDIO

2022-11-25.17-10-34.mp4

@fabOnReact
Copy link
Contributor

fabOnReact commented Nov 25, 2022

iOS Text accessibilityLabel never announced with Fabric (main branch)

cc @blavalla

This is an iOS issue in the main branch, the Text accessibilityLabel is set to always equal the Text attributed.string

- (NSString *)accessibilityLabel
{
return self.attributedText.string;
}

Android - Text correctly announcing accessibilityLabel

ENABLE AUDIO

2022-11-25.21-22-30.mp4

iOS - not announcing accessibilityLabel

ENABLE AUDIO

VID_20221125_212309.mp4

@github-actions
Copy link

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.

@github-actions github-actions bot added the Stale There has been a lack of activity on this issue and it may be closed soon. label Aug 13, 2023
@github-actions
Copy link

github-actions bot commented Sep 5, 2023

This issue was closed because it has been stalled for 7 days with no activity.

@github-actions github-actions bot closed this as completed Sep 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs: Triage 🔍 Platform: Android Android applications. Platform: iOS iOS applications. Stale There has been a lack of activity on this issue and it may be closed soon.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants