[Android] AutomationId no longer works

[GeorgeDarakchiev]

Description

When AutomationId is set to any MAUI element\view it is not set as a ContentDescription to the Android element\view.
You can find project showing the issue here: https://github.com/GeorgeDarakchiev/ReproProjects/tree/main/AutomationIdRepro/AutomationIdRepro

It seems like it is caused by PR: #3835 where there is
else if (info.ContentDescription == virtualView.AutomationId) info.ContentDescription = null;

Anyway, this leads to UI Automation frameworks (like Appium) not being able to locate elements by Automaton\Accessibility Id (depending on the naming in the framework).

In this image you can see on the left how it was before, and on the right how it is now (no content description)

Steps to Reproduce

1. Run the app from here: https://github.com/GeorgeDarakchiev/ReproProjects/tree/main/AutomationIdRepro/AutomationIdRepro
2. Use any automation tool like Appium to check the content description.

Version with bug

Preview 12 (current)

Last version that worked well

Preview 11

Affected platforms

Android

Affected platform versions

All

Did you find any workaround?

As a workaround SemanticProperties.Description can be set, but this leads to issues on iOS.

Relevant log output

No response

[StuartFerguson]

Is there an ETA for this issue being fixed?

[PureWeen]

I don't think this is something that we are going to be able to change. Using contentDesc for AutomationId was a very incorrect decision because it completely breaks accessibility.

Basically what you say here

As a workaround SemanticProperties.Description can be set, but this leads to issues on iOS.

Is what breaks on Android accessibility wise.

My recommendations here would be to either

1. Hook into ViewHandler.ViewMapper.ModifyMapping("AutomationId", ) and map the AutomationId to something like "Tag" and then use that inside appium instead
2. if you want to replace our behavior the MauiAccessibilityDelegateCompat is public so you can inherit from that and opt out of the behavior we added in the PR you referenced

[StuartFerguson]

@PureWeen do you have an example of option 1 you have mentioned?

[PureWeen]

1. ViewHandler.ViewMapper.ModifyMapping("AutomationId", )

public partial class App : Application
{
	public App()
	{
		InitializeComponent();

		MainPage = new AppShell();


#if ANDROID
        ViewHandler.ViewMapper.ModifyMapping("AutomationId", (handler, view, previousAction) =>
        {
            if (view is Android.Views.View androidView)
            {
                androidView.SetTag(42, view.AutomationId);
            }
        });
#endif
    }
}

[StuartFerguson]

I realise the initial issue was for Android but is there equivalent code for the other platforms (iOS, MacOS and Windows) or are they not affected by the change and can still use AutomationId ?

[PureWeen]

I realise the initial issue was for Android but is there equivalent code for the other platforms (iOS, MacOS and Windows) or are they not affected by the change and can still use AutomationId ?

iOS/Windows/MacOS shouldn't be affected because all of those platforms have a specific property for this purpose and we still map AutomationId to those. Android is the only one that doesn't provide anything here for us to use.

[GeorgeDarakchiev]

Hi @PureWeen

This is troubling for me:

I don't think this is something that we are going to be able to change.

I read this as: The most important property for testability (and every QA) will never work for Android. It seems like developers have to choose between testability or accessibility, but you are not giving us the option to do that. For some testability is more important than accessibility.
Furthermore surely there should be a way for you to accommodate both properties (yes, I know they both use one native property under the hood). For example if only AutomationId is set -> ContentDescription is set, if only SemanticProperties.Description is set ->ContentDescription, if both are set the SemanticProperties.Description is used.
It would be great if you can share how AutomationId breaks accessibility - as I understand it if I set AutomationId it will set ContentDescription and this is what the screen reader will read?

Also, can you please share how do you write your automation tests? Do you use AutomationId or something else?

P.S. The workaround is not so feasible as the ViewHandler.ViewMapper.ModifyMapping method is not called for controls that have their own Mapper, event if it is based on the ViewHandler. For example it is not called for ContentView or ImageButton and for them it should be done separately. This is tedious, there is no documentation and I am spending days just to be able to do something as simple as setting an AutomationId and continue with creating a basic test for a MAUI app, while in Xamarin there weren't such issues.

Microsoft.Maui.Handlers.ViewHandler.ViewMapper.ModifyMapping(nameof(IView.AutomationId), (h, v, a) => SetAutomationId(v));
Microsoft.Maui.Handlers.ContentViewHandler.Mapper.ModifyMapping(nameof(IView.AutomationId), (h, v, a) => SetAutomationId(v));
Microsoft.Maui.Handlers.ImageButtonHandler.Mapper.ModifyMapping(nameof(IView.AutomationId), (h, v, a) => SetAutomationId(v));

[PureWeen]

The difficulty you ran into with hooking into Mappers definitely needs to be remedied

We'll need to review the cases with Appium a bit more. Because I see that even in appium's documentation they reference content-desc . Does searching by Tag not work for your scenario?

[amolbhushan]

@PureWeen any timelines to review the cases with Appium? This is blocking us from writing automation for our product app.

[PureWeen]

@amolbhushan can you switch to using tag?

The other option here would be to add your own attached property and then you can add a mapper for that property that you map to contentDesc

#if ANDROID
        ViewHandler.ViewMapper.ModifyMapping("OurAutomationId", (handler, view, previousAction) =>
        {
            if (view is Android.Views.View androidView)
            {
                androidView.ContentDescription = view.AutomationId;
            }
        });
#endif

[amolbhushan]

tag

Not sure what you mean by using tag?

[PureWeen]

tag

Not sure what you mean by using tag?

#4714 (comment)

Appium has a few different ways beyond the contentDesc for searching for elements
https://appium.io/docs/en/commands/element/find-elements/

Tag being one of them

[GeorgeDarakchiev]

Hi @PureWeen
Here is some general information about Appium and why using Tag is not feasible.

Appium has a server-client architecture, where the Appium server is started and communicates with the client (called driver). For each platform (Android, iOS, etc.) there is a different driver. Each driver under the hook is using the given platform Automation framework. For example for iOS the XCUITest Driver is using the XCTest that comes with Xcode.
For Android however there are two frameworks that can be used Espresso and UIAutomator. The UI Automator is recommended, but the Espresso cannot be used with Maui and Xamarin as the first step of its setup is "Open your app’s build.gradle file .." and we don't have access to this from maui. So the UI Automator cannot search by tags and that is why it is not usable as a workaround.

Anyway, here is more info on what we are trying to achieve with Appium and Maui.
As Maui is a cross-platform solution - we want to create cross-platform tests and Appium is the best solution (for us) in the market for this purpose. We can write one test and execute it on all 4 platforms - Android, iOS, Windows, Mac.
Imagine a login screen with two Entry controls and one Button and we want to create a test for login. The best way to ensure that it works for all platforms and that the correct Entry is located is to set AutomationId to all elements - "UsernameEntry" and "PasswordEntry".
Using other ways for searching is way harder and leads to ugly and hard to maintain code - for example if searching by ClassName - it would be different in all platforms and there would be need for Ifs like If in Android search for EditText, if in iOS search for UITextField. Searching by ClassName however is not recommended as at some point index would be used - for the second Entry - and that makes the test prone to failure if the page under test is changed.
So that is way we desperately need AutomationId to work on all platforms. Below you can see a sample code of how a test would look like if AutomationId works and how it looks now as we need to search differently on different platforms:

        [Test]
        public void LoginTest()
        {
            AppiumWebElement userNameEntry = App.Driver.FindElementByAccessibilityId("UserNameEntry");
            userNameEntry.SendKeys("MyUsername");

            AppiumWebElement passwordEntry = App.Driver.FindElementByAccessibilityId("PaswordEntry");
            passwordEntry.SendKeys("MyPassword");

            App.Driver.FindElementByAccessibilityId("LoginButton").Click();
        }

       [Test]
        public void LoginTest()
        {
            AppiumWebElement userNameEntry = null;
            switch (Settings.Device.Platform)
            {
                case MobilePlatform.IOS:
                case MobilePlatform.MacOS:
                    userNameEntry = App.Driver.FindElementByAccessibilityId("UserNameEntry");
                    break;
                case MobilePlatform.Android:
                    userNameEntry = App.Driver.FindElementByClassName("EditText");
                    break;
                case MobilePlatform.Windows:
                    userNameEntry = App.Driver.FindElementByClassName("Edit");
                    break;
            }
            userNameEntry.SendKeys("MyUsername");

            AppiumWebElement passwordEntry = null;
            switch (Settings.Device.Platform)
            {
                case MobilePlatform.IOS:
                case MobilePlatform.MacOS:
                    userNameEntry = App.Driver.FindElementByAccessibilityId("PaswordEntry");
                    break;
                case MobilePlatform.Android:
                    userNameEntry = App.Driver.FindElementsByClassName("EditText")[1];
                    break;
                case MobilePlatform.Windows:
                    userNameEntry = App.Driver.FindElementsByClassName("Edit")[1];
                    break;
            }
            passwordEntry.SendKeys("MyPassword");

            App.Driver.FindElementByAccessibilityId("LoginButton").Click();
        }

Looking forward to your replay.

[PureWeen]

@GeorgeDarakchiev I don't currently have Appium setup locally. Can you test if this works
MauiApp27.zip
?

I implemented this code based on what I found here
https://dev.to/nextlevelbeard/an-end-to-the-abuse-of-accessibility-ids-5d2j

If this doesn't work can you send the project back to me with Appium setup and failing?

Also, thank you for taking the time to give me such thorough explanations :-) I haven't done a lot with Appium so our current workaround is based on Xamarin.UITests which works fine but Xamarin.UITest uses a different mechanism for searching than Appium does :-/

[GeorgeDarakchiev]

I can confirm that contend-description is set with the AutomationId from the App.

Also you can easily inspect the app without having to create tests or some fancy Appium setup.

You just need to follow the steps below:

1. Deploy and start the app on emulator
2. Start Appium Server (can be downloaded from here)
3. Start Appium Inspector (can be downloaded from here)
4. In the Inspector add the following capabilities
   • deviceName -> [your_emulator_name]
   • platformName - > Android
5. Click "Start Session"

Then you will be able to inspect the app and see the tree, whether elements have content description and search for elements.

[PureWeen]

@GeorgeDarakchiev so the above zip file worked for you? You were able to use FindElementByAccessibilityId successfully?

[PureWeen]

@StuartFerguson @amolbhushan @GeorgeDarakchiev

Does the project here make it so AutomationId's work for you on appium?
https://github.com/dotnet/maui/files/8586594/MauiApp27.zip

@GeorgeDarakchiev
I think your last post says that it does but I'm just making sure

[GeorgeDarakchiev]

Yes, it works in the project.