Accessibility: Gesture bindings missing textual description

[RastislavKish]

Hello!

First of all, thanks for this amazing project! 😁

I'm a blind user. The app works pretty well with TalkBack, except the gesture bindings. Right now, each binding is represented by a button, which as far as I understand contains just the icon of the action to be performed, i.e. an app icon if the user configured an app to be launched, or an action icon if they bound one of the supported operations.

But there is currently no textual information which would tell the screenreader what's bound to the specific gesture. I can think of two solutions here:

• Use the contentDescription attribute on the binding button to textually describe the bound action. The binding configuration screen already has nice textual descriptions, so if you just use these, it should work. There will be no visual changes, the attribute is used only by screenreaders.
• Redesign the representation of the bindings such that the buttons will include the textual description. This is a visual change to the interface, which could perhaps benefit also the sighted users in being more clear about what's assigned to the particular gesture. Especially if you at some point decide to implement using app shortcuts, this would be very cool, but with all the different actions supported by individual apps, the shortcut's name may be more representative than the icons.

Reproduction steps

1. Open the launcher settings on the gesture bindings tab
2. Turn on TalkBack by holding the up and down volume keys together for few seconds
3. Use left and right swipes to navigate around the interface, and find the button located between the gesture description and its respective "Remove binding" button

Expected behavior

TalkBack should read aloud the action associated with the binding as the button's title, just like it's reading "Choose app" if there's no assigned action.

Actual behavior

The button is read just as "button".

Environment

Xiaomi redmi note 12 pro 5G, Android 14, TalkBack 16, μLauncher 0.2.3

[jrpie]

Thanks for reaching out!

First of all, thanks for this amazing project! 😁

Thank you, nice to hear that! The original idea is by Finn Glas though; I'm just maintaining this fork and doing some improvements.

I'm a blind user. The app works pretty well with TalkBack, except the gesture bindings. Right now, each binding is represented by a button, which as far as I understand contains just the icon of the action to be performed, i.e. an app icon if the user configured an app to be launched, or an action icon if they bound one of the supported operations.

I never thought this app would work very well with accessibility tools - I didn't believe the gesture-based approach was a particularly good fit for such use cases - glad to hear that it does work for you!

Yes, the interface indeed only shows an icon of the action. I definitely should set a contentDescription (sorry for forgetting about that!). Adding text to the layout won't work since there is not enough space (especially on small screens)

[jrpie]

This will be fixed in the next version. Are you aware of any other accessibility issues? (I really need to figure out how to use a screen reader)

[RastislavKish]

Thank you for the quick response and fix! 😁

The original idea is by Finn Glas though; I'm just maintaining this fork and doing some improvements.

Having ideas is great, but evolving things and keeping them in shape often takes much higher effort and is very important. We have tons of launchers, but only handful actively maintained and thus actually useful. You're doing a great job!

I didn't believe the gesture-based approach was a particularly good fit for such use cases

Ach, quite the opposite, actually. 😃 At least when it comes to the blind. You know, we're not particularly good at immediate pinpointing icons in a large group, but speed-sketching on the screen, no problem! 😃 And we're actually damn fast at that.

Screenreaders that blind folks use on smartphones are based on gestures, and over the time, some of them have developed really wild and pretty creative gesture systems, where you can by few motions invoke literally hundreds of actions.

Unfortunately, very few are open-source, so I personally just use ordinary launchers with gestures and powerful search. For the apps I use most often, I have assigned gestures and can quick-open them, for the rest, I use the launcher search to quick-execute what I need. μLauncher is very powerful in both.

Are you aware of any other accessibility issues?

There are no major issues which would have significant impact on the user experience. Perhaps one thing that could be improved is the app list. Right now, each app has a button with the app icon, and a label carrying the app name. Both perform the same functionality as far as I understand. However, for the screenreader, they're separate objects, so when the user swipes through the list, they pass through each app twice (once they hit the icon button and once the label). And the icon button does not have a description, which is not necessary in this case because the label is literally next to it, but a new user may wonder what is the purpose of the button.

To be honest I'm not particularly sure what solution would be the most correct. Usually, the best you can do is using standard semantic components for anything you need to represent, so if there for instance was a standard component with the purpose of carrying an icon and its description, it would be the right thing to use. But I never bothered about icons in my apps, so I have no idea if there is such a component and what styling options it offers. 😃

Another approach is to just mark the icons as unimportant for screenreaders. I can't recall now the name of the attribute, but View does have one which will communicate to the screenreader: "hey, this is just some unimportant static component, feel free to ignore it." and Talkback should just pass by. Of course this can be only done if the two objects really have identical functionality.

Another thing that could be added, but is more of a pleasant bonus rather than necessity, are a11y actions. Often in programs, you have certain components on which you cann perform different actions. For instance, the app list offers for each app the option to add it to favorites, open information about the app, hide the app, uninstall the app etc. You can offer these also as accessibility actions, in that case, the screenreader user will be able to swipe up / down on the item and directly choose the operation they want to perform, without opening the standard menu by a long press.

However, in this case the menu is nicely accessible and focused right away when displayed, so the a11y actions are more of a formality than actual necessity. There are apps which display these menus in a completely different part of the window, or require you to drag the item somewhere to do something with it, that's the kind of scenario where this is really significant.

I really need to figure out how to use a screen reader

Basically, you move backward / forward by swiping left / right, this is similar to pressing the Tab key on the computer. If you want to activate something, you double-tap, if you want to go back, you swipe down and left, like a horizontally reversed capital L shape (or at least I think this is the default configuration 😅). You can also focus things by touching them. Generally just sliding your finger on the screen or touching the screen won't send touch events to the app, but will focus and read what's under your finger.

If you want to pass gestures to the app, you need to use two fingers next to each other, the line will be drawn between them. This is commonly used for scrolling long lists, where we swipe up / down with two fingers, but it also works for passing swipes into the apps. Note tapping does not work this way.

Swiping up / down can have different functions, which you can switch between by making up-down / down-up gestures. By default, it controls navigation by characters, but can be switched to navigation by words, lines, headings, links and if the focused component has a11y actions, this will cycle between them and double-tapping will activate the chosen action.

If you need to type on a keyboard, IF the keyboard is screenreader accessible, you can slide your finger on it and when you focus some letter and put your finger up, it will be written down, no need to double-tap except for several special keys. In other words, you can use the keyboard pretty much like you would normally. There are more advanced forms of blind touch-screen typing which are much faster, but those are not important in this context.

These are the fundamentals, which should help you get started and perform basic things. 🙂

[jrpie]

Perhaps one thing that could be improved is the app list.

I guess this is the android:importantForAccessiblity property (?)
By the way, there is a text-based layout of the app list, which would probably work much better with accessibility.

Many thanks for the detailed explanation!

[RastislavKish]

I have compiled the current development version and installed on my second device. Works very nicely! 😁 The bindings are read as they should, and the app list also works flawlessly.

I've discovered one more thing, next to the search field, there is a checkbox which currently does not have any description.

But this looks really good. Thank you!

[jrpie]

I have compiled the current development version and installed on my second device. Works very nicely! 😁

Thanks for testing 👍🏼

I've discovered one more thing, next to the search field, there is a checkbox which currently does not have any description.

oh, that's is a button to toggle between favorite apps and all apps. I added a content description for it.

[jrpie]

By the way, do you think µLauncher is particularly well suited for blind users? This is definitely not the target audience I had in mind (as I said, I naively imagined it would be utterly useless for such use cases), but if you think it works well, I could add a sentence about that to the description on the app stores. (If it works for you, but you wouldn't recommend it to other blind people, I'll just leave the description as it is).

[RastislavKish]

oh, that's is a button to toggle between favorite apps and all apps. I added a content description for it.

I've updated my installation, works perfectly! 😁 Thank you so much!

By the way, do you think µLauncher is particularly well suited for blind users?

This is a very good question. I think it definitely is something that blind users may potentially be interested in. But at the same time, just like in the sighted community, the needs vary.

I use my smartphone pretty heavily on travels. As such, I know it from top to bottom, and my main priority is being able to use it efficiently on the go. That means being able to spawn things as swiftly as possible, and µLauncher is very strong in this regard. And I think many other blind people who rely on their smartphones could see it similarly. Some of them may use the gesture systems of the proprietary screenreaders, these have an advantage in being accessibility services, so they can capture gestures anywhere in the system. But at the same time, they usually don't have a search system of µLauncher, which is also very important, and most of them come as apks, many even technically skilled people are not willing to sideload these days and prefer apps from stores, just for the convenience.

So yes, µLauncher has multiple strengths which may be particularly attractive to blind users, plus it's free, open-source and privacy respecting.

But at the same time, there is a large group of blind folks who don't use their smartphones regularly. They primarily use their computers, and have smartphones for calling and several useful apps like OCR, image describing, messaging etc. For this group, having an organised home screen with folders and icons for their most used apps may be a more reasonable solution, because their priority is not efficiency but readability.

Nova launcher is somewhere in the middle between these two. It has many functions for building a layout that makes most sense to you, while it also offers quite a few gestures and has usable search capabilities. It has also very thorough a11y support, so it's very popular in the blind community. It's the golden middle grond for everyone.

I was using it myself, I built a very elaborate system of home screens, folders and icons. But over time, I found that I most often use only the gestures or the search, I sporadically used icons and when I did, I was usually just wasting time, because I could do the same faster with search. Eventually, the recent events at Nova launcher forced me to finally look for an open-source alternative, and wen I found µLauncher is not just usable, but actually better in both gestures and search than Nova, because it's designed for efficiency, it made me very interested.

So, back to your original question 🙂 Yes, I do think µLauncher is particularly suited for blind people, with the addition, that it's particularly suited for the blind people seeking efficiency. I genuinely do not know how large is this group. I have a blog and will definitely be writing about µLauncher, I'm myself curious what will the community think about it.

I could add a sentence about that to the description on the app stores.

One thing you could add anyway, and not just to the app stores but also the readme, is accessibility being a feature. This not just makes the project look professional, but for blind people, it's a very valuable information. I think quite a few would be interested what alternative launchers are available to them, but with such complex and visual programs like launchers, a11y can often be hit or miss. You can't just look through screenshot and pick what you like the most, you need to sit down, install all the actively developed options and then go through them one by one. But not everyone has enough time resources for doing a detailed mass-analysis, so if a project explicitly mentions it's accessible, it can speed things up a lot.

Another thing that may be worth adding is some documentation on how to best use the launcher with TalkBack. I'm not sure how many people realize the screenreader now allows passing gestures to apps, I don't remember this being communicated anywhere. It used to be impossible, but now it works and it's really useful, exactly for situations like this.

Also most people do know that swiping up / down with two fingers helps with scrolling, but I'm again not sure how many realize how does this work in the background, that it's actually another form of touch pass and you can even do curved gestures like > or < with it, which is weird but quite cool. 😃

[jrpie]

Thank you for the detailed response! Yes, adding a note that µLauncher aims to be accessible sounds good.
I guess that oppinion doesn't differ too much from that of sighted people: Some really like µLauncher's approach, but for most people the speedup does not justify learning 35 gesture bindings. µLauncher definitely is oppinionated software for a somewhat niche audience.

Could you send me a link if you write that blog post? I'd be interested about responses, too. Also I'd like to add a link to the documentation, I guess it would be nice to have some ressource for how to set it best to work with a screen reader. (In my talk back config, gesture passthrough was indeed not enabled by default, making µLauncher unusable).