WPCampus and Tenon, supported by many partners and community members, published “Gutenberg Accessibility Audit Report”.
In late 2018, WPCampus released a request for proposals to conduct an accessibility audit of the WordPress block editor, also known as Gutenberg. In early 2019, they have announced our selection of Tenon, LLC to conduct the audit, at a cost of $31,200. Now Tenon’s report on the accessibility of the new editor is published.
According to the official release, Tenon’s technical audit methodology is derived from the collective experiences of Tenon’s consulting staff who, all together, have performed nearly 300 accessibility audits of websites,
software, and hardware.
This report was made possible by the extraordinary generosity of more than 100 members of the WordPress community, who collectively contributed $10,264 to fund this project, WPCampus stated.
In an official release, WPCampus highlighted the role of Matt Mullenweg and Automattic, saying they have pledged to ensure that the audit would be fully funded and are covering the balance of the cost of this effort.
“In addition, this audit would not be possible without the members of our vendor selection committee. These amazing and talented people donated their time and energy to review proposals, ask smart questions, and ensure that the report we’re sharing today would be valuable to our community”, WPCampus published on the official web site.
On December 6, 2018, version 5.0 of WordPress was released which includes the new Gutenberg editor. However, work on Gutenberg began in Spring of 2017 with accessibility related issues being logged in the Gutenberg project’s GitHub repo shortly thereafter. In March 2018, the WordPress accessibility team began testing Gutenberg in earnest and logging issues they discovered during testing.
In November 2018, WPCampus issued an RFP to find a provider to do an accessibility audit on Gutenberg.
Specifically, the WPCampus community sought to identify:
▪ Potential legal risk (if any) posed by upgrading to Gutenberg
▪ Potential challenges when supporting the new editor, particularly for assistive technology users
▪ Potential impact of Gutenberg on the accessibility of public-facing web content
On December 20, 2018 Tenon’s proposal was accepted. The proposal included a technical audit of the Gutenberg experience as well as user-based testing with persons with disabilities. Testing began on January 15, 2019 and concluded on March 1, 2019. This Executive Summary document provides a description of Tenon’s audit of Gutenberg, our methodology, key findings, and a summary of the data uncovered during testing.
Key Findings
During usability testing of Gutenberg, participants only had an average completion rate of 63% across three tasks, with an average System Usability Scale (SUS) score of 46. The scores within the UX Report indicate that Gutenberg can be described, at best, as “Poor” to “OK”. The usability test results are backed by the data uncovered during technical review.
During the technical review, Tenon staff logged a total of 90 issues across 16 components tested. As the data in this report will show, there is an average of 18% failed check items per component, with 63% of issues impacting at least 3 user populations. 69% of issues are Medium or High severity. Finally, 53% of the issues logged in the Technical Report are for Level A Success Criteria.
On the ability to generate accessible markup with Gutenberg
For the most part, the markup generated from the Gutenberg editor is clean, semantically correct and accessible. It’s clear that a lot of attention has been paid to things like using proper headings and lists in the output, which makes it easy for users to create accessible content without having to know HTML.
However, it’s still relatively easy for users to unknowingly create accessibility problems, for example, by using the wrong heading levels within a post, creating multiple posts with the same title, or adding videos which autoplay. The editor could do more to warn users when they do this kind of thing.
Some specific limitations in the visual interface also make it difficult for certain kinds of content to be fully accessible, because it doesn’t provide the tools to define all necessary semantics and metadata. The Video Block interface, for example, doesn’t provide a way to associate a captions file with the video; the Table Block interface doesn’t allow users to define a caption, header rows, header cells, or scopes. Gutenberg users can only add this information by using the HTML editor, but this, of course, requires them to have sufficient HTML knowledge.
For more details, check out the official WPCampus site.