Large Scale Benchmarking

Why Large Scale Benchmarking?

Results from single web pages and web sites are not enough for larger comparison of eGovernment services between political regions.

To compare results of eGovernment policies between countries or between individual regions within each country, results from a set of web sites are needed. Such results allows us to identify both good practices and fuel promotion of desirable qualities. Regions with most potential for improvement can learn from the best practices and learn for example what makes the web sites in their neighboring country more accessible then theirs, why do their score lower on transparency compared to the other municipalities in their country and so on.

Additionally, when benchmarking is repeated at regular intervals, it allows detection of improvement. Information is available, such as if new policies or technologies have had the anticipated effect. As an example, if the public web sites in country has become more accessible with introduction of new accessibility policies.

What is here today

Several studies have been carried out.

For accessibility studies such as "Measuring Progress of eAccessibility in Europe" (MeAC), the UK Cabinet Office have received a lot of attention.
These studies are carried out by human experts and can never be completely unbiased. Humans may (subconsciously) favor some web sites or particular types of web sites, which clearly could influence the results.

Manual studies are hard to repeat for two main reasons:

• the evaluation itself is costly and
• there is no guarantee that two humans evaluatuting the same page reaches the same result as manual tests often rely on human judgement. As an example, it is considered good accessibility to have a text clear and consistant. However, what is clear and consistant may be highly individual.

Furthermore, those studies are carried out as one-off studies meaning that there are no results over time available and there is no data available on how web accessibility has evolved over time.

Additional studies on eGovernment such as the "UN e-Government Survey 2008 – From e-Government to Connected Governance" exist.

What will eGovMon bring?

eGovMon consists of a tool for running automated tests at regular intervals. In the first project phase data on the accessibility of Norwegian municipality web sites is collected.

Research challenges

Several research challenges exists such as:

• Which set of web sites represents the region or country.
• Which web pages represent the web site.
• How can the evaluation be carried out both with good performance and statistically sound.
• How can the indicators which are normally evaluated manually be evaluated automatically.
• How can the results be stored and presented efficiently.

Presentation of results

The presentation of the huge amount of data derived with the monitor is subject to further research. Such interface is crucial to support the take-up of the results from the project and to allow for collaborative improvements of the reports and of the indicators prepared.

A central challenge is how to present results from all of the four different indicators in a way that enables the researchers, policy makers, and practitioners to gain a deeper understanding needed for practical web site improvements and for policy decisions, and to facilitate their interactions.