Introduction
Owing to some work we have recently undertaken to analyse Welsh Government support for business innovation we have been playing around with whether or not we can assess the impact and relevance of the Welsh Well-being of Future Generations Act and ultimately developing a system for measuring awareness of the aims of the Act in the business population of Wales.
The Well-being of Future Generations Act (2015) consists of seven principles that are now embodied in law as requirement and permission to consider all legislation in terms of its impact upon future generations. The Future Generations Commissioner has responsibility for overseeing the beneficial impact of the legislation.
The seven principles:
- A prosperous Wales.
- A resilient Wales.
- A healthier Wales.
- A more equal Wales.
- A Wales of more cohesive communities.
- A Wales of vibrant culture and thriving Welsh language.
- A globally responsible Wales.
A good example of these principles in action is the way in which they inform the public procurement activities of Welsh government and other public bodies in Wales and the provision of public support for business development and innovation in Wales. Our project is to design and test a method for measuring the spread of these principles by examining the extent and degree to which the principles are referenced or embodied in the digital actions of organisations in Wales.
A particular issue with political ambitions of this scope and scale is providing evidence of impact, this is particularly difficult where some key outcomes are not necessarily quantitative in nature. Similarly, it is very difficult for government to measure awareness of such issues in a systematic way. Particularly when a key constituency is not the public but the wider business community. To help address these issues Etic Lab is developing an innovative approach based upon its business listening platform (ForesightSI) and a new tool designed to measure the visibility and distribution of the principles.
Development of the Welsh Future Generations Score
The method we employ here is based upon the use of Topic Modelling, a well understood statistical technique for identifying ‘topics’ occurring in a collection of texts. We also deploy the concept of sentence encoding to create a model used to measure the semantic similarity of large samples of unstructured text. Etic Lab created a reference set of written materials that explain and represent the values and principles underpinning the legislation drawn from the literature and other written materials describing the act and its implementation produced by Welsh Government. A normalising process, by which we were able to find co-incident topics that arise in the wild alongside the stated values of the AFGA was also possible. This repository was then used to train the model to measure texts produced by different organisations. The end product of this process is a metric, a number between 0 and 1 that reflects the degree of similarly between the material produced by an individual organisation and the content of the reference WFGA-encoded set. In other words a measure of the extent and degree to which the content of a companies reflect or embody the span of issues represented in discussions of the legislation.
Testing the WFG Score
Etic Lab LLP maintains a structured a sample of Welsh companies that reflects the distribution of different business sectors within each Unitary Authority in Wales, which means we can provide robust, reproducible and valid analysis a variety of topics concerning the Welsh economy. The pre-existing metrics have recently been to measure Digital Maturity, Digital Growth, Innovation ESG and the distribution of the Welsh language on commercial websites across Wales.
We used our sample of Welsh businesses and the ForesightSI platform to retrieve data from company websites and process these data to create a Welsh Future Generations Score (WFG) for each company. Then we performed a series of experiments to explore the nature of the WFG metric and its possible use cases.
Experiment One: Distribution of the WFG metric
The first experiment was designed to test whether or not we were measuring something that actually occurs in that (some) companies do make reference to the principles and concepts contained in the public writings of Welsh Government. A score of 0 would suggest the organisation under examination exhibits no support for WFGA values while a score of one 1 suggests a total alignment with the values of the legislation. To do this we took the WSG scores for our national sample of 5431 companies and plotted them to determine the range of scores across the business community as a whole.
As can be seen in the chart above, when we look at our sample of Welsh companies the WFG scores appear approximately normally distributed. This shows that the metric is measuring something that appears to a degree in the digital presence of most companies and that different companies have different scores. Overall a score of between 0.3 and 0.4 is by far the most common with more companies having a score lower than this rather than higher.
This means the metric is useful because, as can sometimes be the case ,the scores are not concentrated at either end of the distribution; that is almost no companies score very highly or very low. In which event we would need to refine the model or accept that the concepts have no currency in analysing Welsh businesses. What we have found is that concepts are widely used and we can look to use the metric to discriminate between different groups of companies for a variety of purposes.
Experiment Two: Distribution of the WFG metric by business sector
Given that different companies have different scores as we have found with our other measures such as Innovation and Digital Maturity we next decided to ask the question; Do businesses in different business sectors demonstrate on aggregate, a different level of the WSG metric?
In order to answer this question we segregated our sample of companies into 7 top level standard industrial classification (SIC) groups. As can be seen from the chart below, the WFG metric does in fact distinguish between the different business sectors with respect to the ‘visibility’ of Welsh Future Generations Act values. These data reflect the very different levels of engagement with the principles set out in the 2015 act in the different business sectors. It is especially significant that the sector with the highest relative WFG score is so unevenly distributed across Wales, concentrated as it is in the south of the country and the Capital region in particular.
It would follow from these data that we can expect the population of businesses in the different Unitary Authorities of Wales would have varying WFG scores as we have found to be the case with other measures such as Innovation, Digital Maturity and of course the prevalence of the Welsh language on business websites.
Experiment Three: Distribution of WFG metric by Unitary Authority
Given that each Unitary Authority in Wales has a distinctive portfolio of companies reflecting the diversity of their local economy it is likely that this will be reflected in the aggregate score on our new metric at the Unitary Authority level.
In fact, as can be seen from the graph, the average WFG score does vary between the different unitary authorities and unsurprisingly it is companies located in the capital city that tend to have the highest scores on the metric. Although the differences in WFG appear small, given the quality of the sample (its size and structure), the percentage difference between locations is significant especially with respect to those authorities at different ends of this distribution. Perhaps the most economical interpretation of this effect is that at this stage in the lifetime of the legislation, the greatest impact is to be found in companies that have had the most contact with the public sector.
In order to test the hypothesis that the WFG scores would be highest in companies that have been in relatively direct contact with Welsh government we carried out one final experiment.
Experiment Four: Welsh Government Support
For this study we compared the scores using our new metric drawn from a sample of companies that had engaged with a Welsh Government-funded business support programme with the scores from our national sample of companies. It is worth noting that we are probably underestimating the effect of companies interacting with Welsh government upon WFG scores using the approach.
First, because the companies that engaged with the business support programme varied widely in terms of the amount of time, money and effort that the engagement involved.
Secondly because our national sample will doubtless include some companies that have also interacted with Welsh Government. Nevertheless it is reasonable to look for the impact of support programmes funded by Welsh Government to have had an impact reflected in our new metric.
The graph below shows that the WFG scores for our sample of 340 companies that had taken part in a Welsh Government funded innovation support programme are shifted to the right in comparison with the national sample. Companies in direct contact with the government therefore seem to be more likely than the general population of companies in Wales to have score or 0.4 or higher. These results should be considered as consistent with the earlier experiments and strongly suggestive that contact with Welsh government programmes does influence scores on our new metric.
Summary: The Welsh Future Generations Act Score (WFG)
The Welsh Future Generations (WFG) Score is an experimental metric – we trained a model to recognise the extent to which a company’s communications to themselves and the public reflects the values of the Wellbeing of Future Generations Act, as defined by the Government’s publications explaining their aims and intentions with the legislation. Initial tests of the metric demonstrate that it can be used to measure WFG at scale across the business economy of Wales. The scores obtained exhibit a normal distribution and can be seen to vary systematically by business sector and geography. Finally we have produced indicative evidence that the measure is impacted when a company has direct contract with a Welsh Government funded programme for businesses.
Possible uses
- Detect changes in the national population of businesses with respect to this metric over time.
- Provide baseline data against which to examine the impact of any interventions. For example, grant aid to improve the use of Welsh language in business or innovation can be evaluated against a matched control group and measured in a particular target group.
- An automated and robust process to measure WFG and use alongside other metrics such as ESG, Welsh Language, Innovation etc… at any scale when measuring the digital presence of public services or private businesses.
