This blog describes the dataset provided by Burning Glass which was used to create the Skills Chart shown above.

The data

Burning Glass collects job advertisements from over 40,000 websites daily.[1] Each advert is assigned an occupation code, using the Standard Occupational Classification (SOC) system. And any skills and software programs mentioned in the advert are extracted. The term ‘skill’ is used here in its broadest sense. Some skills are requests for experience in particular industries; other skills refer to specialised knowledge; and finally some skills describe an area of work. Burning Glass classify each skill as either 'baseline' or 'specialised'. 

The Skills Chart above is based on adverts collected during the first four months of 2015. There is an average of 5 skills or software programs in each advert and a total of over 8,000 unique skills and programs.

The benefits

The scraped data has a number of benefits over existing survey data. First and foremost, it can provide a timelier indicator of skills demanded by UK employers. The Skills Chart is based on data from early 2015. However, the chart could be based on the very latest data and updated regularly. The official statistics relating to skills come from the UK Commission’s Employer Skills Survey (ESS) but the survey is only conducted every two years.

The skills extracted from job adverts also provide a much richer picture of skill demands. The Employer Skills Survey only asks employers about a small number of broad skill groups. By contrast, the data from job postings can be used to measure the demand for thousands of different skills. Moreover, the skills are extracted directly from adverts, rather than being classified into pre-defined and potentially ill-suited categories. The sheer number of adverts also allows us to identify the typical skills demanded for single occupation group or a single industry.

The top 30 chart also sheds new light on the relative popularity of various software programs. Understanding which programs are rising or falling in popularity may be helpful to both businesses (who are considering which programs to purchase) and workers (who are looking to upskill).

The limitations

As with any novel approach, the data is not without limitations and these should be borne in mind when viewing the Top 30 Skills Chart.

The chart shows the skills demanded by employers, but does not show skill shortages. The latter may be arguably of greater use to policymakers but would require gathering information on the skills of applicants. While the ESS does ask employers about skills shortages, this is only in relation to a small number of broad skill groups which also limits its usefulness.

The dataset includes thousands of skills, and as a result, some skills are not independent of others. This means that apparent changes in a skill’s demand may simply be due to changes in specificity. For example, the set of skills includes both management and several specific types of management, such as project management and sales management. A decline in management may not mean that employers have reduced their demand for this skill, but rather that employers now prefer to request specific forms of management. This limitation could be overcome by aggregating certain skills together.  

Finally, the Skills Chart and underlying dataset only capture skills for jobs that are advertised online. Some jobs be advertised through other channels or not advertised at all. For example, in creative occupations a piece of work may be commissioned directly from an artist. That said, Burning Glass estimate that the adverts they collected in 2014 equated to approximately 85% of all vacancies measured by the ONS in the same year, where the latter was based on a survey.

The potential

Despite the noted limitations, the Skills Chart and underlying dataset could act as a useful complement to the Employer Skills Survey. They could provide a more detailed and timely picture of skills demanded by UK employers.

There are several other applications of the dataset that could be useful for UK policymakers. For example, the skills and salary data could be combined to derive the marginal value of each skill. These values could be tracked overtime, in the same way that prices of goods are tracked in the Consumer Price Index. The values could provide an alternative measure of skill shortages.

The data could also be used to identify new occupations types and industries by tracking the emergence of new skills, or new combinations of skills. Finally, the data on software programs could be used to measure the tech intensity of both specific industries and the UK economy as a whole. 

[1] Duplicate postings are deleted.