Code Unknown - The Problem with CPV
Over the last few weeks I've been developing some code to automatically extract data from the EU's Tender Electronic Daily (ted.europa.eu) in order to provide analysis of how architectural services are procured across the European Union.
Whilst at first glance the TED website might seem relatively easy to navigate and intuitive (compared to some notable examples), once you try to extract more precise information, it starts to fall apart.
The ability to set up custom searches and publish these as RSS feeds is extremely useful - although the RSS feeds appear to be limited to 30 items - the amount of information that these feeds contain is very limited.
For example, a typical tender notice published as an RSS feed will appear as follows:
188452-2013: UK-Liverpool: Architectural design services
Publication date (PD): 08-06-2013 | Deadline (DT): 08-07-2013 | Document (TD): Contract notice
However, the full notice contains a substantial amount of additional information in a standardised format. The "Data" tab of the page to which this notice leads includes the following:
Title: UK-Liverpool: Architectural design services Document number: 188452-2013 Publication date: 08/06/2013 OJ S: 110 Place: LIVERPOOL Authority name: University of Liverpool Construction Company (ULCCO) Original language: EN Heading: Member states - Service contract - Contract notice - Restricted procedure Country: UK Type of authority: 8 - Other Document sent: 06/06/2013 Deadline: 08/07/2013 Contract: 4 - Service contract Procedure: 2 - Restricted procedure Document: 3 - Contract notice Regulation: Z - Not specified Type of bid: Z - Not specified Award criteria: 2 - The most economic tender CPV code: 71220000 - Architectural design services Original CPV code: 71220000 - Architectural design services NUTS code: UKD52 Directive: Classical Directive (2004/18/EC)
Which is a far more useful set of data. It would be far more transparent if the above could be included within the <description> tag of the RSS feed.
One other problem that this exercise has highlighted is the seemingly random application of CPV codes (Common Procurement Vocabulary). It appears that these are being applied to notices in different ways.
Let's take two examples from two different notices with what would seem to be similar titles:
185014-2013: UK-Ipswich: Architectural and related services
Publication date (PD): 06-06-2013 | Deadline (DT): 04-07-2013 | Document (TD): Contract notice
CPV Code: 71200000 - Architectural and related services, 71220000 - Architectural design services
...and...
186741-2013: UK-St Albans: Architectural, construction, engineering and inspection services
Publication date (PD): 07-06-2013 | Deadline (DT): 10-07-2013 | Document (TD): Contract notice
CPV Code: 71000000 - Architectural, construction, engineering and inspection services
The first notice has correctly identified the subcategory of service they are intending to procure, the second has simply lumped the whole thing under the primary category, making it very difficult to locate (particularly as the 71000000 code includes everything from "Laboratory services" to "Blacksmith services").
A search for all tenders in the five most recent editions using the 71000000 code shows a total of 43 notices (everything from "Health and social work services" to "Road bridge construction work", but including a number of "Architectural design services"), whereas the same search reveals just 11 notices using the 71200000 code.
The trouble is that searching on the basis of CPV codes does not reveal subcategories of those master categories. So, searching for tenders within the 71000000 category does not show those contained within the 71200000 subcategory.
A further problem is the incorrect assignment of codes. Searching for tenders using the 71200000 code shows the following notices:
186427-2013: UK-Liverpool: IT services: consulting, software development, Internet and support
which has 183 CPV codes assigned to it; so many to make the categorisation useless.