Almost three-quarters (71%) of financial services organisations are now using web scraping to gather external data to guide their decision-making. This is one of the key findings from a new Oxylabs white paper, which demonstrates how web scraping is shaping the sector.
Oxylabs surveyed 500 UK-based and 501 US-based senior data decision-makers from UK and US financial services companies, gathering insights on attitudes towards data collection in the sector.
Internal data (74%) remains the most popular source for businesses, but there are clear signs web scraping is set to overtake it in the near future. Over two-thirds of respondents (68%) plan to increase their data budgets in the coming year, while 80% believe they will focus more strongly on web scraping in the same period. Other methods, such as third-party data aggregators (used by 56%) remain an important part of the data-gathering ecosystem but are now a long way behind web scraping and internal data.
Despite web scraping’s rise, there remain some concerns among decision-makers about how to implement it in a safe, cost-effective, and compliant manner. Almost four in ten (38%) state risk and legal complications as the main factors discouraging them from using web scraping. Alongside this, 36% are concerned about budget constraints or the need to introduce complicated new technology, and 34% believe they lack the technical know-how to make web scraping a success.