Neil Crofts
BBC
Neil Crofts is a senior software engineer at the BBC in the Audience Platform Data team responsible for processing the billions of messages audiences create when interacting with BBC online services.
Previously he worked on the BBC Red Button Digital Text service and prior to that has developed software in a number of other industries including semiconductors, automotive (commercial and F1) and telecoms.
Talk: 7 serverless recipes for big data architectures
The BBC Audience Platform Data team collects, transforms and delivers billions of events each day from audience interactions with mobile apps and web sites such as BBC News, BBC Sport and iPlayer.
We will share our experience of the 7 critical steps we took to migrate from Spark clusters with Map-Reduce to a serverless architecture. We will cover the good, the bad and the ugly, including lessons learnt along the way. These steps are also applicable to new architectures.
We have been running in production for nearly a year and have achieved our goal of real-time analysis at up to 25000 transactions per second. This allows us to meet the demands of even the biggest news days, as well as seeing significant savings on our cloud infrastructure.
