tl;dr:Biggest challenge was organizational - meeting demands from various stakeholders. This was tackled by creating a cross-departmental working group. Articles outlines how deferring JS had significant impact on performance and tools used to monitor it.
Click the link in this tweet if paywalled.