tl;dr:“I thought I’d share some of the more “surprising” lessons after churning through just north of 500 million tokens, by my estimate.” Lessons include: (1) When it comes to prompts, less is more. (2) You don’t need langchain. You probably don’t even need anything else OpenAI has released in their API in the last year. (3) Improving the latency with streaming API and showing users variable-speed typed words is actually a big UX innovation with ChatGPT.
tl;dr:Ken finds it hard to gauge experience when hiring and has developed 3 strategies to help: (1) Case Studies: Prepare a 1-2 page story that lays out, a particular technical scenario in deliberately broad brushstrokes, and then ask the candidate to figure out what they’d do. (2) Three Why’s Technique: Practice of asking someone to describe something, and then pressing them three more times for more details. (3) Ask them to Break the Rules: A more specific instantiation of a famous interview question “What important truth do very few people agree with you on?”
tl;dr:(1) You don’t need hundreds of engineers to build a great product. (2) Simple Outperformed Smart. (3) Our highest impact findings would always come within the first and last few hours of the audit. (4) Writing secure software has gotten remarkably easier in the last 10 years. (5) All the really bad security vulnerabilities were obvious. And more.