Meta Developer Programs

Despite having worked at Udacity, I never got the chance to work on education content. I consumed a lot of it, and I thought a lot about what made the good stuff good, but I never built any of it. That all changed when a previous coworker reached out. The crew that had spun off from Udacity as Aptly needed help with a program they were working on, and I had just finished up a project, so I thought I'd check it out.

It turned out they were working on Meta's professional certificate programs on Coursera, five of them, in five different stacks: Android, iOS, back end, front end, and databases. Complete career-change programs meant to take someone from zero to hireable. The content had already been created. What the program needed was more quiz questions, enough that quizzes could be randomized and actually test whether a learner understood the material rather than whether they'd seen the question before. Aptly had a few other contractors on the work, and they were all specialists, an iOS developer here, a database person there, some front-end knowledge in between. Nobody knew Android. So I did what I do best. I learned the thing nobody else knew and figured out how to apply it.

I didn't design these programs and I didn't write the lessons. What I did do was read every technical claim in five curricula with the most adversarial eye on the team, asking of each lesson what the learner should now be able to do, and how we would know.

On paper I was in over my head. Kotlin and the Android lifecycle in one track, Swift in another, then Python back ends, React front ends, and SQL and schema design. So I learned each one the way the learner would, dropped into the same lessons at the same point, figuring out what was actually going on. It turns out I'm very good at that, learning a thing alongside where a learner would, staying just far enough ahead to write the exam. Every piece of online-education work I've done since runs on that skill, and this project is where I found it. It's the same muscle I later used writing and appearing on camera for Microsoft's developer certificates.

Quiz writing sounds like the small job on a course team. But think of the quizzes you've taken. Most of them a good test-taker can fake their way through without learning a thing, because two of the wrong answers are jokes and the right one is the longest. That's a pencil-whip, and the difference between a pencil-whip and a real question is the distractors: the wrong answers. A good distractor is plausible, comes from material the course actually covered, and is wrong in the exact way a real misconception is wrong. You can't write one without understanding both the right answer and the specific ways a learner half-understands it, which means understanding the lesson, and every lesson before it.

The bar I set came out of that same seat. Too many quizzes test what was said in this video: the vocabulary, the exact phrasing. I wrote to a different standard: a good quiz should be answerable by someone with domain expertise who has never seen the content. If a working Android developer who never watched a single lesson can't ace my Android quiz, then my quiz is measuring memory, not understanding. The hard part was the videos themselves: they were small and atomic, and when you've only got a two minute video, how much meat is there to test against? Writing distractors that were plausible without reaching past the lesson took clever working. At Coursera scale the difference matters, because one lazy question gets served to everyone who ever takes the course, and many of them are betting a career change on it.