This is one of those “N technical things every programmer must read” lists, except that “programmer” is way too broad a term and the styles of writing people find helpful for them are too different for any such list to contain a non-zero number of items (if you want the entire list to be helpful to everyone). So here’s a list of some things you might want to read, and why you might (or might not) want to read them.
This post on why making a competitor to Google search is a post in classic Alex Clemmer style. The post looks at a position that’s commonly believed (web search isn’t all that hard and someone should come up with a better Google) and explains why that’s not an obviously correct position. That’s also a common theme of his comments elsewhere, such as these comments on, stack ranking at MS, implementing POSIX on Windows, the size of the Windows codebase, Bond, and Bing.
He’s sort of a modern mini-MSFT, in that it’s incisive commentary on MS and MS related ventures.
Explorations of various areas, often Python related, such as this this series on the Python interpreter and this series on the CPython peephole optimizer. Also, thoughts on broader topics like debugging and learning.
Often detailed, with inline code that’s meant to be read and understood (with the help of exposition that’s generally quite clear).
Computer related projects, by which I mean things like reconstructing the Cray-1A and building mechanical computers. Rarely updated, presumably due to the amount of work that goes into the creations, but almost always interesting.
The blog posts tend to be high-level, more like pitch decks than design docs, but there’s often source code available if you want more detail.
This is a quarterly publication from RC. Posts vary from floating point implementations in various languages to how git works to image processing.
I wonder why web publications like this don’t get more press. There’s been a bit of a revival lately, and we’ve seen plenty of high quality publications, from high-profile efforts like The Macro to unpublicized gems like Snowsuit, but you don’t really see people talking about these much. Or I don’t, anyway.
A lot of great material on how engineering companies should be run. He has a lot of ideas that sound like common sense, e.g., choose boring technology, until you realize that it’s actually uncommon to find opinions that are so sensible.
Mostly distilled wisdom (as opposed to, say, detailed explanations of code).
A mix of things from writing a 64-bit kernel from scratch shortly after learning assembly to a high-level overview of computer systems. Rarely updated, with few posts, but each post has a lot to think about.
I think of this as “the C++ blog”, but it’s much wider ranging that that. It’s too wide ranging for me to sum up, but if I had to commit to a description I might say that it’s a collection of deep dives into various topics, often (but not always) relatively low-level, along with short blurbs about books, often (but not always) technical.
The book reviews tend to be easy reading, but the programming blog posts are often a mix of code and exposition that really demands your attention; usually not a light read.
A wide variety of bite-sized technical tidbits, from how integer division behavior varies by language to data corruption that isn’t corrected by Ethernet or TCP checksums. Usually bite-sized and easily read.
Low-level. A good example of a relatively high-level post from this blog is this post on the low fragmentation heap in Windows. Posts like how to hack a pinball machine and how to design a 386 compatible dev board are typical.
Posts are often quite detailed, with schematic/circuit diagrams. This is relatively heavy reading and I try to have pen and paper handy when I’m reading this blog.
Another “not exactly a blog”, but it’s more informative than most blogs, not to mention more entertaining. This is the best “blog” on the pervasive brokenness of modern software that I know of.
Write-ups of papers that (should) have an impact on how people write software, like this paper on what causes failures in distributed systems or this paper on what makes people feel productive. Not updated much, but Greg still blogs on his personal site.
The posts tend to be extended abstracts that tease you into reading the paper, rather than detailed explanations of the methodology and results.
Explanations of how Linux works, as well as other low-level topics. This particular blog seems to be on hiatus, but “0xAX” seems to have picked up the slack with the linux-insides project.
If you’ve read Love’s book on Linux, Duarte’s explanations are similar, but tend to be more about the idea and less about the implementation. They’re also heavier on providing diagrams and context. “0xAX” is a lot more focused on walking through the code than either Love or Duarte.
Jessica is probably better known for her talks than her blog? Her talks are great! My favorite is probably this talk with explains different concurrency models in an easy to understand way, but the blog also has a lot of material I like.
As is the case with her talks, the diagrams often take a concept and clarify it, making something that wasn’t obvious seem very obvious in retrospect.
I think of this as the “C is harder than you think, even if you think C is really hard” blog, although the blog actually covers a lot more than that. Some commonly covered topics are fuzzing, compiler optimization, and testing in general.
Posts tend to be conceptual. There’s often code as examples, but the code tends to be light and easy to read, making Regehr’s blog a relatively smooth and easy read even though it covers a lot of important ideas.
A lot of posts about networking, generally written so that they make sense even with minimal networking background. I wish more people with this kind of knowledge (in depth knowledge of systems, not just networking knowledge in particular) would write up explanations for a general audience. Also has interesting non-networking content, like this post on Finnish elections.
AFAICT, the theme is “things Julia has learned recently”, which can be anything from Huffman coding to how to be happy when working in a remote job. When the posts are on a topic I don’t already know, I learn something new. When they’re on a topic I know, they remind me that the topic is exciting and contains a lot of wonder and mystery.
Many posts have more questions than answers, and are more of a live-blogged exploration of a topic than an explanation of the topic.
Technical explorations of various topics, with a systems-y bent. Kubernetes. Git push. Syscalls in Rust. Also, some musings on programming in general.
The technical explorations often get into enough nitty gritty detail that this is something you probably want to sit down to read, as opposed to skim on your phone.
90% of Kyle’s posts are explanations of distributed systems testing, which expose bugs in real systems that most of us rely on. The other 10% are musings on programming that are as rigorous as Kyle’s posts on distributed systems. Possibly the most educational programming blog of all time.
For those of us without a distributed systems background, understanding posts often requires a bit of Googling, despite the extensive explanations in the posts.
A mix of theory and wisdom from a distributed systems engineer on EBS at Amazon. The theory posts tend to be relatively short and easy to swallow; not at all intimidating, as theory sometimes is.
This used to be a blog about random experiments Marek was doing, like this post on bitsliced SipHash. Since Marek joined Cloudflare, this has turned into a list of things Marek has learned while working in Cloudflare’s networking stack, like this story about debugging slow downloads.
Posts tend to be relatively short, but with enough technical specifics that they’re not light reads.
Lengthy and very-detailed explanations of technical topics, mixed in with a wide variety of other posts.
The selection of topics is eclectic, and explained at a level of detail such that you’ll come away with a solid understanding of the topic. The explanations are usually fine grained enough that it’s hard to miss what’s going on, even if you’re a beginner programmer.
More than you ever wanted to know about writing fast code for the JVM, from GV affects data structures to the subtleties of volatile reads.
Posts tend to involve lots of Java code, but the takeaways are often language agnostic.
Adventures in signal processing. Everything from deblurring barcodes to figuring out what those signals from helicopters mean. If I’d known that signals and systems could be this interesting, I would have paid more attention in class.
Some content on Lisp, and some on low-level optimizations, with a trend towards low-level optimizations.
Posts are usually relatively long and self-contained explanations of technical ideas with very little fluff.
Years of debugging stories from a long-time SRE, along with stories about big company nonsense. The stories often have details that make them sound like they come from Google, but anyone who’s worked at Microsoft, IBM, Oracle, or another large company will find them familiar.
This reminds me a bit of Google’s SRE book, except that the content is ordered chronologically instead of by topic, and it’s conveyed through personal stories rather than impersonal case studies.
Homemade electronics projects from vim on a mechanical typewriter to building an electrobalance to proof spirits.
Posts tend to have a fair bit of detail, down to diagrams explaining parts of circuits, but the posts aren’t as detailed as specs. But there are usually links to resources that will teach you enough to reproduce the project, if you want.
This is one of the few programming blogs where I regularly go back and re-read posts from the archive. I learn something new every time. Posts span the entire stack, from how individual programmers can improve at programming to how orgs can improve at recruiting. I re-read that last post before posting the link here and this bit jumped out:
Well, in case you hadn’t noticed, they’re kicking our butts at recruiting. Even in our own backyard. Professor Ed Lazowska at the University of Washington told us last year that Google’s getting about 3 times as many UW hires as we are. A candidate at last week’s recruiting trip told me that of the nine or ten students he considered to be the best programmers at the UW, about half of them went to Google; only two went to Amazon, and the rest went to “no-name” places.
Actually, his story had one more interesting tidbit: he said that although Microsoft is considered one of the top three places to work by the UW CS students (along with Google and Amazon), he claims that Microsoft is hiring lots of mediocre programmers. He said they gave offers to a whole bunch of programmers who he knows aren’t any good — and this guy was my strongest interviewee of the trip, so I was inclined to trust his judgment. He said that in his eyes, this disqualified Microsoft as a potential employer.
That’s not to say we don’t lose candidates to Microsoft. We do! Microsoft has determined that Amazon is very good at talent assessment, but crappy at selling the candidates and clinching the deal. So when Microsoft hears from a candidate that they’ve got a full-time offer from us, Microsoft doesn’t even interview the person. They take the candidate for a ride in the company hummer, have execs wine and dine them, let them spend the day with the team they’re going to join, show them the private office with a door they’ll get so they can concentrate on innovation… it’s a straight sell job after we’ve made an offer.
This is exactly what happened to me with Microsoft – I had a number of offers from other companies (though not Amazon), and someone from Microsoft called me up and sold me on Microsoft. I technically had an interview, but the interview was basically a sales job. Basically every time I re-read a Steve Yegge post, I notice that the post reflects some recent experience of mine.
A mix of technical posts about security and BSD and commentary on how broken software is. Some examples of the latter are this post on automation failure and this post on how Netflix handles CDs.
Even when there’s code, the posts tend to be about a high-level idea that just happens to be illustrated by the code, which makes this a lighter read than you’d expect from sheer amount of code.
As far as I know, Rebecca doesn’t have a programming blog, but if you look at her apparently off-the-cuff comments on other people’s posts as a blog, it’s one of the best written programming blogs out there. She used to be prolific on Piaw’s Buzz (and probably elsewhere, although I don’t know where), and you occasionally see comments elsewhere, like on this Steve Yegge blog post about brilliant engineers1. I wish I could write like that.
This isn’t updated anymore, but I find the archives to be fun reading for insight into what people were thinking about microprocessors and computer architecture over the past two decades. It can be a bit depressing to see that the same benchmarking controversies we had 15 years ago are being repeated today, sometimes with the same players. If anything, I’d say that the average benchmark you see passed around today is worse than what you would have seen 15 years ago, even though the industry as a whole has learned a lot about benchmarking since then.
In-depth explanations on how V8 works and how various constructs get optimized by a compiler dev on the V8 team. If I knew compilers were this interesting, I would have taken a compilers class back when I was in college.
Often takes topics that are considered hard and explains them in a way that makes them seem easy. Lots of diagrams, where appropriate, and detailed exposition on all the tricky bits.
Mostly dormant since the author started doing art, but the archives have a lot of great content about hardware, low-level software, and general programming-related topics that aren’t strictly programming.
90% of the time, when I get the desire to write a post about a common misconception software folks have about hardware, Yossi has already written the post and taken a lot of flak for it so I don’t have to :-).
I also really like Yossi’s career advice, like this response to Patrick McKenzie and this post on how managers get what they want and not what they ask for.
Common themes include:
I still sort of can’t believe that anyone reads my writing on purpose. If I had to think of one flattering thing to about my blog, it would be that even though my blog posts are often substantially longer than Steve Yegge’s, I have literally not seen a single person complain about the length in internet comments. I expect that’s a self-defeating prophecy, though :-).
Note that this list is relatively tilted towards blogs I find to be underrated. So it doesn’t include, for example, the high scalability blog, mechanical sympathy, or Patrick McKenzie even though I think they’re great. In that case, you might say that it’s strange that I have folks like Steve Yegge and Kyle Kingsbury listed. What I can say? I still consider them underrated. This list also doesn’t include blogs that mostly aren’t about programming, so it doesn’t include, for example, Ben Kuhn’s excellent blog.
Anyway, that’s all for now, but this list is pretty much off the top of my head, so I’ll add more as more blogs come to mind. I’ll also keep this list updated with what I’m reading as I find new blogs. Please please please suggest other blogs I might like, and don’t assume that I already know about a blog because it’s popular. Just for example, I had no idea who either Jeff Atwood or Zed Shaw were until a few years ago, and were and still are probably two of the most well known programming bloggers in existence. Even with centralized link aggregators like HN and reddit, blog discovery has become haphazard and random with the decline of blogrolls and blogging as a dialogue rather than a monologue. Also, please don’t assume that I don’t want to read something just because it’s different from the kind of blog I normally read. I’d love to read more from UX or front-end folks; I just don’t know where to find that kind of thing!
This post was inspired by the two posts Julia Evans has on blogs she reads and by the Chicago undergraduate mathematics bibliography, which I’ve found to be the most useful set of book reviews I’ve ever encountered.
Thanks to Bartłomiej Filipek and Sean Barrett for suggestions on what to add to the list. I haven’t had time to write them up, but I’ll probably add https://fgiesen.wordpress.com/, http://fabiensanglard.net/, http://preshing.com/, http://huonw.github.io/, and https://randomascii.wordpress.com/, among others. Also, thanks to Lindsey Kuper for discussion/corrections.
Quote follows below, since I can see from my analytics data that relatively few people click any individual link, and people seem especially unlikely to click a link to read a comment on a blog, even if the comment is great:
The key here is “principally,” and that I am describing motivation, not self-evaluation. The question is, what’s driving you? What gets you working? If its just trying to show that you’re good, then you won’t be. It has to be something else too, or it won’t get you through the concentrated decade of training it takes to get to that level.
Look at the history of the person we’re all presuming Steve Yegge is talking about. He graduated (with honors) in 1990 and started at Google in 1999. So he worked a long time before he got to the level of Google’s star. When I was at Google I hung out on Sunday afternoons with a similar superstar. Nobody else was reliably there on Sunday; but he always was, so I could count on having someone to talk to. On some Sundays he came to work even when he had unquestionably legitimate reasons for not feeling well, but he still came to work. Why didn’t he go home like any normal person would? It wasn’t that he was trying to prove himself; he’d done that long ago. What was driving him?
The only way I can describe it is one word: fury. What was he doing every Sunday? He was reviewing various APIs that were being proposed as standards by more junior programmers, and he was always finding things wrong with them. What he would talk about, or rather, rage about, on these Sunday afternoons was always about some idiocy or another that someone was trying make standard, and what was wrong with it, how it had to be fixed up, etc, etc. He was always in a high dudgeon over it all.
What made him come to work when he was feeling sick and dizzy and nobody, not even Larry and Sergey with their legendary impatience, not even them, I mean nobody would have thought less of him if he had just gone home & gone to sleep? He seemed to be driven, not by ambition, but by fear that if he stopped paying attention, something idiotically wrong (in his eyes) might get past him, and become the standard, and that was just unbearable, the thought made him so incoherently angry at the sheer wrongness of it, that he had to stay awake and prevent it from happening no matter how legitimately bad he was feeling at the time.
It made me think of Paul Graham’s comment: “What do I mean by good people? One of the best tricks I learned during our startup was a rule for deciding who to hire. Could you describe the person as an animal?… I mean someone who takes their work a little too seriously; someone who does what they do so well that they pass right through professional and cross over into obsessive.
What it means specifically depends on the job: a salesperson who just won’t take no for an answer; a hacker who will stay up till 4:00 AM rather than go to bed leaving code with a bug in it; a PR person who will cold-call New York Times reporters on their cell phones; a graphic designer who feels physical pain when something is two millimeters out of place.”
I think a corollary of this characterization is that if you really want to be “an animal,” what you have cultivate in yourself is partly ambition, but it is partly also self-knowledge. As Paul Graham says, there are different kinds of animals. The obsessive graphic designer might be unconcerned about an API that is less than it could be, while the programming superstar might pass by, or create, a terrible graphic design without the slightest twinge of misgiving.
Therefore, key question is: are you working on the thing you care about most? If its wrong, is it unbearable to you? Nothing but deep seated fury will propel you to the level of a superstar. Getting there hurts too much; mere desire to be good is not enough. If its not in you, its not in you. You have to be propelled by elemental wrath. Nothing less will do.
Or it might be in you, but just not in this domain. You have to find what you care about, and not just what you care about, but what you care about violently: you can’t fake it.
(Also, if you do have it in you, you still have to choose your boss carefully. No matter how good you are, it may not be trivial to find someone you can work for. There’s more to say here; but I’ll have to leave it for another comment.)
Another clarification of my assertion “if you’re wondering if you’re good, then you’re not” should perhaps be said “if you need reassurance from someone else that you’re good, then you’re not.” One characteristic of these “animals” is that they are such obsessive perfectionists that their own internal standards so far outstrip anything that anyone else could hold them to, that no ordinary person (i.e. ordinary boss) can evaluate them. As Steve Yegge said, they don’t go for interviews. They do evaluate each other – at Google the superstars all reviewed each other’s code, reportedly brutally – but I don’t think they cared about the judgments of anyone who wasn’t in their circle or at their level.
I agree with Steve Yegge’s assertion that there are an enormously important (small) group of people who are just on another level, and ordinary smart hardworking people just aren’t the same. Here’s another way to explain why there should be a quantum jump – perhaps I’ve been using this discussion to build up this idea: its the difference between people who are still trying to do well on a test administered by someone else, and the people who have found in themselves the ability to grade their own test, more carefully, with more obsessive perfectionism, than anyone else could possibly impose on them.
School, for all it teaches, may have one bad lasting effect on people: it gives them the idea that good people get A’s on tests, and better ones get A+’s on tests, and the very best get A++’s. Then you get the idea that you go out into the real world, and your boss is kind of super-professor, who takes over the grading of the test. Joel Spolsky is accepting that role, being boss as super-professor, grading his employees tests for them, telling them whether they are good.
But the problem is that in the real world, the very most valuable, most effective people aren’t the ones who are trying to get A+++’s on the test you give them. The very best people are the ones who can make up their own test with harder problems on it than you could ever think of, and you’d have to have studied for the same ten years they have to be able even to know how to grade their answers.
That’s a problem, incidentally, with the idea of a meritocracy. School gives you an idea of a ladder of merit that reaches to the top. But it can’t reach all the way to the top, because someone has to measure the rungs. At the top you’re not just being judged on how high you are on the ladder. You’re also being judged on your ability to “grade your own test”; that is to say, your trustworthiness. People start asking whether you will enforce your own standards even if no one is imposing them on you. They have to! because at the top people get given jobs with the kind of responsibility where no one can possibly correct you if you screw up. I’m giving you an image of someone who is working himself sick, literally, trying grade everyone else’s work. In the end there is only so much he can do, and he does want to go home and go to bed sometimes. That means he wants people under him who are not merely good, but can be trusted not to need to be graded. Somebody has to watch the watchers, and in the end, the watchers have to watch themselves.