Evergreen Notes
Evergreen Notes
Evergreen notes
Evergreen notes are written and organized to evolve, contribute, and accumulate over time, across projects. This is an unusual way to think about writing notes: Most people take only transient notes. That’s because these practices aren’t about writing notes; they’re about effectively developing insight: “Better note-taking” misses the point; what matters is “better thinking”. When done well, these notes can be quite valuable: Evergreen note-writing as fundamental unit of knowledge work.
It’s hard to write notes that are worth developing over time. These principles help:
- Evergreen notes should be atomic
- Evergreen notes should be concept-oriented
- Evergreen notes should be densely linked
- Prefer associative ontologies to hierarchical taxonomies
- Write notes for yourself by default, disregarding audience
This concept is of course enormously indebted to the notion of a Zettelkasten. See Similarities and differences between evergreen note-writing and Zettelkasten.
Implementing an evergreen note practice
See:
- Write about what you read to internalize texts deeply
- Taxonomy of note types
- A reading inbox to capture possibly-useful references
- A writing inbox for transient and incomplete notes
- Executable strategy for writing
References
Many students and academic writers think like the early ship owners when it comes to note-taking. They handle their ideas and findings in the way it makes immediate sense: If they read an interesting sentence, they underline it. If they have a comment to make, they write it into the margins. If they have an idea, they write it into their notebook, and if an article seems important enough, they make the effort and write an excerpt. Working like this will leave you with a lot of different notes in many different places. Writing, then, means to rely heavily on your brain to remember where and when these notes were written down.
Evergreen notes should be atomic
It’s best to create notes which are only about one thing—but which, as much as possible, capture the entirety of that thing.
This way, it’s easier to form connections across topics and contexts. If your notes are too broad, you might not notice when you encounter some new idea about one of the notions contained within, and links to that note will be muddied. If your notes are too fragmented, you’ll also fragment your link network, which may make it harder to see certain connections. Evergreen notes should be densely linked
There’s no clear litmus test or correct answer here—just a bunch of tradeoffs.
The notion is quite similar to the software engineering principle of separation of concerns, which suggests that modules should only be “about” one thing, so that they’re more easily reusable. But likewise, if you fragment modules too much, you’ll have a cohesion problem. In this way, Evergreen note titles are like APIs.
References
The underlying principle I’d call the principle of atomicity: put things which belong together in a Zettel, but try to separate concerns from one another. For example, I might collect a list of assumptions in one Zettel which serves as an overview. like hard determinism . A related argument and its conclusion will be kept in another Zettel. Moral responsibility under hard determinism is a good example. I can re-use the arguments without buying into the assumptions because the arguments are of sufficiently general form. Atomicity fosters re-use which in turn multiplies the amount of connections in the network of Zettels.
Evergreen notes should be concept-oriented
It’s best to factor Evergreen notes by concept (rather than by author, book, event, project, topic, etc). This way, you discover connections across books and domains as you update and link to the note over time (Evergreen notes should be densely linked).
The most straightforward way to take notes is to start a new note for each book, each project, or each research topic. Because each note covers many concepts, it can be hard to find what you’ve written when a concept comes up again later: you have to remember the name of each book or project which dealt with the topic (by contrast: Evergreen notes should be atomic).
When you read another book which discusses the same concept, you’ll write a new note on that book. With this approach, there’s no accumulation (contra Knowledge work should accrete). Your new thoughts on the concept don’t combine with the old ones to form a stronger whole: you just have a scattered set of notes on the concept, perhaps referring to it by different names, each embedded in some larger document.
It’s not just about accumulation. There’s also no pressure to synthesize your new ideas on the concept with your prior thoughts about it. Is there tension between them? Is some powerful distillation only visible when all these ideas are considered simultaneously? Understanding requires effortful engagement
If we read two books about exactly the same topic, we might easily link our notes about those two together. But novel connections tend to appear where they’re not quite so expected. When arranging notes by concept, you may make surprising links between ideas that came up in very different books. You might never have noticed that those books were related before—and indeed, they might not have been, except for this one point.
Organizing by concept makes note-taking a little harder, but in a useful way: when writing new notes, we have to find where they fit into the whole. So we explore some part of our prior web of notes, which may lead us somewhere unexpected.
Over time, we accumulate notes which we can combine in increasingly complex ways (Evergreen note titles are like APIs) to produce novel insights (Evergreen note-writing helps insight accumulate).
References
When you’ve taken two texts apart already, a Zettelkasten will help you draw connections between them, see their similarities and oppositions. Thereby, you’ll be able to distill a bunch of texts and find out something new for yourself with time.
you’ll generate new ideas by following connections and exploring a part of your web of notes. The non-apparent connections are generally more beneficial to creative thinking than the obvious ones as they generate greater surprise.
when you analyze a text, you decompose its web-like whole into pieces and keep track of their relations to one another.
We could try to generalize the experiences of Paris, Florence, New York under general concepts like “art” or “exhibitions,” or “crowding” (inter-actionistic), or “mass,” or “freedom” or “education,” in order to see how the slip box reacts. Usually it is more fruitful to look for formulations of problems that relate heterogeneous things with each other.
In the old system, the question is: Under which topic do I store this note? In the new system, the question is: In which context will I want to stumble upon it again?
Fleeting literature notes can make sense if you need an extra step to understand or grasp an idea, but they will not help you in the later stages of the writing process, as no underlined sentence will ever present itself when you need it in the development of an argument.
Many students and academic writers think like the early ship owners when it comes to note-taking. They handle their ideas and findings in the way it makes immediate sense: If they read an interesting sentence, they underline it. If they have a comment to make, they write it into the margins. If they have an idea, they write it into their notebook, and if an article seems important enough, they make the effort and write an excerpt. Working like this will leave you with a lot of different notes in many different places. Writing, then, means to rely heavily on your brain to remember where and when these notes were written down. A text must then be conceptualised independently from these notes, which explains why so many resort to brainstorming to arrange the resources afterwards according to this preconceived idea.
Evergreen notes should be densely linked
If we push ourselves to add lots of links between our notes, that makes us think expansively about what other concepts might be related to what we’re thinking about. It creates pressure to think carefully about how ideas relate to each other (see Understanding requires effortful engagement and Evergreen notes should be concept-oriented). It’ll also help you internalize the ideas more deeply through Elaborative encoding.
Finding the right links requires reading old notes, so it’s also an organic mechanism for intermittently reviewing the notes we’ve written (Evergreen note maintenance approximates spaced repetition). This may lead to surprising discoveries (Notes should surprise you).
And by recording the connections, we document how we came to our conclusions, which may be useful to us (or our colleagues) later. As much as is possible, we should Prefer fine-grained associations. By contrast, Tags are an ineffective association structure.
When just reading through our notes, the connections offer many paths to move through idea-space. The temptation is to navigate hierarchically, but the links cut across fields and topics. Prefer associative ontologies to hierarchical taxonomies
Luhmann actually argues that…
In comparison with this structure, which offers possibilities of connection that can be actualized, the importance of what has actually been noted is secondary.
You don’t necessarily have to link to notes you’ve already written: Backlinks can be used to implicitly define nodes in knowledge management systems. It feels high-friction to stop and add a new note whenever it feels necessary; it’s very freeing to be able to link to a stub. (see also Evergreen notes permit smooth incremental progress in writing (“incremental writing”)).
Aside from the ongoing value of the captured links, they may help you shepherd your attention while drafting: Release valves for non-linear thought may support improved linear output.
References
2. Possibility of linking (Verweisungsmöglichkeiten). Since all papers have fixed numbers, you can add as many references to them as you may want. Central concepts can have many links which show on which other contexts we can find materials relevant for them. Through references, we can, without too work or paper, solve the problem of multiple storage. Given this technique, it is less important where we place a new note. If there are several possibilities, we can solve the problem as we wish and just record the connection by a link or reference. Often the context in which we are working suggests a multiplicity of links to other notes. This is especially the case when the card index is already voluminous. In such cases it is important to capture the connections radially, as it were, but at the same time also by right away recording back links in the slips that are being linked to. In this working procedure, the content that we take note of is usually also enriched
In any case, communication becomes more fruitful when we succeed to activate the internal network of links at the occasion of writing notes or making queries. Memory does not function as the sum of point by point accesses, but rather utilizes internal relationships and becomes fruitful only at this level of the reduction of its own complexity. In this way, more information becomes available at this isolated moment of an search impulse than one had in mind. There is also more information than was ever stored in the form of notes, The slip box provides combinatorial possibilities which were never planned, never preconceived, or conceived in this way.
If we ask, for instance, why on the one hand museums are empty, while on the other hand exhibitions of paintings by Monet, Picasso, or Medici are too crowded, the slip box accepts this question under the perspective of “preference for what is temporally limited.” The connections that already exist internally are, of course, selective, as this example was to prove. They also do not fall into the limits of what is obvious because we must cross the border between the one who takes note and the slip box itself.
Evergreen note titles are like APIs
When Evergreen notes are factored and titled well, those titles become an abstraction for the note itself. The entire note’s ideas can then be referenced using that handle (see Concept handles, after Alexander). In fact, this property itself functions as a kind of litmus: as you develops ideas in notes over time and improve the “APIs,” you’ll be able to write individual notes which abstract over increasingly large subtrees (e.g. Enacted experiences have incredible potential as a mass medium, Evergreen note-writing as fundamental unit of knowledge work).
Some effective note “API design” techniques: separation of concerns (Evergreen notes should be atomic), sharp titles (Prefer note titles with complete phrases to sharpen claims), and positive framings (Prefer positive note titles to promote systematic theory).
Related: Grounded claims, after Qian et al
References
Prefer note titles with complete phrases to sharpen claims
When writing Evergreen notes, I’ve found that using complete phrases as note titles helps maintain concept-orientation (Evergreen notes should be concept-oriented). For example: Educational objectives often subvert themselves, Evergreen notes permit smooth incremental progress in writing (“incremental writing”).
These are often declarative or imperative phrases making a strong claim. This puts pressure on me to adequately support the claim in the body. If I write a note but struggle to summarize it in a sharp title, that’s often a sign that my thinking is muddy or that this note is about several topics (contra Evergreen notes should be atomic). In both cases, the solution is to break the ideas down and write about the bits I understand best first.
Questions also make good note titles because that position creates pressure to make the question get to the core of the matter. Some questions really are evergreen (To what extent is exceptional ability heritable?); others are more ephemeral creative prompts (How might the mnemonic medium enable readers in genres outside platform knowledge?). The goal with the latter type of note is to eventually drop the question mark, refactoring it into declarative/imperative notes.
A few common exceptions to this policy:
- I use nouns and noun phrases in note titles only to define core terms (which other notes generally orbit around). Examples: Executable strategy, Enacted experience, Enabling environment
- Outline notes (see Create speculative outlines while you write) like §Note-writing systems
- For more, see Taxonomy of note types
I often begin by writing a note without knowing what the title will be. The title often emerges from the text as it’s written. When a note suggests a strong title with a clear claim, that’s a good sign that it’s starting to make sense. Related: Evergreen note titles are like APIs
Prefer positive note titles to promote systematic theory
When writing Evergreen notes, it’s tempting to write notes like “X is bad” or “Y doesn’t work.” Prefer instead to write note titles which express the property or requirement in a positive sense.
If you’ve thought them through, negatively-oriented notes usually contain a descriptive theory of why the missing/flawed property is important. By bringing that theory to the foreground, you make it easier to see the claims in a systematic context, which, in turn, makes it easier to build on.
For example:
- Metacognitive supports require dynamic, participatory environments, not “Passive environments can’t offer good metacognitive support”
- Use phones to collect and triage, not (usually) to read, not “Don’t read on your phone”
Evergreen notes are a safe place to develop wild ideas
When you have some inkling about a novel idea, it’s tempting to try to immediately write down the idea and develop it in-place. But often, that’s not possible, practically or emotionally: the idea may just not be solid enough yet to attack directly. The blank page may feel intimidating; the claims may still feel mushy.
Instead, nurture the wild idea and let it develop over time by incrementally writing Evergreen notes about small facets of the idea. Those notes have much tighter scope: they just have to describe one atomic concept (Evergreen notes should be atomic, Evergreen notes should be concept-oriented).
The idea doesn’t even initially have to be related to any pre-existing line of thought. But over time, you can incrementally connect it to other concepts, old or new. See also: Spaced repetition may be a helpful tool to incrementally develop inklings.
You can Create speculative outlines while you write to tie those pieces together, and in time, they’ll accumulate into a more coherent whole. (Notes should surprise you and Knowledge work should accrete).
By contrast: Brainstorming may often substitute for missing insight accretion systems
References
Steven Johnson, who wrote an insightful book about how people in science and in general come up with genuine new ideas, calls it the “slow hunch.” As a precondition to make use of this intuition, he emphasises the importance of experimental spaces where ideas can freely mingle (Johnson 2011). A laboratory with open-minded colleagues can be such a space, much as intellectuals and artists freely discussed ideas in the cafés of old Paris. I would add the slip-box as such a space in which ideas can mingle freely, so they can give birth to new ones.
Every intellectual endeavour starts from an already existing preconception, which then can be transformed during further inquires and can serve as a starting point for following endeavours. Basically, that is what Hans-Georg Gadamer called the hermeneutic circle (Gadamer 2004).
Because he didn’t pre-judge the fittingness of new ideas or try to put them in order but just let them flow in, these ideas sometimes came in so fast he couldn’t write them down quickly enough. The subject matter, a whole metaphysics, was so enormous the flow had turned into an avalanche. The slips kept expanding in every direction so that the more he saw the more he saw there was to see. It was like a Venturi effect which pulled ideas into it endlessly, on and on. He saw there were a million things to read, a million leads to follow… too much… too much… and not enough time in one life to get it all together. Snowed under.
Evergreen notes permit smooth incremental progress in writing (“incremental writing”)
Evergreen notes’ atomic size (Evergreen notes should be atomic) and link structures (Evergreen notes should be densely linked) make it easy to stop and resume work. This helps us Close open loops.
These small, self-contained notes represent regular checkpoints. Each note takes only a few minutes to write, but because they’re Evergreen notes, each note is solid ground to stand on—fairly complete relative to its own concept (Evergreen notes should be concept-oriented). Of course, we’ll iterate on their contents over time, but each time we do, that note will remain a mostly-complete, self-contained unit. They’re Short assignments, after Anne Lamott.
By contrast, when we’re working on a large work-in-progress manuscript, we’re juggling many ideas in various states of completion. Different parts of the document are at different levels of fidelity. The document is large enough that it’s easy to lose one’s place or to forget where other relevant points are when one returns. Starting and stopping work for the day feel like heavy tasks, drawing heavily on working memory.
References
As the outcome of each task is written down and possible connections become visible, it is easy to pick up the work any time where we left it without having to keep it in mind all the time.
All this enables us to later pick up a task exactly where we stopped without the need to “keep in mind” that there still was something to do. That is one of the main advantages of thinking in writing – everything is externalised anyway.
Only if nothing else is lingering in our working memory and taking up valuable mental resources can we experience what Allen calls a “mind like water” - the state where we can focus on the work right in front of us without getting distracted by competing thoughts.
Evergreen note-writing as fundamental unit of knowledge work
If you had to set one metric to use as a leading indicator for yourself as a knowledge worker, the best I know might be the number of Evergreen notes written per day. Note-writing can be a virtuosic skill, but Most people use notes as a bucket for storage or scratch thoughts and Note-writing practices are generally ineffective.
- Evergreen note-writing helps insight accumulate
- Evergreen note-writing helps reading efforts accumulate
- Note-writing helps writing accumulate: these notes are the fuel for the Executable strategy for writing, particularly if you Create speculative outlines while you write.
A caveat: “Better note-taking” misses the point; what matters is “better thinking”
References
If writing is the medium of research and studying nothing else than research, then there is no reason not to work as if nothing else counts than writing.
Focusing on writing as if nothing else counts does not necessarily mean you should do everything else less well, but it certainly makes you do everything else differently. Having a clear, tangible purpose when you attend a lecture, discussion or seminar will make you more engaged and sharpen your focus.
Even if you decide never to write a single line of a manuscript, you will improve your reading, thinking and other intellectual skills just by doing everything as if nothing counts other than writing.
Evergreen note-writing helps insight accumulate
Much of the day-to-day thinking involved in creative work is simply lost, like sand castles in the tide. Ephemerality can actually be useful in low-fidelity thought, but it’s simply an accidental property in many cases. We should do our serious thinking in the form of Evergreen notes so that the thinking accumulates.
Leaps of insight emerge from prior thought. So where does that thought happen? It could happen in your head, or in a series of fleeting sketches in the pages of your notebook, but Knowledge work should accrete, and those mechanisms are awfully lossy.
Consider some hypothetical leap of insight you’d like to be able to make. To make that leap, you’ll typically need to evolve many independent, partially-formed ideas simultaneously, until they suddenly converge in a flash of inspiration. If you need to iterate on more than a few pieces at once, you may struggle to keep them all in your head.
By contrast, because Evergreen notes should be atomic, they’re small enough in scope that you can start and finish one note in well under half an hour (see Evergreen notes permit smooth incremental progress in writing (“incremental writing”)). Yet each note you write represents an increment in your thinking about that specific idea, and each note enriches the broader network of links (Evergreen notes should be densely linked). Because these are Evergreen notes, you now have a clear place to stand as you iterate on this specific idea.
The notes you write will interact with materials you read (Evergreen note-writing helps reading efforts accumulate) and will produce the foundations of new manuscripts (Executable strategy for writing).
And if you can’t write even one atomic note on the idea you have, Spaced repetition may be a helpful tool to incrementally develop inklings.
Related: “Better note-taking” misses the point; what matters is “better thinking”
References
Naturally, independence presupposes a minimal measure of intrinsic complexity. The slip box needs a number of years in order to reach critical mass. Until then, it functions as a mere container from which we can retrieve what we put in. This changes with its growth in size and complexity. On the one hand, the number of approaches and occasions for questions increases. The slip box becomes a universal instrument.
Evergreen note-writing helps reading efforts accumulate
It’s important to Write about what you read to internalize texts deeply, but instead of just writing about the specific book you’re reading, you can (and should) write your notes such that your reading observations accumulate over time as they interact with each other and with your own ideas (see Evergreen note-writing helps insight accumulate, Knowledge work should accrete).
This is also why we write Evergreen notes: so that if we encounter a book which discusses a concept we’ve already written about, we’re pushed to integrate new ideas with our prior conception. Certainly, we normally do this when we read, but we’re limited to our faulty memory of other works which might be related. The externalized note-taking system substantially removes this limitation.
This is why part of why Evergreen notes should be concept-oriented: so that the structure of our notes pushes us to notice the relationships between the ideas in different texts—and in our own work (see Evergreen notes should be densely linked, Notes should surprise you).
The notes you write will also produce the foundations of new manuscripts (Executable strategy for writing).
This is one reason for Evergreen note-writing as fundamental unit of knowledge work.
Evergreen note maintenance approximates spaced repetition
Because writing Evergreen notes with dense associative structure (see Evergreen notes should be densely linked) requires that we constantly reread and revise our past writing, this type of note-taking approximates spaced repetition.
In particular, the spaced repetition follows your present interests. If you stop reading or writing about a given topic, you’ll mostly never revisit it. If you’re regularly reading or writing about a topic, you’ll revisit that prior material fairly constantly.
This isn’t an efficient spaced repetition, memory-wise: you’re not really taking advantage of the Testing effect. But it does take advantage of something like the Generation effect, and it may be a useful lens to think about managing your attention across the corpus of ideas you accumulate over time. Specifically, this practice will encourage you to repeatedly give attention to past ideas which seem relevant to your present work. And because you may find yourself revising your notes on those past ideas, the attention may be quite effortful.
This same effect occurs when maintaining systems which involve Transclusion.
References
We learn something not only when we connect it to prior knowledge and try to understand its broader implications (elaboration), but also when we try to retrieve it at different times (spacing) in different contexts (variation), ideally with the help of chance (contextual interference) and with a deliberate effort (retrieval). The slip-box not only provides us with the opportunity to learn in this proven way, it forces us to do exactly what is recommended just by using it.
Similarities and differences between evergreen note-writing and Zettelkasten
My practice of writing Evergreen notes is heavily inspired by Niklas Luhmann’s Zettelkasten practice and its contemporary advocates. I use a different term both because there are some distinctions and because I want to give myself space to explore ideas in this space apart from the culture surrounding Zettelkasten, which has its own prior values and proclivities.
Key similarities
- concept-orientation (Evergreen notes should be concept-oriented)
- atomicity (Evergreen notes should be atomic)
- emphasis on linkage (Evergreen notes should be densely linked)
- serendipity as a virtue (Notes should surprise you)
- the centrality of one’s own ideas and their development over time, rather than merely accumulating summaries of others’ ideas (Evergreen notes are a safe place to develop wild ideas, Do your own thinking, contra Commonplace books)
- emphasis on using one’s own words, even when describing others’ ideas (Write about what you read to internalize texts deeply)
Key differences
- My approach includes a few methods surrounding the note-writing which expand its scope:
- a method for capturing scraps which would not yet qualify as Zettel, then revising them over time: A writing inbox for transient and incomplete notes
- an unusual method for managing that inbox: Spaced repetition can lower the stakes around destructive inbox-maintenance operations
- a method for systematically revisiting questions and proto-notes over time: Spaced repetition may be a helpful tool to incrementally develop inklings
- a concrete daily practice: My morning writing practice
- More generally, my approach uses a broader Taxonomy of note types which describes a hierarchy and methodology for capturing ideas very early and incrementally developing clusters of notes into increasingly higher-level representations. By contrast, Zettels are mostly meant to represent a few “middle” levels of this hierarchy.
- Titles are very important indexes in my system. Zettels normally have numeric identifiers; modern adherents may give their notes titles, but apparently not with the same approach of “creating APIs.” Evergreen note titles are like APIs
- Contextual backlinks significantly change the nature of “links,” allowing emergent behaviors like Backlinks can be used to implicitly define nodes in knowledge management systems
- I’ve embedded a Spaced repetition memory system into my notes: The mnemonic medium can be extended to one’s personal notes. This has substantially changed the medium, but in ways I don’t yet understand very well.
- My notes are publicly accessible, and I integrate them into public conversation. I don’t understand the impact of this practice yet, but it certainly changes my relationship to the notes in a significant way and creates valuable discussion. There are likely some interesting networked intelligence angles here. I know of no prominent “live” public Zettelkasten. Luhmann’s is archived, but that’s a different relationship—he’s not using it in conversation, then using that interaction to inform his writing.
One final difference, this one a touch pointy: the primary purpose of my system is to develop ideas in my core creative projects. Most people in the contemporary Zettelkasten culture seem to use their systems primarily to write notes about others’ ideas. If they’re developing their own ideas with them, those ideas are an interesting hobby, not their core creative work. All this falls afoul of the issues around People who write extensively about note-writing rarely have a serious context of use. I don’t know how, exactly, but my context of use substantially shapes the note-writing practice.
Zettelkasten
The 20th-century German sociologist Niklas Luhmann managed to publish 70 books. He credits much of his success to his Zettelkasten, or “slip box.” It’s an unusual system for developing ideas over long periods of time by slowly iterating on thousands of atomic slips of paper, all densely linked to each other. Over time, it evolved into what Luhmann considered to be an independent thought partner in his research, capable of carrying on a conversation with him and eliciting ideas which genuinely surprised him.
Though Luhmann is often mentioned most in association with the concept, it apparently significantly predates him:
Born out of the commonplace tradition with modifications by Conrad Gessner (1516-1565) and descriptions by Johann Jacob Moser (1701–1785), the Zettelkasten, a German word translated as “slip box”, is generally a collection of highly curated atomic notes collected on slips of paper or index cards.
Arno Schmidt, a modernist German author, used this method extensively and published a book called Zettels Traum (“Slip Dream”) which interpolates scholarly commentary (in a zettel-ish style) with a primary narrative. (video, article)
See also: Similarities and differences between evergreen note-writing and Zettelkasten
References
Knowledge Work and Thinking
Knowledge work should accrete
Many activities in Knowledge work seem to be ephemeral efforts, their outputs mostly discarded after they’re completed.
You might wake up to a really tricky email and realize that it connects to something you’ve been thinking about for a while. You might spend an hour writing a careful reply, capturing your latest thinking. And now… it lives in your “sent” folder, and briefly in the impression on your and your colleague’s mind. The effort accumulates only insofar as that work subtly influences your and your colleague’s thinking over time.
Likewise, Most people take only transient notes, though with effective practices, they’re an essential foundation; see Evergreen note-writing as fundamental unit of knowledge work.
We should strive to design practices systems which yield compounding returns on our efforts as they accumulate over time.
A Spaced repetition memory system achieves this for memory: when you find information useful, you can invest a little effort to make sure you always have it available. Over time, one’s spaced repetition library accumulates thousands of questions, and (I strongly suspect) that knowledge makes it easier to be an effective knowledge worker later.
Hamming illustrates this point vividly:
You observe that most great scientists have tremendous drive. I worked for ten years with John Tukey at Bell Labs. He had tremendous drive. One day about three or four years after I joined, I discovered that John Tukey was slightly younger than I was. John was a genius and I clearly was not. Well I went storming into Bode’s office and said, “How can anybody my age know as much as John Tukey does?” He leaned back in his chair, put his hands behind his head, grinned slightly, and said, “You would be surprised Hamming, how much you would know if you worked as hard as he did that many years.” I simply slunk out of the office!
What Bode was saying was this: “Knowledge and productivity are like compound interest.” Given two people of approximately the same ability and one person who works 10% more than the other, the latter will more than twice outproduce the former. The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity - it is very much like compound interest. I don’t want to give you a rate, but it is a very high rate. Given two people with exactly the same ability, the one person who manages day in and day out to get in one more hour of thinking will be tremendously more productive over a lifetime. I took Bode’s remark to heart; I spent a good deal more of my time for some years trying to work a bit harder and I found, in fact, I could get more work done.
References
But most importantly, without a permanent reservoir of ideas, you will not be able to develop any major ideas over a longer period of time because you are restricting yourself either to the length of a single project or the capacity of your memory. Exceptional ideas need much more than that.
On Twitter, you don’t build anything.
Core practices in knowledge work are often ad-hoc
Knowledge workers’ routinely involves complex, underspecified tasks like understanding the basics of a new industry or writing a memo on some strategic consideration. Even when these responsibilities are core to their work, knowledge workers’ approach to these tasks is often ad-hoc—created on the spot based on past experience, instincts, aphorisms, and whim. These practices are the opposite of a Executable strategy.
This way of working is highly contingent on both practitioner and context, so it’s difficult for knowledge workers to share and build knowledge about these practices.
Without consistent ground to stand on, it’s difficult for knowledge workers to evaluate and develop their own performance relative to these core tasks. Athletes and musicians pursue virtuosity in fundamental skills much more rigorously than knowledge workers do
Because these processes aren’t considered systematically, knowledge workers typically don’t structure the outputs of these tasks so that they will accumulate into something more valuable over time (contra Knowledge work should accrete).
Leaps of insight emerge from prior thought
When looking at someone else’s stroke of genius, you see only the end product. You don’t see how much kindling was burnt before that sudden realization was possible. In part, that’s because even our own epiphanies don’t feel like they emerge as a natural consequence of prior efforts—but they do!
Leaps of insight depend on having accumulated lots of prior thought on those topics. Sometimes that accumulation happens entirely within our subconscious (our “subconscious back burners,” as May-Li would say), but it’s helpful to design our external cognitive systems such that our day-to-day noodling can accrete (see Knowledge work should accrete).
One practical implication of this notion: Evergreen note-writing as fundamental unit of knowledge work.
References
focus lies almost always on the few exceptional moments where we write a lengthy piece, a book, an article or, as students, the essays and theses we have to hand in.
The things you are supposed to find in your head by brainstorming usually don’t have their origins in there. Rather, they come from the outside: through reading, having discussions and listening to others, through all the things that could have been accompanied and often even would have been improved by writing.
Do your own thinking
Beware: it’s too easy to let others’ schema and ideas dominate your own. It’s hard to hear yourself think.
When reading, the default is to let the author do your thinking for you. Per Schopenhauer:
When we read someone else thinks for us: we merely repeat his mental process. … Accordingly in reading we are for the most part absolved of the work of thinking. … It stems from this that whoever reads very much and almost the whole day, but in between recovers by thoughtless pastime, gradually loses the ability to think on his own – as someone who always rides forgets in the end how to walk. But such is the case of many scholars: they have read themselves stupid. For constant reading immediately taken up again in every free moment is even more mentally paralysing than constant manual labour, since in the latter we can still muse about our own thoughts. But just as a coiled spring finally loses its elasticity through the sustained pressure of a foreign body, so too the mind through the constant force of other people’s thoughts.
That’s bad from an epistemological perspective, but it’s particularly bad for achieving novel insight. How can you think thoughts which have never been thought before if you’re reliant on others’ thinking?
Per Kant:
Enlightenment is the human being’s emergence from his self-incurred minority. Minority is inability to make use of one’s own understanding without direction from another. This minority is self-incurred when its cause lies not in lack of understanding but in lack of resolution and courage to use it without direction from another. Sapere aude! Dare to be wise!
One key antidote: Write about what you read to internalize texts deeply
References
His first and most dramatic step was to stop talking. He decided that he simply would not speak to anyone nor allow anyone to speak to him until he felt that he knew what words he wanted to use and what they meant. Thus, he would be forced really to understand what he was thinking, to think thoughts that were based solely on his experience, and to avoid parroting un-truths learned from others. His silence lasted almost two years. “Out of this intense period of silent thought emerged in embryo most of the great philosophical and mathematical innovations that have made his fame and. moved the world forward a litrle,” concludes Hatch.
I had believed that texts contain information, and that it was my job to make the information accessible.
It is easy {in the world to live after the world’s opinion}; it is easy {in solitude to live after our own}; but the great man is he who {in the midst of the crowd} keeps {with perfect sweetness} {the independence of solitude}.
Don’t wait for the author to hammer you over the head. Instead, from the very beginning, constantly generate hypotheses (“the main point of the book is that…”) and questions (“How does the author know that…?”) about the book.
Making brief notes about these can help. As you read, try to confirm your hypotheses and answer your questions. Once you finish, review these.
Many eminent thinkers need a writing surface to think
Many of the most effective people I know—alive and dead—seem unable to do serious thinking without a writing surface in front of them. It seems to extend cognition somehow: perhaps it effectively extends one’s Span of working memory, or perhaps moving the fingers somehow contributes to the thinking.
For instance, in an interview with Charles Weiner, Richard Feynman said that for him (1973) the paper’s not just a record of work done in his head:
Weiner: (Referring to Feynman’s journals) And so this represents the record of the day-to-day work.
Feynman: I actually did the work on the paper.
Weiner: That s right. It wasn’t a record of what you had done but it is the work.
Feynman: It’s the doing it — it’s the scrap paper.
Weiner: Well, the work was done in your head but the record of it is still here.
Feynman: No, it’s not a record, not really, it’s working. You have to work on paper and this is the paper. OK?
Grothendieck, an eminent 20th century mathematician, is said not to be able to think without writing (2007):
He was improvising, in his fast and elegant handwriting. He said that he couldn’t think without writing. I, myself, would find it more convenient first to close my eyes and think, or maybe just lie down, but he could not think this way, he had to take a sheet of paper, and he started writing. He wrote X → S, passing the pen several times on it, you see, until the characters and arrow became very thick. He somehow enjoyed the sight of these objects.
G wrote:
The role of writing is not to record the results of research, but is the process itself of research. I have always made every effort to describe in the most meticulous way possible, using mathematical language, these images and the understanding they bring. It is in this continuous effort to articulate the inarticulable, to define what is yet unclear, that the particular dynamic of mathematical work (and perhaps as well all creative intellectual work) is perhaps found.
References
Writing forces sharper understanding
Writing is a great way to put pressure on your thinking: it’s hard to summarize something you don’t sharply understand. By trying to explain an idea, you’ll naturally try multiple framings, flesh out its edges, and see new connections. This is part of why Evergreen note-writing helps insight accumulate and why you should Write about what you read to internalize texts deeply.
The additional step of making associations and integrating that writing with prior notes (i.e. to create Evergreen notes, particularly since Evergreen notes should be concept-oriented) makes this effect even more powerful because you have to understand how a given idea relates to other ideas. And when you’re comparing the new ideas to the old, you can see what’s not being said in the new work.
This practice is a rough kind of metacognitive support: Metacognitive supports as cognitive scaffolding.
This observation appears to be true even for non-prose writing: Many eminent thinkers need a writing surface to think
References
Kant, I. (1996). An answer to the question: What is enlightenment? In A. Wood (Ed.), & M. J. Gregor (Trans.), Practical philosophy (pp. 11–22). https://doi.org/10.1017/CBO9780511813306.005 (Original work published 1784)
Don DeLillo, on why he became a writer (via Michael Nielsen):
I have an idea but I’m not sure I believe it. Maybe I wanted to learn how to think. Writing is a concentrated form of thinking. I don’t know what I think about certain subjects, even today, until I sit down and try to write about them. Maybe I wanted to find more rigorous ways of thinking. We’re talking now about the earliest writing I did and about the power of language to counteract the wallow of late adolescence, to define things, define muddled experience in economical ways. Let’s not forget that writing is convenient. It requires the simplest tools. A young writer sees that with words and sentences on a piece of paper that costs less than a penny he can place himself more clearly in the world. Words on a page, that’s all it takes to help him separate himself from the forces around him, streets and people and pressures and feelings. He learns to think about these things, to ride his own sentences into new perceptions. How much of this did I feel at the time? Maybe just an inkling, an instinct. Writing was mainly an unnameable urge, an urge partly propelled by the writers I was reading at the time.
Ahrens, S. (2017). How to Take Smart Notes: One Simple Technique to Boost Writing, Learning and Thinking – for Students, Academics and Nonfiction Book Writers.
If we try to fool ourselves here and write down incomprehensible words, we will detect it in the next step when we try to turn our literature notes into permanent notes and try to connect them with others.
Writing notes and sorting them into the slip-box is nothing other than an attempt to understand the wider meaning of something. The slip-box forces us to ask numerous elaborating questions: What does it mean? How does it connect to … ? What is the difference between … ? What is it similar to?
Experienced academic readers usually read a text with questions in mind and try to relate it to other possible approaches, while inexperienced readers tend to adopt the question of a text and the frames of the argument and take it as a given. What good readers can do is spot the limitations of a particular approach and see what is not mentioned in the text.
Barbara Tuchmann, quoted in The Independent Scholars Handbook - Richard Gross:
“I do all my own research,” she said, “though reviewers have speculated that I must have a band of hirelings. I like to be led by a footnote onto something I never thought of. I rarely photocopy research materials be- cause, for me, note-taking is learning, distilling. That’s the whole essence of the business, In taking notes, you have to discard what you don’t need. If you photocopy it, you haven’t chewed it.”
Concept handles, after Alexander
A “concept handle” is a memorable noun phrase representing a complex, often abstract topic. For example: “prisoner’s dilemma,” “Overton window,” “belief in belief,” etc. In my own writing, examples include Enabling environment, Enacted experience, etc. The “concept handle” is a concept handle for itself, coined by Scott Alexander.
Successful concept handles can really amplify a vague idea which many people sort of understand but can’t point to and talk about. If you give that vague notion a crisp, catchy name, you can unlock a lot of conversation and reflection. Per Alexander:
I’m not too likely to discover some entirely new social phenomenon that nobody’s ever thought about before. But there are a lot of things people have vague nebulous ideas about that they can’t quite put into words. Changing those into crystal-clear ideas they can manipulate and discuss with others is a big deal.
If you figure out something interesting and very briefly cram it into somebody else’s head, don’t waste that! Give it a nice concept-handle so that they’ll remember it and be able to use it to solve other problems!
One way to think about concept handles is as “APIs for concepts.” In this sense, the idea connects to my thinking about Evergreen note titles are like APIs.
Q. What’s an example of a concept handle?
A. (prisoner’s dilemma, enacted experience, etc…)
Q. What’s the networked-intelligence reason why concept handles are important?
A. They let people coordinate discussion around an idea that was previously nebulous.
References
Let ideas and beliefs emerge organically
Beware preconceived notions. Do your own thinking.
- Producing work more quickly by using existing work
- Evergreen note-writing as fundamental unit of knowledge work
- Create speculative outlines while you write
- Executable strategy for writing
- Epistemology, originality
- Use notes to avoid preconceived conclusions
- Leaps of insight emerge from prior thought
- Brainstorming may often substitute for missing insight accretion systems
- Prefer associative ontologies to hierarchical taxonomies
- Evergreen notes are a safe place to develop wild ideas
- Morale:
- responding to authentic interest and thought instead of thinking “I should write about X”
- no more “what should I write about?”
Brainstorming may often substitute for missing insight accretion systems
Brainstorming with others can create a fluid social context to exchange ideas, but when we brainstorm alone, we’re creating an expansive space to summon what we already “almost know.” The ideas we can have are primarily limited by our prior thinking and ideation in this space (see Leaps of insight emerge from prior thought).
A brainstorm’s generative, uncritical mindset might lead to surprising results, but often we also use brainstorming as a practical way to brain-dump all our ideas about a subject in one place. That may be a sign that we haven’t designed our knowledge systems as described in Knowledge work should accrete. With Evergreen notes and dense associative structure (see Evergreen notes should be densely linked), our ideas are collected and distilled continuously. Evergreen note-writing helps insight accumulate, so there’s less need for brainstorming because those insights have already emerged—and captured in an ongoing fashion—in one’s day-to-day work.
See also: Evergreen notes are a safe place to develop wild ideas
References
As proper note-taking is rarely taught or discussed, it is no wonder that almost every guide on writing recommends to start with brainstorming. If you haven’t written along the way, the brain is indeed the only place to turn to. On its own, it is not such a great choice: it is neither objective nor reliable – two quite important aspects in academic or nonfiction writing. The promotion of brainstorming as a starting point is all the more surprising as it is not the origin of most ideas: The things you are supposed to find in your head by brainstorming usually don’t have their origins in there.
Original thought requires solitude
Grothendieck, “The importance of Solitude”, via Michael Nielsen:
In those critical years I learned how to be alone. But even this formulation doesn’t really capture my meaning. I didn’t, in any literal sense, learn to be alone, for the simple reason that this knowledge had never been unlearned during my childhood. It is a basic capacity in all of us from the day of our birth. However these three years of work in isolation 1945-1948, when I was thrown onto my own resources, following guidelines which I myself had spontaneously invented, instilled in me a strong degree of confidence, unassuming yet enduring in my ability to do mathematics, which owes nothing to any consensus or to the fashions which pass as law. By this I mean to say: to reach out in my own way to the things I wished to learn, rather than relying on the notions of the consensus, overt or tacit, coming from a more or less extended clan of which I found myself a member, or which for any other reason laid claim to be taken as an authority. This silent consensus had informed me both at the lycee and at the university, that one shouldn’t bother worrying about what was really meant when using a term like “volume” which was “obviously self-evident”, “generally known,” “in problematic” etc… it is in this gesture of “going beyond” to be in oneself rather than the pawn of a consensus, the refusal to stay within a rigid circle that others have drawn around one – it is in this solitary act that one finds true creativity. All others things follow as a matter of course.
Since then I’ve had the chance in the world of mathematics that bid me welcome, to meet quite a number of people, both among my “elders” and among young people in my general age group who were more brilliant, much more ‘gifted’ than I was. I admired the facility with which they picked up, as if at play, new ideas, juggling them as if familiar with them from the cradle – while for myself I felt clumsy, even oafish, wandering painfully up an arduous track, like a dumb ox faced with an amorphous mountain of things I had to learn (so I was assured) things I felt incapable of understanding the essentials or following through to the end. Indeed, there was little about me that identified the kind of bright student who wins at prestigious competitions or assimilates almost by sleight of hand, the most forbidding subjects.
In fact, most of these comrades who I gauged to be more brilliant than I have gone on to become distinguished mathematicians. Still from the perspective of thirty or thirty five years, I can state that their imprint upon the mathematics of our time has not been very profound. They’ve done all things, often beautiful things in a context that was already set out before them, which they had no inclination to disturb. Without being aware of it, they’ve remained prisoners of those invisible and despotic circles which delimit the universe of a certain milieu in a given era. To have broken these bounds they would have to rediscover in themselves that capability which was their birthright, as it was mine: The capacity to be alone.
Deresiewicz, W. (2010, March 1). Solitude and Leadership. The American Scholar.:
But it seems to me that solitude is the very essence of leadership. The position of the leader is ultimately an intensely solitary, even intensely lonely one. However many people you may consult, you are the one who has to make the hard decisions. And at such moments, all you really have is yourself.
Susan Sontag (As Consciousness Is Harnessed to Flesh: Journals and Notebooks, 1964-1980; 7/19/77)
One can never be alone enough to write. To see better.
Q. In what sense does Grothendieck claim he didn’t exactly learn to be alone?
A. He views this as a basic capacity endowed at birth, and—for some people—unlearned over time.
Q. What’s the primary reason Grothendieck views solitude as so important?
A. Solitude is necessary to “go beyond” the social consensus of what is known, what is worth asking, what one should aim for.
Q. What cautionary tale does Grothendieck observe in many of his preternaturally gifted young peers?
A. Decades later they’ve become professionals, but their work isn’t quite profound, because they remain prisoners of the context others have set out for them.
Henrik Karlsson - Cultivating a state of mind where new ideas are born
Conversation with others often emphasizes the most well-understood elements of an idea
There’s an unintuitive danger in talking about an emerging idea with others. The clearest, most familiar parts are the ones which you’ll have the easiest time communicating and which your conversation partner will have the easiest time grasping. Often, those notions are already somewhat mainstream or even clichéd; others are likely to have lots of cached thoughts around that idea, and they’ll tend to interpret it incrementally.
But if you’re doing something original, the most interesting elements are the ones which others—and you!—understand least well. Particularly early on, you may not be able to articulate the new element you’re reaching for very clearly. It may just sound like an unusual adverb choice or an innocuous-seeming qualifier. In any case: because their replies will tend to emphasize the most mainstream elements and pass over the elements you least understand, conversation will often drag you back towards the mainstream. It’s a kind of “regression to the mean” for ideas.
Of course, the best colleagues and collaborators actively avoid this trap! One of my favorite Michael Nielsen behaviors is that if he hears me talking about some idea that seems fairly banal, he’ll deliberately tug at the places where I’m straining to reach past typical interpretations.
This idea emerged from an early 2018 conversation with Bret Victor. He’d just returned from a visit to a major research lab, where he’d presented and discussed some of his work. I asked whether the conversations had been useful, and he glumly replied that he felt he needed to go isolate himself in a cabin for a week to begin to hear his own thoughts again. This was quite a confusing reply in the moment, but I soon experienced this challenge myself in conversations around my Khan Academy research and understood what (I think!) he meant.
In this 2016 email he makes a similar remark.
A creative person chooses their work by following an internal compass, which they must trust to point in a meaningful and valuable direction. For me, this compass is a fragile thing, easily confused by interfering magnetic fields. Any time I’m in a social situation where I’m expected to explain myself — “no, it’s not Internet of Things… no, it’s not Augmented Reality… no, it’s not Artificial Intelligence… no, well, I don’t know…” — my compass gets twisted around and I lose all grasp on why I am. It’s an awful feeling, and can sometimes take days to recover.
Deep understanding requires (and is a result of) intense personal connection
Learning matters insofar as it influences our future thoughts and actions. In fact, because Understanding requires effortful engagement, learning is only a consequence of a rich tapestry of thoughts and actions woven with those concepts.
Seen through this lens, perfunctory learning activities (“fine, I guess I’ll attend that workshop”) often subvert their own goal. Without personal connection to the ideas, participants’ intellectual participation will be relatively shallow. The tapestry will be woven too loosely; it will soon unravel. Little will be deeply understood.
When an activity’s real purpose is something intrinsically meaningful to the participant, their earnest intellectual participation will naturally produce effortful engagement with the ideas involved. Deeper understanding was not the goal, but it will likely occur anyway. Gumption transcends willpower and confidence.
For curiosity-driven activities in particular, personal connection is the literal precursor; the activities continue only as long as that connection does. The connection between the activity and the intellectual participation is particularly pure in these cases.
John Littlewood (the English mathematician) wrote:
I have tried to learn mathematics outside my fields of interest; after any interval I had to begin all over again.
Jonathan Blow on a HN comment:
- More broadly, the best and most creative work comes from a root of joy and excitement. If you lose your ability to feel joy and excitement about programming-related things, you’ll be unable to do the best work. That this issue is separate from and parallel to burnout! If you are burned out, you might still be able to feel the joy and excitement briefly at the start of a project/idea, but they will fade quickly as the reality of day-to-day work sets in. Alternatively, if you are not burned out but also do not have a sense of wonder, it is likely you will never get yourself started on the good work.
References
Deep understanding requires detailed knowledge of fundamentals
In response to People seem to forget most of what they read, and they mostly don’t notice, many suggest that they don’t want detailed recall. They do most of their reading “to get a general picture,” or “just to get a conceptual understanding.” That might sometimes be possible, but in many cases it’s not possible to really understand a concept without a firm grasp of the details on which it’s built.
The intuitive argument:
Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding. Imagine meeting a person who told you they “had a broad conceptual understanding” of how to speak French, but it turned out they didn’t know the meaning of “bonjour”, “au revoir”, or “tres bien”. You’d think their claim to have a broad conceptual understanding of French was hilarious.”
How can we develop transformative tools for thought? - How important is memory, anyway?
A more concrete argument is that conceptual understanding is mostly about connections—understanding how elements relate to each other, causes, effects, implications, constraints, tendencies, etc. You can’t understand these high-order relationships without familiarity with their constituents.
Another argument draws on our understanding of human information processing: Expertise requires building sophisticated chunk recoding schemes.
A related, simpler claim: Memory augmentation may make it easier to learn complex topics by decreasing working memory load
Q. Reductio ad absurdum argument against someone who only wants a “broad conceptual understanding,” not detailed recall?
A. That’s like wanting to have a “broad conceptual understanding” of French without knowing the meaning of “bonjour.”
Q. Connectivist argument that deep understanding requires detailed knowledge of fundamentals?
A. Conceptual understanding is largely about relationships; can’t learn the edges without knowing the nodes.
References
Gumption transcends willpower and confidence
It’s nice to be disciplined and ambitious, but having gumption is even better. Gumption is the inspired, agentic feeling for an activity which makes it no longer require willpower or confidence.
Gumption comes easily when activities have intrinsically meaningful purposes. It’s readily sapped by nonsense. Related: Enabling environments’ activities directly serve an intrinsically meaningful purpose
References
Note-Taking Practices
Note-writing practices are generally ineffective
Knowledge work should accrete, but Most people take only transient notes.
In part because Note-writing practices provide weak feedback, people don’t even notice how ineffective their note-writing practices are. They develop some baseline level of note-writing skill and mostly stay there (People generally develop skills to a plateau and then stop). Expert performance is not well-defined, so it’s not obvious that people aren’t performing well (Salience of improvement drives skill development). All this is true of other core knowledge work skills, too: Core practices in knowledge work are often ad-hoc.
Much of what’s written about trying to improve these practices is misguided: “Better note-taking” misses the point; what matters is “better thinking”
By contrast: Evergreen note-writing as fundamental unit of knowledge work
Note-writing practices provide weak feedback
One reason why Note-writing practices are generally ineffective may be that note systems generally offer poor feedback.
If one starts a spaced repetition practice, they’ll get strong feedback every day: if they write a bad question, it’ll bother them immediately and regularly thereafter; they’ll feel (to some extent) their growing retention of a given topic.
By contrast, in note-taking the feedback is very delayed: in typical practice, when you write a note, you may not see it again for weeks. The feedback is also ambiguous: if a note helps you distill some insight (or fail to), that usually won’t be especially evident.
In general, people don’t have a clear picture of what a note should be, so it’s not clear when a given note fails to conform to that standard.
Evergreen notes offer tighter and higher-signal feedback loops because you’re constantly revising and consulting past notes. It’s somewhat easier to notice if a prior note is difficult to revise. The Executable strategy for writing also creates stronger feedback on one’s note-taking practices.
References
There is another reason that note-taking flies mostly under the radar: We don’t experience any immediate negative feedback if we do it badly.
Note-writing can be a virtuosic skill
Most people don’t seem to take note-writing very seriously as a skill. For some, that’s because they see it as a simple utility (e.g. Most people use notes as a bucket for storage or scratch thoughts). For others, their indifference may arise because they notice that Note-writing practices are generally ineffective.
But some note-writing practices are vastly more effective than others. It’s possible to become much, much more skilled at writing notes, and that skill translates into valuable output: Evergreen note-writing as fundamental unit of knowledge work.
Most people take only transient notes
In contrast to Evergreen notes, Most people use notes as a bucket for storage or scratch thoughts. These are very convenient to write, but after a year of writing such notes, they’ll just have a pile of dissociated notes. The notes won’t have added up to anything: they’re more like fuel, written and discarded to help the author process their ongoing experiences.
Fleeting notes are valuable scratchpads to temporarily support working memory, but Knowledge work should accrete, so we should view them as “messy-thought” inputs for the “neat-thought” notes they’ll inform (Khoe).
This is one reason why Note-writing practices are generally ineffective.
References
Most people use notes as a bucket for storage or scratch thoughts
People don’t want to forget an idea or a conversation or a task or quote from a book, so they write it down in Evernote or something. What’s the purpose? Probably: “To make sure I don’t forget.” Maybe: “Just writing it down helps me remember.”
In this conception, notes are a way to Close open loops, not to accumulate insight. The “real” work happens outside the notes; the notes are just a reference system which stores information which might help, or a write-once pile of “messy” thoughts (Khoe).
These are not Evergreen notes. Most “storage-oriented” notes will never be useful again (Most people take only transient notes). More importantly, this framing misses that it’s possible for note-writing to be the “real” work (Evergreen note-writing as fundamental unit of knowledge work).
References
People who write extensively about note-writing rarely have a serious context of use
Many bloggers and “life-hackers” have made a full-time job of suggesting how you should organize your journal, or how you should most effectively Write about what you read to internalize texts deeply. We should take this advice seriously insofar as those practices have helped the authors achieve meaningful creative work: “Better note-taking” misses the point; what matters is “better thinking”
But most people who write about note-taking don’t seem particularly accomplished in their own fields, whatever those may be. In fact, most such writers aren’t applying their notes to some exogenous creative problem: their primary creative work is writing about productivity. These writers offer advice on note-taking to help scientists and executives with the challenges of their work, but the advice was developed in a context disconnected from those external realities. There are two related problems here: Effective system design requires insights drawn from serious contexts of use, and Powerful enabling environments usually arise as a byproduct of projects pursuing their own intrinsically meaningful purposes.
Luhmann, by contrast, barely wrote about his Zettelkasten: he focused on his prolific research output, then published a couple small essays about his practices near the end of his career.
John Locke is one interesting counter-example. Evidently a prolific note-taker, he published a book on his “New Method of Making Common-Place Books”.
I’m not quite guilty of this problem myself, but I certainly slip into this behavior for weeks at a time. This is a cautionary note. Related: The most effective readers and thinkers I know don’t take notes when reading.
References
Taxonomy of note types
TODO: flesh this out; write a note for each note type; etc
For me, the practice of writing and revising notes is, at its core, about trying to move up the following rough ladder:
- Ephemeral scratchings in Daily working log
- Prompts and incomplete notes in writing inbox (A writing inbox for transient and incomplete notes), which are meant to possibly become Evergreen notes. These prompts may accumulate writing over time via spaced repetition (see Spaced repetition may be a helpful tool to incrementally develop inklings)
- Evergreen notes, in increasingly complex stages of development:
- stubs implicitly defined through backlinks (Backlinks can be used to implicitly define nodes in knowledge management systems); e.g. Note-writing system, Audiobooks
- simple definitions for terms of art, little of my own added; e.g. Span of absolute judgment, Spacing effect
- Bridge notes narrowly relate two adjacent terms
- precise, narrow declarative notes (Prefer note titles with complete phrases to sharpen claims); e.g. Human channel capacity increases with stimulus dimensionality
- sometimes these are framed as questions, when evidence is too inconclusive to frame sharply; e.g. To what extent is exceptional ability heritable?, To what extent can application prompts supplant recall prompts in the mnemonic medium?
- higher-level APIs (Evergreen note titles are like APIs)
- notes abstracting over many other notes, e.g. Educational games are a doomed approach to creating enabling environments, Reading texts on computers is unpleasant
- personal terms of art, e.g. Enacted experience
- “Outline notes”, e.g. §Enabling environments, games, and the Primer
Note types outside this ladder:
- Proper noun notes
- “Literature notes”, titled after a single work and meant primarily as linkages to other more durable notes, and as targets for backlinks. I write these roughly as “outline notes,” except for someone else’s ideas. For example: Miller - The magical number seven, plus or minus two
- Likewise, but less commonly, I also have “person notes” (e.g. Anand Agarawala) and “business notes” (e.g. Confluent)
- These note types are weakly evergreen. I may add to them over time, but because they aren’t concept-oriented (Evergreen notes should be concept-oriented), they’re not as useful to build on as an evergreen note. Non-trivial writing about proper nouns typically gets factored into separate evergreen notes which can be used in multiple places.
- “Log” notes, which accumulate ephemeral observations about a specific practice, system, or project over time. They’re akin to a Daily working log, but sliced by some topic of interest rather than by date. e.g. Log: personal mnemonic medium
- With better transclusion or support for Contextual backlinks, such notes could be written directly in one’s Daily working log, and “log” notes could be defined as a query over such notes.
Tactically speaking, I usually denote a note’s “type” with a tag.
Don’t over-obsess or over-formalize this stuff. Remember: “Better note-taking” misses the point; what matters is “better thinking”.
Literature notes are secondary and separate
Because Evergreen notes should be concept-oriented reference-specific notes should be both brief and clearly separated from the note archive. They primarily exist to help you write durable notes.
Literature notes are typically a lightweight synthesis of observations collected while reading (see How to collect observations while reading). We can keep a quick summary of the work and those observations around for later literature lookup, but the bulk of the value will already have been absorbed into our lasting notes (see How to process reading annotations into evergreen notes). The reference-specific notes are mostly useful for their links into those lasting notes (Are literature notes necessary if we have automatic universal backlinks?).
There’s an important philosophical reason why we should keep these literature notes separate from our durable notes. The archive of lasting notes is the place where you Do your own thinking: you’ve interpreted others’ ideas into your own structure of knowledge. Direct quotes are fairly rare; durable notes are intentionally expressed in your own words. By contrast, literature notes are often mostly the author’s thoughts. They tend to lean on direct quotes, and even when our interpretation is offered, it’s in the context of the author’s ontology and claim system. It’s hard to hear yourself think, so we should clearly separate the space where we do our own thinking from these more direct representations of others’ thoughts.
References
You need to take some form of literature note that captures your understanding of the text, so you have something in front of your eyes while you are making the slip-box note. But don’t turn it into a project in itself. Literature notes are short and meant to help with writing slip-box notes. Everything else either helps to get to this point or is a distraction.
I always have a slip of paper at hand, on which I note down the ideas of certain pages. On the backside I write down the bibliographic details. After finishing the book I go through my notes and think how these notes might be relevant for already written notes in the slip-box. It means that I always read with an eye towards possible connections in the slip-box.
Are literature notes necessary if we have automatic universal backlinks?
Literature notes are secondary and separate, but are they necessary at all? Are they only necessary because you might want to start a walk through your notes from a specific reference that happened to come up, and it’s awkward to find all the notes which refer to a specific reference?
In other words, is this just a hold-over from the physical Zettelkasten? If we have a system which easily allows us to see—perhaps without even asking—all the notes associated with a reference, would we still need to keep literature notes at all?
One reason to advocate for literature notes is that it would create an opportunity to explicitly curate key durable notes associated with a note. This might be important for cutting through the noise, especially if a reference is mentioned frequently.
Would such curated associations even need to be textual? Is it sufficient to form metadata-only associations? Advantages and disadvantages of using notes to form associations in content
Related: How should note tagging practices change with ranked link visualization?
References
A writing inbox for transient and incomplete notes
Even if you aspires to write Evergreen notes, most notes begin as transient notes. You should be able to capture thoughts without friction (Close open loops), then reliably develop them into evergreen notes over time (Knowledge work should accrete). This implies two important mechanisms:
- a quick way to capture transient notes which clearly isolates them from evergreen notes; and
- a place to put notes you want to develop further and a practice which reliably drains it (Inboxes only work if you trust how they’re drained)
I use a “writing inbox” for this purpose. Undeveloped ideas, excerpts from my Daily working log, notes from reading, one-line prompts, etc all begin in that queue. During My morning writing practice, I’ll look through notes in this inbox and spend time developing any that strike me. On most days, I spend the majority of my writing time in this way.
Many notes in my writing inbox end up as evergreen notes, but that’s not appropriate (or possible) for all of them. If a note doesn’t seem sufficiently interesting after a few looks, it’s best to archive or delete it. (A challenge here: Triage strategies for maintaining inboxes (e.g. Inbox Zero) are often too brittle)
While I’m at my computer, I capture notes directly into my writing inbox. I also feed it with: Pocket memo pad to capture into writing inbox while out.
References
Daily working log
Each day, I start a note titled with that day’s date; e.g. 2020-03-12. It captures ephemera throughout the day: reflections, scratch work, etc. It’s an intentional dumping ground, a release valve so that there’s always “a place to put that thing.”
In the Taxonomy of note types, this is the lowest-fidelity layer, ephemeral by design. But as scratch thoughts look like they might have legs, they get extracted into a note in my writing inbox (A writing inbox for transient and incomplete notes). Sometimes I’ll use the daily working log as a drafting space, and I can extract sections roughly-as-is into Evergreen notes.
For me, the important bits are:
- This is a space where I can put anything and feel zero friction. (Close open loops)
- I feel a natural pressure to extract anything which wants to “outlive the day”—to move up the Taxonomy of note types.
Because the daily working log is also a live note, it also functions as a useful stub for Contextual backlinks: Backlinks can be used to implicitly define nodes in knowledge management systems.
Bridge notes narrowly relate two adjacent terms
Say that another person has thought deeply about a similar idea to one of your own, and their conclusions seem mostly similar. You each have your own terms of art for various attributes of the theory. It’s tempting to subsume the other’s terms into your own, but there are probably subtle differences between your conceptions and theirs which would be lost in the process. You could just use the other person’s terms, but sometimes it’s hard to go in new directions when anchored to prior terminology which may carries many connotations. And yet you don’t want to be constantly referring to both sets of terms.
One solution is to create a note which relates your term of art to another similar term of art. This establishes a “bridge” between the theories, describing where you overlap and where you differ. The respective terminologies can maintain their “sovereignty” with their own main notes—the bridge note exists just to relate the two. For example: Similarities and differences between evergreen note-writing and Zettelkasten.
Such notes remind me of entries in a relational database many-to-many join table.
References
Write notes for yourself by default, disregarding audience
Because Evergreen notes can be used as part of a strategy for writing public work (Executable strategy for writing), it’s tempting to “save time” by writing notes in publishable form. That might mean providing all the necessary background to understand some (boring to you) idea, or self-censoring, or adding lots of qualifiers, or spending lots of effort on clarity. Many of these practices can be somewhat useful as part of your own thinking process—for instance, clearer writing usually involves clearer thinking. But I find it substantially increases the overhead and effort in writing, often to the point of producing blockage.
More concretely, this manifests as a common failure mode for me when I’m writing notes as part of explicit preparation for some public writing. I’ll often try to do both jobs at once. That is, I might be writing atomic-style notes (Evergreen notes should be atomic) but I try to write them as if they’re sections in a larger essay or work. Or even just: I try to write things with all the context and clear prose needed for an outsider to understand what I’m talking about. Then I often find that I can’t write anything at all! Better to write at a level where I can produce something, then use that to lever myself upward. (Evergreen notes permit smooth incremental progress in writing (“incremental writing”))
When it’s a topic I understand well, I can write notes for both myself and an audience simultaneously. But that sometimes produces the false impression that I can pull this off all the time! To avoid that false impression, I’ll write notes for myself “by default,” and only “opt into” writing notes for an audience explicitly.
Q. What bad habit do I often fall into when writing evergreen notes in preparation for public writing?
A. I’ll try to do my public writing as part of my first pass on the notes.
Q. Why do I often find myself stuck when I try to write evergreen notes as publishable prose for an audience?
A. When a topic is hard enough to distill on its own, the extra cognitive load of considering a reader overwhelms me.
Associations and Linking
Prefer associative ontologies to hierarchical taxonomies
Let structure emerge organically. When it’s imposed from the start, you prematurely constrain what may emerge and artificially compress the nuanced relationships between ideas.
Our file systems, organizational structures, and libraries suggest that hierarchical categories are the natural structure of the world. But often items belong in many places. And items relate to other items in very different hierarchical categories.
Worse, by presorting things into well-specified categories, we necessarily fuzz their edges. Things don’t always fit exactly. Maybe once enough new ideas are collected, a new category would emerge… except you can’t see its shape because everything’s already been sorted. And because everything’s already been sorted, further sorting requires undoing existing structure.
It’s better to let networks of related ideas to gradually emerge, unlabeled: Let ideas and beliefs emerge organically. Once you can see the shape, then you can think about its character. This is one reason why Evergreen notes are a safe place to develop wild ideas.
But beware: Tags are an ineffective association structure.
One consequence of following this advice: It’s hard to navigate to unlinked “neighbors” in associative note systems.
References
Our ineptitude in getting at the record is largely caused by the artificiality of systems of indexing. When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used; one has to have rules as to which path will locate it, and the rules are cumbersome. Having found one item, moreover, one has to emerge from the system and re-enter on a new path. The human mind does not work that way. It operates by association. With one item in its grasp, it snaps instantly to the next that is suggested by the association of thoughts, in accordance with some intricate web of trails carried by the cells of the brain.
Topic clusters emerge by themselves, especially surrounding keywords or tags. The resulting archive fits the way you think because it grew according to your interests. Also, things are labeled in a way especially meaningful to you, not anybody else. This is all about personal information management, so personalization is a must, and increasing idiosyncrasy will likely make things better.
Prefer explicit associations to inferred associations
When building an interconnected personal knowledge base (e.g. Evergreen notes should be densely linked), is it critical to build all the associations by hand? Tools like DEVONthink use machine learning to suggest “related” notes. And on the web, one’s history (perhaps even contextualized within a single tab’s timeline) can suggest some relationships between content. Or: can we just use search functionality when we want to navigate around a set of ideas?
Tools like these can be helpful for scaffolding the associative process, and they can serve as a second place to look for linkages, but we shouldn’t rely on them to do all the work.
First: the process of thinking about the relationships between items is part of how you can Do your own thinking. It forces you to engage with new material more deeply.
Second: you want your associations to be high-signal so that you don’t have to try to evaluate some unordered list of links. Being able to quickly evaluate such a list without navigating among the items requires that you remember what all those items contain. Tightly-curated associations will be higher-signal.
Third: your terminology will evolve over time, so text-based linkages aren’t going to help long-term.
References
Even though the Zettelkasten makes suggestions here, too, for example based on joint literature references, making good cross-references is a matter of serious thinking and a crucial part of the development of thoughts.
In the Zettelkasten, keywords can easily be added to a note like tags and will then show up in the index. They should be chosen carefully and sparsely. Luhmann would add the number of one or two (rarely more) notes next to a keyword in the index (Schmidt 2013, 171).
It’s unlikely you write notes today as you did 6 years ago. You use different terms.
Every full text search presents tons of Zettel notes. Because all are seemingly equal (and connected in the same way: not at all), you have to either rely on your memory to distinguish useful from useless notes, or look through all of them every time.
Prefer fine-grained associations
Links between materials in an information system (e.g. Evergreen notes should be densely linked) can be fine-grained (like a citation in the middle of a sentence of a paper) or coarse-grained (like a “see also” section).
It’s generally better to make fine-grained associations. For instance, rather than evaluating a jumbled list of papers related to the paper you’re reading, it’s more helpful to see that I noticed paper X relates to paragraph N.
This is particularly true when links are bidirectional, since you’ll need help to see why the “backwards” relationship makes sense.
Prefer labeled associations
Edges linking nodes in information systems (e.g. Evergreen notes should be densely linked) should themselves be labeled in some way which contextualizes the association. It’s often not enough to say “X is related to Y”: better to say “X goes into more detail about Y in the context of Z.”
The extra context will be a cue for your memory or evaluation of the linked material, which will help you effectively navigate the knowledge graph and retrace your former thought processes.
Tags are an ineffective association structure
Tags are an easy way to relate heterogenous items, but they’re quite a low-signal way of describing relationships.
All items with a given tag are presented as being related… but it’s hard to see how. They’re just a jumbled, unordered list.
Some of those items are more relevant to a particular tag’s topic than others, so we should Prefer explicit associations to inferred associations.
Some of those items have only a few sentences touching on a tag, but the tag is associated with the whole item. We should Prefer fine-grained associations. Relatedly, tags are often pretty vague or broad. Better to link to associate ideas more precisely.
And sometimes, it would be very helpful to have a few words of context about why an item relates to a particular tag. We should Prefer labeled associations.
Backlinks can be used to implicitly define nodes in knowledge management systems
Systems which display backlinks to a node permit a new behavior: you can define a new node extensionally (rather than intensionally) by simply linking to it from many other nodes—even before it has any content.
I first noticed this in Conor White-Sullivan’s behavior with Roam Research. He wrote about our conversations in several notes throughout his system (ephemeral daily logs, feature lists, etc). As he was doing that, he wrote certain noun phrases (e.g. my name) as links. Those nodes had no content of their own, but as he did this across several days, they began to develop an implicit definition within his system, expressed through the backlinks.
This effect requires Contextual backlinks: a simple list of backlinks won’t implicitly define a node very effectively. You need to be able to see the context around the backlink to understand what’s being implied.
I find this particularly useful for terms of art and proper nouns of all sorts.
I’m also experimenting with this technique in my cooking notes; see e.g. Cabbage (but the backlinks will not unfortunately be visible on my public notes, since they’re from my private weekly journals).
Contextual backlinks
A contextual backlink displays not only a reference from another location, but the specific context around that reference—for instance, the page of a book or the referencing paragraph.
Related: Prefer fine-grained associations
Indexed references vs. tags
Tagging is common in contemporary information systems, but Tags are an ineffective association structure. One more effective historical antecedent is the index in publishing.
Indexes don’t strive to include every relevant page number for a given term; instead, it includes a handful of top references. By contrast, the listing of items with a particular tag often becomes quite unwieldy.
Indexes can also include other editorial content. For instance, an entry might note “See also: …”
Both index entries and sophisticated tagging systems can be hierarchical.
In Zettelkasten
Luhmann kept his index cards tightly curated. They were meant mostly as a jumping-off point: the inter-note associations are more important.
How should note tagging practices change with ranked link visualization?
References
Because it should not be used as an archive, where we just take out what we put in, but as a system to think with, the references between the notes are much more important than the references from the index to a single note.
Only tags that are specific to the objects I use and mention in a note are worthy: To take precise actions over a long distance I need a sniper rifle and not a shotgun.
Searching on a topic in your archive is like firing a shotgun into the woods and hoping that there will be food on the table somehow. I need a sniper rifle, night vision goggles, and infrared satellite pictures as if I have cheated the hell out of Counter-Strike. (I never did by the way.) There is some sneaky, precious game out there.
How should note tagging practices change with ranked link visualization?
In their indexes, Zettelkasten practitioners try to tag only the most important few notes for a given topic, then they rely on inter-note linkages to navigate from there (see Indexed references vs. tags). This keeps the entry high-signal (see Tags are an ineffective association structure).
But this may just be a consequence of current interfaces for displaying tagged data. Conceptually, what’s happening here is that in the writer’s mind, many notes are tagged with a given tag, but some notes are particularly relevant to that tag, and the writer doesn’t want to lose track of those. So they create a curated representation of that tag to replace the system’s indifferent “list it all” visualization.
But if you could view a tag’s network structure, arranged by PageRank or something, I suspect this would be come much less important.
Related: Are literature notes necessary if we have automatic universal backlinks?
Advantages and disadvantages of using notes to form associations in content
If we’re reading a web page and notice that it relates to some PDF on our hard drive, we have no real way to make that association. But we can create a note which links to both resources.
This is different from the memex solution
Writing a note is good because it forces you to Do your own thinking, but it’s a bit heavy because Evergreen notes should be concept-oriented. So to write the note, we have to extract the conceptual link between the two, name it, and arrange it in some way that might attract future linkages.
When we Write about what you read to internalize texts deeply, we have two layers of notes: there are lightweight notes about the reading itself in our reference library, then there are higher-fidelity Evergreen notes in our note archive. It may be valuable to support two layers of noting about association fidelity as well: one could make lightweight associations between materials, then encode the valuable observations into durable notes over time.
References
Notes should surprise you
If reading and writing notes doesn’t lead to surprises, what’s the point?
If we just wanted to remember things, we have spaced repetition for that. If we just wanted to understand a particular idea thoroughly in some local context, we wouldn’t bother maintaining a system of notes over time.
This is why we have dense networks of links (Evergreen notes should be densely linked): so that searches help us see unexpected connections.
This is why we take Evergreen notes should be concept-oriented: so that when writing about an idea that seems new, we stumble onto what we’ve already written about it (perhaps unexpectedly).
References
One of the most basic presuppositions of communication is that the partners can mutually surprise each other. Only in the way can information be produced in the respective other.
If you look something up in your Zettelkasten, you need to get unexpected results in order to form new thoughts. Surprise is the key ingredient here, as I pointed out in my introductory post on this topic. The links between notes make this possible since you’ll generate new ideas by following connections and exploring a part of your web of notes. The non-apparent connections are generally more beneficial to creative thinking than the obvious ones as they generate greater surprise. While your mind usually continues to work with the obvious, your Zettelkasten instead shows you the bizarre. It sparks your imagination and blows your mind as it confronts you with the unexpected.
Also, it opens up opportunities to connect thoughts over the course of years which in turn will generate moments of surprise. This eventually leads to discoveries of unforeseen connections and enables you to think out of the box.
Reading and Writing
Write about what you read to internalize texts deeply
If you want to deeply internalize something you’re reading, the best way I know is to write about it:
- because Understanding requires effortful engagement
- and in particular, Writing forces sharper understanding
- remember: Collecting material feels more useful than it usually is
- because otherwise you won’t Do your own thinking
- because Knowledge work should accrete and Evergreen note-writing helps reading efforts accumulate
- indeed: Most people read ineffectively
For deep understanding, it’s not enough to just highlight or write marginalia in books: there isn’t much pressure to synthesize, connect, or to get to the heart of things. And they don’t add up to anything over time as you read more. Instead, write Evergreen notes as you read.
But of course, it doesn’t always make sense to read in this way: much of the time you’re not really trying to internalize the text deeply, and text may not be worthy of that much attention: The best way to read is highly contextual.
Also, it’s worth noting: The most effective readers and thinkers I know don’t take notes when reading. Speaking at least for myself, experience has suggested that I need more support to effectively engage with what I’m reading.
Method
Our broad approach is an alternating cycle:
- Collect passages that seem interesting and thoughts that emerge while reading: How to collect observations while reading
- Process clusters of those passages and thoughts into lasting notes:How to process reading annotations into evergreen notes
References
It is impossible to think without writing; at least it is impossible in any sophisticated or networked (anschlußfähig) fashion.
Notes on paper, or on a computer screen … do not make contemporary physics or other kinds of intellectual endeavour easier, they make it possible.
How to collect observations while reading
It’s important to Write about what you read to internalize texts deeply, but it’s distracting to switch back and forth between reading and writing polished notes. Instead, collect insights in a lightweight way while you read. You can put them in A writing inbox for transient and incomplete notes. That’ll Close open loops, and you’ll process them later (see How to process reading annotations into evergreen notes).
Annotations—even inline marginalia which include your own writing—have very little informational value. They’re atomized; they don’t relate to each other; they don’t add up to anything; they’re ultra-compressed; they’re largely unedited. That’s fine: think of them as just a reminder. They say “hey, look at this passage,” with a few words of context to jog your memory about what the passage was about.
Since you’re going to write lasting notes anyway, annotations need carry just enough information to recreate your mental context in that moment of reading. You wouldn’t want to rely on that long-term, since then you’d just have a huge pile of hooks you’d have to “follow” anytime you wanted to think about your experience with that book.
When processing these observations, you’ll want to be able to see the big picture and see clusters of ideas, so it’s helpful to collect annotations in a manipulable fashion.
Concretely, the approach I’m trying:
- Physical books:
- Write couple-word thoughts on a slip of A7 paper (Pocket memo pad to capture into writing inbox while out)
- Draw a dot or line in the margin of interesting passages and dogear the page.
- Web articles:
- Copy+paste interesting excerpts into a single working note in my writing Inbox.
- Or perhaps use the Bear excerpter, in combination with the marker tool.
- Digital books and PDFs:
- Use in-app highlighters
- Export all highlights into a working note in Inbox to cluster
TODO
I find the digital solutions quite unsatisfying: it’s slow and heavy browsing between annotations in these solutions.
References
The text inspired a thought, and the inspiring part is already marked in the text.
taking notes on my Mac while reading just doesn’t work for me. My state while typing is too different from the state I’m in when I read print. Going back and forth requires heavy switching of mental gears. First, this wears me out after a while. Second, this switching ruins the focus: I cannot follow the text properly. That’s why I take notes on paper and mark the passage I want to refer to with a little * in the page’s margin.
How to process reading annotations into evergreen notes
It’s important to Write about what you read to internalize texts deeply. While reading, you’ve marked passages that seem relevant, and you’ve scribbled notes with your thoughts (How to collect observations while reading). Now we’ll process all that into lasting notes.
First: what notes should even get written? We’ll write Evergreen notes should be concept-oriented, so what are the key concepts? You need to take a step back and form a picture of the overall structure of the ideas. Concretely, you might do that by clustering your scraps into piles and observing the structure that emerges. Or you might sketch a mind map or a visual outline. The structure you observe does not have to match the book’s structure: it’s whatever makes sense relative to your own personal ontology (Do your own thinking).
Once you have a picture of the concepts at play, you’ll begin an iterative process of note-writing. Here I’ve summarized Christian Tietze’s process, which I’m presently adopting / adapting:
- Write a broad note which captures the “big idea” of one of your clusters.
- Are there multiple big ideas? Write multiple broad notes to maintain Evergreen notes should be atomic.
- Write finer-grained notes: Look through the individual scraps in that cluster. Write notes which capture more nuanced atomic ideas within that cluster.
- Connect: Search for relevant past notes which relate to these new notes. Link, merge, and revise as necessary to represent your new, synthesized conception of those ideas.
- Revise: Return to the broad note and improve your summary based on what you’ve learned writing the detailed notes and the details you’ve unpacked, if it’s possible to do so without muddying their focus. Remove detailed notes that are no longer necessary; update others based on what you learned writing your updated broad note if appropriate.
- Loop
References
Second, I find out if a cluster’s main point has too many prerequisites to stand alone. It might be a conclusion which draws from lots of assumptions or from complex models I’d need to explain. I prepare the conclusion first and then branch off into other notes to capture all the necessary ideas. This is where links come in handy: the details point back to the concept note and the concept note mentions its detail branches.
Clusters don’t lean onto the book’s outline. A book’s index for example collects references, not caring about the table of contents or the flow of ideas. For definitions of terms a similar approach is useful: collect usage examples in the text and definitions themselves to get a clear picture of the term’s meaning. Clusters can be topic-based, too, just like an index.
This is what I call ‘orthogonal to the content’: they don’t adhere to the succession of pages and sections. Instead, clusters form themselves around any purpose you deem fitting.
Making sure you will be able to find this note later by either linking to it from your index or by making a link to it on a note that you use as an entry point to a discussion or topic and is itself linked to the index.
Most people read ineffectively
- Knowledge work should accrete, but most reading does not
- People seem to forget most of what they read, and they mostly don’t notice
- People often struggle to remember details of prose text because they never processed them in the first place
- Evergreen note-writing helps reading efforts accumulate, but most people don’t write about what they read
- Of those that do, Note-writing practices are generally ineffective
- However! The most effective readers and thinkers I know don’t take notes when reading
- Skillful reading is often non-linear, but most people read linearly
References
Skillful reading is often non-linear
Books are almost always written with highly linear structures, and the form of the medium itself encourages the reader to move through the text linearly. But skillful readers rarely read linearly. Sometimes they read with intention—looking for how the book can help them answer a specific question. Or they might do a sparse first pass to understand the book’s structure (see Inspectional reading). Or they might start with the index and focus on the most relevant passages.
It’s interesting to think about what kind of medium would support “best practices” by default here.
An interesting contrary argument in McCutcheon (2015):
Ancient authors incorporated this linearity of the scroll into the totality of the hermeneutics for a text. They constructed poetry books without a clear linear narrative linking the individual poems, but relied on the scroll to force readers to make interpretive sense of this variatio of content, since they had to proceed through the poetry book in that manner.
References
The purpose of reading books like these is to gain information. Here, finding out what happens — as quickly and easily as possible — is your main goal. So unless you’re stuck in prison with nothing else to do, NEVER read a non-fiction book from beginning to end.
Instead, when you’re reading for information, you should ALWAYS jump ahead, skip around, and use every available strategy to discover, then to understand, and finally to rememberwhat the writer has to say. This is how you’ll get the most out of a book in the smallest amount of time.
People seem to forget most of what they read, and they mostly don’t notice
It seems that most people can remember only a few high-level details of a book weeks later—if that. A typical reader might spend hours finishing some serious non-fiction—then maybe it comes up at a dinner party, and they find you can remember like three sentences. Basically no detailed recall. Barely the gist! (((Scattered notes|Reading comprehension.md|1,99|1,114))) (((Scattered notes|Reading comprehension.md|1,99|1,114)))
What’s more: people seem surprised when this happens. They seem to consistently overestimate how much they’re absorbing from a book.
Part of the problem here is People often struggle to remember details of prose text because they never processed them in the first place.
See How rapidly do people forget practical knowledge?
This observation is unfortunate for many reasons, but among them: Deep understanding requires detailed knowledge of fundamentals and Complex ideas may be hard to learn in part because their components overflow working memory.
For common objections: Many people view memory as unimportant to deep creative work.
References
People often struggle to remember details of prose text because they never processed them in the first place
People seem to forget most of what they read, and they mostly don’t notice, but this isn’t just a process of long-term memory decay. In my personal experiences, and in my experiences working with students, some details are apparently forgotten because they were never really processed in the first place. The reader’s eye just skidded right over some sentence or paragraph, and the idea was never perceived at all, or so little attention was paid that the idea was never really processed. Sometimes the trouble is that the reader didn’t understand what a sentence was saying, but didn’t realize that or didn’t interrogate it; in either case, the idea will not be remembered. The problem here isn’t that these ideas can’t be recalled a day or a week later; it’s that they can’t be recalled ten seconds later—the idea never made it that far.
“gue sit amet vitae eros. Integer tristique, justo quis dapibus mollis, dolor lacus vestibulum lectus, vel molestie nunc libero sit amet pur” ↗
Per [ [ How to Read a Book - Adler and vaccumsan elementum sementary reading” problem: the reader can’t state “what the text says” (as opposed to what it means, or why it’s being said).
Of course, separately, people will often struggle to remember details because they didn’t understand those details—they didn’t know what the prose meant, and that makes it much more difficult to remember, particularly if the recall task requires even the mildest of Transfer learning.
See 2023-03-31 Patreon letter - Memory systems and problem-solving practice for my observations of this with a student.
Adjunct questions improve comprehension of related but untested content; it’s proposed that this is in part because these questions put the reader into a more attentive state. See also The mnemonic medium may push readers to read more slowly and attentively.
References
Collecting material feels more useful than it usually is
Accumulating tabs, saving PDFs, and making bookmarks feels like progress, but we systematically overrate its value. Understanding requires effortful engagement; you are not likely to draw much understanding from a folder of barely-skimmed PDFs.
We collect material because it’s easy, and because it quells the anxiety that we’ll never find what we’re looking at again. But really, we’re often just making things worse, burying important materials in tons of secondary matter we just “don’t want to lose.” This notion is in contrast to Knowledge work should accrete.
This is a first step to conquer Collector’s Fallacy: to realize that having a text at hand does nothing to increase our knowledge.
Instead, we should Write about what you read to internalize texts deeply, because Evergreen note-writing helps reading efforts accumulate. And to help steer ourselves effectively (contra Note-writing practices provide weak feedback), we should process collected materials in short iteration cycles, rather than letting them pile up for long periods. But! Keep in mind that Most texts aren’t worth writing detailed notes about.
Often a good compromise is to use spaced repetition to cheaply internalize a few key details; you can come back and write real notes later if the material turns out to be valuable. See e.g. Deciding to remember something with a spaced repetition system is (aspirationally) a lightweight gesture
References
Because ‘to know about something’ isn’t the same as ‘knowing something’. Just knowing about a thing is less than superficial since knowing about is merely to be certain of its existence, nothing more. Ultimately, this fake-knowledge is hindering us on our road to true excellence. Until we merge the contents, the information, ideas, and thoughts of other people into our own knowledge, we haven’t really learned a thing. We don’t change ourselves if we don’t learn, so merely filing things away doesn’t lead us anywhere.
Just like photocopying is self-rewarding and addictive, I argue that we fall into the same trap of false comfort when we bookmark web pages and sort the bookmarks into folders or tagged categories. Bookmarking a web page is satisfying because we get rid of the fear of losing access to the information. I get into detail in another post .
This is a first step to conquer Collector’s Fallacy: to realize that having a text at hand does nothing to increase our knowledge. We have to work with it instead. Reading alone won’t suffice: we have to create notes, too, to create real, sustainable knowledge.
Especially when we start to research something new, Eco recommends we read and highlight texts right after we create copies. If we train ourselves to process photocopied texts soon, we get a feeling of how much we can really handle.
Shorter cycles of research, reading, and knowledge assimilation are better than long ones. With every full cycle from research to knowledge assimilation, we learn more about the topic. When we know more, our decisions are more informed, thus our research gets more efficient. If, on the other hand, we take home a big pile of material to read and process, some of it will turn out be useless once we finished parts of the pile.
The marks which can make a difference to their organisations are on the knowledge workers not on the pieces of paper. This is what it means to inform - to change the form of a person or a device such that they act differently (ideally more effectively) on their environment.
A reading inbox to capture possibly-useful references
To avoid a proliferation of anxiety-inducing browser tabs and a terrifying folder of PDFs, it’s important to have an automatic procedure for capturing references to readings which might prove useful.
Once captured, each item in the inbox either:
- gets trashed (doesn’t look like it’s worth a detailed read after all)
- gets read in a serious fashion (i.e. Write about what you read to internalize texts deeply)
- gets read shallowly and filed in the reference library
- (maybe) gets added to some other list like “recipes to be cooked”
Importantly, this isn’t a “someday maybe” list. It doesn’t accumulate indefinitely, because then it wouldn’t be a reliable way to Close open loops.
So, when constructing a reading inbox, the important considerations are:
- zero-friction capture for books, articles, web pages (to easily close that loop)
- zero-friction to view the reading corresponding to an inbox item
- zero-friction listing across item type
- the inbox should encourage lingering items to be removed (e.g. it should be obvious when one has been passed over many times)
Interestingly, no existing “read later” or reference management system fits these criteria. They’re usually siloed by content type, and none of them encourages lingering items to be removed. See also: Beware automatic import into the reading inbox.
The reading inbox is an important release valve for things I encounter when on my smartphone (see Use phones to collect and triage, not (usually) to read).
Related: Incremental reading
References
The Inbox is the place to hold the items we either want to or need to pay attention to. A lot of stuff will never reach our inbox; we can shut off the noise outside.
Some things that found their way onto the reading lists turn out to be useless. Toss them. Putting items on the reading list is a tiny commitment only: we commit to pay attention to them later, but we don’t need to hold on to them if they don’t withstand a critical look.
Beware automatic import into the reading inbox
It’s easy to treat RSS subscriptions, email newsletters, etc as interchangeable with one’s reading inbox, but they should be clearly separated. (see A reading inbox to capture possibly-useful references)
Inboxes only work if you trust how they’re drained. That is, it’s important that everything added is either read or removed, so that if a reference seems interesting, you can add it to that list with the confidence that you’ll return to it later. One effective way to ensure this property is to add everything yourself.
If you automatically import content, then when you sit down to read, you’ll see lots of content that you have no immediate connection to. That material will be interspersed with content you intentionally added because you had a strong emotional connection, diluting the latter.
If you automatically import content, the reading inbox may intermittently become intimidatingly long. That discourages you from opening it, which makes it less reliable.
If you automatically import content, it becomes more expensive to groom your reading inbox. To decide whether to trash or carefully read those items, you have to first get some sense of what they are. By contrast, if you’ve added an item manually, you usually already have some sense of what it is or have some context which indicates something about its value. It’s momentum-sapping to switch back and forth between evaluating items to be read and actually reading them.
If you add content yourself, you’ll provide your own backpressure: you’ll have some awareness of how large your reading inbox has gotten, and so you might think twice about adding something of dubious value.
References
The web is full of noise, and so is this fraction of the web I subscribed to. It’s up to me to find some signal . That means I have to decide what I want to have knowledge about, and what is dismissible.
Close open loops
Tasks left undone, observations left unrecorded, replies yet to be written—these swirl about our minds, as if we’re rehearsing them over and over again to make sure they’re not forgotten. To get rid of this nagging and create a “mind like water” (to use the term in Allen, 2015), build systems to reliably close these open loops.
For instance: for operational to-dos, this means
- You should be able to record a task anywhere
- You regularly drain tasks from this list
- You regularly delegate, refactor, or delete tasks which you can’t prioritize
Taken together, these properties ensure that when you record a task, you can stop thinking about it. Ubiquitous capture isn’t enough, as most to-do systems demonstrate. If you don’t regularly review your task list and decide to delete or re-strategize lingering tasks, you won’t be able to trust that you’ll follow up on tasks you record.
See also A reading inbox to capture possibly-useful references.
References
Executable strategy for writing
A naive writing process begins with a rough inkling about what one wants to write and a blank page. Progress from this point requires an enormous amount of activation energy and cognitive effort: there’s nothing external, so you must juggle all of the piece-to-be in your head.
By contrast, if you’ve already written lots of concept-oriented Evergreen notes around the topic, your task is more like editing than composition. You can make an outline by shuffling the note titles, write notes on any missing material, and edit them together into a narrative. In fact, because you can Create speculative outlines while you write, you might find that the first of these steps is already accomplished, too. And writing each note isn’t hard: Evergreen notes permit smooth incremental progress in writing (“incremental writing”).
Instead of having a task like “write an outline of the first chapter,” you have a task like “find notes which seem relevant.” Each step feels doable. This is an executable strategy (see Executable strategy). But beware—don’t let this strategy “poison” the initial note-writing process: Write notes for yourself by default, disregarding audience.
I describe two approaches here: an undirected version, in which writing projects emerge organically from daily work; and a directed version, in which you’re trying to write about something specific.
Undirected version
- Write durable notes continuously while reading and thinking. (Evergreen note-writing as fundamental unit of knowledge work)
- Each time you add a note, add a link to it to an outline, creating one if necessary (Create speculative outlines while you write).
- Eventually, you’ll feel excited about fleshing out one of those outlines. (Let ideas and beliefs emerge organically)
- Write new notes to fill in missing pieces of the outline.
- Concatenate all the note texts together to get an initial manuscript
- Rewrite it.
Directed version
- Review notes related to your topic (and a step or two beyond those—Notes should surprise you)
- Write an outline
- Attach existing notes to each point in the outline; write new notes as needed.
- Concatenate all the note texts together to get an initial manuscript
- Rewrite it.
One other nice benefit of this approach: Evergreen notes lower the emotional stakes in editing manuscripts.
References
To see with clarity if your research backs up your text’s structure sufficiently, the next step is to assign notes from your Zettelkasten to the items of your outline. When an item of your outline seems to be neglected because you don’t have enough notes that fit, you can continue your research, focusing on the missing pieces. As soon as you’re confident you got enough coverage for a start, you string the notes’s contents together according to the outline. Thus you create the very first draft. That’s all it takes to move from a plan to outline to manuscript. Then you begin to re-write, organize the material and start to make the text coherent.
There’s no magic involved in writing texts with the help of a well-fed Zettelkasten. To compile a first draft you put the contents of selected notes at the appropriate places in the outline, putting meat on the bones of your text’s skeleton. That’s how a Zettelkasten helps you complete your first draft.
Create speculative outlines while you write
When you write a new note, add it to one or more outlines you’re maintaining, creating a new one if necessary. Substantially-complete writing projects will naturally emerge.
Normally, we start an outline when we start a writing project. This forces us to start with a blank page. By contrast, if we write new notes every day and notice how they relate to each other, these can accumulate into potential writing projects. When an outline feels “ripe,” we can pluck it and turn it into a manuscript without the exerting herculean start-up effort that comes with a blank page.
Maintaining already-written notes in an outline is comparatively easy: just look at a pair of notes and ask: which comes first? (Pirsig)
Furthermore, to start a writing project with a blank outline, we need to have a topic and some angle in mind. We can Use notes to avoid preconceived conclusions.
References
Developing arguments and ideas bottom-up instead of top-down is the first and most important step to opening ourselves up for insight.
“When I am stuck for one moment, I leave it and do something else.” When Luhmann was asked what else he did when he was stuck, his answer was: “Well, writing other books. I always work on different manuscripts at the same time. With this method, to work on different things simultaneously, I never encounter any mental blockages.”
When I create a Zettel, I search through the folder for an outline that could make use of this Zettel. If I don’t have an outline in my folder that fits, I create one.
Now I can see articles, books, and other writing projects emerge as a consequence that I read texts and create Zettels without any specific intention in doing so.
The book on writing I mentioned came to be in the same manner. I work on a book on nutrition, so I decided to research how to write a book and writing in general. Each note I turned into a Zettel got its place in an outline. After a while, I satisfied my need for readings on that topic. When I looked at the outline I realized that I had notes worth a book already. I replaced the IDs with the content of the Zettels. Voilá. A manuscript was ready.
Instead of asking Where does this metaphysics of the universe begin? — which was a virtually impossible question -all he had to do was just hold up two slips and ask, Which comes first? This was easy and he always seemed to get an answer. Then he would take a third slip, compare it with the first one, and ask again, Which comes first? If the new slip came after the first one he compared it with the second. Then he had a three-slip organization. He kept repeating the process with slip after slip.
Focus, Creativity, and Motivation
Effective deep work depends on both time and intensity
It’s tough for many people to block off four contiguous hours for focused work, and even harder for many to avoid spending much of that time checking email or social media. But these things are doable, fairly mechanically: schedule the time; turn off the WiFi; block the web sites; etc. The trouble is that this isn’t enough. It’s relatively easy to zone out—not in a creative day-dreaming fashion, but in a scattered, dull fashion.
It’s very difficult to actually maintain your sharp, vivid attention on a problem for hours at a time. But that’s what’s necessary. Getting to the desk and clearing obvious nuisances out of the way is only table stakes. One hour of high intensity will often produce much more than five hours of dull focus, even if the latter includes no gratuitous distraction.
Process over product, and one solution is to focus on measuring inputs instead. Measuring the time spent on a problem is a good start (see e.g. notes on Pomodoro technique); measuring the time spent authentically, without distraction, is better; but really what matters is closer to “time multiplied by intensity”, as Cal Newport observes in Deep Work.
Getting this intensity depends on more than just the right behavior during the actual moments of work:
Dullness
Buddhism offers an interesting model of mental acuity. It suggests that a key challenge to progressing in Meditation is “dullness,” (“laya” in Sanskrit) characterized by weak attention, “like trying to see through a dense fog.” It becomes an obstacle during longer meditation sessions because the mind is used to a certain amount of stimulation to occupy its attention, and that’s not supplied during meditation.
This is a familiar problem when doing difficult intellectual work, too: Dullness and distraction in creative work may arise from the same causes as in meditation.
See Moments of consciousness model, after Culadasa for a model which attempts to explain this phenomenon.
Q. Name a few cures for dullness in meditation.
A. e.g. Deep breaths, exhaled through pursed lips; tense all muscles until trembling, then relax; meditate standing; meditate walking; cold water
Q. Why do common cures (“tense all muscles”) work against dullness in meditation?
A. They provide increased stimulus for the mind, decreasing the proportion of non-perceiving moments.
References
Dullness and distraction in creative work may arise from the same causes as in meditation
It sounds obvious in retrospect: Meditation instructors say all the time that the point is not to achieve attention and awareness just while in explicit meditation sessions, but to bring that same quality of mind to everyday life. I’d understood that in terms of recognizing Impermanence and Non-identification. But somehow I hadn’t connected that to problems I encounter when doing difficult creative work.
When I’m doing some difficult design or research, it’s very common for me to struggle with feelings of tiredness. I think this is just Dullness. And of course I often experience the progression of distraction → forgetting → mind-wandering, as described in Moments of consciousness model, after Culadasa.
These models help explain why I don’t wrestle with dullness or distraction/forgetting nearly as often when doing other activities. When I’m doing difficult creative work, I’m staring at the same object of attention for an extended period, just like in a meditation session; and just like the breath, the work rapidly ceases to produce novel stimuli. I’m staring at the same difficult paragraph I was staring at five minutes ago. And just as with the breath, when the object of attention is not generating many perceiving moments, the mind will naturally scan for other objects in awareness which might be more stimulating. Alternately (or additionally), the proportion of perceiving moments will simply drop, which in turn causes Dullness.
The cure for these problems in creative work is the same as the cure for these problems in meditation:
- cultivating continuous introspective awareness (e.g. through setting intention to audit attention every few breaths)
- cultivating heightened sensitivity to the object of attention, intention to perceive ever greater level of detail
- increasing the proportion of perceiving moments to create more buffer against dullness, e.g. by expanding the scope of awareness
Of course, the kind of distraction I’m describing here is mostly internal—the kind I generate for myself. Paul Graham points out that often, “distraction is not a static obstacle that you avoid like you might avoid a rock in the road. Distraction seeks you out.”
Q. In what sense do my meditation experiences explain my sleepiness-at-work experiences?
A. In both instances, you’re trying to maintain attention on a low-stimulation object; without training, the proportion of non-perceiving moments will climb.
Q. Why am I more likely to suffer from dullness and distraction when doing challenging creative work than in other activities?
A. When stuck on a creative problem, the object of attention isn’t naturally changing, so it generates increasingly subtle stimuli.
Displacement activity
Displacement activities are self-soothing activities which people (and animals) perform when they’re highly motivated to perform multiple behaviors, as well as when they’re unable to or uncertain how to proceed. The behaviors are also associated with stress and anxiety.
For example, people scratch their heads when they’re not sure which option to choose. Or they respond to urgent emails (or refresh social media) when they’re feeling anxious about challenging intellectual work: see also Culturally default behaviors fill spare time with others’ ideas, People prefer doing to thinking. Animals tend to groom, preen, or scratch in similar situations.
See also Team environments make it easy to fool yourself with displacement activities.
Q. When do animals perform displacement activities?
A. When stressed or anxious; when uncertain or torn about how to proceed.
Q. How do knowledge workers manifest displacement activities?
A. Doing lots of simple urgent tasks instead of the highly uncertain but important tasks.
People prefer doing to thinking
The phrase comes from a paper from Timothy Wilson and colleagues, describing a variety of studies in which participants were asked to sit and think in a bare room for 15 minutes. In one of the studies, the room contained a device they could use to shock themselves, and almost half did—even though they previously answered that they’d pay to avoid a shock!
Related:
- Culturally default behaviors fill spare time with others’ ideas
- Many people find difficult creative work mostly unpleasant in the moment
- It’s hard to hear yourself think
Q. Vivid experiment demonstrating that people prefer doing to thinking?
A. Wilson et al asked people to sit and think for 15 minutes but offered them a device they could use to shock themselves, and many did, despite previously stating that they’d pay money to avoid a shock!
References
Get playful
An exhortation closely related to Get curious, but suited to moments of action.
Don’t worry about doing it “right,” don’t worry about “how long it’s taking me,” don’t worry about “solving” the problem or doing something “important.” Just enjoy the interaction with the problem and the impulses you feel relative to it. Act on those impulses; watch what happens; act on the new impulses. Turn up the temperature on the distribution.
Examples
It’s hard not to think of someone like Jacob Collier when I think about this (see e.g.this interview). He seems to be living a constantly-playful life of musical exploration. I find it helpful to think about him as an exemplar here.
Obsession as high-order bit
Amazing creative work typically emerges from individuals who experience creation or discovery in their field as profoundly meaningful. This involves an almost spiritual devotion—seeking moments of transcendence beyond the mundane to encounter something timeless and revealing about reality’s underlying patterns.
Michael Nielsen captures this sentiment through his phrase “searching for the numinous.”
Einstein’s 1918 reflection on Max Planck illustrates this principle: “The state of mind which enables a man to do work of this kind is akin to that of the religious worshiper or the lover; the daily effort comes from no deliberate intention or program, but straight from the heart.”
Adam Mostroianni’s essay “An invitation to a secret society” describes this approach as “doing things the beautiful way”—pursuing work for intrinsic meaning rather than external rewards like money, ambition, or utilitarian benefit.
Related concepts explore how Tools for thought should be designed to foster intrinsically meaningful purposes and how powerful enabling environments often emerge as byproducts of projects driven by genuine purpose rather than external motivation. Cultivating curiosity remains essential to this orientation.
Skill Development
People generally develop skills to a plateau and then stop
People build thousands of skills in their lives—chopping vegetables, reading books, handwriting, making a budget, etc. These skills usually don’t improve linearly over a person’s life. Instead, people focus on a skill for some initial period, then it reaches some “good enough” plateau, then it mostly stays there. Some significant later event might cause the skill to suddenly start improving again, but it’s generally a punctuated equilibrium.
This happens because Naive approaches to practice rapidly plateau. Once that plateau is reached, it takes renewed effort to keep making progress: Performance plateaus often require a change in approach to surmount. At that point, the extra effort may not be worth it, or it might not be clear how to improve. Or the possibility of improvement may not be salient: Salience of improvement drives skill development.
Per Thorndike (1921, p. 178):
The main reason why we write slowly and illegibly, add slowly and with frequent errors, delay our answers to simple questions and our easy decisions between courses of action, … forget people’s names and our own engagements, lose our tempers, and the like, is not that we are doing the best that we are capable of in that particular. It is that we have too many other improvements to make, or do not know how to direct our practice, or do not really care enough about improving, or some mixture of these three conditions.
It’s rarely the case that people are anywhere near their limit, or even that marginal improvement is terribly difficult. One fun example he cites: Aschaffenburg (1896) administered daily speed tests to experienced type-setters, and without any other inducement, observed them improve their speed each day.
One big manifestation of this observation: Athletes and musicians pursue virtuosity in fundamental skills much more rigorously than knowledge workers do.
References
In their classic studies of Morse Code operators, Bryan and Harter (1897, 1899) identified plateaus in skill acquisition, when for long periods subjects seemed unable to attain further improvements. (p. 365)
Salience of improvement drives skill development
Everyone knows what virtuosic piano playing sounds like. When an amateur plays, he naturally compares himself to recording he’s heard—and may quite painfully feel how much improvement is possible! By contrast, when people make complex decisions, they usually lack a visceral sense that they’re an amateur. Military officers are trained to make strategic and decisive decisions, but that’s not particularly salient to a small business owner. Decision-making, as an abstract skill, wouldn’t appear on the “skill weightlifting” menu.
I suspect that this is one key reason why People generally develop skills to a plateau and then stop, and in particular why Athletes and musicians pursue virtuosity in fundamental skills much more rigorously than knowledge workers do. If you’ve never heard of a Spaced repetition memory system, it’s probably not obvious that memory can be made a solved problem. If you’ve never heard of Evergreen notes, it’s probably not obvious that Knowledge workers usually have no specific methods for developing ideas over time.
Executable strategy
In creative work and in life, many goals seem unpredictable and unwieldy, reliant on hope and luck: starting a habit, getting in shape, writing a book, doing a research project, etc. (see Core practices in knowledge work are often ad-hoc)
Cleaning the kitchen isn’t like that. Even clearing out one’s inbox isn’t like that (Triage strategies for maintaining inboxes (e.g. Inbox Zero) are often too brittle). For many prosaic goals, you have executable strategies, which reliably achieve the goal with a predictable, manageable amount of effort. They’re a kind of metacognitive support (Metacognitive supports as cognitive scaffolding).
To construct an executable strategy, we must factor the task into activities such that:
- completing more of those activities will reliably bring you closer to the goal
- each activity consumes a predictable amount of effort
- each activity feels doable
- little effort is required to select and plan an activity
One common choice is to set daily goals for a certain number of hours at work. Success with this strategy requires a clear theory of how those hours will inexorably accumulate to the desired outcome. Simply spending some number of hours on a project is a fairly weak constraint: it’s easy to work with focus many hours unproductively.
Another common choice is to discretize the creative output. If you’re trying to write an essay, you might aim to write a certain number of words per day; if you’re designing an interface, you might aim to design one element of the UI per day. But these steps don’t consume a predictable amount of effort, and they often don’t feel doable. If you’re stuck in your writing because you’ve become confused about one of your ideas, you won’t be able to write 500 words for your manuscript. Instead, you need to spend more time thinking about the idea. A manuscript is a challenging place to do that. By contrast, see Evergreen note-writing as fundamental unit of knowledge work.
List of executable strategies
- Executable strategy for writing
- Spaced repetition memory systems make memory a choice
- How to process reading annotations into evergreen notes
- Starting a habit
Desirable difficulties, after Bjork
Training activities can often be made more effective by introducing difficulties.
Some examples which Robert A. Bjork (1994) cites as improving long-term performance in experiments:
- Shuffling heterogenous tasks rather than grouping repetitions by task type
- Varying the parameters of a task
- Varying the environmental context of learning sessions
- Creating “contextual interference” which requires more attention: e.g. by providing resources which differ in structural organization
- Reducing the frequency of feedback (e.g. by providing aggregate feedback every N trials or by reducing the frequency of feedback over time)
One consistent theme with all of these manipulations is that they reduce performance during the training activity itself, but they yield better longer-term outcomes. This reversal probably makes it difficult to adopt some of these manipulations in learning activities, since they’ll make learners feel like they aren’t learning as well. e.g. (Bjork, 1994 again):
Such a conditioning process, over time, can act to shift the trainer toward manipulations that increase the rate of correct responding — that make the trainee’s life easier, so to speak. Doing that, of course, will move the trainer away from introducing the types of desirable difficulties summarized in the preceding section.
Worse, there may be difficult institutional barriers: trainers may be evaluated by immediate (not long-term) performance; or they may not have a chance to observe long-term performance.
Besides this short-term reversal, students may avoid “desirable difficulties” because they believe their memory/understanding to be stronger than it really is. Bjork suggests that “training is frequently non-optimal because it fails to incorporate the variability, delays, uncertainties, and other challenges the learner can be expected to face in a real-world job setting of some kind” (see also Transfer learning).
Q. What counter-intuitive effect does increased difficulty have on performance in training tasks?
A. It typically harms performance during training but improves it in long-term measures.
Q. Why might learners believe that introducing difficulties into practice sessions makes them learn less well?
A. Added difficulties will often harm performance during practice (while increasing long-term performance).
Q. Why might coaches be institutionally disincentivized from adding desirable difficulties to practice activities?
A. They might be evaluated by students’ short-term performance.
Q. Why might trainers be unable to see the long-term impact of difficulties added to training exercises?
A. The training timeline may be too short to see improvements which may come from more difficult training activities. Alternately, they may never get to see real-world performance of their pupils.
References
Expertise requires building sophisticated chunk recoding schemes
In many fields, experts become experts mostly by developing more sophisticated mental representations (Mental representations, after Ericsson and Pool), which amounts to increasing the size of their mental chunks (“Chunks” in human cognition). This increases their information processing capacity (Human channel capacity increases with bits-per-chunk, Recoding can increase chunk size). This happens through practice: Good practice encodes more effective chunk recoding schemes
What sets expert performers apart from everyone else is the quality and quantity of their mental representations.
(Ericsson and Pool, 2016, p. 62, not well-cited)
For example, the model developed by Simon and Gilmartin (1973) suggests that chess masters have encoded order tens of thousands of chunks. (See also Chase and Simon - Perception in chess)
Knowledge work often requires solving search problems. Ericsson and Pool suggest that expert search performance comes from more complex chunk schemas (2016, p. 70-72). The argument’s not made very strongly, but because Human channel capacity increases with bits-per-chunk, this would seem to explain superior culling and feedback-uptake performance.
References
Daily Routines and Productivity
My daily routine
Working independently means the burden and opportunity of creating my own structure. I’ve found ritual and routine essential to guide me.
When my days don’t go well, it’s often because something derailed me in the morning, and I never really got back on track. The high-order bit for my productivity is whether I complete a deeply-focused morning creative block. So my day is structured around making intense creative mornings happen.
- ~7:00: Wake; shower; walk, train, feed my dog; make coffee
- ~7:45 – 13:45: Uninterrupted morning working block
- Default environment: WiFi off; Forest on my phone; alone at my desk (It’s hard to hear yourself think)
- No meetings; working in 25m/5m pomodoros (see notes on Pomodoro technique); no extended breaks (I find maintaining momentum is more important than combatting fatigue)
- Start a Daily working log, write for a minute or so about how I’m feeling and my intentions for the day; look at my Menu for the week, get a sense of what I’d like to dig into.
- Dig into whatever seems most exciting creatively, usually sticking to one task for the entire block
- Details here vary depending on my projects and their status. Sometimes it’ll be My morning writing practice; sometimes it’s designing or building prototypes; sometimes it’s reading through a stack of material about something I’m trying to understand.
- If I start to run out of steam (usually around pomodoro 10), I’ll switch to administrative and procedural tasks, like email time: I’ll spend ~30m ~3 times a week replying to email (and generally ignore it otherwise).
- I eat lunch at my desk. It’s usually Chopped salad or Quiche.
- After this block, I think of my workday as effectively “done” and free myself from any further sense of obligation.
- ~2 – 3: Decompression walk with my dog. Often also walking with a friend or taking a meeting by phone at this time.
- ~3 – 6:30: Unstructured time. Napping, socialization, meditating, exercising, reading, more walking.
- ~6:30 – 8: Cooking, dinner.
- 8 – 10:30: (If a friend wasn’t over for dinner) piano, reading, time with Sara.
My morning writing practice
As part of my My daily routine, mornings are often spent writing and revising Evergreen notes. This is typically the most challenging work I do all day, so I like to do it when I have the most clarity and focus. It’s not for “note-taking” in a traditional sense—writing down other people’s ideas, or recording things that happened—it’s for developing ideas. (i.e. Most people use notes as a bucket for storage or scratch thoughts vs. Evergreen note-writing helps insight accumulate)
Unless I have something in mind that I’m particularly excited to write about, I usually begin by opening my writing inbox (A writing inbox for transient and incomplete notes) and flipping through those prompts and incomplete notes. If any strike me, I’ll draft Evergreen notes about them. This may happen over multiple days: I may flesh out a note considerably, then run out of steam and leave it in my inbox to finish another day.
If my inbox is relatively low, I’ll get out my memo pad (Pocket memo pad to capture into writing inbox while out) and fill my inbox with those notes. I don’t force it: if none of the prompts seem interesting, I’ll archive the ones which seem most boring and move on.
After working through my writing inbox, I’ll focus my attention on my primary creative projects and ask myself prompts like:
- what are the most important unknowns for this project?
- what new ideas am I excited about?
- what are the most interesting things I know about this project?
For these prompts, I’ll use my Daily working log as a scratch space, splatting a dozen or so one-liners into a haphazard bulleted list. After emptying my head, I’ll write about any that seem interesting. Usually that leads to rabbit holes which consume the rest of the session. I’ll add promising stragglers to my writing inbox for future days.
If those prompts don’t feel fruitful, I’ll use the time to Write about what you read to internalize texts deeply. I’ve usually got a backlog of books and articles I’ve read but for which I haven’t yet written Evergreen notes. If the prompts don’t feel fruitful for several days in a row, that’s a sign that I need to shake things up: my inputs aren’t high-variance enough, or I’m not giving myself the right kind of creative space, or I may need to re-evaluate my projects. My writing inbox should always feel like a cornucopia.
I take 5-minute breaks to get up and move around every 25 minutes, but even with those breaks, I usually can’t continue this practice longer than 2-3 hours. Sometimes I can do another session later in the day.
The high-order bit for my productivity is whether I complete a deeply-focused morning creative block
There’s so much productivity advice out there. You can get lost for months reading blogs of people optimizing their work with special journals and note-taking systems. I’ve spent tons of time measuring and optimizing my schedule throughout the day, trying to eke out an extra hour of work. All these things are useful, no doubt, and for some types of work they’re perhaps what’s most essential. But my experience has been that really one thing determines whether or not I have a “good day” at work. If I sit down at my desk in the morning for an uninterrupted 4-6 hour working session, and manage to “sink into” a deep state of focus and clarity on some creative project, I’ll probably have a great day creatively. If my mind is scattered and never settles on any particular problem, it rarely matters how much I “optimize” the rest of my day—that will not be a day of meaningful creative progress.
The first hurdle was simply clearing my morning schedule consistently, so that I always have a 5-6 hour contiguous working block. That’s pretty easy (for me) to do. And of course the next obvious obstacles are the things “productivity hackers” often write about: blocking distracting stuff on my computer really does help, so long as one isn’t too rigid about it. A writing practice helps me get clear on my projects and their goals, so that I’m reasonably likely to have some clear sense of what to do when I sit down at my desk.
But still, I often find that I sit down to my desk for four hours without ever really settling into any particular line of creative work. That’s not because I’m checking Twitter or my email or anything obvious like that. It’s just that I’ll jump around between various questions without ever committing to any one, or I’ll find myself solving some convenient problem which actually isn’t important, or my mind will simply wander. Or I’ll experience Dullness. This is a problem because Effective deep work depends on both time and intensity.
Meditation certainly helps here: Dullness and distraction in creative work may arise from the same causes as in meditation. The key does seem to be continuous metacognitive awareness. It also helps to Get curious and Get playful. And part of it, I observe, is just practice. I spent many years doing quite task-oriented work. The project of moving into increasingly challenging creative work is in part the project of getting comfortable being lost, and still making progress while remaining lost. Deep research requires a slower pace than tech industry work
Another strategy that’s helped: It’s easier to remain focused when collaborating live.
Related: It’s hard to hear yourself think.
Pomodoro technique
In reference to my own practice:
- 2023-02 February: now a usual day is: 25m pomo for morning pages (usually ~7:30 AM); 30m meditation; then 5-6x 55m pomos, stepping down if focus is poor; finishing ~2:30–3PM (2024-04 April update: still doing this, working well)
- 2022-11 November: now using 55m pomos when starting before 10AM; 45m before noon; 25m after, following data from Project - pomodoro experiment; practically speaking this usually means my morning goes: 55 55 45 45 45 25 25 25 ( ~5.5hrs active time in ~6hrs of clock time)
- Switched back to 25m/5m pomos in Jan 2022 as part of Program - finish my workday in the morning; focusing on longer contiguous blocks (6hr instead of 4hr, no long breaks) and higher-quality deep attention on harder tasks rather than trying to optimize efficiency
- Switched to 40m/5m pomos on 2021-07-12
- So goal for the morning block is 6 45m pomos.
- 8h of working time is 12 pomos
- 55m of break time spread within those 12
- Switched to 35m/5m pomos on 2020-01-22
- Goal for the morning with this block size is 7 40m pomos
- 8h of working time is ~14 pomos
- 65m of break time spread within those 14
Short assignments, after Anne Lamott
In Bird by Bird, the first strategy Anne Lamott shares is what she calls “short assignments”, which she visualizes with a tiny picture frame:
…all I have to do is to write down as much as I can see through a one-inch picture frame. This is all I have to bite off for the time being. All I am going to do right now, for example, is…to describe the main character the very first time we meet her, when she first walks out the front door and onto the porch. I am not even going to describe the expression on her face when she first notices the blind dog sitting behind the wheel of her car—just what can see through the one-inch picture frame, just one paragraph describing this woman, in the town where I grew up, the first time we encounter her.
I find this a helpful frame when I’m struggling with research. Sometimes the problem I really care about—what I’m really curious about—feels too enormous. It overwhelms me, and I don’t see how to get a handle on it. I’ll distract myself with Displacement activity because I don’t feel like I have a path to making progress.
How applicable is this to research? Anne Lamott writes:
E. L. Doctorow once said that “writing a novel is like driving a car at night. You can see only as far as your headlights, but you can make the whole trip that way.” You don’t have to see where you’re going, you don’t have to see your destination or everything you will pass along the way. You just have to see two or three feet ahead of you. This is right up there with the best advice about writing, or life, I have ever heard.
I feel a bit conflicted about this. One the one hand, deep understanding often feels like it comes from accumulating lots of small increments. On the other hand, synthesis and reframing are often the essential moves, and they seem somewhat incompatible with the one-inch frame.
Q. Lamott’s childhood story used to illustrate short assignments?
A. Her 10 y.o. brother weeping, overwhelmed by the hugeness of a big assignment on birds due the next day, which he’d put off for months; her father says: “Bird by bird, buddy. Just take it bird by bird.”
This is some piece of the motivation behind Evergreen note-writing helps insight accumulate.
Deep research requires a slower pace than tech industry work
Anyone working in an industry for a while will become accustomed to that culture—its processes, its norms, its values, its tacit knowledge. Much of this is incredibly valuable, of course, but these ideas can also represent constraints. There are some important impedances here between tech industry culture and research culture. In particular, tech culture is calibrated to a much faster pace. This can lead to impatience or early abandonment when confronting problems which require a researcher’s pace.
A “huge project” for a Silicon Valley tech person may be a year or two long; a “huge project” for a researcher may last a decade. Persistence with a difficult problem may require tens of hours for a tech person and hundreds of hours for a researcher. It’s not that the tech people are constitutionally lazy or something like that: in that industry, it’s usually a bad idea to spend many hundreds of hours thinking about a single problem. But foundational insights often do require more patient, focused thought than tech culture would support, so people from tech culture may struggle to sit with a fundamental problem for the time required to make progress. More broadly: San Francisco tech culture makes research hard.
Personally, coming from the tech industry, I’ve noticed that my expectations around the pace of progress are seriously miscalibrated for many research problems. I’ll feel like I’ve been banging my head against a question forever, but it’s only been a few tens of hours—that’s nothing in this space! If I continue expecting results at the rate I’ve been expecting them, I suspect that I’ll both miss the results I’m looking for and also drive myself nuts. An important aspiration for me is to get much more comfortable slowing down.
One related special case: Inappropriate time pressures often harm creative work
References
So many people give up, saying “I’m not good at proof”. When really {it has nothing to do with not being good at proof, but at not being good at dealing with being stuck and uncertain, and learning from ideas that don’t immediately succeed}.
As Xerox Chief Scientist Jack Goldman told Xerox execs: “If you hire me, you will get nothing of business value in five years. But if you don’t have something of value in ten years, you’ll know you’ve hired the wrong guy.”
San Francisco tech culture makes research hard
I love San Francisco dearly. I love the optimism, the energy, the earnestness, the ambition. But most of the time, the culture here isn’t really what I need to make progress in my research. Often, it’s actively harmful.
The main thing I struggle with is deliberateness. Deep research requires a slower pace than tech industry work. Others are moving very quickly, and conversation often implies that I should be too. It often feels hard to give longer-term ideas time and space to develop while living in this social context. When I talk to people about what I’m thinking about or where I’m stuck, the questions they ask are usually about action, motion—and, often, incrementalism. Sometimes the stereotypical bias to action can be very helpful, even for a researcher! But often these framings aren’t what’s needed. Often I just need to sit with the problem for another hundred hours. And what I need from a conversation partner is a provocative question, a creative reframing, a thoughtful reference, etc.
Another issue is the prevailing relationship to understanding. People here like to be knowledgeable, in the sense of being able to discuss topics at parties or blog about them. But there’s often a pervasive shallowness at play. The social epistemics of what it means to know something are weak, instrumental. People want to understand something enough to use others’ results, not necessarily to discover something new. When I say “what does it mean to learn something?” my epistemics are very different from almost all my conversation partners here. This means conversation here rarely helps—and in fact often distorts my sense of what I’m looking for.
Social capital dynamics here are misaligned with what I’m aiming for. People are celebrated for shipping, for growing, for huge $ amounts, for going viral, etc. Those things are all fine, but they’re not exactly what I’m doing. My big achievement for a month might be to understand some important new detail of a research project. This culture is not terribly excited about that.
Other SF subcultures don’t seem to have this problem
Rob Ochshorn pointed out something interesting: the Bay Area food culture sure seems to value deliberateness. “Slow food”, after all! Also consider John Muir, an absolute hero of the region, and the associated Sierra Club. It’s interesting that these subcultures can co-exist. Maybe I can give myself an infusion of these “slower” cultures while still playing in the tech scene.
References
Memory and Cognition
Many people view memory as unimportant to deep creative work
When telling others that Spaced repetition memory systems make memory a choice, people often react quite negatively: “What’s the point of memorizing all that stuff? Rote knowledge isn’t what matters: I want conceptual understanding, creativity, artistry, etc!”
One response to this is to point out that in fact, Spaced repetition memory systems can be used to develop conceptual understanding. But it’s also important to take seriously the claims against the value of memorization. Some responses there:
- Basic cognitive tasks rely on memory
- Some kinds of creative insight depend on long-term memory
- Memory augmentation may make it easier to learn complex topics by decreasing working memory load
- Deep understanding requires detailed knowledge of fundamentals
- Expertise requires building sophisticated chunk recoding schemes
- People underrate the importance of memory because of bad experiences with rote memorization in school
- Memory systems help you focus on deeper engagement by automating away rote memorization
- Memory augmentation can accelerate the unpleasant early stages of learning a subject
More generally: Spaced repetition and creativity
References
How rapidly do people forget practical knowledge?
This is an intentionally quite vague question.
In Quantum Country: QCVC questions are initially forgotten at very different rates
A more conservative figure in medicine:
Medical students forget roughly {25–35} percent of basic science knowledge after {one year}, more than {50} percent by {the next year} (Custers, 2010), and {80–85} percent after {25 years} (Custers & ten Cate, 2011).
Reading natural text in a contrived experimental setting, Amlund et al (1986) find < 50% free recall minutes later, even when readers knew it was a test.
References
Spacing effect
You’ll remember material more reliably if you study it on separate occasions, with some space in between—rather than if you spend the same amount of time cramming it all in one evening.
To put this another way, successive reinforcements flatten the Forgetting curve, so you can wait longer and longer between each review. Well-timed spacings flatten that curve to a greater degree.
Spaced repetition memory system algorithms taken advantage of this to implement efficient learning systems.
Optimal spacing
The intersession intervals shouldn’t be increased without bound, since that decreases the likelihood that you’ll remember the material when reviewing, which in turn diminishes the reinforcement effect on your memory. The optimal ISI depends on the retention interval (RI)—the time between the final study session and the test.
Mozer and Lindsey (2016) derive this power-law relationship for individual study sessions from various empirical data sets:
Optimal ISI = 0.097 * RI^0.812
Which has this shape:
It looks roughly linear to me, honestly. Their data sets don’t include anything outside of a 1 year RI, so we shouldn’t trust the function beyond this range.
Also, this function doesn’t seem to fit Cepeda et al 2008’s data very well:
e.g. they found that OISI ~= 5 for RI = 35, but the power law fit finds OISI = 1.8, which empirically performed much worse in that study.
Collected empirical evidence
Kornel (2009), in an experiment involving GRE-type vocabulary:
Combining the three experiments, 90% of participants learned more in the spaced conditions than the massed conditions, whereas only 6% of participants showed the reverse pattern.
Possible explanatory theories for the spacing effect
- “Encoding variability”: spacing varies the context, which leads to richer, more diverse encodings representing those contexts
- Spacing demands less concentrated effort and focus than massed study.
- Less accessible memories are more reinforced by retrieval, which may enhance learning via Two-component model of memory
- “Predictive-utility”: if an item is typically retrieved on a short interval, your mind assumes it’s no longer needed after that interval elapses; longer intervals establish longer periods of need.
Q. What term is used in spacing effect studies to refer to the time between sessions?
A. Intersession interval (ISI)
Q. What does ISI stand for in spacing effect papers?
A. Intersession interval.
Q. What term is used in spacing effect studies to refer to the time between the final study session and the test?
A. Retention interval (RI)
Q. Distinguish “intersession interval” and “retention interval” in spacing effect studies.
A. The former refers to time between study sessions and the latter to the time between the last study session and the test.
Q. What’s the “spacing function” refer to in spacing effect literature?
A. Recall accuracy as a function of intersession interval
Q. What’s the characteristic shape of the spacing function?
A. A hill: a relatively sharp initial increase in accuracy followed by a slow decline.
Q. What’s the “optimal ISI” refer to relative to the spacing effect?
A. The peak of the spacing function: the ISI which produces the highest recall accuracy.
Q. The optimal ISI depends strongly on what other interval?
A. The retention interval (Cepeda et al, 2006; via Mozer et al, 2009)
Q. What’s the central claim of encoding variability theories for the spacing effect?
A. Spacing produces better recall because it encodes memory traces with a wider variety of psychological contexts, providing more opportunities for overlap with recall contexts.
Q. Why would having memory traces involving a wider variety of psychological contexts lead to more reliable recall?
A. Encoding specificity principle
Q. In encoding variability theory, why not increase the ISI without bound to get maximum context variability?
A. As ISI increases, retrieval becomes lossier because each study context overlaps less with the next.
Q. What’s the central claim of predictive utility theories for the spacing effect?
A. The mind “learns” how long memories are needed according to their access patterns; longer study intervals encourage longer storage.
References
Testing effect
When you try to recall some detail from memory, that act strengthens your memory of that detail. When exploited as a learning activity, this is called “retrieval practice.”
Experiment has demonstrated this effect even when the correct answer is not provided (though Retrieval practice may be less effective without feedback), and even when the test-taker is given “open-book” access to find a correct answer. The effect has been demonstrated in many fields and at many age levels.
This suggests a significantly different role for tests. In typical classrooms, teachers and students imagine that learning happens during lectures, or while reading the material. The tests are there to assess that learning. But in fact, the tests themselves are an important part of the learning process.
Versus other study activities
Retrieval practice leads to more durable long-term memory than simply studying material by e.g. re-reading, despite the fact that students will be less successful during practice itself
It also seems to lead to more durable memory
The testing effect may be diminished or inverted for immediate tests
Format
The testing effect is generally measured to be more pronounced for production tests (short answer, essay) than for discrimination (multiple choice / true-false) (e.g. Kang et al, 2007). This may be due to the Generation effect.
A few cites to follow up on here from Agarwal, P. K., Nunes, L. D., & Blunt, J. R. (2021). Retrieval Practice Consistently Benefits Student Learning: A Systematic Review of Applied Research in Schools and Classrooms. Educational Psychology Review.:
- Carpenter, S. K., Lund, T. J. S., Coffman, C. R., Armstrong, P. I., Lamm, M. H., & Reason, R. D. (2016). A classroom study on the relationship between student achievement and retrieval-enhanced learning. Educational Psychology Review, 28(2), 353–375.
- Niedermeyer, F. C., & Sullivan, H. J. (1972). Differential effects of individual and group testing strategies in an objectives-based instructional program. Journal of Educational Measurement, 9(3), 199–204.
- Weinstein, Y., Nunes, L. K., & Karpicke, J. D. (2016). On the placement of practice questions during study. Journal of Experimental Psychology: Applied, 22(1), 72–84.
n.b. that Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and synthesis. Psychological Bulletin, 144(7), 710–756 find a medium-large effect (d=0.58) for transfer across initial and final test formats
Reviews
Pan, S. C., & Rickard, T. C. (2018). Transfer of test-enhanced learning: Meta-analytic review and synthesis. Psychological Bulletin, 144(7), 710–756, focused specifically on Retrieval practice and transfer learning
Branwen, G. (2009). Spaced Repetition for Efficient Learning. Retrieved December 16, 2019, from https://www.gwern.net/Spaced-repetition
Commentary
Roediger, H. L., & Karpicke, J. D. (2018). Reflections on the Resurgence of Interest in the Testing Effect. Perspectives on Psychological Science, 13(2), 236–241. https://doi.org/10.1177/1745691617718873
Primary research
- Butler, A. (2010). Repeated Testing Produces Superior Transfer of Learning Relative to Repeated Studying. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36, 1118–1133.
- Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19(4–5), 528–558
- Karpicke, J. D., & Blunt, J. R. (2011). Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping. Science, 331(6018), 772–775
- Karpicke, J. D., & Smith, M. A. (2012). Separate mnemonic effects of retrieval practice and elaborative encoding. Journal of Memory and Language, 67(1), 17–29
- van Gog, T., Kester, L., Dirkx, K., Hoogerheide, V., Boerboom, J., & Verkoeijen, P. P. J. L. (2015). Testing After Worked Example Study Does Not Enhance Delayed Problem-Solving Performance Compared to Restudy. Educational Psychology Review, 27(2), 265–289
- Rohrer, D., Dedrick, R. F., Hartwig, M. K., & Cheung, C.-N. (2020). A randomized controlled trial of interleaved mathematics practice. Journal of Educational Psychology, 112(1), 40–52
- Agarwal, P. (2019). Retrieval Practice & Bloom’s Taxonomy: Do Students Need Fact Knowledge Before Higher Order Learning? Journal of Educational Psychology, 111, 189–209
- Examines relationship between factual learning and higher-order learning; finds that matched practice/test modalities generally does best (consistent with Transfer-appropriate processing); mixed practice may be a good compromise.
- Yeo, D. J., & Fazio, L. K. (2019). The optimal learning strategy depends on learning goals and processes: Retrieval practice versus worked examples. Journal of Educational Psychology, 111(1), 73–90
Queue
Roediger, H. L., & Butler, A. C. (2011). The critical role of retrieval practice in long-term retention. Trends in Cognitive Sciences, 15(1), 20–27. https://doi.org/10.1016/j.tics.2010.09.003
Elaborative encoding
“Elaborative encoding” describes what we do when we relate knowledge to existing memories or experiences. Making these connections is thought to improve recall, particularly when connections are made to especially distinctive and emotionally-connected targets.
It’s most often used in mnemonic methods, where people might remember numbers by relating them to celebrities, places, smells, etc.
Some Spaced repetition memory system users explicitly write prompts to promote elaborative encoding. Experiments by Karpicke and Smith (2012) suggest this may not be adding much, vs. retrieval practice alone. Related: Retrieval practice appears to be a more effective learning activity than elaborative encoding.
References
Queue
Bradshaw, G. L., & Anderson, J. R. (1982). Elaborative encoding as an explanation of levels of processing. Journal of Verbal Learning and Verbal Behavior, 21(2), 165–174. https://doi.org/10.1016/S0022-5371(82)90531-X
Span of working memory
A person’s span of working memory is the theoretical maximum number of items (“Chunks” in human cognition) they can remember simultaneously, without committing any to long-term memory.
==TODO: I’m conflating “immediate memory” and “working memory” here, without fully understanding the relationship. Some researchers seem to treat these terms interchangeably, while others
It’s not a precisely measurable quantity: the definition of an item is fuzzy, and subjects’ spans vary somewhat depending on context and item type. Usually experimenters will talk about a more specific working memory span, like “reading span,” which refers to a task in which {subjects read a sequence of sentences and try to remember the last word in each} (see review of task types, Conway et al, 2005).
While the value does vary by task and context, it appears to be surprisingly consistent. There’s considerable debate among cognitive psychologists about the mean. Early-to-mid-20th century experimenters (e.g. Crannell and Parrish, 1957) put the value around 5-7, but more recent experimenters suggest (e.g. Cowan, 2001) that the value is closer to {4}, and that higher values indicated subjects subtly recoding stimuli into larger chunks (see Human channel capacity increases with bits-per-chunk).
Related: Complex ideas may be hard to learn in part because their components overflow working memory
References
Channel capacity of humans as information processors
One way to examine the limits of human information processing is to ask how much information can a person can reproduce from some stimulus they observe. In this framing, we can model the observer as a communications channel using tools from information theory. This figure
A perfect communications channel could reproduce any input you gave it. In practice, most channels (including humans) produce more errors as inputs contain more information. The behavior is usually asymptotic: a channel transmits its input perfectly until some threshold. Past that threshold, which we call the {channel capacity}, the correlation between outputs and inputs falls, and the total number of bits of transmitted information remains constant.
Experiments on human Span of absolute judgment can be used to model humans in this way. Miller’s review (1956) of the empirical data suggested that human channel capacity for unidimensional stimuli is about {2.6} bits.
For example, here’s a figure from Miller (1956, p. 83), using experimental data from Pollack (1952, 1953) on human absolute judgment of pitches, reframed with an information-theoretic approach.
Q. If you know a subject’s span of absolute judgment (for single-item, unidimensional magnitudes), how would you find their channel capacity?
A. channel capacity = log_2(span of absolute judgment)
Q. Why is the span of absolute judgment related to human channel capacity by a log2 relation?A. Channel capacity is expressed in bits. If the span of absolute judgment is 8 categories, you need log2(8) bits to represent every state.
References
Human channel capacity increases with bits-per-chunk
One common workaround for Channel capacity of humans as information processors appears to be making a sequence of smaller observations, rather than a single complex absolute judgment. This only works if you can hold the sequence in your head, so it’s limited by your Span of working memory. Happily, Working memory span is mostly independent of item complexity. So you can increase your effective channel capacity by increasing the number of bits in each observed chunk (“Chunks” in human cognition).
In this figure depicting data from Pollack (1953), channel capacity expands almost linearly with bits-per-chunk
This effect is still limited by the Span of absolute judgment, so to expand bits-per-chunk beyond 5, you’ll need to make chunks multidimensional (Human channel capacity increases with stimulus dimensionality).
Q. How does human channel capacity vary for a sequence of elements, as the number of bits transmitted in each element increases?
A. It increases roughly linearly.
Q. Why does it matter that the span of working memory is roughly independent of the span of absolute judgment?
A. It suggests that we can hold more information in working memory by increasing the “chunk” size of the items held in memory.
References
Learning increasingly complex ideas may amount to forming larger effective chunk sizes
Complex ideas may be hard to learn in part because their components overflow working memory. But Human channel capacity increases with bits-per-chunk, and Recoding can increase chunk size. So when you’re finally able to learn something that’s eluded you, it may be because you’ve finally encoded large enough chunks (“Chunks” in human cognition) representing the constituents.
Miller suggested a narrower interpretation of this notion in his paper introducing the term (1956, p. 95):
It seems probable that even memorization can be studied in these terms. The process of memorizing may be simply the formation of chunks, or groups of items that go together, until there are few enough chunks so that we can recall all the items.
Using spaced repetition systems to see through a piece of mathematics - Michael Nielsen:
I think that what’s going on is what psychologists call chunking. People who intensively study a subject gradually start to build mental libraries of “chunks” – large-scale patterns that they recognize and use to reason. … Experts begin to think, perhaps only semi-consciously, using such chunks. The conventional representations – words or symbols in mathematics, or moves on a chessboard – are still there, but they are somehow secondary.
So, my informal pop-psychology explanation is that when I’m doing mathematics really well, in the deeply internalized state I described earlier, I’m mostly using such higher-level chunks, and that’s why it no longer seems symbolic or verbal or even visual. I’m not entirely conscious of what’s going on – it’s more a sense of just playing around a lot with the various objects, trying things out, trying to find unexpected connections. But, presumably, what’s underlying the process is these chunked patterns.
References
Complex ideas may be hard to learn in part because their components overflow working memory
Ideas typically build on other ideas. Some directly combine a few prior insights; others might be difficult to express without special notation or terminology. When an idea has few dependencies (low Element interactivity), you can introduce them all, then immediately express the idea: green means go; red means go; a green or red stoplight pointing right means you can or can’t turn right.
But if you’ve just been introduced to a zoo of new terms, you probably won’t absorb much from a sentence which uses many of those terms at once. You can’t juggle all those terms in working memory at once, and you haven’t had time to encode any of them more durably. And so: Memory augmentation may make it easier to learn complex topics by decreasing working memory load.
It’s not just a matter of timing. People seem to forget most of what they read, and they mostly don’t notice, so even if you gave yourself time to absorb prerequisite ideas, you’d likely forget many of them.
This may be part of why some topics are so challenging to understand. If you’re trying to learn quantum computing, that requires new notation, new terms, and new concepts. It’s as if you’re reading a book which begins in English, but you find more and more Spanish words and sentences sprinkled in, until suddenly you notice that the whole book’s in Spanish, and you don’t understand any of it. You’ll need to absorb the vocabulary and grammar before you can follow the plot.
Any relatively complicated activity requires holding more information in our heads than short-term memory allows.
(Ericsson and Pool, 2016, p. 61)
More concretely: Learning increasingly complex ideas may amount to forming larger effective chunk sizes
References
Memory augmentation may make it easier to learn complex topics by decreasing working memory load
Complex ideas may be hard to learn in part because their components overflow working memory. Given that Spaced repetition memory systems make memory a choice, memory augmentation may be one interesting solution.
Related:
- The same notion, from a more emotional perspective: Memory augmentation can accelerate the unpleasant early stages of learning a subject
- A related claim, specifically about deep expertise: Deep understanding requires detailed knowledge of fundamentals
- A possible part of an underlying mechanism here: Learning increasingly complex ideas may amount to forming larger effective chunk sizes
- Cognitive load theory makes more general predictions about learning being more difficult in these contexts, beyond memory limitations
- And in particular it suggests problems for topics with high Element interactivity.
- Another possible mechanism: Forward testing effect
References
What makes these subjects difficult? In fact, individually many of the underlying ideas are not too complicated for people with a technical background. But the ideas come in an overwhelming number, a tsunami of unfamiliar concepts and notation. People must learn in rapid succession of qubits, the bra-ket notation, Hadamard gates, controlled-not gates, and many, many other abstract, unfamiliar notions. They’re imbibing an entire new language. Even if they can follow at first, understanding later ideas requires fluency with all the earlier ideas. It’s overwhelming and eventually disheartening.
Memory augmentation can accelerate the unpleasant early stages of learning a subject
One common negative reaction to working with a Spaced repetition memory system is: “I’d much rather get my hands dirty: then I’ll naturally end up remembering whatever’s important.” (Enabling environments focus on creating opportunities for growth and action, not on skill-building) But when you’re first interacting with a new subject, it’s hard to take any meaningful steps at all: at least for a while, you often can’t hold enough of the new terms and ideas in your head simultaneously long enough to do anything meaningful (related: Novices in enabling environments often can’t do what’s enabled). In this context, a Spaced repetition memory system can help by accelerating you through this awkward, unpleasant stage to the point where you can actually have a meaningful experience with the material (related: Enabling environments’ activities directly serve an intrinsically meaningful purpose).
For instance, a meaningful way to learn French might be to have conversations with French speakers. But if you’ve just started, you don’t know enough of the language to have a real conversation—or at least, the experience may be so uncomfortable that you won’t repeat it. Memory systems can help you rapidly skip ahead to the learning stage in which a meaningful conversation becomes possible.
This is one good retort to Many people view memory as unimportant to deep creative work.
Related: The mnemonic medium works well for linear primers in established fields
Q. Why are SRMs often particularly helpful in the early stages of learning a subject?
A. In those stages, your grasp of the core terms and ideas is so wobbly that you can’t yet do anything meaningful. SRMs can accelerate you past that.
Q. Why might an SRM help you have meaningful experiences more rapidly if you’ve just started learning French?
A. It can help you more rapidly reach the point where you can have a conversation with someone in French.
References
Spaced Repetition
Spaced repetition memory systems make memory a choice
Memory is normally something which happens by chance. Often, it feels a bit like the object of a helpless prayer: you might be reading a book, and think to yourself “oh boy, I’d better remember this.”
One fascinating consequence of using a Spaced repetition memory system is that they make memory a choice. Once you’ve adopted a memory practice, if you want to remember something, you can simply cause it to happen: just take a few moments to write a question about it. Over the next few weeks, you’ll encode it durably into long-term memory. Moreover, it’ll only take you a few minutes cumulatively: Spaced repetition memory systems are extremely efficient. So not only do these systems make memory a choice, but they make the choice very low-stakes: Deciding to remember something with a spaced repetition system is (aspirationally) a lightweight gesture.
This radically changes one’s relationship with memory! Core practices in knowledge work are often ad-hoc, and “remembering details” is a good example of a typically ad-hoc practice. But spaced repetition memory systems are an authentic Executable strategy for remembering specific details.
I often visualize this property embodied in a magic wand. When you feel an impulse of interest arise within you, you just point the magic wand at the object, and you’ll remember it effortlessly. It’s a tool which serves your intellectual excitement.
One of the important pieces here is that you can use the same “move” with basically anything: Spaced repetition memory systems offer a generalized medium and context for practice.
They’re not as great a strategy for solving People seem to forget most of what they read, and they mostly don’t notice, since it’s quite effortful to choose the details to remember and to write effective questions for an entire text. This is where the Mnemonic medium aspires to help.
References
Spaced repetition memory systems are extremely efficient
Using a Spaced repetition memory system, you can memorize the answers thousands of questions in just a few minutes a day. This is the unintuitive consequence of an exponential at work in the Spacing effect: Spaced repetition yields (what feel like) exponential returns for small increases in effort.
Each time you remember an answer correctly, that question’s test interval increases geometrically, forming an exponential curve: 5 days, 2 weeks, 1 month, 2 months, 4 months, etc. If you add questions at a constant rate, and each question’s probability of occurring on a given day diminishes exponential with time, then you’ll have a roughly constant number of questions to review on a given day.
References
Spaced repetition yields (what feel like) exponential returns for small increases in effort
Spaced repetition memory systems are extremely efficient, but that phrase doesn’t get across the nature of the efficiency. There are lots of things you might try to improve yourself—reading more books, talking to experts, etc. These things will all help, of course, but they also yield diminishing returns fairly rapidly. Spaced repetition doesn’t work like this. If you spend a few extra minutes reviewing prompts about something you’ve read, you’ll get vastly more out of the material. Each time you subsequently review (constant time), you’ll be able to wait longer until the next review (exponential increase… or at least an increasing increase).
There are some problems with this observation. The exponential phenomenon can be largely seen as an artifact of the review schedule. It’s expanding exponentially, so memory stability seems to be expanding exponentially. But Plots of demonstrated retention exaggerate exponential benefit of practice. You really want some to look at some kind of continuous “memory stability” parameter, describing (say) the half-life or the time to fall to 90%. And then you want to look at the impact of repetition on that parameter.
That said, the model fit in Ye, J., Su, J., & Cao, Y. (2022). A Stochastic Shortest Path Algorithm for Optimizing Spaced Repetition Scheduling. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 4381–4390 for a language learning data set does indeed predict exponential increase in half-life with each repetition (well, compounding increases with a slowly falling geometric constant): Calculator Suite - GeoGebra
References
Given that the essay takes about 4 or so hours to read, this suggests that a less than 50% overhead in time commitment can provide many months or years of retention for almost all the important details in the essay.
This is the big, counterintuitive advantage of spaced repetition: you get exponential returns for increased effort. On average, every extra minute of effort spent in review provides more and more benefit. This is in sharp contrast with most experiences in life, where we run into diminishing returns. For instance, ordinarily if you increase the amount of time you spend reading by 50%, you expect to get no more than 50% extra out of it, and possibly much less. But with the mnemonic medium when you increase the amount of time you spend reading by 50%, you may get 10x as much out of it. Of course, we don’t quite mean those numbers literally. But it does convey the key idea of getting a strongly non-linear return. It’s a change in the quality of the medium.
Spaced repetition mechanics create a sense of effortlessness
The core mechanics of a Spaced repetition memory system remove decision-making and gumption from the critical path. Say that you’d like to study the molecular pathway of cell metabolism. Without an SRS, you’d need to make a plan like “I study cell biology on Tuesday evenings,” remember that plan, and summon the will to execute the plan repeatedly. But if you have an active SRS practice, you can throw some prompts into your library and be confident that you’ll see them again over time.
You don’t need to decide how often you’ll study them. You don’t need to exert willpower to study those particular prompts—only to show up for daily SRS practice. Your decision about how to mark a prompt also isn’t weighty: that choice just fiddles a knob; the prompt will reappear in any case. The feeling, in gestalt, is one of effortless action (Wu wei).
This is very unlike the feeling one has when, say, maintaining an inbox, which is full of weighty decisions: Triage strategies for maintaining inboxes (e.g. Inbox Zero) are often too brittle.
This feeling of effortless is one reason why I’m interested in Spaced everything: I’d like to bring it to more than just memory prompts.
Kawara, an inspiration-focused spaced repetition app, pitches this feeling well: “Kawara doesn’t let me forget, but it doesn’t pressure me to remember.”
There’s a connection here to “autoplay” functionality in YouTube or Netflix, wherein the software drives you, rather than the other way around. In those contexts, it’s a somewhat adversarial relationship; the aspiration here is to “program yourself” in a way you yourself would endorse. (see also Spaced repetition systems can be used to program attention)
Q. How do spaced repetition mechanics cultivate a sense of effortlessness?
A. By removing weighty decisions and demands on willpower.
References
Spaced repetition memory systems offer a generalized medium and context for practice
If your creative and intellectual goals are incredibly focused, then you can generally find a more compelling and effective way to practice than a Spaced repetition memory system. For instance, if you’re trying to learn a foreign language, you might seek out literature or films at your reading level and consume that instead—that would probably be much more interesting. But one of the big advantages of existing SRMs is that they’re fully general. If you can write a practice task for it, you can toss it into the bucket with everything else. And if you have a review habit, you’ll end up practicing it. You don’t have to start a new habit to make it happen. This generality is a key part of Spaced repetition memory systems make memory a choice.
It’s interesting to ponder, though, whether/how one might incrementally specialize systems like Anki for better targeted practice. Plugins and custom note types are a step in this direction. I believe we can probably do much, much better.
Deciding to remember something with a spaced repetition system is (aspirationally) a lightweight gesture
People who haven’t actually used a Spaced repetition memory system often think of it as a tool you might apply “when you want to memorize something.” But this is an awful way to understand how to use an efficient memory system. Without augmentation, explicitly memorizing information is quite onerous, so it’s not something people often do. It’s reserved for extremely important details, details which are worthy of memorization’s high costs.
But Spaced repetition memory systems are extremely efficient. Deciding to remember something is not a high-stakes decision; it’s a decision that’ll cost a fraction of a minute over the next couple years. These systems wouldn’t be very interesting if you only used them to memorize the kinds of material you already memorize, because most people don’t explicitly memorize very much. Not only do Spaced repetition memory systems make memory a choice, but they make it an almost costless choice. Emotionally, it’s closer to choosing what to highlight or write marginalia next to on a book page. And practically speaking, when a once-expensive resource becomes nearly costless, surprising things can happen (e.g. electricity).
The way I add material to memory systems feels like a gesture. It’s something I do habitually, often almost unconsciously. It’s usually not explicitly purposeful—more like a way of mentally “underlining” information. It’s much like how I use the Twitter “like” button. It’s not a bookmark; it’s not a vote; it’s not costly; liking a tweet is a habitual, unconscious way I indicate interest.
Spaced repetition systems can be used to program attention
Spaced repetition memory systems make memory a choice, but the computerized component’s value lies specifically in dynamically scheduling and selecting questions to be reviewed. In some sense, the efficacy of a Spaced repetition memory system comes from its power to program your attention (Programmable attention). Think: “{cron} for your mind.”
Manually making decisions about which cards to review would be far too taxing on a per-card basis. The transaction cost is too high. When that work is mostly outsourced, you can make a coarser decision—to devote your attention to SRS practice for 10 minutes—and then let your attention be directed by the machine within that block.
Systematically, we can generalize spaced repetition to:
- a priority queue of microtasks
- (for memory tasks, in SM-2: a simple due date)
- an interactive environment which presents sufficiently high-priority tasks
- (for memory tasks, a flashcard UI which shows “due” cards)
- feedback actions which modify a task’s subsequent priority
- (for memory tasks, forgotten / remember modify the next due interval)
Within a traditional flashcard-style system, you can use this observation to go far beyond memorization: see Spaced repetition memory systems can be used to prompt application, synthesis, and creation and Spaced repetition may be a helpful tool to develop or change habits. Spaced repetition prompt design is about designing tasks for your future self.
But the core concept—automatically arranging and presenting tasks according to some expanding schedule—can be instantiated in many interfaces and domains. I call this notion Spaced everything.
As a pianist, I have a huge number of technical exercises that I maintain: e.g. scales, argpeggios, and patterns played in variations across each key. I only want to work on exercises for 20-30 minutes a day. Which ones should I do? You can imagine a system which:
- keeps track of each exercise, including axes of variation (blocked, arpeggiated; keys; modes)
- priority is a function of both recency and the previous maximum tempo I achieved, relative to my target tempo
- presents a prioritized queue of 20-30 minutes worth of exercises
- perhaps the exercise durations are estimated by measuring my practice
- perhaps it can be configured to present at most 5 variations of any single exercise, to ensure breadth
- solicits feedback from me in terms of a subjective rating and/or a maximum tempo marking
It’s interesting to imagine a single interface malleable enough that I could define my piano exercises above as one sort of routine, and a SRS memory system as another routine—both special cases of a single general primitive.
Some examples:
- Incremental reading
- Timeful text
- Spaced repetition may be a helpful tool to incrementally develop inklings
- Spaced repetition can lower the stakes around destructive inbox-maintenance operations (to-do lists, email, reading lists, etc)
- Readwise
Related:
- For applications which can use the simple SRS flashcard format, see Unusual applications of spaced repetition memory systems
- Spaced repetition systems as catechism
- OS-level spaced repetition system
References
Related
Evergreen note maintenance approximates spaced repetition
Triage strategies for maintaining inboxes (e.g. Inbox Zero) are often too brittle, vs. using spaced-repetition to “approximate” inbox grooming.
I use this concept to engage with my implementation of A reading inbox to capture possibly-useful references
Programmable attention
Michael Nielsen has suggested that the review sessions of a Spaced repetition memory system don’t just help you remember things: it orchestrates your repeated attention over time across hundreds of tiny tasks, too many to manage by hand. Systems like these are a form of programmable attention.
More generally, environments can be designed to modify their occupants attention by e.g. lowering executive overhead, mitigating unhelpful habits or biases, orchestrating the attention of multiple people in concert, distributing known-good attentional strategies, etc. You use simpler forms of programmable attention all the time: inboxes with snooze and alarm features; bots which remind you of things; Twitter is a kind of programmable attention. What is the core properties of such systems? What is their potential reach?
Such systems are often focused on productivity, but I believe they can be used to support creative work—reading, thinking, expressing, problem-solving.
For more, seen specifically through the lens of spaced repetition: Spaced repetition systems can be used to program attention
This term is evocative, but it has unfortunate connotations of roboticism and alienation. I think I’ll ultimately want to find another.
- computer-supported attention? computer-augmented attention? (vague)
- attention playbooks (too static in implication)
- cybernetic attention (slightly more accurate, maybe, but also with unpleasant connotations)
- Michael has also suggested “designed attention” / “designer attention” and (2022-11-15) “algorithmic attention.”
- I like that “designer attention” evokes “designer drugs”—altered states of consciousness, superpowers—in addition to implying a wide possibility space for inventing arrangements of attention.
- “Algorithmic attention” places emphasis on the authored arrangement of the attention—the attention algorithm. Relative to other terms, “algorithm” also imples a more hands-off relationship, and perhaps opacity.
Maybe it’s better to focus on finding great terms for specific instantiations of programmable attention—e.g. “coordinated attention” for ideas around collective intelligence, etc.
References
Coaching is paying someone to program your attention
Twitter is a kind of programmable attention
When you choose whom to follow on Twitter, you’re choosing what types of mindsets and aesthetics to expose yourself to on a regular basis. You’re choosing what types of conversations to have. You’re choosing to be reminded regularly of certain things, and not of others. This is a kind of Programmable attention.
Spaced repetition memory systems can be used to develop conceptual understanding
The most obvious way to use a Spaced repetition memory system is to memorize simple facts: names, definitions, numerical constants, etc. The vast majority of memory system users appear to only ever use them for this purpose. But spaced repetition isn’t just useful for memorizing simple facts. The same mechanism can be used to deeply understand complex material, particularly if you write our own prompts (How important is it to write your own spaced repetition memory prompts?). Such conceptually-oriented prompts might test you on connections, implications, causes, consequences. These broader ideas don’t naively feel like facts which can be memorized—but they often are. Besides the benefit of memorizing these conceptual ideas, engaging with higher-level relationships in this way over time can keep you in contact with the topic and help you internalize it more deeply. (related: The mnemonic medium keeps readers in contact with material over time)
Writing good spaced repetition memory prompts is hard; writing good conceptual prompts is much more so. The techniques used to encode abstract, conceptual knowledge are not well or widely understood. See Important attributes of good spaced repetition memory prompts for more. Particularly relevant here: Spaced repetition memory prompts should encode ideas from multiple angles.
This is one important retort to Many people view memory as unimportant to deep creative work.
It’s worth noting, too, that the cogpsy basis of spaced repetition (the Testing effect) has been demonstrated for conceptual knowledge as well as for declarative knowledge (see Karpicke and Blunt, 2011). But should you practice integrative tasks for conceptual learning instead of atomic tasks (per Spaced repetition memory prompts should usually focus on one atomic unit)? Probably, at least in part: How complex should tasks be for test-enhanced learning?
It’s possible that this type of information may be much “cheaper” to remember with spaced repetition schedules: Conceptual information may have much slower optimal spaced repetition schedules.
References
Spaced repetition memory systems can be used to prompt application, synthesis, and creation
A Spaced repetition memory system like Anki is primarily designed to help people memorize a lot of declarative knowledge, like vocabulary. But the same mechanisms can be used to create relatively unorthodox cards which prompt application, synthesis, and creation.
One limit to these types of questions is that because you authored them, you have to leave the context relatively vague: “apply the lens of utilitarianism to a recent decision,” rather than “apply the lens of utilitarianism to the death penalty.” The latter question’s not very helpful if you wrote it: you would have already thought through the answer, so it’d really just be a memory prompt when you saw it again later. This limitation makes the idea described here promising: The mnemonic medium can help readers apply what they’ve learned through simple application prompts.
Related:
- Unusual applications of spaced repetition memory systems
- Spaced repetition may be a helpful tool to develop or change habits
- Spaced repetition systems can be used to program attention
References
Spaced repetition may be a helpful tool to develop or change habits
Imagine that you read an article which suggests something like this:
Going around in circles with a challenging prioritization exercise? If none of the factors seems especially decisive on their own, you’re probably not going to get anywhere with a big pro/con table. You’ll need to either create new choices, identify more powerful axes of evaluation, or accept that you can’t make a strongly-informed choice.
You could memorize pieces of this with a Spaced repetition memory system, but you can also use the same system to help “install” this habit in your mind. Here are some example questions you might try:
Q: Recall a situation in which you went around in circles endlessly trying to prioritize various options. (Generate one you haven’t thought about before for this question)
A: blank
Q: Imagine an unusual series of concrete steps you might take next time you find yourself stuck in an unclear prioritization exercise. (Generate a series of steps that you haven’t generated before)
A: blank
Q: Imagine a specific situation in which you would have trouble escaping a prioritization quagmire by finding a single decisive factor. (Generate one you haven’t thought about before)
A: blank
After you’ve answered this type of prompt about a specific habit a few times, it’s often helpful to make much more specific prompts to reinforce that habit. For example (via Kanjun, 2020-07-01): “When I pick up my toothbrush in the morning…” “…get my phone and open Anki.”
This is an example of Spaced repetition systems can be used to program attention.
References
Spaced repetition may be a helpful tool to incrementally develop inklings
I’m often struck by an interesting question or notion in conversation or on a walk. In many cases, I can’t write anything terribly insightful on that topic in that moment: I certainly can’t write a good Evergreen notes. I don’t have anything useful to say about the notion yet—it just seems awfully interesting.
What action should I take now? How can I arrange to develop that inkling over time? I could create a “to-do” or block out time to think about this question, but that’s often not what’s called for. Instead, often what I need is marination: let’s come back in a few days, see what bubbles up.
I can capture the notion in A writing inbox for transient and incomplete notes, but it’ll rapidly become a pile of unwieldy scraps which I’ll come to ignore (Inboxes only work if you trust how they’re drained).
Spaced repetition systems can be used to program attention, so such mechanisms might be helpful here. In such a system, I might:
- Sit down to my morning writing and see a small handful of writing prompts for the day, drawn from my writing inbox
- I can choose to append whatever’s top of mind about any of them (perhaps they obscure what was written previously until I’ve added new material)
- Once I’ve done that, I can mark the prompt as “fruitful,” meaning it’ll come up again relatively soon, or “unfruitful,” in which case the system will increase its interval substantially.
- Alternately, I might take a moment to convert one of these prompts into one or more Evergreen notes.
- Any prompts which I simply ignore will have their intervals increased, but perhaps not as substantially as explicitly-unfruitful prompts.
By taking advantage of the exponential nature of spaced repetition intervals, one could make incremental progress on potentially hundreds of prompts, while considering only a few on any given day.
This would represent a system for incremental thinking.
Related: Evergreen note-writing helps insight accumulate.
Implementations
Rice Issa has published a simple implementation.
References
Unusual applications of spaced repetition memory systems
While a Spaced repetition memory system is primarily designed to help people remember facts, their flashcard mechanism can be used for a variety of other purposes.
- Salience prompts
- Unusual objects for recall tasks, more about creating a context for reflection than about literal recall:
- Something clever or beautiful a writer or artist did: why did it work? What might have made them think of it? What was its effect on you?
- Flashes of insight: what was the critical observation which unlocked a profound realization? What was the context?
- Model-breaking moments: someone said something you didn’t expect them to say—it’s a different way of looking at the world. What made them say that? What’s the delta between that way of seeing and your own?
- Changes in intention: what made you realize that your old intention isn’t serving you well? Why wasn’t it serving you well? What’s your replacement? What difficult changes does this mean?
- Failures: very precisely, what was your error? What were the proximate causes of that error? What were the causes of those causes? When those causes are present in the future, what strategies do you intend to use to have a different outcome?
- Tough past decisions: what were the key factors that helped you make a call? How did you frame the decision? What were you surprised by in hindsight? (via MN, 2020-06-31)
- Good operational patterns (e.g. in email or meetings): what effect did the pattern have on how you and others felt or acted? When might it be appropriate (or not)? What seem to be important considerations for its success/failure? In what types of future situations do you intend to try the pattern? (via MN, 2020-06-31)
- Memorizing quotes: use intermediate prompts which e.g. present only the first letter of each word (via Divia Eden, 2020-07-01)
- Spaced repetition may be a helpful tool to develop or change habits
- Broader cognitive task types: Spaced repetition memory systems can be used to prompt application, synthesis, and creation
- Visualization exercises to reinforce happy memories; e.g. front: “visualize your trip to Trapani with Sara”; back: photo(s) (via Taylor Rogalski, 2020-06-11)
- See also Nick Cammarata with peak experiences: https://twitter.com/nickcammarata/status/1315129231102234625?s=20
- Prompts to stay in touch; e.g. front: “visualize your friend Rob; is there anything you’d like to say?”; back:
imessage://URL to send him a message if you like (via Taylor Rogalski, 2020-06-11) - Motor memory prompts; e.g. play a C# minor scale on your thigh
- Aesthetic kindling; e.g. front: an interesting image you found while browsing the web, perhaps with a non-prompt like “what do you find striking about this?”; back: nothing (see also Kawara, which centers on this type of prompt)
- Lightweight, optional, non-urgent tasks: “have you looked at who’s linked to your blog recently? maybe there’s something interesting there: (link)”
- Exposing yourself to thoughts you flinch away from: e.g. “Project X failed because I knew that Sal opposed it from the start, and I just let that fester instead of addressing it.” Using cloze deletions to make yourself complete the observation may be effective here. (via Divia Eden, 2020-07-01)
Outside of the simple flashcard format, the general spaced repetition mechanism can be applied to many domains; see Spaced repetition systems can be used to program attention for more. Related: Spaced repetition systems as catechism.
On emotional reinforcement
Ava (bookbear) suggests that love is, in many ways, about repetition—choosing the same person or idea or way of being over and over again.
I haven’t figured out how to write about this without it feeling too squishy, but spaced repetition has been a really interesting venue for experimenting with this sort of feeling. I have lots of prompts whose primary purpose is reinforcing my emotional connection to a person, or a place, or an ideal, or an idea. It sometimes feels like a misshapen emotional crutch, but sometimes it feels like it expands my capacity to love!
Spaced repetition systems as catechism
In James (“Brad”) DeLong - Quantum Country interview - 2019-11-22, Brad jokingly suggested that the Mnemonic medium represents a new kind of “catechism.” It’s an amusing comparison, but it’s also worth pondering seriously!
A typical example from the Westminster Shorter Catechism :
Q1: What is the chief end of man?
A1: Man’s chief end is to glorify God, and to enjoy Him for ever.
On their surface, catechisms are about memorizing doctrinal knowledge, but they also effect a change in identity through repeated exposure.
Likewise, a Spaced repetition memory system’s review sessions help readers remember the material of a book, but they also trigger readers to re-engage with material they’ve read over time. That’s practically useful because many readers will see new connections after they’ve sat with the content for a few weeks. But maybe it also fosters a change in identity: you’re not just a person who read that essay one time, you’re a “student of that topic” in some more continuous sense.
My own experiences here are mixed. Sometimes I’ll see a question for the twentieth time and answer by rote, with no emotional connection at all to the original source. Other times I’ll find myself wondering new questions about the topic, feeling gradually more “in contact with” that topic over time.
That’s all an indirect effect, creating a change in identity by memorizing details associated with that identity. But it may also be possible to use spaced repetition systems to program one’s identity more directly. Sticking with quantum computing for the moment, one simple example card might be: “At this instant, what unsolved question do I instinctively find most fascinating about quantum computing?” (there’s nothing on the back). If you saw this question on a regular basis over the next few months, would you identify with the problems of the space more personally?
You can imagine creating cards about new habits (“Think of a new concrete situation in which I’ll have trouble leaving space for others to speak.”) or values (“What’s an unusual recent situation in which you thought on the century scale?”); see Spaced repetition may be a helpful tool to develop or change habits. More exotic Spaced everything systems can be used to schedule arbitrary fine-grained tasks associated with some new identity, like iteratively reaching out to interesting people in a new field.
At this point, it feels like we’ve moved quite far away from catechism, but this Anglican catechism has the same flavor:
Q: What is your Name?
A: …
Q: Who gave you this Name?
A: My Godfathers and Godmothers in my Baptism; wherein I was made a member of Christ, the child of God, and an inheritor of the kingdom of heaven.
A nice suggestion from Michael Nielsen: (2023-04-04):
There’s a Stanford anthropologist who wrote a book “Making God real” who argues that it’s the repetition of those acts which makes God real to believers
Another related observation from Michael Nielsen (2021-11-10):
In everyday life an astounding number of things happen. A tiny few seem really significant. Memory systems let you distill those things out, so that you will return to them again and again. They’re a way of concentrating your experience.
Related: OS-level spaced repetition system, Unusual applications of spaced repetition memory systems
References
Spaced repetition prompt design is about designing tasks for your future self
Writing good spaced repetition memory prompts is hard, but here’s one useful mental model. When you make a prompt for a Spaced repetition memory system, you are giving your future self a recurring task. Prompt design is task design.
So if you’re writing prompts to help learn a particular idea, you must design tasks which reinforce your understanding of that idea when you perform them. If you’re writing prompts to support creative work, you must design tasks which cause you to notice possibilities or connections, and so on.
“Capturing” an idea with spaced repetition prompts is like translation
The process feels surprisingly similar to translating text between languages. When translating a passage, you’re searching for words which, when read, light up a similar set of bulbs in readers’ minds to those which might have been activated by the original language. It’s not a rote operation. If the passage involves allusion, metaphor, or humor, you won’t translate literally; you’ll try to find words which recreate the experience of reading the original for a member of a foreign culture.
When writing SRS prompts to help you internalize an idea, you’re performing something similar to language translation: which tasks, when performed, require lighting the bulbs which are activated when you have that idea “fully loaded” into your mind?
How do we go about designing such a task? If the idea is fairly simple, it may be possible to directly conceive a task which reliably lights all those bulbs. But when the idea has too many important facets, it’s hard to design a task which reliably stimulates all those elements. So it’s best to break concepts down into their simplest units: Spaced repetition memory prompts should usually focus on one atomic unit.
OS-level spaced repetition system
Computer operating systems have come with a predictable set of personal information management tools for decades: an address book, a calendar, an e-mail client, some basic note-taking function, files and folders, etc. These are structured differently from siloed “apps,” which typically aim to subsume some workflow from start to finish. This basic OS software is more general-purpose, each both a tool and a service, connected throughout the OS via API-powered integrations. You add an event to your calendar from an email, autocomplete a contact’s name within a chat app, save and open files to the same folder from many apps, and so on.
What if there were an “OS-level” Spaced repetition memory system? What if, rather than living “inside an app’s shoebox”, as in Anki and other existing tools, prompts were framed more like files in folders—readable and writable throughout the system?
Web articles could surface interleaved prompts, written by the author as in the Mnemonic medium or perhaps by readers as an annotation layer. These mnemonic annotations might be the authorial product of an individual with a strong perspective (as in a literary book review); or they might be crowdsourced, as on Genius / Hypothesis. You’d import these prompts as you read, just as your browser forms a history as you read.
Your PDF and e-book reader’s annotations could naturally be surfaced in this centralized SRS, rather than remaining siloed in some inaccessible sidebar.
When you jot notes in your daily meetings, you could tag key insights with a special tag to surface them to this system (as in The mnemonic medium can be extended to one’s personal notes)—perhaps a future word processor’s formatting bar would include buttons for bold, italic, underline… and “to be reviewed.”
Just as modern operating systems may create tentative calendar events or contacts based on chat messages or emails, the system may create tentative SRS prompts based on links you’ve bookmarked or phrases you’ve searched for repeatedly.
But these ideas become much more interesting once you think of SRS as useful for much more than memorization (see Spaced repetition memory systems can be used to prompt application, synthesis, and creation), and still more powerful when you consider that Spaced repetition systems can be used to program attention.
For example, you can raise your smart watch today and say: “remind me to write about my idea that SRS could be framed as an OS-level service.” That’s a one-time reminder. But with this OS-level SRS, you could raise your watch and say: “remember to reflect: what novel contexts might benefit from an OS-level SRS service?” That would create not a one-time “to-do” but a prompt for repeated reflection over time. (See Spaced repetition may be a helpful tool to incrementally develop inklings for more.)
References
Traditional spaced repetition memory prompts are atomized
The prompts in a Spaced repetition memory system are an unordered, unstructured set. Each prompt is intentionally quite fine-grained and atomic, since that’s what seems to work best for effective memorization ((Spaced repetition memory prompts should usually focus on one atomic unit). But this lack of structure creates a feeling of wandering through a forest, able to see only one leaf at a time.
In some domains, this type of atomization is appropriate. For instance, if you’re studying hiragana, or a large set of foreign-language vocabulary, the inter-prompt structure may not be terribly meaningful—you really do have a huge sack of atoms to memorize.
But in other domains—say, physics or math—the atoms only have meaning as part of a broader structure. When prompts in these domains work well, I suspect it’s because they’re hanging on some invisible, broader structure that’s already in the reviewer’s head.
This atomization is the primary reason that Spaced repetition memory prompts alone are a poor communications medium. It also limits their suitability for long-term personal notes: Existing spaced repetition systems discourage evergreen notes.
See also: The mnemonic medium gives structure to normally-atomized spaced repetition memory prompts.
Existing spaced repetition systems discourage evergreen notes
Though notes in a Spaced repetition memory system are atomic in the same way as Evergreen notes (Evergreen notes should be atomic), they’re in many ways too atomized (Traditional spaced repetition memory prompts are atomized). The form discourages incremental synthesis and distillation.
The questions float in an undifferentiated mist, detached from any intrinsically meaningful context and not linked to relevant neighbors (Evergreen notes should be densely linked), and not especially meant to be accessed except within the review experience. They’re not meant to be durable, growing units; they’re meant to be disposable flotsam. All this could be fine, if they had a clear relationship with a separate system for Evergreen notes, but they don’t.
Happily: The mnemonic medium can be extended to one’s personal notes.
References
I start to identify open problems, questions that I’d personally like answered, but which don’t yet seem to have been answered. I identify tricks, observations that seem pregnant with possibility, but whose import I don’t yet know. And, sometimes, I identify what seem to me to be field-wide blind spots. I add questions about all these to Anki as well. In this way, Anki is a medium supporting my creative research. It has some shortcomings as such a medium, since it’s not designed with supporting creative work in mind – it’s not, for instance, equipped for lengthy, free-form exploration inside a scratch space.
Spaced repetition memory prompts alone are a poor communications medium
Traditional spaced repetition memory prompts are atomized. Each little prompt must stand on its own, presentable in any order at any time. But that makes it difficult to use the prompts to effectively communicate an idea which is itself highly structured and ordered. In practice, ideas generally depend on other ideas, and emotional salience requires relating those ideas to a meaningful larger whole (contra Deep understanding requires (and is a result of) intense personal connection). Narrative excels in these regards (Narrative as cognitive scaffolding).
One important implication: Studying another person’s spaced repetition memory prompts is usually ineffective.
This observation is one of the primary motivations for the Mnemonic medium: The mnemonic medium gives structure to normally-atomized spaced repetition memory prompts
Writing Good Prompts
How to write good prompts: using spaced repetition to create understanding
Excerpt from M. C. Escher, Metamorphosis III (1968)
How to write good prompts: using spaced repetition to create understanding
Andy Matuschak
December 2020
Contents
-
The central role of retrieval practice
-
Properties of effective retrieval practice prompts
-
A recipe for chicken stock
-
Factual knowledge
-
Simple facts
-
Lists
-
Cues and elaborative encoding
-
Interpretation; the “more than you think” rule of thumb
-
Procedural knowledge
-
Conceptual knowledge
-
Open lists
-
Salience prompts
-
Prompt-writing, in practice
-
Iterative prompt-writing
-
Litmus tests
-
Revising prompts over time
This guide and its embedded spaced repetition system were made possible by a crowd-funded research grant from my Patreon community. If you find my work interesting, you can become a member to get ongoing behind-the-scenes updates and early access to new work.
Special thanks to my sponsor-level patrons:
Adam Wiggins,
Andrew Sutherland,
Bert Muthalaly,
Calvin French-Owen,
Dwight Crow, fnnch,
James Hill-Khurana,
Lambda AI Hardware,
Ludwig Petersson,
Mickey McManus, Mintter,
Patrick Collison, Paul
Sutter, Peter Hartree,
Sana Labs,
Shripriya Mahesh, Tim O’Reilly.
As a child, I had a goofy recurring daydream: maybe if I type just the right sequence of keys, the computer would beep a few times in sly recognition, then a hidden world would suddenly unlock before my eyes. I’d find myself with new powers which I could use to transcend my humdrum life.
Such fantasies probably came from playing too many video games. But the feelings I have when using spaced repetition systems are strikingly similar. At their best, these systems feel like magic.This guide assumes basic familiarity with spaced repetition systems. For an introduction, see Michael Nielsen, Augmenting Long-term Memory (2018), which is also the source of the phrase “makes memory a choice.” Memory ceases to be a haphazard phenomenon, something you hope happens: spaced repetition systems make memory a choice. Used well, they can accelerate learning, facilitate creative work, and more. But like in my childhood daydreams, these wonders unfold only when you press just the right sequence of keys, producing just the right incantation. That is, when you manage to write good prompts—the questions and answers you review during practice sessions.
Spaced repetition systems work only as well as the prompts you give them. And especially when new to these systems, you’re likely to give them mostly bad prompts. It often won’t even be clear which prompts are bad and why, much less how to improve them. My early experiments with spaced repetition systems felt much like my childhood daydreams: prodding a dusty old artifact, hoping it’ll suddenly spring to life and reveal its magic.
Happily, prompt-writing does not require arcane secrets. It’s possible to understand somewhat systematically what makes a given prompt effective or ineffective. From that basis, you can understand how to write good prompts. Now, there are many ways to use spaced repetition systems, and so there are many ways to write good prompts. This guide aims to help you create understanding in the context of an informational resource like an article or talk. By that I mean writing prompts not only to durably internalize the overt knowledge presented by the author, but also to produce and reinforce understandings of your own, understandings which you can carry into your life and creative work.
For readers who are new to spaced repetition, this guide will help you overcome common problems that often lead people to abandon these systems. In later sections, we’ll cover some unusual prompt-writing perspectives which may help more experienced readers deepen their practice.If you don’t have a spaced repetition system, I’d suggest downloading Anki and reading Michael’s aforementioned essay. Our discussion will focus on high-level principles, so you can follow along using any spaced repetition system you like. Let’s get started.
The central role of retrieval practice
No matter the application, it’s helpful to remember that when you write a prompt in a spaced repetition system, you are giving your future self a recurring task. Prompt design is task design.
If a prompt “works,” it’s because performing that task changes you in some useful way. It’s worth trying to understand the mechanisms behind those changes, so you can design tasks which produce the kind of change you want.
The most common mechanism of change for spaced repetition learning tasks is called retrieval practice. In brief: when you attempt to recall some knowledge from memory, the act of retrieval tends to reinforce those memories.For more background, see Roediger and Karpicke, The Power of Testing Memory (2006). Gwern Branwen’s article on spaced repetition is a good popular overview. You’ll forget that knowledge more slowly. With a few retrievals strategically spaced over time, you can effectively halt forgetting. The physical mechanisms are not yet understood, but hundreds of cognitive scientists have explored this effect experimentally, reproducing the central findings across various subjects, knowledge types (factual, conceptual, procedural, motor), and testing modalities (multiple choice, short answer, oral examination).
The value of fluent recall isn’t just in memorizing facts. Many of these experiments tested students not with parroted memory questions but by asking them to make inferences, draw concept mapsSee e.g. Karpicke and Blunt, Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping (2011); and Blunt and Karpicke, Learning With Retrieval-Based Concept Mapping (2014)., or answer open-ended questions. In these studies, improved recall translated into improved general understanding and problem-solving ability.
Retrieval is the key element which distinguishes this effective mode of practice from typical study habits. Simply reminding yourself of material (for instance by re-reading it) yields much weaker memory and problem-solving performance. The learning produced by retrieval is called the “testing effect” because it occurs when you explicitly test yourself, reaching within to recall some knowledge from the tangle of your mind. Such tests look like typical school exams, but in some sense they’re the opposite: retrieval practice is about testing your knowledge to produce learning, rather than to assess learning.
Spaced repetition systems are designed to facilitate this effect. If you want prompts to reinforce your understanding of some topic, you must learn to write prompts which collectively invoke retrieval practice of all the key details.
We’ll have to step outside the scientific literature to understand how to write good prompts: existing evidence stops well short of exact guidance. In lieu of that, I’ve distilled the advice in this guide from my personal experience writing thousands of prompts, grounded where possible in experimental evidence.
For more background on the mnemonic medium, see Matuschak and Nielsen, How can we develop transformative tools for thought? (2019).This guide is an example of what Michael Nielsen and I have called a mnemonic medium. It exemplifies its own advice through spaced repetition prompts interleaved directly into the text. If you’re reading this, you’ve probably already used a spaced repetition system. This guide’s system, Orbit, works similarly.If you have an existing spaced repetition practice, you may find it annoying to review prompts in two places. As Orbit matures, we’ll release import / export tools to solve this problem. But it has a deeper aspiration: by integrating expert-authored prompts into the reading experience, authors can write texts which readers can deeply internalize with relatively little effort. If you’re an author, then, this guide may help you learn how to write good prompts both for your personal practice and also for publications you write using Orbit. You can of course read this guide without answering the embedded prompts, but I hope you’ll give it a try.
These embedded prompts are part of an ongoing research project. The first experiment in the mnemonic medium was Quantum Country, a primer on quantum computation. Quantum Country is concrete and technical: definitions, notation, laws. By contrast, this guide mostly presents heuristics, mental models, and advice. The embedded prompts therefore play quite different roles in these two contexts. You may not need to memorize precise definitions here, but I believe the prompts will help you internalize the guide’s ideas and put them into action. On the other hand, this guide’s material may be too contingent and too personal to benefit from author-provided prompts. It’s an experiment, and I invite you to tell me about your experiences.
One important limitation is worth noting. This guide describes how to write prompts which produce and reinforce understandings of your own, going beyond what the author explicitly provides. Orbit doesn’t yet offer readers the ability to remix author-provided prompts or add their own. Future work will expand the system in that direction.
Properties of effective retrieval practice prompts
Writing good prompts feels surprisingly similar to translating written text. When translating prose into another language, you’re asking: which words, when read, would light a similar set of bulbs in readers’ minds? It’s not a rote operation. If the passage involves allusion, metaphor, or humor, you won’t translate literally. You’ll try to find words which recreate the experience of reading the original for a member of a foreign culture.
When writing spaced repetition prompts meant to invoke retrieval practice, you’re doing something similar to language translation. You’re asking: which tasks, when performed in aggregate, require lighting the bulbs which are activated when you have that idea “fully loaded” into your mind?
The retrieval practice mechanism implies some core properties of effective prompts. We’ll review them briefly here, and the rest of this guide will illustrate them through many examples.
These properties aren’t laws of nature. They’re more like rules you might learn in an English class. Good writers can (and should!) strategically break the rules of grammar to produce interesting effects. But you need to have enough experience to understand why doing something different makes sense in a given context.
Retrieval practice prompts should be focused. A question or answer involving too much detail will dull your concentration and stimulate incomplete retrievals, leaving some bulbs unlit. Unfocused questions also make it harder to check whether you remembered all parts of the answer and to note places where you differed. It’s usually best to focus on one detail at a time.
Retrieval practice prompts should be precise about what they’re asking for. Vague questions will elicit vague answers, which won’t reliably light the bulbs you’re targeting.
Retrieval practice prompts should produce consistent answers, lighting the same bulbs each time you perform the task. Otherwise, you may run afoul of an interference phenomenon called “retrieval-induced forgetting”This effect has been produced in many experiments but is not yet well understood. For an overview, see Murayama et al, Forgetting as a consequence of retrieval: a meta-analytic review of retrieval-induced forgetting (2014).: what you remember during practice is reinforced, but other related knowledge which you didn’t recall is actually inhibited. Now, there is a useful type of prompt which involves generating new answers with each repetition, but such prompts leverage a different theory of change. We’ll discuss them briefly later in this guide.
SuperMemo’s algorithms (also used by most other major systems) are tuned for 90% accuracy. Each review would likely have a larger impact on your memory if you targeted much lower accuracy numbers—see e.g. Carpenter et al, Using Spacing to Enhance Diverse Forms of Learning (2012). Higher accuracy targets trade efficiency for reliability.Retrieval practice prompts should be tractable. To avoid interference-driven churn and recurring annoyance in your review sessions, you should strive to write prompts which you can almost always answer correctly. This often means breaking the task down, or adding cues.
Retrieval practice prompts should be effortful. It’s important that the prompt actually involves retrieving the answer from memory. You shouldn’t be able to trivially infer the answer. Cues are helpful, as we’ll discuss later—just don’t “give the answer away.” In fact, effort appears to be an important factor in the effects of retrieval practice.For more on the notion that difficult retrievals have a greater impact than easier retrievals, see the discussion in Bjork and Bjork, A New Theory of Disuse and an Old Theory of Stimulus Fluctuation (1992). Pyc and Rawson, Testing the retrieval effort hypothesis: Does greater difficulty correctly recalling information lead to higher levels of memory? (2009) offers some focused experimental tests of this theory, which they coin the “retrieval effort hypothesis.” That’s one motivation for spacing reviews out over time: if it’s too easy to recall the answer, retrieval practice has little effect.
Achieving these properties is mostly about writing tightly-scoped questions. When a prompt’s scope is too broad, you’ll usually have problems: retrieval will often lack a focused target; you may produce imprecise or inconsistent answers; you may find the prompt intractable. But writing tightly-scoped questions is surprisingly difficult. You’ll need to break knowledge down into its discrete components so that you can build those pieces back up as prompts for retrieval practice. This decomposition also makes review more efficient. The schedule will rapidly remove easy material from regular practice while ensuring you frequently review the components you find most difficult.
Now imagine you’ve just read a long passage on a new topic. What, specifically, would have to be true for you to say you “know” it? To continue the translation metaphor, you must learn to “read” the language of knowledge—recognizing nouns and verbs, sentence structures, narrative arcs—so that you can write their analogues in the translated language. Some details are essential; some are trivial. And you can’t stop with what’s on the page: a good translator will notice allusions and draw connections of their own.
So we must learn two skills to write effective retrieval practice prompts: how to characterize exactly what knowledge we’ll reinforce, and how to ask questions which reinforce that knowledge.
A recipe for chicken stock
Our discussion so far has been awfully abstract. We’ll continue by analyzing a concrete example: a recipe for chicken stock.
A recipe may seem like a fairly trivial target for prompt-writing, and in some sense that’s true. It’s a conveniently short and self-contained example. But in fact, my spaced repetition library contains hundreds of prompts capturing foundational recipes, techniques, and observations from the kitchen. This is itself an essential prompt-writing skill to build—noticing unusual but meaningful applications for prompts—so I’ll briefly describe my experience.
I’d cooked fairly seriously for about a decade before I began to use spaced repetition, and of course I naturally internalized many core techniques and ratios. Yet whenever I was making anything complex, I’d constantly pause to consult references, which made it difficult to move with creativity and ease. I rarely felt “flow” while cooking. My experiences felt surprisingly similar to my first few years learning to program, in which I encountered exactly the same problems. With years of full-time attention, I automatically internalized all the core knowledge I needed day-to-day as a programmer. I’m sure that I’d eventually do the same in the kitchen, but since cooking has only my part-time attention, the process might take a few more decades.
I started writing prompts about core cooking knowledge three years ago, and it’s qualitatively changed my life in the kitchen. These prompts have accelerated my development of a deeply satisfying ability: to show up at the market, choose what looks great in that moment, and improvise a complex meal with confidence. If the sunchokes look good, I know they’d pair beautifully with the mustard greens I see nearby, and I know what else I need to buy to prepare those vegetables as I imagine. When I get home, I already know how to execute the meal; I can move easily about the kitchen, not hesitating to look something up every few minutes. Despite what this guide’s lengthy discussion might suggest, these prompts don’t take me much time to write. Every week or two I’ll trip on something interesting and spend a few minutes writing prompts about it. That’s been enough to produce a huge impact.
At a decent restaurant, even simple foods often taste much better than most home cooks’ renditions. Sautéed vegetables seem richer; grains seem richer; sauces seem more luscious. One key reason for this is stock, a flavorful liquid building block. Restaurants often use stocks in situations where home cooks might use water: adding a bit of steam to sautéed vegetables, thinning a purée, simmering whole grains, etc. Stocks are also the base of many sauces, soups, and braises.
Stock is made by simmering flavorful ingredients in water. By varying the ingredients, we can produce different types of stock: chicken stock, vegetable stock, mushroom stock, pork stock, and so on. But unlike a typical broth, stock isn’t meant to have a distinctive flavor that can stand on its own. Instead, its job is to provide a versatile foundation for other preparations.
One of the most useful stocks is chicken stock. When used to prepare vegetables, chicken stock doesn’t make them taste like chicken: it makes them taste more savory and complete. It also adds a luxurious texture because it’s rich in gelatin from the chicken bones. Chicken stock takes only a few minutes of active time to make, and in a typical kitchen, it’s basically free: the primary ingredient is chicken bones, which you can naturally accumulate in your freezer if you cook chicken regularly.
Recipe
- 2lbs (~1kg) chicken bones
- 2qt (~2L) water
- 1 onion, roughly chopped
- 2 carrots, roughly chopped
- 2 ribs of celery, roughly chopped
- 4 cloves garlic, smashed
- half a bunch of fresh parsley
- Combine all the ingredients in a large pot.
- Bring to a simmer on low heat (this will take about an hour). We use low heat to produce a bright, clean flavor: at higher temperatures, the stock will both taste and look duller.
- Lower heat to maintain a bare simmer for an hour and a half.
- Strain, wait until cool, then transfer to storage containers.
Chicken stock will keep for a week in the fridge or indefinitely in the freezer. There will be a cap of fat on the stock; skim that off before using the stock, and deploy the fat in place of oil or butter in any savory cooking situation.
This recipe can be scaled up or down to the quantity of chicken bones you have. The basic ratio is a pound of bones to a quart of water. The vegetables are flexible in choice and ratio.
Variations
For a more French flavor profile, replace the celery with leeks and add any/all of bay leaves, black peppercorns, and thyme. For a deeper flavor, roast the bones and vegetables first to make what’s called a “brown chicken stock” (the recipe above is for a “white chicken stock,” which is more delicate but also more versatile).
What to do with chicken stock
A few ideas for what you might do with your chicken stock:
- Cook barley, farro, couscous and other grains in it.
- Purée with roasted vegetables to make soup.
- Wilt hearty greens like kale, chard, or collards in oil, then add a bit of stock and cover to steam through.
- After roasting or pan-searing meat, deglaze the pan with stock to make a quick sauce.
To organize our efforts, it’s helpful to ask: what would it mean to “know” this material? I’d suggest that someone who “knows” this material should:
- know how to make and store chicken stock
- know what stock is and (at least shallowly) understand why and when it matters
- know the role and significance of chicken stock, specifically
- know some ways one might use chicken stock, both generally and with some specific examples
- know of a few common variations and when they might be used
Some of this knowledge is factual; some of it is procedural; some of it is conceptual. We’ll see strategies for dealing with each of these types of knowledge.
But understanding is inherently personal. Really “knowing” something often involves going beyond what’s on the page to connect it to your life, other ideas you’re exploring, and other activities you find meaningful. We’ll also look at how to write questions of that kind.
If you’re a vegetarian, I hope you can look past the discussion of bones: choosing this example involved many trade-offs.In this guide, we’ll imagine that you’re an interested home cook who’s never made stock before. Naturally, if you’re an experienced cook, you’d probably need only a few of these prompts. And of course, if you don’t cook at all, you’d write none of these prompts! Try to read the examples as demonstrations of how you might internalize a resource deeply without much prior fluency.
To demonstrate a wide array of principles, we’ll treat this material quite exhaustively. But it’s worth noting that in practice, you usually won’t study resources as systematically as this. You’ll jump around, focusing only on the parts which seem most valuable. You may return to a resource on a few occasions, writing more prompts as you understand what’s most relevant. That’s good! Exhaustiveness may seem righteous in a shallow sense, but an obsession with completionism will drain your gumption and waste attention which could be better spent elsewhere. We’ll return to this issue in greater depth later.
How to make and store chicken stock: factual and procedural knowledge
Chicken stock
Recipe
Variations
What to do with chicken stock
A few ideas for what you might do with your chicken stock:
We’ll begin with the “recipe” section of this passage, which describes how to make and store chicken stock. It’s interesting to contrast this section’s text with the more conceptual initial paragraphs. As a form, recipes already involve a somewhat more explicit knowledge structure than you’d find in ordinary prose. This will give us a bit of a scaffold to get started.
To know how to make chicken stock, you must know what ingredients to collect. We’ll begin there.
This type of knowledge is mostly factual. There aren’t a lot of concepts or relationships here: it’s mostly just a bunch of raw information you need to know.
Simple facts
We could write a prompt which simply asks: “What do you need to make chicken stock?” But this isn’t precise enough: should we recall the quantities or just the names of the ingredients? How much chicken stock are we making? This isn’t focused enough: because it’s asking for so many details simultaneously, it’s unlikely to sharply activate all the memories you want to reinforce. And because it’s asking for so much, it’s liable to lead to inconsistency and intractability: each time you answer, you’ll remember some details and forget others. The inconsistent activations will tend to erode your memory.
We’ll need to break the ingredients list down into the elements which must actually be learned.
The first line is about the chicken component. If you’d never heard of stock before, you could begin by simply clarifying what kind of chicken we use:
Q. What type of chicken parts are used in stock?
A. Bones.
This question focuses on just one detail. It’s precise about what it wants (“What type of chicken parts…?”), and it should produce a consistent answer over time. It’s plenty tractable, but it still demands the effort of retrieving the answer from memory.
Writing a simple factual prompt like that naturally tickles a neighbor you might consider adding: the explanation prompt. I write prompts like this when a detail seems likely to be challenging or when the explanation itself is interesting. A more experienced cook likely wouldn’t bother with the first question, but they might still find this one useful.
Q. Why do we use bones to make chicken stock?
A. They’re full of gelatin, which produces a rich texture.
The explanation question will reinforce the knowledge captured by the factual question. Perhaps more importantly, explanations make facts more meaningful. A prompt like this offers a hook to connect the fact to other ideas you may pick up in later cooking adventures. For instance, a Chinese dish of jellied chicken feet might inspire you to try using feet as some of the bones in your stock (as in fact, they’re the most gelatin-rich part).
Note that the answer is found in a different section of the text! Writing prompts often involves hopping around as you work to identify the puzzle pieces and put them together.
Now, this explanation prompt is a good first try, but it could be more precise. It would be reasonable to answer “Why do we use bones to make chicken stock?” with “Because bones are economical.” You want to write questions which cause you to unambiguously retrieve the information you have in mind. This more precise prompt is better:
Q. How do bones produce a chicken stock’s rich texture?
A. They’re full of gelatin.
You may notice that one of the prompts in this review set doesn’t satisfy the consistency property. That prompt isn’t trying to engage retrieval practice, at least not like we’ve seen so far. Perhaps it doesn’t work very well—such prompts are a bit of an experiment! We’ll discuss them more later.
Lists
Back to the ingredients list. Chicken stock obviously involves chicken and water: that follows from the definition of stock (which we’ll handle later). But then there are all the aromatic flavorings, which are more variable and depend somewhat on application. Understanding the ingredients in terms of functional grouping will help us internalize the structure of the recipe:
Q. Chicken stock is made with chicken, water, and what other category of ingredients?
A. Aromatics.
From here, we might ask:
Q. What are typical aromatics used in chicken stock?
A. onion, carrots, celery, garlic, and parsley
But unless you’re an experienced cook, you’ll probably find this prompt intractable, or you may remember ingredients inconsistently. Unordered lists like this can be challenging to translate into good prompts.
One good strategy is to create a set of questions which require you to fill in a missing element of the list:
Q. Typical chicken stock aromatics:
???
carrots
celery
garlic
parsley
A. Onion
Q. Typical chicken stock aromatics:
onion
???
celery
garlic
parsley
A. Carrots
… and so on. Note that I keep the list in the same order. When each element has a consistent (though arbitrary) location, you’ll end up learning the list’s visual “shape” to some extent as you repeat these prompts. That will help your recall.
Most spaced repetition software has a special function which can rapidly generate sets of fill-in-the-blank prompts like this. In the software interfaces, these prompts are often called “cloze deletions.” In each review session, the software will only ask you to fill in one blank. This behavior is important because without it, one variant would “give away” the answer to another.
Ultimately, of course, you want to be able to recall all the ingredients on demand, not just one. Much of the time, these single-element deletions will be enough to achieve that goal. But with complex ideas, you may find you need to add integrative prompts after you’ve thoroughly practiced their discrete components. In the case of lists, you can imagine that the system could ask you to fill in more blanks simultaneously as your memory of the individual entries improves. I don’t know of any general memory system which does that, but it seems worth trying. This is one example of a general opportunity to incorporate more sophisticated knowledge modeling into these systems.
Another way to help yourself understand lists like this is to write explanation prompts for each of the components: A quick answer: carrot provides vegetal sweetness; like salt, this sugar brightens other flavors.why is carrot a good aromatic for chicken stock? If you know the answer to this question for each ingredient, you’ll have an easier time generating the list on demand, perhaps without any of the cloze deletions. And as with simple facts, explanations make knowledge more meaningful. In this case, the recipe doesn’t say, so you’d need to do some research to write questions of this kind.
Cues and elaborative encoding
If you find yourself struggling with these prompts, it can be helpful to add a cue, like this:
Q. Typical chicken stock aromatics:
- onion - carrots - celery - garlic - **??? (herb)** A. Parsley
But make sure the cue doesn’t render the prompt trivial: it’s important that you exert some effort to retrieve the answer from memory. Consider this prompt:
Q. Typical chicken stock aromatics:
- onion - **??? (rhymes with parrots)** - celery - garlic - parsley A. Carrots
This prompt doesn’t require you to retrieve any knowledge about chicken stock: there’s only one vegetable which rhymes with “parrots.” You could answer this without having ever read the recipe. By contrast, the previous prompt’s cue—“herb”—leaves your memory with work to do. There are many herbs, and you’ll still need to remember which one the recipe specified.
That said, “rhymes with parrots” may be a useful mnemonic device to add to the answer, offering an association to help you in the future:
Q. Typical chicken stock aromatics:
- onion - **???** - celery - garlic - parsley A. carrots (rhymes with “parrots”: picture a flock of parrots flying with carrots in their mouths, dropping them into a pot of stock)
Such cues engage another memory phenomenon cognitive scientists have explored experimentally: you make information easier to recall when you connect it to other memories. This process is called elaborative encodingSee e.g. Bradshaw and Anderson, Elaborative Encoding as an Explanation of Levels of Processing (1982).. The members of an ingredient list can be difficult to relate to anything meaningful. In such cases, you can still leverage elaborative encoding by fabricating an association as a mnemonic device. Vivid associations work best, so it’s helpful to find relationships involving visuals, meaningful personal experiences, or emotions like humor and disgust.
Putting these mnemonic devices in the answer field is a useful general trick when the information you’re trying to remember seems relatively arbitrary or isolated. By putting the association in parentheses, you’re making it clear that the focus of the prompt is on remembering the answer: the association is an extra tidbit you might engage with, or not.
This particular prompt is unlikely to require any extra help, but if you really find yourself struggling, you can add a prompt specifically intended to reinforce the association:
Q. Mnemonic device for carrots in chicken stock?
A. rhymes with “parrots”: picture a flock of parrots flying with carrots in their mouths, dropping them into a pot of stock
Instead of fabricating a mnemonic device, you can also leverage elaborative encoding by adding imagery to your prompts. The same fill-in-the-blank idea applies quite well to images. You’d probably find this prompt much easier to remember over time than its textual equivalent:
Q. Chicken stock ingredients:
A. Carrots
Of course, cues and elaborative encoding add extra effort. You don’t need to use them all the time: you’d probably find that exhausting. But they’re useful techniques to deploy if you suspect a prompt will prove difficult to remember, or if you’ve noticed yourself struggling in practice.
Interpretation; the “more than you think” rule of thumb
Now we’ve covered the ingredients which go into the stock. What about quantities?
Well, you could try something like: “How much chicken bone is in a batch of chicken stock?” But what’s a “batch”? This question isn’t precise enough. We could appeal to the particular recipe, asking “How much chicken bone is in Andy’s recipe for chicken stock?” But that’s not actually what we want to know: the recipe notes that you’ll want to adjust the recipe for the quantity of chicken bones you’re using. Writing good prompts often involves interpretation, a first step to creating understanding beyond what’s explicitly written.
Q. What’s the ratio of chicken bones to water in chicken stock?
A. A quart of water per pound of bones
Q. How much onion to use in chicken stock per pound of chicken bones?
A. Half an onion per pound
Q. How much carrots/celery to use in chicken stock per pound of chicken bones?
A. 1 carrot/celery rib per pound
Q. How much garlic to use in chicken stock per pound of chicken bones?
A. 2 smashed cloves per pound
What about the parsley? Here’s another instance of interpretation: “a bunch” of parsley is not a consistent unit anyway. So I won’t bother writing prompts about parsley’s quantity; I’ll just grab a handful.
Notice how I’ve broken the ingredient list down into many questions here, each focused and precise. I’ve noticed that people often feel a compulsion to economize on the number of prompts they write. Prompts seem to carry a per-unit “price,” so people naturally try to write fewer questions which cover more ground. But that’s counter-productive. Unless you explicitly decide to exclude certain information, the number of “units of raw knowledge” is fixed, a constant of the territory. When you write coarser prompts in smaller quantity, you’re not reducing the amount you have to learn. You’re just making the material harder to review.
Prompts are cheaper than you probably imagine. An easy prompt will consume 10–30 seconds across the entire first year of practice, and much less in each subsequent year. Until you’ve internalized that observation, try to adopt this rule of thumb: write more prompts than feels natural.
Now, prompts are cheap, but they’re not free. Besides their time cost, they have an emotional cost: no one wants to spend time reviewing a bunch of boring material they already know. So if you’re writing prompts for a subject that’s already quite familiar, you should use fewer prompts—not because it’s always safe to write coarser questions for familiar topics, but because there’s less marginal knowledge you need to capture.
For example, you may notice that I haven’t written prompts about how the ingredients are to be prepared before they’re added to the stock. No need, I feel: roughly chopping vegetables and smashing garlic before using those ingredients as aromatics is the natural inference for most cooks who are serious enough to make stock. But if that’s not a trivial inference for you, prompt away.
Relatedly, the appropriate scale of a “focused” prompt depends on the scale of the concepts you’ve internalized. This particular set of aromatics is so deeply familiar as a group that I’d write prompts which treat it as a unit (“Italian aromatics”) instead of memorizing individual ingredients and quantities.
As you build fluency in increasingly complex concepts, you can write increasingly complex prompts while keeping each focused on what feels like a single detail. In fact, the ability to think in terms of increasingly complex “chunks” appears to be a significant part of what expertise actually is.For a compelling demonstration, see Chase and Simon, Perception in chess (1973), which experimentally demonstrates how chess masters operate in terms of larger chunks. Viewed through this lens, one role for memory systems is to accelerate the process of increasing your effective chunk size in a topic.
.
Procedural knowledge
Now that we’ve got prompts for the ingredients, let’s look at the steps of the recipe. This is procedural knowledge—knowledge needed to perform specific tasks, more knowing how than knowing what.
In some sense, procedures are lists. So we can start by using the cloze-deletion method we used for the ingredient list:
Q:
???
Bring to a simmer on low heat (this will take about an hour). We use low heat to produce a bright, clean flavor: at higher temperatures, the stock will both taste and look duller.
Lower heat to maintain a bare simmer for an hour and a half.
Strain, wait until cool, then transfer to storage containers.
A. Combine all the ingredients in a large pot.
… and so on, for each of the four steps.
This is an awfully unfocused prompt. It includes many unimportant details which distract from the knowledge you actually intend to retrieve. In my experience, wordy prompts like these tend to dull my concentration and produce vague, distracted answers.
We can improve this prompt somewhat by simply removing words. If necessary, we can add extra prompts to cover any details we removed.
Q:
1. ??? 2. Slowly bring to a simmer 3. Maintain bare simmer for 90m 4. Strain, wait until cool, then store A. Combine all ingredients
This is better. But editing down the procedure helps us make a few observations about this knowledge. First: a few keywords (or word groups) carry the critical details of the procedure: slowly bring to a simmer, then maintain bare simmer for 90m. If you know those four bolded phrases, you basically know this procedure. The other words are just a skeleton. Which brings us to a second observation: the first and fourth steps aren’t worth writing prompts about. If you know what stock is (and we’ll get to that), you know you’re simmering a bunch of ingredients in a pot. When the stock is finished simmering, the final step is common sense. What else would you do?
After highlighting keywords in this way, prompt-writing often feels like playing Jeopardy: can you phrase that in the form of a question? Here’s a revised set of questions which capture my understanding of this procedure.
Q. At what speed should you heat a pot of ingredients for chicken stock?
A. Slowly.
Q. When making chicken stock, when should you lower the heat?
A. After the pot reaches a simmer.
Q. When making chicken stock, what should you do after the pot reaches a simmer?
A. Lower the temperature to a bare simmer.
Q. How long must chicken stock simmer?
A. 90m.
Procedures can often be broken down into keywords like this. What are the important verbs, and when should you move between them? What are the key adjectives, adverbs, subjects, objects?
In our stock recipe, the verbs aren’t very important: “bring,” “lower,” “strain.” You’re cooking ingredients in water at various temperatures, so those actions are obvious. But conditions or heuristics describing when to move between verbs are important: first when the water reaches a simmer, then after ninety minutes passes. And while it’s not worth capturing the fact that you’re heating the pot, the adverb “slowly” is indeed important.
The recipe is quite linear, but more complex procedures may branch, including conditions or special cases which could trigger some alternate path in the flowchart. Such predicate structures are usually worth capturing. If the branching is sufficiently complex, you might consider drawing a flowchart and using that in your prompt.
Is the keyword-based approach better than the list-based approach? That depends on how important the discrete details are and how intuitive they are for you. The keyword-based approach emphasizes the individual knowledge components more strongly, which is a good idea if precision is important. Also, I usually find focused questions like these more pleasant to answer. But if an outline is all you need, then the list approach is probably simpler.
It’s worth noting that in our editing, we’ve left behind a few useful details from the original recipe. Let’s re-add them with separate prompts:
Q. How long should it take to heat a batch of chicken stock (with 2lbs of bones)?
A. About an hour
This knowledge isn’t essential, but “heads-up!”-type information can be quite useful when learning procedures. If you know this detail, you’ll leave yourself enough time when making stock, you won’t be confused when your pot takes forever to heat up, and you might notice that you’re using too high a temperature if the pot boils in fifteen minutes.
Q. Why does Andy’s recipe claim we should prepare chicken stock over low heat?
A. “Brighter, cleaner” flavor.
Explanation-type prompts are especially valuable when studying procedures: they can help you avoid rote learning and build a deeper understanding. Note that in this particular case, we were only able to traverse one level of “why.” Why low heat? A cleaner flavor, says the recipe. Why does low heat produce a cleaner flavor? It doesn’t say. You might find yourself wondering what “brighter” or “cleaner” really even means. Writing this prompt revealed a gap in our understanding: how useful! We could dig into this question, but we’ll choose to keep moving for the moment. We indicate that by explicitly phrasing the answer as tentative, reliant on an external claim.
It’s helpful to add similar caveats to your prompts whenever you’re working with subjective, provisional, or incomplete information. In some sense, most of the prompts we’ve been writing are provisional: other recipes will probably do things somewhat differently. So we could phrase everything more tentatively, but in practice that’s more distracting than it’s worth. It’s usually enough to record where the information came from. Most spaced repetition systems have metadata fields you could use to link these prompts to the original chicken stock recipe.
Exercise: how to store chicken stock?
We’ve written prompts about how to make chicken stock. Now it’s your turn. Use what we’ve learned so far to write prompts representing your understanding of this paragraph on storage notes:
This will yield about 1.5 qt. Chicken stock will keep for a week in the fridge or indefinitely in the freezer. Once chilled, there will be a cap of fat on the stock; skim that off before using the stock, and deploy the fat in place of oil or butter in any savory cooking situation.
You can use this text box as a scratch area. What you write won’t be transmitted or saved.
Here are my prompts:
Click to reveal
Q. 2 lbs of chicken bones yields roughly ??? qt stock
A. 1.5 qt
Q. ??? lbs of chicken bones yields roughly 1.5 qt stock
A. 2 lbs
Q. How long will chicken stock keep in the fridge?
A. A week
Q. How long will chicken stock keep in the freezer?
A. Indefinitely
Q. Once chilled, what should I do before I use a fresh batch of chicken stock?
A. Remove the cap of fat
Q. What should I do with skimmed fat from chicken stock?
A. Keep and use as savory cooking fat
How do these prompts compare to yours? Did I include any details you didn’t cover? Did you cover any details I missed? How do those differences strike you? How does the scope of your prompts compare to these?
What stock is and why it matters: conceptual knowledge
A wheel that can be turned though nothing else moves with it, is not a part of the mechanism.
— Ludwig Wittgenstein, Philosophical Investigations
Now we turn our attention to the first few paragraphs of that recipe. Definitions seem like the simplest things to write prompts for: after all, isn’t that how flashcards are most commonly used in school?
For instance, what about a pair of definition prompts, like this?:
Q. What is stock?
A. A flavorful liquid building block.
Q. Culinary term for flavorful liquid building block?
A. Stock.
Leaving aside quibbles about focus, precision, and consistency, we could probably memorize the verbatim answer to the first question. The reverse definition can help you remember the name for this term, which is often useful. But the ability to parrot these answers isn’t at all the same as knowing what stock is.
Stock is a concept. Knowing the meaning of a concept like “stock” is different from high school flashcard knowledge—for instance knowing that “correre” in Italian means “to run.” To internalize a concept, you need to understand its components and relationships. Your goal is to design a set of prompts which collectively trace the edges of “stock.”
I’ll now introduce some useful lenses for understanding concepts. You won’t necessarily need to use every lens for each concept you encounter. Think of them as a toolkit for identifying elements which seem most important to you. Most of these example prompts are best suited to a novice cook. Experienced cooks usually already know what stock is, though they don’t necessarily know how to make it.
Attributes and tendencies: What makes stock, stock? What’s always, sometimes, and never true of stock?
Q. How are stocks usually made?
A. Simmering flavorful ingredients in water.
Q. Why don’t stocks usually have a distinctive flavor?
A. To make them more versatile.
Similarities and differences: Knowing what stock is requires knowing what relates and distinguishes it from other adjacent concepts.
Q. How is stock different from soup broth?
A. Soup broth has a more complete flavor; stock isn’t meant to stand on its own.
Parts and wholes: What are some examples of stocks? Are there important “sub-concepts” of stocks? Is “stock” a part of some broader category? Visualize a Venn diagram, even if the edges are fuzzy.
Q. Name at least three examples of stock:
A. e.g. chicken, vegetable, mushroom, pork
Q. Stock is rarely served directly; it’s best thought of as a ??? (construction metaphor)
A. Building block.
Causes and effects: What does stock do? What causes it to do that? What doesn’t it do? When is it used?
Q. Why do restaurants use stock as a cooking medium instead of water? (name two reasons)
A. Adds flavor, improves texture.
Q. Stocks are a common base for… (name at least two)
A. e.g. sauces, soups, braises
Q. Restaurants often use stock as a cooking medium where home cooks might use ???.
A. Water
Significance and implications: Why does stock matter? What does it suggest? Make the concept personally meaningful.
Q. What liquid building block explains why simple restaurant dishes are often tastier than home renditions?
A. Stock.
Q. What should I ask myself if I notice I’m using water in savory cooking?
A. “Should I use stock instead?”
This last prompt isn’t stated in the recipe: it’s an example of what might be an understanding of your own. Notice, too, that its aim is more behavioral than intellectual. We’ll have more to say about using prompts to change behavior in our section on how to use chicken stock.
Exercise: what chicken stock is and why it matters
As an exercise in modeling conceptual knowledge, try writing prompts representing your understanding of this paragraph introducing chicken stock and explaining its role:
One of the most ubiquitous and useful stocks is chicken stock. Now, the point of chicken stock isn’t to make everything taste like chicken. When used to prepare vegetables, for example, chicken stock makes them taste more complete, adding background voices which harmonize with the primary flavors. It also adds a luxurious texture because it’s rich in gelatin extracted from the chicken bones. Chicken stock takes only a few minutes of active time to make, and in a typical kitchen, it’s basically free: the primary ingredient is chicken bones, which you can naturally accumulate in your freezer if you cook chicken regularly.
You can use this text box as a scratch area. What you write won’t be transmitted or saved.
Here are some prompts I might write:
Click to reveal
Q. Chicken stock doesn’t make vegetables taste like chicken; it makes them taste more ??? (according to Andy’s recipe)
A. “complete”
Q. Chicken stock makes vegetables taste “more complete” by adding ??? which ??? with the primary flavors (music metaphor)
A. supporting voices; harmonize
Q. Besides improving flavor, chicken stock adds a luxurious ??? to dishes
A. texture
Q. Flavor and diet issues aside, why use chicken stock instead of vegetable or mushroom stocks?
A. Chicken stock has gelatin, which creates a luxurious texture
Q. Why is chicken stock economical?
A. Its main ingredient (bones) accumulates for free if you cook with chicken
Q. What should I do with the carcass of a roast chicken?
A. Freeze it and make chicken stock
There was one prompt which I found I wanted to write but couldn’t, at least while using only the information in the recipe:
Q. When to use chicken stock versus other types of meat stock?
A. ???
Prompt-writing can helpfully reveal such gaps in our understanding. You don’t need to stick with one resource: follow your nose; Google around; consult other references. Even if you don’t decide to follow up on the missing information immediately, you can guide future exploration by sensitizing yourself to feelings of curiosity and gaps in understanding.
You’ll notice that I don’t necessarily use all the lenses I introduced in the previous section. For instance, I didn’t feel there were any useful part/whole prompts to ask. And the most important attribute of chicken stock is that it’s made with chicken, but it would be silly to ask about that.
Using chicken stock: open lists and salience prompts
There’s no sense in making a batch of chicken stock unless you know how to use it. The recipe includes a list of suggestions. How might we create understanding around this type of knowledge?
Open lists
We learned some techniques for representing lists when we were writing prompts for our recipe’s ingredients. Could we use a similar technique here?
Q. Things to do with chicken stock:
???
wilt and steam hearty greens
make purée soups
deglaze pans
A. Cook grains in it
This prompt doesn’t work. Anything could reasonably go in that first slot. You might notice, too, that this list feels so much more arbitrary than the list of ingredients in chicken stock. The difference is that this is an open list. You don’t really mean to memorize this exact list, though it might be useful to name a few members on demand. No, if you keep cooking, you’ll be adding to this list your whole life. By contrast, the list of ingredients in our recipe has a fixed set of members: it’s a closed list.
I think of closed lists as a complex fact, almost an equation:
radius of earth = 6,371 km
chicken stock ingredients = onions, carrots, celery, garlic, parsley.
In fact, if you’re an experienced cook, you probably think of the ingredients in chicken stock as an open list—if that’s true, it’d be better to represent them that way! This is a common fate for closed lists in human-scale concepts. I like to think of open lists like tags—like the tags you might use in a system for digital bookmarks. My mental filing cabinet has a tag called “way to use chicken stock,” and I’ve fastened that tag to some notes about making purée soups.
When I encode this type of knowledge, I find three types of prompts consistently helpful. First I write prompts focused on each of the tagged items, linking from the instance to the tag. Then I might separately write prompts about the tag itself, perhaps inspired by patterns I notice in its instances. Finally, I often write a prompt which fuzzily links from the tag to its instances by asking for examples.
For example, this prompt links an instance to the tag:
Q. When puréeing vegetables to make soup, how can I produce a richer flavor without adding fat?
A. Thin the purée with chicken stock instead of water.
After we’ve written several prompts like this, a prompt about a pattern in the tag might suggest itself:
Q. What should I ask myself if I notice I’m using water in savory cooking?
A. “Should I use stock instead?”
It’s often useful to be able to summon examples of a tag on demand. We can write a prompt which fuzzily links the tag to instances:
Q. Name two ways you might use chicken stock.
A. e.g. cooking grains, steaming hearty greens, making purée soups, deglazing pans
This type of prompt is easy to write, so it’s tempting to write something similar and be done with it. But prompts like this usually don’t work well without other prompts supporting it. You’ll probably find yourself giving the same examples each time. In the absence of additional prompts, you’ll likely forget about the other instances. If you do change your answer each time, the prompt won’t satisfy the consistency property we’ve introduced, and interference effects may leave your memory unreliable without other supporting prompts.
When you’ve just learned a particularly open-ended concept—one which could apply to many instances—you can turn example-generating prompts like the one above into creative prompts like this one:
Q. Name a vegetable purée soup which you might try making with chicken stock (give an answer you haven’t given before)
A. e.g. potato, parsnip, celeriac, sunchoke, salsify, squash, carrot, pepper, lentil…
The novelty admonition is an interesting trick: “give an answer you haven’t given before.” Sure, after a year or two, maybe you’ll re-use a vegetable without realizing it. That’s fine. But note that you probably can’t write a prompt like this one about contexts to use chicken stock, unless you have enough prior experience to generate plenty of different answers.
This isn’t really a retrieval prompt anymore. Creative prompts are more like a textbook exercise, asking you to apply what you’ve learned in a new way. Unlike the other prompts we’ve seen, the goal here is to avoid retrieving an answer from memory: you’re meant to think creatively for a few moments. Since your answer’s different each time, retrieval practice won’t consistently reinforce your memory of any particular response. Instead, it will reinforce whatever knowledge you consistently use when generating an answer. Your novel responses may also make meaningful associations which strengthen your memory through elaborative encoding. And those associations may be particularly sticky because ofSee Slamecka and Graf, The Generation Effect: Delineation of a Phenomenon (1978). another memory phenomenon called the generation effect: you remember information better when you generated it yourself.
Michael Nielsen and I have experimented with application-focused spaced repetition prompts in Quantum mechanics distilled, but we don’t yet feel we understand them.I’ve described a few mechanisms which creative prompts might employ, but I should be clear that these prompts are much less well understood than the retrieval-focused prompts we’ve examined so far. What specific effects, if any, do such tasks have on our memory and understanding? Through what mechanisms? In what situations and with what properties are they most useful? What repetition schedule should be used for such tasks? These remain open research questions.
Salience prompts and the Baader-Meinhof phenomenon
Besides their impact on memory and understanding, many of the prompts we’ve been discussing about have another important effect: they keep you in contact with an idea over time.
Have you ever heard about something for the first time, then suddenly noticed it everywhere? Maybe you learn an unusual word like “mellifluous,” then spot it several times in the following days. In 1994 a pseudonymous internet commenter dubbed this the Baader-Meinhof Phenomenon (after being surprised to see two independent references to that terrorist group within a day). Stanford linguistics professor Arnold Zwicky suggests that this effect is a kind of selective attention: new ideas are particularly salient, so we notice them more readily. They haven’t really become more common—they just seem that way.
Sometimes this effect isn’t helpful. You can probably remember a time when a friend learned a new idea, and then everything became an (inappropriate) nail for their new hammer. But to really internalize a new idea, you need to bring it into your life and make meaning with it. In particular, if it’s a new skill, you probably haven’t really understood it until you’ve put it into practice several times on your own.
Salience typically fades over time. If you don’t soon have a chance to connect that new idea to something meaningful in your life, you may stop noticing opportunities so readily. The dynamic seems similar to the problem of forgetting knowledge over time. So one valuable use for spaced repetition prompts is to keep ideas salient, top of mind, over longer periods of time. Gwern Branwen has pointed out** In private communication. that such prompts are effectively trying to extend the Baader-Meinhof phenomenon and control it for a purpose.
We’ve already written a few prompts which focus on salience:
Q. What should I ask myself if I notice I’m using water in savory cooking?
A. “Should I use stock instead?”
Q. What should I do with the carcass of a roast chicken?
A. Freeze it and make chicken stock
Q. Name a vegetable purée soup which you might try making with chicken stock (give an answer you haven’t given before)
A. e.g. potato, parsnip, celeriac, sunchoke, salsify, squash, carrot, pepper, lentil…
The point of those prompts isn’t really to “know” those answers intellectually. It’s to cue certain ideas, which in turn may prompt new thoughts or create new behaviors. Viewed in this way, the point of repeating these prompts over time is to keep the relevant ideas salient until they have a chance to connect to something meaningful in your life. As economist Brad DeLong suggests, review sessions are surprisingly like a secular catechism.
Many of the Orbit prompts in this guide are of this kind. They’re meant to keep you in contact with these ideas until you can make sense of them as you write your own prompts. Odd as it may seem, I often write such prompts about my own ideas in the course of my creative work. They help me muse on an inkling or question over weeks and months, until it can hopefully grow into something more. This is one way prompt-writing can create understanding which extends beyond simply capturing knowledge from a text.
It’s often helpful to phrase salience prompts around contexts where those ideas might be meaningful in your life. For example, many cooks usually buy individual parts of a chicken instead of whole birds. Perhaps you’d like to start buying whole chickens so that you can use the bones for stock. This prompt demonstrates how you might target a specific situation:
Q. To help keep my freezer full of chicken stock, I’ll make sure to ??? instead of ??? when buying chicken.
A. buy whole birds; buying parts
This example helps illustrate a broader issue. Just because you can answer a factual question about an idea, that doesn’t mean the idea will spontaneously occur to you when it’s useful.This is one facet of a broad problem in educational psychology called transfer of learning: how do people transfer what they learn in one situation to a different one? I don’t have a well-grounded understanding of the difference, but in my experience, context-laden prompts (like this last example) help the leap from theory to practice. I suspect this is another reason it’s important to densely connect new ideas to old ones, as we did in the conceptual knowledge section: roughly speaking, more connections means more opportunities to trigger new knowledge.
What makes a spaced repetition prompt most likely to change your behavior or prompt new thoughts? What schedule should one use for repeating such prompts? If the objective isn’t memory, the schedule shouldn’t be tuned by “forgotten” and “remembered” buttons—so what should replace them? These remain open questions.
Exercise: variations
The chicken stock recipe includes a paragraph on variations, which are in some sense an open list. Write prompts representing your understanding of the knowledge in this paragraph:
For a more French flavor profile, replace the celery with leeks and add any/all of bay leaves, black peppercorns, and thyme. For a deeper flavor, roast the bones and vegetables first to make what’s called a “brown chicken stock” (the original recipe is for a “white chicken stock,” which is more delicately flavored, making it versatile but less intense).
You can use this text box as a scratch area. What you write won’t be transmitted or saved.
Writing prompts, in practice
Iterative prompt-writing
This guide aspires to demonstrate a wide variety of techniques, so I’ve deliberately analyzed the chicken stock recipe quite exhaustively. But in practice, if you were examining a recipe for the first time, I certainly wouldn’t recommend writing dozens of prompts at once like we’ve done here. If you try to analyze everything you read so comprehensively, you’re likely to waste time and burn yourself out.
Those issues aside, it’s hard to write good prompts on your first exposure to new ideas. You’re still developing a sense of which details are important and which are not—both objectively, and to you personally. You likely don’t know which elements will be particularly challenging to remember (and hence worth extra reinforcement). You may not understand the ideas well enough to write prompts which access their “essence”, or which capture subtle implications. And you may need to live with new ideas for a while before you can write prompts which connect them vibrantly with whatever really matters to you.
All this suggests an iterative approach.
Say you’re reading an article that seems interesting. Try setting yourself an accessible goal: on your first pass, aim to write a small number of prompts (say, 5-10) about whatever seems most important, meaningful, or useful.
I find that such goals change the way I read even casual texts. When first adopting spaced repetition practice, I felt like I “should” write prompts about everything. This made reading a chore. By contrast, it feels quite freeing to aim for just a few key prompts at a time.As Michael Nielsen notes, similar lightweight prompt-writing goals can enliven seminars, professional conversations, events, and so on. I read a notch more actively, noticing a tickle in the back of my mind: “Ooh, that’s a juicy bit! Let’s get that one!”
If the material is fairly simple, you may be able to write these prompts while you read. But for texts which are challenging or on an unfamiliar topic, it may be too disruptive to switch back and forth. In such cases it’s better to highlight or make note of the most important details. Then you can write prompts about your understanding of those details in a batch at the end or at a suitable stopping point. For these tougher topics, I find it’s best to focus initially on prompts about basic details you can build on: raw facts, terms, notation, etc.
Books are more complicated: there are many kinds of books and many ways to read them. This is true of articles, too, of course, but books amplify the variance. For one thing, you’re less likely to read a book linearly than an article. And, of course, they’re longer, so a handful of prompts will rarely suffice. The best prompt-writing approach will depend on how and why you’re reading the book, but in general, if I’m trying to internalize a non-fiction book, I’ll often begin by aiming to write a few key prompts on my first pass through a chapter or major section.
For many resources, one pass of prompt-writing is all that’s worth doing, at least initially. But if you have a rich text which you’re trying to internalize thoroughly, it’s often valuable to make multiple passes, even in the first reading session. That doesn’t necessarily mean doubling down on effort: just write another handful of apparently-useful prompts each time. For a vivid account of this process in mathematics, see Michael Nielsen, Using spaced repetition systems to see through a piece of mathematics (2019).With each iteration, you’ll likely find yourself able to understand (and write prompts for) increasingly complex details. You may notice your attention drawn to patterns, connections, and bigger-picture insights. Even better: you may begin to focus on your own observations and questions, rather than those of the author. But it’s also important to notice if you feel yourself becoming restless. There’s no deep virtue in writing a prompt about every detail. In fact, it’s much more important to remain responsive to your sense of curiosity and interest.
Piotr Wozniak, a pioneer of spaced repetition, has been developing a system he calls incremental reading which attempts to actively support this kind of iterative, incremental prompt writing.If you notice a feeling of duty or completionism, remind yourself that you can always write more prompts later. In fact, they’ll probably be better if you do: motivated by something meaningful, like a new connection or a gap in your understanding.
Let’s consider our chicken stock recipe again for a moment. If I were an aspiring cook who had never heard of stock before, I’d probably write a few prompts about what stock is and why it matters: those details seem useful beyond the scope of this single recipe, and they connect to happy dining experiences I’ve had. That’s probably all I’d do until I actually made a batch of stock for myself. At that point, I’d know which steps were obvious and which made me consult the recipe. If I found I wanted to make stock again, I’d write another batch of prompts to recall details like ingredient ratios and times. I’d try to notice places where I found myself straining, vaguely aware that I’d “read something about this” but unsure of the details. As I used my first batch of stock in subsequent dishes, I might then write prompts about those experiences. And so on.
Litmus tests
While you’re drafting prose, a spell checker and grammar checker can help you avoid some simple classes of error. Such tools don’t yet exist for prompt-writing, so it’s helpful to collect simple tests which can serve a similar function.
False positives: How might you produce the correct answer without really knowing the information you intend to know?
Discourage pattern matching. If you write a long question with unusual words or cues, you might eventually memorize the shape of that question and learn its corresponding answer—not because you’re really thinking about the knowledge involved, but through a mechanical pattern association. Cloze deletions seem particularly susceptible to this problem, especially when created by copying and editing passages from texts. This is best avoided by keeping questions short and simple.
Avoid binary prompts. Questions which ask for a yes/no or this/that answer tend to require little effort and produce shallow understanding. I find I can often answer such questions without really understanding what they mean. Binary prompts are best rephrased as more open-ended prompts. For instance, the first of these can be improved by transforming it into the second:
Q. Does chicken stock typically make vegetable dishes taste like chicken?
A. No.
Q. How does chicken stock affects the flavor of vegetable dishes? (according to Andy’s recipe)
A. It makes them taste more “complete.”
Improving a binary prompt often involves connecting it to something else, like an example or an implication. The lenses in the conceptual knowledge section are useful for this.
False negatives: How might you know the information the prompt intends to capture but fail to produce the correct answer? Such failures are often caused by not including enough context.
It’s easy to accidentally write a question which has correct answers besides the one you intend. You must include enough context that reasonable alternative answers are clearly wrong, while not including so much context that you encourage pattern matching or dilute the question’s focus.
For example, if you’ve just read a recipe for making an omelette, it might feel natural to ask: “What’s the first step to cook an omelette?” The answer might seem obvious relative to the recipe you just read: step one is clearly “heat butter in pan”! But six months from now, when you come back to this question, there are many reasonable answers: whisk eggs; heat butter in a pan; mince mushrooms for filling; etc.
One solution is to give the question extremely precise context: “What’s the first step in the Bon Appetit Jun ’18 omelette recipe?” But this framing suggests the knowledge is much more provincial than it really is. When possible, general knowledge should be expressed generally, so long as you can avoid ambiguity. This may mean finding another angle on the question; for instance: “When making an omelette, how must the pan be prepared before you add the eggs?”
False negatives often feel like the worst nonsense from school exams: “Oh, yes, that answer is correct—but it’s not the one we were looking for. Try again?” Soren Bjornstad points out that a prompt which fails to exclude alternative correct answers requires that you also memorize “what the prompt is asking.”
Revising prompts over time
It’s often tough to diagnose issues with prompts while you’re writing them. Problems may become apparent only upon review, and sometimes only once a prompt’s repetition interval has grown to many months. Prompt-writing involves long feedback loops. So just as it’s important to write prompts incrementally over time, it’s also important to revise prompts incrementally over time, as you notice problems and opportunities.
In your review sessions, be alert to feeling an internal “sigh” at a prompt. Often you’ll think: “oh, jeez, this prompt—I can never remember the answer.” Or “whenever this prompt comes up, I know the answer, but I don’t really understand what it means.” Listen for those reactions and use them to drive your revision. To avoid disrupting your review session, most spaced repetition systems allow you to flag a prompt as needing revision during a review. Then once your session is finished, you can view a list of flagged prompts and improve them.
The analogy to sentences is drawn from Matuschak and Nielsen, How can we develop transformative tools for thought? (2019).Learning to write good prompts is like learning to write good sentences. Each of these skills sometimes seems trivial, but each can be developed to a virtuosic level. And whether you’re writing prompts or writing sentences, revision is a holistic endeavor. When editing prose, you can sometimes focus your attention on a single sentence. But to fix an awkward line, you may find yourself merging several sentences together, modifying your narrative devices, or changing broad textual structures. I find a similar observation applies to editing spaced repetition prompts. A prompt can sometimes be improved in isolation, but as my understanding shifts I’ll often want to revise holistically—merge a few prompts here, reframe others there, split these into finer details. If you’ve attempted the exercises, you may notice that it’s easier to revise across question boundaries when composing multiple questions in the same text field. As an experiment, I’ve written almost all new prompts in 2020 as simple “Q. / A.” lines (like the examples in this guide) embedded in plaintext notes, using an old-fashioned text editor instead of a dedicated interface. I find I prefer this approach in most situations. In the future, I may release tools which allow others to write prompts in this way.Unfortunately, most spaced repetition interfaces treat each prompt as a sovereign unit, which makes this kind of high-level revision difficult. It’s as if you’re being asked to write a paper by submitting sentence one, then sentence two, and so on, revising only by submitting a request to edit a specific sentence number. Future systems may improve upon this limitation, but in the meantime, I’ve found I can revise prompts more effectively by simply keeping a holistic aspiration in mind.
In this guide, I’ve analyzed an example quite exhaustively to illustrate a wide array of principles, and I’ve advised you to write more prompts than might feel natural. So I’d like to close by offering a contrary admonition.
I believe the most important thing to “optimize” in spaced repetition practice is the emotional connection to your review sessions and their contents. It’s worth learning how to create prompts which effectively represent many different kinds of understanding, but a prompt isn’t worth reviewing just because it satisfies all the properties I’ve described here. If you find yourself reviewing something you don’t care about anymore, you should act. Sometimes upon reflection you’ll remember why you cared about the idea in the first place, and you can revise the prompt to cue that motivation. But most of the time the correct way to revise such prompts is to delete them.
Another way to approach this advice is to think about its reverse: what material should you write prompts about? When are these systems worth using? Many people feel paralyzed when getting started with spaced repetition, intrigued but unsure where it applies in their life. Others get started by trying to memorize trivia they feel they “should” know, like the names of all the U.S. presidents. Boredom and abandonment typically ensue. The best way to begin is to use these systems to help you do something that really matters to you—for example, as a lever to more deeply understand ideas connected to your core creative work. With time and experience, you’ll internalize the benefits and costs of spaced repetition, which may let you identify other useful applications (like I did with cooking). If you don’t see a way to use spaced repetition systems to help you do something that matters to you, then you probably shouldn’t bother using these systems at all.
Further reading
These resources have been especially useful to me as I’ve developed an understanding of how to write good prompts:
- Piotr Wozniak’s Effective learning: Twenty rules of formulating knowledge and the more detailed Knowledge structuring and representation in learning based on active recall tackle the same topic as this guide from a different perspective.
- Michael Nielsen’s Augmenting Long-term Memory: a thorough review of how spaced repetition works and why you should care; more details on how to practically integrate prompt-writing into your reading practices, particularly when reading academic literature; notes on creative applications; and much more.
- Michael Nielsen’s Using spaced repetition systems to see through a piece of mathematics demonstrates how to iteratively deepen one’s understanding of a piece of mathematics, using spaced repetition prompts as a lever.
For more perspectives on this and related topics, see:
- Soren Bjornstad’s series on memory systems helpfully covers many practical topics in maintaining a prompt library, including some advice on prompt-writing.
- Nicky Case’s How To Remember Anything Forever-ish introduces spaced repetition and covers some prompt-writing techniques through delightful illustrations.
- How can we develop transformative tools for thought? (from Michael Nielsen and me) discusses the challenges of prompt-writing with particular focus on the “mnemonic medium,” which involves embedding prompts in narrative prose.
Acknowledgements
My thanks to Peter Hartree, Michael Nielsen, Ben Reinhardt, and Can Sar for helpful feedback on this guide; to the many attendees of the prompt-writing workshops I held while developing this guide; and to Gwern Branwen and Taylor Rogalski for helpful discussions on prompt-writing which informed this work. I’m particularly grateful to Michael Nielsen for years of conversations and collaborations around memory systems, which have shaped all aspects of how I think about the subject.
This guide (and Orbit, its embedded spaced repetition system) were made possible by a crowd-funded research grant from my Patreon community. If you find my work interesting, you can become a member to get ongoing behind-the-scenes updates and early access to new work.
Special thanks to my sponsor-level patrons:
Adam Wiggins,
Andrew Sutherland,
Bert Muthalaly,
Calvin French-Owen,
Dwight Crow, fnnch,
James Hill-Khurana,
Lambda AI Hardware,
Ludwig Petersson,
Mickey McManus, Mintter,
Patrick Collison, Paul
Sutter, Peter Hartree,
Sana Labs,
Shripriya Mahesh, Tim O’Reilly.
Licensing and attribution
This work is licensed under CC BY-NC 4.0, which means that you can copy, share, and build on this essay (with attribution), but not sell it.
In academic work, please cite this as:
Andy Matuschak, “How to write good prompts: using spaced repetition to create understanding”, https://andymatuschak.org/prompts, San Francisco (2020).
Writing good spaced repetition memory prompts is hard
People regard flashcards as something trivial from their school days, so they don’t take writing them very seriously. But it’s awfully hard to write good prompts for a Spaced repetition memory system. For example, good prompts:
- access an idea from multiple angles
- capture one precise thought (likely reflective of “Chunks” in human cognition)
- avoid unintentional ambiguity
- are concise
- get to what really matters about the topic, not just what’s easy to memorize
For more, see: Important attributes of good spaced repetition memory prompts
It’s harder than people think
Unfortunately, it’s not obvious when the prompts you’ve written are bad, so people often don’t realize that their prompts are bad. This can cause them to underrate the performance or overrate the tedium of spaced repetition memory practice. More: To what extent do review sessions offer prompt-writing feedback?
One solution: The mnemonic medium may help scaffold prompt-writing through author-provided prompts
It’s taxing even if you know how
Even if one develops the skill to write good prompts, it’s quite time-consuming and cognitively taxing to do it. I believe that this is another significant barrier to widespread adoption.
One solution: The mnemonic medium supplies expert-authored prompts to remove the burden of prompt-writing. Or, maybe Using machine learning to generate good spaced repetition prompts from explanatory text.
References
Important attributes of good spaced repetition memory prompts
This note collects ideas about how to encode knowledge into Spaced repetition memory system prompts, both to support memory of facts but also to foster richer understanding (Spaced repetition memory systems can be used to develop conceptual understanding).
See How to write good prompts (2020) for a published manuscript on this topic.
Basic attributes of prompt-writing:
- Spaced repetition memory prompts should be concise
- Spaced repetition memory prompts should usually focus on one atomic unit
- Spaced repetition memory prompts should encode ideas from multiple angles
- Spaced repetition memory prompts should connect and relate ideas
- Spaced repetition memory prompts should unambiguously produce a specific answer
- Prompts must make clear what “shape” of answer they expect
- Spaced repetition memory prompts should ensure reviewers must retrieve answers from memory
- Avoid yes-no spaced repetition memory prompts
- Spaced repetition memory prompts should be written to discourage shallow “pattern matching”
- Beware set/enumeration SRS prompts
- Cloze deletion prompts seem to produce less understanding than question-answer pairs in spaced repetition memory systems
- Expert response heuristic, after Issa Rice
“Covering” material:
Meta / mental models:
- A pattern language of prompt-writing
- Rule of thumb: write more spaced repetition prompts than seems natural
- Spaced repetition prompt design is about designing tasks for your future self
Related: The “reflected essay” metaphor for the goal for the mnemonic medium
References
Spaced repetition memory prompts should usually focus on one atomic unit
If you’ve just learned a new way to cook peas, you might write a prompt for a Spaced repetition memory system like: “Q. How do I cook peas sous vide?” “A. 18m @ 70°C”. But I find that questions like this are usually a struggle—I’ll frequently forget the answer. Questions which draw on multiple independent ideas seem to trip over this problem.
Note, however, that the research literature on broader applications of the Testing effect paints a more ambivalent picture: How complex should tasks be for test-enhanced learning?
Why is it important that questions focus on one idea? Michael Nielsen (2018) suggests:
I suspect it’s partly about focus. When I made mistakes with the combined question, I was often a little fuzzy about where exactly my mistake was. That meant I didn’t focus sharply enough on the mistake, and so didn’t learn as much from my failure. When I fail with the atomic questions my mind knows exactly where to focus.
What constitutes “one idea”? That depends on your prior context; “one idea” seems to consist of one step on top of whatever knowledge you’ve already deeply internalized. Another way to look at that might be relative to “Chunks” in human cognition: a single idea is expressed in one “chunk,” whatever that might mean for you.
The fix in such instances is simple: break the question down into multiple simpler questions. Then, perhaps, add another question which integrates those two simpler questions. (This seems like an opportunity for Spaced repetition and knowledge modeling). The process of writing questions which focus precisely on one idea does seem to help sharpen my focus on the key elements of a topic (one angle on Writing one’s own spaced repetition prompts seems to promote understanding).
Wozniak (1999) supplies a good example (also demonstrating Spaced repetition memory prompts should be concise):
Ill-formulated knowledge - Complex and wordy
Q: What are the characteristics of the Dead Sea?
A: Salt lake located on the border between Israel and Jordan. Its shoreline is the lowest point on the Earth’s surface, averaging 396 m below sea level. It is 74 km long. It is seven times as salty (30% by volume) as the ocean. Its density keeps swimmers afloat. Only simple organisms can live in its saline waters
Well-formulated knowledge - Simple and specific
Q: Where is the Dead Sea located?
A: on the border between Israel and Jordan
Q: What is the lowest point on the Earth’s surface?
A: The Dead Sea shoreline
Q: What is the average level on which the Dead Sea is located?
A: 400 meters (below sea level)
Q: How long is the Dead Sea?
A: 70 km
Q: How much saltier is the Dead Sea than the oceans?
A: 7 times
Q: What is the volume content of salt in the Dead Sea?
A: 30%
Q: Why can the Dead Sea keep swimmers afloat?
A: due to high salt content
Q: Why is the Dead Sea called Dead?
A: because only simple organisms can live in it
Q: Why only simple organisms can live in the Dead Sea?
A: because of high salt content
Q. SRS prompts should usually focus on one atomic unit. How is “one atomic unit” defined?
A. Depends on the complexity of your prior knowledge. One idea is a simple combination of chunks you’ve already encoded.
Q. Why does MN think SRS prompts focused on more than one idea don’t work well?
A. When making a mistake with a combined question, it’s harder to focus sharply on exactly where the mistake was.
References
The Mnemonic Medium
The mnemonic medium gives structure to normally-atomized spaced repetition memory prompts
Spaced repetition memory prompts alone are a poor communications medium, but the Mnemonic medium gives the prompts order and embeds them in a larger whole by embedding them in a prose narrative. This context allows readers to build understanding in an authored, structured manner. When readers return to the prompts, the review is anchored in that narrative experience: Mnemonic medium prompts rely on invoking external experiences (from narrative, from real-world experience). When you answer those questions, you’re recalling the details in the context of the larger narrative.
The narrative also gives the prompts a more salient purpose, which is important because Deep understanding requires (and is a result of) intense personal connection.
See also the discussion in Expanding the scope of memory systems: what types of understanding can they be used for? within How can we develop transformative technologies for thought?
Interleaved practice may help anchor the prompts in the narrative
In Project - spring 2022 demo - a peritextual mnemonic medium and subsequent projects, I’ve been developing a new iteration of the mnemonic medium which emphasize interactions in the marginalia. Naively, one might imagine this design would strengthen the effect I describe in this note: the prompts are more strongly embedded in the prose narrative because they’re appearing immediately alongside that prose.
But these designs also de-emphasize interleaved review (Mnemonic medium prompts are interleaved into the reading experience), focusing on the task of helping readers save prompts they find interesting to review later. I worry that the interleaved review may actually be quite important to the effect described in this note. Without it, the prompts saved while reading may feel atomized, similar to those in downloadable SRS decks: Traditional spaced repetition memory prompts are atomized.
References
The mnemonic medium works well for linear primers in established fields
When people are initially learning about a topic, it’s a particularly valuable time to augment their learning, and in particular their memory: e.g. Memory augmentation can accelerate the unpleasant early stages of learning a subject.
The Mnemonic medium is particularly well suited to this context because The initial mnemonic medium is implicitly authoritarian in premise: it assumes you want to defer to the author’s authority on the subject and memorize every prompt. That assumption is disproportionately likely to hold for a primer, when the reader may not have many ideas or preconceptions of their own about the subject. The reader doesn’t want to jump around or pick and choose—they don’t know enough to do that. They want to follow a clear path laid out before them.
I think it’s also important that the primer be covering a well-established field, so that its table of contents is broadly not up for much dispute or authorial idiosyncrasy. When the field is not well-established, readers will be more likely to second-guess the author’s choice of what’s important.
I think these reasons are in part why Quantum Country worked relatively well.
Q. Why was it important to the success of its mnemonic medium that Quantum Country was a primer?
A. The medium assumes that readers want to substantially defer to the author about what to learn and how; a primer’s the best setting for that.
Q. Why was it important to the success of its mnemonic medium that Quantum Country was about a well-established field?
A. No need to second-guess the table of contents: if readers want to understand the basics of quantum computing, they can trust that this is the stuff they need to learn.
Learning and Metacognition
Learning requires metacognition
To successfully learn something new, people must evaluate their understanding, monitor for confusion or inconsistency, plan what to do next based on those observations, and coordinate that plan’s execution. This often falls under the category of “metacognition,” though Metacognition is too imprecise a category: prefer to unbundle its phenomena.
In a classroom or professional setting, an expert might perform some of these tasks for a learner (Metacognitive supports as cognitive scaffolding), but when a learner’s on their own, these metacognitive activities may be taxing or beyond reach. Unfortunately, Most explanatory media make participants run their own feedback loops.
Metacognitive supports as cognitive scaffolding
Learning requires metacognition, but environments can take some of that metacognitive burden off of learners’ shoulders.
In a classroom setting, teachers do much of the monitoring, evaluation, planning, and executive control. Of course, they sometimes do too much of it, but in many situations, this can free students to focus on the material at hand.
On Quantum Country, our interleaved review sessions help readers notice when they’re not absorbing the material and give them a sense of ease when their memory is solid. The environment also plans and executes subsequent memory practice accordingly.
Games are particularly good at this:
Metacognitive supports require dynamic, participatory environments
Metacognition is inherently dynamic, so to design an environment which offers metacognitive supports, that environment must dynamically behave and respond to participants’ engagement with the material.
Environments which are dynamic but non-participatory (like most “explorables”) have nothing to behave and respond to.
Environments which are participatory but non-dynamic (like Make Magazine) can’t alter their message or behavior in response to participants’ actions.
This is one reason that Most explanatory media make participants run their own feedback loops: most explanatory media are static.
Static media sometimes try to provide metacognitive supports by presenting fixed metacognitive “programs” for participants to “execute,” like “if you’re unfamiliar with the use of XOR in this context, consult page 240.” These are obviously limited; they still make the participant do lots of work.
Most explanatory media make participants run their own feedback loops
Learning requires metacognition. When learning something new from a text, readers must constantly ask themselves: did I understand that? what questions can I ask myself to check my understanding? should I reread that passage? should I consult a reference for background on that? etc.
In other words, readers have to run their own feedback loops. When readers are focused on challenging object-level material, they may not be able to effectively perform this kind of metacognition; conversely, too much effort spent on evaluation, planning, and executive control may make it difficult to engage with difficult object-level material.
The situation is perhaps slightly worse in video formats. In books, readers are in complete control of the pace, but in videos, there’s a default pace, and viewers must actively decide to subvert it.
My theory as to why this is true: Metacognitive supports require dynamic, participatory environments.
References
Educational objectives often subvert themselves
Some “educational” activities have intrinsically meaningful purposes, but in most educational environments, the primary concern is cause others/oneself to know something, which is generally not an intrinsically meaningful purpose (contra Enabling environments’ activities directly serve an intrinsically meaningful purpose).
A fixation on learning outcomes is a fixation on what would normally be the effect of a deeper cause: an intrinsically meaningful purpose involving that material. By attempting to produce the effect without the cause, the teacher makes the students into dependents. He’s the source not only of expertise but also of purpose. In such a relationship, the teacher’s role is defined by his superiority. This often manifests as (unintentional) condescension.
Internally-modulated learning is self-actualizing; externally-modulated learning is self-abnegating. Students sense the abnegation and often respond either by disengaging or by shrinking their sense of intellectual responsibility. Both of these behaviors magnify the asymmetry between the teacher and student, which in turn magnifies (intentional or unintentional) sense of superiority the teacher conveys.
Deep understanding requires (and is a result of) intense personal connection. Condescension and external dependence are unlikely to produce such a connection. Often, the teacher/author themselves doesn’t have such a connection to the material, which makes the condescension worse (see Authored environments are significantly colored by authors’ motivations). Per Bret:
I don’t imagine that many designers are inherently fascinated by counting to ten, or addition. So material on those subjects is created neither for direct empowerment or sharing the love, but from a standpoint of “I know this concept, and you don’t this concept, and I think you need to know what I know.” That creates an imbalance, an asymmetry, between a “teacher” and a “student”. And that asymmetry often expresses itself as condescension.
Related:
- Enabling environments focus on creating opportunities for growth and action, not on skill-building
- Educational games are a doomed approach to creating enabling environments
- Educational environments usually don’t involve original thought
- Enabling environments focus on doing what’s enabled
References
Of course, in some sense I’m quite enamoured of goals like “enabling people” … But I’m also very suspicious of such goals. I think > 99.9% of the time they end up patronizing. The only thing I know of which consistently gets away from that failure pattern is to make the primary goal something else, something that’s intrinsically important.
E.g., if you run the Apollo program you’ll certainly be enabling people. But it’ll be secondary to getting to the moon.
Enabling Environments
Enabling environment
An enabling environment significantly expands its participants’ capacity to do things they find meaningful and important.
Schools ostensibly aspire to this purpose, but Educational objectives often subvert themselves in large part because Enabling environments’ activities directly serve an intrinsically meaningful purpose. In general, Enabling environments focus on creating opportunities for growth and action, not on skill-building.
Many other social institutions represent powerful enabling environments. Highly functional corporations are often great examples of enabling environments. In these organizations, new employees might feel far more personally capable than they ever had before, even after many years of experience. Likewise, Y Combinator is an enabling environment.
Great software environments are enabling environments. Photoshop expands experts’ range of artistic expression and unlocks previously-rarefied photo enhancement techniques for novices. Software development tools enable teenagers to make games and distribute them to millions at zero marginal cost. By contrast, Most games aren’t enabling environments, and Educational games are a doomed approach to creating enabling environments.
Books and videos rarely deliver here: Mass mediums are typically bad at helping people translate ideas to practice.
A collection of densely-connected Evergreen notes can be an enabling environment for the author: Evergreen note-writing helps insight accumulate. (See also Evergreen note-writing as fundamental unit of knowledge work)
Designing enabling environments
Enabling environments are generally authored, but Powerful enabling environments usually arise as a byproduct of projects pursuing their own intrinsically meaningful purposes. Authored environments are significantly colored by authors’ motivations; that often means Powerful enabling environments focus on expert use.
Designing new enabling environments can be framed as designing a University++
Challenges in authoring enabling environments:
Some mechanisms for designing these environments TODO expand into notes:
- design representations which expand the range of action for a participant’s existing expertise / efforts; e.g.:
- Photoshop’s content-aware resize tool
- checklists in airplanes/hospitals
- design representations which expand the upper bounds on participants’ capacity; e.g.:
- Arabic numerals
- non-linear video editing
- design representations which subsume the expressive range of existing representations but with lower / different effort / expertise demands; e.g.:
- Figma’s vector network bezier tool vs. Photoshop’s pen tool
- automatic retain counting in Rust and Objective-C vs. manual retain counting
- design representations which subsume the expressive range of existing representations but with smoother effort / expertise on-ramps; e.g.:
- SICP
- Minecraft’s 3D editor vs. pre-existing voxel editors
- (many existing environments here, like SICP and most executable notebooks, interplay weakly with where the enabled action happens, which significantly limits their power)
- (here’s the opportunity for dynamic Cognitive scaffolding, Enacted experiences amplify the power of narrative, and some other Primer design elements)
Enabling environments focus on creating opportunities for growth and action, not on skill-building
A great research lab isn’t fixated on instilling heaps of specific knowledge and skills in its graduate students. It enables by creating great opportunities for personal growth, and by highlighting bridges to opportunities for action based on that growth. This is generally true of an effective Enabling environment.
Critically, such environments aren’t about pushing growth for growth’s sake.Enabling environments focus on doing what’s enabled. Y Combinator founders are primarily trying to start a successful company, not to achieve personal growth (Enabling environments’ activities directly serve an intrinsically meaningful purpose). And insofar as the environment creates personal growth, it will generally emphasize the ultimate goal in all things (e.g. making that startup successful), rather than focusing on “practice.”
This heuristic gives us a new way to view existing media. For instance, books are rarely focused on creating opportunities for personal growth or action: they’re generally about communicating knowledge, in an abstract context. This leads us to a compelling prompt:
What would it mean to design “books” which are primarily about creating growth opportunities for people, and bridges to opportunities for action based on that growth?
See e.g. Moore method.
References
Powerful enabling environments focus on expert use
Because an Enabling environment can help people do new things, it’s tempting to design environments which aspire to help novices enter a discipline, perhaps through simplified versions of its activities. This approach is usually quite limited.
Powerful enabling environments usually arise as a byproduct of projects pursuing their own intrinsically meaningful purposes. Those purposes are usually best served by enabling experts. From that position, environments may also include structures to help novices pursue those same purposes.
When environments focus on enabling simplified versions of an activity, the goal often becomes skill-building itself. This tends to subvert its own purpose (see Educational objectives often subvert themselves). Environments designed for e.g. solving important problems in a field will almost automatically avoid this trap.
Representation design is one of the most important parts of designing enabling environments TODO write note. Representations designed specifically for simplified versions of an activity usually won’t also enable expert practice. That means these representations have limited power. (contra Focus on power over scale for transformative system design) Worse: they’ll often be discontiguous with representations used by experts. But representations designed for experts often can service novices. When apprentices are building their skills in environments of authentic practice, they may use simplified representations. But because those representations were conceived in the context of expert practice, they’re often contiguous with expert representations so they can evolve smoothly. (See also Most dynamic representations developed for communication aren’t very enabling)
Examples and counter-examples
Research labs can be powerful enabling environments. They may include lots of structures which help junior scholars develop: reading groups, colloquiums, writing workshops, etc. When they’re working well, these activities are all about producing better research—not simply to build skills. Many of these activities (e.g. colloquiums) may actually be more important for experienced scholars. They were created for that purpose, and then perhaps additional structures were added to make them more accessible to junior faculty. Participation in these activities is participation in the discipline, for everyone. These activities grow with their participants.
Mathematica enables high-school students to visualize and manipulate the data from simple experiments. But more importantly, it helps professional scientists do their work more effectively. A tool designed primarily for the students probably wouldn’t help the pros do better science; this tool, designed for the pros, also helps the students do better science—and it grows with them to the frontiers of knowledge.
SimCity is fun—at least skill-building is not its primary purpose—but its representations encode many assumptions so fundamentally that they can’t be smoothly evolved to the representations of experts. If your goal is to do urban planning, SimCity will not help you much.
Likewise, Logo enables children to access ideas in differential geometry. This is wonderful! But professional geometers don’t seem to find Logo’s representations relevant for their research. So the children can only be said to be doing differential geometry in a limited sense because to do math is to ask and answer original questions, and this environment doesn’t seem to help much with that. Papert wasn’t interested in helping differential geometers first and foremost, and he didn’t necessarily have the expertise to do so, but it’s interesting to imagine an alternative Logo designed around powerful computational representations for geometers?
References
If you create Mathematica you’ll certainly be enabling people. But it’ll be secondary to doing kick-ass mathematics / theoretical physics.
Scalability and non-scalability of ideas are interesting. Rocky’s Boots is still one of the best ever games that provide profound learning experiences. The extension of this to Robot Odyssey didn’t work because the logic and wires programming didn’t scale well enough — the bang per effort dropped off precipitously
Powerful enabling environments usually arise as a byproduct of projects pursuing their own intrinsically meaningful purposes
The Apollo program was an incredibly powerful Enabling environment, but it did not emerge from a project aiming to give scientists lots of great opportunities for personal growth. Rather, it was about putting people on the moon (and, er, saving the world from the Soviets). The enabling environment was a byproduct of that deeply meaningful effort.
Likewise, when Pixar created its revolutionary animation tools, many teams had been working on computer graphics for years, but Pixar’s systems emerged from a zealous pursuit of a storytelling dream: Pixar’s movies and technology development act as coupled flywheels.
Cathedrals! University research labs! Mathematica! They all follow this pattern.
Practically speaking, such contexts provide deeply meaningful feedback: Effective system design requires insights drawn from serious contexts of use. They also avoid the issues described in Authored environments are significantly colored by authors’ motivations. But perhaps most importantly, these projects also provide the intense personal connection which makes great work possible.
Some implications:
- Powerful enabling environments focus on expert use
- Enabling environments focus on doing what’s enabled
Is it possible to make the tail wag the dog? To initiate a project pursuing some intrinsically meaningful purpose in order to reap the enabling environments which emerge in that context? It’s not clear. The most likely failure mode is that the resulting project wouldn’t really create the intense personal connection required. But this is what we’re trying for with Ladder.
Building on Seymour Papert. (2005). You Can’t Think About Thinking Without Thinking About Thinking About Something. Contemporary Issues in Technology and Teacher Education, 5(3), 366–367.: you can’t teach children “logical thinking” in a vacuum, in an abstract sense; in the same way, you can’t make “tools for thought” in an abstract sense. You have to make a tool for thinking about something in particular; likewise, you have to understand logical thinking about something in particular.
References
The most powerful tools are not developed in isolation. Rather, they arise as part of projects done for their own, intrinsic reasons. Think of the art of stained glass windows, developed in service of God in the great cathedrals. Or of the development of computer animation in service of story by Pixar. These larger goals orient the development of the tools, ensuring they can be used seriously. This sounds like a platitude, but is often violated. “Tools” for mathematics or art or etc are often developed by people who are not deeply active in the area themselves. Unless they do extremely intensive user research—effectively, a collaboration with serious users—it’s extremely difficult for them to build anything other than plausible-seeming toys.
To this end, we will develop a series of ambitious media projects. These will—indeed, must!—be intrinsically worth doing in their own right. But they will also serve as a vehicle for the development of tools for thought.
DNA origami is a powerful emerging tool; it didn’t come out of a field centered on tool-making, but rather Paul Rothemund invented it as a means to the end of making self-assembling computers. The Scanning Tunneling Microscope didn’t come out of the field of microscopy, but rather Heinrich Rohrer wanted to help his colleagues fabricate Josephson junctions and needed a better spectrometer.
I don’t think you can start with “text” or “programming” and get very far. I think it’s always better to have something important and big you want to do better with — eventually this provides clues to various kinds of media (including “languages”) that need to be invented to help. This is what people miss. McCarthy wasn’t trying to invent Lisp, he was trying to create ways to make an “Advice Taker”. Doug wasn’t trying to do hypertext, he was trying to synergize human effort for good.
Effective system design requires insights drawn from serious contexts of use
Scrappy prototypes are great: they allow scrappy iteration and quick evaluation. But many critical insights will only emerge in the context of a serious creative problem that’s not about the system itself. This is a key claim of Insight through making.
That sounds like standard practice: of course systems have to be evaluated! But most system designers don’t take “serious” seriously: Tool-makers usually lack connection to a serious context of use.
Observing how your theories (represented in systems) interact with reality can yield insights which help improve your theories. The character of those insights will depend on the context in which the system was used. If the system isn’t used seriously, the insights will be more like those which a pure theorist could have seen. Those were possible without actually building a system.
Pixar’s a good example of an organization which creates serious contexts of use, which in turn drive system design: Pixar’s movies and technology development act as coupled flywheels.
Common challenges:
- Great tool-makers are often not great tool-users, and vice-versa
- Groups researching tools for thought need enough capacity to build prototypes suitable for serious work
Related theory:
- Powerful enabling environments usually arise as a byproduct of projects pursuing their own intrinsically meaningful purposes
- Authored environments are significantly colored by authors’ motivations
- Powerful enabling environments focus on expert use
References
Concretely: suppose you want to build tools for subject X (say X = differential geometry). Unless you are deeply involved in practicing that subject, it’s going to be extremely difficult to build good tools. It’ll be much like trying to build new tools for carpentry without actually doing any carpentry yourself. This is perhaps part of why tools like Mathematica work quite well – the principal designer, Stephen Wolfram, has genuine research interests in mathematics and physics. Of course, not all parts of Mathematica work equally well; some parts feel like toys, and it seems likely those are the ones not being used seriously internal to the company.
It aims us at relevant problems, not just exercises or toy-scale problems.
It keeps us honest about success and failure, so that we don’t fool ourselves so easily.
It makes us face the wholeproblem, not just the easy or mathematical parts. In computational geometry, for example, we can’t avoid the cases of collinear point triples or coplanar point quadru- ples. We can’t assume away ill-conditioned cases.
Facing the whole problem in turn forces us to learn or develop new computer science, often in areas we otherwise never would have addressed.
Besides all of that, it is just plain fun to look over the shoulders of those discovering how proteins work, or designing submarines, or fabricating on the nanometer scale.
Software-based researchers often strive to build systems containing high-level ideas that are likely to generalize, since those make for more compelling academic papers. However, we believe that trying to be too general actually hinders scale and sustainability. To build long-lasting software that can organically grow a large userbase, one must instead start specific.
In 2009 we created Python Tutor with a very specific goal in mind: to provide a convenient way for students and instructors (such as ourselves) to walk through Python code step-by-step and see the values of variables.
Enacted Experiences
Enacted experience
An enacted experience is an experience which participants feel they’ve brought about—but which, in reality, is the largely-specified expression of an author’s intentions. This is a powerful mechanism because Enacted experiences can create intense personal connection to authored targets.
(n.b. I use the term “enaction” differently from Bruner: Jerome Bruner and enaction)
Examples and counter-examples
Many elements of a Y Combinator batch are enacted experiences: carefully-arranged dinners, timelines, pressures, etc. combine to produce intended experiences. Savvy professionals often create enacted experiences in meetings: they structure the agenda and framing so that attendees will inevitably arrive at the desired beliefs or conclusions, but each person will feel that they brought that about for themselves.
Good video games are structured so that at any given moment, players feel as though their cumulative actions have created their present experience. And they feel their ongoing actions enact future experiences—they’ll bring those states into being. This sense will occur in any Participatory environment, but in games, designers can exert strong authorial control over which experiences players will enact.
Game designers like Jenova Chen, Frank Lantz, and Jonathan Blow create fiddles which play their players. They create environments in which players are the ones pressing the buttons, and players feel intuitively that they’re bringing each moment about, but the buttons are arranged such that players’ actions typically create exactly the intended experience. In good games, players will feel they’ve enacted the authored experience even when they had no real agency. This gives game designers unique opportunities for expression: Games are an aesthetic medium of action.
This is an unusual property for a media form. In film and in books, the viewer/reader feels at any given moment that the author has created that moment. If the work is done very skillfully, the viewer/reader may feel that the characters or their environment have created that moment. But viewers/readers certainly don’t feel that their actions created the moment.
The closest non-game media analogue might be immersive theater, but participation is often too passive to create this effect.
Software environments are participatory, and users generally believe they bring experiences in those systems about, but those experiences are usually not the controlled expression of the software designer’s intentions. So software environments rarely produce enacted experiences.
Elements
- Definitionally, any environment which produces an enacted experience must be a Participatory environment.
- The joy of being the cause
- Enacted experiences require participant-situated cause and effect
- Enacted experiences require blocking on participant action
- Enacted experiences require tight action–feedback loops
Many enacted experiences use metacognitive scaffolds (Metacognitive supports as cognitive scaffolding) to steer participants toward the desired experiences, but not all do. Playing a piece from sheet music and executing a recipe are examples of enacted experiences which might involve no scaffolding.
Enacted experiences exist on a spectrum of participation and authorial control
Challenges
Prospects
- Enacted experiences can bootstrap active participation in enabling environments
- Enacted experiences have incredible potential as a mass medium
Enacted experiences can create intense personal connection to authored targets
The creator of an Enacted experience can induce intense personal connection to a specific experience they have authored. One important consequence: Enacted experiences amplify the power of narrative.
In Journey, for instance, players can do little but walk forward up the mountain. The sequence of environments they traverse are carefully choreographed so that most players experience an emotional hero’s journey, culminating in rebirth and divine transcendence. It’s not a passive experience: the player took every step with the character. So the character’s ascension is felt particularly deeply.
Great movies and books can be quite moving, but the stakes I feel in those media are nowhere near as intense. I’m empathizing with the characters, but I don’t become the characters. There’s no contingency; I’m not making any of this happen; nothing I can do will change anything.
Social environments can create intense personal connections, but generally not to specific authored experiences.
Likewise, software environments can create intense personal connections, but they’re usually not to specific authored experiences. Snapchat’s streaks are one interesting counter-example.
This relates to Situated Learning - Lave and Wenger’s arguments that Situated learning is a process of identity construction/transformation. I think they’d also suggest that my framing here is too didactic; they’d advocate legitimate participation in a task in community:
When the process of increasing participation is not the primary motivation for learning, it is often because “didactic caretakers” assume responsibility for motivating newcomers. In such circumstances, the focus of attention shifts from coparticipating in practice to acting upon the person-to-be-changed.
References
Enacted experiences can bootstrap active participation in enabling environments
Enabling environments focus on doing what’s enabled, but Novices in enabling environments often can’t do what’s enabled. A well-designed Enacted experience can allow participants to immediately experience doing what an environment enables.
For instance, say a software company has special testing infrastructure which enables their engineers to make large changes confidently. A manager might introduce a new engineer to that infrastructure by assigning him a feature which the manager knows will require modifying well-tested code. The engineer will see that adjacent code includes references to the testing framework and will integrate his new feature accordingly. Code review might create useful discussions about the tests. The testing infrastructure enables the engineer to land his new feature confidently. Because of this experience, he can test future features with ease.
By comparison, imagine that the manager hadn’t done this. Someone else asks the engineer to extend a feature which doesn’t already use the special testing infrastructure. The engineer might not think to use it. Someone might point it out to him in code review and send him some introductory documentation, but reading that is an activity about his goal, so he’s less connected to it. He tries to just dive in, but he finds the testing infrastructure challenging to integrate, since his module has no examples to modify.
Unfortunately, Enacted experiences are hard to author and Enacted experiences are hard to distribute. Still, Enacted experiences have incredible potential as a mass medium. I see this as the central promise of media like the Primer: The Primer++ is embedded in a field, bootstrapping participation through enacted experience.
Situated Learning - Lave and Wenger argues a subtle point about what I’m describing here: it’s important that the activity is legitimate. The manager’s assigning a feature which really does need to get built, primarily because they want help building it—not primarily as a “training exercise”. The (somewhat implicit) claim is that these sorts of activities will produce more effective transfer and greater identity transformation.
Enacted experiences have incredible potential as a mass medium
I believe it’s possible to create an Enacted experience with the primary purpose of communicating ideas, values, and practices—akin to doing the “job” of a book. This is a powerful proposition because Enacted experiences can create intense personal connection to authored targets and Enacted experiences can bootstrap active participation in enabling environments.
Avenues:
- Apprenticeships and other social institutions (e.g. YC) do this today, but those aren’t mass media forms (Enacted experiences are hard to distribute)
- Video games are a mass medium, but they’re used almost exclusively to convey aesthetic experiences, not as part of an enabling environment
- Meditation apps are a promising reference: Guided meditation apps deliver enacted experiences in a mass medium
I see this as one of the core ideas of The Primer++.
There seems to be some essential tension here with Interaction is a cost center in interface design.
Y Combinator
Y Combinator is a startup incubator which, besides funding, offers its participants a structured environment meant to accelerate both their company and their personal capacity. Many of the activities focus on conveying a honed set of practices and values: e.g. interviews and dinners with successful past founders transmit cultural knowledge almost mythically; a canonical set of books and talks do likewise; the tight Demo Day timeline mandates a certain way of doing; office hours provide direct coaching in the same principles.
Y Combinator is an Enabling environment: founders regularly report finding themselves suddenly able to do much more—and not just because of the funds. It’s like a school in some shallow sense, but the key difference is that all of a founder’s activities revolve around building the startup they’re ridiculously passionate about—not around learning some abstract set of skills which are meant to enable something intrinsically meaningful later (Enabling environments’ activities directly serve an intrinsically meaningful purpose).
Y Combinator is also an Enacted experience. By structuring their lives around the dinners, milestones, talks, and office hours, founders fairly consistently absorb a certain set of practices and values, and at events like Demo Day, they fairly consistently experience a certain set of emotional experiences. But many of these experiences don’t feel “forced” from the outside; many of the experiences feel like something the founders brought about—even though they wouldn’t have been accessible outside the context of YC (Enacted experiences can bootstrap active participation in enabling environments). These activities aren’t framed as “practice.” They’re framed as doing (Enabling environments focus on creating opportunities for growth and action, not on skill-building).
Y Combinator also offers “Startup School,” a pared-down distance learning version of the same program. My impression is that it doesn’t offer the same consistency in enacted experience—but it would be very powerful if it did (see Enacted experiences have incredible potential as a mass medium).
Transformative Tools for Thought
How important is memory?
People tend to fall into two buckets when told of the mnemonic medium. One group is fascinated by the idea, and wants to try it out. The second group is skeptical or even repulsed. In caricature, they say: “Why should I care about memory? I want deeper kinds of understanding! Can’t I just look stuff up on the internet? I want creativity! I want conceptual understanding! I want to know how to solve important problems! Only dull, detail-obsessed grinds focus on rote memory.”
It’s worth thinking hard about such objections. To develop the best possible memory system we need to understand and address the underlying concerns. In part, this means digging down far enough to identify the mistaken or superficial parts of these concerns. It also means distilling as sharply as possible the truth in the concerns. Doing both will help us improve and go beyond the current prototype mnemonic medium.
One response to such objections is the argument from lived experience. In the past, one of us (MN) has often helped students learn technical subjects such as quantum mechanics. He noticed that people often think they’re getting stuck on esoteric, complex issues. But, as suggested in the introduction to this essay, often what’s really going on is that they’re having a hard time with basic notation and terminology. It’s difficult to understand quantum mechanics when you’re unclear about every third word or piece of notation. Every sentence is a struggle.
It’s like they’re trying to compose a beautiful sonnet in French, but only know 200 words of French. They’re frustrated and think the trouble is the difficulty of finding a good theme, striking sentiments and images, and so on. But really the issue is that they have only 200 words with which to compose.
At the time, MN’s somewhat self-satisfied belief was that if people only focused more on remembering the basics, and worried less about the “difficult” high-level issues, they’d find the high-level issues took care of themselves. What he didn’t realize is that this also applied to him. When he began using the memory system Anki to read papers in new fields, he found it almost unsettling how much easier Anki made learning the basics of such subjects. And it made him start wondering if memory was often a binding constraint in learning new fields.1
One particularly common negative response to the mnemonic medium is that people don’t want to remember “unimportant details”, and are just looking for “a broad, conceptual understanding”. It’s difficult to know what to make of this argument. Bluntly, it seems likely that such people are fooling themselves, confusing a sense of enjoyment with any sort of durable understanding.
Imagine meeting a person who told you they “had a broad conceptual understanding” of how to speak French, but it turned out they didn’t know the meaning of “bonjour”, “au revoir”, or “tres bien”. You’d think their claim to have a broad conceptual understanding of French was hilarious. If you want to understand a subject in any real sense you need to know the details of the fundamentals. What’s more, that means not just knowing them immediately after reading. It means internalizing them for the long term.2
A better model is that conceptual mastery is actually enabled by a mastery of details. One user of Quantum Country told us that she found the experience of reading unexpectedly relaxing, because she “no longer had to worry” about whether she would remember the content. She simply trusted that the medium itself would ensure that she did. And she reported that she was instead able to spend more of her time on conceptual issues.
When people respond to the mnemonic medium with “why do you focus on all that boring memory stuff?”, they are missing the point. By largely automating away the problem of memory, the mnemonic medium makes it easier for people to spend more time focusing on other parts of learning, such as conceptual issues.
Another common argument against spaced repetition systems is that it’s better to rely on natural repetition. For instance, if you’re learning a programming language, the argument goes, you shouldn’t memorize every detail of that language. Instead, as you use the language in real projects you’ll naturally repeatedly use, and eventually commit to memory, those parts of the language most important to learn.
There are important partial truths in this. It is good to use what you’re learning as part of your creative projects. Indeed, an ideal memory system might help that happen, prompting you as you work, rather than in an artificial card-based environment. Furthermore, a common failure mode with memory systems is that people attempt to memorize things they’re unlikely to ever have any use for. For instance, it’s no good (but surprisingly common) for someone to memorize lots of details of a programming language they plan to use for just one small project. Or to memorize details “just in case” they ever need them. These patterns are mistakes.
But the truths of the last paragraph also have limits. If you’re learning French, but don’t know any French speakers, then waiting for “natural opportunities” to speak just won’t work. And even if you do have (or create) opportunities to speak, it’s desirable to accelerate the awkward, uncomfortable early stages that form such a barrier to using the language.
It’s in this phase that memory systems shine. They can accelerate people through the awkward early stages of learning a subject. Ideally, they’ll support and enable work on creative projects. For this to work well takes good heuristics for what any given person should commit to memory; what is good for one person to memorize may be bad for another. Working such heuristics out is an ongoing challenge in the design of memory systems.
(Incidentally, a surprising number of people say they are “repulsed”, or some similarly strong word, by spaced-repetition memory systems. Their line of argument is usually some variant on: it is claimed that spaced-repetition systems help with memory; if that is true I must use the systems; but I hate using the systems. The response is to deny the first step of the argument. Of course, the mistake is elsewhere: there is absolutely no reason anyone “should” use such systems, even if they help with memory. Someone who hates using them should simply choose not to do so. Using memory systems is not a moral imperative!)
An immense amount of research has been done on the relationship of memory to mastery. Much of this research is detailed and context specific. But at the level of broader conclusions, one especially interesting series of studies was done in the 1970s by Herbert Simon and his collaborators. They studied chess players, and discovered3 that when master chess players look at a position in chess they don’t see it in terms of the individual pieces, a rook here, a pawn there. Instead, over years of playing and analyzing games the players learn to recognize somewhere between 25,000 and 100,000 patterns of chess pieces. These much more elaborate “chunks” are combinations of pieces that the players perceive as a unity, and are able to reason about at a higher level of abstraction than the individual pieces. At least in part it’s the ability to recognize and reason about these chunks which made their gameplay so much better than novices. Furthermore, although Simon did this work in the context of chess, subsequent studies have found similar results in other areas of expertise.4 It seems plausible, though needs further study, that the mnemonic medium can help speed up the acquisition of such chunks, and so the acquisition of mastery.
So, does all this mean we’re fans of rote memory, the kind of forced memorization common schools?
Of course not. What we do believe is that many people’s dislike of rote memorization has led them to a generalized dislike of memory, and consequently to underrate the role it plays in cognition. Memory is, in fact, a central part of cognition. But the right response to this is not immense amounts of dreary rote memorization. Rather, it’s to use good tools and good judgment to memorize what truly matters.
We’ve identified some ways in which criticisms of memory systems are mistaken or miss the point. But what about the ways in which those criticisms are insightful? What are the shortcomings of memory systems? In what ways should we be wary of them?
We’ve already implicitly mentioned a few points in this vein. Think about problems like the need to avoid orphan questions. Or to make sure that users don’t merely learn surface features of questions. These are ways in which memory systems can fail, if used poorly. Here’s a few more key concerns about memory systems:
Memory systems don’t make it easy to decide what to memorize: Most obviously, we meet a lot of people who use memory systems for poorly chosen purposes. The following is surprisingly close to a transcript of a conversation we’ve both had many times:
“I don’t like [memory system]. I tried to memorize the countries in Africa, and it was boring.”
“Why were you trying to remember the countries in Africa?”
[blank look of confusion.]
It’s easy to poke fun at this kind of thing. But we’ve both done the equivalent in our own memory practices. Even some users of Quantum Country seem to be going through the motions out of some misplaced sense of duty. The question “what will be beneficial to memorize” is fundamental, and answering that question well is not trivial.
What’s the real impact of the mnemonic medium on people’s cognition? How does it change people’s behavior? A famous boxer is supposed to have said that everyone has a plan until they get punched in the face. Regular users of memory systems sometimes report that while they can remember answers when being tested by their system, that doesn’t mean they can recall them when they really need them. There can be a tip-of-the-tongue feeling of “Oh, I know this”, but not actual recall, much less the fluent facility one ultimately wants for effective action. Furthermore, the user may not even recognize opportunities to use what they have learned. More broadly: memory is not an end-goal in itself. It’s embedded in a larger context: things like creative problem-solving, problem-finding, and all the many ways there are of taking action in the world. We suspect the impact of memory systems will vary a lot, depending on their design. They may be used as crutches for people to lean on. Or they may be used to greatly enable people to develop other parts of their cognition. We don’t yet understand very well how to ensure they’re enablers, rather than crutches. But later in the essay we’ll describe some other tools for thought that, when integrated with memory systems, may better enable this transition to more effective action.
Footnotes
-
See here and here for more on learning new fields using Anki. The last four paragraphs are adapted from Augmenting Long-term Memory (2018). ↩ -
The last two paragraphs are adapted from our forthcoming mnemonic essay: Andy Matuschak and Michael Nielsen, How quantum teleportation works (2019). ↩ -
See, e.g., William G. Chase and Herbert A. Simon, Perception in Chess (1973). Some fascinating earlier work in a related vein was done by Adrian D. de Groot, and summarized in his book Thought and Choice in Chess (1965). ↩ -
We’ve met many mathematicians and physicists who say that one reason they went into mathematics or physics is because they hated the rote memorization common in many subjects, and preferred subjects where it is possible to derive everything from scratch. But in conversation it quickly becomes evident that they have memorized an enormous number of concepts, connections, and facts in their discipline. It’s fascinating these people are so blind to the central role memory plays in their own thinking. ↩