It's so fascinating how this world is so interconnected. I think our universe is a strongly connected dense graph (more on that here). In this article, I'll try to show how productivity tools (GTD), kanban dashboards, note-taking apps (and PKMs), databases and distributed systems (PubSub/Kafka) are just representations of the same underlying phenomenon. It's all about information floating through space and time. We just need to have a system for capturing ideas and then effectively using these bits of data.
Searching for the Best Tools
Taking notes is super important. Most engineers create cheat sheets, maintain personal wiki or markdown docs, take meeting notes, etc. I think all my career it's been a constant struggle to find the best tools to capture notes effectively in order to make good use of them later.
1. Plane Docs and Stickers
The most basic option to capture notes is to create plane docs (e.g. MS Word or Google Docs) and write everything there. But it's pretty hard to organise all of these docs. It could be a good starting point, after all that's how books or blog posts work.
Another option I was doing for a while was using Google Keep-style apps (stickers). It's super easy to create a note there but fairly hard to encounter it later, there's kind of no structure there.
2. Folders vs Tags / Avoiding Hierarchy
Some apps like Evernote are great for capturing info. The only downside for me is that they used to enforce tree-like (folders) structures. I think folders is not how the real world works because it forces you to have only one way of categorising your note. It needs to be at least labels like in GMail (each email could belong to multiple categories) - using hashtags is a good example of having multiple tags/labels. Also, I noticed that any kind of rigid structure prevents me from finding a place for new thoughts and ideas. I would simply not gonna write an important idea down because it takes a lot of effort to find a space for it in the hierarchy.
3. Knowledge Graphs
To me, graphs seem like a natural solution. You can write multiple thoughts and simply connect them. The more connections you create, the easier it will be to find relevant information. Wiki is a good example, as Emacs Org Mode, Workflowy, Obsidian, Roam Research, etc. I'll come back to the graphs in a bit.
4. Getting Things Done / Org Mode
Another massive rabbit hole, which requires many separate articles, is tools to organise yourself, including calendars, TODO lists, kanban boards and many more. To me, it's been a constant struggle to find good tools and organise a system that works more-less effectively.
I've read the GTD book by David Allen three times (thanks to Dmitri Osintsev for recommending it) and passed some internal workshops at Google (which teaches you to build the GTD system on top of Gmail). That book helped me a lot, it has a lot of great advice relevant even today.
Even though tools to organise yourself (and your team) and PKMs are different things, to me, it's just various facets of the same underlying concept. This is all about information going through space and time. It's all about effectively ingesting the data, making it accessible to make decisions or produce some output.
For a while I was even thinking of building some flexible minimalist tool myself, trying to incorporate GTD technics and knowledge graphs in one place. Likely I found Emacs Org Mode and Workflowy (web outliner, inspired by Org mode), which I used for a while (thanks Andrei Kashin for recommending). These are super flexible tools, and you can build almost any system you want on top of that.
5. PKMs, Zettlekasten, Roam
Actually it's been just a few years ago when I came across Zettlekasten (thanks Kate Ustiuzhanina) and tools like Roam Research and Obsidian (thanks Alex Zimin), and the concept of personal knowledge management.
Basic idea is to write down various ideas (from your perspective), and link them together. You can explore this graph of ideas. Moving forward through regular links (like in wiki) and backwards through backlinks (list of links pointing to our page). Backlinks turned out to be incredibly useful feature.
My Use Cases of PKMs
1. Quick Capture
Quite often, when showering, talking to friends, flying to San Francisco, or listening to an audiobook, I come across interesting ideas. But I would totally forget about them. At some point, I realised that capturing these ideas is super important. And the process needs to be extremely easy. I usually just send myself a message in Telegram, and then later it ends up in daily notes in Roam. The notes don't have to be pretty; mine are just 20% baked.
2. Notes Related to People
A long time ago, when I started working at Google, my manager Kostas created a doc for 1:1 notes. It's basically a log, shared only between the two of us, where we added everything that seemed important at a time every time we met. E.g. things you're working on, current challenges, advice on how to approach things, what to improve, etc.
Over time this doc turned out to be tremendously important. I could now look back a few years ago and see what were the challenges at the time and maybe give a piece of advice to somebody else because I was in the same shoes at some point. Now I'm trying to incorporate this practice with people I work with and interact with regularly.
3. CRM / dossier
With time I realised a lot of people recommend me so many interesting things, so I started making notes related to a particular person. At first, it could sound creepy to have a "dossier", but it could definitely improve the overall experience. E.g. you can prepare better for the next meeting. It also helps with the retention of ideas.
If you want, you can build a CRM (customer relationship management) on top of that quickly. E.g. it could notify you when you haven't met someone for a while, etc.
PKM tools like Roam make these sorts of things easy. You simply write down a bit of knowledge (e.g. good advice someone gave you) and link it to a person (as easy as creating a hashtag). I usually link an idea with the date (automatic if you're writing to the daily capture), person, location (e.g. "grabbing coffee at Kings Cross") and any related things. The more connections you make, the easier it will be to find this later.
4. Journaling
I find journaling extremely useful. It's like self-therapy. Also, it's super interesting to read old notes, you can see how your mindset has changed, etc.
There're different ways of journaling. I usually do that when I'm stuck with something:
- Write my "brain dump" down as text. Then drink some coffee.
- Then cluster similar thoughts together. Drink some coffee.
- Refactor each cluster. Just replace all these 5 sentences trying to tell you the same thing with one concise version. Drink more coffee.
- I look at each problem/sentence separately and come up with action items, takeaways or rules for life.
Another thing I like is to write the top positive things and the top negative things. It same as in tim.blog/pyr (thanks Alisa for recommending).
5. Helps Writing Articles
These knowledge graphs are super helpful when it comes to writing. You can simply follow along a path through ideas and have a new chapter ready.
Basically select an arbitrary chain of ideas from your graph, and your new chapter is ready!
6. Reflecting
I love deep questions. Even something simple like "whats the biggest challenge to achieve this?" or "where do you want to be (or the project) in one year"? I try collecting questions like that to build a framework to navigate through uncertainty (in business or in life). PKMs help me to bring structure to that, collect the ideas and link them together.
List of PKMs
Here's a short list of PKMs that I'm aware of. I'll leave a comment around describing it. I use Roam Research, but it's not fantastic. There could be better alternatives for your needs.
- Roam Research - "As easy to use as a document. As powerful as a graph database. Roam helps you organize your research for the long haul."
- Obsidian - "Obsidian is a powerful and extensible knowledge base that works on top of your local folder of plain text files."
- Notion - "A new tool that blends your everyday work apps into one. It's the all-in-one workspace for you and your team."
- Foam - "Foam is a personal knowledge management and sharing system inspired by Roam Research, built on Visual Studio Code and GitHub."
- Craft - "When your work needs to wow, Craft gives you the tools to make it magnificent."
- Logseq - "Logseq is a privacy-first, open-source knowledge base that works on top of local plain-text Markdown and Org-mode files."
- Dendron - "Knowledge Management. Redefined. Lightning fast. Open source. Built to scale."
- RemNote - "A note-taking tool. But there’s also flashcards, PDFs, backlinks, and more - to help you study, stay organized, and think."
- Dynalist - "The best outlining app for your best work."
- Emanote - "enables you to create beautiful websites – such as personal webpage, blog, wiki, Zettelkasten, notebook, knowledge-base, documentation, etc."
- Neuron - "Future-proof note-taking and publishing based on Zettelkasten (superseded by Emanote)."
- Zotero - "Your personal research assistant. Zotero is a free, easy-to-use tool to help you collect, organize, annotate, cite, and share research."
- Workflowy - "Workflowy offers a simpler way to stay organized. If you have a crazy job or an ambitious project, we will be your trusty sidekick."
- UPD: TiddlyWiki - "Non-linear notebook for capturing, organising and sharing complex information." (thanks Pesho Ivanov for suggesting)
- UPD: Tana - "all-in-one workspace. Built for fast-moving startups and professionals, to get more done faster."
- UPD: Capacities - "A studio for your mind. A place to make sense of the world and create amazing things."
- UPD: Anytype - "the everything app for those who celebrate trust & autonomy" (thanks for Brendon Wong for sharing last three)
More technical ideas
1. Database view
In the world of databases, there's a really powerful yet underrated concept of views (e.g. logical view or materialized view). Basically, it's a virtual table defined by a query. Usually, it's used as a public interface to your database.
E.g. you're managing a massive database, and some team needs only a list of countries where your customers are from to show in their dashboard. Instead of granting access to your entire system (or to the Customers table), you can create a logical view (virtual table) that has only the countries of the customers, and share it with that team. If the data in the main database has been updated, the view will also be up to date.
You can leverage views to avoid creating extra microservices (e.g. writing a lot of boilerplate in Java), not super effectively doing the same thing of getting a list of countries from the Customers table. Creating a view solves this problem.
In PKMs, you can also use the same approach and execute custom queries. Here's a conceptual example in SQL of getting a TODO list (top 10 TODOs ordered by importance and urgency) from your notes.
Most PKMs (e.g. Roam) use Datalog instead of SQL, but you got the idea.
Having this approach gives a massive advantage:
- you can simply click on the TODO, and it will bring you precisely to the context, to the note where this TODO has been created. I.e. you don't need to write the description of the issue again and again in your separate task tracker. This saves a ton of time.
- Everything will be in perfect sync. E.g. your kanban board will be created on top of your knowledge graph. If you update one thing, it automatically updates everything else, so you don't have any discrepancy.
When you have a separate TODO list or issue tracker, separate notes, separate key results. It takes a lot of effort to keep everything in sync.
2. Optimising for Read vs Write / PubSub / GPT-3
I feel some sort of discrepancy between wiki-style pages (markdown docs) and quick capture log. Sometimes I prefer the first, sometimes the second, and completely not sure how to deal with both at the same time.
Quick capture is basically an append-only event log, you're appending more and more data. It's optimised for writing. E.g. you can simply add a sentence "There're elephants in Africa", and that's it.
Wiki-style pages or markdown docs are optimised for reading. You don't need to scan three years of your logs to filter out only information for a particular topic. You can just read this topic directly. But it's harder to write/update, because you would probably have two different page, one about elephants and one about Africa. I.e. every new idea would most-likely need to be written multiple times (in different pages).
I realised this when I was using the PubSub (publish/subscribe) architecture (e.g. Apache Kafka) in distributed systems. You have input streams of append only events (topics), and you can subscripe to them. You can recreate a state from event log - it's basically a functional programming approach in the realm of databases (avoiding mutable state like in read/write databases. See great talk by Martin Kleppmann). You can easily build a database on top of this streaming system (e.g. ksqldb or materialize, see explaination).
There's no easy way of converting input log into wiki-page reports (except maybe via custom queries, see the TODO list example above). But I believe large language models (LLM) like GPT-3 should be able to cover this gap easily. You can simply tell it "here's my 3 years of logs, please create me a report about startups". So it would go through all your records and would create you a wiki-page about a particular topic which you can read. I.e. your only job would be to write down facts and new ideas. And it would be producing you great write-ups about anything you want.
It seems there're tools like mem.ai (kudos Szymon Zmyślony for recommending) wich are moving in that direction. Looking forward to try more!
3. Semantic Search / Embeddings / GPT-3
Backlinks are amazing and connect all your ideas, but it would be nice to have an automatic way of finding related ideas in your knowledge graph. I think more and more people start realising it, also we have a large language models boom now. I think it should be relatively easy to do. I noticed that more and more tools (e.g. Notion) starting to add AI-generated summary and semantic search.
On one side, it should be relatively easy to do a simple version of that. You can calculate embeddings (something from Hugging Face - thanks Michal Kazmierski for the idea) for each sentence that you have. This give you a representation of idea in the vector space. Then you can simply find ideas close to each other semantically.
UPD: Another observation (thought provoked by David Perell). While amount of notes grows with time linearly O(n), the amount of possible connections could grow up to O(n^2). I.e. the more notes you make, the much much more links you can make most likely. So we definitely need some AI automation to help us make new connections.
But I think that just semantic search might not gonna work well. Here's a counter example. E.g. elephants and Africa are definitely connected via "there are elephants in Africa", but "elephants" and "Africa" are very different things semantically. I.e this approach can find similar things, but can't always establish connections between very different things.
Maybe leveraging semantic search is good enough already, but it would be nice to dive deeper into that. If you have insights, please let me know!
4. Connecting people with similar thoughts
It's cool to publish your knowledge graph and see graphs of other people, what they are interested in, etc. Kind of compare your mind maps.
At the same time, you can use the same semantic search and connect people with similar ideas. That would be dope! And some people (Davey Morse from plexusnotes.com) are working on that (thanks Sasha Lopoukhine for letting me know).
Connecting people with similar ideas would empower our civilisation and push progress forward. But we must keep in mind how to avoid the polarisation of our society.
5. Collaborative Editing
The majority of note-taking apps are currently really bad at collaborative editing. Notion is OK, but others are terrible. I would love to have a knowledge graph for the team where we could edit everything simultaneously and leave comments (similar to Google Docs, but in the knowledge graph). I don't see why this can't be done.
I hope all popular tools would have it soon. If you're building something like that yourself, CRDTs is a good starting point (great video by Martin Kleppmann).
6. Quantifying Information
There's something about knowing where you can read a lot of interesting things and where you can skip because it's not super informative. I love how Grammarly shows which sentences in the email are important, and which are not.
I know there're classical math tools allowing you to measure how much information a particular text has. But LLMs probably would be more practical in terms of guiding you where you can get the most of insights.
Would be curious to know your ideas about it.
My Wishlist
Areas where PKMs need to improve:
- Semantic search of similar ideas.
- AI-powered transform from (quick capture log) into (wiki-style reports).
- Collaborative editing and comments.
- Loading time is really bad often.
- Mobile apps are not fantastic.
- Managing what is public and shared, what is personal.
Resources
- Building a Second Brain by Tiago Forte.
(quite nice overview video) - How to Take Smart Notes by Sönke Ahrens.
(great overview video using Obsidian) - Getting Things Done by David Alen - not PKM, but classical GTD.
- Designing Data-Intensive Applications by Martin Kleppmann - not PKM but a lot about distributed systems.
- 100 Roam tips - nice YouTube clip about Roam.
(nice Roam overview) - YouTube has a ton of stuff on Roam and other tools.
Tiago Forte and Ali Abdaal channels could be a good place to start.
Conclusion
It's all about information floating through space and time. We just need to have a system for capturing things, and then effectively using these bits of data.
To me it's absolutely fascinating that everything is so intertwined: databases, stream processing systems (kafka/pub sub), functional programming, note taking, organising apps (e.g. TODO lists, kanban dashboards, calendars, etc).
I'm super curious to see what's next in this space!
What are your thoughts?
Please help me improve my writing 🙏