This combo makes smart note systems magical.

Hey folks, nowadays I cannot open a social media app without getting bombarded by those false 10x improvement gains for when you're using AI coding assistance. And it's getting to me

assistance. And it's getting to me because the gains are there, but they are nowhere near 10x, especially during coding. Uh when I work on large code

coding. Uh when I work on large code bases, it's like more like a on a good day 10% to 30% gain. Um but I do have some areas where those 10x gains are

real. One of them is rapidly prototyping

real. One of them is rapidly prototyping stuff like from idea to a hack together web app to showcase something or to explore alternatives. Sure, the 10x gain

explore alternatives. Sure, the 10x gain is there. Uh but during coding, not so

is there. Uh but during coding, not so much. There are however other areas in

much. There are however other areas in my uh professional life where the 10x gains are real and today I want to showcase some of those uh with you. Now,

in one of my previous videos, I mentioned that these LLM agents are kind of a new programmable machine where you like program in English and you have to

deal with all kinds of quirks like the non-determinism the meh so so um instruction following stuff like that. But once you like get used to

like that. But once you like get used to those quirks, you can actually get them to do useful stuff. Today I want to show you some of those workflows that I implement in my daily work and I want to

focus on one specific area which is my uh the place where I do all my thinking and all my writing which is my obsidian vault. So let's first take a quick look

vault. So let's first take a quick look at what that exactly is and uh how I work with it. So what is this smart note takingaking system I'm talking about?

Well, it's based on the Zetlecasta system and if you want to Google it a bit you can Google for smart notes or zetle casten. This is a book by Sunk

zetle casten. This is a book by Sunk Arens. It's a bit dry, a bit dense, but

Arens. It's a bit dry, a bit dense, but it covers the essential. But basically,

uh the it's my second brain. It's like

an offline copy of my brain. It contains

a lot of notes. Uh you can treat it a bit like Wikipedia with a big difference that I wrote each and every page on in that Wikipedia. It's also hyperl so

that Wikipedia. It's also hyperl so notes reference other notes and you can traverse the graph like that. uh and

that's a I found it to be a really powerful way to uh strengthen my thinking and my writing and my creative process. And as a tool or for tools, I

process. And as a tool or for tools, I use Obsidian, which is something a lot of people use, but I love it and it's especially useful for developers because

it is markdown based. So if you know Markdown syntax, you'll know your way around Obsidian.

It's highly extensible. It has a open plug-in market where you can find plugins for whatever. If you like to draw in Excal, it has a plug-in stuff like that. And it's also very friendly

like that. And it's also very friendly to write your own plugins. We'll go over a couple uh I wrote myself later in this video, but it's like a developer's dream. So, let's quickly go over

dream. So, let's quickly go over Obsidian so you have an idea what this means in practice. Uh this is my vault, my Obsidian vault. And these are all notes I wrote. And let's click open some

notes. So you can just use markdown

notes. So you can just use markdown syntax to add images to uh create lists, write text, whatever. And the powerful part is right here. Uh so this is one of my notes

and you can link through just like on Wikipedia to different related notes and then you can click on them and navigate them quickly. So it's a very cool way to

them quickly. So it's a very cool way to reference notes and like start start down a search of linked notes. And

there's also kinds of cool things like this graph view that highlights all the notes in the system and you can see like things you think and write a lot about and things sort of like in isolation

that you maybe should explore further.

So this obsidian tool helps with this whole smart note takingaking system u approach. Uh that's enough details. Now

approach. Uh that's enough details. Now

let's dive in some use cases. Now,

before we get started, there's one super important uh thing I want to highlight, and that is that uh when you let uh an

an LLM agent into your vault, into your smart node system, do not let it think for you.

This second brain thing is like made to augment your brain. But if you just let an LLM uh vomit word vomit all over it uh without you having any input, you

lose all all value of this uh system. So

handle with care. I use LLMs heavily in my note-taking system, but I do not let them create new content.

All right. So the first use case where I heavily use uh LLM agents uh in my writing thinking smart note system is

actually in searching uh because one of the things I found pretty frustrating in Obsidian is the limited search functionality. If you take a look here

functionality. If you take a look here at the search it gives you like search on a file name on a tag on keywords uh stuff like that but it's all like

text based heavy based stuff. So if I want to look at notes that I wrote about LLMs, uh it's pretty like just looking for the text LLMs and it's not a smart

search or a semantic search. So that was the first uh thought thing that I had in my brain like wait a second all these u neural net things are pretty

good at semantic search stuff. So um

that's the first project I tackled and let's take a look at what exactly I did.

Uh this is the architecture of my semantic search. Uh and basically I have

semantic search. Uh and basically I have my Obsidian vault which is uh a folder with lots of markdown files and the

Obsidian application.

What I did is I uh vipcoded a plugin.

It's a very open system and it's all TypeScript. So it's pretty easy to to

TypeScript. So it's pretty easy to to get started. I vipcoded a plugin that

get started. I vipcoded a plugin that would u look at the file I have open and do like a semantic search. So go through my obsidian vault and look for relevant

other notes that are relevant to this topic but not only on keywords but like on the meaning of the file. So that's

what semantic search means. And in the background this plug-in just calls cloud code uh my CLI application with like some kind of system prompt that says you

have to uh do a semantic search and that then returns JSON and the plug-in renders everything. We'll show that in a

renders everything. We'll show that in a minute. But so it's a plugin that calls

minute. But so it's a plugin that calls clot my AI coding agent which like internally talks to an LM in the cloud.

But that's the basic architecture. Now

what does this agent do? It has a vector database in the background. So I set up a vector database. I used Chroma DB, but the technology is not very important.

But that's one that was really easy to set up if you're looking for uh vector databases and embedding based model searches.

Uh so I installed Chromeab and I installed an MCP server in my cloud that allows the agent to talk to this uh

vector database. Now a vector database

vector database. Now a vector database and embedding models are basically a system where given a document like one

of my uh obsidian nodes just converts that document into a number sequence of numbers actually a vector uh which is

that's like called an embedding. It's

a vector representing the meaning of the document. It's it's quite similar

document. It's it's quite similar actually to the internal mechanisms of a large language model. They they also have like embeddings somewhere in there.

But so yeah, Chromad converts a document to an embedding and then stores that in a persistent storage, a vector database which is really good at doing

vector-based comparisons and some searches. And the

powerful thing that this uh setup enables is similarity search. So I can uh one once I have filled this chromma

uh with all my documents I wrote I've coded some uh Python for that. So, uh,

it fills the Chroma database with all my, uh, embeddings for all my documents.

And then I just use the similarity search in the MCP server the in the Chroma endpoint that just like takes a text query string or an entire node for

or whatever and that can do a similarity search on all my existing nodes and then return it to the agent which returns it to the plugin and which can show like all the relevant nodes uh for its search

query. So that was a pretty cool

query. So that was a pretty cool project. I think it took me two hours to

project. I think it took me two hours to set up everything and maybe an hour to tweak the the system prompt that I have to send to cloud code. So this was half day's work and now I have semantic

search in Obsidian. So let's do a quick demo.

Uh and let's maybe use today's script.

So I I wrote a quick script but it has like no links to nothing whatsoever.

Uh so let's now start my plug-in which is uh this brain augmented brain thing the semantic context fetcher and let's just uh highlight my current document

and let's start the search. This takes a couple of seconds.

Okay. And it's done. So let's take a look at the results. It just renders like a list of all kinds of notes that are relevant. And uh if you watch this

are relevant. And uh if you watch this entire video, all these notes will be very relevant. Semantic search is what

very relevant. Semantic search is what we're talking about here. There's a

note. You can even click it and uh open the note. So yeah, that was a really

the note. So yeah, that was a really cool and useful uh vibe coding experiment uh setting up this semantic search in uh Obsidian and it has already

boost my like exploring my uh smart node system uh for free because I don't pay anything, not a paid plug-in, nothing.

Okay, there's another really cool use case where I use LLM agents inside my vault and that is when I am learning actually not when I'm creating new

notes. So whenever I do a consulting gig

notes. So whenever I do a consulting gig and I have to talk about organization design or like team structure or something, I've done work on that before. I read a lot about it. I wrote a

before. I read a lot about it. I wrote a lot about it and it's somewhere in my vault. Now before I had uh these agents

vault. Now before I had uh these agents in my workflow, I would just open up a note, start reading it, start navigating it, and start like refreshing my memory.

But uh that's a very boring way to freshen up your memory. What I found what was really cool is to like generate interactive quizzes on the fly based on

a topic. And that's what these uh LLM

a topic. And that's what these uh LLM agents also allow me to do. Now uh the the theory behind it is from this book.

It's called Make It Stick by Peter C.

Brown and others. But um like the theory is the best way to learn is to do spaced and interled retrieval practice which is a very fancy term of just mixing up the

topics and having like little cards with questions and answers and like the the ones you know well go to the back of the stack. The ones you struggle with stay

stack. The ones you struggle with stay at the front. And I I built like a a quiz master basically uh using my uh AI agent uh my cloud code and yeah I have

it generate quizzes for me on the fly.

So uh yeah let's take a quick look. So

yeah let's see what this looks like. Let

me run a generate quiz uh command. It's

a slash command and we'll take a look at the prompt in a second. Uh and we want the topic to be team topologies.

Topologies that's a difficult word. And

let's generate three questions.

And now it's going to do a semantic search. So also using Chroma to look for

search. So also using Chroma to look for uh notes related to team topology and then uh create a quiz based on it. And

in the meantime, let's dive into the system prompt. So here's a system prompt

system prompt. So here's a system prompt which basically says we you're a quiz master. You're going to make a quiz. you

master. You're going to make a quiz. you

need to use ChromeB to uh actually look for relevant documents and then come up with some questions and then run as an interactive quiz

master and it even scores and gives like further study uh tips but uh yeah that's a bit too much of detail. So let's uh let's do one of these quizzes and let's

run it interactively. I also have like a vibe coded quiz app that's a bit easier on the tokens than this interactive mode. But yeah,

the sky is the limit there if you start doing this. So yeah, here's the quiz. We

doing this. So yeah, here's the quiz. We

have three questions on team topologies and they're all based on my vault. So

let's quickly run through it. C, nope.

D, no. But A and B both could actually work.

What I like about this interactive mode is um if I like am stuck and I don't know, yeah, I know what this question is about. Let's now say please give a hint.

about. Let's now say please give a hint.

So I did not program anything about a hint, but seeing that this is an LLM, it can just like interpret my command and start freewheeling. Security nightmare

start freewheeling. Security nightmare again.

But as you can see, it like gives some uh uh hints and stuff. So yeah, this runs a quiz and gives you a score, stuff like that. But it's a really fun way to

like that. But it's a really fun way to interact with your knowledge base, with your Obsidian vault. All right, another cool example I want to show you is how I

deal with all the work I did before I had my smart node system. So, I've been creating content and especially making courses and workshops for I think 10

plus years now. And before I had my smart notes, uh the only artifact I had was PowerPoints, slide decks. So, uh

that's not very good uh to search through, that's not very good to write blog posts for YouTube content. So, I

needed to figure out a way to get from bunch of slide decks, huge slide decks, like super dry uh educational stuff to u having text that I could use for YouTube

scripts, for blog posts, stuff like that. And that sounds like something

that. And that sounds like something LLMs might be good at, right? So, uh I started playing around with it and I came up with a pretty cool system. Uh

so, let's uh go over that.

Um, most modern LLMs are multimodal, which means that they work just as fine with text as they do with audio and uh, images. And Cloud Code, the tool I use

images. And Cloud Code, the tool I use the most, uh, works perfectly fine with images. So, what I did is I pre-process

images. So, what I did is I pre-process my slide text, my PowerPoints into folders where like every slide gets an

export in an image, a PNG file. So I

have a huge folder with 200 300 images and then I have my uh LLM agent like batch those interpret every each and every slide uh and like interpret the

text on there, interpret the meaning on there and write like a short summary um in a markdown file in my vault somewhere. And then I have like one huge

somewhere. And then I have like one huge markdown file, one note containing like the transcript of these slide decks. And

while not perfect, they are a very big time saver. Like

it's like 300 slide 300 slide deck condensed into a single markdown file and I just have to scroll over both once to do a cross check which is like 20 minutes work instead of 4 hours of work.

So this is one of those places where I really get 10x improvements u during my non-coding activities. So yeah, let's do

non-coding activities. So yeah, let's do a quick demo.

Uh I did a course on software design a couple of years ago and this is the full 200 slide deck. I would not bore you with that but I have like a three slide

alternative lightweight demo uh and it contains some u uh some references to a great talk by Jessica Kerr. But as you can see, this

Jessica Kerr. But as you can see, this slide is just an image with a speaker Jessica Kerr and like some uh handwriting

on a on a whiteboard uh which is like design until you understand the problem, write code until you realize you don't and then notes to the original software confir So, it's all it's all very

visual.

And then I have some other slides. Maybe

let's go over them to uh get the full picture. There's another slide about

picture. There's another slide about Kent Beck's Tidy First, a very good book if you haven't read it. Um, also a good person to follow if you like working

with uh LLM coding agents. But yeah,

there's this like a mathematical formula and some references to books and substack posts, stuff like that. And

then there's another slide which is like a text dump on uh what domain driven design is, stuff like that. So uh yeah, so that's the slide deck. This is a very

small version, but this scales to 200 300. And now let's reverse engineer

300. And now let's reverse engineer those slides using clot code. So, let's

run clot.

Always run in and dangerously skip permissions if you know what you're doing. And if you don't, please don't

doing. And if you don't, please don't touch this. Uh, so I have my reverse

touch this. Uh, so I have my reverse engineer slide deck.

And I want it to reverse engineer the what folder name was it? Slides

demo.

And I want it into reverse engineered markdown.

And now it's going to take a look at all those images and create a markdown file that resembles the contents of that slide deck. I wrote a I wrote the system

slide deck. I wrote a I wrote the system prompt so that it batches in chunks of 10. So when I hit my clot limit, I don't

10. So when I hit my clot limit, I don't lose all progress.

But as you can see, it's uh it's like very autonomous. It just OCRs every

very autonomous. It just OCRs every image and writes like a summary and then dumps everything in a markdown file.

That's like a small note I want to make.

This is the only place where I let an LLM write stuff into my vault, but I made like 200% sure that it writes only

text that's found in the slides and that it like writes in my style. So, I had uh Clal analyze my entire vault for like writing patterns, tone of voice, stuff

like that. And I had it come up with a

like that. And I had it come up with a system prompt like with the the summary of my writing style and that I also pipe in when I have it write stuff. So it

writes in my tone of voice.

So it finished up on the markdown file.

So now let's take a look at what came out of it. There should be a reversed engineered note. Here we go.

engineered note. Here we go.

So it has some meta metadata but it also has like uh chapter per or yeah a section per slide

and if you can see this is like the first uh slide about with Jessica curse quote or the original uh software engineering conference quote. Yeah, as

you can see, it parsed that human written uh it parsed that uh written text just fine. It also links to some

notes that I have or not yet have, but uh it's trying to link uh topics. And

here you go. That's the second slide with the tidy first reference with the Constantine's equivalence thing. It

parsed the formula just fine. Even put

it in a code block. Cool. And then it explains a bit uh tries to explain a bit. This is something I would very much

bit. This is something I would very much proofread uh before I would push this to production and it parsed the third slide just fine.

Dense wall of text with some highlights and it took like the highlights into account. So yeah, that's how I use LLM

account. So yeah, that's how I use LLM agents to give a first draft of my courseware which is where my real 10x improvements have been in the non-coding

area. Now, if any of these examples

area. Now, if any of these examples sounds cool uh and you want to try them yourself, I provide full links to the code and to the prompts in the description below. So, take a look down

description below. So, take a look down there. So, those were some examples

there. So, those were some examples where I like introduced an LLM into my Smart Note system. And again, I do not let it do any thinking for me, but it

has 10xed some of the tedious, boring work, or it allowed me to explore my vault in uh ways that were not possible before, or at least not for free. Uh so

yeah, do you use LLM outside of coding?

Have you found 10x improvements somewhere else? Uh please let me know in

somewhere else? Uh please let me know in the comments below, and I'll see you again next time. Bye-bye.