Make it Work

As we were wrapping up TalosCon 2024

I was in the pub talking to one of my fans,

Rio Kierkels, a DevOps Specialist

from the Netherlands.

And at one point Rio said, you should

talk to Mischa van den Burg.

So here we are today.

Thank you, Rio.

And it's great talking to you, Mischa.

Welcome.

Thank you.

Thank you, Gerhard.

So we talked a little bit about this

before we started recording.

The more I was researching the content

that you produce and the blog post,

the more excited I was getting.

I knew this was going to

be an amazing conversation.

You've put so much good stuff out there,

not just in terms of content,

but in terms of ideas and you live them,

which I find very inspiring.

Like you are the embodiment of

everything that you stand for.

And it's amazing.

Thank you.

That's the biggest

compliment I've had in a long time.

Thank you.

Four years ago, you switched

from nurse to DevOps engineer.

That's right.

Why DevOps?

I had been living what you

can call it a colorful life.

I've tried out several things.

I moved to Norway.

I worked in the oil industry there.

I worked in management.

Then I lost my job in that industry and I

was very fed up with it.

So I wanted to try something else.

So I transitioned to nursing.

I wanted to work more with people and

like feeling that I

made a change in the world.

And I did.

Every day I was at the beds with the

patients and I could literally make their

world a little bit better by just being

there for them and

putting in a lot of effort.

But I'd also noticed that like my

personality, I'm rather introverted and

it was costing me a lot of energy.

And I wasn't sure if I was going to

be able to do this like

for the rest of my life.

Throughout this whole period, I was

always ever since I was a kid tinkering

with computers and scripting,

automating, especially video games.

I was very into online RPGs.

And at some point I figured out that you

could write scripts for those RPGs.

So I could mine my ores and earn

my in-game money while

I was doing my homework.

Also Diablo II: Resurrected bot, seriously,

like what I saw that like, hey, I think

that's what that was.

That was used to be my favorite game.

Still is by the way, I'm still playing that.

So yeah, like you like you and I never

built a bot by the way for Diablo II

Resurrected, but you did. Amazing.

Well, I didn't build it.

It was an open source project and I was

involved with it, but

by no means built it.

But I did run it and

made videos about it.

So I did do that.

That's when like my fascination with

computers and

automation especially started.

And that escalated to me having like

renting servers in Germany and having

hundreds of accounts playing the games for me.

And then selling that in-game currency

for a little bit of extra money.

I could by no means live off it, but it

was just so cool to know

that I had this army of bots.

playing for me as I was doing my work.

So in that process, I learned Python

coding, Linux. Everything was running

running on Linux, and

I was doing all of this in my free time

and I was always like, well

what if I could make this my job?

Like not automated games specifically,

but like working with infrastructure and

working with code, automation.

And I tried a few times, but it was

always like, well, you don't have a CS

degree and you don't have

experience in the field.

And this was when I was living in Norway.

So that was very tough to get into, but

at some point I was again reaching that

point in my nursing career where I felt

this is not going to be the thing either.

And then I just went, I asked myself the question:

What do I want to do?

And I answered that by saying:

what do I like to do in my free time?

And that was what came out.

And then to get back to your question,

I started talking to people

who I knew in the IT industry.

And one of them said, I

think DevOps is your thing.

I think that's what

you're describing to me.

You're what you like

about code and automation.

I think DevOps is the thing.

And that put me on the track on DevOps.

And as soon as I heard that word for the

first time, I was basically sold.

Okay.

So as you were discovering DevOps,

as you're getting more and more into it,

what are the fun things that you didn't

expect to begin with,

but you really enjoyed?

You've been doing this for four years

now, five years, four years?

I'm going into my fourth year now, yeah.

I really love to learn, and learn new

skills, new technologies.

And I hadn't fully appreciated when

I started out that this field,

or IT in general, is just such

a constantly changing field.

And you're basically required to tinker

in the evenings just to stay up to date.

It gives this constant intellectual

stimulation for me and something to keep

up with and to share about.

Which is something that I didn't

realize in the beginning.

I thought, okay, I have a

skill set I need to learn.

I learn it.

And then that's what I'm going to do.

But that's one thing that really

surprised me and that

I still enjoy a lot.

What about the hardest part?

I did have this vision that I would sit

behind my computer and code and tinker

with what I like to do.

And that was it.

And I heavily underestimated

the soft skills part of it.

So that in DevOps, you're all about

breaking down silos and helping teams to

reach their goals as a DevOps Engineer.

And I didn't realize how much

interpersonal communication and meetings

and all of that is involved with that.

It's not necessarily the hardest part

about it, but that was something that I

had to like readjust myself to.

But then my nursing skills, and my

project management skills like

really came to shine there.

So that's, I think, part of my

success as a DevOps Engineer.

That background that I could apply those

skills in those environments.

And yeah, another hard part about DevOps

is that it's sometimes hard for me to

realize that not everybody is

enthusiastic about these things as I am.

So if you come into a company and yeah,

we're all about Kubernetes now.

Let's start.

Let's do everything Kubernetes.

And this is why.

And then it's like: hold on there cowboy.

You said the K word.

I said the K word.

Get out of here.

Mischa will no longer join this meeting.

He said the K word.

That can be a bit challenging.

And that's one thing you have to learn

if you're passionate about something that

not everybody is going to be as

passionate about as you are.

What does it mean

"DevOps - breaking down silos" in practice?

Do you have a story or an example to give

that illustrates that concept?

We were basically the cloud platform team

and then we had development teams relying

on us running that platform logically.

What would happen is that we would get a

ticket and then: oh, there's

a problem with the database.

And then it wasn't necessarily that

they were saying you should fix it.

It wasn't necessarily implied that there

was this expectation

that we just handle it.

But the way it was working in this

company is that there wasn't much

collaboration between those divisions.

So even though we were a DevOps platform

team, it was again, just Dev throwing

stuff over the fence to Ops.

What I emphasized was: is this urgent?

How urgent is this?

Okay.

Well, it can wait a day, okay.

Let's plan a meeting,

and let's go over it together.

And I would, I would, of course, look

into it myself and see if there was like

something immediately going wrong

with the database itself.

But then I would actually jump on a call

with the developer who

sent in this ticket,

and rather than just me fixing it,

I would do it together with him.

And not necessarily because

I didn't know what to do, but I was actually

showing genuine interest in

what is the application doing,

and how does it work.

And when you start doing that, then the

people, what they're working on, they

start to get very enthusiastic about it.

And in my experience, then you get into a

bit of a different mindset together

instead of like, well, there's this error

and we're fixing it.

It becomes more of a personal project that you're

a fun thing that you're solving together.

And when you do that,

you become more relaxed.

And then in my case, we found

the solution much faster.

And more often than not, developers were

pointing out stuff, like we were

screen sharing, and we were in the monitoring.

And I thought that I had seen

that I had analyzed

the problem and I thought I had it,

but he said: "Hey, what about that one?"

And I hadn't,

that one hadn't occurred to me,

and we dug into it,

and then that was the problem.

So that's a very long

winded explanation, I think.

But that's one thing that I have always

emphasized is: okay, we are DevOps team,

it's not supposed to be that way,

but how do you actually implementing that

collaboration between the two?

And for me, this was one way of doing

that, to actually do it

in meetings, in calls together.

That is a great example.

Great example.

It starts with working together on a problem,

not you throwing the problem

over the fence, and letting me handle it.

Because that in itself is the first step

towards building a silo.

Yeah.

The second step is accepting

the ticket, and going with it.

Don't do that.

Try to have the conversation.

Try to work on it together,

and be open to working together.

Full stop.

Yes.

We are in this together.

It's not me, and I'll figure it out,

and you'll just get the end result.

We will go through it together.

Bonus points if you can record it.

One of the best things which I've done is

recording internal meetings like this,

because we are a remote team.

Jump on Riverside, or any

remote software exactly like this.

Do the recording, do the screen sharing,

do some minimal editing,

and then you have a video of how you

solved the problem together.

Yes.

Post it internally and it's a reference

for example:

what does it mean to be on call?

How do you handle an incident?

What happens when you take

an application into production?

If a migration, if a database migration

fails, how do you tackle it?

How do you build preview environments?

Why do you need preview environments?

All sorts of questions like these that

come up, build a knowledge

base, a wiki of some sort,

A Zettelkasten. It's a system for

wikis, but still something

that captures that knowledge.

And it's not different

people doing things in a corner.

It's people coming together, recording

that, and then sharing that knowledge.

And with all the tech that we have

nowadays, starting

with AI, transcription,

this is easier than ever.

You still have to do it.

Totally agree.

And it's one of the main emphases I have

when I work somewhere is

usually when I jump on a team,

the documentation sucks.

And I always make it a point

of improving that from day one.

Like there's always the onboarding

process where you can't really do much

the first couple of days.

I always start with reading the

documentation, asking questions and

improving it already as I go along.

That's amazing.

Document first,

automate second.

But always document.

I used to do things

wrong for many, many years.

I used to like automation.

There's this Makefile.

How do you mean?

It's like self-explanatory.

And people would tell

me, no, where's the docs?

Who needs docs when

everything is here in the Makefile?

It took me a while, but I finally got it.

Documentation first, automation second.

Are there any hard-earned lessons, any

hard-earned DevOps lessons that you would

like to share with us?

It's really important to keep an online

presence and keep that well up to date.

So the moment you start, like you can

always lose your job.

And the moment you lose your job and you

haven't updated your

LinkedIn profile for three years,

or even not having a

LinkedIn profile at all,

is one of the biggest mistakes you can

make as an engineer, in my

opinion, in this day and age.

And I don't necessarily

like that it's the case.

Like I still am the guy when I'm on the

street and I'm taking a

picture with my cell phone.

I feel weird.

It feels still like this

abnormal thing to do to me.

Like I wasn't a

social media person at all.

But once I recognize the potential it has

and the importance it has

these days for your career,

then I think this is one hard-earned

lesson that I would like to give

everybody to share your journeys,

even if it's just one post a week.

But be active online.

It will only only give you benefits.

Build your personal brand.

Whatever that is, whatever works for you.

If it's TikTok, if it's Reels, if it's

Instagram, it doesn't really matter.

Twitter, Blue Sky, say

Twitter... X, shall I say or Xitter.

I've heard that's like a new thing, which

is like catching apparently.

Xitter, that's a funny one.

In Mandarin, it

actually means something else.

So anyway, I just read recently,

the point is whatever works for

you, videos, podcasts - do that.

And if it's too much to start one,

or maintain one, join one.

There's so many around.

Yes.

So do whatever you can do to

put good content out there.

Do that.

Personal blog. Does not matter what it is

as long as you do that,

because it's like compound interest.

You're putting in the

bank of your personal brand.

It's you. Be who you

are and show the world.

Exactly.

If you have a stack of 200 CVs,

and there's one person there

with a thousand followers and a

personal brand, he literally has a better

chance of getting the job.

Again, I don't necessarily like it this

way, but believe me, it is this way.

Another hard-earned lesson that you've

shared, and I picked it up from the

various YouTube videos

that you posted and blog posts, is:

you waited too long to start a home lab.

Yeah. I'm an academic-oriented person.

I studied at university.

I love reading books and getting into

theory and writing notes

and getting more and more

and more information in my head.

But when I started learning Linux

seriously, professionally,

I'm ashamed to say this, but I wasn't

entering the commands into

the command line for months.

I was just reading about

them and I said, "Oh yeah, okay,

well that's how the file system works."

Great.

But when I sat down and actually tried to

do it the first time,

it wasn't that great.

It hadn't stuck in my mind at all.

One huge mistake I made that I wasn't

applying what I was learning in practice,

and you really need to do that in order

to make things stick.

And one of the best ways to do that is to

start a home lab in

whatever shape or form that is.

Anyone can do it with any hardware.

That's it.

It's becoming more and more accessible

and you will learn so much.

So just go and learn.

Have fun.

Explore.

Tinker.

Exactly.

Yeah, I also always used to associate

this word "home lab"

with these huge server racks

with blinking lights.

That's what you see on YouTube.

I never understood that you could

actually do this on

cheaper hardware or on something

that you already have.

So my students, I always

say, Okay, go to your closet.

There's 99% sure there is one old laptop

in there with maybe four

to eight gigabytes of RAM.

Install Ubuntu server on it and you have a home lab.

You have a learning environment where you

can learn enough skills to get a job.

That's literally all the

hardware you need to get a job.

I went through every

single article on your blog.

It's amazing.

So thank you first of all for taking the

time to write them,

and then to share them.

And out of all of them, there's one that

resonated with me the most.

Do you want to guess which one?

My favourite one is:

"I'm in love with my work"

So I'm not sure if that's the one,

or it's "The power of writing".

It's either that one or the other one.

Okay, so I'm going to read a quote and

you'll recognize it, I think, after

the second sentence.

Once you decide on your occupation, you

must immerse yourself in your work.

Yeah, I know.

You have to fall in love with your work.

Never complain about your job.

You must dedicate your

life to mastering your skill.

That's the secret of success and it is

the key to being regarded honorably.

Who said that?

Who wrote that?

Jiro.

Jiro.

Who's Jiro?

Tell us about Jiro.

Jiro.

Oh, Jiro is the protagonist or the main

character of the

documentary "Jiro Dreams of Sushi."

And when I first saw this documentary, I

did not appreciate how

deep this actually is.

But Jiro is a sushi chef who operates out

of a subway station in

this tiny little restaurant

with like four seats in it.

And he is the master of

sushi of the entire world.

Like there's nobody better than him.

And I know nothing about sushi.

I have no interest in sushi.

But this documentary is the story of him

like committing to his craft

of becoming a sushi master.

And what they call in Japan a "shokunin".

Someone who is completely dedicated to

their craft and just doing

the same thing over and over and

over and over again and making it perfect

and even more perfect the next day.

That is the shortest

description I could give you.

That's a great one.

So this restaurant is in

Tokyo, by the way, Japan.

So if you're there, I forget the name of

the underground station,

but you will find it because there's just

one like this in the

entire world actually.

It's a three Michelin star restaurant.

Only one of its kind in the world.

And some say Jiro is the best sushi chef

that has ever lived.

Now, this documentary was recorded, it

was actually was it came out in 2011.

Jiro was 85 years old.

And he's been doing this.

He's been doing sushi

every day since he was 19.

Yeah.

Today, Jiro turns 99.

I say today when we're recording this

actually maybe a few

weeks ago, he turned 99.

I'm wondering, is he still making sushi?

Well, the last time I checked, that's a

couple of years ago.

Then he was also in his 90s and he was

still there every day in his restaurant.

That's amazing.

I didn't realize this, but in the

Japanese culture, apparently retired is

not that well understood.

No.

When we first talked, I had a title in

mind for our conversation.

Do you remember what that was?

The Note Taking Santa Claus.

The Note Taking Santa Claus.

That's it.

Okay, we will get to that in a minute.

I hope or a few minutes we'll see.

But after watching "Jiro Dreams of Sushi",

and reading all your blog,

I came up with a new title proposal.

Let's see if it's as

good as DevOps Sushi.

I was thinking:

"The Bear that Dreams of DevOps".

The Bear that Dreams of DevOps.

Wow.

Okay.

Why is the bear an important thing for you?

Because I know it is, but why is it?

Because people listening

to us, they won't know.

You really have read all of my blog.

I'm so honored that someone...

This also is featured on the blog and why

I chose the logo at the time that I had.

And Mischa is a Russian

name that means"little bear".

So I'm conveniently

ignoring the little part here,

but it might not seem on the camera, but

I'm Dutch and I'm almost two meters tall.

So I'm a pretty big guy.

And in Norwegian, they

have this word called Bamse,

which also means teddy bear.

But they usually describe that with

for long, big hairy guys.

And well, if you're watching this on a

video, you'll see the connection there.

For our listeners, imagine Thor.

Wait, let me get my hammer.

Exactly. Yeah.

That's the only

thing missing from the picture.

Yeah...

So another thing which I watched over the

weekend on top of "Jiro Dreams of Sushi"

which is a great movie,

10 out of 10, highly recommended,

is "Why I use Kubernetes for my Homelab."

A video that you

posted not that long ago.

We'll put a link in the show notes.

And I'm wondering if you

could take us back to the day

that you decided to use

Kubernetes for your home lab.

I think it's a very interesting choice.

How did it start?

That was when I was

working in my first DevOps role,

and we were using Ansible for everything.

and we were using Ansible for everything.

And they had built this super, actually

very impressive system

of deploying Docker servers and then

having three Docker servers,

having running the same container and

then putting them there.

and then putting them there.

But when one of those

containers would die,

then we would have to

then we would have to

go in and fix it, right?

Because you're limited by the tooling,

even though there's probably ways of

doing it with Ansible alone.

That was my reality of the day, of the

situation I was working in.

And when I then for the first time

created a Kubernetes cluster,

and I created a

deployment with three replicas,

and I killed one of those containers,

and I killed one of those containers

and then it came up

automatically, I was sold.

And then when I did a rolling update,

where I would update the image tag,

and then it would go one by one, and then

update the containers.

And if something would go

wrong, it would roll it back.

That's when I fell

totally in love with it.

And I knew I could not go back.

and I knew I could not go back.

For me, that was the

moment where I thought,

this is what I want to work with.

this is what I want to work with.

When I then decided to do my

home lab, to create a home lab,

like it wasn't even a thought,

like it wasn't even a thought,

I didn't ever even

consider anything else.

It was just, yeah, I'm

specializing in Kubernetes.

specializing in Kubernetes.

Of course we need to run

Of course, we need to run

Kubernetes in my home lab.

Kubernetes in my home lab.

Even though it's completely

overkill, and it makes no sense,

it had to be done on Kubernetes.

Well, actually, I think

it makes a lot of sense.

And you talk about this

in your video as well.

Why it makes sense,

and why it is important.

And the majority would react that way.

It's overkill.

Of course it's overkill.

Why not use Docker?

You started with K3S.

Why?

Part of me wanted to go the kube-adm way,

Part of me wanted to go the Kubeadm way

and just do it properly.

and just do it properly.

Even kube-adm is a bit

easier than doing it the hard way,

like compiling the

binaries yourself, etc.

binaries yourself, etc.

Still a goal that I

have for maybe next year.

have for maybe next year.

I went for K3s because I recognized

I went for K3S because I recognized that

the sooner I got up and running,

and actually working

and actually working

with the Kubernetes API,

and feeling that I had

and feeling that I had

something stable to work with,

the quicker I did that,

the more I would learn,

the more I would learn,

the quicker I would start learning.

the quicker I would start learning.

So I think it might felt

like a shortcut at that time.

but I thought long-term and I thought,

But I thought long-term, and I thought,

well, with kube-adm, like,

like running your own CNI

running your own CNI, and having to

update your certificates,

like, it will work in the beginning,

but if something breaks,

then I'm just sitting

there tinkering for hours.

And I wanted something that was a little

bit more thought through,

and K3S was a very good option for that.

Why not Kind?

I had a few Linux machines

I had a few Linux machines,

I had a few Linux machines

and I really like this

idea of that k3s does,

where you install the binary,

and then you can get a token,

and then you can add

another node to your cluster.

And I actually, you can of

course do this with Kind as well.

But I really like this idea of

maintaining a Linux machine,

maintaining a Linux machine

where you install something on,

where you install something on,

but you still have the

possibility of SSHing into it,

and you still are a Linux administrator

at that point, right?

And I really like that balance,

that I could have those

Linux machines running Linux,

I could tinker with it,

I could still run other

things on that machine as well,

alongside K3S,

and then just expand my

home lab as I went along.

I really like that concept.

Okay, so you mentioned

SSH, which is interesting,

because then you switch to Talos,

because then you switched to Talos

and Talos prides

itself on not having SSH.

Yes.

Why did you switch to Talos?

Well, I mean, that was

that stage in my home lab.

And then later, I was running

applications that I was thinking,

well, this is

actually running very stable,

and my home lab has

now escalated to a point,

where I don't want to

live without it anymore.

It's actually containing applications

that I want to keep running 24 seven,

that I want to have

accessible from the internet.

And then the question

of security comes around.

And you need to, when you start exposing

things to the internet,

you need to really

know what you're doing.

It's a very dangerous situation.

If you don't know what you're doing,

and you're just opening ports and letting

people into your cluster,

they could break out of your

containers and do bad things.

So at that point, I felt, okay, I want to

have things more secure.

And then Talos came on my radar,

which is production grade

security out of the box.

And it wasn't necessarily because I felt

incapable of securing my own cluster.

It was more a combination

of learning a new technology

and also having this peace of mind,

knowing that what I was

doing was actually secure.

So then I switched to Talos

and yeah, I'm never going back.

I think it's been amazing so far.

If someone wants to get really good at

Kubernetes and start a homelab today.

Yeah.

How would you recommend

that they do that?

I'm currently in the process

of building a course for this,

the Kubernetes Homelab.

It's 60% done and people are responding

really well to it in the community.

They love it.

And the approach I give there is,

first I have my

Kubernetes Fundamentals course

where they run Rancher Desktop

on their own machine

and then to learn Kubernetes.

And then the next step is

to actually get a laptop,

a Raspberry Pi or some

sort of dedicated hardware

and then run k3s on it.

That is the approach that I advise to do.

So first learn it in a local setting

and then move to dedicated hardware,

use k3s and learn

GitOps as soon as possible.

So where could people find this course?

The course is currently

available in my community,

KubeCraft that I started at

the beginning of this year,

where I invite engineers to come in and

engage with each other

and where I also have my note-taking

material and my Kubernetes courses.

Obviously we'll add a

link in the show notes.

I had a quick look.

I haven't joined the community yet.

I'll put yet there.

I've seen that it's

on skool.com with a k.

Can you tell us a little bit about that?

Well, maybe I can just briefly describe

why I even started doing this.

As a YouTuber, I was just making videos

because I like sharing knowledge

and then at some point people were

actually telling me that:

"Whoa, we really like your

approach to explaining things

and how you approach things.

Don't you have a course on X or Y?"

And when I realized that my audience

started asking me for this,

I thought: "Well, maybe I do have some

potential of going

beyond sharing on YouTube."

So at that point, I was

like any engineer considering:

"Oh, I should build my

own course platform."

Let's code my own platform.

And then you spend a weekend on it and

then you start looking into

authentication and user

management and all that stuff.

And then I figure:

"Okay, maybe that's not the way."

And then I researched the platforms and

of course there's

Udemy, there's Kajabi, etc.

But what Skool really emphasizes is

combining education with community.

And when I heard that sentence, I was like:

"Yep, this is it."

Because I don't just want people to buy a

course of me and then that's it.

I mean, that's fine, but it's much better

if you can create a group of people

who are gathering around

their interests for those courses

and then can get support from me,

but more importantly from the collective

knowledge of that community.

They can actually draw from each other

and discuss tactics with each other.

And that is proven to

be very powerful so far.

So this community, when you started

earlier this year, started at zero.

Where is it today?

We are currently at 401 members.

Wow, so just past 400, okay.

So it'd be interesting to see how many

members there will be

by the time this comes out.

And you'll have the

delta to see how that grows.

I also know that

obviously this is a paid community

because of the value that you get.

But there's also a free community that

you have also on Skool.

The free community now has 2000 members.

And I only started that a month ago.

Wow!

So that has been very

successful in that sense.

Well, I do ask, I charge a monthly price

for membership of the premium community.

The services that I provide for KubeCraft.

I decided: well, I'll

create this free community

that people can at least come together

and meet people who are...

They're watching my content.

So there is for sure some level of

overlap in their interests,

in note-taking, DevOps and such like.

So people can come there, make friends,

and they can discover the platform.

And I give away a lot of material in that

free community as well.

And then if they like, if

they see the value of it,

then maybe they can consider

joining the premium community.

Okay, great.

We will make sure to add

both links in the show notes

so people can go and

check them both out to see

where they want to start

and what they want to do next.

What would you say is the main reason

that people go from

free to the premium one?

I think there are two kinds of people in

the world in terms of employment.

There is the guy who shows up at work,

does his work, goes home,

and then plays video games

or does whatever he wants

and then he shows back at work again.

And then there's the guy

who is entering the stand-up

who excitedly talks about a new feature

that came out on Talos

or he managed to get this self-hosted app

up and running in his homelab.

And he talks about that.

And I'm obviously the second guy

who likes to tinker with what he does

in his free time as well.

Well, imagine putting 400

people like that together,

all sharing what they're discovering

and what they're learning.

And what you get from

that is a sense of belonging.

I was never able to find this place of

geeks and nerds sharing homelab stuff

and learning Kubernetes and learning my

approaches to

note-taking and personal branding

and doing that together.

And I really found my tribe in this.

I created something that wasn't there and

I really feel at home there.

And there is, we have calls

there as well a few times a week

where you can ask me questions directly

on how I approach things.

But what's even better, I don't try to be

the guru of the community.

I'm all about letting the

community answering things.

Because I can only know so much.

I only have this experience.

When you put 10 experienced engineers in

a call and when you come in there,

maybe someone who wants

to transition to DevOps

and you're in that

meeting and you're saying,

"Oh, guys, I have this

problem in my homelab."

or "At work",

or, "I don't know what my next

certification should be."

If you have all that

experience sitting there in a meeting,

you're going to get the answers and

you're going to get good answers.

And it's amazing.

I'm amazed how well it all works and how

much progress people are making.

Coming back to the technical side, we

have a couple more

things to go through before

I'm very keen to start screen sharing,

and showing our homelab,

because that's what this is about.

Talking about it, but then also showing

what that means in practice,

which I'm very excited about.

What was the hardest issue

that you've hit with Talos?

When I set up my Talos cluster, I had

baked a custom image.

There's the image builder, and then you

can add certain drivers that you need.

But I did that, and I apparently didn't

document that properly,

because I need the

iSCSI tools for my Synology

to be able to provision

storage on the cluster.

And I had updated my Kubernetes cluster

with the Talos update,

so new Talos images are

loaded onto the nodes.

And all of a sudden, all

of my storage was broken,

and I can be honest, I panicked.

Fortunately, everything

is backed up to the cloud.

But that was a big problem at that point.

And then after a lot of debugging,

thinking it was Synology

pods on the cluster itself

that maybe got updated

or weren't compatible,

I finally realized, oh yeah, I had that

iSCSI driver that was missing.

So that was the biggest one.

Yeah.

Anything that happened in the recent

update that you haven't posted yet?

I ran into something where I would

initiate a reboot of the node,

and then in the CLI, it will then, say,

now waiting for the node to come back up.

And then I could see in my closet where

the nodes are standing, it was up.

It was up and running.

But it wouldn't work.

I would have to turn it off and on again,

and then it would reboot properly.

Like, I decided not to spend too much

time into why that was.

At least I knew how to fix it.

But that was an

interesting one that I had this time.

So this is a tip from the experts.

Turn it off and on

again, and it will fix itself.

That's one.

Which version did you

upgrade from and to?

I went from Talos

version 1.7.5 to 1.8.2.

That's it.

Yeah.

I assumed this was it.

From 1.7 to 1.8.

Oh, really?

I did went from 1.7.5 to 1.7.7.

So I was a good boy, and I followed the

latest patch upgrade,

and then went to the next minor version.

But you had the same

problem too, apparently.

So I don't do in-place upgrades.

It's one of my hard rules.

I never do that.

Actually, my clusters, they have a date

on the cluster name,

which tells me exactly

when that cluster was created.

And I never upgrade minors.

I rarely upgrade patches.

I just need to know when.

Even Kubernetes, sometimes I never

upgrade, for example,

from one Kubernetes minor

to another Kubernetes minor,

even though most experiences are good.

The problem is everything else that you

have running in the cluster

is very difficult to

reason about the combination

of the different tools that you have,

because it's not just Kubernetes,

not just the operating system.

There's also everything else.

What I found is that if

you do in-place upgrades,

you're risking it.

The more upgrades that work,

the more likely you will

hit a bug at some point

where you wish you hadn't done that.

Yeah.

If you don't do in-place upgrades,

it forces you to encode

the system in a certain way.

It forces you to restore from backups.

It forces you to have a certain approach,

which is operationally more mature.

It's a bit harder, but you

can sleep safely at night

knowing that you can restore everything,

even if it was to burn down,

because everything is configured,

everything is backed up.

You just point to a new cluster,

everything gets restored

or you create a new cluster,

you give it a new config

and it boots as a new cluster.

Most of my workloads,

they have built in a

way to restore themselves.

What that means is that

in the init container,

when a workload starts,

it looks if it's the

first time that it's booted,

it's like it's a bootstrap phase.

And if it's a bootstrap phase,

it will go and restore

from an S3 compatible API.

Nice.

In my case, I use two.

I use Backblaze B2 and Cloudflare R2.

And I do this via rclone.

So rclone is the first thing that restores,

even MySQL.

You'd be surprised

how well this stuff works

for even for large data,

because you're pulling

down from object storage

that is supposed to be performant.

Backblaze is not as

performant as Cloudflare R2,

or at least in my experience,

but I have both because you

don't want to put your eggs

in one basket, especially backups.

So I have two of

everything, another rule that I have.

The other one is don't

do in-place upgrades.

It's not worth it.

So when you do an upgrade,

then you would actually just spin up an

entirely new cluster?

Okay.

So also just like completely new,

well, not new, but

like different hardware,

spin up a new cluster,

and then you decommission the old one,

and that becomes your staging cluster.

I'm thinking of it blue-green,

but long-term blue-green.

I'm always between blue and green.

And it removes all the pressure

from having to do the

upgrades straight away.

It'll either work or not work.

No, I'm setting things up.

It's almost like buying a new car,

but in this case, it's more practical

because it's not a car.

You do need to have a bit more hardware,

but you have that anyway,

because you have generations of hardware.

You have your old laptop and the older

laptop and whatever.

So that's my case as well.

Also, very controversial,

my clusters are single-node.

Ooh, spicy.

I know, right?

Yeah, you can get some pretty big

single-node instances.

And unless you really

understand your networking

and can deal with etcd,

and you can do uneven numbers,

knowing how Raft works

and how leader election

works and all of that,

you need three, five, seven,

you're pushing it a bit.

If you go beyond nine,

you're basically like

in a whole new world.

I think very few have more than nine

nodes in their homelabs.

But that should be the absolute limit

for many reasons.

These systems are so advanced

that they will continuously converge,

and you can have problems

and not even know that

you're having problems.

That's how refined they're getting.

And usually when you get into DNS

and if you get into network latencies

and switch upgrades,

and you have to deal

with all these scenarios.

Single-node, you don't.

In that metric,

I am currently at four,

but I had five for a particular reason.

But you have a single control plane?

Currently, yes.

Right.

So that's slightly different.

So single control plane,

that's almost like the middle ground.

So the control plane is not HA.

In my case, my control plane is not HA,

but all my workloads fit

on the control plane node.

Yeah, I see.

And then I have two right now,

and then basically one

is blue, one is green.

So I have two home labs,

and I just switch between them.

And it takes me months to

go from one to the other.

So nothing is rushed.

I can experiment, I can try things out.

And if one was to break,

I always have like a

spare that I can go back to.

That's the other thing.

I used to have a staging cluster

and that ran exactly the same.

And I would just then update the GitOps

for the production one

if everything went right.

However, I actually

started needing to expand

my production cluster with extra nodes.

And I cannibalized some from my staging.

And so that one's not

operational anymore.

And to give you an

alternative perspective

on not doing in-place upgrades,

I found that it makes

me think about my systems

to have them more

resilient and easier to restore.

And I'm not in the stage where I have

auto restore like you,

but what I emphasize a

lot on my current cluster

is I don't want to have anything

persistent outside of databases.

So my databases are

running in with the EDB operator,

Enterprise DB.

And then using Barman,

I'm also storing those in object storage.

And that has been a really good exercise

like four times already.

I lost all of my databases

because I was a bit too experimental.

Tinkering.

You were tinkering.

Yeah, it happens.

That's how you know that

you're tinkering the right way.

When you break stuff,

you're pushing the limits.

So I've actually gained a lot of

confidence in my setup now

that I can actually

just delete the whole thing

and then restore it from object storage

because everything is in databases.

And anyone who's listening to this

databases on Kubernetes

is easier than you think.

And it works extremely well with the

Barman object store.

It's that.

I'm so, so excited about it.

It's super cool.

In my last Homelab

presentation, which I gave at Taloscon,

I use CloudNativePG, but

also Barman and object storage.

And that's exactly how I migrated from

homelab to production.

Because in this case on homelab,

I was running a workload

that was backing out via Barman

to R2, Cloudflare R2 in this case.

And when I set up production,

I was restoring

everything exactly the same way.

Would you recommend iscsi-tools?

Would you recommend it?

Like does it work well,

the extension in Talos

and like the whole setup?

Because I'm thinking of

setting up the same thing.

I don't have that yet.

But would you recommend it?

Like I can't speak to

iSCSI tools specifically

because I just run that and it works.

So I have a Synology NAS,

and then I use the iSCSI tools in Talos

that needs to be installed.

And then there is a CSI driver for

Synology for Kubernetes.

And that allows me to

provision persistent volumes

on my Synology NAS.

And I must say I'm really

impressed how well that works.

I know that you started with Argo CD,

but you're currently using Flux.

Yes.

Why?

I work with Argo CD at work.

And when I first learned Kubernetes,

I was super excited about Argo with UI.

And still, it's like a fantastic tool.

That was my first GitOps exposure.

But then I'm a

specialist in Microsoft Azure,

and I'm also a Microsoft MVP.

So I focus a lot on

Azure Kubernetes Service.

And in terms of

enterprise-grade Kubernetes,

I like to, like, as I mentioned before,

I like to explore and experiment,

but I also need to focus myself.

If not, I get too broad

and I just get overwhelmed.

So in terms of

production-grade Kubernetes,

I decided to focus on

that ecosystem for a while,

which has been a good

decision in my case.

And the native GitOps

offering in AKS uses Flux,

which I was very surprised by

when I first tried that out.

Like: "Oh, that's an interesting choice.

Why wouldn't they go for Argo CD?"

And I don't have the

answer to that question.

But all I know is that when I started it,

I was really surprised how well it works

and how well it fit me.

It fit me better than Argo CD.

What specifically

about it fit you better?

Argo CD has a beautiful UI,

which makes the

navigation of a Kubernetes cluster

possible almost

exclusively through the UI.

So you can do almost

everything through the Argo CD UI.

And I noticed at work as well

that I was kind of

losing my CLI ninja skills

in terms of kubectl

because I was deploying

so much through Argo CD,

which was, I saw as a problem.

And what I like with Flux

is that it's much more CLI focused.

And of course, like, usually you're not

actually interacting

with the cluster in that way.

But in terms of my homelab experience,

I want to be in there and tinkering

and just talking to my cluster directly.

And then the Flux CLI

felt very intuitive to me,

even though Argo CD also

has a CLI, I'm aware of that.

But the way how Flux does

it with the custom resources

such as Helm releases and Kustomizations,

and being able to check those resources

and Git repos, all of that,

I really like the way how that is set up.

And also, I completely love the way

how Flux bootstraps onto clusters,

how you can make your cluster like a

self-feeding machine.

It's just super awesome.

Okay, interesting.

How was the migration

from Argo CD to Flux?

Because you had like a bunch of things

all declared in Argo CD, I imagine.

Was it straightforward to migrate to Flux?

At that point, when I was using Argo CD,

I was getting very Helm focused.

And at some point, I realized,

well, not everything that I want to run

has Helm charts readily available to me.

So I need to dig deeper into this.

Then at the same time,

I was discovering Flux.

And there is a lot of

emphasis on Kustomize.

Basically, everything

works with Kustomizations.

That was basically an incentive to

finally learn Kustomize.

I had been putting it

off for the longest time.

I just tore everything down and rewrote

everything in Kustomize,

which has been a really good experience

in terms of my development.

And Kubernetes learning.

When Rio mentioned you,

he caught my attention.

And that was him mentioning how you spent

nine years in Norway.

Roaming the wilderness.

Yeah.

Reconnecting with yourself,

reconnecting with the nature

and remembering what is really important.

Well, I must say, like, it wasn't that I

was doing that full time, right?

I was just living

a normal life and working.

But basically, if I wasn't

like tinkering with computers

during the evenings in the week,

every holiday, every weekend,

every opportunity I got

was spent being in nature.

was spent being in nature.

This will sound so cliché.

But one of the reasons that got me into

this was the movie "Into the Wild".

And just that phrase, "into the wild,"

I have since become less of a fan of

Mr. Christopher McCandless,

who actually turned

out to be rather stupid.

But when I was young, I didn't actually

appreciate his mistakes.

Let's keep it that way.

But that idea of being

out and being self-reliant,

that really applied

to me, and spoke to me.

And...

I have since come to realize that

that is also not possible.

So that's also why I was...

that I went back, basically.

But the idea of the experience of just

strapping on a backpack

and driving your car somewhere with a

vague idea of what you're going to do.

And then I would just bring a sack of

oatmeal and my fishing pole.

And I have since turned plant based,

but then I was still

killing animals and eating them.

But that did allow me to be out there

alone on a self-reliant matter.

And when you do that, then you...

No phone, alone,

in the wilderness, no people.

First of all, you become

really aware what you're doing,

because if you make a wrong step and you

fall, you break your leg.

And that is almost

certainly going to be your death.

So you have to be...

You have to know what you're doing.

And you have to be really

aware of what you're doing.

And when you're out there

not talking to people,

just there with your own

thoughts, things emerge

that would not emerge in the

day-to-day stress & busyness

that we all experience.

And I've had some very

deep philosophical insights

and discoveries about my own life.

Just sitting there, by the water, staring

into the fire that I built myself...

for hours on end.

And you'll be amazed what

the mind then brings up to you.

So that's an indication

of what that was like.

Do you still do that?

I mean, do you still make time for going

out there and being with yourself?

No phone, no people.

I think an extension of this whole

wilderness roaming that I did

was that I got really into meditation and

Buddhist meditation specifically.

And I have now found ways

of getting that experience

decoupled from being in nature that way.

So I'm a very consistent meditator.

I meditate for several

hours on end sometimes.

And I moved back to the Netherlands,

and here nature is much

more difficult to find.

It's here, I have

nice areas around me,

but I do really miss being in the forest

and just knowing that

you are alone there.

And the idea that you cannot

meet anyone,

or it's not likely

that you'll meet everyone.

Like here in the Netherlands, there's

always someone around you.

I did go back to

Norway a couple-- one year ago.

I spent three weeks

driving around and just roaming

and having the time of my life.

Last year, I didn't do it, and now it's

really itching to go back.

So I try.

I know that each of us has a

place where we feel at home.

And it's rarely where home is.

It's usually like somewhere maybe nearby,

or it can be a

different country for sure.

But when you're there,

you feel like you're home,

and it's very hard to explain because

everything is the way

it's supposed to be.

Everything smells right, it sounds right,

you feel very at ease with yourself.

Is Norway that place for you?

That's a deep question.

That's one that I...

Like Norway is

extremely close to my heart.

Like one experience that I had

was when I moved back home,

I hadn't visited for two years.

And then last year, I went there

and I drove off the boat

and it felt like it didn't feel special.

That was interesting.

It felt like, oh, I'm here again.

So in that sense, yes, it does feel like home.

But I'm also really starting

to like my little boring life

in just a one-room apartment

and focusing on my

and focusing on my

business and the technology

and having a more

routine life in that sense.

I think this is potentially

the most important takeaway

from our conversation.

Going outside.

Slowing down.

You don't necessarily have to go outside,

but usually it helps.

Slowing down your mind, and...

finding your happy place.

It's usually in your mind.

The happy place is in

your mind, by the way.

Outside, walking, being...

free from distractions

usually helps that surface.

But ultimately, you

still have to do the work

and you still have to settle your mind

and go through all the steps.

But it's definitely out there.

Everyone has it.

Yes. And nature is a

great facilitator for that.

And I think in my case, nature was the

gateway to inner peace.

Nature was the place where

I could disconnect that way

and sit by the fire and experience that.

And still, of course,

still nature is that for me.

Last year I was there,

I walked 150 kilometers

in the mountains and just alone.

It makes you feel very small

when you're in those big mountains

and a thunderstorm breaks out

and it puts so many

things into perspective.

And how powerful those forces are

and how little we are

in comparison to that.

That is also very healthy for me

to have that humbling

experience every once in a while.

That's it. And when you come back

to whatever you normally do,

you'll be so much more productive

and things will flow a lot better

because your mind took a break.

Absolutely.

Okay. Well, Mischa, thank you very much

for joining me today.

It's been a pleasure talking to you,

looking at your homelab, you know...

Going through that

stopping and starting phase.

That was very, very fun.

And I'm very much

looking forward to the next one

because I haven't shared mine yet.

I mean, I really look

forward to seeing your homelab.

And we have to do a round three.

We will. Yeah. Yeah.

For sure. Thank you.

I'll see you next time.

Thank you, Gerhard. Bye-bye.