Pairing-up on a CDN PURGE with Elixir
Download MP3The focus for today is to get this Pipedream global purge working. And this pull request starts it. It doesn't have that many changes. My focus was to describe the problem. So we don't want to merge it as is, but this will give us every commit to this branch, which is on the changelog.com repository on the upstream one.
Gerhard:Any change will automatically deploy to production, which we don't want because this is exactly the sort of behavior that we don't want from a pull request or a branch. But for the purpose of today, we want that fast feedback loop. The application that we're deploying to is not production. It is the one that has been set up on the site, so 20250505. And this is the one that will replace the current production once we are happy with everything.
Gerhard:But for now, it's our experimental new production that will have this new feature. We can test it in production without it being the real production when it comes to traffic. But as far as pipedream.changelog.com is concerned, that is pointing to this application. So one thing that would help is to look at the headers that comes back from this request, and you can see the same cache status. And in this case, you can see that as a hit.
Gerhard:No bypass. Exactly. There's no cookie. So this is exactly and if you do it again, you can try another request. Hit one, still the same one.
Gerhard:The first one was stale. Okay. So that that explains it. The first request, if you look at it, it was stale Right. Which is exactly, you know, one thing that we're trying to improve, so that it's not the request that refreshes the page, we can explicitly purge the cache.
Gerhard:Mhmm. So that one, if you look at the TTL, it had a negative two six six. So that was stale for about four minutes and a bit, that's in seconds.
Jerod:Okay. That means that it's been stale for a while, but there hasn't been a request for it, so it hasn't Correct. A reason to refresh, and now
Gerhard:That is correct.
Jerod:We hit it, and so now it's gonna be
Gerhard:44 default. Yeah. And if you hit it again, let's see what happens since we were talking. What do we see back? Yeah.
Gerhard:So it was stale for three seconds.
Jerod:Barely.
Gerhard:Because there isn't yet. There there I mean, if you if you hit it again now, you'll see that it's fresh. Right. It has refreshed in the background. The t t r is now fifty seconds, so the countdown has begun.
Gerhard:Now if you want, we can try a purge really quickly to see how that behaves. Just understand the mechanics. It's on the same Mhmm. Tab that we were looking before.
Jerod:Like dash x purge or something?
Gerhard:Yep. That's the one.
Jerod:Oh, you gotta do. Purge.
Gerhard:That's the one. Purged. There we go. And if you run the request again, you'll see that it's miss. So that's the purge.
Gerhard:It shows us how it behaves in practice. This is just one instance, but honestly, we need to do it across all Right. The CDN instances we want to purge. So that's that's what makes this more challenging.
Jerod:Which was why you have this DNS lookup to get all of the instances. Yes. So this assumes because the address is dot internal, this assumes that you are on Fly's network. Right?
Gerhard:That's correct. Yes. We're using the Fly DNS. It has some extra records like this one, for example, which is an IPv6 record
Jerod:Mhmm.
Gerhard:For the application name dot internal. And when you hit that request, there's a for the record, if you see there is like a link above the private networking. It was in just after a screenshot. That's the one. So this shows us the fly internal DNS.
Gerhard:And if you scroll a little bit further down, it tells us what is available, whether it's IPv6 Mhmm. Or text DNS records, what do they contain. So this is a way to discover properties about applications, including a list of all applications within Fly that your org has. We are looking at the first one. Right.
Gerhard:Appname.internal, which gives us those private addresses for all the machines.
Jerod:And there's your dig command that you put in the pull request.
Gerhard:Nice. Because you have WireGuard set up. Very nice. I do. Someone prepared.
Jerod:So ready.
Gerhard:Very nice. Oh, yeah.
Jerod:Be ready to rock.
Gerhard:That's perfect. Nice. How would we solve this the elixir way?
Jerod:Okay. So here you have effectively, what you're doing is here's your mash one liner. So we're taking this
Gerhard:Mhmm.
Jerod:Which effectively just does that exact same dig request, gets all these addresses, loops over them, and runs a curl command for each one with that x purge and the address. Right? Yes. And then it prints it out. So we basically just need to get this into our app the Elixir way, as you said.
Jerod:Yeah. The first thing we're gonna need is a way of doing that same dig, or effectively the same DNS lookup, which I already have done. I don't know if that's cheating or not, but Claude wrote it for me, and I just tested it and it works. So we have that code, I can just walk you through that, and we can run it. We're basically replacing fastly dot purge with pipely dot purge or whatever we call it.
Jerod:Yep. Pipe dream dot purge. First let's talk about the DNS thing. So this is not my code, but it's probably better than I could write it. It's just a new module that Claude wrote with an AAAA method, and returns the addresses.
Jerod:And I've test driven this, I know it works. Here it is right here. So we already have the DNS bits done. Super interesting, but for those who are interested in the way it works, it's basically just using Erlang's inetres module, wrapping that in some stuff to format, etcetera, and get us back those addresses. Those addresses do match, correct?
Jerod:Yeah, they look the same.
Gerhard:Yeah.
Jerod:Yeah. So we're good there. So we have the DNS lookup finished. The way we currently purge with Fastly is I have a Fastly module. It's very similar to what we're gonna do here, pass in a URL or a schema struct to purge it from Fastly's cache.
Jerod:And Fastly's cache used to we probably go back in the history here because this used to be a purge request as well. And they actually changed it to an HTTP post at one point without telling us. And so our my my Fastly leap urges were failing for a long time because I was doing the old method and I think they deprecated it. Maybe they told some people and, you know, I don't watch their blog enough or something. But if we look at this, there we go.
Jerod:Adjust fast leap urged. So eight months ago, had to go in and rewrite this. See if I can look at that diff. And you can see this used to be
Gerhard:HTTP. I can see the
Jerod:request. Purge. We're fake then we can steal some of this back. That's basically what we'll do with our Pypely purge is this method instead of this method. The way this works is you if it has a URL or binary string, it just calls that.
Jerod:And if it has an actual object
Gerhard:Mhmm.
Jerod:Well, I guess it's not objects in Elixir. If it has a struct, then it will basically be like, oh, this is an episode, and now I'm gonna be smart about what I have to purge for an episode. So my question for you is, and like, if there's an episode, it's going to purge the audio file, the plus plus file, and the cover art. So if we edit an episode, we say, hey, purge this episode, you know, there's multiple files that need to actually be purged. What I don't know with Pipely if we wanna be this fancy or not, because in your example, you're just purging the root URL.
Jerod:I assume that's just as an example. But do we wanna be precision and say purge this URL, or do you wanna just tell that instance to just purge everything, and the next request is gonna be fresh no matter what?
Gerhard:So I think it would make sense to have the same purge that we have here. Let's let's have a little bit of a sidebar between Pipedream and Pipedream. Okay.
Jerod:I've been wondering because it seems like you're using them somewhat interchangeably.
Gerhard:Yeah. Pipedream, the way I think of Pipedream, is the specific implementation for Changelog.
Jerod:Okay. That's part
Gerhard:of the Pipedream. Pipedream. Exactly. Pipedreamchangelog.com. Okay.
Gerhard:And as a project that anyone can use, the open source project, that's why it was Pipely because of the name. Right? Pipedream.com is taken, so pipely.tech is the open source variant that anyone can do their version and they can call it whatever they want. In our case, it's Pipe Dream. I really like that name.
Gerhard:I think it's a really good name.
Jerod:Yeah.
Gerhard:And it's just close to Changelog. But globally, it doesn't make, I think, as much sense as Pipely.
Jerod:Okay. So Pipe Dream is what I should be referring to because that's our instance of Pipely.
Gerhard:Exactly. It's also like the subdomain, pipedream.changelog.com.
Jerod:Right.
Gerhard:So pipe dream itself, it fronts both static assets, feeds, and the application. So it has like these three back ends. Mhmm. So what it means is that when we want to purge something about an episode, we have to hit all the URLs that make up the episode so that we explicitly purge them. Right.
Gerhard:So I think this is correct. Okay. The question is, how do we determine what the URLs are?
Jerod:Right. Which that logic is embedded already right here. So we're basically gonna create a pipe dream module that very much mimics this module, only this function will be different. And this function will call the DNS function to get the a records, the AAAA records, and loop over them, which is what your curl command is doing, which is pretty straightforward, right? But then all this this logic, we're just gonna basically copy over and do it over there.
Jerod:And then we have all these fastly.purge call sites, which I think I have a search up here. So then throughout the app, we already have the logic in place of, like, here's when when we purge things.
Gerhard:Mhmm.
Jerod:So I imagine we just go through and replace fastly.purge, or maybe for now, we just put it right next to fastly. Purge, pipe dream dot purge. You know, then we just have both. And so that logic is already there. That's why I said this would be pretty easy feature.
Jerod:So I would start by creating a pipe dream module then. And because I'm lazy, I would probably just copy paste this into it and start to replace stuff.
Gerhard:This is one of my favorite ways of starting. Just copy paste. Yeah.
Jerod:So it's
Gerhard:like, you know, it's
Jerod:not fastly as cache.
Gerhard:It's pipe dream
Jerod:cache, you know, like you're
Gerhard:just Some thought has gone into this. Right? So you're just basically building on top of that thought. So this is great. Yes.
Jerod:And then this is like the actual purge endpoint. I just don't think we need this for this.
Gerhard:Yeah. Okay. Yeah. No. Because we will need to determine what the purge endpoint is for every single instance, indeed.
Gerhard:Yeah.
Jerod:And I think these logic this logic is all identical. Like, there's no reason to change any of these functions, so let's just like hide all those. And then this is the only function that really matters. Because all of these, I know this because I wrote it, all of these basically find all the URLs for those structs, and then call this over and over again for all the URLs. And so this is the only one that really matters.
Jerod:And since it's the only one that really matters, I'd probably would write a test for it. So I would probably just not wanna have anything there and write it fresh. And
Gerhard:then Look at that. Proper TDD. This this is what TDD's supposed to be.
Jerod:Yeah, man.
Gerhard:You write your failing test, and then you write the implementation.
Jerod:That's nice. So test the change log. And look at that. I still I I have I'll duplicate this file Perfect. And call it PipeDream test.
Jerod:Oops. I deleted the wrong part. Pipe Dream test. Alright. And role plays fastly with Pipe Dream.
Jerod:It's gonna be a little bit more complicated because we will have a loop in there. And purge when given a URL string. Okay. So the way this was running is basically we mock, because this is effectively a unit test, we just wanna make sure the right HTTP post call happens. This is on the Fastly side.
Jerod:And we mock it, and then we set up a URL, we call it, and then we assert that it was called appropriately. Not, you know, like all unit tests, not super useful, but still good enough to think through it. Alright. So there's pipe dream. It gives a URL, purge with the URL.
Jerod:Now the mock is actually the thing we're gonna think through, so we're not gonna call HTTP post. And by the way, this is changelog.http, which is basically a wrapper around h t p poison that allows us to, like, pass certain things. Most basically, I used to have HTTP poison calls, like, throughout the application. And then there were times where, you know, SSL and TLS things would fail, and I would have problems with my outbound request to, like, Fastly or Stripe or whatever. And so I just had to inject, for instance, this option into, like, every single HTTP poison call.
Jerod:And that's when I was like, well, let's just have our own HTTP module that just wraps these HTTP poison requests, so I can always have my options in there. So that's the entire point of this module. It's pretty worthless. But
Gerhard:Well, I think as a as an interface and as an, like, abstraction is really helpful in the sense that if the implementation was to change, the wrapper would stay the same, maybe they'd have another one, but the idea is like the the interaction between the actual library and our code is abstracted away, sits behind this nice interface. So I can see the utility of it.
Jerod:Yeah. That's why I do it. There's not when you look at it, you're like, okay, you're basically calling the same exact function inside this and passing the same things, only here I have these shared options, which all of these use. And so I can inject my own options that are kinda global to my HTTP request, And that's kinda I
Gerhard:think this makes sense.
Jerod:Yeah. So here's how you call that. So request method URL body headers and options. And so instead of post now, we're gonna inspect request. And request has one, two, three, four, five, but the last three are all optional arguments.
Jerod:And so we're gonna just call it basically with a method and a URL, because our purge request doesn't have anything special. Now, may wanna include, like, a token or something to make sure that we're the ones purging, but that would be like a phase two for me, I think.
Gerhard:Indeed. Same I'm thinking you'd buy to
Jerod:the exact And same then make it.
Gerhard:That's it. Right. That's it. Yeah. Exactly.
Gerhard:Oh, yeah. Okay. That's it.
Jerod:And what does it return? Well, this is interesting because we're actually gonna call this n times, where n equals however many pipe dream instances we have. Whereas on fastly.purge, it's a single u URL. So do we put our loop in here, or do we assert it's called a certain amount of times? That's a question that I'm still thinking through.
Gerhard:I think asserting how many number of times it's called. So a loop, a minimal loop in my mind is more than one, which would be two. So as long as there is there's like two elements somewhere, and as long as this request is sent twice, it means the loop mechanics are correct.
Jerod:Right.
Gerhard:Now, we could obviously, you know, like, just mess about with the actual code. We can just, like, call it twice, but I don't think we should. I think we should in in in implement the loop properly. Yeah. To begin with, whenever I would use the new TDD, I would get the first version working correctly, which is like one request.
Gerhard:Right. And then I would test for the loop. In that, if there's two elements, I get two requests. Right. But I would like build on top of that one request.
Gerhard:I think one request to begin with, think it's, as a starting point, is is good enough because we still haven't implemented anything on the other side. And once we have implemented some code, we can add the looping part because I'm still not sure how the looping part is going to work. Like, are we going to pass the arguments in the function? How are you going to tell it what endpoints need purging?
Jerod:So my plan would be basically, like, call out to DNS to get list of addresses, loop over set addresses
Gerhard:Yeah.
Jerod:And then, like, call h t p dot request once per address.
Gerhard:Right. But because you're TDD ing this Right. You have your first problem. Like calling out DNS Yeah. Would be problematic.
Gerhard:So instead, I'll just pass it in. Think functional. As functionally, you have purge, and the second argument is a list of endpoints to purge. That you can very easily test.
Jerod:Oh, you want me to add it in here.
Gerhard:Oh, yeah. That's exactly what I'm thinking.
Jerod:I don't like that because that changes this call site for all of these. Like, these are all gonna be calling purge, And they actually don't care about the the address. This is like, you know, addresses or instances. I'll probably bypass that. And honestly, I would probably just change this to with mocks and have two mocks instead of one.
Jerod:And I would mock the DNS request, and then I'd mock this one. So change log dot dns. So now we're just mocking out this. And we're going to return the addresses that we want, and so we can test different things.
Gerhard:I see.
Jerod:Okay. So we can do that, and I might be doing this. You give it a singular host name, and it returns okay. And then we're gonna return eight six seven five three zero nine, which is not a valid IPv6 address, but it won't matter because we're not calling anything real. Is that how that works?
Jerod:Oh, it returns a list of addresses. So we'll do it your way in a certain extent as we were oh, I did return a list. I just got one of them in there. So that's kinda your way, kinda my way. So happy middle ground there.
Jerod:And then end, and then this. I feel like the syntax is probably wrong. Is that right? Yep. Is that looking right?
Gerhard:The first one looks right. Yes.
Jerod:This, and then an empty list, and then another list of the actual functions you wanna mock. And that's your first one. And so here, and then an empty list, and then a another list. Let's just, like, even see if this compiles. Well, it fails, so it did compile.
Jerod:Okay. So I think that syntax is gonna be okay. The URL is fine. K.pipedream.purge. And we'd expect this to fail.
Jerod:But we'd expect it to return or we would expect it to call it with IP. Eight six seven five three zero nine. Right? Mhmm.
Gerhard:And the port. 9,000. And the port.
Jerod:And now is that gonna be like a hard coded thing for now or what?
Gerhard:How's that? Well, we could pass it through the environment variable like another configuration. But I think for now hard coded is fine.
Jerod:And we expect this purge to be called with two arguments.
Gerhard:We can try one to see if when that works, and then we can add another one, another I p v six.
Jerod:Yeah. So we expect this to fail, and it does. So then we come over and gotta make it pass. To make it pass, we say change log.dns.aaa. And now we needed to make another decision, which is, like, how are we getting this address into there, which is our this one.
Jerod:Like, is this stored anywhere in the application? We're gonna hard we can do this for now?
Gerhard:Currently, but we yeah. But we definitely need to configure this. It needs to be configuration property for sure, because this will change.
Jerod:We're effectively getting back a list of IPs. I'll just store that. Change log dot DNS. Alright?
Gerhard:Doesn't return any errors. Like, who Right.
Jerod:So the way I would write this is actually I would create it as a case statement. Yep. And I would say, okay, addresses, And then I would put that here, do stuff. And then I would say error and reason. And then obviously, that would have to do something, but we don't have to figure that out yet.
Jerod:So here's where our logic goes for that. And now we want to call our HTTP dot request purge. And we gotta construct now that actual oh, this is a list of addresses. Mhmm.
Gerhard:Yep. So we
Jerod:have our loop. Our loop would be in here. But for now, we're returning one. So let's just do the one. So we just pattern match the one.
Jerod:We hard code the port, like you said. And then we might have like a full resolved URL being passed into this and not like a a path name. So we might need to fix that actually in all these and change these to paths instead of URLs. Let's see what it looks like. Alright.
Jerod:So I expected it to get called, it did not get called. That's one of the problems with mocking is, like, you don't actually see exactly what did get called. It just says, did not get called. Let's actually assert that this got called. Let's just take a step back from this and assert called changelog.dns with what?
Jerod:With
Gerhard:The name of the application, I think. Yeah. Don't know.
Jerod:The hard
Gerhard:code one,
Jerod:which is asserted. You can also tell it you don't care what it was called with, but we'll do that. Yep. And let's just see if that gets called. I mean, that should get called.
Jerod:Right? What am I doing wrong here? Called expects more. So how does call work? Oh, you actually call it.
Gerhard:Dot a a a. Must be.
Jerod:Yeah. Dot a a
Gerhard:a. Mhmm.
Jerod:Assert call that. That should pass. Alright. So that passes. And then we wanna assert that this gets called.
Jerod:So it's getting called, but I I bet if I tell it I don't care what it is, what you do like this, I think, which means the second argument doesn't matter, then that should pass.
Gerhard:Yes. Look at the URL that we're passing Yes. On line 14.
Jerod:Yeah. So actually, good point. So actually, what should be happening is it should be doing this, which isn't right, but is what we coded.
Gerhard:Yeah. Good good point.
Jerod:Yep. So what we're doing is our module is getting full URLs, but what we want is paths. Right? The way Pipedream will work is we actually just want this to be the path to the file.
Gerhard:Actually, now that I think of it, no. Because Pipedream, when it when we are serving static assets, there is a FQD and there's a host, and the host is c d n two dot change dot dot com.
Jerod:So this actually needs to be chain like, so Pipely needs a change.
Gerhard:That one is correct because we are purging the root URL, like like like the home But
Jerod:what if we aren't? What if we're purging an m p three file? Do you take a full qualified domain name and, like, pat like, an absolute URL there
Gerhard:or what? I see what you mean. Yeah. Yeah. Yeah.
Gerhard:Well, in that case, we need to set the host header separately because the host header is not going to be captured in the FQDN. So in the FQDN right there, we're passing the I p v six, and then the host is separate from that.
Jerod:And Pipely needs the host header in order to know what to search? Exactly. Which Which origin.
Gerhard:Back end, exactly. Which origin, yes, the purge is for. Okay. Because we are hitting we're hitting, in that case, varnish directly
Jerod:Right.
Gerhard:And it needs information. Okay. So what do I do with this request?
Jerod:And we can't use the host name here, because then it'll hit whatever instance The public one. Yeah. Need the actual all the instances. So we need the address.
Gerhard:Exactly. Yeah.
Jerod:Okay. Yes. But we don't want the full URL. We want this to be the So I think
Gerhard:the full URL is correct, but then we need to deconstruct it before we make the request. So from the full URL, we take the path, we put the host from the URL in the in the in the host header, but then the request goes to the I p v six.
Jerod:I'm thinking through this for all of these. Yeah. Because these will be like CDN or they might just be app name.
Gerhard:So I think that's fine. Right? Because it's everything will happen in the purge function. The URL, that's the one that that's the part the variable that we need to deconstruct.
Jerod:Yeah. We gotta take the URL
Gerhard:three parts. Yeah. Exactly. We don't change anything above or, like, before the URL. This is just, like, a specific purge implementation that is, like, again, PipeDream specific.
Jerod:Hostname. Path.
Gerhard:And then that's it. Because the I p v six, the address we get from DNS. Right. So that's it. We just need to get the host name and the path.
Gerhard:Okay. But we don't care about the protocol.
Jerod:Okay. So now we have a host and a path. Then we call it the DNS, and then we make the request with the address and the path, and then we have to pass the host name as a header. And so headers are the fourth option. So we'll pass it an empty body, because this is a purge request.
Jerod:Yep. And then a list of headers. And what's the header called? Just host. Yep.
Jerod:And the value is host. And if we were to test drive that, we would say, okay, we expect this to get called with an empty body. This is correct because we don't want the full path. Then we expect a host to be this. Correct?
Gerhard:Without the protocol. Without the protocol. CDN, yeah. Mhmm.
Jerod:See how we're Yeah, looks right. Request with four is undefined. Well, what what did I do wrong there?
Gerhard:So all that was That's the
Jerod:URL body. How is it undefined? What changed all the h t oh. Oh, I didn't mock that. That's why I'm mocking it with two with two arguments.
Jerod:I need to mock it with four arguments so that it actually defines that for me. So we changed our actual API a little. Okay. Still failing, but we got past that thing. We wanted this to be called.
Jerod:Unfortunately, with these, you're just asserting the mock is called, and so you don't actually have the Yeah. That's annoying. Yeah. Like, we wanna know what it was called with, not that it just wasn't called correctly. And so here's where I would probably get myself
Gerhard:in the
Jerod:middle and just hop in there and call it this. And just call changelog.pipedream.purge. Oh, look at that. We didn't get that far because my pattern matching's wrong. Oh, because it's returning more than one address.
Gerhard:Yeah. But look, those are like the real those are the real IPs.
Jerod:Yeah. Because I'm actually just calling this, the the mock is gone, and so the DNS resolves. So I see. Okay. That's why.
Gerhard:So then you need multiple, I think. Right?
Jerod:Yeah. Yeah. To do this. We might have just been better off doing our loop. Yep.
Jerod:For now, I can just run this real quick again and just see what happens. So yes. So addresses, this should be a list, and path, that's And host.
Gerhard:And host?
Jerod:That's correct. Perfect. So so we're sitting pretty there. Okay. So that was the problem.
Gerhard:So we need the loop.
Jerod:Get rid of this, and then we're gonna say, basically, for it's for address addresses Mhmm. Do? Yeah.
Gerhard:Yeah. Looking good.
Jerod:This should work, because we're returning one in the list there. Still failing. Pooh. The path
Gerhard:the path is correct. Yeah. That's actually a good good way of checking whether this is even being called.
Jerod:See, if it passes, why you tell it that this doesn't match This is what's failing is we're not constructing that correctly.
Gerhard:Okay. So can we see how we're constructing it? We can. So that's this function
Jerod:right here. So the address Could
Gerhard:it be that the path has two forward slashes instead
Jerod:of one? Oh, it totally could.
Gerhard:Usually, that's what that's what it is. Yeah. Can you add another forward slash?
Jerod:I think you drill
Gerhard:it. Or that's
Jerod:that one. That was it. It was like this, slash that uploads.
Gerhard:That's what experience looks like.
Jerod:Yeah. Totally.
Gerhard:The fact that
Jerod:you knew that, and as soon as you said it, was like, you're absolutely right.
Gerhard:So it's this.
Jerod:Is the problem.
Gerhard:Nice. Nice.
Jerod:Nice. So what's prettier? I think I think just that.
Gerhard:I'm wondering whether we could use some sort of utility to assemble them, because if a path doesn't have a forward slash, it's going to fail. Mhmm.
Jerod:Yeah. And I think we could do, like, URI dot construct or something.
Gerhard:The opposite, basically, of Yeah. URI parse. Yeah.
Jerod:So, like, you just build the map, and you call it, and then it's, toString or something.
Gerhard:If you think about it, the only thing that we need to do is right? We have already have the URI. So if you go back in the code, really, what we only need to change do you see there when the URI parse? Mhmm. We would only need to change the protocol and the host.
Jerod:Oh, yeah. Set this to 9,000, you mean?
Gerhard:Exactly. So, like, in the URI, right, like, we deconstruct it, we want to inject its HTTP. It'll always be HTTP. There'll be no TLS, and we need to use the I p v six and the port, but everything else remains the same.
Jerod:Yeah. Did you set the host then to the to the address, basically?
Gerhard:You set the host to the address, yes, then and you also changed the protocol. I don't know what the protocol is, by the way, because I think we we so we need to explicitly make it HTTP because HTTPS is going to fail. So, you know, when you had, like, URI pars, you were debugging this in the terminal
Jerod:Yeah.
Gerhard:In your console. If you were to do URI parse, what are the components that you get back? That's the one. So the scheme needs to be set to HTTP, the authority doesn't really matter. The authority, I think, that is used for the scheme for HTTPS and the host.
Gerhard:You need to capture the host and then change the host. But otherwise, everything is right here.
Jerod:You change the host and then support in the scheme.
Gerhard:Correct. Yes. Yes. But you still want to keep hold of the original host because we need to pass that as a header.
Jerod:Alright. So I think what we can do is say, grab the host.
Gerhard:Can we just do maybe URI? Because we will be reusing it.
Jerod:I mean, yeah, you could do this and you could say host equals URI dot host. And now we can manipulate URI. You're not actually manipulating it because we know that it's functional. But we can now say, new URI equals map dot put.
Gerhard:Think it's inside the loop. Right? Because for every address.
Jerod:True. Because we wanna have the address there. So new URI equals map dot put key value. I think you can call put in and get this is where my these are my elixirs, rusty. Mhmm.
Jerod:And put in and you can put multiple things in. Takes a data structure up half and the new value. How do I put multiple Yeah. New values at the same time? Put in only updates one.
Jerod:Yeah. Chaining. I don't want a chain. Map dot merge, that's what I'm looking for.
Gerhard:Mhmm.
Jerod:Mhmm. Alright. So make map dot merge, and we're gonna take the current host oh, sorry, the current URI. Get rid of that. I misspelled merge.
Jerod:URI. And we're gonna merge on to our URI. The scheme Yep. Which is gonna be HTTP.
Gerhard:Yes. The
Jerod:port, which is gonna be nine Yes. And the host, which is gonna be address. Yes. Correct.
Gerhard:That's it. And then
Jerod:we get rid of this, and we say new URI.
Gerhard:Mhmm. Perfect. Look at that.
Jerod:Assumes that h t poison will accept a URI struct, and it it would I mean, why wouldn't it? But we'll find out. So should this then just pass the test like it already did before? Are we failing this again? Confirm that's our problem.
Jerod:Yes.
Gerhard:Okay.
Jerod:And so now let's do logger dot warn. New URI. And I'm not sure when you run the tests inside here. Do we get the loggers? We should probably get them.
Jerod:Host is this. Ports 9,000 scheme. We might have to call that URI to string thing in case maybe h t poison won't take it. So URI to string. That's fine with me.
Jerod:Call that right here. URI Yep. To string. It should pass.
Gerhard:Yeah. Let's see. Yay. We did a thing. No way.
Gerhard:Nice.
Jerod:So that's with one. Now let's Yes. Imagine that we're returning more than one.
Gerhard:Yes.
Jerod:Another. This does not matter what this is, actually.
Gerhard:Sure.
Jerod:But then we just assert call again. I mean, this is push lead, but why not? This will work. And this should just say another one. This might not be a valid URL either.
Gerhard:Yeah. I don't think it is. Yeah. I mean, just put underscores or whatever or dots, another dot one or second IP. Let's just give it a name which is descriptive maybe.
Gerhard:First machine, second machine, something like that. First dot machine, literally, first dot machine, second dot machine. And that's And
Jerod:this is first dot machine, and this is second. It's a
Gerhard:fly o machine in second. Yep. Perfect.
Jerod:Yeah. And this will work. Cool.
Gerhard:Look at that. Can you change the assert to do, like, I don't know, third machine or something like that? Or can you change one of them so that we see it fail?
Jerod:Yes.
Gerhard:It doesn't really matter which one.
Jerod:Mhmm. Yeah. Nice. That fails it.
Gerhard:Okay.
Jerod:Okay. So that URL so that works. And in the case of an error well, I'm not doing anything with errors in this case. Oh, I am. I'm actually sending over to
Gerhard:Let's the do something there.
Jerod:Yeah, exactly. Let's just do pipe dream.
Gerhard:Yeah. Purge failing. So I think we'd like to know what came back from the DNS. Right? Because it's it's almost like a DNS related failure.
Gerhard:If I'm looking at the case Yeah. The
Jerod:case is
Gerhard:on the DNS. So I would maybe add a little bit of detail around, just like to give us, like, an indication of where the problem might be.
Jerod:Right. We would say like
Gerhard:PipeDream perch failing.
Jerod:PipeDream perch. This is really the DNS lookup that fails.
Gerhard:Yeah. I think that would be a better
Jerod:error message. We're not actually getting we're not doing anything. Like fighting and forgetting on these, which is probably appropriate, like, you have some sort of a non response, like, I don't know, 400 or a five zero three or something from one of the instances, like, just move on. Right? We're not actually gonna
Gerhard:I think we should capture that because otherwise we won't know. But for now, if we capture the DNS resolution error, which is what was happening on one four six. Yeah.
Jerod:Right. I could just say due to DNS, and then we can just say extra can pass this reason right there. And that will pass it to century. So it'll show up in the metadata. DNS resolution?
Gerhard:Yeah. It's always DNS. That's actually a
Jerod:better way to say it. Pipe? Yeah. It's always DNS.
Gerhard:Yeah. Look at
Jerod:that. Yeah.
Gerhard:Very nice. Okay. I like that. Should we test this maybe?
Jerod:So when given a URL string it's really testing I guess it's just testing the case where when DNS fails, basically. And so we had to mock it differently, so we're getting to a point where, like, we get, you know, when DNS fails, you're kind of it's fine. And we're gonna do another with mocks. And in this case, it's gonna return error, and that's not actually gonna get called.
Gerhard:We don't need HTTP. Do we still need to do HTTP? Do we need to assert that it's not called?
Jerod:We can test that it's not called. Yeah. Okay. We still can mock it and just test that it's not called. And we're gonna say assert called.
Jerod:Actually, what we have to do then, we're gonna test now. Yeah. See, now you're getting to where this is why I don't I stop short a lot of times, because what you're now you're gonna test that the sentry thing was called. Assert not called, assert called.
Gerhard:That's okay.
Jerod:Now we're mock that. Yeah, exactly. Uh-huh. So now we're add to our mocks. This mock for sentry, pass it to this empty list that we don't know why the empty list is there, but we do know that it is there.
Jerod:Capture message, which is a function which which takes two arguments. So underscore underscore Yep. There we go.
Gerhard:Okay. Yep.
Jerod:Oh, I didn't finish this one. So here I can say refute called, I think. Mhmm. You might even just be able to do that. I got another bug up here somewhere.
Gerhard:Oh, you have another comma. You have a comma two commas in the function. True.
Jerod:Yep. Yeah. That's it. So this should get called this should not get called Mhmm. And this should get called, which is sentry dot capture message.
Jerod:As long as we're calling it, we're good. Alright. And then I would probably throw this one at the bottom because I tend to think happy path first and then, like, slow path or not happy later. Now you think we should test it, like, against the reality?
Gerhard:HTTP. Yeah. Well, I think we still should do the HTTP part. I mean, we can leave it later to see
Jerod:Which part are you talking about?
Gerhard:When HTTP requests fail, we should track if purchase fail.
Jerod:Okay. What are the possible responses coming from back? Oh, it's gonna be so then we can do another case statement on this. And then if it's okay, we don't care. We do nothing.
Gerhard:I mean, if you wanted to log, that would be okay, but we don't really care. You're right. We care about the set path. Right. Error.
Gerhard:We do want the response.
Jerod:Response. What do we wanna do now? We want a sentry capture message?
Gerhard:Sentry capture, yes.
Jerod:Alright. Pipe Dream purge failing instance, maybe,
Gerhard:and pass it the address. Yes. That'll be very helpful.
Jerod:And then extra could be the response. Yes. So that's gonna be the entire HTTP response from Mhmm. HTTP poison. And then we didn't test drive that.
Jerod:We actually did it backwards. When given a URL string, maybe do another one, test when an instance fails or when an instance Fails to purge? Yeah. Fails to purge, and then we're gonna get some copy paste here. So let's do this whole copy paste and change it.
Gerhard:And I think it also changed the description for the happy path, but we can come back to that later. Yeah.
Jerod:So in this case, we wanna make this HTTP request be an error.
Gerhard:An error. Yep. Error 503, whatever. Yeah. Something like that.
Gerhard:Mhmm.
Jerod:And that's all fine. And then we call it. That's good. And then we just wanna make sure that we call this
Gerhard:Fails and maybe the second one will also actually, both of them will need to fail in this case.
Jerod:Yeah. Well, I think we've
Gerhard:We could also check that only one, for example, fails because that's more likely to happen. Mhmm.
Jerod:Let's just do one machine so I can get rid of this second assertion. Sure. And then we'll assert this entry with call. Yes. Which we already have written down here.
Jerod:Mhmm. And that should pass. Not mocked. Oh, I didn't mock
Gerhard:it Yeah. In the you haven't mocked it yet.
Jerod:I mocked it in this one. Okay. Nice.
Gerhard:We're Nice.
Jerod:We're all green. We have a 100% test coverage, I think Mhmm. Ish for this one functional.
Gerhard:I think it's good. This is good. Yeah. This is good. This is an important part, and we know that it has failed in the past, and it's annoying when it does, because debugging this is not fun.
Jerod:Yes. Have I ever used this? I do. Sometimes I just wanna make sure and clean up a little bit. I think we're good.
Jerod:Mhmm. Okay. Yeah. Should we actually just fire this up and run it against
Gerhard:We should.
Jerod:Yeah. Let's do that. So what's the best way of doing that? Obviously, in the terminal makes sense Yeah. For the actual call.
Jerod:Then maybe in a different terminal, we can do our curls.
Gerhard:So what I would do, I would try and push this into production to see how it behaves, which is that commit and push. And then while that happens, while the deploy happens, we can, like, try the local.
Jerod:One issue I have is you have a different branch than I have. I had a local branch, and you have a Okay. This branch. So how can I hook my local branch up to yours? Because you
Gerhard:already So
Jerod:pushed this. I already have a commit as well on the DNS stuff, and I'll add a commit here for Okay. This stuff.
Gerhard:Right. So if you add a commit to this stuff, then you can rebase your local branch on top of the remote branch, and then push it again.
Jerod:What do we call this one? Add pipe dream. Purge? Purge.
Gerhard:That's okay for now. We can come back and refine it later. Nice. Alright. So now if we're locally
Jerod:So yes. On branch. Yep. I'm on my PipeLeafHurge branch, which is only local.
Gerhard:So you want to fetch the remote branch. So let's say git fetch origin. Yep. Okay. And I can do git rebase on top of that.
Gerhard:And now that they think of it, actually, no. What you want to do is check out that branch. So git checkout that branch and rebase that branch on top of your branch. Good. That's okay.
Jerod:Did you change something there? Mustaf?
Gerhard:Yes. I did. It's the application instance I've changed. Yes.
Jerod:Alright. And now rebase my Pipely Purge branch on right now?
Gerhard:Now rebase yeah. On top of Pipely Purge. Yes. That's correct. Yes.
Jerod:Cool. So here's yours. Here's mine. Perfect. Let's run let's run the next test real quick.
Jerod:Yep. I did have an unused alias. No big deal. By the way, these will run-in CI as well before True.
Gerhard:We deploy. But you're right. Local test is always quicker. Even though wondering I'm by how much? How much how long does it take, by the way, your test to run locally?
Gerhard:Nineteen point seven seconds. Lovely. So let's push it and see how long it will take in CI. Oh. Okay.
Gerhard:That's okay. Oh, can force. Force push it.
Jerod:Force push it. I have at least favorite thing to do. Why? It just feels so pushy, you know. It's like
Gerhard:pushy. No. It But I'm say I'm saying, Jared, push it. I know you are. I'm asking for it.
Gerhard:You are. So this is good. This is good. Alright. So let's just confirm that the CI is running.
Gerhard:I'm sure it is, but let's just double check. There you go. Perfect. Force pushed. Some checks haven't completed yet.
Gerhard:That's exactly what you want to see. So this is perfect. Now we can come back, and we can finish the local testing and wait for it to do its thing.
Jerod:Yeah. So I think
Gerhard:So the first thing which I think we should do is check a URL that will be cached. It can be it can be the same m p three one if you want, the one that you had in the tests, or anyone. You can pick any URL.
Jerod:Let's do slash news, and this will be cache, because I've just hit it a couple times. Right? Or I'm logged in, so maybe not.
Gerhard:Maybe not. But I'll just do this.
Jerod:There's one.
Gerhard:Yep.
Jerod:Hits four. Yes. That's my fifth hit. Then we wanna say change log .pipedream.purge. Bam.
Jerod:Okay. Okay. Okay. Okay. Okay.
Jerod:Okay. Okay. Now, this should be new. Right?
Gerhard:Should be miss.
Jerod:Miss. This should
Gerhard:be a miss. Miss. Look at that. Look at that.
Jerod:Nice. We're good. Nice. Nice. Nice.
Jerod:Good.
Gerhard:Look at that first time. So how about we try one where the host differs? So actually, now to think of it, PipeDream, the host, yeah, CDN two, but let's try static assets where the origin is a different one, yeah?
Jerod:Where the origin is different than PipeDream.
Gerhard:The origin is different than the application. In this case, the origin is the origin will be the static assets.
Jerod:Wow. So you want CDN two?
Gerhard:Exactly. Yes. Yep. Nice.
Jerod:So miss. Uh-huh. Hit. Hit. Hit.
Gerhard:Look at that. Beautiful. K. Purge.
Jerod:And now purge. I'm gonna Mhmm. Just copy this. I don't have that. And purge.
Jerod:Okay. Okay. Okay. Okay. Okay.
Jerod:Uh-oh. Hit.
Gerhard:Not good. Not good. Not good. So what happened? Oh, this
Jerod:has changed all the place. That's fine. Right?
Gerhard:Yeah. This is fine. This that's exactly where what the origin is.
Jerod:Right. That's the Cloudflare back end. Right?
Gerhard:Yeah. So this is correct. Cdntoangel.com. We are using the host. Can you do another request?
Gerhard:Sure. Just to double check.
Jerod:Hit 5.
Gerhard:Yeah. Okay. No. This was this was not purged. Yeah.
Gerhard:This was definitely not purged. Five more seconds. Can we go back to the purge again? It feels like something is missing. C d n two.
Gerhard:I think I know what's happening here. Okay. So this is a okay. Can you open up the Pipely repository? And if you go to Varnish, the Varnish directory, and if you open up pipedream.chain.com.
Gerhard:Mhmm. That's the one. And if you scroll down, what we're looking for okay. So that's still fine. Varnish health check, that's fine.
Gerhard:We have the host. That is correct. We have all those redirects. That is correct. If we keep going Practical AI redirects purging, return purge.
Gerhard:If request method purge. Okay. But is is that Okay. I'll need to look into this.
Jerod:Okay. I'll need to to look into this
Gerhard:to see exactly why the purge doesn't use the correct back end.
Jerod:Mhmm.
Gerhard:I mean, it really should just use the URL, so I don't if you let's just check one more thing. If you go back to pipe lead the top level, like the top level directory. Yes. And then if you go into test, and if you go into VTC, and if you go into purge there you go. Test purge method.
Gerhard:So I'm testing just a standard purge and, like, just like no host, it doesn't take the host into account.
Jerod:Right.
Gerhard:I need to add more tests here to see what's happening. So this would be my follow-up. Mhmm.
Jerod:So next steps on my end would be twofold. The first one would be to deploy this pipe here not purge everywhere Fastly dot purge exists in the application. And then the application will be doing that. And the second one would be refactoring this to well, when I do that, I could just run those in the background as a background job.
Gerhard:Yes.
Jerod:And or I can make this parallel so that it just calls them all at the exact same time and, you know, task dot async event effectively. There'll be two improvements. Well, one's an improvement and the other one's actually necessary. Otherwise, it's not actually purging. So those would be our Yes.
Jerod:Steps on the application side.
Gerhard:Indeed. And then setting this, the port and the host, the CDN, basically, the application name via config rather than hard coding. Yes. But that that's like a tiny one because this will change. We could leave it here, but whenever we do this, it's just basically spread through the entire code base.
Gerhard:I'm trying to centralize all the configs, and that's usually, like, in the config file. I think we would only set this in production because you would never want to trigger a purge if you're running a development instance of this, not even by accident, which is why I think that config should be only set in production. Now if that isn't set, I think it's okay for it to fail, right, if you were to ever trigger a purge. So we can set that to local host, that can default to local host in development.
Jerod:The other next step would be to add a token Yes. So that we're the
Gerhard:only one that
Jerod:can do purges.
Gerhard:Correct. That is correct.
Jerod:Yes. And is that just supported at the varnish level already or?
Gerhard:So we would need to set up a in most likely in the environment variable. Right? It's like a pre shared key
Jerod:Mhmm.
Gerhard:That is set as a secret in production, and then we set the same secret on the application. So again, it will be just like another config. But is that something
Jerod:that Pipely currently supports or that Varnish supports? Okay.
Gerhard:Not currently. So if we go back to the Pipely repository, and if you look at the readme, if you scroll down, there we go. This is where we are and this is what's coming next. Require auth for purge request. There we go.
Gerhard:That was the next one on my list. Mhmm. And part of that, I can also fix purges to actually work. Good. So that that is literally the next thing.
Jerod:K.
Gerhard:Now, look at what else is left before this is finished, which I'm very excited about. So sending logs to s three, and we can see the logs, how they appear in Honeycomb today, so we can look at that in a minute. There are some redirects in the Fastly VCL that need to be imported into Pipedream as well. And then we do one more review and clean up with all the contributors to make sure that we're in a good place. Mhmm.
Gerhard:And then the plan is to tag and ship an RC one, most likely in the next few weeks, and then start routing production traffic through a part of production traffic through these RCs. So we see how it behaves so that if anything doesn't work correctly, if there's any issues, anything like that, we can either fix them or, you know, roll back the production traffic and then fix them, whatever needs to happen there. So that by the time we meet, we have had these steps where 80% of the production traffic is being routed through Pipe Dream, and then maybe when we're live on stage, we do the final one.
Jerod:Mhmm.
Gerhard:So that would be like one more thing. Let's like ship this thing for real into production.
Jerod:Right.
Gerhard:Everything will be tagged, and then part of the tagging, update the DNS, maybe somehow figure out how to do that really easily so that we finish, like, that last 20% with everybody on stage. And the DNS is just like yeah. The weights. Right? We say eight eight eight requests out of 10 will go to the new one, and then 100%, like, all requests go to the new pipe dream.
Gerhard:So that will be the plan.
Jerod:We're living the dream, man.
Gerhard:I think we are. We're so close. Really, really close. So would you be interested to see what the Honeycomb logs look like currently? Sure.
Gerhard:And there's three new boards, Piply content, Piply requests, and Piply service.
Jerod:Okay. Which one do you wanna pick? Service.
Gerhard:What is the first impression that you're getting out of this? Does it make sense?
Jerod:Yes. Request to buy data center. This makes sense because we've hitting DFW.
Gerhard:Yes.
Jerod:Hits and misses on different origins, so that makes sense. Be cool to have an overall you have an overall hit and miss.
Gerhard:So we separate them by origin.
Jerod:Right.
Gerhard:But we wouldn't we don't need to if we don't want to. But the origin, as you know, it is important, like, is it the application? Is it the is it the static assets that's changed log. Place? Or is it we could also do it by host, for example, but I think that's because if you know, it's the same host for both feeds and application requests.
Jerod:Right.
Gerhard:But it is the origin that always differs between where the request is going. And I think that empty string, those 19 requests right there Uh-huh. I think those are our purges.
Jerod:Oh. Mhmm. Right.
Gerhard:Or I see the origin, I think that's where we have something, some some work work to do.
Jerod:Yeah. This makes sense. I think eventually it'd be cool to have one that was just an overall hit miss ratio for the entire
Gerhard:Perfect.
Jerod:But that would be in addition to this. I would also keep this. Yeah. Nice. P 90, this is response times.
Gerhard:Response time. Yes.
Jerod:Yep. Cool. Based on server data center.
Gerhard:This is like the which Pipedream instance is being hit? Mhmm. JNB Johannesburg. That's the slowest one currently
Jerod:Mhmm.
Gerhard:At three seconds.
Jerod:Quite a bit than everything else. Six x slower than the next slowest.
Gerhard:Yes. And the question is, is it the actual client? Because sometimes clients are slow. Right? So some if you remember, we debug this a while back where someone was downloading an m p three from a watch, and Yeah.
Gerhard:Those requests were slow.
Jerod:Not much about it.
Gerhard:Yeah. Exactly. So sometimes it can be the clients.
Jerod:Right.
Gerhard:And then size is the data being transferred in bytes.
Jerod:Gotcha. So these are the two one's duration, this one's based on traffic.
Gerhard:Exactly. Traffic,
Jerod:yes. Transfer size. Cool.
Gerhard:Correct. Correct.
Jerod:This all makes sense to me, and then we go to Yeah. Requests. Mhmm. And this is Get requests. A purge request.
Gerhard:That's where you can see the purge request there. Yep. So I think this is what you wanted, not split by Yeah. Yeah. Okay.
Gerhard:So we will make the change. Not a problem. Those that you see there, that's basically me running a benchmark. Mhmm. Because you can see get request 200 and, yep, 40Four's5Hundreds.
Gerhard:Yep.
Jerod:Cool.
Gerhard:Mhmm. We can see this is where you can see, like, the the the bots trying to dot ENV. Mhmm. Favicon, maybe we should have a favicon. That seems to be a genuine like, we should have it.
Gerhard:So Yeah. A redirect or something along those lines, that
Jerod:should We go in do have one of those, but I think it's on the CDN. And so I think in our HTML, we tell them to grab it from the CDN versus from the app.
Gerhard:So I think we just we just need like a VCL config Yeah. To rewrite that. Nice. And the last one, I think, was it the content that or is this the content board?
Jerod:This is the request board.
Gerhard:Yeah. Requests. Yeah. Let's have a look at the content. Think that's also not another interesting one.
Jerod:Uh-huh.
Gerhard:And the most interesting part is we have GYP data. Okay. So you can see most requests, GB.
Jerod:Cool. So we have everything we need. Was that hard?
Gerhard:No. Not really. Yeah. No. It was just a matter of wiring Jio to Licity, getting it in a way that, you know, like the token, I had to sign up for an account with MaxMind.
Gerhard:But no, it was not difficult.
Jerod:Okay. OHA, our number one user agent. OHA.
Gerhard:Yeah. That's the benchmarking tool. Yeah. That's why you can see all of them. Hyperping as well, that's what we use for synthetic monitoring.
Jerod:Right.
Gerhard:And Beta Stack as well. We have to both Hyperping and Beta Stack, they're just testing. The website is up. Yeah. Now, as you can imagine, this doesn't have a lot hackney.
Gerhard:Interesting. Look at that. 27 requests. That's me. That's you running purges.
Jerod:That's
Gerhard:me. That's you running purges. Look at that.
Jerod:Mhmm.
Gerhard:Yeah. Very nice.
Jerod:Very
Gerhard:cool. So these make sense. I'll make that one change, and then yeah. I'm excited for what comes next.
Jerod:Yeah, man. We are getting there.
Gerhard:How does it feel like this moment, realizing how close we are?
Jerod:Feels good, you know. There's we have plenty of time, I think. We have more than a month to get it done. We got six weeks, maybe. Before we have to be on stage in Denver.
Jerod:I think we're gonna get there. I think we're not gonna be super nervous about it, because by then we'll have, you know, enough traffic flowing through it that it won't be like a complete switchover moment where we have to suddenly cancel the show and like run to our computers and figure out what went wrong. That feels good. Change was as straightforward as I thought it would be, so that's always nice to just have a pretty easy purge so that we don't have to even wait sixty seconds. We can just immediately ship those episodes out, you know.
Gerhard:Yeah. That would be so good.
Jerod:And that's great.
Gerhard:That's what I want. Yeah. Very nice. And remember how this started with us not getting enough hits on the important content?
Jerod:Yes. I mean,
Gerhard:that that that's how this journey started. So what I'm going to do now is run a benchmark that is going to show how well these actually work. So I'll do I'll do benchmark c d n two, and I'm going to run it twice. What we would expect to see, this will send 2,000 requests, and we would expect 1,999 to be hits and only one to be a miss.
Jerod:Mhmm.
Gerhard:So if you do a refresh now, all the logs should be there. Look at the spike. Hits 2,000. Okay. Why is his tooth?
Gerhard:So we're getting more hits than we should.
Jerod:It must have
Gerhard:already been in there. Exactly. It was a stale one.
Jerod:That's exactly right. Zero misses.
Gerhard:Zero misses. But there you go. So we just served a stale one. So this is what we wanted for
Jerod:Nice.
Gerhard:Like, all this time. Right? Like 99% this case, a 100% hits.
Jerod:A 100% hits. That's all I wanted. There you go. That's all I wanted. There you go.
Jerod:There you go. Cache everything all the time, and don't don't don't do misses. No misses. Stop missing.
Gerhard:In this case, a 100% hits.
Jerod:A 100% hits. That's all I wanted. There you go. That's all I wanted. There you go.
Gerhard:There you go.
Jerod:Cache everything all the time and don't don't don't do misses. No misses. Stop missing.
Gerhard:There you go. Yeah. There you go. So we're finally there. This works as it should.
Gerhard:It just makes me so happy. Love it. Alright. I'm thinking you could bond to
Jerod:the exact same way. And then make it
Gerhard:That's it. Right. That's it. Yeah. Exactly.
Gerhard:Oh, yeah. Okay. That's it.
Creators and Guests
