Speaker 0: Hello and welcome to another addition of Black Hills information security talking about news. We have a cast of characters here to talk about the crowds thing, but before we get started, I just wanna point out, you're gonna be fine. It's gonna be okay. Take deep breaths. Unless you're in an airport, then you're screwed.

Or if you're running bit defender. If you're running bit defender, I saw somebody recommend on Twitter it's time to just like, walk out of the office and give up at that point. But other than those things, it's gonna be okay. We think most likely, So today we wanted to put this together because ambulance chasing is fun. We wanted to put together a bunch of people that kind of understand what's going on.

So we could talk about it, but also comm commensurate if you're on discord, I wanna see those memes folks? Like, what are some of the best memes you're seeing from this crowds thing? This is cath. It helps us get through through these things. If I come across a flip and I'm joking and all of that, I'm not doing that to make light of anybody that's dealing with this situation and really struggling today.

I'm doing it because we're trying to use this as laughter as the best medicine and we're laughing hopefully with you. We're not laughing at anybody I also want make it very clear that also goes for the engineers at Crowds strike. It's fun to make memes it's fun to make jokes every once in a while, but I want you to understand on the onset before we get started on this. Software dev for very large software projects is incredibly difficult. Kernel Dev is even more difficult.

So please have a heart for the engineers. As far as the salespeople people that are trying to capitalize on this, don't don't give them no quarter at all. But honestly, we really wanna make sure that everyone understands this comes from a place of love. We're joking about it. We're trying to get through it.

And it's kind of a little bit of gallo humor. I have Jo T to help talk about Colonel stuff and what's going on. Patterson Cake is here, runs our Ir practice at Phi. How many reboots is your son up to now. We're it

Speaker 1: we're 18 and counting.

Speaker 0: We're at 18 counting. 1 of those solutions was to keep rebooting the system 15 times and then magically the network stack would come up. Get an update from Crowds strike. His son is doing that for us, he's on 18. Now Patterson anytime any...

Like we get into a wall in this. It kinda gets into that just shout out how many reboots were out. We just need to know because I was worried that that was just a lot of, like, hot air, but we're 19... 19 counting. I got John Hammond.

If there's anybody that knows malware knows virus knows with going on, except for today. All like, we're all like, what the hell? So I called up, John, and I'm like, dude, What the hell going on? He's like. I don't know.

I haven't slept much, man. It's just really, really super busy, but we're really happy that John is here to help us talk about it. Matt is also on who... By the way, Matt works for Black Hills information security

Speaker 2: by the wearing this today.

Speaker 0: He is he is a full time employee of Black Hills information security much of the soc and surprised to other people at Black Hills information security. But he also is, like, bypass and exploit kernel level stuff at B I. Derek been around for a long time at B his. I think he's like, employee number 6 or 7.

Speaker 2: Something.

Speaker 0: And he just keep showing up to these things. But he also is the lead on our sock services, so we wanna bring him in on this, and then we've got Jason. Who you all know, Jason is just like, here at kind of when we get 2 technical, dropping in questions, bring in hot takes, buddy little jokes and little jab, who And behind the scenes we have Ryan, and can everyone send Ryan, your your love because he has courage. He should not even be here. His boss is an asshole for making him come to work today.

And, he's he put this all together for us. So round applause. For, Ryan for making this happen, we really really appreciate you, Ryan. He makes us look good and sound good, even whenever he doesn't feel like doing it. So, John, you gotta jump off because you got a very, very, very busy day.

I first learned about this on Twitter, I jumped on and I saw a tweet from you early this morning about you're, like, hey, there appears to be some weird things happening. And I noticed it was, like, 05:00 in the morning, and that tweet was 3 or 4 hours old. I was like, what the hell? So you've been working on this quite a bit. You're probably gonna have 1 hell of a day.

That wanna give us kind of a rundown of when you first started hearing about it some... Because you're 1 of the first people in the industry that I know of that was kind of, like, throwing the alarm belt. You kinda give us a little bit of... Timeline from your perspective about what's happened here?

Speaker 3: Yeah. Thank you so much. Well, hey, I'll admit. I'm over in the Pacific time. Kicking it over in California.

So it was like, 10 or 11:30 ish, and I was thinking like, well, goodness. We gotta get a video out tomorrow for some other thing that I wanted to do, But then I saw this fire start to burn in on the subreddit crowds strike? Crowds strike subreddit. And folks trying to, hey, starting to flag it, and I really have the hunch. This is gonna absolutely imp.

Speaker 0: Now were you seeing some of the the the earliest stuff that I saw appeared to start in Australia and New Zealand. Is that kinda of where you started seeing it pop up there?

Speaker 3: I know. Due to the time zone, those folks are awake and still tracking things are the day? I'll admit. I think the earliest I saw was 09:57PM Pacific, and that was just the starting gun on the crowds subreddit. But I believe everyone chi in, but all the comments and responses to said, hey, yeah.

We're already down and have been fighting fires here in Australia and New Zealand whatever. Etcetera.

Speaker 2: Mh.

Speaker 3: And then Us, and I think the rest of the world started to wake up to it slowly but surely. What else can I color the picture on here for?

Speaker 0: So so whenever you first started seeing this pop up, what was the first ink? I mean, we know that as there was something I mean, it seemed to me like we knew very early on. It was Microsoft and crowds together. That's what the issue was. Right?

But it took a little while to figure out exactly what was the sis driver? Like, how long did that kind take from Has. That everything is on fire. Here's possibly what it's causing to causing it to be on fire? Was that, like, an hour 2 hours, how long do you think that time window was between Oops sees?

Too. Here's what we gotta do to deal with it.

Speaker 3: It's actually pretty neat. I think you can see, and I'm trying to see if I can track it down in the reddit post just as well there was I think about a 20 minute delta from when... Hey, some sales engineer representative from Crowds said, hey, we're aware we're seeing widespread reports pretty good bath, etcetera, etcetera. We'll try to get a a tech alert or whatever they call their advisor messaging and knowledge based stuff. So that was a 20 minute delta, pretty And then about 15 minutes following the technical alert came out.

And then around that time, I would say 40 to 50 minutes, maybe an hour is when they identified. Hey, we are seeing this cs s agent dot sis colonel driver being problematic. It looks like I... And I'm not super a thousand percent confident on whether it's loading, some of these c hyphen 00291 Dot whatever dryers pile.

Speaker 2: Yeah Yeah.

Speaker 3: But those are the hiccups. Those are the issues. So you've got folks trying to share some quick scrappy workarounds and mitigation of, hey, just rename the folder path, rename it so that way it won't load the driver. Bear in mind that will completely triple crowds strikes, so your mileage may vary with that. I was kinda going back and forth screaming and shouting on Twitter when I saw Brody from, I think the director of Overwatch.

On at Crowds strikes screw. Saying, like, look, here's a little bit more pointed a little bit more targeted. Make sure you can get into safe mode and use the commands to remove the C29001 sis suffix, colonel files and drivers. And that was much sharper and much more accurate than just rename the folder path

Speaker 0: and Yeah because it doesn't

Speaker 3: much strike

Speaker 0: Judge completely break crowds strike at that. Right. Right. Right. Though.

Speaker 3: I don't wanna, hey, ramble for too long. So if I'm up he's strike look.

Speaker 0: No No. Please. Please please. No.

Speaker 3: But, obviously, I think a lot of folks will raise their hand and say, hang on wait a second. You really want me to do that manually across... However, many 50000 points. Yes. That is the crux of the issue, and you can expand that and extend that to whatever...

Scale and size you might like. And folks are saying, like, well, wait a second. That only works if you aren't using bit bitlocker encryption,

Speaker 0: it's going harder

Speaker 3: to do that at scale because right, you'd need the bitlocker recovery key to be able to get into, that safe mode to.

Speaker 0: On on reddit, that I'm hearing, like systems administrators that are straight up, Like, yeah. I need to get the recovery keys. It's on my notebook computer. Which is blue screened, and I don't have recovery keys for that system. Like, it's bad.

Like, if... I mean I was joking about the whole bit bitlocker thing, but This is 1 of those situations where our security products are getting in the way of our security products at point.

Speaker 2: Not just bitlocker. It's probably any encryption.

Speaker 0: It is any. Right? Got it. Just watching discord explode with the memes. Keep them going, keep them annoying.

I don't know memes. I just saw the Steve Ko meme pop up. And I never thought I'd see a Steve Ko meme in an It security webcast, but here we are in Ko Khaki.

Speaker 3: May I... I'm sorry. I don't mean to stop no. I'd I I'd love to have all of your hot take in your ex. Expert opinion pigeon because I know there's been some chatter where folks are saying, hey, you can use group policy, push out some automated tasking and then forced the safe mode for configuration change, blah blah blah, and that is...

I think what some folks are saying, oh, might be some to this headache.

Speaker 0: But wait wait a minute know. If we look at this, you can't even boot. Like more back isn't coming up.

Speaker 1: Right.

Speaker 0: So group policy, I don't know who... Unless there's some kind of new thing that's out there And Like you said, if you're on discord with us. If you have a link about that, that would be great. But, apparently, 1 of the things they said is reboot the system 15 times, the network stack will come back up, Crowds strike can reach out and pull down an update that nukes this file and solves the issue. We're testing that now.

Patterson, what do we at for reboots?

Speaker 1: I, I fear at, 18 or 19, my my volunteer staff, quit. Yeah. There... My volunteer staff is directly struggling with this issue in the enterprise, and so and all seriousness, They they were testing this as a potential solution and and it's just a a non starter. I I've seen reports that it's worked for some folk.

I have 2 2 direct contacts and 2 significant enterprises in both have failed to see any resolution literally up to 18, 19 reboots on a handful systems. Yeah. Your mileage may vary, but that is the pretty clearly not a silver bullet at this point unfortunately.

Speaker 0: Yeah. That that that test of 2 systems that we're going on not looking good. George just pointed out, let's go to 50. Let's see what that does. Maybe that'll do it.

Speaker 1: You never know. You never know.

Speaker 0: You never know.

Speaker 4: Once you get to the hundred, it's that's working out here.

Speaker 0: Yeah. Just.

Speaker 5: A quick question for everyone here. Like, it it used to be Info tech Twitter was the place to go and So find answers and solutions. Like, is there a centralized place right now where people were...

Speaker 0: I'm I still went... I still went to Twitter. And read it. Those are the 2 places that I'm at. This might get me back on the Twitter because that's where I'm getting probably the best information and the best communication with some of the top people on this, and I don't know what you all think, but it's reddit and Twitter for me is where I'm I'm getting all of my information on that.

And, of course, for the His discord server.

Speaker 4: I would also add Linkedin into this. I'd not want to usually promote Linkedin for a threat intel, but, be very... It's actually really surprising how much sharing has been done. Just trying to follow along There's been a lot of sharing of code in reverse engineering, kinda figure it all out through Linkedin. And so Put Linkedin.

Yeah.

Speaker 6: Yeah. I'd I'd I echo that to you. A Twitter Twitter and Linkedin combination there, I I tend to lean into Twitter first, and then, lean on my friends like Matt. Yeah, so, yeah. Who is an employee of Phi by the way, just for the wreck.

Oh Yeah.

Speaker 2: I've heard that.

Speaker 6: Yeah. So it it's it's kind of interesting that... Matt and I work very deeply with lots of malware development. So the minute, we heard, of this issue. We both were like, we know exactly what this is.

So oh, my camera's mirrored apparently. Okay. Does it... Which is my best side. Is it this side with this

Speaker 4: but it's the side that has the hat on? You need the hat.

Speaker 6: Yeah. I need the hat. Right? I'm, I'm going to mirror my camera. So that was scary.

Speaker 0: My god. That's nausea. Yeah.

Speaker 4: I saw a lot of people.

Speaker 0: Can you give us a little bit of information like, okay. So we know that this... This file is a channel file, And I scrambled all over the Internet trying to figure out what the hell a channel file that's actually, how I opened up my conversation with John and invited him on. And but, you know, because Crowds people are saying, oh, it's a channel file. It's not like a full update.

So what I was able to find out from channel files from, Mike Fell, you stay ready, who's over at trusted, a good friend of B his. Is this is a file for minor updates, definition updates, very, very small tactical updates. And the thing that I was kind of working off of and trying to figure out is this this something that's generated through automated means like God help me for saying this artificial intelligence? And threaten until pushing updates down systems or is it manual. And he said based on his information, which may be out of date by a couple of years it was very much manual.

It was a manual like file for these updates that could pushed down, but like little quick tweaks. So it wasn't an automated thing, and then people started cracking this thing open, and it was full of null. It was full of zeros. Now Can you and Matt talk a little bit about where does crowds strike exist as it relates to the hurdle? And, like, what what the hell going on with AAA file full of null coming into it and like, how that impacts the kernel itself?

Speaker 6: Yeah. Yeah. Absolutely. So so when you're looking at, these defensive products, a modern day, Ed, D, marketing acronym of choice. There are always, we're typically in in in well developed products, layers of that product that operate in the kernel of the operating system and also operate in what we call the user land space.

Right? Kernel development, when you when you look at... So the channel file that we're actually talking about has an extension of dot sis. Are in in Windows land, that means it's a kernel driver. Okay?

Kernel... A driver development is a very specialized skill. It is, it is a skill that you have to get exactly right. Because when you're developing code in the kernel, there are no safeguards. You are responsible for managing all the memory allocations you are responsible for for managing all the resources and releasing them correctly.

You are responsible for managing the memory space, of the a driver that you are developing and maintaining and you... And you are God at driver level. You can write to anything in the kernel and the user space of the operating system that you want to write to. So it is a very high trust development environment. Okay?

So if we go back historically for a minute, there there was a time pre Windows vista where most antivirus products at that time used to put hooks into the kernel, Ss, there's, it's a des distributor table that various kernel calls we're using, and they with

Speaker 0: it was it the global interrupt table that they were hooking into? Wolf that's local.

Speaker 2: It for each job.

Speaker 0: Okay. Go ahead.

Speaker 6: Yeah. It was global. And, This Microsoft started getting upset with the community because if they made a mistake doing this, they would destabilize the kernel and result in in what we're seeing today, right, which is a blue screen of death. Otherwise known as as a colonel crash. Right?

And so Microsoft, with the release of Vista kicked the the virus community at that time out. Of the kernel, and they introduced the technology called Patch. Patch, basically, said, you can no longer directly patch the des descriptive tables in the kernel. You're not allowed to period. And if you do, we will bug check the system, and others, we'll will crash the system.

Right? So naturally, the virus community at that time, and the emerging, you know, Ed x d community eventually. They were not too happy with that. And so what happened as we move forward is These, vendors started doing what's called user land hooking in their products. So they would patch into in the user space not the kernel space, the D in windows that we all know and love as nt D l dot d, which means they were redirecting not Api calls in the Windows operating system so that they could get telemetry about what's going on in the system.

At the same time, They were petition Microsoft. You've got to let us back into the kernel. We really really don't like this situation of having to hook every process and do all this extra work in user land just because you won't let us tamper with the Colonel. Eventually, and I don't know what actually transpired, of course, I'm not a my Microsoft employee, but eventually, Microsoft Rene and said, okay, we are going to give you some limited means for you to develop drivers in the Kernel and listen to telemetry. And because that was their primary request.

And the telemetry came in the form of what's called kernel callback notifications. And there's multiple different kernel callback notifications that are involved with colonel drivers. There are.

Speaker 0: 1 I wanna jump in real quick on that history. There was a huge dust up between S semantic. And Mca at that time because they were the main antivirus vendors. And basically, what they were planning on doing was just exploiting the Windows kernel to get the level of access that they need it and before it

Speaker 2: actually right

Speaker 0: really ugly in the streets because that was the threat antitrust. We're just gonna write exploits for it. Microsoft caved, and this is the kind of inter in a immediate intermediary solution. So go ahead, keep going, jo.

Speaker 6: Yeah. Yeah. So that's exactly right. So These kernel callback notifications are in the form of a table in memory. And what happens is that when a driver installs itself and gets loaded, it will register a callback notification that that driver wants to listen to.

And basically, it's telling the Windows kernel when certain events happen, tell me about that event. And there are names for these things. There is a process create notified routine, a create thread notify routine, a load image routine, a object register callback routine as well as a registry, activity routine. So the community was relatively happy that they got the telemetry back that they need it, But bear in mind, the when you are developing drivers that listen to these callback notifications, The kernel is still vulnerable if that driver that it receives the notification does not... Do the right thing, essentially and process that notification in a timely manner and allow the kernel to continue operate.

Right? Now the worst of all possible scenarios which we are experiencing right now is if the kernel notification callback routine in the developed driver crashes or it is not able to be called. Right? And from our understanding to The colonel notification callback that was registered by the crowds strike driver, called out for it to make a notification, and encountered a driver that had bad or missing coat. And if that immediately causes the Windows kernel to crash.

By design because it's a destabilizing effect. And so it it's a mixture of things that are going on here. Right? It's necessary for Ed and X to receive telemetry via these kernel callback notification registration events, but there's still a danger and it's very critical that the driver is developed very carefully to process these events, in a timely manner and to not destabilize the k. Right?

So this is going to be where the discussion is and, you know, you can go further than this, to understand this is not the only component of Eds and X. While they are si telemetry, those drivers are also interacting with the service process that's running in user, typically, and, you know, providing that telemetry back to Crowds strike, maybe getting some signature data, that kind of thing. That that you know, a lot more of the activity of defending is happening in user. That kernel driver is typically acting mostly as a notification facility. Right?

So that... That's what we're seeing here. That's that's as near as I can understand what happened. Now... Our information, and, I think Matt can put up a slide on this is that the kernel driver that was pushed with the update, got installed, was in fact full of null characters, which means that the put notifications that immediately hit that driver would have hit essentially no coat.

And that absolutely would cause an immediate blue screen of of, deaf. And, there has been some reverse engineering that has already occurred to actually support this. Matt, I wonder if you could put some of that up

Speaker 0: can you share your screen and talk about your your things come?

Speaker 4: Yes. I actually just posted the pictures in our chats through shared? Yep. So we go to the other picture for a second. You'll kinda see that there's already under the exception code and access violation, which she's kinda of what Jo with talk about.

But... Yeah. To you point about the update being full of Null. What we kinda see right here was this was paused because as Josh mentioned that Arnold was expecting something. And by doing...

I wanna stress a little bit of research and a little bit of digging before this, it looks like it was expecting an actual unit code structure of an address, rather it got back this error code. And because of that, it crashed because obviously, the update was

Speaker 6: full of Null and didn't have any routines or functions. So that way, it didn't return valid result. Causing this screen? Yeah. Just just to put a finer point on that, when you see an exception that is access violation in Windows what that typically means is and I see these all the time because I spend life in a debug.

What that typically means is that what A register in the Cpu has received an address with all zeros in it, it has tried to fetch memory at that address. And that is by definition and access violation because there is no address of 00000, whatever. Right? And this level. Just to be clear, there is no structured exception handling.

Like, when you're working

Speaker 4: if there's no Not

Speaker 0: There is no recovery.

Speaker 6: Like... Or it's no recovery colonel, there is no such thing it's structured exception handling

Speaker 0: coming. Yeah. It's just not not a thing at all. So the gotcha which

Speaker 4: is why Microsoft is so kind of always been cautious about letting third parties, develop drivers, especially for this reason. When you see this a lot with, you know, in the gaming industry, especially the graphic card drivers and stuff, 1 small thing crashes a computer, and all of a sudden, if your code is introducing that variance. Send you a large player basic cannot play and your game is dead in the water. But what I actually wanted to kinda of show is that second picture now, which kind of let me down, there must be some kind of issue with it. If you see it under fail bucket Id.

This unknown function because of that structure, it was supposed to jump to an address. To continue on the code. But because it was full of zeros and it returned to Null, it actually changed and caused to jump to an unknown address so far, this is what I've been able to blow, and that's actually been part of the root cause of this crash.

Speaker 6: Right. And so the only way to recover from a situation where you have a bad driver in the kernel that already has pre existing callbacks in a running system. You've gotta remember that the driver itself was written over, but the pre existing callback already in the callback tables. And so as soon as that first notification callback went to the new driver immediate screen. Right?

Mh. The remedy is remove the bad driver. Okay? So that, crowds strikes user code then once it boots up Presumably, I don't notice for a fact, but other crowds abstract customers probably know this. The user level code would reach out for a new update.

And as I understand it right now, Crowds strike has the corrected patched version of the driver again you'll continue on your way. Right.

Speaker 0: So I wanna ask you guys a little bit about load order. Right? So it sounds like the majority of the Windows kernel fires up, but But we're not getting network stacks going enough to be able to go to crowds stretch to pull this off. Tried doing the 15 reboot trick. Right?

It seems to me like, this is, like, where the where these drivers load. Yeah. Is actually before the network stack

Speaker 6: It is that you you will not get a stable kernel with this bad driver in place. I mean, in my opinion, Matt, Go ahead.

Speaker 4: Well, so obviously, when we look at an Ed, we think about it, they wanna know what it's going across the wire as well as what's going

Speaker 3: quickly across...

Speaker 0: Well that boot. Process. Right?

Speaker 4: So they put... They're kind of putting themselves at a higher priority to be loaded first over the network stack. And that's kind of where this problem of, you know, process You even it on a reboot, it's not getting the update is because by the time, network stack gets called. The crowds D or... So the crowds drivers already.

Loaded and it crashes.

Speaker 6: Right? And if it's a driver full of null, it'll crash as soon as it loads, tries to load that drive driver So you're

Speaker 4: I think we've said it a lot, but I think the most interesting part of this whole takeaway is a driver update full of null. Driver. That's what I

Speaker 0: keep going back to. This doesn't seem like a developer screwed up and writing their code poorly. This almost feels like the file was corrupted.

Speaker 3: Or

Speaker 6: something something something in there.

Speaker 4: I've I've heard a lot of mixed things about people saying that if it's Ai generated. I don't believe that I think...

Speaker 0: What other what not from what I'm hearing, this is not a file that Ai generates automatically. That's... And I and like I said, the people, like, if we're talking Mike, or you stay ready on Twitter. Mike knows this stuff really well from a crowds perspective use to work there did all kinds of research for B his. He's not trusted Sec.

It it's not Ai generated. As near as I can tell.

Speaker 6: No. But something in that Qa process obviously went a aw. For this file ended up with Null in it. Right?

Speaker 4: These things happen, I mean, I recently just upgraded my whole home infrastructure. I got the, you know, dream machine pro. And when I plugged it in, had did you an update? The update corrupted the whole os ass. I had to, you know, ref flash the entire thing.

It's not unheard of, but I just think it was some kind of secured, you know, safety check before it went light.

Speaker 0: Yeah. And we have a lot of people saying well, someone f up the code. I really don't think the code was left up. I think it's the build process. So the Ci pipeline somewhere between point a to point z.

Like I said, this doesn't look like hands on keyboard, Oops c's, I used gets at. Get ass. It's like... So way, like, more basic than that. Like it...

It's definitely an Oops c's. Right? But it's not poor code development, yes no

Speaker 4: No 1 forgot comma sort to speak. And is...

Speaker 0: So I got I got hand it over to John. John has to jump off. John, do you have anything else to ask add because by the way I wanna say thank you so much for giving 30 minutes of your time today. I know that you're gonna be real busy. You've been up all night, and we've appreciate you coming on.

Is there anything that you gotta say before you have to jump off though.

Speaker 3: Oh, goodness. Well, hey, I don't think so. Thank you so much for letting me be here with you all. It's kinda interesting to think on and speculate as to, okay, What were the root cause or what are these impacts and things as it's unfolding, at the end of the day, I think I would offer, you know, just a gentle reminder, than the other side across the screen. It's still another person.

So big hug ops to crowds strike or fight fires, the best that they can, and everyone that is going to be doing this slowly in recovery and efforts. You know what it's still it's still all of us in the same fight. So we'll be there to.

Speaker 0: Also, I wanna throw a shout out to Hunters for not taking advantage and, like trump to do marketing and saying, look, Crowds sucks were, because there are vendors that are doing that. And by the way, trying don't don't party with those people. Right? We just don't wanna party with those 8 holes. So be sure to check it out.

Also, I'd like to point out Hunter has a neighborhood walk program, if you wanna give things a try and play with it, and John is on Youtube as all of you know because you're probably all here because of him. So Thank so much, Dude. Get out of here and get back to your day and and good luck and happy hunting.

Speaker 3: Thanks, all. See you soon.

Speaker 4: Does see where

Speaker 0: jeff black Jason's got some questions. Go ahead.

Speaker 5: My question is, how do I explain this to regular people? What happened?

Speaker 0: Honestly, I would just say it was a bad update that corrupted the operating system. Yeah. Antivirus works at a very low level in your in your cute, antivirus screwed up, like, something happened with it, and it broke it. That's that's how I would try to describe it to friends and family. To be honest.

I don't know anybody else has a better analogy.

Speaker 6: Know, I I think that's probably the easiest way to explain it, but what what's what's I'm interested in following now is, you How exactly Microsoft's gonna respond to the developer community and to all of the X Ed and defensive product community? In terms of, ideas for putting guard rails, more guard rails around the tunnel without pissing them off. Right? I think it's going to get to be a very interesting conversation because, you know, frankly in a in a perfect world, this shouldn't shouldn't be allowed to happen. Right?

And and you know, we we'd rather not see an entire operating system destabilize by a single Kernel driver, I also worry greatly in our community and have for a long time actually that the kernel driver supply chain, is an extremely vulnerable and highly trusted part of well everyday It operation

Speaker 0: that's very certainly because of the compromise that happened all the way back at Windows Vista. Right? Yeah. You either let third party vendors into the Colonel or you would let them out. And Seriously, I think that Microsoft as much as they were being, you know, very, very, very mono in trying to lock vendors out of the colonel.

There was some damn valid reasons. Yeah. To keep them out of it as well.

Speaker 6: Well, those of us who lived through that period of time. We remember back in... You know, Windows Xp days just how unstable the kernel would get when you put lay it on some of these products. And the reason was they were hooking colonel tables, and sometimes they messed it up. Know, and it's...

You know.

Speaker 5: I I wasn't in It or anything like that years ago, and I remember how alt my computer crashed all the time. It, like the blue screen all the time, and then at some point it's stopping being blue screen.

Speaker 0: Probably around SP2. And Yeah. So if you're looking at Windows xp, if we're looking at what protections were put in place, it used to be pre Sp 2 on Xp that any application could access any region of memory anywhere else in memory. So you had a lot of things like video game trainers that could actually hook right into another process memory and start changing it that could mess with the kernel. Sp 2, they started putting in protections to try to shut that down.

When Vista came out, Microsoft shutting it down completely and like what Jo talked about is that compromise. Right? So you probably did notice things getting more stable, and more secure. But now kind of, like, an open question, I'd love to get people a disc... Their opinion on this too.

Do you think that this is gonna push Microsoft to shut this down even further.

Speaker 4: I scored I have my hot take is that this is definitely gonna start a conversation. You know, Microsoft. I I mean, Microsoft... I'm not... Microsoft does not like me, so I'm gonna try to be as blood as I can be, but have their

Speaker 0: what we do here. Right? So...

Speaker 2: But they have their own product to

Speaker 4: in the space and Ed, and it doesn't really, like, what Josh describes because they built the Os, they have their own ability to kind of built their own in kind of methods inside there to get the same level telemetry for the kernel without having to hook it. So they are, you know, ahead of

Speaker 0: kind of.

Speaker 4: They are, and they're competing with a market space. Here. I think this gives them enough fire to kind of say, maybe we need to revamp this pulse. See, and so pushed people out. I mean, I wouldn't see that happen because I wouldn't work eyes because all of a sudden, their products gonna start looking a lot more interesting and a lot more accurate.

Speaker 0: Or do you think that they're gonna do something like if you're crowds striker or another vendor, you're gonna have to hook to Amc? And it's like this is what you look into?

Speaker 6: No. I... You know, I'd like to see what I would like to see and if I were at Microsoft right now, I would be advocating for this, I would like to see a kernel Api developed. That had a series of driver integrity checks built into it for every driver load, kind of like live dynamic qa as the system booted.

Speaker 0: Do that. But that is definitely gonna be a performance hit. Right?

Speaker 4: No Do you want that? But

Speaker 6: but Only on boot if they could pull it off that the integrity checks are only on

Speaker 0: there's a chance for the system to recover. Correct. Oh, okay.

Speaker 6: But that's that's what I... You know, I think is kind of a responsible middle ground here. To maintain the telemetry that the vendors need, but also introduce additional safeguards in the driver load process so that potentially these things would not happen.

Speaker 0: Right? That's interesting. Yes, Jason. Your muted your mute. But I'm glad you raised your hand.

Speaker 5: Why does stream work? Youtube work, discord work and all these other things work right now. Linux

Speaker 0: Linux. Yeah. Linux.

Speaker 6: Well, or any uni may stop operating.

Speaker 2: Yeah. Just be glad that this doesn't bring down all the Linux... Server. And Very different. Yeah.

Speaker 0: Let's let's answer that question a little bit different. Right? If we're looking at Crowds strike Foul, Falcon server The vast majority like overwhelming majority of Falcon instances are endpoints like people sitting down working at their computers. When you're looking at a lot of server infrastructure, that's actually running the Internet. Right?

A lot of that is running on Linux. And if it is running Windows, a lot of that server infrastructure, which we don't see as mock for, like, these types of services, it isn't gonna be running Crowds strike. In those, like, those critical services that are out there. So that's generally why we're not seeing things go down at that level. Now.

What's going on with the Faa and everything else, my guess is that the core infrastructure for banking, the core infrastructure for airlines, the core infrastructure for a lot of things is just fine but the systems that the people use to run those apps, those critical services every day are down. Sort of the people can't monitor if the people can't maintain if the people can't poke into those things, then you've got stop airplane. Then you've got to stop financial transactions, then you've gotta stop these things from progressing. So, yes, there are absolutely servers that are windows servers that are running crowds that is in fact, the thing. But in our experience in testing, it is exceedingly rare for a very high like, like high fidelity, high availability, high bandwidth service to be running on a windows system with A falcon on it.

Speaker 6: And by the way, Microsoft, if you listen to my idea, you can write me a large, check. I will...

Speaker 0: I will

Speaker 6: definitely fully accept that.

Speaker 0: Oh, somebody just said, so should I reboot my computer 15 times? I'm seeing that? Seeing people that have gotten up to 20 and it hasn't worked. So it you can do it if you're real desperate. The other thing that I wanna talk about I wanna take a pivot, from what's going on here, from a technical under the hood perspective, and let's let's take a step out.

You're now an enterprise, you're an analyst, your a cis Okay. This is great. How it actually impacted the colonel, where the virus hooked into the colonel? That's all fine. That's all good.

How do I recover? And right now there is... I'm I'm gonna tell you right out the gate that recovery is bleak. So right now, the recovery officially from Crowds is you have to physically log in to a workstation as administrator, which I'm gonna come back to here in just a couple seconds. You have to boot the system into safe mode.

Then you have to delete a file seconds and reboot it, and it will come back. That is like 4 or 5 sentences, that sounds easy. Here's the instructions. Boot into safe mode navigate to the drivers Crowds strike directory, locate the Sis files, delete it, reboot it, and then you're back. So you're seeing on Reddit and you're seeing on some people in Twitter, where they're talking about it in terms of they have a hundred thousand notes.

And best case scenario this takes 5 minutes per node to do. That is a ridiculous amount of time. To be able to go through and reboot all of those systems manually. Jason, you had a question,

Speaker 5: Does this include people that work from home?

Speaker 0: Yes. But include people that work from home. So worst case scenario for desktop support teams is they're sending instructions to people. On how to move their computer into safe boot

Speaker 2: or just out new laptop.

Speaker 5: But but if you can't use your computer to get that information? Then how are you getting the information to fix the computer?

Speaker 0: There there we go circular firing squad. Yeah. And by the way, Jason, it gets worse because a lot of organizations have been told to put hard drive encryption on their computer system. And using like Bitlocker or something like that. You can't even boot it.

Speaker 2: And these kinds of massive problems it. Know, was a totally different problem back in the day, but I remember when the the seed files were stolen for our

Speaker 0: pay. That's right.

Speaker 2: And, you know, I worked for a large company at... And we had a hundred and 50000 employees that need your new Rsa tokens. So...

Speaker 0: Remember, Rsa didn't replace those tokens for a bunch of their customers. They didn't renew because you're were out. Shipyard, and it was Dod. Yeah. So that deal.

That sucks. Yeah. So Ryan, can you share the thing I just shot you about bypassing bit defender? Or sorry, Bit bitlocker? So bit bitlocker, this is opposed by what is it S Cali, that you can cycle through B to you get a recovery screen.

Navigate to troubleshoot advanced option start setting, restarts, skip the bitlocker first and second attempt. Navigate back to troubleshoot advanced options command prompt, Bc at its set default minimal safe boot. This, I have seen pop up in 4 or 5 different places I've seen it on Reddit. I've seen it on Twitter. I've even seen it on our on our on our discord server.

And I am seeing people to get around bit bitlocker. Sorry, referred to it as a bit to better. I'm seeing people that have tried to bypass bitlocker that this works. This is Anecdotal, you can try to do it as a desperate measures. If you...

Especially if you have a critical system that has, you know, app absolutely like the recovery keys you need to get access to this computer system. But this thing terri me because this isn't even 5 minutes to do. It's gonna take up half an hour to 45 minutes to do.

Speaker 2: You know, kia escrow for the win. Right? Yeah.

Speaker 0: He's Mh I wanted to bring in Patterson and Derek a little bit more on this. So we're saying you gotta sit down and you gotta log in into to all your Windows computer systems guys. What if you what if you're... I mean, if you go into safe mode. Right?

You don't need to have an administrator password for safe mode on some systems or can you kind of walk through what is the logic of that? What if you're using lapse? For administrator, like, password randomization in your environment? It's not an issue of just an admin sitting down? Like, What are some recommended kind of approaches for trying to handle this?

If you're seriously a helpdesk desk desktop administrator. You're staring down this. What are some pitfalls? What are some things that you've gotta you gotta start figuring out right now?

Speaker 2: Oh, Lord. I've been so far removed from break fix stuff. Like, III honestly don't know the answer to the question of does safe mode, you, we're still require an administrator path. I would say no based on experience, but I mean, if it were me, and I was an enterprise, and I was dealing with because this is an incident Right? This is, you know, basically a done all the service.

I would say I'd probably be sending out new new equipment. And having it cross ship. I know it's expensive, but I just don't know how you're gonna get remote workers to successfully. You'd spend more money on the phone with folks I think, trying to figure it out. I'd just...

I... And then, you know, I've seen people saying in chat, I've been trying to, you know, read the fire hose of, for the the folks that are like as Crowds strike gonna be financially responsible. I'm sure in their end user license agreement that there's... We don't take responsibility for outage kind of language. Right?

Speaker 0: You I I am equally sure that there's going to be judges. They're gonna throw that shit right out the window though.

Speaker 2: But I mean, as far as, like, retain customers. Yeah.

Speaker 0: III, you know what and and so I wanna continue to talk about recovery. We'll talk about the out Especially to crowds right here in a second gaining I just... So ditch Oh, go ahead.

Speaker 2: No. Just those manual instructions are really un 10 for for... Mean, any anyone who has remote workforce stuff.

Speaker 6: Yeah yeah.

Speaker 0: We've got we've got people freaking out there, like, excuse me what? Do... So you completely can bypass bit bitlocker using these techniques. Apparently, someone was sitting on a 0 day for bit bitlocker. Today's is their data shine.

Maybe. I don't know. I would like to get some more confirmation

Speaker 4: as

Speaker 0: to whether or not this actually works. So some other solutions kind of kicking kicking this out there. If you if you don't, if you're not using hard drive encryption, You can always have a linux system on an Iso that run some scripts to automatically mount the hard drive and delete these files. Yeah. That is 1.

It's just a Usb drive. It once again, if you're dealing with remote workers, are you're gonna ship them Usb drives, and it's gonna get there on Monday or Tuesday and then they it's gonna get scary. The other 1 that I've seen Banter about is pixie boot. But we Derek and I were talking about this this morning.

Speaker 2: Been so you gotta be on the same land. Right? I mean,

Speaker 0: rather be on the same.

Speaker 2: Yeah. I just... I I just...

Speaker 6: It's not gonna work for remote workers as Top Absolutely. Scale is difficult.

Speaker 0: Yeah.

Speaker 6: Yeah. And it just... Yeah. It dip... That be

Speaker 2: the beeping in, You're not gonna be able to pick boo.

Speaker 6: Right. Internet. I think I think the biggest biggest challenge here is is the bit bitlocker challenge. Right? Because, you know, if you if you have to go recovery keys or you go to Safe mate.

I mean, I know the hacks out there, but, you know, who knows if it's gonna work.

Speaker 0: But like this. Somebody said it doesn't bypass bitlocker. It just boots normally with a trusted platform module to unlock the drive. You still need to have the admin password.

Speaker 2: Yeah. Yeah.

Speaker 0: Yep. Yeah. But still kinda kinda does. I so

Speaker 1: Can can I...

Speaker 0: Linux girl last, what is pixie boot. So Pixie boot is where Instead of your computer starting its operating system from the hard drive, it goes on the network through a broadcast protocol. And it downloads an image of an operating system to load from the network rather than the hard drive

Speaker 2: typically over T.

Speaker 6: Yeah. T, it's very... Common. And and basically, what happens there is do you you put a option in your Dhcp server, which Yeah. Which points at common the network server that supplies the operating system image.

Speaker 0: Yes. Daniel just have we need to pixie Boot on content delivery networks. I'm like, there's nothing

Speaker 6: Yes and by thought about that,

Speaker 0: that makes me feel warm and fuzzy.

Speaker 6: By the way, the acronym is is PXE even though if we all say Pixie. But... Yeah. Yeah.

Speaker 4: It Yeah. Sounds cooler. Right, Off.

Speaker 2: Does sound cool.

Speaker 4: I feel

Speaker 6: like it does sound cooler.

Speaker 5: I feel like Patterson quietly in the background finding a fixed for this and then the very end he's like, alright. So here's how...

Speaker 0: Here it is. He's the ninja that's quiet in the back of the fight. He's jumping in. What do you got, Patterson?

Speaker 1: I I I'm for Oh, I'm I'm I'm border on captain obvious, but I can't help myself in that, as an incident responder, You, this is the crap we've been warning you about. And and but I I just wanna say, please forgive me, you know, never let a good crisis go to waste. All of the things that we're wrestling with out of band comm, prioritization of business critical resources, backups, resilience, This may be the wrong time, but that's just top of mind for me to to walk out the other side of this as an incredible learning opportunity, to redefine and or, define ways that you're going to confront these kinds of issues because this is, and, this is denial of service at scale And again, it is... It's like, the the world's greatest tabletop exercise.

Speaker 2: Yeah. Like the Ara code. Right? Like, Oh, my gosh.

Speaker 0: It's like everyone gets to play saudi of Ara. Yeah. I ordered it out this morning. On the bright side, lots of companies get a live ransom wear recovery test this morning. Yeah.

Because you... You're worried about ransom, this is the same. Like, these, like, just like Patterson said, these are the same steps. If you can do this well. You're doing awesome.

Keep going, but I'm guessing a lot of organizations can't do this well. And unfortunately, there's no amount of money. That you can pay somebody that they will give you, like you pay them at Bitcoin and they give you a recovery key that recovers your ops. Like, there is no really cool reset button with the Ransom ware group where they're, like, give us a million dollars and you get everything back. That doesn't exist today.

Speaker 6: Yeah. Watch out for the fishing tactics they're gonna go flowing really.

Speaker 0: Pitching tactics. A little bit. Before John left, he showed a bunch of domains that were registered, like, within the last 24 hours. And here some of them, crowds strike blue screen dot com. Cloud crowds strike 0 day dot com.

Crowds strike BS0D dot com. Crowds strike doom day dot com. Crowds strike fix dot com. Crowds strike down dot site. Crowds strike token dot com.

So it's brace yourselves. Like you think this is bad like the attackers are gonna take advantage of this? This is coming right now. So, Derek, do you wanna... You were talking about that with me on the phone?

Do you wanna talk a little bit more about this? Like, the

Speaker 2: I mean, we just saw this. Oh, gosh. I just read about this yesterday on another, can't I remember the the the software package. We actually had a sock customer that bell for something similar where, you know, you Google the problem. And it's, like, here's the solution of the problem.

So you couple either, like, fishing or social or a search engine optimization, and you get the the fix or the update, and well, it's not the fix or the update.

Speaker 6: Right at

Speaker 2: all it's backdoor. And, yeah. So that's what I worry about. Those are just some of the Io c's that are known now, and I imagine there'll be more to come.

Speaker 0: So from a defense perspective, 1 of the things you might wanna do with your Dns security provider is basically looking at the age of domains slick. Like, what is the age agent's domain? I mean, you should be... And this goes back to what Patterson said. This is all shit.

You should have been doing too. Right? Like, isn't like, oh, god. No. We never saw this coming.

Speaker 2: Take advantage of

Speaker 0: This is that really hurts me is like the past people that are listening to this webcast are lighting up discord and I cannot keep up with what's going on on meeting right now. They're not the ones that worry me the most. The ones that worried me the most, so the ones that logged into Cnn dr report, whatever. And they're like, good god. What happens?

What is this? And they're starting from ground 0 right now.

Speaker 2: Yeah.

Speaker 0: That is what really scares me.

Speaker 5: So I believe we're gonna have a new back and bridges and inject card. Do.

Speaker 4: Yeah. I but I think we 1 the card is gonna be.

Speaker 0: We did a we

Speaker 1: did a tabletop yesterday, and and we actually pulled the Ed, you know, endpoint security, and then they rolled the failure and everyone on in the audience was inc. Are always works. I'm

Speaker 6: like, no. Really.

Speaker 0: Back to him to get you like

Speaker 3: me see

Speaker 1: Sometimes things don't go as planned.

Speaker 4: Like Yeah.

Speaker 0: I what timing is that?

Speaker 2: So somebody actually... I just call something going by, so updates are bad. Right? I mean, these are the kinds of things got that make you know, organizations think, well, now we're not gonna update because look what happened. And right answer either.

Speaker 6: So so the new the new... Back to us some breaches cards. I think think it needs to be named Colonel Driver of F buy. That's what we need.

Speaker 4: So at the

Speaker 1: at the the first crowds deployment ever did. A long time ago, we had issues with, basically denial service through over saturation of internet connectivity based on updates. And then you could group your updates thereafter. You could actually manage distribution of Crowds strike updates to endpoints based on grouping. And and you not do that anymore, and you exert some manual control to say, testing

Speaker 2: know. Somebody posted a minute ago, and III just caught it, and I'm not sure if it's true that even if you were trying to stay behind, like that this basically, you got this update either way. And that's 1 thing I never really agreed with in the... Certainly for personal users when Windows did it, but know, I I never really agreed with forcing updates on organizations. And I know there's ways around it for sure, but, you know I think the automatic update thing It needs to be an option.

It shouldn't shouldn't on.

Speaker 0: Okay. So I'm gonna throw this out there. I'm gonna disagree. This is bad. No question.

But God it if takes us back 10 years with auto

Speaker 2: That's political it. That's what's gonna happen.

Speaker 0: We don't... We we can't have that.

Speaker 2: Right? No disagree. No, but with follow up.

Speaker 0: But we've used to be I used to teach way back in mid 2000. It was like, you should always test and validate your updates for, you know, a couple of weeks before you roll it out to production, and it's like, we ain't got time for that

Speaker 2: I I stopped. Saying that and more of, said in it and it does apply to this too. Just get get good at fixing it when it breaks. Right? Immediately deploy to, like, your It team that would be, you know, responsible for fixing it Right?

We're get to auto

Speaker 1: out q You do go ahead 1 crowds and hunters, and then at least only 1 third of your environment

Speaker 2: already the space Yeah. Yeah. Well, that's an idea.

Speaker 1: Let in depth, not really.

Speaker 0: So I you know, have joke about that, dude? We joke about that, but I guarantee... And... Is that a bad idea today. To basically say that we have multiple offices.

We're gonna use different Ed r's to make sure that we don't go down a hundred percent.

Speaker 1: We used to recommend doing a bifurcation of endpoint security on workstations versus servers, partially for that reason.

Speaker 4: Yeah.

Speaker 1: Not a, again, not a terrible idea. Sure.

Speaker 0: And the article I was, trying to remember during my class, but I remember Dan Gear and Bruce Schneider, the mono culture, paper, where they were talking about the dangers of mono culture. Like. We're still there. Right? That is absolutely a problem that exists.

But I I just don't know where the hell this goes. I don't think it goes anywhere good. Like, I think that you're gonna have Ceos are gonna be, like, rip and replace crowds strike now.

Speaker 6: Oh, yeah.

Speaker 0: And Crowds strike, just to be honest, Matt, I wanna get your take, off I wanna get your take They aren't a garbage Ed. Right? Like they're not bottom feeder. Right? So we might be seeing 1 of the better Ed r's in the market.

Get completely blown out of the water because of 1 stupid Ci mistake. And I'm not dude Crowds strike doesn't pay me. Right? That I get no money from them. In fact, we have a love hate relationship with Crowds strike.

We absolutely do

Speaker 2: love to hate us.

Speaker 0: They love to hate us. They keep shutting down our upping licenses little researching. At the end of the day, I mean, I'm gonna throw this out there, everybody. God damn. They're a pain in the ass.

That's a good thing.

Speaker 2: Come Right?

Speaker 0: So you want it to be a pain in our ass.

Speaker 2: And, like, if they handle it in a away. Like if I was a ceo or they handle it in a way where you know, they take ownership and and and and try and make it as right as possible. I don't know that I would say. Alright. Because this might happen to any product.

Speaker 0: You Yes. That's the key. If you think you're gonna move off to another vendor and they're never going to have this. I've got this. This great poster that I saw today where they went through and they listed out all of the other vendors that have had Ed vendors.

That have had the same type of issue. They just aren't as big. Ryan, could we kick this up?

Speaker 2: Yeah, with

Speaker 0: it Quick. Yeah.

Speaker 2: Yeah. It's just... It's just not. Ed vendors as well. Right?

Speaker 6: It's not.

Speaker 2: I think that this could happen to any enterprise, class software, But.

Speaker 6: It's and any product at all that involves itself with the tunnel, which is a lot.

Speaker 2: You know, And I've seen thousands of networks since I've been at. At black hills like both large and small. And if you think the really large shops aren't held together with, like, duct tape and pops sticks behind the same wrong, Right? They are are. And so I could just imagine this happened because a filed didn't transfer right.

Nobody paid a etch. Right?

Speaker 0: Yeah. So I... So the tweet talks about blue screen of death in what is this cortex. Another 1 in K. Another 1 in s semantic endpoint protection.

Another 1 in Sop net filter. So, you know, God damn at Crowds strike. You're putting me into a situation where I have to defend you. I did not want to be at this point today. I did not wanna be at this sprint today.

So

Speaker 6: no. I I

Speaker 1: don't... Was 1

Speaker 4: I wanted to jump into. I know going it back a bit. A lot of Ed vendors have the ability with their updates to actually do staggered updates where you can do grouping of updates or, you know, push live to a big group or you can actually set it to that way, depending on new settings, you know, you can have a small subs section, get the latest bleeding edge updates and then your court brew. So like, when to say this whole thing about, like, people are gonna stop doing updates. I think the only thing people that benefit from that, are threat actors when you do that, and I wanna stress that.

There are some settings and controls like free prod that can deploy this. I think well, this is gonna definitely have a bigger conversation, how is your update policy set up? We... For a lot of time in security focusing on, you know, How you're rolling out these updates, How they deployed how they're being tuned. We focus a lot about that, but no 1 really focuses it on update tests.

And I think this is a great opportunity to look at these products. Because I know a lot of Ed across the space have the ability to do, like, too staggered behind updates from the bleeding edge or the current 1. Yeah. And I think this... This is a great opportunity that if you somehow we're able to set your thing not always push the live up update blindly and have it go to a small control test.

You might be 1 of the few companies that are not having this issue today.

Speaker 0: Yep. Yep. Now the big question that I'm getting from my family is, like, when do we buy on the dip? III honestly this is... III hope I'm wrong, but I think this is, like Crowds strikes going go down.

It's not gonna go out of business completely, but it's it's going to go down. Like, there is... Ceos having very blunt conversations with their tech team, and they do not give a shit about any of the things that we just said, and they're like, we need blood. We need a head on the platter and crowds strikes going to be the sac artificial lamb. I I think that that's going to happen.

You think the dip already happened possibly. Right? It might be. I don't know. But III would not people...

I would not look at this as like, 2 weeks, everything's going to be okay. It might be. It might be, but this is this is bad. Especially if it gets to the point where flights are down for, like, the next, like week or if it led to people dying. Like, there's hospitals that went down, operating rooms that went down.

Dialysis machines that went down, at least that's what we're seeing. Right? A lot of that shit's not confirmed, But, you know, standard outage, companies can recover from. If you actually led to loss of life in you're a hospital, and you got a story of another hospital that went down because of this product. Right wrong or in different small mistake, anybody can make that mistake.

If you choose to keep that software in your environment and something like this happens again, The liability is not going to be on the vendor. The liability is going to be on the hospital that decided to keep that vendor that had something like this happening in the past. And once again, I don't think that's fair to Crowds strike. I don't think it is, but there's a lot of organizations and lawyers that are looking at it through that type of lens. And they have to do something.

And I... And that's that's my big fear about what we're seeing this right now. Yeah.

Speaker 5: I, I have a lot of empathy for the people are gonna work the crowds strike booth that black hat.

Speaker 2: Oh, yeah. Yeah. It's gonna

Speaker 0: be Page even just close it down. Yeah. It might be entirely possible that Crowds strike this isn't worth it. Because think about all the memes and the people on Twitter, they're gonna be going by and taking a picture of them flipping off the crowds strike booth. Right?

That's gonna be the hot thing to do at Black hat.

Speaker 5: I'm gonna plug

Speaker 0: Crowds strike. If you're thinking of going to Block hat, don't booth

Speaker 2: Well, that's nice it. We can not... We can... As a community, we're not always the nicest foe.

Speaker 0: No No. No. No. No. No.

Speaker 1: Can can I Can I suggest that you prepare your speech now when somebody says, you deployed Ed on my biomedical systems and risk the lives of patients that not to minimize that concern, but I'm here to tell you for a certainty if Ransom was detonated on your biomedical systems, it would directly impact your patients? So be prepared to have a logical conversation about the cost benefit of Ed versus exploitation versus threat actor acts this

Speaker 0: and right

Speaker 1: cannot overs stress that... And a logical conversation, not an emotional conversation about there is a risk this they're running Ed, and it's this. There's a risk to not running Ed and it's it's it's this.

Speaker 0: Well, and I'm also gonna kind of, like, r on that a little bit. If you're a crowds strike right now, you need to within the next 12 hours. Be saying, what are you doing moving forward that you were not doing a week ago. Yeah. You cannot say that an intern screwed up.

You cannot say that an admin screwed up. You cannot say that somebody oops seed. You've gotta say what are we doing that's fundamentally different moving from this moment on, that we were not doing last week. And I also think being very open about what happened and saying, oh, there was a problem in our Ci pipeline Not good enough. Like, with something like this, you're gonna have to say here's specifically where the boo boo is.

And I would recommend looking at Man, and whenever they were breached and of course, that led into the whole solar winds thing. He is open as you possibly can. Like, if you're trying to minimize this by doing obscurity, if you're gonna try to minimize this with this community by trying to double speak corporate speak and going around it, you're f. Not only are you f you deserve to be aft. If you get in front of this and you get very detailed technical explanations of where the program broke down the Ci pipeline, The validation process broke down and what you're doing to rectify that, you may survive this.

I don't have a lot of hope for Crowds to do that. I I just seriously don't. But I think when this community, like, we react better to Sunshine, which is weird because we're It geek and Sunshine makes us burn in ways and impressive that are uncomfortable. But the more information that you give us on what's going on on this, The more we're going to trust you moving forward and your only marketable thing that you have at Crowds strike right now is trust. And if you start burning that with corporate double speak, you're done.

Speaker 6: Yeah. IIII wanna not to not to be shameless plugging, but I'm gonna do a slight plug. Back in February. I released, blog, February 20 second, called, initial access operations. Part 1, and it talks about the Windows endpoint to affairs technology landscape.

So aside from the things that I've said today and that Matt Ar said today on this, news cast. You know, it it behaves us to to do a little research and read up on some of these things to to to to you know, get get educated about some of the things that are going on in the systems that we're managing every day. So I just wanna to put that out there. Go read it, and I think you'll learn a few things and it helps reinforce some of the stuff that I said today.

Speaker 0: Alright. Alright. Everybody. Let's wrap it up We've gone a little bit over. Thank you so much to those of you that joined.

Thanks to John for joining. Thanks for Ryan for being here when you stick with Covid, it's not cool that he's here, speaking of processes that need to be fixed. We need people that can back him up, but he's so good. No. He's he's asleep.

Don't tell him. He's getting a big head is what you're saying. He's got Covid. He's not gonna remember any of this anyway. But but seriously, I appreciate you guys jumping on and really the community jumping on.

I think we had, like, 3000 people come on, which is huge. We really appreciate all of you in the community. You blew up my desk discord in ways I didn't know was possible. I couldn't keep up with you. But we appreciate all of you and good luck and, god bless to every single 1 of you.

It's gonna be a rough weekend. Take care.