Summarized using AI

Practical SaaS Security

Michael Buckbee • October 06, 2022 • Los Angeles, CA • Talk

In this presentation titled "Practical SaaS Security" delivered by Michael Buckbee at The Rails SaaS Conference 2022, the speaker shares valuable insights on managing security for web applications, specifically in the context of Software as a Service (SaaS) environments. The talk focuses on the threats posed by malicious actors, the functionality and importance of Web Application Firewalls (WAF), and introduces a new WAF toolkit called Wafris.

Key Points Discussed:

  • Incident Response: Buckbee opens with a dramatic depiction of the stress and chaos during a security incident, underscoring the need for effective communication and rapid response.
  • Threat Actors: The term 'threat actors' is emphasized, indicating a shift from the traditional label of "hackers". Buckbee outlines their methodologies, notably using resources like proxy servers and botnets to launch attacks against vulnerable sites.
  • Web Application Firewalls (WAF):
    • Explained as a critical component in defending against incoming threats by acting as a rule-based filter between clients and application servers.
    • Discusses how WAFs can be configured to manage traffic and block malicious requests based on defined rules.
  • Real-World Incidents: During the session, Buckbee shares several real-life case studies:
    • DDoS Attacks: Examples of different types of distributed denial-of-service attacks faced by companies, highlighting how they affected operations and customer relations.
    • Intrusion Events: Emphasizes hidden attacks like credential stuffing, which often go unnoticed until significant damage is done.
    • Each case study illustrates challenges encountered and strategies utilized to mitigate the impact of these attacks.
  • Survivorship Bias: Discusses the tendency to focus on highly publicized incidents rather than the common, unreported intrusions affecting the majority.
  • Wafris Toolkit: Buckbee introduces Wafris, a new open-source WAF service designed to offer lightweight, efficient security solutions for SaaS applications through Redis integration. He highlights its benefits, such as real-time reporting and simplicity of implementation.

Conclusion:

The presentation concludes with the call to action for the developer community, emphasizing that security should be a collective responsibility and highlighting the importance of being proactive rather than reactive in addressing security vulnerabilities. Buckbee concludes by encouraging engagement and collaboration from attendees to advance security practices within the SaaS ecosystem.

Practical SaaS Security
Michael Buckbee • Los Angeles, CA • Talk

Date: October 06, 2022
Published: unknown
Announced: unknown

Mike joined us at The Rails SaaS Conference in Los Angeles, California to share practical lessons learned managing security for hundreds of businesses and also released a brand new Web Application Firewall (WAF) toolkit called Wafris!

You can join us in Athens on June 1–2, 2023. Details are available at https://railssaas.com. To follow announcements of future events, follow http://twitter.com/railssaas.

Thank you to our corporate sponsors who made the event and the production of this video possible:

EVENT SPONSOR
ClickFunnels (https://clickfunnels.com)

PRODUCTION SPONSORS
Buzzsprout (https://www.buzzsprout.com)
Geocodio (https://www.geocod.io)

POST-PRODUCTION SPONSORS
Render (https://render.com)
PlanetScale (https://planetscale.com)
Evil Martians (https://evilmartians.com)
Entri (https://www.entri.com)

0:00 — “The Incident”
1:52 — Threat Actors
6:20 — Web Application Firewalls (WAF)
8:37 — Real-World Incident 1
12:40 — Real-World Incident 2
16:43 — Real-World Incident 3
18:48 — Real-World Incident 4
23:17 — Survivorship Bias
27:27 — Vulnerabilities you don’t expect
29:53 — Introducing Wafris

The Rails SaaS Conference 2022

00:00:48.079 all right a world of betrayal
00:01:01.399 a world of pain
00:01:08.700 the web app is no longer your own world can you hear me it's been 48 hours since
00:01:15.240 we lost control of this site I haven't slept and my cats have started giving me advice
00:01:22.340 a team of developers would stretch to the Limit I've never seen an attack like this it's
00:01:29.640 huge do you think it'd be easier to just get new jobs than deal with this yeah let's get tacos
00:01:52.920 I'm the person who gets on a call when there's an incident and on that call it's most likely your boss maybe some
00:01:59.700 board members maybe some people from your legal counsel and what I say is I
00:02:06.600 see this all the time it's gonna be okay let's get started so the reason I'm on that call is I run
00:02:14.640 a service called expedited with stands for web application firewall and we
00:02:19.980 mostly work inside Cloud environments mostly on Heroku and mostly larger Enterprise type sites but before we get
00:02:26.220 into like details of a web application firewall I really want to talk about the people we're trying to protect your
00:02:31.620 sites against and those are threat actors we're in Hollywood these are not actors actors these are the polite way
00:02:39.300 of saying hackers um hackers gets used in a lot of different ways so we rename the variable
00:02:44.519 to threat actors and now serious security people say threat actors instead so
00:02:50.040 now you're all serious security people and this is the number one tool of
00:02:55.140 threat actors which is resources I always try to think in terms of what it is they're trying to do and how we can
00:03:01.920 overcome that so it's very easy to get lost in all the technical aspects of this and really it's a resource thing
00:03:08.519 and so money is the big thing they use as a resource I don't know about yourself and quick side note I took this
00:03:15.060 image from the treasury site and it's got a lot of cool weird security features in it so you can't print off your own so what is it that threat
00:03:21.959 actors are buying they are buying proxy servers so they
00:03:27.420 run scripts the work through a proxy that attacks your rail server now proxy
00:03:33.060 is one of those terms like database like there's lots of different things to do something similar it's important to keep them straight
00:03:39.120 because we're going to be a little bit confusing otherwise this is a forward proxy you know what a proxy is a forward
00:03:44.700 proxy is if you're making a request and you know you're using it versus a reverse proxy where it's invisible to
00:03:50.760 you further these the term for this is anonymous proxy and what exactly is that
00:03:57.900 it is this this is a real list of proxy servers you could take this list any one of these IPS plug it in and use it
00:04:07.319 and now a question who here does not have kids anybody a couple people
00:04:13.080 I'm Gonna Let You in a secret which is that kids want more than anything in the world to download the sketchiest
00:04:18.900 weirdest mobile applications to every device you own
00:04:25.199 the laughter confirms it and the way those are monetized quite often especially in the Android ecosystem is
00:04:31.380 that they actually have a proxy built into them those are then harvested resold and they end up in a list like
00:04:38.160 this same your parents government Thanksgiving it's coming up you got to do tech support
00:04:44.000 their machines use this proxies servers get compromised even a lot of times web
00:04:49.620 extensions get compromised so that's where proxies come from all right so we have proxies we have a
00:04:57.120 script we're a threat actor now we need a botnet so what I want to show you is
00:05:02.820 an actual botnet and we're going to write it right now so pay attention but be responsible
00:05:09.360 um this is a botnet so this is curl curls uh Unix utility ships on your Mac
00:05:15.120 that we're all using and the dash X says Hey use this proxy this is the IP address and Port from the first one in
00:05:21.360 the list from the other slide and if we just swap that out over and over again and made requests over and over again we
00:05:27.360 could take out a site that would look like it's a worldwide attack now we're rubyists most of us I think so
00:05:35.400 this is what that looks like in Ruby this is a slightly more sophisticated one where we have an array
00:05:40.800 we're attacking a URL this is a little pseudo Cody and I use HTTP RV for it but
00:05:46.080 whatever HTTP Library you like they all support proxies now
00:05:51.180 if you were thinking oh this is a global botnet this is very cool there's a sense of like crude sophistication with all of
00:05:58.680 these uh and by that I mean this is not something that you're
00:06:03.720 dealing with an evil genius in most cases this is something that they're trying to hit a bunch of sites the
00:06:09.539 dumbest way they can to figure out who has been the laziest in terms of protecting them and that's really what
00:06:15.120 lets them get into things so now that we've talked about threat actors let's talk about the other side
00:06:20.460 of this which is web application firewalls this is a term just like proxy or database this is a category of
00:06:26.340 software Services I run one but there's others and as a developer if you do not know
00:06:32.220 what a web application firewall was before coming in here you are not alone majority of developers I know don't
00:06:39.120 really think about it that it's not something that's part of their stack it's not something that they use or interact with or think about so in most
00:06:46.740 cases we have Chrome and we have a rail server they talk to each other seems straightforward
00:06:51.900 the WAFF sits in between these and it actually is where your SSL connections
00:06:57.240 are terminated and what Chrome talks to and then on the back end it takes that and then talks to your actual
00:07:04.080 application server so it's one step removed from how the requests normally go
00:07:10.139 now here we're adding in the bits from before so this is a script from a threat actor interacting through proxies that
00:07:17.039 are hitting a weft and now the WAFF is making decisions and that's all a laugh is it's a rules engine that's applied to
00:07:23.580 incoming traffic we're in La there's a lot of clubs imagine there's a club someone walks up to it and the bouncer
00:07:30.180 says to them hey no mesh no mesh tank tops so maybe that person responds do you
00:07:37.560 know who my dad is like this this mesh tank top it costs more than your salary for a week but it
00:07:43.380 doesn't matter because they have a rule so we similarly have rules in a while and one of the most basic rules is who
00:07:50.280 gets in and who gets out so you can imagine a situation where you know like you're an IP address and I say you
00:07:56.580 cannot come in no offense so and then on this side we have someone else who's in
00:08:01.680 IP address we said you're on the VIP list you can certainly come in but nobody else can and that is really my
00:08:07.919 definition of a laugh if you do not have the ability to go and type an IP address in and block someone from accessing your
00:08:15.240 site it's not a WAFF I mentioned that because there's a lot of like waffleg services and other proxies and things
00:08:21.539 but that's really what it comes down to now thousands of customers a lot of
00:08:27.660 empirical information around what a tax assesses get what those look like all
00:08:33.419 sorts of things that are very valuable to share but there's two problems the first of which is that everybody wants
00:08:39.000 to know what happened to everybody else and nobody wants to share what happened to them
00:08:45.000 this is a real problem because it means we can't learn from all of these things and it's not that people are like
00:08:51.480 calorie or afraid but there's a lot of weirdness around security incidents in particular there's a lot of
00:08:57.360 untested legal exposure just quick question who's here from uh the EU anybody
00:09:04.320 couple people hiding somewhere so in the EU they have the gdpr which is a
00:09:11.100 strict set of data security regulations that say like all sorts of things you're supposed to do or at least achieve in
00:09:16.560 the US who is here from the U.S state anybody it's everybody do you know the number of
00:09:22.380 emails involved in a data breach that means you need to contact your State's Attorney General
00:09:27.720 nobody the actual laws and about half the U.S states that have something very similar
00:09:34.260 to that and lots and lots of other stuff so people are worried about having that
00:09:39.839 more vulnerabilities more disclosures lots of issues so my solution to that is
00:09:45.899 we're going to do based on a true story so these case studies I'm about to show are real so the names and countries and
00:09:53.580 industries maybe not but the actual attack details have actually happened and it's something that I've helped
00:09:59.399 people out with so these are incidents now incidents are
00:10:05.040 what came from the video in the beginning there are things that take your site down to the point that there's calls and people are upset and all sorts
00:10:11.519 of things and I'm going to break these up again trying to work the resources from the ones that occur the least to
00:10:17.160 the ones that I see the most and so the first is distributed denial of service attack which is the proxies that the
00:10:23.820 threat actors are using and it has demands these are actually really rare I think that's interesting because when I
00:10:30.120 talk to most people this is what they think of first they think someone bad is going to come and try to extort me for
00:10:35.940 something and that is what happened to Bull Capital the fine people who have a
00:10:41.160 financial services company that does really aggressive like AI investing kind of stuff and one evening Denmark time
00:10:47.220 threat actors came into their Discord and they said we would like a thousand Monero please uh you may not be familiar
00:10:53.940 with Monero it is an untraceable crypto and it was recently used to trace a Trader at the NSA so your mileage may
00:11:08.940 pronounce that properly or 150 000 US Dollars roughly um their site was down for about 24
00:11:14.399 hours and got on a video call with them the founder of the company was chain
00:11:19.860 smoking and in the midst of is Copenhagen lofts in the smoke his hand was shaking and saying we must
00:11:27.060 we must get the site back online and Q is about ready to pay
00:11:32.160 said okay looked in the eye as much as you can across zoom and said
00:11:37.920 I see this all the time it's gonna be okay let's get to work and
00:11:43.800 we did and the first thing we did was nothing technical we talked about his business and starting from that point we
00:11:50.640 found all his customers were from the EU how is advertising work how all the
00:11:56.940 pieces came together and we made a plan we did some Geo blocking some other things and we stopped the attack now
00:12:03.720 that may sound underwhelming especially from a technical sense and that is because in this particular case the WAFF
00:12:10.980 is the perfect thing to fix this stuff in the same way that you're trying to store information long term you use a
00:12:16.740 database if someone came up to you and said like yeah I've got a ram drive that's on a USB stick and I stick it
00:12:22.140 into the laptop you'd be like you're a crazy person that is not how you do this similar thing here okay so the next
00:12:30.120 category distributed denial of service attacks with no demands now these are a lot more varied this happens quite often
00:12:37.860 in like weird circumstances so we'll go through a couple different scenarios here so this was a therapy as a service
00:12:44.220 site and they were hit with the largest DDOS attack that I have ever seen uh it was
00:12:50.700 over a billion requests a day their site was down for over 72 hours and it was a
00:12:57.060 very dicey situation it was dicey because they're a therapists that founded it they're not technical people and then
00:13:03.779 they had contract Developers and the owners did not understand
00:13:09.139 why the developers would let the situation occur so
00:13:14.459 to give them some perspective on this very tense call I did a breakdown that was pretty much like this which is that
00:13:20.279 a billion requests a day works its way down to about nearly 12 000 tax a second
00:13:26.220 that is a lot to handle and it's not the rails as slow or can't handle these things but that the infrastructure
00:13:31.560 needed for this kind of stuff if you're actually trying to build it up to sustain it is more than what you have
00:13:37.740 it's not just more of what you have in place it's an entirely different set of infrastructure it's not you need at
00:13:44.519 least bandwidth you need database connections you need all sorts of other things you don't have in place and most
00:13:51.360 importantly even if you had the perfect plan in place to go from a hundred
00:13:56.459 requests a second to 11 500 requests a second during an attack you can't you
00:14:03.480 can't because the attack is so large and consumes so much resources you can't get an SSH connection you can't SSH to your
00:14:10.560 servers you can't deploy you can't find logs to stuff because everything is breaking down because literally it's
00:14:17.339 dropping packets so your SSH request doesn't go through nothing works so you can't even make changes to your
00:14:23.279 environment all right so let's look at what these requests look like so to try
00:14:28.560 to complicate things the attacker wanted to defeat any caching that was going on so they used a hash function if you're
00:14:36.120 not familiar with hash functions this is md5 which is an older one but you put a value in and you get a consistently
00:14:42.600 sized value out that is very hard to reverse engineer you can't take this top value of like C4 ca4238 and turn that
00:14:51.000 back into one in any reasonable way okay so then they made a billion requests
00:14:56.459 that looked like this each GB get request now broke any possible caching and these are
00:15:02.579 404 now for a foreign not found it's still taking up a bunch of resources it still
00:15:09.060 has to be SSL terminated so running through your routes file routes and rails can get freaky you can do any sort
00:15:15.060 of execution on it and because it's not found it's running through your whole routes file for every request then you
00:15:21.060 have to do string parsing and concatenation to get a page that under normal circumstances would say like oh
00:15:26.699 hey it looks like you're looking for slash nine eed2e that's not found and finally get a
00:15:32.519 response to HTML which takes bandwidth now here's the sophisticated and unsophisticated part of this which is
00:15:37.920 that I just explained this whole hash thing seems very complicated why are they doing this does anybody know
00:15:43.199 another set of values from one to a billion that are unique literally the number is one to a billion
00:15:52.199 they could have just done this would have had the exact same results again crude sophistication happening
00:15:58.440 here and how we stopped this was you know rails most year routes you know what they are they're the resource
00:16:04.680 routes you have products you have users so we white listed all the real routes and let everything else get kicked out
00:16:09.899 and we're able to bring them back in line now a couple takeaways they never
00:16:15.300 found out who did this whether it was a disgruntled ex-employee whether it was a
00:16:21.120 customer just very odd never had a ransom demand and something else to imagine is that
00:16:27.120 they weren't sure if they would actually know us they had a ransom demand imagine your support team they get an email that
00:16:33.660 comes from a throwaway Gmail account talking about Monero talking about a DDOS attack would they even know what
00:16:40.680 that means it's hard to tell all right so next case the Irene brand candy and
00:16:48.839 confectionary Sarah massive candy wholesaler and they had an API that was pushed out to a lot
00:16:56.759 of mobile clients for Commerce and they had no rate limits on the API and the reason for that was that they did not
00:17:03.120 want to stop anyone from shoveling money at them as fast as possible which makes sense but one of their customers pushed a
00:17:10.260 really bad update which is like a loop and a loop and that one customer suddenly started making a hundred times
00:17:16.319 more requests than all their other customers combined very bad lots of people are upset they're losing
00:17:22.559 money so they made a rough decision which is it's an API they'll add API keys they
00:17:29.220 revoked that API key so now they've replaced fairly fast queries which are
00:17:35.580 like hey give me all the attributes associated with product.e1234 with re-authentication attempts
00:17:42.240 reauthentication attempts are deliberately slow uh they're talking about bcrypts kind of stuff they're slow
00:17:47.640 to stop like timing attacks and other security things so that's bad but also these are all mobile apps so what
00:17:54.299 happens to mobile apps that are on flaky connections they retry themselves again and again
00:17:59.400 and again so they went from a hundred times more traffic than the rest to a thousand times more traffic than the
00:18:04.440 rest and that's when I got on a call with them and said yeah I see this all the time so
00:18:11.340 um so did that were able to fix things for them credential stuffing attacks this is by far the most common reason I
00:18:18.900 see sites go offline which is kind of weird because it's not deliberate this is something that we see used all
00:18:25.020 the time for any site maybe it's they want to do fraud of some kind anything with money maybe that's spam emails
00:18:31.620 they're trying to get out any kind of resources or a lot of times it seems like there's nothing involved and what
00:18:36.900 they're actually doing is they've gotten a list of email addresses and passwords from some other data breach and they're
00:18:42.120 essentially testing it for cleanliness to see like oh are these still active is this something that's happening so it
00:18:48.600 brings us to the National Sports League National Sports League had pay-per-views so I don't know anyone
00:18:55.380 buys pay-per-views is anybody who does which doesn't seem like the right crowd for it
00:19:01.860 happy with the price of them and the answer is going to be no people hate how much pay-per-views cost they
00:19:08.880 think the price is too high and there's sort of a grudge pricing aspect to it but pay-per-views are bottom line and
00:19:14.880 interesting thing is they typically aren't like you put a credit card in what happens is you put in the
00:19:19.919 credentials of your cable provider into there it does this whole back-end reconciliation and then you get the cost
00:19:27.480 on your next cable bill well that makes a huge Vector for fraud because they aren't directly interacting
00:19:34.559 with credit cards so many of the protections aren't there so National Sports League had literally millions and
00:19:40.679 millions of attempts trying to break into these because as soon as the attackers are able to crack one they
00:19:45.720 could turn around and sell it for a couple bucks and they were making tons of money so this is what the traffic
00:19:51.780 looked like red is attackers and then blue is legitimate so this is bad on both fronts certainly
00:19:58.860 they don't want all these attackers but also the attackers are hitting them so hard it's pushing out the legitimate
00:20:04.140 users so nobody's happy now pay-per-views are very time sensitive
00:20:09.240 they have to be sold before the event happens and they're no good afterwards
00:20:14.340 so their National Sports League they're in the cloud they already have a massive
00:20:19.380 infrastructure and so they just threw more resources at this double triple the service and the
00:20:26.580 instinct is Right which is like sight down Hulk must fix site just like going for it but you see on the right here all
00:20:35.280 they really did was enable their attackers to make even more money because that's where the extra resources
00:20:41.340 were going and the number of legitimate attacks stayed the same the instinct's right but
00:20:46.980 the application is wrong you cannot infrastructure your way out of these attacks because it only makes it worse
00:20:53.100 so they realize that and they said what we're going to do is we're going to do rate limiting now they started at two
00:21:02.280 attempts per hour so someone's logging in they mess it up twice they're they're blocked they can't log in again
00:21:09.299 for some period of time and on the zoom call you could hear in the distance almost like the support team just start
00:21:15.840 crying because so many legitimate users were being locked out and so support cases
00:21:22.020 went way up so I said okay this isn't going to work and they switch it to five attempts an hour which is pretty generous in the scheme of things now
00:21:29.760 a thousand proxies five attempts an hour five thousand times twenty four hundred and twenty thousand attempts a day
00:21:36.960 I'm gonna put out there it's a career limiting news that if your CEO says oh we have to stop this attack and you're
00:21:43.140 like fix the boss we switched it you can only do 120 000 a day
00:21:48.840 so it's not something that sits well with non-technical people because it sounds ludicrous and it is and then also
00:21:55.440 it's not something that sat well with the threat actors and you know what they did they hit back and spent five dollars
00:22:03.059 spend five dollars bought roughly another couple thousand good proxies at which point I spoke to them I see it
00:22:10.380 all the time Gunner called them started going through their logs and this is what we saw
00:22:15.480 does anybody see anything wrong with this user agent this is what was coming in on the request
00:22:24.000 yeah so I can rephrase it that I think would help people which is can anyone spot the problem with this user agent if
00:22:30.960 you saw it today in your logs yeah this is old so this user agent
00:22:37.679 version string is from January of 2022 and this is a very common thing we see which is that
00:22:43.559 you know a lot of times user agents are faked very easy to fake user agents but
00:22:48.720 a lot of times like headless Chrome instances that are using these attacks and things they lag and since Chrome
00:22:54.179 auto updates it's not a big deal and to pick these out and also from the
00:22:59.220 support side it's very easy to fix that if someone calls like calls in and says like I can't log in because it's giving
00:23:04.860 me this weirder say just restart your computer they're restarting they can get back in So something that worked for
00:23:10.500 them now that was incidents that was things that take down your site all that so
00:23:17.760 I want to jump back to something so World War II that was not a great segue but we'll just go with
00:23:25.559 marketing back to World War II they had planes going out and not so many planes were coming back and so statisticians
00:23:32.039 the developers of their day were brought in to try to see what they could be done with this and they were shown an image
00:23:37.679 like this which was hey here's the planes we have and here's all the holes where should we put the armor
00:23:43.320 and where should they put the armor and although like on the right one they see like all the holes sort of overlapping
00:23:49.500 well they all put it in the wrong spot you have to put the armor where there's
00:23:56.100 no holes because those are the planes that came back planes that come back all the holes
00:24:01.860 represent is where it doesn't matter if it gets shot similarly I'm trying to tell you these are all
00:24:08.940 sort of dramatic and cool these big incidents but I don't think this is really what people need to worry about I
00:24:14.340 think what they need to worry about are intrusions so intrusions are something that happens
00:24:21.179 to 100 of your sites it's happening right now it happens constantly uh so
00:24:26.400 these are hidden attacks they won't take down your site you probably won't know that they've occurred and if you do see
00:24:32.280 them in your logs you may dismiss them so let's talk about Oliver's comfy and cool
00:24:38.640 meal fashion which is where I bought this number from all right so
00:24:44.760 this is a long line so the IP address is in yellow the red is the attack and
00:24:51.480 we're running rails this is something you're quite likely to see in your logs it's unlikely that this is going to show
00:24:58.200 up in Google analytics or plausible or whatever other JavaScript based analytics your system you're using
00:25:04.440 because it's a bot it's trying not to trigger that stuff but this is literally
00:25:09.659 a botnet that just scanned your site taking down the operating system the framework the host versions of those and
00:25:17.100 we see these IP addresses again and again and again so they don't stop with this they come
00:25:23.340 back with other things but if we could stop this we'd be in much better shape
00:25:29.059 similarly there's a lot of credential stealing this is also hidden for most people for the same reasons but this is
00:25:36.419 trying to take anything all your environment variables your secrets build artifacts and the way that you normally
00:25:42.600 find out about this oh we're starting so this is actual beta this is a traversal
00:25:49.320 attack and so this is just looking through all the different you know paths within your web application you pull
00:25:54.419 this out interesting thing about these is they can even affect static websites um that's sort of almost the classical
00:26:00.539 way these things are leaked which is that someone makes a marketing site it's Jekyll or it's Hugo they accidentally
00:26:06.600 leave the EnV file in there and then it gets pulled so how do people find out about these it's not taking the site
00:26:13.080 down they find out about them because they get a massive build from AWS because people go on they take your
00:26:19.200 credentials they mine crypto I hear moneros very popular um and you know they go on their way
00:26:25.980 it's a research Source utilization thing the other thing that happens is if they can get into your build system because
00:26:31.260 they steal your git credentials or whatever else they will inject JavaScript not a customer mine but
00:26:36.299 Ticketmaster had a massive issue with this where someone was able to add a
00:26:42.000 one-line statement to a Javascript file and Ticketmaster that was essentially a credit card skimmer
00:26:48.299 and the only reason they found out about it was the bank that was doing the clearing of the transactions so like all
00:26:54.480 of these are coming from Ticketmaster informed them they did an audit and they found I think if I'm remembering
00:26:59.940 correctly it was named something like ie6 uh shim you know version one two
00:27:06.120 three kind of thing so in the other place people learn about this stuff is have I
00:27:11.220 been boned which is a website that helps with data breaches a place you don't want to show up if you're a SAS
00:27:16.980 application so they leverage these intrusion attempts of all different kinds into a breach into all sorts of
00:27:24.539 bad things for your site now I want to talk about another exploit
00:27:30.659 that's in this vein that's going to be a little weird and the reason it's weird is that it's for Java so there's a log
00:27:38.820 for J vulnerability I don't know if anyone has seen the string familiar with this
00:27:44.520 this string is death so it's weird to talk about this but this is a dangerous strength so this
00:27:50.760 exploits a vulnerability in log for J which is a Java logging utility and
00:27:56.039 every single site on the internet has basically been tested for this and so what happens is
00:28:01.620 if it's able to be written into the log file of a Java log4j service it runs out
00:28:08.580 to this IP address and downloads the payload the payload is a Java class which gets executed and then they own
00:28:15.960 your machine um so goal here is let's play a game let's play a game if we can get this
00:28:21.059 into your log files so here I've cleverly added at gmail to the dangerous string and I would put this into your
00:28:27.600 email forms would this make it into your logs No Maybe
00:28:33.179 what if did a HTTP get now this one 404 would
00:28:39.299 this make it into your logs Maybe how about this we're gonna do curl and
00:28:44.820 we're going to set the user agent to be the dangerous string easy enough
00:28:50.580 now maybe I made it into your logs maybe it didn't work but we have rail SAS
00:28:58.080 and that's cheap but we're all running rails apps so it doesn't really matter except that the ecosystem of SAS apps is
00:29:04.020 now SAS apps on top of SAS apps so here we have a client it's interacting with the web server
00:29:10.020 and that web service that's your web service has a bunch of SAS apps that you use a logging service transactional
00:29:16.440 email content delivery Network and then each of those also has a bunch of SAS apps they use so your transaction or
00:29:22.919 email service also has a log service and monitoring software and all sorts of things and if you're in a situation where your
00:29:29.460 transactional email services log service happen to be log4j which is non-run
00:29:34.559 reasonable assumption it runs out to the purple bit gets a malware the log service is compromised and what is the
00:29:41.279 number one use for transactional email that's right password resets and so the
00:29:47.940 attackers would have the password reset URL an email address for all of your customers
00:29:53.700 so this is a new thing when a lot of these Services were put together we
00:29:59.580 never really considered this but security is a community activity and
00:30:05.220 across all the different Frameworks and across all of our different work we're doing we we have to think about it that
00:30:11.760 way so this is our weft toolkit these are all the things that a waft provides blocking by geography but all these
00:30:18.240 different things now a problem none of this is in rails none of this
00:30:24.120 comes by default it's all something you have to add in separate and most of the people that use this they're thinking
00:30:29.399 we're too small to need that we don't need it and you know that's where I see
00:30:34.860 people just get blindsided constantly by or you know it's a big company and they have a staging server they're like oh we
00:30:40.200 don't really need it so I think this is due to a brutal resource asymmetry which is that any attacker can truly write a
00:30:47.279 script we did it at the beginning of the talk to contact 10 000 websites throw an exploit into all of them and
00:30:54.240 just see what gets returned or to see whether or not you know they happen to have a bad day that day and checked in
00:31:00.240 their credentials what we don't have is a coherent response to this and when I say we I mean literally the people in
00:31:05.279 this room the people running SAS applications we don't have a way to act collectively there is no tool or service
00:31:11.880 that fits in this Gap we have some existing open source Solutions which a lot of times if you read through them
00:31:18.000 are basically someone being angry at you for not like grapping through your logs aggressively enough
00:31:23.039 not an easy solution and then on the other side you have this the world where I live which is Enterprise life options
00:31:28.440 which are expensive they're enterprisey by definition so how do we change this how do we shift this around to where
00:31:34.919 security ships in every single app well we need a new kind of WAFF and here at
00:31:40.919 rail sass I'm happy to announce that we're making one it's open source it's an in-app laugh
00:31:46.919 and it runs on top of redis so this is the architecture for it so
00:31:53.520 this runs inside your application so there's a Wafers client that works with rails and talks to reddit's and then
00:32:00.360 separately there's an application that talks back to the Reddit server that's
00:32:05.460 the admin so this pulls it out of the context of your main application so it's
00:32:10.860 very easy to implement the client is very thin the rules engine which is what defines a lap is all held in redis so
00:32:18.480 it's very fast and it's designed to be just very lightweight and something you
00:32:24.000 can drop in something that can do automatic intrusion blocking so if a DOT
00:32:29.220 EnV file happens you know to be requested on your site rails will just let that be served
00:32:35.520 we would like that to be stopped this would stop it this is also one of those like career limiting moves to explain
00:32:41.760 this to a non-technical person that yeah like a giant botnet just scanned us and tried to get our credentials but we're
00:32:47.279 fine with it because it didn't actually work today doesn't sit well waffris with blocked
00:32:52.500 inside p and you'd be going on with yourself the other thing we can do because we're based on redis instead of just log files
00:32:59.399 is we can do real reporting not logging and we can do this in real time so in a normal context
00:33:05.820 you have a laugh you have a rail server and then you have a log service and you have to look through the log to see hey
00:33:11.820 what were the IP addresses that were coming in we have to dig through them I showed some of the logs from before and how we pulled bits out well that billion
00:33:19.679 request DDOS attack though I mentioned that would have generated right around
00:33:24.779 500 gigabytes of logs if you've ever tried to deal with 500 gigabytes of log files it is not an easy task so that
00:33:32.460 same amount of information which is really just the IP addresses how often they made requests and when they made
00:33:37.860 them it's about 250 kilobytes inside of redis because it boils down to just the IP
00:33:44.519 address and incremental counter yeah so it's a much more efficient use of space and more than that it's the actual
00:33:51.120 information you need so that you can then go in and so this is a real screen from a real version of Wafers we've had
00:33:57.779 live and running for a couple months and you can go in find the IP address
00:34:04.380 IP address see how many requests it's made and then one click block it so by using the built-in redis data
00:34:10.619 structures we're able to radically simplify all of this to something that's easily used now
00:34:17.460 I think something we've maybe forgotten as a group is that rails really set the tone for a lot of web framework stuff
00:34:23.820 all the things like convention over configuration secure defaults you know all the tooling that we love and my hope
00:34:31.560 is to try to get the same thing to happen for security for rails to be a leader in this
00:34:39.560 all right um so how are we going to do this well part of
00:34:45.300 this is you know leading by example and again we think security is a community
00:34:50.520 activity we need to bring along the other Frameworks with us which is why we're also going to have clients for the
00:34:56.520 other web Frameworks and the wafris admin application is a rails app
00:35:01.980 that is still used to administer this uh we have plans and people in place
00:35:07.140 working on these clients for wafers in case you're doing any other sort of development or you have friends and
00:35:13.320 deliberately designing this to be very easy to drop in to any hosting environment it works it's designed to work with the
00:35:21.599 lowest common denominator redis so a redis maybe it's free maybe it's single digit dollars a month from your provider
00:35:28.619 all that and one last exciting thing which is that bullet train Andrews open source Ruby and rail sass framework is
00:35:35.700 going to include it by default so very happy about that all that being said we still need your help myself Ryan
00:35:43.020 Castillo who's sitting down here on the front co-founder uh we're looking for 10 sites to try this like
00:35:49.079 literally I'm not putting my email up address up there I want you to find me here and tell me you would like to do
00:35:54.119 this now these don't have to be production sites but just any site you have that you'd be interested in running wafferson
00:36:01.079 to try to test it out to help us actually make this happen we also want to talk to anybody who's interested in
00:36:06.720 trying to get the message out of security by default meeting with rails and lastly we just want to hear your
00:36:12.660 security stories I've already heard multiple stories when I've talked about hey here's what I'm giving a talk on people say oh yeah we had the weirdest
00:36:19.920 incident last year when this thing happened we'd love to hear more of that and that's where I'll end it please help
00:36:25.859 me thank you so much
00:36:30.960 foreign
Explore all talks recorded at The Rails SaaS Conference 2022
+4