Summarized using AI

A survey of recent RubyGems CVEs

Samuel Giddins • September 13, 2024 • Sarajevo, Bosnia and Herzegovina • Talk

In this talk titled A Survey of Recent RubyGems CVEs, Samuel Giddins, a security engineer and maintainer of RubyGems, bundler, and rubygems.org, offers insights into the vulnerabilities present in the RubyGems ecosystem, which serves as a vital package manager and gem host for Ruby software.

Key Points Discussed:

- Importance of Security: Giddins emphasizes the necessity of understanding security vulnerabilities in software that developers frequently interact with, particularly RubyGems.

- Understanding CVEs: Common Vulnerabilities and Exposures (CVEs) are identified and catalogued vulnerabilities that can have serious implications for software security. The talk aims to familiarize attendees with how these vulnerabilities can manifest and be mitigated.

- History of Vulnerabilities: Through various examples, Giddins presents a timeline of CVEs affecting RubyGems, highlighting significant vulnerabilities like CVE-2007-0469, which was related to insufficient input validation in file extraction functions.

- Common Vulnerabilities: The session outlines several common weaknesses, including directory traversals, arbitrary YAML deserialization, denial of service issues, and dependency confusion vulnerabilities found in bundler and the RubyGems repository.

- Response and Mitigation Strategies: Giddins shares insights into how the RubyGems team responds to security incidents, mentioning a rigorous process for dealing with vulnerability reports, including triage, fixing, and subsequent disclosure.

- Continuous Improvement: He discusses ongoing efforts to enhance RubyGems security through initiatives like the HackerOne program for responsible disclosure of vulnerabilities and third-party audits. Giddins also emphasizes the support from sponsors and organizations that help facilitate these improvements.

- Future Directions: The talk concludes with a call to action for developers to engage more with the RubyGems security team and adopt better practices in gem publishing to reduce exposure to vulnerabilities.

Conclusions:

The session highlights the evolving nature of security threats in the RubyGems ecosystem and the Ruby community's commitment to addressing these challenges proactively. Giddins encourages developers to contribute to making the ecosystem more secure and supports efforts to automate secure gem publishing.

Overall, Samuel Giddins aims to raise awareness and share knowledge about the ongoing work to secure the RubyGems infrastructure, advocating for a collaborative approach to security in the software development lifecycle.

A survey of recent RubyGems CVEs
Samuel Giddins • Sarajevo, Bosnia and Herzegovina • Talk

Date: September 13, 2024
Published: January 13, 2025
Announced: unknown

RubyGems, like any sufficiently-used piece of software, has its fair share of bugs. Being a package manager (and gem host), many of those bugs turn out to have security implications. Let's take a tour of recent RubyGems RubyGems.org vulnerabilities, and learn how we're keeping the ecosystem safe.

Marshal, insufficient input validation, symlink traversal, oh my! Over the past couple of years, there's been a slow trickle of CVEs announced, covering both RubyGems RubyGems.org. Let's go on a quick tour of those vulnerabilities, covering their lifecycle from discovery to mitigation to announcement. We'll dive into some patterns that have started to emerge, and discuss the steps the RubyGems team is taking to keep the Ruby ecosystem secure in an increasingly adversarial world.

EuRuKo 2024

00:00:10.400 uh so I know that everyone loves to talk about security right after lunch and
00:00:15.759 without a coffee break so if you bear with me I will try and make this as
00:00:21.640 entertaining as possible while still being informative and you know sticking
00:00:26.760 to the things that I said I would talk about so little bit about me uh I am at Essie
00:00:34.760 giddens everywhere on the internet um my name is Samuel and I am a
00:00:41.480 maintainer of ruby gems bundler and rubygems.org and I've been contributing
00:00:46.920 bugs to the aform mention projects for about a
00:00:52.680 decade and uh as mentioned I am currently the security engineer and Resident at Ruby Central so it is my
00:01:00.680 full-time job to make the Ruby packaging ecosystem as secure as
00:01:07.759 possible and it's all of your jobs to make that as difficult for me as
00:01:12.799 possible I guess so some quick goals for today um
00:01:18.000 going to talk a bit about the history of vulnerabilities in a very well tested
00:01:23.159 piece of infrastructure um that being ruby gems sort of try and scare you a bit
00:01:30.240 about the big bad dangerous world out there and uh maybe give you some
00:01:36.479 nightmare fuel that you can send to your boss and say uh you know maybe we should
00:01:43.799 support this work more I don't know if that was too honest I'm sorry we'll
00:01:49.079 we'll just you know give you some history and make this fun so what is a
00:01:58.600 cve um before we get there okay survey of Ruby Jem
00:02:05.200 CVS we've got uh issues with Marshall insufficient input validation
00:02:12.120 simn traversals dot dot dot oh my there's so many
00:02:18.680 vulnerabilities that have cropped up in this ecosystem over the past almost 20
00:02:25.360 years and over the past few years there's been a slow trickle of new cves
00:02:30.879 that the team has announced covering both Ruby jems and rubygems.org and
00:02:36.239 we'll take a quick tour through those vulnerabilities talk a bit about their life cycle and uh see the patterns that
00:02:44.440 have started to emerge and think a bit about what we can do to make things
00:02:49.640 better going forward so what's a
00:02:55.560 cve show of hands first if you've heard the three-letter acronym
00:03:03.680 CBE that's most of you good um anyone brave enough to volunteer what they
00:03:09.040 think it stands for vulnerability something with
00:03:15.599 vulnerability you are
00:03:24.000 correct Okay so we've we've got another word in there we've got common we've got
00:03:34.879 vulnerabilities and exposures great so now I'm going to quote a bit
00:03:41.920 from mit's overly serious website about what cve
00:03:48.920 are what's a cve well the mission of the cve registered Mark program is to
00:03:56.200 identify Define and catalog publicly disclosed cyber security
00:04:02.959 vulnerabilities that was written by someone who is a
00:04:09.280 marketer there are also these things called cve identifiers which is an alpha numeric
00:04:16.120 string that identifies a publicly disclosed vulnerability and sort of colloquially
00:04:22.800 we security people refer to individual vulnerabilities as ACV
00:04:31.960 what is it it's a it's an instance of one or more weaknesses AKA something bad
00:04:38.880 in a product AKA a piece of software that can be exploited so something bad
00:04:47.479 can unintentionally happen causing a negative impact to
00:04:54.039 confidentiality Integrity or availability the negative impact is the thing that makes it
00:05:01.479 bad um and a set of conditions or behaviors that allows the violation of
00:05:09.680 an explicit or implicit security policy so vulnerability is something that
00:05:15.720 breaks a promise about how that product works so that would be promises
00:05:23.680 like we don't expose your password to other people or
00:05:30.639 you know we don't uh we don't let other people pretend to
00:05:36.400 be you we don't let other people upload new rails gems if they're not supposed
00:05:42.680 to stuff like that so some more some more questions uh
00:05:48.199 who here has run Jen G
00:05:53.560 stall I I see some people who didn't put their hands up which is frankly shocking
00:05:59.360 to me me um I want to hear about how you're using Ruby if you've never run gem install similarly have you ever put this
00:06:07.720 line in a gem file okay most of
00:06:14.360 you um so this is sort of why why you should care about this talk about
00:06:22.120 security uh it's because you have used the products that I work on things like
00:06:29.960 Ruby jems bundler and Ruby j.org and any vulnerabilities in those projects they
00:06:37.080 could affect you talk to your doctor and see if
00:06:42.560 security is right for you um to those that don't know in
00:06:49.160 America we love putting advertisements for very expensive medications on TV and they all sound like that it's
00:06:57.879 wild so another shocker ruby gems like any other
00:07:04.199 sufficiently used piece of software has its fair share of bugs I didn't write
00:07:09.960 them of course I would never write a bug someone else wrote them and mysteriously
00:07:15.039 placed them in the software and being a package manager and
00:07:21.639 a gem host many of those bugs turn out to have security
00:07:27.400 implications and so now this overie is out of the way we can dive in and talk about the first ruby gem
00:07:35.919 cve cve identifier cve 2007
00:07:42.160 0469 nice the description of the CV is that the extract files function in install.
00:07:49.759 RB in Ruby jems before Ruby jems version 0.9.1 does not check whether files exist
00:07:57.360 before overwriting them which allows user assisted remote attackers to overwrite arbitrary files cause a denial
00:08:04.080 of service or execute arbitrary code via crafted gem packages AKA really bad
00:08:10.400 stuff I'm curious was anyone here using ruby gems before version
00:08:20.560 0.9.1 no oldtimers okay that is actually surprising to me I wasn't
00:08:27.199 either so this V ability was published in January of
00:08:35.360 2007 17 and a half years ago and it turns out that it manifested a pretty
00:08:42.240 common weakness a vulnerability of directory
00:08:48.000 traversal that was fixed via improved input validation if you've ever read uh
00:08:54.839 patch notes for a security release of software this sort of taxonomy should seem pretty
00:09:01.040 common you know here's here's the weakness the the classification of it
00:09:08.120 and here's how it was fixed and it turns out this is a pretty
00:09:13.680 common one um assumptions about untrusted input AKA random gems can't be
00:09:22.480 trusted and we fix it by not trusting it so in this case we had ruby gems
00:09:30.360 expecting files in a gem to all be under that Gem's directory makes sense you install a gem its files belong to that
00:09:38.160 gem but that means that files in the tarball uh gems are actually just gzip
00:09:46.519 tarballs inside gzip tarballs it's real fun um but ruby gems
00:09:52.959 expected that all the entries in that tarball would say not be absolute paths
00:10:01.079 or not have dot dot in them to ascend the directory
00:10:06.800 structure and it didn't check that assumption when it was unpacking gems so
00:10:11.920 I could give you a gem and if I could guess where it would be installed I could say
00:10:20.200 override literally any file on your system pretty
00:10:26.839 bad as I mentioned this was from from 2007 it was found and fixed
00:10:34.560 and you know the description of the bug and and the fix were all published in
00:10:41.880 2007 um ruby gems and its Associated projects have been around for a while
00:10:48.959 and a lot of the history of its vulnerabilities can be traced the fact
00:10:54.000 that they date to a a more Humane time on the internet
00:11:00.160 um a time when we didn't talk too much about
00:11:05.360 security and we didn't assume that everything was adversarial a time when you would
00:11:11.959 download software off of forums and run it because it looked
00:11:18.760 cool it was a long time ago landscape has kind of changed if we fast forward a few years
00:11:27.000 we can take a look at our worst ever cve who was around for that time when
00:11:34.600 Ruby jems just like was down for multiple days and all work stopped and
00:11:43.079 you couldn't do anything that was that sure was a time
00:11:48.160 wasn't it yeah so what
00:11:53.800 happened 11 and A2 years ago is somebody pushed a jet
00:12:00.120 called exploit to rubygems.org and they were able to get arbitrary
00:12:07.279 remote code execution running they ran a bunch of code that
00:12:13.160 uploaded all of the config files on rubygems.org to paste
00:12:21.040 then this was problematic um funny enough this actually didn't
00:12:28.560 even get it to own cve identifier the root cause was the uh great yaml remote
00:12:35.880 code execution vulnerability um anyone who was deploying a rails app back in January of
00:12:43.079 2013 probably has traumatic memories about
00:12:48.320 this can anyone confirm I'm seeing some
00:12:55.480 nodding and so you know everyone was too busy fixing and verifying that nothing
00:13:01.279 was tampered with to do something like file to get a cve identifier
00:13:07.519 assigned um and part of that involved rebuilding 100% of the rubygems.org
00:13:14.440 infrastructure because who knew what code had been run on the
00:13:21.519 server oops uh this was also back in the day when you didn't have containers and
00:13:29.120 kubernetes and all of that good stuff there was a VPS that had been
00:13:36.120 like hadn't been rebooted for a couple of years that was the
00:13:42.079 server and so there had to be another like the server
00:13:48.079 built in some ways the world has gotten a bit better than it was back
00:13:53.959 then and almost half of that response time and the same for all future
00:14:00.079 rubygems.org cve responses was spent on verifying that we couldn't find any
00:14:07.000 evidence that the vulnerability was actually exploited notice I said we
00:14:12.839 can't find any evidence we can never be sure uh one of the fun things about
00:14:20.079 security vulnerabilities is sometimes they can be used to cover their own
00:14:25.240 tracks and so you uh you sort of have to look for evidence that something
00:14:33.160 happened so how did how did we go about doing that well the rubygems.org server
00:14:40.240 uploaded all gems to an S3 bucket if the server was compromised the
00:14:46.680 S3 bucket could have also been compromised so we had to check the check sums of every file in S3 and make sure
00:14:54.399 it didn't change we to do that there were some trusted mirrors around the world that people were running and we
00:15:00.839 downloaded all of their gems and checks on them and compared them to what was still sitting in the S3 bucket um we
00:15:08.240 also used the check sums stored in the rubygems.org database and in the
00:15:14.279 database dumps that we make publicly available verified that there were no
00:15:20.639 other instances matching the pattern of the vulnerability which in this case was
00:15:25.959 yaml that used some action dispatch uh class
00:15:33.000 to uh execute arbitrary code when the Amo was deserialized which happened because of
00:15:39.959 course gem specs in do gems are
00:15:46.120 yaml the design of the system keeps me employed that's what I'll say so yeah we
00:15:53.440 were verifying that there was no Rogue yamal um in other incidences we've
00:15:59.399 verified that you know no other published versions that are there uh are matching the pattern of malicious
00:16:06.720 activity and also we have to check our failed requests and make sure that none of them match the pattern either but
00:16:13.800 then don't get uploaded because you someone wrote a bug in their exploit or
00:16:20.759 something and it turns out that I have run through this exact same Playbook a dozen times since joining
00:16:28.680 this security team this Playbook has been around for over a decade and it
00:16:34.279 turns out it's not specific to rubygems.org it's pretty common practice
00:16:39.680 for any application that exists to serve
00:16:47.800 content so diving in a bit to the most common
00:16:53.759 weaknesses and I'm going to split these into the you know three different
00:16:59.680 applications in question starting with ruby
00:17:05.400 gems so dating back to literally our first cve there have been
00:17:11.039 vulnerabilities with directory traversal unpacking gems directory traversals just
00:17:16.600 a fancy word for you can write and read files you shouldn't have access to um
00:17:23.839 couple of ways that this has happened we had bad uh
00:17:29.360 tar entry file names things like absolute paths or relative paths with
00:17:35.360 dot dot in them we had people writing Sim links to
00:17:40.559 outside of the gems directory and then writing through the Sim link to you know overwrite
00:17:48.960 files had people putting malicious content in the name version or platform
00:17:55.280 of a gem because those weren't validated separately so if you had a gem named uh
00:18:02.880 say you know slash user uh Slash something well you start writing into
00:18:10.400 that directory as I mentioned Sim link
00:18:16.320 directory traversal I bring this up again because there was a period of like five years where someone would find a
00:18:22.480 bug in our Sim link handling every six months report it we'd fix it we'd say oh
00:18:27.520 thank God we're done with all of these bugs and it would repeat again they've stopped coming
00:18:34.760 fortunately um I won't be so naive as to think that
00:18:39.799 means we've actually fixed all of them um arbitrary yaml
00:18:46.880 deserialization uh has come up a few times
00:18:52.760 um moving on we've had people putting terminal control characters into fields
00:19:00.039 of a gem so that when you say gem list or gem search uh you would get arbitrary
00:19:07.760 output uh in your terminal for a while ruby gems shipped
00:19:13.880 both a gem server and a documentation server and there were cross-site
00:19:19.320 scripting vulnerabilities in both of those which led to us just not shipping
00:19:24.960 them by default anymore and then there's been a whole class of denial of service
00:19:32.480 weaknesses the one that's most in Vogue right now is called redos stands for
00:19:38.960 regular expression denial of service and that's you craft a bad string and you make a regular expression match take
00:19:45.360 forever on it um it's become kind of a game that security researchers I think
00:19:50.720 are just playing for fun at this point similarly uh someone found a a way
00:19:58.360 to make the client sanitize incredibly long strings take
00:20:04.280 approximately forever someone uploaded a summary in their gem spec with uh
00:20:10.840 100,000 characters and it took several minutes to print that summary when you search for the
00:20:16.320 gem and someone also found out that our
00:20:21.520 uh tar reader to read gem files would just blindly trust if you said hey
00:20:28.240 here's here's a file with negative length it would go
00:20:34.600 back continue reading in the tar file would hit the file with negative length
00:20:40.200 would go back and continue reading in the tar file on and on and on and on and on
00:20:48.840 forever bundler also had its own fair share of distinct
00:20:54.840 vulnerabilities um the most common class of those has been what's been termed
00:21:00.240 sourcer dependency confusion and it basically boils down to
00:21:06.000 you download a gem from rubygems.org instead of from your company's gem
00:21:11.640 server or you download a gem that is a
00:21:16.720 typo of the one you are expecting to download I mean who here has ever run
00:21:21.840 gem install
00:21:27.039 rils nice and a lot of this is just caused by a
00:21:32.120 lack of name spacing in ruby gems it's a problem we're trying to
00:21:38.000 fix there's also been a class of shell injection bugs where you could write a innocuous looking gem file and run
00:21:44.960 bundle install and when bundle would bundler would try and get clone your uh
00:21:50.840 your gem it would instead run arbitrary commands
00:22:01.080 and finally there's just been all of the Upstream bugs from ruby gems are issues in bundler as
00:22:08.360 well rubygems.org let's see we've had issues with people
00:22:14.400 overwriting content and poisoning the cash relying on things like
00:22:20.360 uh uh malicious uh hash collisions in like
00:22:25.440 the name version and platform of a gem people bypassing access
00:22:38.279 passwords without MFA everyone's favorite arbitrary yaml
00:22:44.880 deserialization a whole other host of Dos vulnerabilities including of course
00:22:50.120 multiple types of bombs um that's just a term for like
00:22:56.320 self-referential unpacking of file the negative numbers and tar files
00:23:02.679 there so many things how do we handle this you know
00:23:07.880 there's constant security concerns in bundler um and being a package manager
00:23:16.360 that means it falls under the agis of I'm sorry for the buzzwords but this
00:23:22.840 pays for my work software supply chain security
00:23:30.159 and this is something that you know governments are now caring about see for
00:23:36.559 example the US government putting out a paper that says C is
00:23:42.520 bad I'm not going to read through this this whole big quote from the US
00:23:48.720 Department of Homeland Security but it boils down to the fact that our random
00:23:55.080 ruby gems have become as a tech technical term critical commercial
00:24:00.480 software according to the US government how do we deal with
00:24:08.640 that over the past couple of years we've received 430 hacker 1 reports which is
00:24:15.600 our responsible disclosure and Bug Bounty program each one of those reports takes time to
00:24:21.880 triage some of them require a lot of testing we fixed 64 real issues leaving
00:24:29.440 so so many reports that are just
00:24:36.320 useless we've uh actually been working on sort of principles for what it means
00:24:43.159 to run a package repository securely and
00:24:48.480 as part of that we've actually just wrapped up a thirdparty audit of rubygems.org
00:24:54.279 which meant we had to scope that work and stay within a super tight budget and
00:25:00.360 now we're coordinating on how they verify that we've actually fixed the 30
00:25:06.640 something findings that they had and then there's the security
00:25:12.039 attacks that we deal with things like people publishing malicious
00:25:17.480 gems people typos squatting um it turns out if you prefix a gem name with active
00:25:23.640 or action people will trust it dependency SE confusion as we
00:25:29.799 mentioned and data exfiltration uh things like trying to
00:25:34.840 send all of your environment variables to paste bin what's old is new again it turns out
00:25:43.440 and then there's the things that are not quite attacks but manifest as sort of vulnerabilities
00:25:50.600 in the system things like uh me as the on call
00:25:55.760 waking up and asking why is this getting so many 500s right
00:26:01.240 now why am I getting paged hey why is there one IP address making 10,000
00:26:07.120 requests per second um now why is the site
00:26:12.760 down if this is you please stop it and it it turns out oh it's a security
00:26:18.200 researcher is hitting every gem for an endpoint on the whole system and you
00:26:25.120 fixed it added a missing index and asked them to politely
00:26:34.600 stop we're running out of time I'm sorry I uh I'm going to skip over some of the
00:26:41.120 stuff that I've been working on um there's a cool project called Ruby jams research ask me about it and tell you
00:26:49.279 that we run an official security program as I mentioned hacker
00:26:55.240 one hacker one.com Ruby Jem if you find a vulnerability report it to us there
00:27:01.720 please not in public we also have a private security email that goes to our
00:27:07.039 entire security team backed by a 247 on call
00:27:12.399 rotation again please get in touch privately if you suspect there's a vulnerability in bundler rubygems.org or
00:27:18.919 Ruby jems give us a chance to fix it and if you find issues with individual gems
00:27:25.279 don't report those to us report them to the gem author you'd be
00:27:31.760 surprised and every time there's a vulnerability we go through this common life cycle there's a report someone
00:27:39.480 tells us and points out that there's an issue we have to triage the report say
00:27:45.159 is this true is this important is this a vulnerability if so stop the bleeding
00:27:52.360 you know like disable uploading Gems or delete the bad thing that someone
00:27:57.600 uploaded fix the source of the vulnerability verify the
00:28:03.840 fix very important to verify we've really easy to think you fixed a bug if
00:28:09.440 you haven't actually tested it we have to assess the impact see if there were other things
00:28:16.840 that happened we didn't know about backfill the fix in you know old versions after we implemented
00:28:23.279 it then finally disclose what happened why and what we're doing to make sure it
00:28:28.679 doesn't happen again there are also a bunch of big
00:28:34.080 projects that we're working on to improve the overall situation of Ruby
00:28:39.760 packaging ecosystem security um big one's called sigstore won't bore you
00:28:45.120 with it um I've spent a bunch of time talking about it
00:28:50.559 recently all this to say that all this work that I've been talking about costs
00:28:55.760 a bunch of money and it is supportive by our generous sponsors um AWS fastly at data dog and
00:29:04.720 Honeybadger Chief among them it's also supported by donations to Ruby Central
00:29:10.279 chiefly from Shopify the German stf and uh op ssfs Alpha
00:29:17.519 Omega and finally AWS they host the
00:29:22.559 service and pay for me to be here um there's also a big thing I've
00:29:28.600 been working on called trusted publishing that lets you automate publishing gems from CI making it less
00:29:35.039 likely that you will be the source of a vulnerability um and if you'd like I can
00:29:41.799 help you set it up for your gems this afternoon and all of this is made
00:29:48.600 possible by contributions from users like you your companies and
00:29:55.200 security-minded organizations like the stf F op ssf AWS
00:30:02.279 Shopify and more I hope that you have enjoyed this
00:30:07.799 Whirlwind sorry for The Rush at the end tour through uh the history of ruby gem
00:30:17.120 cves thank you and uh come chat about ruby gems
Explore all talks recorded at EuRuKo 2024
+39