Search
EuRuKo 2024
Patterns and solutions distilled from 10 years development and maintenance of a big campus software ruby on rails application
Edit
#ruby-on-rails
#background-jobs
#rails-upgrades
#api-documentation
#hotwire
Summarized using AI

Patterns and solutions distilled from 10 years development and maintenance of a big campus software ruby on rails application

Igor Jancev • September 11, 2024 • Sarajevo, Bosnia and Herzegovina • Talk

In this presentation at EuRuKo 2024, Igor Jancev, a senior Ruby developer at the Technical University of Vienna, shares insights gained from a decade of developing and maintaining a substantial Ruby on Rails campus software application, comprising over 380,000 lines of code. The talk highlights the evolution and challenges faced during the application's lifecycle, particularly the migration from Rails 2.3 to Rails 6.1 and the integration of Hotwire Turbo. The following key points are covered:

  • Development History: The application began in 2008, and Jancev joined the project in 2013, primarily focusing on Ruby development despite his previous Java experience.

  • Background Jobs: The speaker discusses the intricacies of background jobs using tools like Resque and Sidekiq. He elaborates on their parallel processing capabilities, specifically addressing the architecture of job management and data exchange with external systems. Jancev emphasizes the importance of a structured approach to migrate jobs iteratively to minimize user disruption.

  • API Documentation: Jancev explains the method for creating comprehensive API documentation that consolidates over 200 API methods across their systems. The solution uses Ruby class methods for documentation, which simplifies the relationship between documentation and code. This API documentation also supports automated testing, enhancing the efficiency of development cycles.

  • CLI Enhancements: The talk covers the implementation of a user-friendly command-line interface utilizing Shopify’s CLI UI gem. This enhancement allows developers to navigate and execute tasks with a more intuitive and visual interface.

  • Migrations and Version Management: Jancev presents a strategy for managing Rails version migrations effectively. He introduces techniques for maintaining a single branch while accommodating different Rails versions, streamlining the migration complexity.

Main Takeaways:

  • Emphasis on starting with the desired end product perspective, ensuring that both the API and user interfaces are built with usability in mind.
  • A focus on simplifying complex codebases, logically structuring job management, and employing efficient migration strategies.
  • Leveraging Ruby’s inherent capabilities to minimize external dependencies, thereby enhancing maintainability and efficiency in development.
  • The importance of gradual transitions in software changes to reduce risks and improve user experience.

Patterns and solutions distilled from 10 years development and maintenance of a big campus software ruby on rails application
Igor Jancev • Sarajevo, Bosnia and Herzegovina • Talk

Date: September 11, 2024
Published: January 13, 2025
Announced: unknown

Did you know that the campus software of the Technical University in Vienna is powered by a >380.000 LOC Ruby on Rails application that was started in 2011?A senior Ruby developer from the team shares patterns and solutions distilled from 10 years of development, maintenance and upgrades.

We all know Ruby and Rails are great tools for startups and fast development of new applications. But did you know that the campus software of the Technical University in Vienna is powered by a >380.000 LOC Ruby on Rails application that was started in 2011 and is actively developed and maintained by a small, dedicated team of Ruby developers ever since?

A senior Ruby developer from this team shares patterns and solutions distilled from more than 10 years of his work on this project, for example how the team gradually migrated the Rails application from Rails 2.3 to Rails 6.1 and Hotwire Turbo, how a big amount of data is kept in sync between the rails applications and many other applications at the university, how to make big changes to the code base without upsetting all users, and much more.

EuRuKo 2024

00:00:07.120 hello everybody thank you very much for the warm welcome so couple of words about me so
00:00:14.599 Carmen already said so my name is Igor yanev I was born here in Saro uh many years ago I had like a a
00:00:24.320 happy uh teenager life until divor started and then I found myself as a
00:00:30.599 refugee in Austria uh so I learned German as fast as I could and I continued my study of
00:00:39.680 um computer science at the Technical University in Vienna finished the studies and since then I work as a
00:00:46.800 software developer uh engineer architect whatever and uh the first like 10 years
00:00:54.079 I worked with as a Java developer and then I became Java architect and
00:01:00.879 then I heard about this new um uh web
00:01:06.760 framework com called Rubi on res and that you can do like a Blog in an afternoon and so I decided to try it out
00:01:15.320 and so I um played with it at home and uh I liked what I see and uh the problem
00:01:22.759 was that um my job as a Java architect somehow didn't uh wasn't so fun anymore
00:01:30.600 because I was just frustrated how uh simple and fast things I can uh do
00:01:37.680 things with rubyan rails and then I go to the job and then I have to fight some
00:01:44.000 XML huge XML files and configurations and everything so so B basically like
00:01:50.680 Ruby destroyed my career as a Java architect because I just F okay I can
00:01:56.840 cannot cannot do it anymore like fulltime so I thought about like wouldn't it be nice if I could like
00:02:05.079 program Ruby and Ruby on Rails like for for for living not just for for fun and
00:02:11.239 for like side projects and at that time it there were not so many so many like
00:02:17.080 um rubian rails project so I did some smaller stuff but then after some time
00:02:24.040 like I heard about um the Technical University in Vienna is searching for
00:02:30.080 developers with Java and Ruby experience and I said okay maybe it's not so bad
00:02:36.120 like could do half half and then I came joined this project and that it turned
00:02:42.680 out that it's like they use Java and Ruby but they need me only for Ruby
00:02:48.040 stuff so it was even better so I could like program um 100% time only Ruby and
00:02:54.280 Ruby on rail so it was like the Dream came true so I could like be paid for
00:03:00.680 like um programming Ruby on Rails and so 10 years fast forward so I'm uh uh
00:03:09.440 actually I did all of this like a freelance uh freelance developer and um
00:03:16.120 two years ago I moved with my wife to Sweden and there we have founded a small
00:03:22.959 company called Mastery bits com and I'm like the software developer in the
00:03:28.000 company my wife is product designer so the creative uh Force so to say and yeah
00:03:35.519 so in case you visit Sweden you could see us maybe at some Midsummer Festival
00:03:43.480 uh with flowers in the in the hair or or you could uh see me in some Park doing
00:03:49.720 my fallong Gong meditation that's what I do for like more than 20 years when I want to like relax and uh it's an
00:03:58.040 ancient Chinese meditation practice like self self-improvement practice um
00:04:04.280 yeah so let's start so today I will talk about um the campus software of the
00:04:12.319 Technical University of Vienna it's from the outside it looks like one application but actually it's one rails
00:04:20.440 rubian rails monolith and like four Java applications and they commun communicate
00:04:27.639 with each other with h GP apis and and
00:04:33.120 this is like the internal part but then then there are like other applications sap and some
00:04:41.039 call managers and uh data warehouse so we have to exchange data with with many
00:04:46.520 other systems as well and I L looked up like um the first commit was made in
00:04:52.880 February 2008 it was rails 2.0 and I joined the project in like 2
00:05:01.520 2013 and uh now we are a team of seven Ruby developers and there are a couple
00:05:08.360 of other teams like Java developers as well so today today I would like to talk
00:05:14.199 I thought what what there are a lot of things I could share in 10 years like work on many things so I choose a couple
00:05:21.600 of things that I think are interesting maybe for other projects as well like uh not very specific uh stuff that uh only
00:05:30.240 only interesting for us so the first one is uh like bed shops so like this a
00:05:37.280 special form from of background jobs um that we have uh like we have like
00:05:43.639 background jobs like sending emails and like some small stuff but we also have some jobs for example for this um data
00:05:51.120 exchange with other systems like sometimes we have to generate some huge XM H XML or Json files or or we have you
00:05:59.680 have to process data like daily multiple times a day like to import uh and
00:06:05.560 synchronize data with s SRP and
00:06:10.599 um many other stuff as well like when some um deadlines for applications and
00:06:17.520 then you have to change some some states so um and so we used uh rescue for many
00:06:24.000 years and now currently we are in the process of migrating to sidekick and I
00:06:29.720 worked on this actually like the I started um like once the migration but
00:06:36.479 then I made a mistake I wanted to do it all at once like to refector the code and migrate all the data and everything
00:06:43.039 and so it was like too complicated but then in the second try like
00:06:48.280 um now we do it in parallel so we have the old system with rescue best B shops
00:06:53.880 and I established the new system with this sides kick and also the new
00:07:00.720 um uh like um base classes and everything and now we are migrating like
00:07:06.840 one job after another so this is like a part of this UI for this new bedops like
00:07:13.680 you have bed shops the categories and some scheduling stuff you can activate deactivate you can start the B shop
00:07:20.840 directly with mouse click and see some progress and if you click like on some
00:07:26.720 details of a Bop then you can see this I call them job runs like uh how long it
00:07:33.080 take is it did it fail or is successful and if I click on details of
00:07:39.319 a job run then I see some some uh like locks that uh were
00:07:46.199 created so it's actually simple like job job run job run log and then in the job
00:07:52.720 the name of the class is saved that is the actual uh application job that is
00:07:59.159 doing the work and uh this is basically how such a
00:08:05.560 bed shop looks like so it's relatively simple for what it does so it's like uh has
00:08:12.960 just a perform method and some logging statement this look like ra logging but
00:08:19.159 actually they log in the database if they started with perform later if they
00:08:24.400 started with perform now they'll just Lo to the um to the rails to the real
00:08:31.840 console so and you can say with total and add you can like deliver information
00:08:38.240 how many items it's uh is processing and where where the job is at the time and
00:08:44.600 then this this is displayed in this progress bar basically
00:08:50.480 and so uh the when I worked on this I thought about what would be the ideal um
00:08:58.079 how do I want a b job to look like so it's uh it hides most of these things
00:09:05.160 like creating job run and uh um like writing to database or whatever the job
00:09:13.160 should look like as simple as possible so it looks like is if it's not using
00:09:18.519 the database or anything at all so that this complexity actually hide it in the
00:09:23.959 in the base classes like application job is the play Base Class for all of the
00:09:29.680 background jobs and uh I often use like this around perform methods like where
00:09:36.720 you can say Okay um before the perform method is called then like the progress
00:09:44.760 bar is refreshed and some start time is saved and exception handling is here and
00:09:51.240 at the end like we save duration and uh refresh and everything so and you see
00:09:57.000 the yield that's the where actually the per form uh met to disc from from the
00:10:02.279 from the job and then we have this bop job that's a subass of application job
00:10:09.360 and uh this is the one that saves to the database that's the GUI that I uh showed
00:10:15.200 you and then here it's again around perform which like takes the scheduling
00:10:21.279 uh and initializes this job run and finishes it or writes the exception logs
00:10:27.760 and and so on so it basically like all of this setup and uh is is in the base
00:10:36.639 classes and when you write bedops you just concentrate on the on the concrete
00:10:41.720 uh um stuff that you want to do and so the other thing is interesting now that
00:10:48.200 we use turbo uh hot fire turbo like before we use some JavaScript for this
00:10:54.040 um uh like progress bar and now um actually there's method like in the base
00:11:00.600 classes refresh GUI progress bar that um like when this progress bar is shown
00:11:05.800 it's only a partial and then when I say at I'm at 20% or 30% it's just broadcast
00:11:13.720 uh this progress to the uh to the page that uh has this progress stream so uh
00:11:21.680 the nice thing about this is uh I can start the job in the rails console for example and it's if you have the Page
00:11:28.519 open you just see the progress running or or multiple people have open the page
00:11:34.880 and um there was a one like heck small heck that I had to do because um like if
00:11:41.560 the job sends these updates very often like maybe it takes only a couple of
00:11:47.120 milliseconds to like um progress one item uh uh then I would bombard the
00:11:54.760 clients with with broadcast messages like very often so by using this rails
00:12:00.040 cach with like six PES in one second you can
00:12:05.440 basically throttle this broadcast to uh it happens only maximum once a second
00:12:11.639 because if the if this provider job ID is it's Unique for all jobs if it's in
00:12:17.760 the cach it doesn't do nothing and if it's not in the cach then it SS send request uh refresh so it's a simple
00:12:26.240 trick to do the throttling of uh these broad broadcast
00:12:32.000 messages so now there is a special form of these um background jobs that we use
00:12:37.639 basic because we have like some uh jobs that process like I don't know tens of
00:12:44.040 thousands of students and do something with them like synchronized data so uh
00:12:49.760 this each of the processing items are independent of of each other so it
00:12:57.320 offers itself to be parall ized so first I had one where I programmed it like
00:13:04.519 thought how could it be done like to have a main job and then to have like a sub job and then to slice the IDS of
00:13:12.880 these students and then give each sub job like a part of it and then like start them with sidekick like each runs
00:13:20.120 in one thread and they do all in parallel and then when they finish like they then the whole job is finished so
00:13:27.480 for this I created this this parallel job processor is it called so that we we
00:13:34.279 need this kind of system in multiple places so it's basically only couple of lines of code like I just uh in create
00:13:43.399 new this parallel job processor with some name uh and then I slice these IDs
00:13:50.399 so usually these are some IDs and then I add like these sub jobs with this uh
00:13:56.759 with this slice and then I set say to this part job processor perform and then
00:14:02.120 I don't have to know how it's like internally implemented so the code for all of these B jobs that use this
00:14:08.839 parallelization is then very simple and it's only how this sub job
00:14:14.639 looks like it just uses some methods like item process so it uh can tell the
00:14:21.279 job process so that it finished this one uh student and so the uh this perform
00:14:29.440 method of this parallel job processor now it basically has a list of these sub chobs what it does is it creates a red
00:14:37.639 um uh unique redish channel name and then it starts each job it gives them
00:14:43.399 this redish Channel and then subscrib subscribes to this reddish Channel and listens to the messages of these sub
00:14:49.959 chops and they could send like log messages or uh say I finished one item
00:14:56.279 or um there's an error or what whatever and then so this uh main job is
00:15:02.399 basically listening to this messages wres to lock so from the outside on our
00:15:08.040 Bop guy you don't see that it's a parallelized job you just seems like a normal job and then when all the jobs
00:15:15.320 complete then just say ready is unsubscribe and it's
00:15:20.600 finished and this like it's simple like these um parallel chobs like they just
00:15:27.199 say ready is publish and there are simple me Json messages uh actually like
00:15:33.399 that only say what type of the message it is like completed or processed or
00:15:40.720 whatever yeah and then there is a special case of this parallel job processor for like sometimes we create
00:15:48.160 these huge files as I said like you don't just process items but you have to create like a big Json file with like
00:15:55.240 all applications or whatever like in this example and so for this uh I have
00:16:01.360 like a parallel file generator that you can tell okay this is the file name I want to have and then these are like the
00:16:09.440 sub chops and then the sub chops create like only a part of the file and then this parallel file generator that like
00:16:16.880 uh merges all these parts into one file and you can specify the limiter and like
00:16:22.360 prefix postfix so you have like um Can configure a little bit the file
00:16:29.240 okay so that was like U our Bop infrastructure that we use uh the next
00:16:35.639 thing I would like to talk is like um how we um create our API documentation
00:16:43.120 because we have as I said we have like one Ruby on Rails monolith and multiple
00:16:48.399 Java applications and we have like more than 200 some API methods and also API
00:16:54.600 methods from these other applications and they have different types and have different diversions and
00:17:01.279 replicating and whatever so uh uh we wanted to have them all in one place and
00:17:09.760 so uh I thought what do I want um like
00:17:14.919 look as an end user like what should the API documentation look like what do I want to have in in this API
00:17:22.360 documentation and from the other side like on the left side how do I want to write this um
00:17:29.480 data for the API documentation so it should be more um near to the code so I
00:17:35.559 don't have to write it somewhere else so in some uh different file so basically I
00:17:42.160 came with with some simple uh rubby class methods like API doc with short
00:17:50.000 description long description so like specifying the parameters and errors
00:17:55.200 that the method can can have and from this information than like uh this API
00:18:02.720 documentation uh with different categories and uh stuff is created with
00:18:08.600 um example well what I wanted to have is like example U URL for this API and also
00:18:18.120 like some statistical data like when when was the last uh uh time this API
00:18:23.799 was called and what's the uh response time and also what makes it easier
00:18:32.400 to work with it is to find um to have like the Ruby controll and and method in
00:18:39.400 the documentation so it's only for administrators so I can just uh copy it
00:18:45.400 and I can go directly to the code so the implementation is actually
00:18:52.000 fairly simple like um I mean it was like implemented it like almost 10 years ago
00:18:58.880 I think or something like that like 2010 15 and uh uh I used uh like two
00:19:07.200 rubby callbacks like inherited and Method edits and class variables and that's actually all like there is um uh
00:19:15.240 model that is extended in this base controller and uh how it works is like
00:19:22.799 um I have um class variable CA last API
00:19:28.120 doc and this is basically a hash of hashes where I save like the information that's
00:19:34.840 provided here like short description response parameters errors and what and so on so because um when Ruby load uh
00:19:43.960 loads the the classes and the methods then it loads them one one by one and
00:19:49.240 then um when when I load a class then I
00:19:54.840 in the inherit then inherited call back is called then I said back to Neil uh
00:20:01.280 some of these uh like category and type and whatever and and then these methods
00:20:07.440 are called like this API do API parameter and so on and these these
00:20:13.400 methods uh basically save the data in this last API
00:20:18.440 dock uh and then when the method edit is called that's uh basically after all of
00:20:25.640 these annotations then comes the method then I just take this data from this
00:20:30.679 last API Dock and save it into some nice uh Ruby uh object like this API method
00:20:37.960 where I have like all the information I need and then I have list uh it's added
00:20:43.440 to the list and then I have a list of these API methods and this is used by the like to generate the HTML uh HTML
00:20:51.480 view so it's actually Rel simple quote and if some uh someone has to add
00:20:58.280 something thing to this or thebag or whatever it's it's easy to find so uh and it works like since 9
00:21:07.600 years so maybe if I would do it today like maybe I would look to have less uh
00:21:14.200 class variables or maybe P it to some something else but it's implementation
00:21:19.760 details what's important for me is like actually how this looks from the outside because this is where most work happens
00:21:27.080 and it should work and how it's implemented inside it could be a little bit nicer but it's uh um it works like
00:21:36.240 this yeah and then we have this base controller where like on every API call
00:21:41.400 this all again this suround action that like saves this duration and saves the
00:21:48.240 information to the database from the request and there is some extra code I
00:21:54.400 uh don't have time to show it now but like uh as you see you don't have to
00:21:59.440 specify here the this URL like the root so the application itself gets from the
00:22:05.799 root information from rails and from the method and the class it can like find out which roote is it so I can show it
00:22:13.320 here in the in the
00:22:18.400 documentation yeah and so nice uh actually features that when you implement it yourself and don't use like
00:22:25.320 some Gem and or generate some XML file and that is like shown externally in
00:22:31.799 this API documentation when you have this data like U it was very easy to integrate this API documentation from
00:22:39.400 java applications for example because they generate something called vadle files web application description
00:22:46.520 language so it's XML file and then basically I had to write a parcel like
00:22:51.880 maybe 780 lines of code pass this data create this API method objects and so I
00:22:58.159 had them in in our view on that that was actually all or for example this API
00:23:04.240 lock it turned out we can use it for automated API tests because if we save
00:23:10.640 the the like where else for each um API that is called then I can can take this
00:23:17.919 I generate a like a file and then I have a test that can take this where else and
00:23:23.240 like do some automated testing yeah so
00:23:29.840 the next thing I would like to talk about is uh like develop a UI
00:23:36.880 um just have to check what okay uh like developer wide uh so we use uh as
00:23:44.880 everybody else probably like we use rake for um like many tasks that are
00:23:50.279 repetitive for one of and uh with rake uh like we have like now more than 160
00:23:57.320 rake tasks and it's um uh sometimes difficult to find what
00:24:03.320 you search for and it loads the whole race environment so it takes a little bit time um uh so uh I discovered
00:24:13.200 actually uh like couple of months ago I discovered a gem from Shopify it's
00:24:18.440 called CLI UI so it helps uh to create
00:24:23.760 like more uh nice um UI for command line
00:24:29.159 tools and it's very simple you can read the documentation so it offers like text
00:24:35.520 formatting progress bars uh like you can run some task in parallel and then you
00:24:41.279 see see it so I created something like that that for for some task that we use
00:24:48.039 more often in in our application like it's called atis from
00:24:53.200 Rubi uh so when you start it like with atis UI then you get like a list of um
00:25:00.200 um tasks that you can perform and then you can like navigate the list with a
00:25:05.960 arrow key so you can print uh say F and then you can filter the list and then
00:25:12.360 when you start it then you can have like this like not a lot of uh output but
00:25:18.360 just to show you like with some icons uh what is it doing right now or use some
00:25:24.080 progress bars for like downloading a file or something like this
00:25:29.120 and uh so I created basically as like um base class for this task for this CLI
00:25:36.720 tasks so one task looks like this like it's you can specify the description and
00:25:43.279 then you have perform method and then you can use some of these uh like helper methods from the base class to to do
00:25:51.080 like um to do this stuff and uh one
00:25:57.000 feature that we also need it because the one thing is with clii do you can do it
00:26:03.360 um um uh you can like uh have interactive um uh like ask
00:26:12.279 the user for some information and then run the code but sometimes you of course
00:26:17.799 need in like continuous integration or in some scripts you just want to run the
00:26:22.919 task with options like we do with with like ra and so I use basically for and
00:26:29.120 this CLI UI for this so to generate like um like there are two task like atis UI
00:26:36.679 for this UI method and Attis exec where I can directly execute the task and so
00:26:43.880 the tour generates basically then the information and if you have some options then you can see the options in the um
00:26:51.799 uh that you can use and uh like uh it was like uh you can specify the options
00:26:59.039 like with a simple um method that and then in the perform method that just ask
00:27:04.559 if the option is here then use it otherwise ask the user so it's like um
00:27:10.679 and this is basically all the code you need for this to to work like have the list of these tasks and then uh like
00:27:18.960 this first answer and then basically just using the list of the task to generate this list of uh options that
00:27:26.279 you can uh invoke and like the second sub command this is this atis exec where
00:27:34.360 it's generating like this um um executable like tasks for for tour so
00:27:42.880 it's not not much cant it adds some uh some
00:27:48.279 user um like it's nice a user interface for
00:27:53.360 for developers as well yes on the last thing I would like to
00:27:58.679 mention uh I didn't have time to to search for there was a blog post like uh
00:28:04.760 many years ago that that I read that you can actually um uh when you do race
00:28:11.919 migrations like it before when we were migrating from La two to race three uh
00:28:18.320 we had a branch and then we had some changes there and different gems and
00:28:23.720 code and sometimes it took months especially like this uh uh Chang from
00:28:29.880 rails 2 to three with the merb and everything so you would have some maybe
00:28:35.000 conflicts and everything and there is actually method where you can um use the
00:28:40.559 same like same branch and run it with two different rails versions so um it
00:28:47.320 works it's very simple you create symbolic link like for example called CH file next uh and then in the G file you
00:28:56.080 have a next method where you ask uh it basically only checks uh if the
00:29:02.799 name of the file is G file. next then it's the next version otherwise it's uh
00:29:09.000 the current version and so when you start with bundle gem file is uh gemile
00:29:15.760 next for example you run real server bundle install then you run it with next
00:29:21.559 is true and so you can in in your gem file you can say if next then use rails
00:29:27.360 like 61 and otherwise Rose ra 52 and for
00:29:33.080 the code we just have another method like in um I think it's boot boot uh RB
00:29:41.480 uh where also ask if this G file is G file next so in case you need different
00:29:47.440 code for the different rails versions then you can like say if next version then do some something else otherwise do
00:29:55.000 this so with in this way you can just say have the same like branch and just
00:30:00.399 start it with rails five and ra six or ra six Rail 7even and it
00:30:05.679 works yes so this is just like uh um
00:30:12.360 uh what we talked about until I what I learned some of the things I learned
00:30:17.679 basically like to do like big migrations uh to establish like the old and the new
00:30:25.120 system in the application and then to move like like B shops for example like one by one so you don't have a stress
00:30:31.840 and you have to do it all at once to start always like with this end product
00:30:37.360 how I want to uh the API to look like from the user perspective the GUI or the
00:30:43.559 API or how would I like to communicate with uh command line tools and then
00:30:50.919 search for ways to implemented in this uh simple API from the outside so to try
00:30:58.639 to hide the complexity in the base classes and try to make this API simple
00:31:04.440 as possible yeah and there is a lot of stuff that Ruby and ruon res offer us like these callbacks and stuff so you
00:31:11.600 don't um have to always reach for some some gems and add more and more
00:31:19.039 dependencies like many of the things like this API documentation is not so much code and it saves that it's it's
00:31:26.159 our code so it's easy to change to to migrate so sometimes you don't need to
00:31:31.559 uh like reach for for some more complicated stuff yeah yeah so that's basically it
00:31:39.159 for today so thank you very much
Igor Jancev
@igorj
Explore all talks recorded at EuRuKo 2024
+39