00:00:14.920
uh
00:00:29.199
she's on the ball oh they are on the ball excellent job hi everyone uh my name is keith um if
00:00:35.440
the toy gets boring i might just do some more jumping jacks and we'll see how we go keith and
00:00:40.879
mario's guide to continuous deployment um it says keith and mario's guide
00:00:46.800
the keen observer in you would notice that there was only one of me this is mario he could make it he's
00:00:53.680
stuck in melbourne but he couldn't make the conference but you could all send him some
00:00:58.879
heart emojis on twitter i should have put his handle up there but it's uh mario music
00:01:05.280
in the bottom right hand corner so send him some love and it'll be great so tomorrow this
00:01:11.280
is me keith pitt um that's my twitter handle uh
00:01:16.400
that fluffy thing on the left isn't part of my beard that's actually um an animal uh
00:01:25.040
once you get a beard start fascinating more about beats um and start with the other kids
00:01:31.680
this guy his name is harley he's from epic mealtime um i like his beat a lot except the
00:01:37.920
outsides are a bit rough for my liking so i cleaned it up and i think if that's i think that's how i would wear the knee
00:01:44.960
um when i rehearsed this i was under time so i thought we could just look at some pictures of beats together
00:01:51.200
uh this is quite nice this is dana from star trek
00:01:57.920
he can't he can't wear a beard um but this guy can it's a pretty it's a lot of gel in that
00:02:04.560
view if you want to know more i would check out beads.org it's just the forum beads
00:02:12.319
i like eating apple products this picture mean i'm not growing a beer this is what i do
00:02:17.760
also this one and more information about me um
00:02:22.800
i won south australian edition of the year in 2007. um those birds weren't there when i won
00:02:30.000
um that was the one that was photoshopped in afterwards but um that's what i did when i won
00:02:38.000
uh myra and i used to work for a company called envato they make them forest grand canyon a
00:02:43.519
couple of slides like that together we worked on a cycle invader studio it was
00:02:50.319
it helps freelancers sell their digital services online um i work at companions bar and i also
00:02:58.879
made this thing called desktop together it's a site that lets you download desktop wallpapers to a dropbox
00:03:05.680
so as you scroll through wallpapers you're like i like the picture of the cat hit download it'll send you dropbox and then you hit your
00:03:12.400
you change your background thing to point to the dropbox folder and your insta wallpapers it's magic
00:03:20.159
plug i work for buildblocks you know um recently it's having a full time on it
00:03:26.000
it's a continuous integration service but it's a little different all the tooling is hosted the
00:03:31.040
ui is hosted but you bring your own service and you get complete control flexibility security you can run it
00:03:37.280
behind your firewall you don't have to deal with like java or tmc installations
00:03:42.319
and if you go to this link you get two months free um of buildbox good box of already
00:03:47.519
essentially done anyway plug's over let's talk about australia um
00:03:53.120
i am from australia it's a little island off the coast of new zealand we say
00:04:08.959
i'm from the left hand side of perth and this is our central business
00:04:14.080
district um it's kind of what most of perth looks like um
00:04:20.400
plug another plug rails camp perth is happening in november 14th 17th if you'd like to
00:04:26.479
come come find me afterwards uh it's pretty much this is the campsite
00:04:31.600
um uh we're kind of in the middle uh we've got a climbing wall some
00:04:36.800
archery we've got a flying fox snorkeling
00:04:42.000
uh we may have some student tickets available so if you're interested in coming to rails camp perth come find me afterwards and
00:04:47.840
i'll give you some more details so more facts about australia before i get onto what you're actually here to see um uh
00:04:56.479
this this gentleman here uh 1954 was bob hawk he scored 2.5 pints of
00:05:03.680
being in 11 seconds um he later became our prime minister
00:05:12.800
um the first police force in australia was made up of the most well-behaved criminals
00:05:19.360
you're doing less murdering today chief police
00:05:27.440
can walk backwards so they can't pull off the little stance
00:05:32.479
uh the average australian will consume 165 000 eggs in the lifetime
00:05:39.280
it's very interesting i know and uh australian billionaire politician cliff palmer wants to build a new
00:05:46.240
titanic a replica titanic 2.
00:06:03.440
all right everyone's hands are for me everyone
00:06:09.199
thank you so much all right put your hands don't put your hands down yet i didn't say anything
00:06:14.319
thank you put your hands down if you don't write tests it's okay to put your hands down i won't
00:06:19.919
shame you much it's okay put okay put your hands down if you don't have a ci server
00:06:28.800
put your hands down if you don't automatically push your master branch to production on successful builds
00:06:35.600
who's got the hand up ah yeah high five high five the guy next
00:06:41.919
year because he had his hand up state uh no it does not count
00:06:48.400
unfortunately it's just just production um so a few people do it and
00:06:53.840
uh you may have heard the terms continuous delivery continuous deployment they are different and it's easy to get
00:06:59.919
confused between the two in continuous delivery testing is automated but the deploy to production
00:07:05.360
is manual in continuous deployment the entire pipeline is automated including the deploy to production
00:07:12.639
so you might be asking yourself what is this continuous deployment voodoo you speak of keith and why would
00:07:17.680
i want to do it well because of yolo
00:07:24.080
well well not really yellow the process is pretty yellow
00:07:29.199
but the code is not i found when i was doing continuous deployment i would be writing some code and i would
00:07:36.800
knew that in about 10 minutes time this code would be on production so i really changed how i approached
00:07:42.000
particular problems i was much more careful in what i did i was much more thorough in what i was trying to achieve uh i wasn't
00:07:49.360
just throwing some code out there and like whoever deploys that next week it's going to be their problem to deal with
00:07:54.479
because i know it's not going to work deployments are tied to my name so keith pushed this it went to production
00:08:01.039
and now he's deleted all the users so i tend to be much more careful in the way that i approach problems um there are three
00:08:09.120
three qualities of continuous deployment that i say are important one of them is fast feedback we want
00:08:14.800
deploys to be fast in this new world of a b testing and
00:08:19.840
those other things that require fast deployments um you want you want you want fast feedback from your production environment
00:08:26.960
so doing continuous deployment makes that work because your entire process is automated you
00:08:32.159
commit you push time passes production and there's really no human involvement
00:08:38.399
you're much more confident in your deploys because these things are automated you've written scripts to handle the entire deployment process
00:08:45.279
migrations are handled for you testing is handled for you and if you trust the tests then you
00:08:50.320
should be able to deploy the production off of master straight away and it i used to do a lot of manual deploys and
00:08:56.720
manual deploys now scan me because i forget i'm going to like have i tagged this thing have i not tagged it what's
00:09:02.640
the tagging structure like have i committed this thing to staging right have i run the smoke tests i've stuffed
00:09:07.760
up a couple of deploys in my time because i'm trying to rush something out and i just forgot to do something so it gives me the confidence
00:09:14.240
and it's scalable if you've got if you're doing maybe one to two deploys to master a day you can scale up to
00:09:20.080
maybe 30 30 deploys a day quite easily um and when you commit to master and the deploy goes out
00:09:26.560
you end up doing much more deploys anyway so it scales really really well um
00:09:32.160
i like venn diagrams let's turn this into a venn diagram great it doesn't really need to be a venn
00:09:37.200
diagram but i thought it would could be one if it was good um another thing there in the middle so
00:09:44.080
um everyone take a photo of this show it's your boss they'll be instantly
00:09:49.440
convinced that this is a good idea um
00:09:54.480
everyone's got photos great uh okay so continuous deployment you as the
00:10:00.160
famous saying goes you can't do continuous deployment without breaking a few omelets so let's talk about what we're going to
00:10:05.680
need for a continuous deployment all these things there's many of them but they're all
00:10:11.360
very very easy let's get into the first one which is feature planning okay monday you come to work your goal
00:10:18.240
is to make this new feature you should talk with your team about how you're going to uh write this feature
00:10:23.600
and deploy this feature all in the same conversation uh you talk about what's the smallest thing that you can ship today let's say
00:10:29.839
this feature is quite large try and break the feature into much smaller features so and ones that you can ship today add
00:10:36.000
this link here boom add this thing here boom boom boom and by end of the week you've done 10 deploys and the feature
00:10:41.279
is complete as what the the customer originally wanted but you've got lots of smaller deploys
00:10:47.600
and small increments in the code base can it be deployed with zero downtime
00:10:52.880
maybe you're adding a new piece of infrastructure to your production servers maybe adding redis or changing databases or something
00:10:59.360
uh you should probably talk about if uh does this feature that we're about to embark on have any implications to zero
00:11:04.480
downtime and we'll learn more about what zero time time is in a few slides time and uh uh
00:11:12.640
how are we to handle database migrations uh if anyone here has tried continuous deployment before
00:11:18.000
you know that database migrations are a pain in the bum so we're going to learn about how to deal with those later on and you just
00:11:24.480
need to talk about them during those conversations so coder views this is a picture of bill gates using
00:11:31.680
the 1980s version of github he's just looking at looking up just looking at paper
00:11:39.120
coderview is really important to continuous deployment and uh github plays a big part of that the pull
00:11:44.800
request system is completely amazing it fundamentally changed how we do code reviews
00:11:49.839
um when mario would do a feature on desktop he'll write some code submit a pull request i'll plus one
00:11:55.920
i'll hit merge when it sorry he hits merge when he's happy and i've plus one um goes to master master goes production
00:12:02.160
boom it's very very simple um but sometimes uh oh actually this is a picture of a pull request
00:12:07.839
of someone adding a flux capacitor to a project
00:12:13.040
yeah that happened um but sometimes pr's can get a little off topic when you're talking about white space or quotes or like hash syntax so there are
00:12:20.240
a couple of gems to kind of just make those decisions for you one of them is kane by square
00:12:25.600
um it does some static analysis it looks at line length and it looks at the abc complexity
00:12:31.279
of your project and will error if it doesn't look quite right and uh this one here rubocop is a
00:12:38.079
robust ruby code and analyzer based on the community ruby star guide it makes sure that using the right hash
00:12:43.920
syntax and you're not doing anything crazy looking um and those two gems will just make the
00:12:49.279
ci process a little bit easier for you beach toggles one of my favorite parts of continuous deployment
00:12:56.079
the idea behind a feature toggle is just a mechanism to flip a feature in production on or off
00:13:02.399
without doing a deploy now how you really do that is up to you could be a database table could be radius it could be environment variables
00:13:07.760
it doesn't really matter as long as there's no real deploy to production involved i like to think about feature toggles as
00:13:15.200
just being really ghetto let's say there's a feature that i want to implement and to get to that feature
00:13:20.480
you have to go through some button or some link i would generally just hide the button in the feature tool i won't really
00:13:25.839
bother about hiding the page or anything like that if the user finds the page they found a
00:13:31.120
broken feature that's great it's nice little surprise treasure for them um a nice broken feature
00:13:37.200
fitlife made this gem called rollout which works with redis that's how he does feature flippers this
00:13:42.800
is the one that i use personally it's very very simple you can define groups of users so i'm going to roll this feature out to
00:13:49.040
developers only or you know this group of users only the other one is uh could flip by pda
00:13:56.000
it's a bit more enterprise version it's a lot more features and it's got a web ui that allow you to control stuff
00:14:01.120
with rollout you have to go into the production console and do it manually automated tests
00:14:09.040
automated deploys need automated tests that's just kind of the way it goes mario and i write just enough tests to
00:14:16.720
make sure that we trust what we're going to do um there was a fat a while ago to like
00:14:22.240
test everything we don't really test everything we test just what we trust like we're not sure about this let's test this
00:14:28.560
just because we've embraced yellow in in you know the way that we do our our
00:14:34.000
processes and not writing not writing tests for everything is a part of that um yeah some uh we also
00:14:42.720
don't write all the tests so we can keep the test fast now i know you're wondering yes i just said
00:14:48.959
i write less tests to make them fast yeah i did but there are other ways to
00:14:55.279
make your test fast aside from just not writing them corey haynes did a talk on fast rails
00:15:01.120
tests uh you can watch that you just google fast test by corey haynes and he'll show you how to write them fast
00:15:07.920
another type of test is called a smoke test now the idea behind a smoke test is that it runs on your production
00:15:14.959
server after a deploy so if you're if you're doing a deploy uh manually you
00:15:22.240
type cap deploy you'll watch the output stream by and then when it's finished you'll generally go to production hit refresh
00:15:28.000
great that's all working the smoke test automates that particular process for you
00:15:33.199
it can be pretty easy it can start by just checking to make sure that the name of your product is on the home page
00:15:38.720
it doesn't have to be too difficult just something to hit the page or hit your key flows like maybe it'll go purchase something with a
00:15:44.880
with a real credit card or maybe it'll i don't know sign up um just something to make sure that your main flows in
00:15:50.399
production are working uh the way that we use that do that is by using capybara um capibara makes this
00:15:55.519
really easy because you just give copybar an endpoint and you start you know like click this button there's this link
00:16:01.600
there that sort of thing um it's even better if you can run those same tests locally against your
00:16:06.800
development machine and on production so if you're just checking to make sure the name of your
00:16:12.320
product is there you can run that run this run these tests locally point them at your local environment
00:16:19.199
it'll make sure that they run and then when we put this script as part of our actual deployment process it'll also
00:16:24.720
check to make sure that it's on production as well after a deploy zero downtime deploys now these are
00:16:31.040
probably this is one of the more harder bits of continuous deployment it's just the act of deploying code to production
00:16:36.639
without a maintenance page um because you because uh we're committing to master a lot more
00:16:42.639
um every commit to master will cause a deploy to production that will just mean more deploys to production um and if you've got quite a
00:16:49.680
bit of traffic seeing this all the time is going to be annoying for your users so to point to production without the
00:16:56.639
users noticing that a deploy ever happened is really important um now if you're on heroku
00:17:03.120
you'll notice that after a deploy the next request is quite slow uh there's a way to fix that using
00:17:08.160
heroku labs preboot thing if you enable this option uh your users won't even know that a deploy happened
00:17:13.760
if you don't turn on maintenance so that's quite a useful thing to to know um if you wanna know how to do zero
00:17:19.199
downtime deploys with unicorn there's a great railscast about it um just go to rousecast and search for zero
00:17:24.400
downtime deployments and uh it'll show you how to do with unicorn
00:17:29.679
now database migrations this is the hardest part of continuous deployment um
00:17:36.320
changing the database while the code is still running is very very very tricky
00:17:41.360
and and there are really two ways to do it the easy way is to use a maintenance page like i told
00:17:47.039
you not to do just before um so the flow would be write some code
00:17:52.160
push the master in your deployment script it would turn on the maintenance page
00:17:57.760
then do the actual deploy to production then turn off the maintenance page that's the easy way to do handle
00:18:04.080
database migrations in a zero downtime process but the hard way is by doing two
00:18:10.320
deploys and you'll know in a second why it's so hard so
00:18:15.360
if let's say you're adding a column this is one of the one of the more common ways of sorry one of the more common things
00:18:20.720
you're doing in a migration you add a column you start off by deploying the migration add a column
00:18:25.919
great and then after that you deploy the code to use the new column that's how you would do at a column in a
00:18:32.640
zero down time deployment process now removing a column
00:18:37.760
is actually very very tricky so you would have to first remove all the code if you want to remove a column you have to remove all
00:18:43.760
the code that uses that column first great secondly you include some dirty dirty hacks
00:18:49.200
and now you're about to find out why database migrations are very tricky
00:18:54.640
this is me removing the notes column from the users table this is pretty innocent i deploy this
00:19:01.280
and i start seeing this everywhere even though i'm not using the column anymore like why is this thing still inserting
00:19:06.400
into the nodes column i do not understand it's because of active record when
00:19:11.679
activerecord boots it actually looks at all the columns of a table and
00:19:17.760
then has caches them internally and when you insert a record it will use those columns if
00:19:23.919
you don't provide a value for one of those columns it'll just use nil so we've we've removed the column from
00:19:29.760
the database we haven't restarted our unicorn workers we haven't restarted rails it still thinks there's a notes column
00:19:36.000
and we're going to get this error like we see now this is what makes zero down deployments very very very tricky
00:19:42.000
and so the way we solve that is just by hacking active record we just get it to ignore the notes column so
00:19:48.880
the process of removing a column with a zero downtime migration is remove the code that uses the column
00:19:54.480
include the dirty hacks deploy that code to production with the hacks
00:19:59.600
deploy the migration that then removes the column and then remove the dirty hacks that's that's a real pain but that's the
00:20:06.640
way that you remove a column with zero down time um there's a great article by pedro
00:20:12.640
pedro bello and he'll uh you know show you how to uh it goes through every single migration type and it will show you how
00:20:19.600
to do it with zero downtime also be aware of database locking if you add an index to a table in mysql
00:20:26.960
there is a good chance that the table will lock and if you're adding a table to your users table or your sales table and
00:20:33.600
those tables lock then in production no one's going to be able to write to those tables for the period that it takes to
00:20:39.280
run like at the index so if your adding index takes 10 10 minutes if you've got a lot of data
00:20:44.559
then you've just completely broken signups and purchases for 10 minutes you can use the large hadron migrator
00:20:50.640
gem from soundcloud that will show you how to do that that will show you how to set that all
00:20:56.240
up now in production stuff sometimes breaks
00:21:04.320
and so you re rolling that back deploys is a very important part of continuous
00:21:09.440
deployment um stuff catches on fire occasionally service catch on fire you know people's
00:21:16.320
wallets catch on fire stuff event i'm sorry aaron should i do it again
00:21:21.600
stuff catches on fire sometimes service touches on fire sometimes
00:21:29.679
and so what i do when stuff catches on fire is uh i generally just try and roll
00:21:35.760
forward um i don't freak out too much about it because i know i'm going to make a mistake eventually so
00:21:41.200
uh what i tend to do is i am the first thing i do is just turn the
00:21:47.280
feature off because we've used a feature flag it's a very very small deploy if we've introduced a bug into this feature i'm
00:21:52.880
like oh crap this is not what we wanted i'll just turn the feature off go back to the code base fix it deploy again
00:21:58.799
turn the feature on and hope that it works another way to do it is just to get revert and then get revert commit to master
00:22:05.760
automatically deploys to production again you've essentially just rolled back a commit
00:22:10.799
github made this really easy recently with the reverb button on proquest i'm not sure if you've seen that but you can
00:22:16.000
merge a pull request and revert that particular merge commit in the ui so you can roll back a deploy um from
00:22:23.520
github and the great thing about github is that fuse pull request the merge pull request button
00:22:28.640
actually becomes your deploy to production button it's very very very cool um and in the bag rules worst case
00:22:33.919
scenario you can always just turn on the maintenance page um even though i'm telling you not to do it
00:22:40.080
you can always do it if you really screw up a deploy um so you can always just do that now
00:22:45.520
monitoring is really really important uh you want to know someone's looking at your stuff when you're not and so um exception tracking is is very very
00:22:52.400
important i'm assuming that you're doing that already um uh performance monitoring with new relic is oh sorry
00:22:58.159
this is bug snag this is my favorite bug tracker i don't worry for them i already know them but i just like it a lot um performance range is very very good
00:23:05.120
with new relic and um you can also do business magic monitoring as well
00:23:10.480
um so instead of looking at how fast a request takes you can start looking at things like sales per minute
00:23:15.600
um and those are very interesting stats to monitor so if you do a deploy and there's a change in your sales or
00:23:21.039
signups maybe they've dropped by five percent after this deploy you may have introduced a bug and not know it um
00:23:27.679
and pages judy makes it really easy to get notified you can plug all those services into pagerduty so if you're so you do a deploy so you
00:23:34.799
commit to master automatically is deployed to production you go to lunch and then you get a phone call because you've just introduced a
00:23:40.720
bug um it's it's pretty powerful i think but bunch would be kind of stuffed um so these are all the
00:23:47.360
ingredients we need for continuous deployment those are all the things we're going to need so let's mix all these ingredients and i'll tell you a story about how i
00:23:53.600
use continuous deployment to make a feature this is this is my current deploy to production
00:24:00.000
script um i don't do anything i just push to master on heroku this is
00:24:05.039
probably the most yolo way of doing continuous deployment i wouldn't recommend doing this but this is really good for raz rumble
00:24:12.880
next thing we're going to add to our deployment script is running the quality checks using kane and rivercock we're gonna sorry
00:24:18.799
robocop we're gonna run our tests we're gonna push to production and we're gonna run our smoke tests that i told you about before
00:24:26.720
now this is even more safer but to make our deployment script even better we're going to push the
00:24:31.919
staging first so we run the quality tests we run our actual unit tests
00:24:37.360
we push the staging we run our smoke test on staging we push the production and run our smoke tests on production
00:24:43.200
and i think it's a pretty good deploy to production script and if we mix these and if we take that deploy script take all our ingredients
00:24:50.080
and uh i'll tell you the story this is me at a conference i had a great idea uh
00:24:56.240
social links social buttons they're very popular these days i wanted to add them to my side project
00:25:02.960
desktoper so i did some code locally this is what i come up with
00:25:08.240
um we've got twitter we've got google we've got facebook and we've got two linkedin buttons
00:25:13.520
just for that extra extra business wallpaper sharing um those recruiters need wallpapers too
00:25:21.679
uh so i was pretty happy with this particular feature it was great i pushed it to a branch and then created
00:25:27.120
a pull request this is mario he's very busy working or reading a book of ponies
00:25:33.039
in the bottom um he's he's pretending to work but i know
00:25:38.320
what he's doing um i submit the pull request uh i usually include a gif with all my pro requests
00:25:44.320
it's part of continuous deployment that gifts are included with all pull requests um so that's that's the gift that i
00:25:49.919
chose um myra was pretty happy with it he tested it locally and i'm sure that it worked fine
00:25:55.039
so
00:26:00.640
ignore that that was a separate project i'm sure so he's happy with it i'm happy with it
00:26:06.960
he's plus one the pull request i hit the merge pull request button which is now now which is
00:26:12.320
our new deploy to production button um this script runs and um
00:26:18.720
plug buildbox would say maybe do it for you um it would do the specs deploy the
00:26:24.400
staging staging smoke test production production smoke tests and if all those things are green
00:26:29.600
then desktop it goes to production and nothing looks any different no one knows i've deployed no one knows
00:26:35.919
anything's changed because i use feature flags i turn on the feature flag for developers only
00:26:42.159
with rollout you can scope to particular groups of users i've scripted to developers only
00:26:47.200
i then turn it on this looks great it's exactly what i expected i'm happy with the feature now i'm going
00:26:53.520
to then turn it on for all of the users and then boom we've both achieved continuous deployment
00:26:59.360
and that's that's a nice gear for you
00:27:07.360
yeah one more one more time wait
00:27:12.400
you know you know how hard it is to do it's it's actually very tricky because you need to look at the camera and do it
00:27:18.640
anyway so you're welcome um uh so we've uh we've got our three
00:27:25.360
things we've got fast feedback confidence scalability have we achieved them yes we have
00:27:31.600
we've got the the fast the fast tests we've got you can actually make your deploys much faster by the way by only doing things
00:27:38.080
that have changed one thing i didn't cover but i i didn't have time to was in your deploy process it's generally
00:27:44.880
push code to production bundle run migrations and then restart
00:27:50.080
if you haven't changed or you've also got asset compilation you can actually change your deploy script so only do the asset compilation
00:27:57.919
if the assets have changed only bundle if bundle has changed um the deploy script for
00:28:03.120
one of our side projects takes a second to get code to production just because it doesn't really do anything and ssh into production and get pulls
00:28:09.360
and just restarts unicorn and those faster points are very very cool so we've got that we've got the
00:28:14.399
confidence because everything is automated all we've got scripts that handle everything we've got monitoring in place
00:28:20.000
and the solution is scalable um because it's scripted we don't have to worry about it we doing 10 manual deploys a day is going
00:28:26.080
to get really annoying really really quickly and because this thing's automated it's really really cool
00:28:33.200
i have some science to show you uh we asked a team that does continuous deployment to
00:28:39.360
give us a get log of the deployment log for a two-week period these are the same size
00:28:44.799
teams same size project and this is the results that we got we found that the team that does
00:28:51.120
continuous deploys actually does five uh 4.7 deploys per
00:28:56.399
day and the other team only does 1.9 and interestingly enough the changes per
00:29:03.279
deploy were much lower on the other team than it was for the manual deploy team so
00:29:08.320
uh there's some more science for you to show your manager so that's kind of it that's
00:29:14.640
continuous deployment um you should always be continuously deploying uh that's so kind of minor mario's motto
00:29:22.320
now so you should should do that you should always be shipping is another motto but
00:29:27.360
it's not as good as this one so that's it continuous deployment keith mario's guide thank you very much
00:29:40.640
thanks keith for the great presentation um any questions
00:29:45.679
well it's because i answered everything that we're going to ask
00:29:50.799
um if not let me try this i have a question ah sure um so uh when when you were starting to
00:29:58.799
do this uh continuous deployment process it was you know i assume a uh incremental process
00:30:04.640
um did you face any challenges implementing this like were there a lot of things which started breaking when you when you first tried to do that
00:30:11.520
do you have to change your workflow in order to accommodate these changes yeah absolutely so in this particular
00:30:16.559
project where we tried this on we actually started on a greenfield project and we decided to do continuous deployment out
00:30:22.000
of the box um the biggest problem i i would say we had was actually heroku um because we're using heroku um it's
00:30:28.960
really hard to run migrations on heroku the migration needs to exist on heroku
00:30:34.320
before you can run it um and so we had to do two deploys for everything which is which was a real
00:30:39.440
pain um the way that we solved that was actually running i shouldn't be telling you this but i
00:30:44.559
will um we actually downloaded the database url from heroku set it locally ran the
00:30:50.480
migrations from our machines and putting them at production so we didn't have to deploy the
00:30:55.600
migration but there was a lot of push back from people in the team as well because they haven't
00:31:00.799
really done this sort of thing before but if you ask anyone in the team now uh i'm sure they'll swear by it and manual
00:31:07.039
deploys scare them so uh that was yeah the biggest problem was probably heroku how do you coordinate uh infrastructure
00:31:14.159
changes like do you have if you're on heroku i guess not but maybe you have experience with teams that are using chef or something like
00:31:20.240
that um how do you coordinate with ops if you need new infrastructure for a new feature the developers are rolling
00:31:26.720
out yeah so um this is the process that we use for like most of our commits like maybe typo
00:31:33.440
change maybe we're going to add a new form but you've got staging there's nothing saying that this week we're going to do manual deploys
00:31:38.720
um you've got these scripts you can run them whenever you like a couple of times we said that we're going to turn off auto deploys
00:31:44.480
this week while we work on this big infrastructure thing um so like it's it's this is just a tool you can use it
00:31:50.000
however you like um but uh no i haven't had any real experience with that sort of thing because we're on heroku
00:31:55.440
um but i'm sure it can be have you ever come done a case where
00:32:00.559
you've had a racing deployment and since somebody commits a branch it's deploying when somebody else
00:32:07.039
merges another pull request ah so yeah so there's two deploys happening at once
00:32:12.399
um yeah we um at the time we're using travis ci and i think we only had one worker so
00:32:17.440
any one bill could run it once so that that's how we solve that problem was it the fact that we're going to do one deploy at a time
00:32:22.960
um but i'm sure there are other ways to do it with your with jenkins and team city i
00:32:29.600
think you can configure your pipeline steps to be only one of these can one at one time um but yeah you just only if you split
00:32:36.720
up the pipeline to multiple steps you can just control when that step gets run um but in that script that i show you you would definitely have that problem
00:32:43.039
if um two deploys run at once i'm not sure how much of that a problem that is on heroku though
00:32:50.799
hello after all of this what do you find to be the most fragile or continuously fragile part of the process
00:33:00.159
that's a really interesting question all the fragile bits we ironed out migrations was tough but then we figured
00:33:07.039
out how to deal with them and so they weren't really fragile anymore the fragility would me would be doing it
00:33:12.159
manually again because i'd have to look at the script and be like okay so we tag it at this point
00:33:17.360
and we deploy this tag to this one and there's that tag particular naming scheme but there's really nothing
00:33:23.440
fragile about it like once you've got those tests in place and um all those checks then it's really really simple
00:33:29.440
um and you know uh manual deploys scare me much more and
00:33:35.440
so not much sorry i'm just saying the same things over and over again but yeah
00:33:44.159
cool we've done no other questions thank you keith thank you very much
00:34:04.840
so
00:34:14.399
you
