00:00:19.529
morning so I got it at five o'clock yesterday morning from New York it's a
00:00:26.589
33 hours door-to-door I need you guys to be aware good morning thank you that's
00:00:34.930
for my benefit okay cool so my name is Brian and today we are going to talk about shipping ruby apps with docker I
00:00:42.429
didn't start a company called code by that I mean both heard of code mining here oh wow that's awesome okay cool coming
00:00:49.809
to use two codes mine will work on filling the gap between the first hands in the second hands I come see me
00:00:56.769
afterwards we'll talk about that okay so we're going to talk about docker just to
00:01:01.899
kind of gauge what people's familiarity is and I can kind of try to tune how detail but I get who's heard of docker
00:01:09.330
okay almost everybody how many people have installed it and played around with it like either
00:01:14.560
locally or somewhere I would say that's about third and how many of you are running it into production like five
00:01:21.910
okay cool so I'm really excited about docker they were really really excited that docker I
00:01:28.270
think docker is going to be something that all of you are using within the
00:01:33.400
next few years and so my goal in this presentation is to try to kind of paint
00:01:38.770
the picture of why that is and get you excited about it too and we'll look at kind of what the
00:01:44.200
options are for getting started with it today so docker is a generic way to run
00:01:51.040
any service anywhere where the enemy is as a little bit of an expert
00:01:56.950
it means anyone service so it won't work for say Mac or Windows services but if you're deploying
00:02:03.490
on the Linux which I imagine many of you are you can use docker and it's a generic way to ship things around and
00:02:10.360
that's where the name comes from its with shipping metaphor so if you think of like cargo containers doctor is based
00:02:16.239
on that metaphor and you put your application into a cargo container then you throw it to the ops team and they
00:02:21.670
can run it anywhere so it's sort of two big components its container based virtualization combined with a generic
00:02:28.750
package for that thing you can use for any service will get it to you exactly what that means in just a little bit so
00:02:36.810
container based virtualization is kind of what makes docker are awesome and this is the big DevOps idea of the last
00:02:46.329
year or two and I think going to be the big DevOps idea for the next couple years it kind of snuck off on a lot of
00:02:54.040
people myself included so a year and a half ago I didn't really know anything about what container based
00:03:00.069
virtualization was but the doctor folks through a conference a couple weeks ago
00:03:07.000
and they got a bunch of presentations from people who are using container based virtualization and apparently
00:03:12.670
everybody who knows a lot of stuff of that office has been using us for a while so all of the Google runs on
00:03:17.950
container based virtualization pretty much all of Twitter on Facebook so the
00:03:23.049
guys that are dealing with DevOps at scale are pretty far along in terms of
00:03:29.079
migrating stuff to container base virtualization I think that's one of the big reasons why it's going to become so
00:03:35.470
prevalent and it scales up and down you don't need to be running nearly as many servers as Google for this to be
00:03:40.840
valuable we're gonna look at how it can be useful even if you have only wants to protect or if you're just developing on
00:03:46.690
your laptop today but you know that if you use approach like this it does scale
00:03:51.730
up I mean that's the big as it gets right so you can start now and then move up with it so there is a project that's
00:03:59.349
been around for five or so years called LXC just stands for Linux containers and
00:04:06.160
this was kind of hub docker got its start for a while it was a to a wrapper
00:04:12.580
a higher level easier to use layer on top of LX e and LX e was doing the hard
00:04:20.650
stuff under the hood I like to see itself was both other layers but docker was sort of the user interface for how I
00:04:27.190
see it's see-through on steroids so it
00:04:33.430
means you get a share kernel and isolated resources so you can only be running one version of the Linux kernel
00:04:39.970
right when you're using container based virtualization so there's no like okay I'm gonna like this kernel but these
00:04:45.310
kernel modules over here and then I'm gonna do something completely different out right here you can do that with a
00:04:51.250
full vm sort of system that a lot of people run on things like AWS right on
00:04:56.530
AWS we take your kernel you can't do that we have container based virtualization but it turns out that if
00:05:02.770
you're using a modern kernel that's fine for almost everything right you don't really I mean me personally I don't
00:05:09.130
really want to be picking what I am not smart enough to actually do that so I just want a good kernel that works and
00:05:14.950
then deal with things that are a little bit higher level in them and then the resources when you're using container
00:05:21.430
based virtualization like LXE or dr isolating so you get a sandbox for
00:05:27.250
resources like kids networking and files so the root of this is like the CH we
00:05:33.760
part the root on your container won't is not the root or host operating system and the pins
00:05:40.479
are different right so we hid one within your container it's not anything to do
00:05:45.520
with k1 on the underlying closed operating system and then you can you can do all sorts of fancy things with
00:05:51.880
the way you configure the networking for each container you can give it access to the hosts networking through a network
00:05:57.970
bridge you can say I think it's doesn't have any networking at all or do you port that Bank and you can also do
00:06:05.139
resource limits so you can assign CPU shares and memory shares to your containers to control how they work when
00:06:10.479
they're competing for resources docker is from a company that is called used to
00:06:16.150
be called the doc cloud and they kind of had a platform as a service that was a
00:06:22.000
key Heroku and they were running this for awhile and the conclusion that they came to was well you know I don't know
00:06:29.050
we're never gonna be more popular than Heroku ever but they sort of realized that the technology they developed under
00:06:35.440
the hood in order to support running that was more valuable than the service they were providing themselves so they
00:06:41.650
made a pretty drastic stack and open sourced this project docker which is the
00:06:47.470
next evolution of their underpinnings of the platform as a service and it's all under either MIT or MIT or BSD license
00:06:55.780
very permissive licensing and gave it away and said okay well we're gonna try to be instead is become a unified
00:07:03.219
standard for how people use container based virtualization if we can do that we'll figure out how to make make a lot
00:07:10.810
of money doing it so then rebranded the company to darker ink the open source it's on go and it consists of two
00:07:18.370
primary components release keys are facing components which is a command
00:07:23.800
line their face and a server daemon that you run which provides a REST API that
00:07:29.320
is talk over age which has some really nice advantages oops
00:07:35.030
under the hood doctors at the top and then there are a few pieces and how to
00:07:41.520
make this work on the left side at a ufs stands for another union file system
00:07:47.490
that's a component that doctor use uses in order to store docker images
00:07:53.100
basically the actual file systems that make up the applications that you're going to run need to be stored somewhere
00:07:58.560
and a ufs is a layered file system that makes that efficient it used to be that
00:08:04.320
docker would talk to alexei which we looked at a little bit earlier and lxc
00:08:09.600
uses seekers and name spaces which are some kernel level features that's really like the weapons-grade stuff that makes
00:08:16.470
this work that's like the hardest components to get absolutely right I think and then but but more recently
00:08:23.580
docker has switched to a component they open source called the container which obviates the need for them to use alexei
00:08:30.900
right now you can switch back and forth and live containers the default but i think it seems like live containers that
00:08:36.780
direction that everything's good things so it just simplifies the stack a little bit okay so what do you get when you run
00:08:43.470
your Ruby application within a docker container isolation we already talked
00:08:49.500
about it's sort of an ephemeral if in the sense that if you're running a rails
00:08:54.840
app usually in the process that you're going to be serving your request from whether it's engine X with passenger or
00:09:02.190
unicorn or something like that it doesn't really need to keep anything
00:09:07.230
over the beyond the lifecycle of that process right so we kind of move to these pseudo share nothing architectures
00:09:13.350
where we're connecting to a database and writing things there but the process themselves are supposed to be disposable
00:09:18.560
and doctor works really well with that it does someone to try not to delete anything that you might need later but
00:09:24.030
basically you can throw away containers they're not one of the big advantages
00:09:30.030
with container virtualization versus say full VM virtualization is the low CPU
00:09:37.260
and memory overhead so if you think about it you know let's say you fire up VirtualBox on your laptop and then you
00:09:44.670
think about how many VMs can I run simultaneously on my laptop like well
00:09:50.790
each of those needs what like half a gigabyte of RAM at a minimum to kind of like run a full system and there a
00:09:57.270
number of gigabytes each so you can run a few of them and that works fine but with docker you can run hundreds because
00:10:04.860
you're not actually needing to store full copies of all these file systems and boot full instances of an operating
00:10:12.480
system it's more like running a process it's a special magical process that's
00:10:17.700
basically than it is running an entirely new operating system and so they also boot up much more quickly if you are
00:10:25.680
booting up a regular VM you have to wait for all of the bunching to do right
00:10:30.840
that's just kind of the way it works but with container virtualization and
00:10:35.880
docker the amount of time it takes to boot a isolated container is measured in milliseconds you basically can't
00:10:42.450
perceive how long it takes to start up a new docker container or throw it away you get the small images with the ufs
00:10:50.370
layering which we talked about in this what's everything be very very high density which means saving a bunch of
00:10:56.730
money and also save a bunch of time because if you can run all of your infrastructure on half as many servers
00:11:02.010
that makes everybody's life easier right there's not always going to be things that need to be dealt with on a per server basis and having fewer servers is
00:11:09.210
a win so doctor has two components there's images and containers or two
00:11:16.890
main concepts this is a little tricky at first I kept getting them confused but
00:11:23.340
the image is a saved version of something that can be booted into it
00:11:29.560
so I think that when I say image can think of like a package docker images like a doctor package and then when you
00:11:35.470
run that package you get a container which is like a process so it consists
00:11:41.529
of a root file system that could be any Linux distribution that's in there and a
00:11:47.259
lot of times people are facing their doctor containers on a bun teeth but you don't have to and then also metadata within the doctor
00:11:54.699
package of how to run this so there's a configuration that'll say okay when I want to run this package I think this is
00:12:01.389
the executable that I should run right pretty simple it's just a little package JSON that's sort of stored there and
00:12:07.930
then they get stored in a registry which is kind of like a package distribution site my doctor runs an official one you
00:12:14.529
can run your own just like would be gem store and push packages to it and they're become publicly available or you
00:12:20.529
can run your own gems your group so if you want to give doctor try this used to
00:12:26.110
be kind of a pain in the butt but if you're on you can install to docker
00:12:31.839
which has everything you need to get started with docker on a Mac it actually runs docker within
00:12:39.040
a regular kind of VirtualBox virtual machine so doctor itself can't run natively on a Mac so that happy is well
00:12:46.389
okay we'll just run one VM which will be our docker or host and then we'll set up
00:12:51.819
the client on your laptop to talk to that VM because it's a client-server architecture the fact that there's a
00:12:57.519
host running at the N it doesn't really matter so boots a doctor will take care of all that you don't need to worry about it if you're running on Linux
00:13:04.149
laptop or a Linux development environment you can like a bug - you can install it with the official docker
00:13:09.610
packages but long story short it's very easy to get started with docker images
00:13:16.920
so I mentioned there is a command-line interface on I'm not gonna go into a ton of details about what you can do with
00:13:23.319
the CLI but just at the high level give you the broad strokes of the main operations they perform the
00:13:30.160
documentation effect this stuff is pretty good your build is kind of the key command
00:13:37.390
these days so docker build is a way to compile a package from a directory so if
00:13:47.620
you have a rails app you're going to go into your rails app directory and let doctor build and the end of that process
00:13:53.380
will be a docker image with doctor package that you can then deploy to any number of places I'll actually show you
00:13:59.440
that in a second docker run is the next step so you've got this package now you
00:14:05.019
want to actually execute it somewhere this could be on your laptop on your staging environment or on your
00:14:10.750
production environment you can customize just a few things so just the things
00:14:15.790
that should vary between those places right so if you have bought into this idea of kind of environment variable
00:14:22.269
based configuration doctor works really well with that people who use for okay might be familiar with the concept of a
00:14:28.420
12 factor application that plays very nicely with the way doctor works and
00:14:34.750
where that ecosystem is hanging so she wanted you I think is when you deploy
00:14:40.779
your application to staging you deploy a package which you've previously tested
00:14:46.270
on CI bite for bite meet us at that exact package and you promote that to staging and then when once you test that
00:14:53.620
package on staging you promote that exact same docker image or package to production bite for bite and the only
00:14:59.950
thing that varies between the staging and production is the configuration in terms of the environment variables or
00:15:06.670
potentially some of the things that your ops people need to control so important things for example you might find your
00:15:12.550
production instances amount of difference at ports than your staging processes docker run you give it ad
00:15:19.480
minimum you just given a docker image and a level to the folks but also unless you provide environment variables and
00:15:24.699
networking Asians that resource limits so you can start a ramp limit and then your
00:15:30.370
passenger process will never be allowed to you consume more than that by doctor
00:15:37.150
yes works just like you would expect display right processes and there's just a bunch of other things that make working with
00:15:43.490
docker images and containers a little bit easier there's a public registry of
00:15:49.040
docker images so if you need to run something like say Redis MongoDB
00:15:54.370
RabbitMQ is it's likely that somebody's already gone through the work to package that up into a docker image which you
00:16:00.740
can just search for the public registry and then dr. Poole will down the back to
00:16:06.950
your docker host so that you can run it anytime you want then dr. containers
00:16:14.210
aren't they have a life cycle you can stop them which will send the signals of the processes to kill them off you can
00:16:19.790
look at the standard out of the standard error using the logs command that's kind of one of those twelve factor
00:16:25.360
application concepts you can look at all your images and you can expect what's on
00:16:30.860
your system so we mentioned docker build how does dr. build have any idea how to
00:16:38.660
specifically package your application into something that can be deployed anywhere the answer is a docker file so this is a
00:16:46.700
really basic syntax that's been developed to also get DSL for just a few
00:16:54.020
instructions you need to give to docker to let it do its job when you're ready to package up an application so this is
00:17:01.880
an example of a complete docker docker file for and a Sinatra application
00:17:08.170
there's not much to it which is kind of the point but we'll look at the sort of the few different types of commands
00:17:14.480
which are present here the first thing is that every docker file starts with a
00:17:19.910
line about where it's from so that that means what image am i starting with you
00:17:25.610
don't really want to day-to-day be creating an image from scratch where you have a root file
00:17:31.940
system with no you need to get everything there for it to work that's a bit overkill so you can
00:17:37.920
start with a either a publicly available doctor image and customize them or you
00:17:43.830
can create your own internal docker image as a base for your company
00:17:49.320
the fusion passenger guys have actually released some publicly available docker images which serve as a great base if
00:17:56.160
you're looking to deploy a rails application so that's right down here I'll talk about those images in a bit
00:18:01.850
run kind of does exactly what you would expect it's just run and then Michelle
00:18:08.390
okay add is a way from moving files from your source code directory your local
00:18:16.590
directory into the docker image right so they're not something magically going to take all of your code and know where to
00:18:23.310
put that into your into your image so you can use the add commands to move
00:18:28.470
files between your source code directory and the docker image ad and then dot on
00:18:34.620
the line there and in the middle is going to move the entire application source tree into the home app web app
00:18:41.430
directory in your resulting image and then you need to tell docker by default
00:18:48.210
what it needs to run if you just a docker run and then the name of your application you can override it what it
00:18:54.570
runs but you want to give it a safe default this spen mining is a process
00:19:00.030
that the fusion passenger passenger full image provides that will start up your
00:19:05.490
application so that's just kind of part of using this base image and then
00:19:11.360
everything works by sort of exposing a network port thank you you're going to
00:19:18.120
get access to so most docker files will define that
00:19:23.550
they're going to do a bunch of stuff any end result of all that becomes a black box right here just say ok when this is
00:19:29.340
done we're going to expose port to the world and that's how you'll be
00:19:34.600
able to use this so to the outside world it doesn't really matter what's happening in magazines the you know the
00:19:39.879
network is the interface between these things so this is the talk about Ruby so
00:19:48.970
let's look at what you can do to get your Ruby applications deployed into docker today okay so I mentioned that
00:19:57.100
that diffuse and passenger guys created some publicly available images to smooth this process I strongly recommend using
00:20:04.779
them especially if you're just getting started with doctor there you can start from this regular of on to you but then
00:20:11.649
you're kind of back to square one in terms of well how am I going to get the right version of Ruby on there am I
00:20:17.950
going to use our VM for rpm or some apps custom actor posit or e that has review
00:20:23.980
point one point two is just not fun to deal with those things right so the
00:20:28.989
passenger guides have kind of already figured that out for you and you just leverage what they've done they have
00:20:35.710
images that have Ruby up through version 2.1 python and you know already there as
00:20:43.299
so as long as you're just dealing with those things you have to do very little customization they also have a build
00:20:48.970
tool chain and comment sea level dependencies like live XML too so you
00:20:56.139
don't have to worry about making sure you have the right version and that stuff installed and the amount of
00:21:01.179
overhead that you get through building on top of this image there are some processes that it's going to spin up by
00:21:07.720
default when you use this this image is your base but all told that's about ten
00:21:12.970
megabytes or so of a fixed cost which compared to the RAM usage of most production rails processes is pretty
00:21:21.009
small these days so when you run it I mentioned the SM processes that are
00:21:27.100
going to take some RAM these are some of the things that it can give out of the box so the Mayan em process
00:21:32.679
is what we saw in the dockerfile as the default executable my in it is just a
00:21:38.410
little python script which handles loading up the planet which actually
00:21:44.710
runs the other dependent services and then there's this edge case that the passenger guys have sort of found where
00:21:52.830
long story short if you boot a docker image and the group process in that
00:21:59.350
image is your application that Matt is going to become hit one and that pig one
00:22:05.260
is not going to be taken care of one of the responsibilities of pig wand which is dealing with whorfin processes so
00:22:13.240
they wrote this by intention which which does that for you but for all intents and purposes you don't need to really worry much about what that's doing so
00:22:20.919
the other things that that image provides our core facilities that you kind of need everywhere right syslog is
00:22:27.549
important to be running because of your Linux distribution thing and it's cause we want them to actually go somewhere
00:22:33.730
you don't want them to just get lost in the ether cron is often important if you
00:22:38.980
want to do something like block rotation or something system level that needs to be done periodically you can also just
00:22:45.880
use something like clockwork and deal with that at the regulator which is what I do and then sshd is kind of an
00:22:53.049
interesting piece right so if you're running a docker container it's not
00:22:58.210
necessarily clear how you go into it and sort of look around what's the inspect ability you can run a
00:23:04.870
new docker container with and you can select a command like bash which is very
00:23:10.630
similar to if you've ever done like Heroku run bash you get a bash shell inside of a heroic environment you can
00:23:17.350
do the same thing with docker you can docker run ash but you're not gonna be able to look at a specific container and
00:23:23.020
say what's this thing doing right now unless you either run sshd or kind of
00:23:29.980
jiggle around with some of the lower-level container virtualization commands in order to
00:23:36.340
to it which is a bit tricky right now hoping that dr. will a doctor attached coming up breathe to say doctor attached
00:23:42.460
and they get a shell within a containers he can inspect it but that has not
00:23:47.770
happened quite yet okay so now I need
00:23:52.840
the demo gods to smile upon me and undo a quick demo this works yes okay so this
00:24:11.770
is just my laptop here and since it's a
00:24:20.110
Mac we're gonna use boots a docker in order to get this whole environment up and running I've already started that up
00:24:26.250
but basically similar to your favorites you can run boot to doctor a month
00:24:32.649
instead of vagrant up and it will just make sure that everything is in its right place and it will actually give
00:24:39.130
you a environment variable that you need
00:24:44.710
to set on your local shell in order to configure for the doctor or client how
00:24:50.649
to talk to the server right so this address here and this port is where the
00:24:55.809
docker server is listening over HTTP for commands when I run docker PFC just does
00:25:01.809
a get request to the server and says okay what are there running processes and then the command line just gets that
00:25:07.750
back at prints that out but because the docker server is running within mmm it's
00:25:15.909
not going to show up it's not going to show up inside of my process tree it's
00:25:23.110
not running on my local box it's inside of the VirtualBox VM so what I've got
00:25:28.539
here is just a simple application that's a Sinatra app
00:25:36.150
and what I'm going to do is package this up into a docker image and then run a
00:25:42.540
version of it for my own purposes so let's see here's the docker file that
00:25:50.010
I'm going to use this is very similar to the one that I showed on the slide the only thing I really changed here is just
00:25:57.810
an optimization on these lines which makes better use of Dockers build cash
00:26:05.280
to not have to do bundle install if the dependencies don't change right so everyone knows like bundle install can
00:26:11.910
kind of take forever and you don't change your gem file that much for reasons that aren't particularly
00:26:17.250
interesting if you push the gem file into your docker image before you run
00:26:22.710
bundle install then doctor is able to cache that more efficiently than if you just omit these lines it would still
00:26:29.370
work but it would do a bundle install every time so let's start with a docker
00:26:41.430
build what I'm gonna do is this force RM which tells it to that way
00:26:52.410
sorry I am going to have it not use the docker cache you can actually see what
00:26:58.270
this looks like if it's going cold here we go
00:27:04.290
cool so it's running it's running the bundle install but it's going to scroll back up to the beginning you can see
00:27:10.750
with each step it's going to run the command and then save an intermediate
00:27:18.100
container so it starts with the passenger full image it's going to
00:27:24.120
remove this file at this file over here each time it's actually caching the
00:27:29.350
result in a beautiful image so that when we go and let this again it'll be much faster when it gets to the bundle
00:27:36.400
installed it runs that now it's showing you the output from the command that Ram
00:27:41.650
and it finishes right so now just to
00:27:46.720
show you if we run this again it's going to be really quick because I don't know
00:27:57.130
why I actually went back to my old install but not that interesting what
00:28:02.680
we'll do now that we have a docker image
00:28:13.419
we can see if you say docker images we can see the hello image is sitting on
00:28:19.959
our local system we can actually execute that and the - P argument tells doctor
00:28:29.409
to map all of the ports that the docker image is supposed to expose and the - D
00:28:35.529
option calls it to run in the background as a daemon as opposed to you in the program and wait around for it so sorry
00:28:43.779
for that let's see if I can make this
00:28:50.700
dr. PS will show me what's running so you can see that the image Iran was the hello image it's running the s p-- in
00:28:57.940
maya in dictum and it's been off for 14 seconds and it actually assigned a
00:29:04.899
dynamic port to map 1080 so the docker file said i'm going to provide 80 that's
00:29:10.809
where i have interesting stuff and then docker when you booted it said okay i
00:29:17.049
need a port to make that port 80 available so obviously if you ran multiple docker images that all provide
00:29:23.259
port 80 they're going to conflict right so it's kind of up to your ops team to figure out what courts they want things
00:29:28.869
to sit on in prague right most application developers don't really care about the important umbers and they
00:29:35.049
shouldn't care they're just going to that's kind of the contract is the app developers provide something that
00:29:40.059
exposes a port and then they ship it to ops who figures out how to actually run it in prod so now with this right so if
00:29:53.109
I hit the root there's no route there so that's just the Sinatra not found page and if you get slash high it says hello
00:29:59.619
Ruby people it works so what I'm gonna do quickly is just
00:30:08.289
make a quick change change the code you can see I've modified the application
00:30:14.830
I'm going to run the build process again and now it worked at the cache so each
00:30:20.740
one of these steps you can see it says using cache that's not actually running that that's why it's really fast including the bundle install use the
00:30:28.059
cache and then when it got to the ad it did not use the cache because the ad
00:30:33.669
step was adding a different version of my code into the docker image so this is just a fast recompile I'm going to run
00:30:40.029
the same run command again and the ports don't collide because it's going to assign another random port to the the
00:30:49.090
new container you can see we have this container over here and gives every container a name it does it you can
00:30:54.909
specify the name or it will give you a randomly generated one so in this case we have romantic hopper and goofy Yona
00:31:04.149
my favorite name is it like a often I seem to get angry Torvalds which makes a
00:31:11.649
lot of sense so you can see this one's been up for two seconds but it's running
00:31:17.470
on port for nine one five four on port for nine one five three it's still
00:31:22.809
saying hello Ruby people but if we change the port to four nine one five four hello read that because people and
00:31:30.640
what you'd do if you were using this to deploy to production is you'd probably keep the old processes around boot up
00:31:37.659
the new ones switch your load balancer to serving up requests from the new port
00:31:43.149
and then remove the old port from the load balancer config and shut it down so
00:31:48.370
that would just be a matter of doing docker kill and it will shut that down
00:31:53.740
now it's going to be gone or you can also address it by its name instead of
00:31:58.990
the container ID and it's all the same so now it's not gonna be available cool
00:32:05.220
that is the demo that's Switchback awesome
00:32:19.380
so okay um so I'm super excited about
00:32:24.910
this and this is the reason why as a ruby developer I feel like for all the
00:32:31.270
years that I've been doing this I've never really had a unit to deliver my
00:32:36.580
application right so for a while I worked on an application that was in JRuby and we built a jar file and gave
00:32:45.370
that to our ops team and for as many things as you can say about Java and it being scary and enterprise-e that was
00:32:51.400
kind of awesome because it didn't really matter what was in that jar file I could test the jar file I knew that it worked
00:32:57.460
I gave it to ops and they could run it the thing is if you're on C Ruby that concept doesn't work at all I guess some
00:33:04.270
people have done work to package their Ruby applications up into like deb files
00:33:09.850
I mean it's just files on disk so you can kind of package them up any way you want but it wasn't ever particularly
00:33:15.520
easy to do that so if you're looking for a delivery unit you run into all these
00:33:22.030
issues right how are you going to deal with things like bundler and what if
00:33:27.280
your dependency your ruby gem dependency actually requires something like Lib XML
00:33:32.620
- that needs to be on a system there's no way in your bundler config and your gem file to say make sure you have this
00:33:38.799
version of Lib xml - installed but you can do that with docker war files or jar
00:33:44.049
files we talked about that you can't use those with anything except the the Java platform and if you use docker you can
00:33:51.790
get all this for any service right you can put Java code into a docker image
00:33:57.520
there's no problem with that at all but you can also put anything else you want into it right so people are packaging up
00:34:03.010
things like MongoDB as a docker image people are packaging up rails applications as docker images and it's
00:34:09.010
all just homogeneous from the standpoint of whoever needs to run it so this opens
00:34:15.730
up a whole world of change in terms of DevOps processes you can use the same
00:34:22.510
image container for local development that you push to your continuous integration
00:34:27.669
environment one idea that we're starting to play with is in our docker images in
00:34:33.669
addition to by default running the code and providing the application creating
00:34:38.800
another executable inside the image called self-test so if you say docker run hello and then you customize the
00:34:46.419
executable and you say okay I want to run self-test instead the docker image is self testing so it's gonna run your
00:34:52.840
suite of tests and put out an exit code of pass or fail whether that image is
00:34:58.270
doing what it's supposed to be doing which is a really powerful concept then you can do things like taking that image building it on CI running the self-test
00:35:05.650
signing it on your CI server as having passed the self test promoting it bite
00:35:11.650
for bite up to staging verifying it with manual queue and staging promoting it bite for bite up to production and
00:35:17.440
really only having the environment variables and potentially your port configurations changing between those
00:35:22.630
things so that whole problem of well it works on my machine or I'm not really sure what this is you know why this is
00:35:28.660
failing on CI kind of goes away and you can really get this benefit of building
00:35:36.550
it once and then running it anywhere within your sort of Linux ecosystem so the fact that it doesn't run on Windows
00:35:42.090
really doesn't bother me all that much this kind of meets all my needs ok this
00:35:48.430
changes everything it's you have to sort of stare at it for
00:35:53.650
a while but why I'm really excited about docker is once you change this one
00:35:58.990
component of your infrastructure it sort of unlocks all of these other benefits through all the other layers so
00:36:06.300
immutable infrastructure is an example of that a lot of people how many of you have
00:36:11.800
used chef or something like that to sort of configure your servers yeah about like half so chefs great but there's
00:36:20.050
this fundamental question that's been nagging at me which is why do we ever run chef against the same server more
00:36:27.609
than once right that only kind of leads to trouble the first time you run chef
00:36:33.400
it's great because you run it and you know exactly what state it's in it ran this script it's deterministic that server is
00:36:38.980
in a known state and you're fine then the problem is you're like oh well now we need to upgrade the version of this
00:36:45.550
package so you change the version of the package and you rerun chef and sometimes it's fine sometimes it fails and you
00:36:51.490
need to iterate on that a number of times but through that whole process you're creating indeterminate state on
00:36:57.100
that VM so the reason we run chef against the same server multiple times is because it's too expensive to
00:37:03.880
actually throw it out every time right to throw it your VM every time on AWS after every chef run people do it but
00:37:12.010
it's not it's not free at least in terms of time right with docker containers and
00:37:18.220
docker images they're so fast that you can actually throw them away every time it makes no sense to you know SSH into a
00:37:26.950
docker container and apt-get install an additional package because now you need
00:37:31.990
this extra package you're just going to throw that container away and build a new image which has the packages that you want so you know exactly what's in
00:37:39.010
all of your containers at all times which is a really powerful concept then
00:37:44.800
it gets a little more crazy right so if you're running all of your code inside of a docker container if you're running
00:37:50.620
MongoDB Redis memcache passenger engine acts all inside of docker containers and
00:37:57.520
there's a host VM right there has to be an underlying operating system that all this is running on and a lot of people
00:38:02.860
are using a bun too for that but if you think about it why do you need a bun to
00:38:07.960
underneath right like why do you need a package manager at all if your packages
00:38:13.270
are actually just docker images maybe you don't need all of that and maybe not
00:38:19.390
having that can actually lead to a simpler and more secure system so you've
00:38:26.110
already got a way to run whatever code you need the host operating system is really just a matter of executing the
00:38:32.950
docker daemon and keeping things running along smoothly so there's a Linux distribution called core OS which is
00:38:39.850
exactly this these guys kind of rethought a distribution from the standpoint of everything running within
00:38:45.160
container virtualization and they stripped out almost everything it's actually auto-updates so you never
00:38:51.870
like just like chrome auto-updates you can have an auto updating host distribution on your servers and it has
00:38:58.080
almost nothing on it except for docker and some canned basic configuration management stuff which is really
00:39:05.460
powerful from the standpoint of security right if you can just get an over the wire update of your host system it's not
00:39:12.600
likely that that's going to break anything if there's very little running on the host itself so it doesn't have a
00:39:17.640
package manager where you're going to be installing things and updating things and dealing with all that complexity you just get an entirely new image and drop
00:39:24.570
that on top of the old version or actually next to the old version of core
00:39:30.000
OS and then it just changes a pointer so that when you restart the the box it'll boot off the new version of the
00:39:36.240
operating system this eliminates entire class of issues okay the Linux
00:39:42.660
distributions actually kind of become unnecessary in the containers too right so the fusion passenger base image that
00:39:48.630
we started with is based on Ubuntu right now but does it need to be right there was
00:39:53.970
nothing in my docker file that was a bun to specific and there's nothing
00:39:59.040
fundamental about the passenger base image that requires a bun to so you could run something that's much smaller
00:40:04.800
than a bun too and get the same benefit somebody could take a micro Linux distribution like say busy box and add
00:40:12.590
Ruby into it and the few c-level dependencies that are needed and now the size of the image for your application
00:40:20.310
drops from hundreds of megabytes down to maybe you can get that down under a hundred megabytes which is really
00:40:25.830
powerful right so now you're like okay we're using docker and we don't need Linux distributions on the host and we
00:40:31.260
don't really need traditional Linux distributions in the containers docker just kind of shifted everything
00:40:37.190
dramatically it goes further you don't really need configuration management if you're running core OS on the host
00:40:44.550
operating system there's not much to manage at all there's a single config file which has things like the users
00:40:50.430
that you need to have available in the few different customizations but that's pretty much it
00:40:56.750
and then for building the images the dockerfile in many cases is all you need you can shell out to a batch script so
00:41:03.320
if you need to install a bunch of packages you can create a dot s H file and run that and it can do a few package
00:41:08.390
installs but for most cases you don't need to be using something like chef or ansible within your docker file to
00:41:14.570
install dependencies because it's pretty simple right we saw how many lines of code that was and then if you go further
00:41:21.110
downstream this starts to affect things like the ROI on service-oriented architectures so a lot of people are
00:41:28.160
talking about micro services these days right and which is kind of a new formation of the service-oriented design
00:41:33.920
concepts they're great for a lot of things but they're not a free lunch especially with respect to deploying
00:41:40.400
them into production deploying five or ten apps is generally harder than
00:41:45.680
deploying a single app but with if you're deploying on something like Heroku it gets a lot easier right so
00:41:51.980
I've worked with teams that use heavily service-oriented architectures on Heroku and Heroku is kind of one of the reasons
00:41:57.410
they're able to do that because Heroku takes care of managing everything you just say okay push this to Heroku now I
00:42:02.720
have my service running Dockers going to unlock those workflows for any environment that you want to deploy into
00:42:08.630
if you wanted to plan your own Hardware you can start to get some of those benefits you can do behind the firewall
00:42:14.000
install so I know multiple companies that have products cloud based products
00:42:19.850
software development tools that they have customers are asking hey can I run this behind my firewall for example like
00:42:27.020
the github enterprise problem right that's traditionally been a huge pain but if everything is running on top of
00:42:33.920
docker in production you can actually just give them a docker image and say okay you're gonna download this image and you're it's a docker run code
00:42:40.520
climate for example and you're going to get a port that exposes code climate that's the contract is it's going to
00:42:46.400
expose port 80 and port 443 and you'll be able to get a working code climate from just that so it really simplifies
00:42:53.740
behind the firewall installs and then the kind of final frontier is a concept
00:43:01.220
that sometimes referred to as datacenter computing so the idea of data center
00:43:06.530
computing is that you're just going to plug in a bunch of Hammad resources into your computing pool and
00:43:14.250
then they're going to be able to run heterogeneous sets of work so when you need more resources you should be able
00:43:19.980
to call up your ops guy and say hey we need to double the number of you know CPUs and RAM that we need but what it's
00:43:27.000
being run for should be abstracted away and just dealt with as a commodity
00:43:32.040
so there's projects core OS has a project called fleet built into it which
00:43:37.470
does this sort of thing you can say fleet run instead of docker run and if you have a fleet cluster it'll just find
00:43:44.430
a place that has the appropriate resources or the appropriate configuration to run that and there's
00:43:50.220
also projects that are a bit more robust like mesas which is a top-level Apache
00:43:55.770
project and used heavily at Twitter where you can build a meso sloughed and
00:44:01.710
just say okay Mesa I need three hundred passenger processes they each need to
00:44:06.960
have two gigabytes of RAM available and two CPU cores and I don't have any idea
00:44:12.210
I don't even want to know where those have to run within our data center but you've got you know a thousand machines
00:44:17.910
figure it out and just mix them together in into whatever resources are available and then give me back a way that I can
00:44:25.290
address them and add them to the load balancer there's a bunch of interesting stuff happening in this data center
00:44:30.630
computing space right now mesas Aurora is a project on top of that fleet and
00:44:35.670
there's a few others as well that are going to make it you know we're gonna get to a point where even on small scale
00:44:41.280
projects you're just plugging resources in and then defining in a declarative
00:44:46.650
way what resources your application needs and as long as your resources are always more than the hardware resources
00:44:52.230
you have are greater than what you need for the software you're fine you don't have to worry about it okay so what's
00:44:58.770
the status of docker and they released 1.0 finally a couple weeks ago it's in
00:45:04.320
production at a good handful of companies it's pretty stable at this
00:45:09.630
point if you want to deploy let's say a rails application on top of docker
00:45:16.140
there's some there's still some sticky bits it's not necessarily that docker has rough edges so much as
00:45:23.820
docker has a limited scope and what you'll find is when you go to deploy things on docker there are other
00:45:30.060
problems that are not within docker propers scope that you're faced with so if you are for example spinning up a
00:45:38.490
bunch of docker containers and you need to add those to a load balancer and then you want to do a deployment like I did
00:45:43.740
on my laptop where you're booting new containers you need to reconfigure your load balancer and drop out the new ones
00:45:48.870
that's not really a problem that Dockers trying to solve you need a tool that's higher level than that to deal with that
00:45:54.570
so this is where a lot of the activity is happening right now there's a whole ecosystem of projects that are being
00:46:00.960
developed that build on top of docker and make that type of stuff possible for
00:46:06.120
the load balancer example there's a project called hip hacci which is a Redis or Etsy uses Redis or Etsy D to
00:46:14.730
configure dynamically configure a load balancer based on whatever you need to be available so you can just talk to a
00:46:21.390
Redis instance and say ok these are the IP is of and the ports of the application right now and reconfiguring
00:46:28.200
your load balancer becomes as simple as running some Redis commands so you can still do that with say Capistrano right
00:46:33.960
as long as you can connect your reticence in so you can reconfigure your load balancer so there's kind of pieces
00:46:39.120
being filled in at each layer of the stack but the ecosystem is not quite mature enough yet where if you're going
00:46:45.240
to run something on dock or you can just say ok I'm gonna use docker and these other three components I'm a big fan of
00:46:51.330
core OS I think that that's gonna be kind of part and parcel with the default docker stack and core OS also provides a
00:46:59.070
component called Etsy D which is a distributed configuration service akin to Apache zookeeper so you can keep all
00:47:05.760
your metadata for where everything is in the environment variables you need inside add CD but even doing that is not
00:47:11.100
trivial right now so there's kind of a few like the exercise is left to the reader bits but the reason I want to get
00:47:18.630
up here and talk about this now is I think you can definitely use it for deploying for developing locally you can
00:47:26.520
definitely start integrating into your CI environments and the ecosystem is getting better and better every month in
00:47:33.510
terms of how you know it's really the issue is not have options for deploying to production it's
00:47:38.820
that there's too many options so so many people have created projects on top of docker that there needs to be some consolidation I think in the ecosystem
00:47:45.630
where it's more like rails and there's like this is the happy path and that's gonna be I think happening over the next
00:47:51.780
year but I will be very surprised if and the vast majority of people in this room are not deploying stuff on top of a
00:47:58.260
docker based system within the next couple years so it's a good thing to start looking at and playing with and
00:48:03.720
getting getting your head around because it's going to have some really big implications all the way up and down the stack that's it thank you very much for
00:48:11.790
your time and attention and we can do some QA yeah so in Sao in development
00:48:36.510
right so what I did there was akin to say deploying something out to a staging environment so I had a docker image that
00:48:43.140
I built from a version of the code I changed the code and then I built a new docker image and deployed that out
00:48:48.570
if you're doing local development what you would probably do instead is create
00:48:54.360
a docker basically make the files that you're coding in available to the docker
00:49:00.060
image so there there are ways that you can wire it up so where you're actually editing your source code you can make
00:49:05.160
changes and then the docker the docker image can kind of just access that without having to rebuild every time you
00:49:12.180
save a file
00:49:17.400
yeah so you're going to it depends on if you're on a Mac or Linux the way you
00:49:22.650
would set it up but yeah you can expose through if you take let's let's put the
00:49:27.780
Mac thing aside if you're let's say you're developing on Linux you can do what's called a bind amount with docker
00:49:34.560
which is the - V option and that just lets you take directory on the host file
00:49:39.810
system which would be your laptop and expose that into the docker container so you just bind mount in the the directory
00:49:46.740
where you're working on the code and then that would be available inside to everything that's running within the docker system if you have if you're on a
00:49:53.430
Mac and you're using vagrant then you have to do like another level of mounting right so you mount from the Mac
00:49:59.130
system into the vagrant VM and then from the vagrant VM into the docker container but it's also possible yeah so I've seen
00:50:17.670
it done a couple ways we have a image that we can build for code climate which runs everything within the same image so
00:50:25.320
it runs Redis MongoDB memcache engine acts passenger all within a single image
00:50:32.070
and you run it and you just get code climate right that's one model for
00:50:37.170
production that's not particularly useful right because you want to manage your database separately potentially on
00:50:42.960
separate physical servers and it doesn't really deal with things like high availability and that sort of thing
00:50:48.540
but it is useful for for example shipping things behind a firewall what I
00:50:54.180
think is likely to be the case is you're gonna have kind of different potentially slightly different package roles
00:51:00.510
depending on where you're deploying to and then in production like staging and
00:51:05.580
production will all kind of be the same so sometimes people will say you can run
00:51:10.830
one you should go on one process per docker container I think for most use
00:51:16.470
cases that's a bit extreme I would think instead about like a role so you have a role of app right and that
00:51:23.970
probably runs maybe it runs an index process which also has a passenger process attached to it but if you
00:51:30.940
discover that you need something like sidekick or you want to run clockwork
00:51:36.550
and do a ruby based Krong system you could create separate images for those things and be able to deploy them all
00:51:42.730
separately but in most cases you don't need that flexibility so I would lean towards keeping things simpler to start
00:51:50.020
which in generally involves putting more into a single docker image to begin with and then splitting things out as you
00:51:56.410
have need because that adds a little bit more complexity yeah yes so docker has
00:52:07.420
the concept of links which is relatively new so you can say okay docker run
00:52:13.690
MongoDB and then docker now run my application but make MongoDB available
00:52:20.170
to it and that's called a link and that only really works if they're running on
00:52:25.210
the same docker host so I think for many use cases it can break down pretty
00:52:31.840
quickly but it can work for like local development so you can run MongoDB in a separate docker container and then use
00:52:38.830
links to expose that to your to your app container so it kinda just depends on
00:52:43.960
whether you're going single hosts or multi hosts is a big variable but then you can always do you know your own your
00:52:50.740
own thing yourself right so this is kind of one of the sticky bits it's like how do you tell everything where everything
00:52:56.200
else is docker lets you do the dynamic port binding that we saw in the example
00:53:01.660
but it can also do static port binding so you can just say okay docker run MongoDB and expose port two seven zero
00:53:09.820
one seven from the container as two seven zero one seven on the host I'm
00:53:14.890
gonna fix that port now if you try to do that twice you're gonna get an error but it does simplify things dramatically
00:53:20.260
because if you don't need to then you can address your database from your app server and say well it's on this host
00:53:26.440
and it's port two seven zero one seven which is the default MongoDB port so you don't necessarily need to jump to fully
00:53:33.100
dynamic configurations for everything right off the bat and so when you wanna go mmm yeah so
00:53:43.000
when you want to go to dynamic service discovery there are a few options as a project called Sky DNS there's a project
00:53:49.359
called there's weather so there's @cd which can can be used to sort of insert
00:53:56.049
information about where services are located and configure processes but you kind of need to glue that together yourself and then they Airbnb team
00:54:04.270
released a couple projects that are used for kind of doing the same sort of thing Apache works for sort of TCP layer load
00:54:11.349
balancing but not necessarily everything can be done like that so that's that's probably the area where them with the
00:54:18.160
most options right now in terms of how you're gonna do it I'm interested to see what the docker team does in terms of
00:55:19.980
the other thing that's nice and I was there's a gate on this issue I'll feed you later after token - this
00:55:30.680
fusing limitation or daughter so that was pretty good OSs
00:55:40.130
- Pete sees not one shared why that's that's the big thing is the same for us
00:55:46.800
both dollhouse leading to the utmost
00:56:00.890
I have a question yeah so when you talk
00:56:09.150
about having darker right four different images so for example if I have a database image then would I be any like
00:56:16.770
you know cases where I might delete the image M on my data be gone ah yeah so
00:56:22.670
that's a good question so with docker by default it will try to save things for
00:56:31.020
you so if your your docker container generates a bunch of files on disk it
00:56:36.299
will keep those around until you tell it to delete those but with the database
00:56:42.960
which is a great example right so you database obviously you need file storage that is has a longer life cycle than
00:56:49.500
just that's that time you ran the database right so for that you would use
00:56:54.569
bind mounts or volumes so the - V flag so you would just create a directory on
00:56:59.640
the host system maybe it's a network attached storage or you know high performance disks you would create a
00:57:06.690
partition for that and it would have a path right so it would be like slash of
00:57:11.880
data for example and then when you boot your MongoDB container you would say
00:57:17.190
okay bind slash data from the underlying host onto the into the container and
00:57:24.540
then everything just writes there so when you stop MongoDB it'll still be there you can use all of the
00:57:30.920
conveniences that you might have available like snap filesystem level snapshot backups all that stuff just
00:57:37.140
works because it's all running in the same kernel right so if it was different kernels then you'd have to get into
00:57:43.319
things like NFS right to like share files back and forth but there's nothing
00:57:48.900
that prevents you from bind mounting the same directory into multiple containers
00:57:54.930
and the kernel will handle that because it's basically just a process that's sort of sandboxed so fine months
00:58:05.560
yes no traveler on a docker
00:58:15.940
super-close thanks Aaron
00:58:23.770
Brian have a question so are you running dock in production and and are you facing any challenges yeah so for us in
00:58:33.500
production we do have that behind the firewall use case that I described we
00:58:41.150
are we have shipped code climate behind the firewall with docker and that is in
00:58:47.180
Prague so for our individual customers that is like back on docker 0.6 which is
00:58:53.330
a little bit crazy though your early adopters for our production instances
00:58:59.990
code climate calm that's a project for kind of actively spinning up right now to move everything over to that and
00:59:06.859
we'll probably try to use core OS for that as well so we just have core OS
00:59:12.440
boxes and at least at the app tier which is important that's actually an important thing to note if you're going
00:59:18.890
to start using docker you don't need to run everything in your entire environment with docker right so for us we have maybe 10 physical servers or
00:59:26.480
something like that a couple of those are database servers or a handful those are database servers those are probably
00:59:32.390
gonna be the last thing to switch over to docker because the benefits are just not quite as extreme it's not like we're
00:59:40.550
actually building we would be building new MongoDB images and deploying them out on a regular basis so those can just
00:59:46.550
sit they have host names and they have static ports and the app tier can still
00:59:51.710
be moved to docker and then address those the same way that addresses them today so we're going to start by doing
00:59:57.349
everything in the app tier which is the workers that do the analysis on our code and then the the passenger processes
01:00:04.510
thank you any other questions O'Brien
01:00:12.770
thank you
01:00:20.859
you
