diff --git a/en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv b/en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv new file mode 100644 index 0000000000..7f564d1d33 --- /dev/null +++ b/en_US.ISO8859-1/captions/2006/mckusick-kernelinternals/mckusick-kernelinternals-1.sbv @@ -0,0 +1,4302 @@ +0:00:09.469,0:00:11.309 +Hello my name is Marshall Kirk McKusick + +0:00:11.309,0:00:15.389 +and I've been around as long as dinosaurs +and mainframes have ruled the world + +0:00:15.389,0:00:18.429 +which is to say the sixties and seventies + +0:00:18.429,0:00:22.460 +however by 1970s a new breed of mammals had began to show up on +the scene, + +0:00:22.460,0:00:24.240 +known as mini computers + +0:00:24.240,0:00:28.230 +although they were just toys in the 1970s they would soon grow + +0:00:28.230,0:00:31.689 +and take over most of the computing market + +0:00:31.689,0:00:33.150 +In 1970 + +0:00:33.150,0:00:37.910 +at AT&T Bell laboratories two researchers Ken +Thompson and Dennis Ritchie began developing the + +0:00:37.910,0:00:39.900 +UNIX operating system + +0:00:39.900,0:00:42.040 +Ken Thompson who had been an alumnus at Berkeley + +0:00:42.040,0:00:46.100 +came back on a sabbotical in 1975 bringing UNIX +with him + +0:00:46.100,0:00:47.539 +In the he was there + +0:00:47.539,0:00:51.330 +he managed to get a number of graduate students interested +in UNIX + +0:00:51.330,0:00:53.940 +and by the time he left in 1976 + + +0:00:53.940,0:00:56.829 +Bill Joy has taken over in running the UNIX system + +0:00:56.829,0:01:00.470 +and in fact continuing to develop software for it. + +0:01:00.470,0:01:04.339 +Bill began packaging up the software that have +been developed under Berkeley UNIX and + +0:01:04.339,0:01:05.779 +and distributing it + +0:01:05.779,0:01:08.040 +as the Berkeley Software Distribution + +0:01:08.040,0:01:12.310 +whose name was quickly shortened to simply BSD + +0:01:12.310,0:01:16.330 +BSD continued to be distributed with +yearly distribution for almost fifteen + +0:01:16.330,0:01:17.490 +years + +0:01:17.490,0:01:21.920 +initially under Bill Joy and later under others including +yours truly. + +0:01:21.920,0:01:24.860 +By the late 1980s interest had began to grow + +0:01:24.860,0:01:27.400 +in freely redistributable software + +0:01:27.400,0:01:30.170 +so a number of us at Berkeley began separating +out + +0:01:30.170,0:01:32.649 +the AT&T proprietary bits of BSD + +0:01:32.649,0:01:35.710 +from those parts that were freely redistributable. + +0:01:35.710,0:01:40.590 +By the time of the final distribution at BSD +in 1992 + +0:01:40.590,0:01:43.620 +the entire distribution was freely redistributable. + +0:01:43.620,0:01:45.909 +I live in a capsule history here + +0:01:45.909,0:01:48.009 +but if you're interested in the entire story + +0:01:48.009,0:01:50.789 +I have this three-and-an-half hour epic + +0:01:50.789,0:01:54.590 +which is available from my website www.mckusick.com + +0:01:54.590,0:01:58.200 +that gives the entire history of Berkeley. + +0:01:58.200,0:02:00.239 +Following the final distribution from Berkeley + +0:02:00.239,0:02:01.450 +two groups sprung up + +0:02:01.450,0:02:03.600 +to continue supporting BSD + +0:02:03.600,0:02:08.080 +the first of this is the NetBSD whose primary +goal was to support + +0:02:08.080,0:02:10.459 +as many different architectures as possible + +0:02:10.459,0:02:14.769 +everything from your microwave oven all way +upto your cray XMP + +0:02:14.769,0:02:19.409 +In fact today NetBSD supports nearly +sixty architectures. + +0:02:19.409,0:02:22.419 +The other group that sprang up was FreeBSD. + +0:02:22.419,0:02:28.239 +Their goal was to bring up BSD and support +as wide set of devices as possible on the + +0:02:28.239,0:02:29.719 +PC architecture. + +0:02:29.719,0:02:36.549 +They also had a goal of trying to make the +the system as easy to install as possible to + +0:02:36.549,0:02:39.309 +attract by a wide group of developers + +0:02:39.309,0:02:42.319 +I chose to primarily work with the FreeBSD +group + +0:02:42.319,0:02:43.740 +both doing software + +0:02:43.740,0:02:46.140 +and also together with George Neville + +0:02:46.140,0:02:51.069 +writing this book ""The Design and Implementation +of the FreeBSD Operating System"". + +0:02:51.069,0:02:52.060 +Together with this book + +0:02:52.060,0:02:53.959 +I developed a course + +0:02:53.959,0:02:56.500 +which runs for twelve chapters + +0:02:56.500,0:02:58.179 +and thirty hours. + +0:02:58.179,0:02:59.749 +The purpose of this video + +0:02:59.749,0:03:01.089 +is to give you a taste + +0:03:01.089,0:03:02.819 +of that course. + +0:03:02.819,0:03:07.249 +What follows are excerpts from the first lecture +of the course + +0:03:07.249,0:03:11.139 +which of course you can also get from my website +www.mckusick.com. + +0:03:11.139,0:03:13.069 + + +0:03:13.069,0:03:17.739 +Enjoy. + +0:03:17.739,0:03:22.239 +This class is nominally about FreeBSD +because well + +0:03:22.239,0:03:26.379 +that's what I know best and that's what +the textbook is organized around + +0:03:26.379,0:03:29.979 +but the fact that the matter is back it's really + +0:03:29.979,0:03:32.339 +a class about your UNIX and that + +0:03:32.339,0:03:36.539 +really covers sort of the broad range of things +in the open source arena as its FreeBSD + +0:03:36.539,0:03:37.689 +in Linux + +0:03:37.689,0:03:38.899 +which of course + +0:03:38.899,0:03:41.159 +you use a lot out + +0:03:41.159,0:03:41.550 +and + +0:03:41.550,0:03:44.349 +it also covers a commercial systems + +0:03:44.349,0:03:46.950 +%uh Solaris, HP-UX, + +0:03:46.950,0:03:49.279 +AIX and so on. + +0:03:49.279,0:03:52.419 +I am going to tend more towards the open +side + +0:03:52.419,0:03:56.389 +open source side of things.So it's really +going to be more FreeBSD in Linux than it's + +0:03:56.389,0:03:57.579 +going to be + +0:03:57.579,0:04:00.849 +Solaris and HP-UX and so on. + +0:04:00.849,0:04:06.959 +For the most part at the level of this course +we're dealing with the interfaces to the system + +0:04:06.959,0:04:07.329 +and + +0:04:07.329,0:04:11.599 +the fact that the matter is a those interfaces are highly +standardized at this point + +0:04:11.599,0:04:12.060 +and + +0:04:12.060,0:04:15.280 +whether it's FreeBSD or Linux or Solaris +or whatever + +0:04:15.280,0:04:19.460 +the Socket system call has to do the same +thing, it has to have the same arguments + +0:04:19.460,0:04:20.150 +in that, + +0:04:20.150,0:04:23.909 +it has to have the same effect + +0:04:23.909,0:04:27.319 +and so until you get down to the really niddy +details + +0:04:27.319,0:04:29.600 +of how they actually go about implementing +that + +0:04:29.600,0:04:31.960 +the differences are relatively minor. + +0:04:31.960,0:04:35.830 +So I would say that sixty to seventy percent +of the material that I'm covering + +0:04:35.830,0:04:40.779 +is just as true for FreeBSD as it would +be for Linux + +0:04:40.779,0:04:42.580 +or for Solaris + +0:04:42.580,0:04:44.659 +%uh AIX is a little bit + +0:04:44.659,0:04:45.629 +sort of off in the weeds + +0:04:45.629,0:04:48.709 +%uh as is HP-UX + +0:04:48.709,0:04:51.099 +but luckily we don't have to worry too much about +that. + +0:04:51.099,0:04:54.569 +Okay so + +0:04:54.569,0:04:59.279 +the other thing is I'm going to assume that +all of you have used the system. I get + +0:04:59.279,0:05:00.910 +really sort of worried when people + +0:05:00.910,0:05:04.249 +you know raise the hands and ""Hey, what's a Shell?"" + +0:05:04.249,0:05:07.990 +or I don't +put a lot of code up but a small piece of code and someone said ""Why + +0:05:07.990,0:05:11.819 +are there two pipe symbols in the middle of +that that If statement?"". + +0:05:11.819,0:05:15.740 +No we're not programming the Shell we're programming +in C. + +0:05:15.740,0:05:19.970 +So hopefully you can tell the difference between +Shell scripts and C code. + +0:05:19.970,0:05:21.990 +okay but I am but am gonna assume + +0:05:21.990,0:05:24.610 +you haven't really looked inside the system. + +0:05:24.610,0:05:28.289 +So I gonna start everything to at a very +high level. + +0:05:28.289,0:05:32.969 +The problem is I have already discovered you come +from a lot of different sort of + +0:05:32.969,0:05:33.819 +backgrounds + +0:05:33.819,0:05:35.180 +and + +0:05:35.180,0:05:36.280 +levels of knowledge + +0:05:36.280,0:05:37.900 +and so + +0:05:37.900,0:05:42.620 +the way that I find works best the sort of +be useful to everybody is that three pass + +0:05:42.620,0:05:43.860 +algorithm + +0:05:43.860,0:05:49.060 +so what I will do is start the first pass a very +broad brush high level + +0:05:49.060,0:05:50.569 +description of what's going on + +0:05:50.569,0:05:54.719 +and then I will go back and i'll go through the +same material again but at a lower level of + +0:05:54.719,0:05:55.300 +detail + +0:05:55.300,0:05:59.939 +then i finally go back and go through a very neatly +low-level of detail + +0:05:59.939,0:06:04.649 +and the fact that is if you are learning new stuff +as I'm doing the high-level thing + +0:06:04.649,0:06:08.649 +you are gonna be utterly washed by the time I get to +low level nearly details + +0:06:08.649,0:06:10.699 +but since I'm going to do it topic by topic + +0:06:10.699,0:06:14.190 +when I get to the end of one of those nearly +low level details + +0:06:14.190,0:06:17.900 +i'll give you a clue as i will say ""Brain +reset, I'm starting a new topic"" so even if + +0:06:17.900,0:06:19.330 +you're completely lost + +0:06:19.330,0:06:23.530 +you can now start listening again plus am gonna get +the broad brush up again. + +0:06:23.530,0:06:27.059 +okay and for those of you that know a lot of +this stuff already + +0:06:27.059,0:06:31.770 +you'll probably find the broad brush rather boring + +0:06:31.770,0:06:35.759 +but by the time i get down nearly low level +details I think you'll actually + +0:06:35.759,0:06:37.860 +pick up some things that you will find + +0:06:37.860,0:06:39.710 +useful and interesting. + +0:06:39.710,0:06:43.759 +So in this way hopefully everybody will +get some + +0:06:43.759,0:06:47.699 +useful percentage of material out of the course. + +0:06:47.699,0:06:49.599 +I am gonna start out by just + +0:06:49.599,0:06:53.089 +walking through and giving you the + +0:06:53.089,0:06:56.919 +outline of what we're going to try and do +here + +0:06:56.919,0:07:01.169 +As i said we're going to go roughly + +0:07:01.169,0:07:03.270 +about two-and-an-half hours of lecture + +0:07:03.270,0:07:04.729 +about two hours forty minutes + +0:07:04.729,0:07:06.499 +per week + +0:07:06.499,0:07:07.619 +and + +0:07:07.619,0:07:11.770 +so we will start off this week with an introduction. + +0:07:11.770,0:07:13.860 +This is as I said we're going to start from the +top + +0:07:13.860,0:07:15.749 +and just start working our way down + +0:07:15.749,0:07:19.350 +so the general thing I'm going to do is +to talk about the interface + +0:07:19.350,0:07:21.439 +%uh which is something that you + +0:07:21.439,0:07:25.319 +are presumably fairly familiar with since +you've worked with that system + +0:07:25.319,0:07:27.249 +and then + +0:07:27.249,0:07:29.739 +sort of layout terminology + +0:07:29.739,0:07:32.080 +although we use normal english words + +0:07:32.080,0:07:34.419 +they have + +0:07:34.419,0:07:38.580 +sometimes rather bizarre meanings compared to their +common usage + +0:07:38.580,0:07:39.220 +and + +0:07:39.220,0:07:42.330 +I will just sort of lay out the terminology +lay out the + +0:07:42.330,0:07:45.750 +the way we talk about how the system is structured + +0:07:45.750,0:07:50.780 +and this week we will also talk about the +basic services ""What is it that the kernel is + +0:07:50.780,0:07:52.929 +providing for us?"" + +0:07:52.929,0:07:54.060 +and then of course + +0:07:54.060,0:07:58.499 +we'll proceed to dive down in and and see how +that is done + +0:07:58.499,0:07:59.970 +so here in + +0:07:59.970,0:08:01.400 +Week 2 + +0:08:01.400,0:08:05.450 +%uh we're gonna look at the system from the +perspective of + +0:08:05.450,0:08:07.039 +something that + +0:08:07.039,0:08:08.720 +manages processes. + +0:08:08.720,0:08:12.170 +One way of looking at the kernel it's really +just a + +0:08:12.170,0:08:16.440 +the resource manager and the resource that +is managing are things going to do with processes + +0:08:16.440,0:08:19.460 +So we'll look at a process, what the structure of +it is + +0:08:19.460,0:08:20.649 +%uh and + +0:08:20.649,0:08:23.559 +talk about the different ways that they can +be structured. + +0:08:23.559,0:08:28.379 +Process can for example be an address space +and can have one thread running in it can have + +0:08:28.379,0:08:29.749 +multiple threads running in it. + +0:08:29.749,0:08:34.620 +we'll talk about the different ways +that we think a process is. + +0:08:34.620,0:08:38.480 +We will look at the management of those processes + + +0:08:38.480,0:08:39.239 +we've got + +0:08:39.239,0:08:42.020 +to lay out the bits and pieces that +need to be managed + +0:08:42.020,0:08:44.660 +%uh and then talk about + +0:08:44.660,0:08:47.190 +how we do that. + +0:08:47.190,0:08:51.740 +we'll talk about jails.. this is something +that you currently find only in FreeBSD + +0:08:51.740,0:08:55.060 +hasn't made it into + +0:08:55.060,0:08:56.320 +Linux yet although + +0:08:56.320,0:09:01.630 +%uh the concept is %uh being actively worked +on so my guess is that you'll see that + +0:09:01.630,0:09:03.500 +%uh fairly soon. + +0:09:03.500,0:09:06.360 +%uh will also then talk about scheduling + +0:09:06.360,0:09:10.579 +which is in essence how we decide what gets +to run, when it gets to run, how long it takes + +0:09:10.579,0:09:13.500 +to run, etc. + +0:09:13.500,0:09:14.330 +okay + +0:09:14.330,0:09:19.020 +The week after that we will go into virtual +memory. + +0:09:19.020,0:09:23.800 +Signals aren't really part of virtual memory +but they didn't fit into next week's + +0:09:23.800,0:09:26.400 +material so I just would drop that at the +beginning + +0:09:26.400,0:09:29.850 +but the bulk of Week 3 is going to +be + +0:09:29.850,0:09:32.019 +the management of Virtual Memory. So we've got + +0:09:32.019,0:09:35.119 +a bunch of physical memory, a bunch of +processes that are + +0:09:35.119,0:09:37.940 +trying to use their address spaces + +0:09:37.940,0:09:39.590 +and we will talk about + +0:09:39.590,0:09:41.410 +essentially how you will make all that work + +0:09:41.410,0:09:43.510 +It's called a virtual memory because it's + +0:09:43.510,0:09:47.420 +sort of a cheat. We promise you the world and +then we deliver you + +0:09:47.420,0:09:51.480 +as small number of pages as we think we +can get away with. + +0:09:51.480,0:09:56.420 +Okay. So the first three weeks then essentially +get us through + +0:09:56.420,0:09:58.340 +looking at the world as if it was all + +0:09:58.340,0:10:00.560 +all about processes + +0:10:00.560,0:10:03.880 +and in Week 4 we change gears. we say +okay well you know + +0:10:03.880,0:10:07.570 +the kernel isn't just about processes. You can sort of look at it orthogonally +and you can + +0:10:07.570,0:10:10.000 +say it's really just a giant I/O switch + +0:10:10.000,0:10:12.910 +it's just like a traffic cop that's just managing +these + +0:10:12.910,0:10:14.860 +I/O streams + +0:10:14.860,0:10:15.450 +and + +0:10:15.450,0:10:18.610 +so let's look at it from that perspective. + +0:10:18.610,0:10:19.310 +And + +0:10:19.310,0:10:24.740 +%uh we'll start with special files, again this +where the interface when you talk about UNIX + +0:10:24.740,0:10:25.880 +systems, when you talk about + +0:10:25.880,0:10:27.950 +what's normally /dev + +0:10:27.950,0:10:34.170 +interface that gets you access +to the various I/O streams that are available + +0:10:34.170,0:10:37.220 +and we'll look at how that's organized and +the structure of it + +0:10:37.220,0:10:41.840 +%uh which is to be fairly simple but in the +last decade has gotten + +0:10:41.840,0:10:43.670 +incredibly complicated. + +0:10:43.670,0:10:48.540 +We will also talk about sudo terminals in +job control + +0:10:48.540,0:10:53.330 +this is about as interesting as watching the +grass grow but unfortunately it's + +0:10:53.330,0:10:55.490 +a major component of the system + +0:10:55.490,0:10:59.520 +and especially people that deal with system +administration have to know far more about + +0:10:59.520,0:11:06.520 +this than they probably ever thought they +wanted to. + +0:11:06.900,0:11:11.430 +Okay we will then continue in Week 5 with +the kernel I/O structure, + +0:11:11.430,0:11:16.090 +%uh We will start with multiplexing of I/O. The +kernel has done this + +0:11:16.090,0:11:17.360 +always + +0:11:17.360,0:11:22.110 +but we will really talking more about how do +we export I/O multiplexing to + +0:11:22.110,0:11:25.970 +user applications. + +0:11:25.970,0:11:29.250 +We will then move into auto configuration strategy + +0:11:29.250,0:11:31.370 +Auto configuration + +0:11:31.370,0:11:32.770 +is what happens + +0:11:32.770,0:11:36.619 +and typically or historically I guess you +could say as the system boots. + +0:11:36.619,0:11:39.500 +so all that stuff that comes out about + +0:11:39.500,0:11:40.810 +what + +0:11:40.810,0:11:43.550 +hardwares are on the machine and how it's all +interconnected + +0:11:43.550,0:11:47.350 +all of that is tied up in auto configuration + +0:11:47.350,0:11:50.040 +and that used to happen just once it boots + +0:11:50.040,0:11:52.000 +but in modern systems today + +0:11:52.000,0:11:55.839 +it's an ongoing process. It happens at boot +but it also happens + +0:11:55.839,0:12:00.550 +anytime you plug a new I/O device, a +PCMCIA card, + +0:12:00.550,0:12:03.680 +you remove a disk or you put in a new disk. + +0:12:03.680,0:12:07.010 +Any sort of activity that changes the the I/O + +0:12:07.010,0:12:08.360 +structure of the machine + +0:12:08.360,0:12:10.870 +auto configuration has to fired back up + +0:12:10.870,0:12:13.050 +and figure out what's disappeared + +0:12:13.050,0:12:18.330 +and cleanup and figure out what new has arrived +to conongiure it in. + +0:12:18.330,0:12:19.320 +and then we'll talk + +0:12:19.320,0:12:23.870 +a little bit about the configuration of the +device driver + +0:12:23.870,0:12:27.390 +this actually gets into an area of that + +0:12:27.390,0:12:28.660 +is + +0:12:28.660,0:12:33.440 +one well let me just give it as a bit +of advice to the class esspecially those of + +0:12:33.440,0:12:36.780 +you who work in system administration. + +0:12:36.780,0:12:42.010 +You'd really want to be careful that +you don't learn too much about device drivers + +0:12:42.010,0:12:44.670 +because there is really these three things that + +0:12:44.670,0:12:48.580 +it's not good to learn about and if you do +learn about it it's really good to keep it + +0:12:48.580,0:12:49.740 +to yourself + +0:12:49.740,0:12:51.949 +because if you become an expert or + +0:12:51.949,0:12:54.960 +viewed as an expert in any of these areas + +0:12:54.960,0:12:59.370 +you will become the designated stocky for +that and your psyche you'll never get to do + +0:12:59.370,0:13:01.760 +anything + +0:13:01.760,0:13:02.610 +but that + +0:13:02.610,0:13:07.360 +The three things that I highly +recommend not learning very much about are + +0:13:07.360,0:13:09.060 +device drivers, + +0:13:09.060,0:13:12.320 +send mail configuration files + +0:13:12.320,0:13:13.970 +or anything having to do + +0:13:13.970,0:13:19.350 +with LDAP or anything in +that general domain + +0:13:19.350,0:13:22.660 +%uh because as I say + +0:13:22.660,0:13:24.900 +that will become your life's work + +0:13:24.900,0:13:25.920 +and + +0:13:25.920,0:13:32.920 +there's other things that you might find more interesting. +""Do you have a question?"" + +0:13:33.870,0:13:36.659 +so one of my students empathized with my point + +0:13:36.659,0:13:39.640 +I believe you said you worked on that mail +system + +0:13:39.640,0:13:43.120 +%uh so you you might know something about +Sum mail configuration files but you don't + +0:13:43.120,0:13:47.850 +have to answer that + +0:13:47.850,0:13:52.100 +so we're going to talk about what a device +driver does and really just the entry + +0:13:52.100,0:13:53.170 +points to it + +0:13:53.170,0:13:57.180 +but we're not going to talk about how you +write such a thing, how you debug such a thing + +0:13:57.180,0:14:01.490 +or much of anything about it. I actually used +to teach an entire class believe it or not + +0:14:01.490,0:14:02.720 +about the device drivers + +0:14:02.720,0:14:05.849 +%uh but then I realized a year i waste +since + +0:14:05.849,0:14:12.580 +i gone through and made a point of forgetting +every slide in that talk. + +0:14:12.580,0:14:16.860 +so then we will move on to Filesystem + +0:14:16.860,0:14:21.540 +and as always will start at the high level +talk about the interface what is it that is + +0:14:21.540,0:14:23.020 +exported out of the system + +0:14:23.020,0:14:27.840 +%uh and then we will start diving down in sea and +how do we go about implementing that + +0:14:27.840,0:14:29.010 +%uh so + +0:14:29.010,0:14:31.010 +we'll start with that + +0:14:31.010,0:14:32.560 +so called + +0:14:32.560,0:14:33.680 +Block I/O system + +0:14:33.680,0:14:36.140 +it's historically been called buffer +cache + +0:14:36.140,0:14:38.590 +and it's still recalled that periodically + +0:14:38.590,0:14:42.720 +and the fact that the matter is that there isn't really +about buffer cache anymore,thers is just one big + +0:14:42.720,0:14:44.620 +cache in it.Its the VM cache + +0:14:44.620,0:14:47.810 +and the Filesystem has a view into it +and + +0:14:47.810,0:14:50.829 +the processes have a view into it but at +the end of the day + +0:14:50.829,0:14:54.660 +you really don't want the same information +on two different + +0:14:54.660,0:14:56.030 +pages of memory + +0:14:56.030,0:14:59.390 +because that just leads to trouble. + +0:14:59.390,0:15:03.390 +But Filesystems think that they have buffers and so +there's been a year where we make + +0:15:03.390,0:15:06.149 +these things that look like what historically +were buffers + +0:15:06.149,0:15:08.830 +that really just happened just really map into VM system + +0:15:08.830,0:15:11.720 +but they're still managed in the way that +they have been + +0:15:11.720,0:15:15.020 +managed historically + +0:15:15.020,0:15:20.670 +We will then get down into Filesystem implementation +the local file system if you will + +0:15:20.670,0:15:23.400 +and into also + +0:15:23.400,0:15:25.730 +soft updates and snapshots. + +0:15:25.730,0:15:26.440 +on this + +0:15:26.440,0:15:31.100 +for the time being is something that you see +only in FreeBSD + +0:15:31.100,0:15:35.310 +the alternative to soft updates is journalling +which is %uh more commonly used + +0:15:35.310,0:15:39.630 +for example what is used by ext3 + +0:15:39.630,0:15:41.179 +so i'll go through soft updates and + +0:15:41.179,0:15:45.260 +a lot of the issues in soft updates are the +same issues that you have to deal with journalling + +0:15:45.260,0:15:48.370 +what is it that we're protecting and how we +go about doing that + +0:15:48.370,0:15:51.150 +and the differences is in the detail. + +0:15:51.150,0:15:54.630 +There is actually a paper in the back to your +notes if this is something that interests + +0:15:54.630,0:15:55.240 +you + +0:15:55.240,0:15:59.930 +%uh it's a comparison of journalling versus +soft updates that was done + +0:15:59.930,0:16:02.120 +about five or eight years ago. + +0:16:02.120,0:16:08.460 +and not to spoil the punch line but the answers +they both work about are the same + +0:16:08.460,0:16:12.500 +Okay the snapshots again is something that +if + +0:16:12.500,0:16:15.920 +you've worked with things like the network +appilance box you're probably quite + +0:16:15.920,0:16:19.640 +aware of what snapshots are and how they do +or don't work for you + +0:16:19.640,0:16:21.959 +this is this the same functionality + +0:16:21.959,0:16:27.380 +%uh in the Filesystem implemented in a +somewhat different way + +0:16:27.380,0:16:28.449 +%uh so this + +0:16:28.449,0:16:31.940 +that Week 6 is really going to be the local +file system + +0:16:31.940,0:16:34.750 +the disk connected to the machine you work. + +0:16:34.750,0:16:39.140 +Week 7 then we get into multiple +Filesystem support so how do we abstract out that + +0:16:39.140,0:16:41.190 +Filesystem layer + +0:16:41.190,0:16:46.430 +and support Multiple Filesystems at the +same time so for example in FreeBSD + +0:16:46.430,0:16:50.199 +you can of course run with their traditional +fast Filesystem + +0:16:50.199,0:16:54.540 +but if you happen to like the Linux Filesystem +better or you have to share a disk + +0:16:54.540,0:16:55.690 +with a Linux machine + +0:16:55.690,0:16:58.310 +you can run the ext2 or ext3 + +0:16:58.310,0:17:01.020 +%uh and it will perfectly happily do that + +0:17:01.020,0:17:01.620 +so + +0:17:01.620,0:17:05.589 +we will have to look then at how do we provide +interface so that we can plug in all these different + +0:17:05.589,0:17:09.260 +Filesystems that we want to support + +0:17:09.260,0:17:12.250 +another area of which there's been a great + +0:17:12.250,0:17:15.309 +deal of growth at least in coding complexity + + +0:17:15.309,0:17:17.840 +is so-called Volume Management + +0:17:17.840,0:17:19.370 +%uh so in the + +0:17:19.370,0:17:24.480 +good old days a Filesystem lived on a disk or +piece of disk and that was that + +0:17:24.480,0:17:26.130 +but this day and age + +0:17:26.130,0:17:31.150 +that won't do any more so we aggregate disks +together by striping them or RAID + +0:17:31.150,0:17:31.980 +arraying them + +0:17:31.980,0:17:33.380 +and we need such really things + +0:17:33.380,0:17:39.210 +and we need a whole layer in the system just to +manage those disks + +0:17:39.210,0:17:44.280 +we'll then get to the as an example of an alternative +Filesystem we're going to talk about the + +0:17:44.280,0:17:46.530 +Network Filesystem or NFS + +0:17:46.530,0:17:48.500 +but that's not because this is + +0:17:48.500,0:17:51.090 +the world's best remote file system + +0:17:51.090,0:17:55.240 +or the cleanest design or any of the thing +properties you might hope that + +0:17:55.240,0:17:57.049 +such a class is this one would have + +0:17:57.049,0:17:58.600 +but it's ubiquitous + +0:17:58.600,0:18:00.210 +very widely used + +0:18:00.210,0:18:01.350 +and + +0:18:01.350,0:18:06.850 +we're going to talk about that one + +0:18:06.850,0:18:07.740 +we'll + +0:18:07.740,0:18:10.970 +then once again switch gears on Week 8 + +0:18:10.970,0:18:17.120 +and turn our attention to of Networking and +Interprocess communication + +0:18:17.120,0:18:18.200 +and + +0:18:18.200,0:18:23.210 +again we'll start from the very top so we'll +go through, we'll go with concepts, the terminology + +0:18:23.210,0:18:24.450 +that gets used + +0:18:24.450,0:18:30.230 +and what's difference between domain +based addressing and an address domain you know + +0:18:30.230,0:18:30.910 +we'll go through + +0:18:30.910,0:18:34.910 +well what the basic IPC services are, + +0:18:34.910,0:18:39.080 +essentially what all the system calls that +has anything to do with networking + +0:18:39.080,0:18:40.590 +and + +0:18:40.590,0:18:43.720 +just sort of describe what each of them are +and I'm going to go through + +0:18:43.720,0:18:45.830 +a somewhat contrived example + +0:18:45.830,0:18:49.840 +it makes use of every one of those interfaces + +0:18:49.840,0:18:52.860 +and just to show how they all conected +together + +0:18:52.860,0:18:54.169 +and for those of you that work + +0:18:54.169,0:18:57.400 +in networking or had done any kind of network +programming + +0:18:57.400,0:19:00.480 +if you're looking for a week to miss and this +Week 8 is the one to miss that's 'cause that's + +0:19:00.480,0:19:02.780 +the sort of most basic + +0:19:02.780,0:19:04.210 +lecture that I'm going to give + +0:19:04.210,0:19:07.910 +If you are not sure whether or not you need to +go through that, there is + +0:19:07.910,0:19:09.540 +one of the papers in the back + +0:19:09.540,0:19:12.620 +it is an introduction to Interprocess communication + +0:19:12.620,0:19:18.279 +%uh read that paper if you say yeah yeah yeah +yeah yeah you are done with Week 8. + +0:19:18.279,0:19:20.590 +on the other hand if you dont come to Week +8 + +0:19:20.590,0:19:22.790 +and then in Week 9 I say + +0:19:22.790,0:19:26.860 +I call on you and say alright what is it + +0:19:26.860,0:19:30.560 +that listen system call does and you +can't tell me + +0:19:30.560,0:19:32.610 +you're gonna get a demerit + +0:19:32.610,0:19:34.340 +okay + +0:19:34.340,0:19:37.770 +then in Week 9 we will get into the actual + +0:19:37.770,0:19:41.419 +networking implementation itself, we go +through system layers as we did + +0:19:41.419,0:19:43.310 +in all the other areas + +0:19:43.310,0:19:44.130 +and + +0:19:44.130,0:19:48.330 +we will spend a significant portion of that +class talking about routing + +0:19:48.330,0:19:50.230 +%uh routing + +0:19:50.230,0:19:53.610 +for those of you that haven't had the pleasure +of dealing with it + +0:19:53.610,0:19:55.540 +is a black heart + +0:19:55.540,0:19:58.050 +or at least a dark science + +0:19:58.050,0:19:59.170 +and + +0:19:59.170,0:19:59.930 +so + +0:19:59.930,0:20:02.490 +we'll talk about it + +0:20:02.490,0:20:06.270 +from the perspective first of all to what +do we do locally within the machine + +0:20:06.270,0:20:10.090 +and then what are some of the bigger strategies +that we can use for doing routing + +0:20:10.090,0:20:11.910 +%uh enterprise + +0:20:11.910,0:20:14.840 +wide routing or + +0:20:14.840,0:20:20.190 +area wide routing some like throughout the +state of California or throughout US whatever + +0:20:20.190,0:20:25.379 +this again like device drivers is really +just sort of a nickel + +0:20:25.379,0:20:26.480 +%uh throught the dark + +0:20:26.480,0:20:27.800 + +0:20:27.800,0:20:31.820 +what what the choices are what that the basic +strategies are that are used + +0:20:31.820,0:20:33.989 +If you're thinking you're going to walk out +of here + +0:20:33.989,0:20:36.110 +knowing how to set up a routing well sorry + +0:20:36.110,0:20:38.430 +we are not going to get that far + +0:20:38.430,0:20:41.559 +but you should at least have a pretty good idea +of what the issues are + +0:20:41.559,0:20:44.430 +and what the general solutions are + +0:20:44.430,0:20:48.950 +then finally in Week 10 well not finally +but next few weeks and + +0:20:48.950,0:20:52.380 +we will go through the Internet Protocols + +0:20:52.380,0:20:54.320 +primarily TCP/IP + +0:20:54.320,0:20:56.560 +and this is + +0:20:56.560,0:20:58.809 +what are the algorithms that are used + +0:20:58.809,0:21:01.030 +and I'm putting a particular emphasis + +0:21:01.030,0:21:03.050 +for this particular class + +0:21:03.050,0:21:05.080 +on + +0:21:05.080,0:21:07.730 +changes that have been made in the protocols + +0:21:07.730,0:21:14.310 +to deal with a lot of attacks that +we've been seeing SYN attacks and + +0:21:14.310,0:21:16.880 +that sort of thing + +0:21:16.880,0:21:19.440 +rather than just a straight + +0:21:19.440,0:21:22.440 +iteration of what the the actual protocols +are + +0:21:22.440,0:21:24.940 +i'll talk primarily about IPv4 + +0:21:24.940,0:21:31.940 +%uh but I will also try and talk a bit about +IPv6 as well + +0:21:33.510,0:21:35.850 +so the first ten weeks are + +0:21:35.850,0:21:38.100 +sort of kernel course + +0:21:38.100,0:21:40.800 +now back with two weeks on at the end + +0:21:40.800,0:21:42.010 +to talk about + +0:21:42.010,0:21:43.990 +through the bigger picture of + +0:21:43.990,0:21:48.240 +System Tuning,Crash dump analysis that level of +thing + +0:21:48.240,0:21:52.940 +The idea is really to consolidate what +we figured out or talked about in the first + +0:21:52.940,0:21:54.710 +ten weeks and + +0:21:54.710,0:21:58.760 +how that applies to tools that are available +to us to + +0:21:58.760,0:22:00.760 +look at what the system is doing, + +0:22:00.760,0:22:02.649 +to analyze what the system is doing + +0:22:02.649,0:22:03.650 +and hopefully + +0:22:03.650,0:22:04.720 +improves + +0:22:04.720,0:22:07.130 +the performance of what the system is doing + +0:22:07.130,0:22:07.750 +and + +0:22:07.750,0:22:12.169 +for the most part the kind of tuning I'm +talking about is not + +0:22:12.169,0:22:14.740 +going in and hack hack hacking your kernel + +0:22:14.740,0:22:16.510 +because the fact that the matter is + +0:22:16.510,0:22:18.600 +you can't do that anyway + +0:22:18.600,0:22:22.340 +%uh so it's more looking at it from the perspective +of saying + +0:22:22.340,0:22:26.390 +is this system running badly because it doesn't +have enough memory on it? + +0:22:26.390,0:22:29.470 +or is it running badly because there isn't enough +I/O capacity? + +0:22:29.470,0:22:33.549 +or is it running badly because the it's got +enough I/O capacity but + +0:22:33.549,0:22:35.940 +certain drives are being overloaded + +0:22:35.940,0:22:37.309 +or is it + +0:22:37.309,0:22:42.220 +being overrun because we're simply trying +to do too much on this machine?,etc. + +0:22:42.220,0:22:45.440 +so that's the sort of level of thing that we're +looking at it + +0:22:45.440,0:22:47.080 +but tied into + +0:22:47.080,0:22:52.130 +lot of concepts that we talked before so we can talk +about active virtual memory + +0:22:52.130,0:22:53.710 +and what that means + +0:22:53.710,0:22:55.120 +and + +0:22:55.120,0:22:58.750 +essentially measure what it is and hopefully +then you will understand the context of what + +0:22:58.750,0:23:00.690 +we talked about in the VM section + +0:23:00.690,0:23:03.990 +what that really means + +0:23:03.990,0:23:07.460 +the Crash dump analysis is is one of these +topics that + +0:23:07.460,0:23:08.730 +you are gonna love or hate + +0:23:08.730,0:23:12.530 +%uh you actually have to deal with crashed +dumps + +0:23:12.530,0:23:13.679 +its people find it invaluable + +0:23:13.679,0:23:15.580 +and if you don't have to deal with Crash dumps + +0:23:15.580,0:23:18.790 +it's an incredible mass of boring detail + +0:23:18.790,0:23:23.240 +the only good part of it is that that's the +whole session is only about an hour or long + +0:23:23.240,0:23:25.529 +If it interests you, listen closely + +0:23:25.529,0:23:28.950 +and if it bores you, well, its only an hour long + +0:23:28.950,0:23:32.880 +okay up lastly we'll talk a little bit about +security issues + +0:23:32.880,0:23:36.250 +again this is really more of the tools that +are available + +0:23:36.250,0:23:40.750 +to deal with security staff as opposed to a +complete tutorial on + +0:23:40.750,0:23:45.120 +how to implement security so those of you +that deal with security + +0:23:45.120,0:23:48.400 +it's just gonna to be sort of security one on +one + +0:23:48.400,0:23:50.029 +%uh for those of you + +0:23:50.029,0:23:51.500 +that have but + +0:23:51.500,0:23:54.399 +you'll have to deal with it but haven't really +thought about it + +0:23:54.399,0:23:58.549 +%uh it'll probably scare you to death and +you wonder have to keep the machines from + +0:23:58.549,0:24:02.840 +being hijacked everyday + +0:24:02.840,0:24:08.030 +Okay that's in essence what we're going +to try and do here + +0:24:08.030,0:24:15.030 +anybody have any comments, questions, thoughts. +Alright, no, well. + +0:24:16.130,0:24:17.840 +Let's get started + +0:24:17.840,0:24:22.180 +we will be begin on page fifteen with an +overview of the kernel. + +0:24:22.180,0:24:26.040 +Hopefully nobody's lost yet. + +0:24:26.040,0:24:29.310 +What's a kernel? Alright. + +0:24:29.310,0:24:31.370 +so starting at the very top + +0:24:31.370,0:24:33.070 +the big broad brush + +0:24:33.070,0:24:35.140 +what we have is + +0:24:35.140,0:24:38.330 +a UNIX virtual machine and + +0:24:38.330,0:24:41.660 +and virtual machines are actually something +that has been around + +0:24:41.660,0:24:44.539 +as a concept since the sixties + +0:24:44.539,0:24:48.919 +the diffrence is really just sort of the level +of the interface that people have dealt with + +0:24:48.919,0:24:51.360 +when they talk about Virtual Machines + +0:24:51.360,0:24:53.610 +in the 1960s + +0:24:53.610,0:24:56.770 +computers were these enormous things you would +have + +0:24:56.770,0:24:58.870 +your computer room would be something that be + +0:24:58.870,0:25:01.909 +three times the size of this conference +room if you had + +0:25:01.909,0:25:03.230 +a computer + +0:25:03.230,0:25:05.530 +the computer itself was this + +0:25:05.530,0:25:07.840 +tall as a refrigerator freezer + +0:25:07.840,0:25:08.950 +imagine + +0:25:08.950,0:25:13.909 +five or eight or ten of these units +side by side that itself made up the computer + +0:25:13.909,0:25:16.080 +that would be one + +0:25:16.080,0:25:20.030 +for the core processor and the one which +should be the floating point unit and several + +0:25:20.030,0:25:24.080 +of them that would be the memory the core +literally + +0:25:24.080,0:25:29.110 +%uh and then they'd be other rows of these +disk drives about the size of a washing + +0:25:29.110,0:25:29.660 +machine + +0:25:29.660,0:25:34.169 +%then behind that since you couldn't store +everything on disk so + +0:25:34.169,0:25:36.300 +that you had rows of tape drives + +0:25:36.300,0:25:37.880 +and then you had this little + +0:25:37.880,0:25:39.610 +set of sort of + +0:25:39.610,0:25:43.330 +munchkins that would run around and and tend +to the machine and they mount tapes and take + +0:25:43.330,0:25:46.710 +off tapes and mount disc packs and remove disc packs +because + +0:25:46.710,0:25:49.760 +the drives themselves were very expensive and +so + +0:25:49.760,0:25:53.110 +you wouldn't just as today we have a + + +0:25:53.110,0:25:56.090 +one spindle that was dedicated just to one set +of platters + +0:25:56.090,0:25:57.130 +you could take out a + +0:25:57.130,0:25:59.460 +set of platters and put in another + +0:25:59.460,0:26:02.540 +hundred megabytes of platters and these are +platters that are + +0:26:02.540,0:26:05.280 +this big around and it's like six or eight +of them and + +0:26:05.280,0:26:09.140 +a giant head assemblies they comes rumbling in and +out + +0:26:09.140,0:26:12.440 +anyway one of these giant giant machines + +0:26:12.440,0:26:17.380 +that costs many millions of dollars would run +at about ten + +0:26:17.380,0:26:21.120 +million instructions per second, 10 mips + +0:26:21.120,0:26:21.630 +and 10 mips + +0:26:21.630,0:26:28.330 +that's was more computing power than anybody +could possibly imagine using in a single application + +0:26:28.330,0:26:28.880 +just + +0:26:28.880,0:26:31.050 +by contrast this this + +0:26:31.050,0:26:34.070 +four-year-old laptop here is probably on +the order of + +0:26:34.070,0:26:36.440 +one or two hundred mips + +0:26:36.440,0:26:37.140 +but anyway + +0:26:37.140,0:26:40.760 +people couldn't really view what we would +do with a lot of computing power + +0:26:40.760,0:26:44.640 +and the other thing was that you didn't have +a notion of sort of operating system that had + +0:26:44.640,0:26:45.890 +applications running on it + +0:26:45.890,0:26:46.760 +because + +0:26:46.760,0:26:50.160 +everybody wanted to write straight to +the raw hardware + +0:26:50.160,0:26:51.750 +and so + +0:26:51.750,0:26:55.900 +IBM who was a big manufacturer +of machines in those days + +0:26:55.900,0:26:59.060 +did what they came up with this thing called +the VM + +0:26:59.060,0:27:00.770 +and this was a little + +0:27:00.770,0:27:02.549 +you'd call an operating system really + +0:27:02.549,0:27:05.130 +but what it did is a cloned + +0:27:05.130,0:27:09.270 +independent copies of the machine worked just +like the original machines you could boot + +0:27:09.270,0:27:11.769 +something that thought it was an operating +system + +0:27:11.769,0:27:13.380 +on top of VM + +0:27:13.380,0:27:16.750 +so you take one least ten that machines and +it would clone + +0:27:16.750,0:27:20.050 +six identical one mip copies + +0:27:20.050,0:27:22.030 +%uh and then you could boot + +0:27:22.030,0:27:24.700 +whatever you wanted on each one of those machines +so + +0:27:24.700,0:27:29.510 +if you were doing database that you do your +database because database cannot run on the raw hardware + +0:27:29.510,0:27:32.920 +or if you're doing payroll who would boot up the payroll +program + +0:27:32.920,0:27:37.950 +%uh or if you actually tried to service the +users you could boot a time sharing batch thing + +0:27:37.950,0:27:40.790 +that would read card images and print +stuff out + +0:27:40.790,0:27:44.460 +or they even had TSO the Time Sharing +Option where you could interactively sit + +0:27:44.460,0:27:45.559 +and type send + +0:27:45.559,0:27:47.560 +stuffs in and get answers back + +0:27:47.560,0:27:48.570 +%uh and + +0:27:48.570,0:27:51.429 +%uh also you could boot TSO so whatever set +of + +0:27:51.429,0:27:52.219 +%um + +0:27:52.219,0:27:55.339 +things you need you could boot them and they ran +independently as if they were running on their + +0:27:55.339,0:27:56.470 +own machine + +0:27:56.470,0:28:03.150 +but all the VM did was it give you an exact +raw copy of the hardware + +0:28:03.150,0:28:04.529 +so when UNIX came along + +0:28:04.529,0:28:07.350 +they sort of like the notion of + +0:28:07.350,0:28:11.509 +providing the concept of independent +things that you could operate in + +0:28:11.509,0:28:13.610 +but they wanted it at a higher level + +0:28:13.610,0:28:15.610 +so you're looking merely to do that + +0:28:15.610,0:28:17.480 +instead of at the raw hardware level + +0:28:17.480,0:28:19.679 +to do it at a process well + +0:28:19.679,0:28:23.799 +and the idea that then was that the interface you +would program to would be what we think of as + +0:28:23.799,0:28:26.090 +a System call interface today + +0:28:26.090,0:28:27.849 +and the idea then was that + +0:28:27.849,0:28:30.740 +you would be given a process or set of processes + +0:28:30.740,0:28:34.990 +and those were independent your process +couldn't affect + +0:28:34.990,0:28:38.830 +the address space of another process reach +over and mess around with the addresses, + +0:28:38.830,0:28:41.030 +you couldn't mess around with their I/O +channels + +0:28:41.030,0:28:43.179 +%uh you could slow them down by + +0:28:43.179,0:28:44.299 +being a pig but + +0:28:44.299,0:28:47.980 +that was about the only way that you could affect +other processes + +0:28:47.980,0:28:48.480 +and + +0:28:48.480,0:28:49.830 +%uh so + +0:28:49.830,0:28:52.669 +what the interfaces that they had there + +0:28:52.669,0:28:58.660 +with one that had these characteristics +he had a a paged virtual address space + +0:28:58.660,0:29:02.980 +so you din't have to know as in the old days how much physical +memory is on the machine and make your application + +0:29:02.980,0:29:04.740 +fit into that amount of memory + +0:29:04.740,0:29:07.950 +you just had what looked like a large + +0:29:07.950,0:29:11.710 +uniform address space even if the underlying +hardware had segments or some other + +0:29:11.710,0:29:13.580 +hardware brain damage + +0:29:13.580,0:29:17.390 +it look to you like he just had a big uniform +address space and + +0:29:17.390,0:29:21.070 +the size of the address space was independent +of the amount of memory that was on the machine + +0:29:21.070,0:29:23.900 +your address space could be bigger than amount of +physical memory + +0:29:23.900,0:29:26.499 +cause we got to move pages around underneath + +0:29:26.499,0:29:29.320 +whatever part address space was actually +active + +0:29:29.320,0:29:34.260 +that is obviously do this if you if +you are trying to run 1 gigabyte of + +0:29:34.260,0:29:35.630 +application on top of + +0:29:35.630,0:29:37.240 +ten megabytes of memory + +0:29:37.240,0:29:40.880 +it's probably going to bring new meaning to +same day service + +0:29:40.880,0:29:45.519 +but if you're willing to wait long enough it +will eventually move the pages around and you will + +0:29:45.519,0:29:49.740 +progress through getting your application run + +0:29:49.740,0:29:53.890 +anothe thing was dealing with software +interrupts + +0:29:53.890,0:29:55.789 +%uh in the old days + +0:29:55.789,0:29:58.749 +you had to understand how the hardware worked + +0:29:58.749,0:30:03.900 +in order to deal with exceptional conditions +so for example to did a divide by zero + +0:30:03.900,0:30:08.170 +the the hardware would have jumped to some +vector location or + +0:30:08.170,0:30:08.630 +something + +0:30:08.630,0:30:12.799 +and you had no khow how that worked and make +sure that you had your program + +0:30:12.799,0:30:16.510 +use just a little bit of assembly language which +set up to deal with that + +0:30:16.510,0:30:19.870 +and UNIX said let's let's get away +from the hardware here + +0:30:19.870,0:30:22.080 +and so they did this thing called signals + +0:30:22.080,0:30:25.700 +the answer is, the finest of the signals is that +if you do divide by zero + +0:30:25.700,0:30:29.529 +%uh you simply register a routine you +want to have called you don't have to know + +0:30:29.529,0:30:31.220 +how the hardware figured it out + +0:30:31.220,0:30:36.740 +you just know that that routine is going to get +called and you can deal with it at that point + +0:30:36.740,0:30:40.960 +well we got set of timers and counters to keep +track of what we're doing, this is really more + +0:30:40.960,0:30:43.490 +for counting than anything else but + +0:30:43.490,0:30:46.970 +applications may want to have access to that. + +0:30:46.970,0:30:51.720 +we have a set of identifiers that we're +going to use for things like accounting, + +0:30:51.720,0:30:54.830 +protection and scheduling and so on + +0:30:54.830,0:30:55.820 +and one of the + +0:30:55.820,0:31:00.320 +the early philosophies of UNIX was that to try +and keep it simple. + +0:31:00.320,0:31:02.630 +operating systems have gotten very baroque + +0:31:02.630,0:31:04.490 +in particular the thing that + +0:31:04.490,0:31:07.350 +pre dated UNIX was a thing called +Multix + +0:31:07.350,0:31:12.820 +%uh Multix was was a joint project between +Honeywell, a big computer manufacturer of the + +0:31:12.820,0:31:15.740 +time at + +0:31:15.740,0:31:17.129 +AT&T bell laboratories + +0:31:17.129,0:31:19.750 +the big industrial labratory at that time + +0:31:19.750,0:31:21.380 +and MIT + +0:31:21.380,0:31:23.430 +a big university then and + +0:31:23.430,0:31:24.690 +still today + +0:31:24.690,0:31:29.259 +%uh and that those three organizations got +together to try and build this + +0:31:29.259,0:31:31.400 +time sharing operating system + +0:31:31.400,0:31:32.280 +and it + +0:31:32.280,0:31:33.770 +it just got bigger and + +0:31:33.770,0:31:37.160 +more grandiose and more complex and never +finished + +0:31:37.160,0:31:38.979 +but as soon as that's what we see + +0:31:38.979,0:31:42.709 +although we know how to do that but also we could +do this other thing too and so then the teared it + +0:31:42.709,0:31:43.429 +apart and + +0:31:43.429,0:31:46.440 +they never really got to something that + +0:31:46.440,0:31:48.210 +could be put into production + +0:31:48.210,0:31:49.919 +and so the + +0:31:49.919,0:31:50.570 +AT&T + +0:31:50.570,0:31:54.340 +Bell laboratories decided to pull out of +that project + +0:31:54.340,0:31:55.940 +and + +0:31:55.940,0:32:00.000 +the two of the people that had been working on +that project, Ken Thompson and Dennis Richie + +0:32:00.000,0:32:04.390 +%uh were sort of bound because they were now +back to typing cards and putting them through + +0:32:04.390,0:32:05.259 +card reader and + +0:32:05.259,0:32:07.960 +they are just used to the idea that you could +actually + +0:32:07.960,0:32:11.559 +sit at an ASSR teletype and interact +with your computer + +0:32:11.559,0:32:13.440 +and so + +0:32:13.440,0:32:18.230 +they found an old %uh PDP-8 soon off in +the quarter that have been abandoned + +0:32:18.230,0:32:22.120 +and started working on this little tiny operating +system which they called UNIX + +0:32:22.120,0:32:26.549 +%uh which eventually moved to the PDP-11 and +became what we have today + +0:32:26.549,0:32:28.050 +but because it was + +0:32:28.050,0:32:32.120 +they were coming first of all from Multix +where everything had been done and + +0:32:32.120,0:32:34.110 +in great grandiose detail + +0:32:34.110,0:32:37.549 +and because they're fundamentally just two +the of them working on it and they wanted to get something + +0:32:37.549,0:32:38.370 +done and + +0:32:38.370,0:32:40.130 +within a year or so + +0:32:40.130,0:32:41.529 +one of their philosophies was + +0:32:41.529,0:32:44.099 +let's find out one way of doing things + +0:32:44.099,0:32:48.180 +let's not have eight ways from Sunday let's just +get the one way + +0:32:48.180,0:32:53.860 +and and that's what we will provide. So what is +the sort of core set of things that we need. + +0:32:53.860,0:32:58.620 +well first thing is when it comes to identifiers, +let's not have you know + +0:32:58.620,0:33:00.430 +eighty thousand different identifiers + +0:33:00.430,0:33:03.140 +so they came up with process identifiers, + +0:33:03.140,0:33:09.620 +user identifier and at that time a single group +identifier and later expanded + +0:33:09.620,0:33:14.200 +and they used that sort of identifies for everything +so its used for counting, used for main + +0:33:14.200,0:33:17.410 +protection decisions, used for scheduling +decisions + +0:33:17.410,0:33:19.470 +and + +0:33:19.470,0:33:24.279 +again it was the simplicity of thing which +was what was driving their decision + +0:33:24.279,0:33:28.840 +but they're really sort of true key idea +that they had + +0:33:28.840,0:33:30.880 +that really made a difference that + +0:33:30.880,0:33:32.539 +that's were set them up side + +0:33:32.539,0:33:34.749 +from what everybody else done before them + +0:33:34.749,0:33:35.450 +and which + +0:33:35.450,0:33:39.740 +in retrospect is something that has been pervasive +more or less ever since + +0:33:39.740,0:33:41.869 +the first of these was the notion + +0:33:41.869,0:33:44.840 +that we have a unique descriptive space + +0:33:44.840,0:33:46.289 +%uh that is + +0:33:46.289,0:33:51.250 +given a descriptor it can reference +any I/O device + +0:33:51.250,0:33:53.650 +or even any kind of I/O channel + +0:33:53.650,0:33:58.270 +so you can have a descriptor for terminal +or descriptor for file or descriptive for + +0:33:58.270,0:34:02.240 +a disk or descriptor for pipe or descriptor +for a socket + +0:34:02.240,0:34:03.500 +and + +0:34:03.500,0:34:04.790 +you don't need to know + +0:34:04.790,0:34:07.940 +what it references in order to be able to read +and write on that thing + +0:34:07.940,0:34:11.290 +so if i hand you a descriptor +can read from that the descriptor or you can write + +0:34:11.290,0:34:13.259 +to that descriptor + +0:34:13.259,0:34:15.189 +and + +0:34:15.189,0:34:17.359 +the correct thing will happen + +0:34:17.359,0:34:19.089 +and you'd say well + +0:34:19.089,0:34:23.629 +that's obvious I mean how else could you +possibly think about doing it + +0:34:23.629,0:34:25.179 +predating UNIX + +0:34:25.179,0:34:28.059 +everything was done with + +0:34:28.059,0:34:29.379 +a little subsystem + +0:34:29.379,0:34:33.419 +that would open a file, read a file, write a +file, close a file + +0:34:33.419,0:34:37.429 +and there was another set of system calls which +would open a terminal,read a terminal, write terminal + +0:34:37.429,0:34:38.089 +,close terminal + +0:34:38.089,0:34:39.210 +and yet another one + +0:34:39.210,0:34:42.409 +which was create a pipe,read a pipe, +write a pipe and so on. + +0:34:42.409,0:34:47.699 +so if you are just dropped at a stupid +program like CAD + +0:34:47.699,0:34:51.579 +you would have to have code in there and say was +my input a terminal which in case I need to + +0:34:51.579,0:34:53.159 +use the read terminal + +0:34:53.159,0:34:57.419 +or is it a file which in case i need +to use read file or is it a pipe in which in case + +0:34:57.419,0:34:59.189 +i need to use read pipe + +0:34:59.189,0:35:01.860 +and so the program itself had to have all +this + +0:35:01.860,0:35:02.859 +coding in it + +0:35:02.859,0:35:04.409 +whereas when they went to + +0:35:04.409,0:35:07.159 +the uniform descriptor space + +0:35:07.159,0:35:09.630 +CAD doesn't know and doesn't need to know +it just says + +0:35:09.630,0:35:10.819 +read my input, + +0:35:10.819,0:35:13.979 +write the output + +0:35:13.979,0:35:17.059 +it works and we add a new type of descriptor + +0:35:17.059,0:35:17.600 +and + +0:35:17.600,0:35:21.700 +CAD just continues to work just as it always +did. + +0:35:21.700,0:35:24.199 +So this proved to be a very powerful construct + +0:35:24.199,0:35:27.019 +and pretty much every operating system after +UNIX + +0:35:27.019,0:35:28.659 +did that and there's + +0:35:28.659,0:35:30.210 +one exception of %uh + +0:35:30.210,0:35:32.549 +large company in the Pacific North-West + +0:35:32.549,0:35:35.830 +that still has not quite uniform descriptor +space + +0:35:35.830,0:35:38.380 +but %uh that's part of their legacy that really + +0:35:38.380,0:35:39.900 +they're working on that. + +0:35:39.900,0:35:42.009 +Longhorn will be here. + +0:35:42.009,0:35:43.939 +and anyway + +0:35:43.939,0:35:46.190 +this set of facilities + +0:35:46.190,0:35:50.150 +makes up the UNIX virtual machine + +0:35:50.150,0:35:51.559 +and + +0:35:51.559,0:35:55.559 +in sometimes we still see virtual machines +being used today in fact we're seeing sort + +0:35:55.559,0:35:56.749 +of a reversion + +0:35:56.749,0:36:01.429 +%uh back to some of the ideas stop in things +like the VMware + +0:36:01.429,0:36:03.079 +which is + +0:36:03.079,0:36:07.029 +essentially allow you to go back to booting +native operating systems again so sort of + +0:36:07.029,0:36:08.280 +interesting to watch + +0:36:08.280,0:36:09.060 +that's the sort of + +0:36:09.060,0:36:12.919 +%uh %uh pendulum of back going back and forth +of what's the correct layer + +0:36:12.919,0:36:14.609 +for for doing + +0:36:14.609,0:36:18.890 +%uh virtual machines + +0:36:18.890,0:36:22.499 +Okay. so far so good. + +0:36:22.499,0:36:24.719 +all right i said there was + +0:36:24.719,0:36:27.160 +two key ideas that UNIX had + +0:36:27.160,0:36:30.279 +the first thing is uniform descriptor +space + +0:36:30.279,0:36:35.819 +the second one which was really critical was +this notion of processes as a commodity + +0:36:35.819,0:36:37.309 +item + +0:36:37.309,0:36:40.220 +so here on Page 17 I tried to lay +it out + +0:36:40.220,0:36:41.090 +the + +0:36:41.090,0:36:44.159 +that the components that make up a process + +0:36:44.159,0:36:45.759 +%uh and + +0:36:45.759,0:36:50.359 +what do I really mean when I say a process as +a commodity item + +0:36:50.359,0:36:53.650 +leading up to + +0:36:53.650,0:36:54.689 +UNIX + +0:36:54.689,0:36:56.800 +the systems that pre-dated it, + +0:36:56.800,0:36:59.200 +processes were these very large + +0:36:59.200,0:37:02.169 +heavyweight expensive thing + +0:37:02.169,0:37:02.779 +and + +0:37:02.779,0:37:04.539 +if you look at + +0:37:04.539,0:37:08.629 +MVS which was the operating system +that ran on IBM for doing multiple processing + +0:37:08.629,0:37:10.509 +and + +0:37:10.509,0:37:13.799 +the system administrator would decide at boot +time + +0:37:13.799,0:37:17.019 +what degree of multiprocessing they wish +to support + +0:37:17.019,0:37:18.140 +so they'd say well + +0:37:18.140,0:37:20.739 +well, let upto six things happen at once + +0:37:20.739,0:37:22.490 +and so as part of booting up + +0:37:22.490,0:37:24.419 +they would create six + +0:37:24.419,0:37:25.349 +processes + +0:37:25.349,0:37:30.059 +and now you as a user if you wanted to do +something let's say you wanted to + +0:37:30.059,0:37:32.009 +compile and ran a program + +0:37:32.009,0:37:34.960 +%uh you would be given a process + +0:37:34.960,0:37:36.019 +and it's up to you + +0:37:36.019,0:37:39.369 +to figure out how to stage what you needed +done + +0:37:39.369,0:37:39.819 +and + +0:37:39.819,0:37:43.930 +%uh that this was often fairly complex + +0:37:43.930,0:37:47.880 +and so you would have to write out all the +steps that you wanted + +0:37:47.880,0:37:50.300 +in this wonderful thing called JCL + +0:37:50.300,0:37:52.259 +Job Control Language and + +0:37:52.259,0:37:56.650 +Job Control Language was send mail configuration +file of the sixties + +0:37:56.650,0:38:00.679 +other where people who sold job that the company +was how to put this stuff together 'cause + +0:38:00.679,0:38:04.189 +like to do is get what one extra space or +missing comma + +0:38:04.189,0:38:05.000 +something in their + +0:38:05.000,0:38:08.630 +and I hope it would just blow up it would +just sort of spit the card deck back at + +0:38:08.630,0:38:09.799 +you and say well + +0:38:09.799,0:38:13.500 +somewhere in there is a mistake that's sort of +the general area of this card + +0:38:13.500,0:38:15.549 +and I can't deal with it. Fix it. + +0:38:15.549,0:38:16.489 +and of course + +0:38:16.489,0:38:20.550 +in those days it wasn't just a matter of hitting +carriage when you know make cariage return you have to + +0:38:20.550,0:38:25.239 +get your deck pull out the card, and type the +new one, put it back in and re-submit it + +0:38:25.239,0:38:28.729 +As heaven forbid you couldnt touch that +card reader you know, it had to be done by + +0:38:28.729,0:38:29.970 +an operator + +0:38:29.970,0:38:32.869 +so the card deck will read through it would +disappear and + +0:38:32.869,0:38:36.800 +you know if you're lucky a few minutes later +if you were not lucky a few hours later + +0:38:36.800,0:38:37.849 +you would get + +0:38:37.849,0:38:39.570 +a print print out + +0:38:39.570,0:38:43.419 +which was what it happened and then you could +look at it and you know + +0:38:43.419,0:38:47.209 +I put a comma in the wrong place I guess +I get to do it all again + +0:38:47.209,0:38:49.930 +so + +0:38:49.930,0:38:54.940 +if you would need to do something +like compiling and running a program + +0:38:54.940,0:38:59.579 +was you'd have to break into these steps. well +I need to run the the preprocessor + +0:38:59.579,0:39:04.670 +and so us clean out whatever gump that was left +over on that process from the previous user + +0:39:04.670,0:39:06.240 +put the preprocessor in there + +0:39:06.240,0:39:10.530 +%uh and then read from this file here let's +say gotta put it somewhere so creative + +0:39:10.530,0:39:12.510 +scratch file over on this disk and + +0:39:12.510,0:39:17.299 +it was excruciating detail like how many cylinders +and how many tracks and this and that + +0:39:17.299,0:39:19.139 +blocks blah blah blah + +0:39:19.139,0:39:23.119 +and don't forget any of those parameters 'cause +it'll spit it out if you do + +0:39:23.119,0:39:26.890 +and so then it would run the first step in that +if its successful then you'd have sitting + +0:39:26.890,0:39:28.899 +in this scratch file that you had created + +0:39:28.899,0:39:33.100 +the output of the preprocessor and then +you'd load the first pass of the compiler + +0:39:33.100,0:39:36.930 +and lets now read from that scratch file +and create this other scratch file over here and + +0:39:36.930,0:39:39.450 +when thats successful and we need to delete that +one + +0:39:39.450,0:39:43.830 +and then load the second pass, put that back +into another scratch file and then we run this + +0:39:43.830,0:39:45.950 +assembler, optimizer then the + +0:39:45.950,0:39:47.750 +loader this and that + +0:39:47.750,0:39:49.410 +finally run the program + +0:39:49.410,0:39:50.900 +and %uh if all goes well + +0:39:50.900,0:39:57.029 +you know at step sixteen out comes the answer + +0:39:57.029,0:39:58.129 +fourty two. so UNIX + +0:39:58.129,0:40:00.819 +said, look this is silly + +0:40:00.819,0:40:02.880 +a lot of this is just + +0:40:02.880,0:40:04.310 +bookkeeping + +0:40:04.310,0:40:07.249 +and computers do bookkeeping really well + +0:40:07.249,0:40:12.179 +and you'll recall yeah but it's going to take +all these cycles it's like + +0:40:12.179,0:40:16.309 +computers are supposed to be labor-saving +devices right so + +0:40:16.309,0:40:20.150 +they came up with this notion that they would +create processes on the fly as needed + +0:40:20.150,0:40:21.159 +you had + +0:40:21.159,0:40:25.549 +you've had a preprocessor in two +steps of the compiler and then + +0:40:25.549,0:40:27.109 +optimizer and then a loader + +0:40:27.109,0:40:29.410 +we just create Boom seven processes + +0:40:29.410,0:40:31.920 +and we connect them together with pipes + +0:40:31.920,0:40:35.180 +and so we take the input and you know run +through in + +0:40:35.180,0:40:38.270 +through the pipes and you know out the end +you get the the + +0:40:38.270,0:40:39.629 +executable + +0:40:39.629,0:40:40.030 +and + +0:40:40.030,0:40:42.880 +we simply create each of these processes + +0:40:42.880,0:40:44.650 +and + +0:40:44.650,0:40:46.549 +so you as a user just + +0:40:46.549,0:40:49.479 +type you know the C compiler just + +0:40:49.479,0:40:52.429 +fork these things pipe them together got the result + +0:40:52.429,0:40:53.640 +and + +0:40:53.640,0:40:57.509 +then once it was done with this processes is +just threw them away so any time you'd create a + +0:40:57.509,0:41:00.479 +new process and it came to you christine clean + +0:41:00.479,0:41:04.239 +and you needed a bunch of things it did +put everything in intermediate files + +0:41:04.239,0:41:07.549 +the fact that matter is in the early days + +0:41:07.549,0:41:08.129 +those computers + +0:41:08.129,0:41:11.910 +didn't really have enough memory to support +all that stuff at once so + +0:41:11.910,0:41:15.809 +behind you those pipes were actually implemented +as files + +0:41:15.809,0:41:19.319 +you didn't have atleast to remember to create +them and delete them + +0:41:19.319,0:41:20.200 +and deal with them + +0:41:20.200,0:41:24.020 +as far as you're concerned it just look stuff flowing through pipes +and of course today it + +0:41:24.020,0:41:24.490 +just + +0:41:24.490,0:41:27.989 +does flow through pipes in memory + +0:41:27.989,0:41:29.439 +okay so + +0:41:29.439,0:41:33.689 +this notion then that that we're just gonna +create processes on the fly is needed and + +0:41:33.689,0:41:35.559 +connect them together as needed + +0:41:35.559,0:41:38.039 +and that was a novel concept + +0:41:38.039,0:41:43.599 +and it wasn't that somehow mysteriously figured +out how to create processes cheaply + +0:41:43.599,0:41:44.839 +cause they hadn't + +0:41:44.839,0:41:46.180 +they were still + +0:41:46.180,0:41:49.959 +really expensive to create + +0:41:49.959,0:41:52.210 +but that extra effort + +0:41:52.210,0:41:53.029 +was + +0:41:53.029,0:41:56.089 +worth it because it was saving a lot of programming +time + +0:41:56.089,0:41:59.809 +so my favorite example is you run ls + +0:41:59.809,0:42:01.810 +so we have to create a process + +0:42:01.810,0:42:04.259 +load the ls binary into it + +0:42:04.259,0:42:06.180 +and it prints a line or two on your screen + +0:42:06.180,0:42:10.609 +and we tear the entire thing down and return +all its resources back to the system + +0:42:10.609,0:42:14.979 +more than ninety percent of the cost of running +ls is creating and destroying the process + +0:42:14.979,0:42:19.239 +a tiny fraction of it is actually running +ls + +0:42:19.239,0:42:24.259 +it goes so fast, who cares right + +0:42:24.259,0:42:25.749 +the point is that + +0:42:25.749,0:42:30.039 +%uh that concept of just creating things as +needed + +0:42:30.039,0:42:31.780 +again was very powerful + +0:42:31.780,0:42:35.709 +and is one that is just pervasive today + +0:42:35.709,0:42:38.639 +okay so what is a process actually made up +of + +0:42:38.639,0:42:43.179 +%uh it gets some amount of CPU time or at +least we do really hope that it gets some + +0:42:43.179,0:42:46.050 +amount of CPU time, the lack of getting +CPU time + +0:42:46.050,0:42:46.670 +that makes it + +0:42:46.670,0:42:47.979 +a pure so sluggish + +0:42:47.979,0:42:49.409 +of course + +0:42:49.409,0:42:51.920 +others really boils down to scheduling + +0:42:51.920,0:42:54.249 +and we're going to talk about scheduling + +0:42:54.249,0:42:56.279 +probably more than you care to + +0:42:56.279,0:42:59.219 +in a couple weeks time + +0:42:59.219,0:43:01.619 +we have the asynchronous events + +0:43:01.619,0:43:04.569 +%uh these are the external events that + +0:43:04.569,0:43:05.659 +are coming in + +0:43:05.659,0:43:07.679 +%uh so + +0:43:07.679,0:43:10.169 +they may be either things that + +0:43:10.169,0:43:14.339 +we're coming in from the outside world like +start, stop and quit + +0:43:14.339,0:43:15.279 +%uh oh + +0:43:15.279,0:43:18.170 +out-of-band arrival notification that kind +of thing + +0:43:18.170,0:43:22.339 +or it may in fact be things that the program +is bringing down upon itself + +0:43:22.339,0:43:25.590 +on such as a segment fault,a divide by zero + +0:43:25.590,0:43:26.910 +and some other + +0:43:26.910,0:43:31.959 +what would normally be viewed as incorrect +operation + +0:43:31.959,0:43:35.849 +and so will talk about that when we talk about +signals + +0:43:35.849,0:43:37.039 +every program + +0:43:37.039,0:43:38.899 +get some amount of memory + +0:43:38.899,0:43:42.659 +it gets an initial amount when it starts +up injured generally allocates more as it + +0:43:42.659,0:43:45.229 +goes along + +0:43:45.229,0:43:49.429 +of course we will deal with up her extensively +will spend an entire week on it + +0:43:49.429,0:43:54.249 +when we talk about how virtual memory is implemented + +0:43:54.249,0:43:54.609 +and + +0:43:54.609,0:43:57.429 +%uh then we get I/O descriptors + +0:43:57.429,0:44:02.259 +I used to say that every program had to have +at least one I/O descriptor since + +0:44:02.259,0:44:04.910 +it absolutely had no input + +0:44:04.910,0:44:06.329 +absolutely no output + +0:44:06.329,0:44:09.049 +then it was sort of pointless + +0:44:09.049,0:44:12.900 +of course I had to have one of my students +come up and point out to me there is a + +0:44:12.900,0:44:13.849 +class of programs + +0:44:13.849,0:44:16.469 +which don't need I/O descriptors + +0:44:16.469,0:44:17.440 +and that is + +0:44:17.440,0:44:19.549 +these things called benchmarks + +0:44:19.549,0:44:23.249 +the just compute something all we really care +about is how long it takes them to compute + +0:44:23.249,0:44:24.959 +we dont actually care what the answer is + +0:44:24.959,0:44:26.019 +In theory we dont + +0:44:26.019,0:44:29.779 +I personally felt like my benchmark stop with +something so I can see it there + +0:44:29.779,0:44:31.489 +doing computing the right thing + +0:44:31.489,0:44:33.169 +but in theory + +0:44:33.169,0:44:35.919 +that wouldn't be necessary + +0:44:35.919,0:44:38.650 +outside of that class of programs + +0:44:38.650,0:44:42.670 +everything needs some sort of descriptors and +of course we'll talk about descriptors + +0:44:42.670,0:44:43.659 +quite extensively + +0:44:43.659,0:44:47.349 +%uh as we go through the I/O subsystem + +0:44:47.349,0:44:50.969 +okay so the executive summary is that processes +are + +0:44:50.969,0:44:54.969 +the fundamental service that is provided by +UNIX + +0:44:54.969,0:44:58.430 +and + +0:44:58.430,0:45:02.849 +what we're going to spend essentially the +next two and a half weeks working on + +0:45:02.849,0:45:04.769 +is + +0:45:04.769,0:45:07.079 +what what makes up processes + +0:45:07.079,0:45:10.180 +we'll go into much more detail about each of these +four points + +0:45:10.180,0:45:11.769 +and + +0:45:11.769,0:45:13.630 +then how do we actually go about + +0:45:13.630,0:45:14.390 +providing that + +0:45:14.390,0:45:16.639 +bit of service + +0:45:16.639,0:45:17.900 +the next thing that I + +0:45:17.900,0:45:22.210 +going to do now is this go through and lay +out some of the terminology that + +0:45:22.210,0:45:23.239 +we have we when + +0:45:23.239,0:45:25.130 +we're talking about processes + +0:45:25.130,0:45:29.229 +so this is sort of the big picture here were +on page eighteen + +0:45:29.229,0:45:30.669 +and + +0:45:30.669,0:45:33.669 +you can see we have sort of three bits that +make up + +0:45:33.669,0:45:36.640 +the system + +0:45:36.640,0:45:39.029 +the currently running user process + +0:45:39.029,0:45:41.180 +and then what we call the top half of the kernel + +0:45:41.180,0:45:43.699 +and the bottom half of the kernel + +0:45:43.699,0:45:47.049 +now this would be a picture for a uniprocessor + +0:45:47.049,0:45:49.299 +so one CPU + +0:45:49.299,0:45:51.209 +if we had a multiprocessor + +0:45:51.209,0:45:54.009 +%uh then we would have + +0:45:54.009,0:45:57.130 +one instance of the kernel + +0:45:57.130,0:45:59.529 +but multiple instances of the user process + +0:45:59.529,0:46:02.879 +but for any given CPU on a multiprocessor + +0:46:02.879,0:46:05.709 +it is running exactly one process + +0:46:05.709,0:46:09.309 +so you may think they were running for four-five +processes all at once + +0:46:09.309,0:46:14.319 +but the fact of the matter is that any instant +in time there's only one process which is + +0:46:14.319,0:46:16.299 +actually running + +0:46:16.299,0:46:18.609 +and + +0:46:18.609,0:46:21.429 +that is the one that we have within the system + +0:46:21.429,0:46:25.199 +now we give the illusion that were running +lots of things because we switch between them + +0:46:25.199,0:46:26.100 +rather quickly + +0:46:26.100,0:46:29.269 +so it looks like things are happening in all +windows at once + +0:46:29.269,0:46:31.430 +but in reality + +0:46:31.430,0:46:33.619 +that's not really happening + +0:46:33.619,0:46:36.440 +okay there is a set of properties that I want to +look at + +0:46:36.440,0:46:40.899 +that had to do with each one of these parts here + +0:46:40.899,0:46:44.359 +but that just to look at it from the +big picture perspective + +0:46:44.359,0:46:45.970 +what you see here + +0:46:45.970,0:46:47.180 +is + +0:46:47.180,0:46:51.549 +there is boundary between the user process +and the top half of the kernel + +0:46:51.549,0:46:54.949 +which is really just like a glorified sovereignty +call + +0:46:54.949,0:46:59.539 +it's a lot like calling into a library routine +like calling strcat, strcpy or something + +0:46:59.539,0:47:00.319 +like that + +0:47:00.319,0:47:03.679 +%uh when you do a system call + +0:47:03.679,0:47:05.650 +we take that same set of parameters + +0:47:05.650,0:47:08.009 +now this is sort of + +0:47:08.009,0:47:09.780 +brick Wall here if you will + +0:47:09.780,0:47:11.380 +that is protecting + +0:47:11.380,0:47:13.680 +the the top half of the kernel + +0:47:13.680,0:47:15.299 +from the application + +0:47:15.299,0:47:18.899 +I'm more go into some detail about how that +actually gets implemented + +0:47:18.899,0:47:22.729 +but in a sense you can think of it +is is there sort of this whaling Wall with little + +0:47:22.729,0:47:24.990 +chinks there you can sort of push a request +through + +0:47:24.990,0:47:28.230 +and somebody other sites where polls that +looks at it and decide whether they're going + +0:47:28.230,0:47:28.690 +to + +0:47:28.690,0:47:30.769 +dained to provide service to you + +0:47:30.769,0:47:34.229 +and if they do then they will send it back + +0:47:34.229,0:47:37.649 +well like a library where you can just sort +of reach in and walk around if you want to + +0:47:37.649,0:47:38.290 +you + +0:47:38.290,0:47:40.950 +good programming practices you don't do that +but + +0:47:40.950,0:47:43.049 +you could + +0:47:43.049,0:47:44.579 +all right so + +0:47:44.579,0:47:49.089 +the the top half of the kernel is really looks +a lot like + +0:47:49.089,0:47:50.509 +a big library + +0:47:50.509,0:47:53.509 +%uh it just happens to be a library of +routines + +0:47:53.509,0:47:57.599 +that deal with things where processes need +to interact with each other + +0:47:57.599,0:48:01.399 +in fact for many people they don't understand +for what's the difference between the C + +0:48:01.399,0:48:03.259 +library and the top half of the kernel + +0:48:03.259,0:48:08.020 +%uh if it's something that you're doing that +no other process needs to know about + +0:48:08.020,0:48:09.799 +then it can be in the C library + +0:48:09.799,0:48:13.829 +so if you call strcat to concatenate two +strings together + +0:48:13.829,0:48:17.599 +nobody else needs to know you're doing that +you don't need to coordinate with anybody + +0:48:17.599,0:48:19.000 +else that you're doing that + +0:48:19.000,0:48:20.160 +it's just happening + +0:48:20.160,0:48:21.979 +so that goes in the C library. + +0:48:21.979,0:48:24.489 +on the other hand if you're reading or writing +the file + +0:48:24.489,0:48:28.029 +there may be other processes that are also +reading and writing that file + +0:48:28.029,0:48:29.910 +and therefore that + +0:48:29.910,0:48:31.579 +has to be done by the kernel + +0:48:31.579,0:48:33.120 +because they can coordinate + +0:48:33.120,0:48:37.189 +on the daemon processes that are trying to access +that file. + +0:48:37.189,0:48:40.529 +so the top of the kernel is pretty straightforward +code + +0:48:40.529,0:48:45.539 +it looks a lot like any other library that +you would write if you look at top half kernel + +0:48:45.539,0:48:49.640 +code you know you see all read,come in +it's got these parameters we Mark around we + +0:48:49.640,0:48:53.719 +get some data that we put it in the buffer and +we return back + +0:48:53.719,0:48:57.470 +in fact writing code for the top half of +the kernel is + +0:48:57.470,0:48:59.729 +not all that difficult to do + +0:48:59.729,0:49:00.989 +%uh it's + +0:49:00.989,0:49:01.959 +you have + +0:49:01.959,0:49:05.939 +for many the same properties that you would +when you're writing user level application + +0:49:05.939,0:49:07.529 +code + +0:49:07.529,0:49:11.779 +the bottom half of the kernel is where things +start to get nasty + +0:49:11.779,0:49:14.820 +thus the bottom half of the kernel is the part +of the system + +0:49:14.820,0:49:18.769 +that deals with all of the asynchronous events +in the system + +0:49:18.769,0:49:22.179 +%uh is things like device drivers, + +0:49:22.179,0:49:23.779 +timers + +0:49:23.779,0:49:25.010 +that level of thing + +0:49:25.010,0:49:28.029 +that are driven by hardware events + +0:49:28.029,0:49:28.659 +so + +0:49:28.659,0:49:31.459 +for example a packet arrives on the network + +0:49:31.459,0:49:33.670 +that causes and interrupt a command + +0:49:33.670,0:49:36.729 +that will be handled by the bottom half of +the kernel + +0:49:36.729,0:49:38.829 +and historically + +0:49:38.829,0:49:43.079 +when an interrupt came in it preempted whatever +else was going on + +0:49:43.079,0:49:45.400 +and it ran until it's finished and then return + +0:49:45.400,0:49:46.539 +and it could not + +0:49:46.539,0:49:49.439 +go to sleep to wait for resources or other +things + +0:49:49.439,0:49:51.339 +%uh in current systems + +0:49:51.339,0:49:54.549 +you can actually go to sleep in a interrupt driver +and wait for + +0:49:54.549,0:49:56.739 +some other activity to complete + +0:49:56.739,0:49:58.259 +it is however + +0:49:58.259,0:50:00.799 +not a good idea to do that + +0:50:00.799,0:50:01.909 +because + +0:50:01.909,0:50:06.739 +the usual case of most of device drivers is they +can finish whatever they're doing in an interrupt + +0:50:06.739,0:50:08.579 +without ever blocking + +0:50:08.579,0:50:09.580 +and so + +0:50:09.580,0:50:13.649 +when an interrupt comes in and we assume that you're +not going to sleep + +0:50:13.649,0:50:14.710 +and if you actually + +0:50:14.710,0:50:17.219 +then go to sleep.oh man + +0:50:17.219,0:50:20.469 +you didnt tell us you're going to do this we +have to go off into a whole lot of other work + +0:50:20.469,0:50:23.029 +that we had originally planned on doing + +0:50:23.029,0:50:25.460 +so if you go to sleep in a device driver + +0:50:25.460,0:50:28.209 +you are taking a very serious performance + +0:50:28.209,0:50:31.019 +so it's highly recommended that you don't +do that + +0:50:31.019,0:50:33.130 +but if you have to you can + +0:50:33.130,0:50:35.809 +on it's because of this historic behavior +or + +0:50:35.809,0:50:39.899 +of not being able to sleep in the bottom half +of the kernel + +0:50:39.899,0:50:42.119 +certain properties that have + +0:50:42.119,0:50:44.769 +%uh taken over in device drivers + +0:50:44.769,0:50:45.940 +and that is + +0:50:45.940,0:50:50.369 +that a device driver should be handed all +the resources it needs to get his job done + +0:50:50.369,0:50:54.490 +you don't give a disk device driver +Go read this + +0:50:54.490,0:50:56.549 +and put it somewhere + +0:50:56.549,0:50:57.580 +you have to say + +0:50:57.580,0:50:59.410 +Go read this particular block + +0:50:59.410,0:51:02.650 +here is a chunk of memory that I want that +my data to put in + +0:51:02.650,0:51:03.959 +and + +0:51:03.959,0:51:06.169 +no to find it when it's done + +0:51:06.169,0:51:06.970 +because + +0:51:06.970,0:51:10.660 +things like allocating memory are classic +places where you end up having to go to sleep + +0:51:10.660,0:51:12.939 +to wait for stuff to happen + +0:51:12.939,0:51:14.449 +and + +0:51:14.449,0:51:16.390 +historically you couldn't do that + +0:51:16.390,0:51:18.640 +even currently don't want to have to do that + +0:51:18.640,0:51:23.400 +so device drivers generally have all +resources pre allocated + +0:51:23.400,0:51:25.169 +and then they can just go + +0:51:25.169,0:51:27.279 +the one place where this doesn't work + +0:51:27.279,0:51:29.029 +is the network + +0:51:29.029,0:51:30.929 +and in particular + +0:51:30.929,0:51:34.630 +you don't know when somebody's going to send +packets to you + +0:51:34.630,0:51:37.040 +you say well you're looking you're open connections + +0:51:37.040,0:51:39.360 +but if you're doing something like IP forwarding + +0:51:39.360,0:51:40.969 +there's no + +0:51:40.969,0:51:45.039 +top half state it's dealing with this packets +they're just coming in on one interface being + +0:51:45.039,0:51:46.719 +sent out on another interface + +0:51:46.719,0:51:50.630 +they never pass through any part of the top +half of the kernel + +0:51:50.630,0:51:53.529 +and so in the case of network device drivers + +0:51:53.529,0:51:56.149 +they need to allocate memory + +0:51:56.149,0:51:56.640 +and + +0:51:56.640,0:51:58.829 +it memory gets into short supply + +0:51:58.829,0:52:01.689 +and they try to allocate memory and it's not +available + +0:52:01.689,0:52:05.049 +they historically coudnt wait for memory to be +available + +0:52:05.049,0:52:08.380 +and even in practice today don't wait + + +0:52:08.380,0:52:09.580 +for memory to become available + +0:52:09.580,0:52:12.469 +they simply drop the packet on the floor + +0:52:12.469,0:52:18.109 +it's like well I didn't have any place to +put it sorry oops + +0:52:18.109,0:52:20.940 +now that doesn't called incorrect behavior + +0:52:20.940,0:52:24.369 +because a higher level protocols what we transmit + +0:52:24.369,0:52:29.140 +but it does cause great performance problems +because we transmissions means that connections + +0:52:29.140,0:52:29.879 +stall + +0:52:29.879,0:52:31.110 +they have to back up + +0:52:31.110,0:52:33.010 +they have to resend data + +0:52:33.010,0:52:33.739 +and so on + +0:52:33.739,0:52:38.739 +so you really want to avoid dropping packets +if you can possibly help it + +0:52:38.739,0:52:42.029 +and consequently + +0:52:42.029,0:52:43.420 +we tend to + +0:52:43.420,0:52:46.499 +pre allocate a certain amount of memory for +the network drivers + +0:52:46.499,0:52:48.299 +%uh and + +0:52:48.299,0:52:52.169 +we try very hard to make sure that we're not +going to run out of memory but + +0:52:52.169,0:52:54.869 +if packets come fast enough and we can't deal +with them + +0:52:54.869,0:52:57.940 +as quickly as they are arriving over short period +of time + +0:52:57.940,0:53:03.489 +we get to the point where we simply have to start +dropping packets + +0:53:03.489,0:53:07.649 +this is a part of kernel that you do not wish to +write code for + +0:53:07.649,0:53:10.919 +because it is extremely difficult +debug + +0:53:10.919,0:53:12.759 +you get these bugs where + +0:53:12.759,0:53:18.779 +the only time it happens is on the third Tuesday +when there's a full moon + +0:53:18.779,0:53:19.300 +and + +0:53:19.300,0:53:24.199 +we have a disk interrupt followed by %uh a +terminal character coming in + +0:53:24.199,0:53:28.289 +and the network packet arriving of size fifteen +twenty two + +0:53:28.289,0:53:30.109 +and when all those things happened + +0:53:30.109,0:53:32.719 +the system panics + +0:53:32.719,0:53:37.380 +and of course there's like the panic +cause you're following some bad pointer + +0:53:37.380,0:53:40.969 +something that should have been there +but was freed some time in the distant past + +0:53:40.969,0:53:42.930 +we are not sure when + +0:53:42.930,0:53:44.049 +and + +0:53:44.049,0:53:47.400 +try to debug things like that is extremely +difficult + +0:53:47.400,0:53:48.509 +mean you can + +0:53:48.509,0:53:52.120 +think well I think I found the problem but +it's not reproduceable + +0:53:52.120,0:53:55.530 +you know you have to wait for the next third +Tuesday with a full moon and blah blah blah + +0:53:55.530,0:53:56.950 +to happen + +0:53:56.950,0:53:57.469 +and + +0:53:57.469,0:54:01.449 +you know so you sort of statistically +gets that you fix that you know I was getting + +0:54:01.449,0:54:03.510 +this bug once every three days + +0:54:03.510,0:54:06.099 +and now it's gone for two weeks without happening + +0:54:06.099,0:54:07.239 +did you fix that? + +0:54:07.239,0:54:08.969 +or if you've been lucky + +0:54:08.969,0:54:10.459 +and and it's + +0:54:10.459,0:54:14.349 +that that coupled with the fact that you're +dealing with hardware + +0:54:14.349,0:54:18.049 +and hardware really works the way it's documented +to work + +0:54:18.049,0:54:21.770 +and so you know they're doing everything that +it says you're supposed to do + +0:54:21.770,0:54:26.260 +it still doesn't work because you didn't set +the fiddle bit over on that other place over + +0:54:26.260,0:54:26.660 +there + +0:54:26.660,0:54:30.479 +that's not documented anywhere but if it's +not said it doesn't work + +0:54:30.479,0:54:33.769 +occasionally + +0:54:33.769,0:54:36.110 +this is another reason that you really want +of avoid + +0:54:36.110,0:54:40.459 +dealing with this part of the system if +you can possibly help + +0:54:40.459,0:54:44.369 +but lets go through and and look at some +of the properties here starting up with at + +0:54:44.369,0:54:45.789 +the user process + +0:54:45.789,0:54:47.980 +%uh we're running with + +0:54:47.980,0:54:51.449 +preemptive scheduling + +0:54:51.449,0:54:53.409 +now there's several caveats here + +0:54:53.409,0:54:55.239 +preemptive scheduling is the default + +0:54:55.239,0:54:56.970 +so called shared scheduler + +0:54:56.970,0:55:01.360 +that is what you normally use there are other +schedulers like the real time scheduler + +0:55:01.360,0:55:02.869 +where what I'm saying isnt that true + +0:55:02.869,0:55:05.709 +%uh will talk about some of the schedulers was +later + +0:55:05.709,0:55:09.930 +but the usual scheduler that you're running +on under UNIX is a shared scheduler + +0:55:09.930,0:55:13.229 +and under the shared scheduler the user applications + +0:55:13.229,0:55:15.159 +run with pre emptive scheduling + +0:55:15.159,0:55:17.449 +and pre emptive scheduling means that + +0:55:17.449,0:55:20.019 +you run at the whim of the system + +0:55:20.019,0:55:21.420 +if they want you to run + +0:55:21.420,0:55:22.140 +you run + +0:55:22.140,0:55:25.490 +once you start running you have no guarantee +of how long you're going to run + +0:55:25.490,0:55:29.370 +it might like to run for three instructions +and then decide it doesn't like you many more + +0:55:29.370,0:55:31.150 +it wants to run something else + +0:55:31.150,0:55:35.920 +while you might get to run for several seconds +and in a row with the with no intervening + +0:55:35.920,0:55:37.469 +things interrupting + +0:55:37.469,0:55:39.719 +you just don't know + +0:55:39.719,0:55:40.969 +and + +0:55:40.969,0:55:42.839 +really all you know is + +0:55:42.839,0:55:43.569 +that + +0:55:43.569,0:55:48.239 +%uh they claim that they're using statistics +and that and that the statistics are fair + +0:55:48.239,0:55:55.059 +and so on average you're going to get a reasonable +amount of time but thats + +0:55:55.059,0:55:57.129 +up to the system you don't control that + +0:55:57.129,0:55:58.439 +the real point here + +0:55:58.439,0:56:01.940 +is that you don't have any way of creating +a critical section + +0:56:01.940,0:56:04.950 +you can say okay I don't want to be interrupted + +0:56:04.950,0:56:07.429 +during this particular sequence of things + +0:56:07.429,0:56:09.809 +so you have to program + +0:56:09.809,0:56:13.469 +assuming that you may be interrupted at any +point + +0:56:13.469,0:56:14.979 +okay + +0:56:14.979,0:56:18.909 +the next thing is that when you're running +in user process + +0:56:18.909,0:56:20.719 +you are running in + +0:56:20.719,0:56:24.150 +with the processor in what's called unprivileged +mode + +0:56:24.150,0:56:28.109 +one of the requirements for running any kind +of a UNIX system + +0:56:28.109,0:56:31.759 +is that you have to have a processor that +support privileged and unprivileged + +0:56:31.759,0:56:33.709 +two different modes of operation + +0:56:33.709,0:56:37.049 +in privilege mode which is what the kernel +runs in + +0:56:37.049,0:56:38.950 +the entire repertoire + +0:56:38.950,0:56:40.869 +of the hardware is available + +0:56:40.869,0:56:45.339 +by this I mean you can set all the register +as you can fiddle with the memory management + +0:56:45.339,0:56:47.460 +unit you can initiate I/O + +0:56:47.460,0:56:50.519 +you can access any register anywhere + +0:56:50.519,0:56:51.919 +etc + +0:56:51.919,0:56:56.540 +when you're running in unprivileged +mode which is what user process run in and + +0:56:56.540,0:57:00.709 +this a large subset of the instructions which +you cannot execute + +0:57:00.709,0:57:03.480 +you cannot initiate I/O on + +0:57:03.480,0:57:04.209 +devices + +0:57:04.209,0:57:06.770 +you cannot change the memory mapping + +0:57:06.770,0:57:10.209 +you cannot access memory that's not part of +your address space + +0:57:10.209,0:57:13.299 +%uh you cannot execute certain instructions +like halt + +0:57:13.299,0:57:15.589 +%uh and + +0:57:15.589,0:57:19.039 +so in general you are protected + +0:57:19.039,0:57:21.789 +from manipulating anything that's outside of your +address space + +0:57:21.789,0:57:23.759 +this of course is desirable because + +0:57:23.759,0:57:27.059 +%uh when you're running in this unprevileged +mode + +0:57:27.059,0:57:28.300 +you're protected + +0:57:28.300,0:57:31.910 +from other processes manipulating you +and they're protected from you manipulating + +0:57:31.910,0:57:33.079 +them + +0:57:33.079,0:57:36.430 +for those of you that have had that misfortune +to have to use + +0:57:36.430,0:57:39.339 +early versions of windows up to about ninety +eight + +0:57:39.339,0:57:42.470 +they always ran with the processor +running in privileged mode + +0:57:42.470,0:57:44.009 +even in applications + +0:57:44.009,0:57:46.459 +and so either maliciously or accidentally + +0:57:46.459,0:57:50.000 +you could stop on other people address space +or you could stop on the kernel + +0:57:50.000,0:57:53.020 +and a lot of the blue screen of death was +people just + +0:57:53.020,0:57:56.319 +following wild pointers and trashing different +parts of the system + +0:57:56.319,0:57:58.819 +taking everything down + +0:57:58.819,0:58:00.020 +it also makes it + +0:58:00.020,0:58:02.320 +far easier to + +0:58:02.320,0:58:05.459 +implement things like viruses and worms and +other things because + +0:58:05.459,0:58:09.619 +user application can we rewrite the boot +block on the disk they can just the write down + +0:58:09.619,0:58:13.109 +and manipulate the registers that allow them +to do whatever they want + +0:58:13.109,0:58:16.730 +whereas when you're running in unprivileged +mode you cant write those kinds of + +0:58:16.730,0:58:20.179 +of things + +0:58:20.179,0:58:24.119 +modern versions of Windows from about +2000 on + +0:58:24.119,0:58:26.630 +now run with privileged and unprevileged mode + +0:58:26.630,0:58:28.649 +but UNIX has always required that + +0:58:28.649,0:58:30.219 +and so when you're running an + +0:58:30.219,0:58:31.319 +the user process + +0:58:31.319,0:58:33.389 +you cannot block i mean + +0:58:33.389,0:58:37.969 +you cannot execute the instructions which +cause a context switching to occur + +0:58:37.969,0:58:40.349 +you can't pick what's going to run next + +0:58:40.349,0:58:43.140 +you can't make that thing run next all you can +do + +0:58:43.140,0:58:45.189 +is go to the operating system and say + +0:58:45.189,0:58:49.269 +hey I've got nothing to do. pick somebody else +to run + +0:58:49.269,0:58:53.449 +the operating system is the think they can +then execute the instructions which cause + +0:58:53.449,0:58:57.609 +a different process to be loaded + +0:58:57.609,0:58:59.049 +and run + +0:58:59.049,0:59:03.400 +alright.finally while you're in a user application you're +running on a user stack + +0:59:03.400,0:59:06.410 +that's part of the user address space + +0:59:06.410,0:59:07.889 +so + +0:59:07.889,0:59:10.819 +part of creating a process gives you a runtime +stack + +0:59:10.819,0:59:14.369 +as part of a virtual address space and so it +can be + +0:59:14.369,0:59:18.199 +more or less up to the limits of the hardware +as big as you want it to be + +0:59:18.199,0:59:19.949 +so if you are running on thirty two-bit processor + +0:59:19.949,0:59:22.819 +you're stack can get 2 gigabytes + +0:59:22.819,0:59:23.319 +and + +0:59:23.319,0:59:26.839 +%uh the what this means is that anytime you +allocate local variables + +0:59:26.839,0:59:28.529 +you don't have to worry about Oh + +0:59:28.529,0:59:30.609 +is that gonna overrun my stack? + +0:59:30.609,0:59:31.610 +so if you need + +0:59:31.610,0:59:35.519 +a hundred thousand double precision floating +point numbers + +0:59:35.519,0:59:37.189 +you can just as a local variable allocate + +0:59:37.189,0:59:40.269 +an array of size a hundred-thousand type +double + +0:59:40.269,0:59:44.029 +and it just decrements your stack pointer by +hundred hundred thousand bytes + +0:59:44.029,0:59:45.009 +a way to go + +0:59:45.009,0:59:47.299 +it's just virtual address space + +0:59:47.299,0:59:49.020 +as you'll see when we get into the kernel + +0:59:49.020,0:59:50.210 +that ceases to be the case diff --git a/en_US.ISO8859-1/captions/2008/asiabsdcon/olander-pcbsd.sbv b/en_US.ISO8859-1/captions/2008/asiabsdcon/olander-pcbsd.sbv new file mode 100644 index 0000000000..95df19c09b --- /dev/null +++ b/en_US.ISO8859-1/captions/2008/asiabsdcon/olander-pcbsd.sbv @@ -0,0 +1,1732 @@ +0:00:05.120,0:00:08.599 +Our first talk is entitled + +0:00:08.599,0:00:12.079 +FreeBSD on the desktop by Matt Orlander + +0:00:12.079,0:00:16.409 +from iXSystems + +0:00:16.409,0:00:17.300 +>> MATT: Thank you. Good morning + +0:00:17.300,0:00:23.349 +I'd like to thank Sato San and George +and the rest of the Asia BSDCon organizers +for inviting us out + +0:00:23.349,0:00:27.099 +we have a little FreeBSD table + +0:00:27.099,0:00:29.189 +and have brought some goodies like + +0:00:29.189,0:00:30.380 +little beasties + +0:00:30.380,0:00:33.010 +you can put in your pocket and walk around Tokyo + +0:00:33.010,0:00:34.110 +and scare people with + +0:00:34.110,0:00:38.339 +%uh, any BSD committers, come on up, and we have +T-shirts for you + +0:00:38.339,0:00:41.490 +and it'll cost - for free - just to say thank +you + +0:00:41.490,0:00:42.729 +%uh this is + +0:00:42.729,0:00:44.640 +FreeBSD on the Desktop + +0:00:44.640,0:00:49.050 +I'm lucky enough to have - + +0:00:49.050,0:00:53.330 +Chris Moore, the founder of the PC-BSD +project here + +0:00:53.330,0:00:56.970 +so really, I don't know why I'm +giving this talk Chris + +0:00:56.970,0:00:59.060 +well, my name is Matt Orlander + +0:00:59.060,0:01:02.940 +I've been working with the BSD since +1998 + +0:01:02.940,0:01:04.970 +when I - uh - + +0:01:04.970,0:01:10.370 +I installed a BSD OS installation at a +little server company in San Jose California + +0:01:10.370,0:01:13.310 +called Telenet System Solutions + +0:01:13.310,0:01:14.280 +later on, I ended up + +0:01:14.280,0:01:16.090 +working full time there + +0:01:16.090,0:01:19.320 +and that they eventually got acquired by BSDi + +0:01:19.320,0:01:23.770 +and so I went on to continue working there + +0:01:23.770,0:01:27.549 +BSDi sold their software assets to WindRiver + +0:01:27.549,0:01:32.540 +and the hardware company lived on with a group +of employees that bought it out + +0:01:32.540,0:01:35.420 +and eventually changed our name back to iXsystems + +0:01:35.420,0:01:39.450 +and now we are very involved trying to support +the FreeBSD project + +0:01:39.450,0:01:41.950 +as well as the other BSDs + +0:01:41.950,0:01:44.229 +I'm on the BSD marketing team + +0:01:44.229,0:01:47.250 +some would say that I don't do enough for that + +0:01:47.250,0:01:50.700 +and of course I do some PC-BSD project +management as well + +0:01:50.700,0:01:56.120 +so people ask us a lot of times +why did we do PC-BSD +Chris can give you + +0:01:56.120,0:01:57.649 +a really good answer about + +0:01:57.649,0:01:59.909 +how he wanted to convert his Mom + +0:01:59.909,0:02:04.360 +and he tried actually to do this +on Linux at first and Linux is such + +0:02:04.360,0:02:10.439 +a fragmented mass, it was much easier for +him to turn to BSD and focus on one + +0:02:10.439,0:02:11.949 +core project + +0:02:11.949,0:02:15.549 +%uh the other reason I usually give is because I + +0:02:15.549,0:02:17.769 +chicks dig BSD + +0:02:17.769,0:02:20.369 +as you can see + +0:02:20.369,0:02:23.779 +that's PC-BSD babe - I don't know if you +can read it + +0:02:23.779,0:02:27.819 +but her site is pcbsdbabe.com +and she was a girl + +0:02:27.819,0:02:32.839 +that installed PC-BSD on one of the early +releases and loved it so much that she + +0:02:32.839,0:02:34.359 +she started a blog + +0:02:34.359,0:02:36.899 +it is pretty entertaining + +0:02:36.899,0:02:40.079 +I won't pull it up here + +0:02:40.079,0:02:44.560 +what is the PC-BSD - I meant to delete that, I swear + +0:02:44.560,0:02:48.159 +PC-BSD it is FreeBSD, but it's not a fork + +0:02:48.159,0:02:51.450 +all we do is take the default FreeBSD +operating system + +0:02:51.450,0:02:54.489 +and add a graphical installer + +0:02:54.489,0:02:59.079 +you know, it's integrated with KDE and +Fluxbox right out of the box + +0:02:59.079,0:03:03.040 +and of course we've got some tools that +Chris and his team have written + +0:03:03.040,0:03:07.159 +to make some of the typical system configuration + +0:03:07.159,0:03:09.729 +utilities a bit easier to use through a GUI + +0:03:09.729,0:03:12.979 +and I'll be showing you some of those + +0:03:12.979,0:03:15.570 +the big claim to fame for PC-BSD + +0:03:15.570,0:03:19.319 +is a different package management system +called PBI + +0:03:19.319,0:03:22.260 +and this is a graphical method of installing + +0:03:22.260,0:03:25.840 +say, like a FreeBSD port or more like a package + +0:03:25.840,0:03:28.759 +in a binary format using a front-end GUI + +0:03:28.759,0:03:31.949 +it contains all the dependencies of the package + +0:03:31.949,0:03:33.519 +in one directory + +0:03:33.519,0:03:37.699 +%uh so that we can save the package through +upgrades and + +0:03:37.699,0:03:40.139 +%uh make it very easy for other users + +0:03:40.139,0:03:41.099 +to install + +0:03:41.099,0:03:43.150 +we're working very hard + +0:03:43.150,0:03:48.059 +to integrate this more closely with FreeBSD's +methodology of ports and packages + +0:03:48.059,0:03:49.390 +and so far + +0:03:49.390,0:03:53.680 +Chris and his team have setup an auto-building PBI server + +0:03:53.680,0:03:56.759 +it's semi-auto, right? + +0:03:56.759,0:03:58.079 +>> CHRIS: it's pretty auto +>> MATT: pretty automatic now + +0:03:58.079,0:04:01.529 +it takes a little manual configuration of a port + +0:04:01.529,0:04:06.370 +and then it tears through the port and generates +the PC-BSD binary + +0:04:06.370,0:04:07.640 +and so now + +0:04:07.640,0:04:11.619 +whenever the port - how many are in their so far now Chris? + +0:04:11.619,0:04:13.219 +>> CHRIS: about a hundred +>> MATT: okay, so that's awesome + +0:04:13.219,0:04:16.989 +so I mean you know it's only hundred but +really for desktop use + +0:04:16.989,0:04:20.419 +%uh how many times you know how many applications +are going to be useful + +0:04:20.419,0:04:22.210 +for your average desktop user + +0:04:22.210,0:04:23.919 +%uh power users, you know, %uh + +0:04:23.919,0:04:30.050 +can just drop in and install a port +or a package so it's not a big deal + +0:04:30.050,0:04:31.860 +so Chris and his team have + +0:04:31.860,0:04:33.259 +setup scripts that tear through + +0:04:33.259,0:04:35.480 +%uh a hundred or so ports so far + +0:04:35.480,0:04:40.819 +and that whenever a port is updated +it will automatically generate the PC-BSD binary + +0:04:40.819,0:04:44.100 +and then it'll notify all of the PC-BSD users + +0:04:44.100,0:04:47.129 +with an annoying pop-up window + +0:04:47.129,0:04:48.219 +that updates are available + +0:04:48.219,0:04:52.889 +but of course you can turn that off + +0:04:52.889,0:04:55.150 +so the goal here is to make it + +0:04:55.150,0:04:57.290 +%uh easier for people to use + +0:04:57.290,0:04:59.870 +FreeBSD that are new to it especially + +0:04:59.870,0:05:03.280 +I think the jury that the PC-BSD community + +0:05:03.280,0:05:05.969 +as I look at forums and talk to people + +0:05:05.969,0:05:07.249 +is Windows users + +0:05:07.249,0:05:11.869 +so we're getting a lot of Windows users +coming on over and I think now we're +getting a lot of Linux users coming over + +0:05:11.869,0:05:14.040 +you know, they like Linux, but + +0:05:14.040,0:05:16.939 +they've tried twenty different flavors + +0:05:16.939,0:05:18.439 +they start to realize that + +0:05:18.439,0:05:22.849 +something's wrong + +0:05:22.849,0:05:25.739 +so we're now that at PC-BSD 1.5 + +0:05:25.739,0:05:28.289 +which includes Xorg 7.3 + +0:05:28.289,0:05:34.909 +3.5.8, actually, you're already bumped up to +like a 1.5.1, right? +[mumbles] + +0:05:34.909,0:05:39.249 +as soon as you get back, he's +been working on that at the hotel + +0:05:39.249,0:05:40.860 +we have a new system updater tool + +0:05:40.860,0:05:43.709 +that updates your PC-BSD system + +0:05:43.709,0:05:44.400 +on the fly + +0:05:44.400,0:05:47.610 +how does that work Chris? So you get +notified there's an update and + +0:05:47.610,0:05:52.229 +I got a call from my office in fact yesterday +that they had done that + +0:05:52.229,0:05:56.699 +the KMail call that I had to look into + +0:05:56.699,0:06:03.699 +but it just notifies you and you can just +download an ISO or download a patch +>> CHRIS: it notifies, downloads and applies the +patch to your system +>> MATT: brilliant +>> CHRIS: on the fly + +0:06:06.479,0:06:08.319 +>> MATT: and of course now we have a WiFi tool + +0:06:08.319,0:06:09.620 +which is really cool + +0:06:09.620,0:06:13.569 +it'll scan and you just double click on that +network + +0:06:13.569,0:06:16.080 +and enter your WPA or WEP key and + +0:06:16.080,0:06:19.489 +it'll remember these + +0:06:19.489,0:06:20.990 +%uh we've got + +0:06:20.990,0:06:22.900 +improvements to the PBI removal tool + +0:06:22.900,0:06:27.249 +who is similar to the add / remove programs +from Windows + +0:06:27.249,0:06:28.699 +so I'll show you that + +0:06:28.699,0:06:31.899 +%uh the new sound detection program + +0:06:31.899,0:06:36.339 +and we now finally have an amd64 build + +0:06:36.339,0:06:40.580 +as of PC-BSD 1.5 + +0:06:40.580,0:06:45.259 +so actually that you can follow along in the +your slides + +0:06:45.259,0:06:52.259 +I'm going to drop out of this and show +you the actual installation + +0:06:56.500,0:07:03.190 +it's like a cooking show + +0:07:03.190,0:07:06.349 +okay so here's our first screen after a very +typical looking + + +0:07:06.349,0:07:08.430 +FreeBSD bootstrap + +0:07:08.430,0:07:11.970 +and we have different languages and of +course keyboard layouts + +0:07:11.970,0:07:15.190 +and we do have Japanese and I believe that +Chris already installed + +0:07:15.190,0:07:16.730 +the Japanese version, is that right + +0:07:16.730,0:07:19.189 +so you have it running upstairs on the laptop? + +0:07:19.189,0:07:21.199 +oh right here? awesome + +0:07:21.199,0:07:25.249 +I would have done that but then I wouldn't +have been able to read anything + +0:07:25.249,0:07:28.009 +we default to allowing submitting + +0:07:28.009,0:07:31.370 +usage statistics to FreeBSD stats + +0:07:31.370,0:07:33.250 +and the reason we do that + +0:07:33.250,0:07:36.530 +is so that when I go to talk to companies +like Adobe + +0:07:36.530,0:07:41.589 +to persuade them to give us a native Flash 9 + +0:07:41.589,0:07:45.629 +which Kirk has already appeared out there + +0:07:45.629,0:07:51.019 +this gives me a place I can point them to +start to persuade for the business case + +0:07:51.019,0:07:53.060 +for them supporting + +0:07:53.060,0:07:54.370 +BSD natively + +0:07:54.370,0:07:59.419 +and I think really close to seeing that this +year + +0:07:59.419,0:08:01.710 +I think Desktop BSD are + +0:08:01.710,0:08:05.169 +I guess you can call them our competitor +although we don't really compete + +0:08:05.169,0:08:07.330 +we collaborate a bit + +0:08:07.330,0:08:12.179 +they just now started defaulting to allowing these +statistics and I saw a big jump in their + +0:08:12.179,0:08:15.659 +BSD stats + +0:08:15.659,0:08:19.860 +here is the license agreement, and all we have +here is the BSD license, some of the Intel +firmware licenses, + +0:08:19.860,0:08:26.860 +and then, that's about it + +0:08:28.249,0:08:33.390 +I'm going to go ahead and agree + +0:08:33.390,0:08:37.300 +here's where I can do a fresh install +or update my existing system + +0:08:37.300,0:08:40.250 +the update basically + +0:08:40.250,0:08:42.379 +%uh Chris, how does the update work? It TARs up + +0:08:42.379,0:08:44.780 +the user's home directory? + +0:08:44.780,0:08:49.000 +[mumbles] + +0:08:49.000,0:08:56.000 +and we've done it many times at work so far +and it works pretty darn well + +0:09:11.250,0:09:17.850 +so there you see, I didn't enter that correctly, +I was just testing that + +0:09:17.850,0:09:19.290 +so now the passwords match up + +0:09:19.290,0:09:21.020 +I go ahead and enter + +0:09:21.020,0:09:28.020 +a default user + +0:09:36.120,0:09:38.680 +you can choose your shells here + +0:09:38.680,0:09:42.180 +and Chris has added Bash as a default +here as well, because + +0:09:42.180,0:09:48.150 +well, the Linux users seem to prefer Bash + +0:09:48.150,0:09:50.980 +here's where I can toggle auto-login, +the Windows people + +0:09:50.980,0:09:56.740 +they're used to it + +0:09:56.740,0:09:57.819 +here's your partitioning + +0:09:57.819,0:10:02.950 +Chris has changed the partitioning a little bit +we used to have something where you actually could go in + +0:10:02.950,0:10:06.210 +and really tweak the partitioning and people were + +0:10:06.210,0:10:08.190 +things and so right now + +0:10:08.190,0:10:10.120 +it is kind of turned off for now + +0:10:10.120,0:10:12.580 +oh you can still do that, ok + +0:10:12.580,0:10:15.740 +what about what about adding + +0:10:15.740,0:10:17.610 +a partitioning tool something that + +0:10:17.610,0:10:19.080 +>> CHRIS: we've talked about it + +0:10:19.080,0:10:21.770 +>> MATT: it's dangerous, it's really dangerous + +0:10:21.770,0:10:25.220 +I'm going to go ahead, since I'm actually on a Mac, + +0:10:25.220,0:10:26.699 +I'm going to use the entire disk + +0:10:26.699,0:10:29.550 +I could customize and go setup my own + +0:10:29.550,0:10:32.060 +partitioning scheme + +0:10:32.060,0:10:35.820 +Chris, when are you going to bring back the server edition? + +0:10:35.820,0:10:40.180 +when we have enough tools to throw at it? +>> CHRIS: we don't have enough tools + +0:10:40.180,0:10:41.099 +>> MATT: what it was + +0:10:41.099,0:10:42.990 +we had a server edition + +0:10:42.990,0:10:44.900 +and we removed it as it wasn't really doing much yet + +0:10:44.900,0:10:49.150 +and basically it dropped you into a default +Fluxbox install + +0:10:49.150,0:10:50.830 +with minimal tools + +0:10:50.830,0:10:52.360 +so you could tweak it + +0:10:52.360,0:10:54.469 +once we bring that back, we'll have %uh + +0:10:54.469,0:10:55.230 +we'll have, based on iXsystems + +0:10:55.230,0:11:00.100 +based on the company's, the sponsors that +come out here + +0:11:00.100,0:11:04.300 +we'll throw together some default partitionings +that we see a lot of + +0:11:04.300,0:11:06.620 +as we %uh, as we ship out servers + +0:11:06.620,0:11:08.169 +and then have them in there as an option + +0:11:08.169,0:11:13.210 +like a LAMP server, a mail server, a DNS or what +have you + +0:11:13.210,0:11:14.390 +this is cool + +0:11:14.390,0:11:18.460 +so now I can go ahead and choose add-ons + +0:11:18.460,0:11:20.910 +and add them on to the system + +0:11:20.910,0:11:23.260 +it's a short list right now + +0:11:23.260,0:11:25.890 +but you've got some cool stuff + +0:11:25.890,0:11:27.299 +I grabbed the source in ports +we have %uh + +0:11:27.299,0:11:29.260 +Opera + +0:11:29.260,0:11:32.689 +they agreed within in forty eight hours +when I asked them + +0:11:32.689,0:11:34.180 +that we could ship with + +0:11:34.180,0:11:37.080 +their Opera binary +they said absolutely no problem + +0:11:37.080,0:11:39.890 +and Adobe agreed that we could ship with Flash + +0:11:39.890,0:11:40.889 +so we do + +0:11:40.889,0:11:43.040 +if you select any of these + +0:11:43.040,0:11:50.040 +%uh I'd have to insert the second disc +so I'll go ahead and skip it + +0:11:50.160,0:11:51.120 +and then, there you go + +0:11:51.120,0:11:57.120 +very secure + +0:11:57.120,0:12:04.120 +so we'll let this cook + +0:12:12.990,0:12:15.300 +so now like the cooking show + +0:12:15.300,0:12:19.020 +I put it in the oven +but I already have one finished + +0:12:19.020,0:12:22.060 +and so here's a finished PC-BSD installation + +0:12:22.060,0:12:26.530 +this is very close to what you would see by default +although I've already gone and installed a few things + +0:12:26.530,0:12:27.899 +noticeably + +0:12:27.899,0:12:32.040 +on the right this is a very %uh this is very +very cool when I show %uh + +0:12:32.040,0:12:35.860 +Windows web developers in particular love this + +0:12:35.860,0:12:39.220 +we can install what we call PAMP + +0:12:39.220,0:12:40.860 +I'm trying to come up with a better name + +0:12:40.860,0:12:43.180 +but for now it's PAMP + +0:12:43.180,0:12:46.880 +and that would of course by the +Apache, MySQL and PHP stack + +0:12:46.880,0:12:48.089 +what's interesting is + +0:12:48.089,0:12:52.590 +the guy, I don't know who wrote this for us +someone did this for us + +0:12:52.590,0:12:54.290 +you did it? + +0:12:54.290,0:12:56.140 +I do know the guy who did it + +0:12:56.140,0:12:58.750 +%uh I thought someone else did it originally? + +0:12:58.750,0:13:01.580 +did you update it? yeah? okay, great + +0:13:01.580,0:13:02.310 +this was great + +0:13:02.310,0:13:05.199 +I just took a class at UC Santa Cruz + +0:13:05.199,0:13:09.190 +I showed the teacher this +and she had me show the entire class + +0:13:09.190,0:13:11.130 +once I had showed her + +0:13:11.130,0:13:12.449 +literally two clicks + +0:13:12.449,0:13:15.980 +we install %uh Apache, PHP, MySQL + +0:13:15.980,0:13:18.060 +and then we get some nice little icons + +0:13:18.060,0:13:20.960 +%uh to start and to stop + +0:13:20.960,0:13:22.580 +we can %uh + +0:13:22.580,0:13:24.199 +just instantly switch + +0:13:24.199,0:13:26.440 +the PHP versions + +0:13:26.440,0:13:28.300 +with the %uh web devs really like + +0:13:28.300,0:13:30.660 +for testing + +0:13:30.660,0:13:32.790 +you get a little drop down to switch the version + +0:13:32.790,0:13:35.710 +especially, I noticed that even in this class + +0:13:35.710,0:13:40.449 +these guys are you know CSS wizards + +0:13:40.449,0:13:43.050 +and xHTML wizards and +they've never seen a command line + +0:13:43.050,0:13:43.959 +and so + +0:13:43.959,0:13:49.069 +everyone was looking at you know +this is my Mac that I use for testing +but I have a PC-BSD laptop + +0:13:49.069,0:13:50.280 +as well + +0:13:50.280,0:13:52.040 +for work +this is my personal Mac + +0:13:52.040,0:13:55.009 +but for work I use my PC-BSD laptop + +0:13:55.009,0:13:56.940 +and so I had that at the class + +0:13:56.940,0:14:00.110 +and I was able to share and +everyone was like what the +hell are you doing? + +0:14:00.110,0:14:02.190 +and so I gave them a quick demo + +0:14:02.190,0:14:02.950 +and %uh + +0:14:02.950,0:14:05.370 +you see here we can put a little icon + +0:14:05.370,0:14:06.299 +down in the tray + +0:14:06.299,0:14:09.770 +that lets us easily start and stop and monitor +the services + +0:14:09.770,0:14:13.420 +%uh for someone that's not familiar +with how these work + +0:14:13.420,0:14:14.989 +we can even go in and edit + +0:14:14.989,0:14:16.229 +the HTTPD.conf + +0:14:16.229,0:14:22.300 +or the PHP.ini + +0:14:22.300,0:14:26.970 +and so this is just one of the one hundred +packages that we have building + +0:14:26.970,0:14:29.050 +and this is a really cool one + +0:14:29.050,0:14:31.460 +and then they can open the web root + +0:14:31.460,0:14:35.040 +or open their personal web root +or toggle the web sites around + +0:14:35.040,0:14:37.549 +so a pretty clever tool for manipulating + +0:14:37.549,0:14:39.160 +Apache, MySQL or PHP + +0:14:39.160,0:14:41.220 +via the GUI + +0:14:41.220,0:14:44.060 +and it installs that %uh that + +0:14:44.060,0:14:48.230 +MySQL GUI admin tool that I've never used + +0:14:48.230,0:14:49.660 +yeah yeah they love that stuff + +0:14:49.660,0:14:52.110 +everyone in the class knew what that was + +0:14:52.110,0:14:54.410 +and I was like, what? + +0:14:54.410,0:14:56.940 +now let me show you the big deal + +0:14:56.940,0:14:58.279 +this is a PBI + +0:14:58.279,0:15:00.320 +that I downloaded + +0:15:00.320,0:15:04.690 +off of PBIDir.com + +0:15:04.690,0:15:06.960 +so this is the this is the GIM + +0:15:06.960,0:15:09.090 +and I'm going to install it + +0:15:09.090,0:15:11.330 +on a very fresh installation + +0:15:11.330,0:15:14.120 +the icon you get in the upper left hand corner + +0:15:14.120,0:15:16.090 +says Get PBIs and %uh + +0:15:16.090,0:15:19.880 +and it just opens up a Konqueror window +directly to the website + +0:15:19.880,0:15:22.210 +I've been talking to Chris about +maybe in the future + +0:15:22.210,0:15:23.089 +having the %uh + +0:15:23.089,0:15:26.820 +maybe grabbing the Desktop BSD's port tool + +0:15:26.820,0:15:28.170 +having that on a tab + +0:15:28.170,0:15:31.410 +and then having a PBI add / remove + +0:15:31.410,0:15:37.360 +on the other tab so that you can just browse +all the programs that you can install +via a port, package or PBI + +0:15:37.360,0:15:40.410 +on one single interface + +0:15:40.410,0:15:43.720 +so I'm going to install the GIMP + +0:15:43.720,0:15:50.720 +I'm going to agree to that nasty license + +0:15:59.090,0:16:01.530 +I love these graphics that Kinsalez does + +0:16:01.530,0:16:08.530 +ok and finished + +0:16:15.030,0:16:16.560 +and there you go + +0:16:16.560,0:16:18.460 +we got the GIMP on there + +0:16:18.460,0:16:21.310 +and it created of course a +menu icon + +0:16:21.310,0:16:23.480 +in here as well + +0:16:23.480,0:16:25.180 +under the GIMP + +0:16:25.180,0:16:31.910 +so now I could easily upgrade this +in fact when the PBI gets updated it will +notify me + +0:16:31.910,0:16:36.590 +that hey there's a new GIMP do you +want to download and install it right now? +and you just click yes and it + +0:16:36.590,0:16:38.540 +downloads and installs +and keeps %uh + +0:16:38.540,0:16:42.780 +your GIMP preferences or whatever +the application preferences that you +already have in there + +0:16:42.780,0:16:47.130 +and then since we're targeting mostly +you know, easy to use + +0:16:47.130,0:16:49.480 +desktop operating systems specifically + +0:16:49.480,0:16:52.869 +for Windows users so we have +a lot of familiar tools for them + +0:16:52.869,0:16:57.050 +if you go into the Start menu and look +at the settings, and here's my software + +0:16:57.050,0:17:00.880 +there's the add / remove that I was telling you about + +0:17:00.880,0:17:02.790 +and so I'll go in and + +0:17:02.790,0:17:09.790 +we will remove it + +0:17:10.190,0:17:12.959 +so here's all the software I have installed + +0:17:12.959,0:17:14.640 +you can see the GIMP there + +0:17:14.640,0:17:17.630 +and really since the GIMP + +0:17:17.630,0:17:21.090 +is really just installed in our + +0:17:21.090,0:17:26.370 +own little directory structure here + +0:17:26.370,0:17:27.610 +so you can see + +0:17:27.610,0:17:28.809 +the GIMP and + +0:17:28.809,0:17:30.009 +all its libs + +0:17:30.009,0:17:37.009 +are hiding in here + +0:17:37.820,0:17:39.530 +that's the %uh, I don't know if you can see that + +0:17:39.530,0:17:42.890 +but it's the automatic directory structure that's created + +0:17:42.890,0:17:49.890 +from the scripts that we use on the auto-building server + +0:17:50.270,0:17:52.570 +and so I go ahead and click remove + +0:17:52.570,0:17:56.970 +and of course all this is really doing +is an rm -rf on there + +0:17:56.970,0:17:58.940 +and bam - gone +and there's a little script + +0:17:58.940,0:18:05.940 +that pulls the icons out of there +and notifies you that it is gone + +0:18:10.330,0:18:15.470 +and we've got a services manager + +0:18:15.470,0:18:18.760 +this is actually PC-BSD 1.4 + +0:18:18.760,0:18:22.460 +but we've got 1.5 running upstairs + +0:18:22.460,0:18:27.850 +that didn't have any PBIs to show on 1.5 + +0:18:27.850,0:18:29.690 +and so here, of course, + +0:18:29.690,0:18:36.400 +anyone who has used Windows you can +see this looks very similar to whatever +that Windows tool is called + +0:18:36.400,0:18:38.540 +so we can start / stop services + +0:18:38.540,0:18:40.290 +that are enabled in the start-up + +0:18:40.290,0:18:41.350 +and so you know + +0:18:41.350,0:18:45.770 +it’s not like this can replace a system administrator +but it sure %uh + +0:18:45.770,0:18:49.580 +well I know that everyone at our office +runs this of course + +0:18:49.580,0:18:52.440 +and most of our family members now are running it + +0:18:52.440,0:18:54.980 +and I was down at + +0:18:54.980,0:18:56.580 +the Luxor in Las Vegas + +0:18:56.580,0:18:59.090 +and I was wearing my FreeBSD shirt + +0:18:59.090,0:19:00.870 +and I'm sitting at the pub + +0:19:00.870,0:19:01.760 +and this guy goes + +0:19:01.760,0:19:03.930 +you've heard of FreeBSD? + +0:19:03.930,0:19:05.120 +and I go 'yeah, dude' + +0:19:05.120,0:19:07.710 +he goes have you heard of PC-BSD? + +0:19:07.710,0:19:09.669 +and I thought it was really cool that + +0:19:09.669,0:19:15.010 +this random bartender down in Vegas is +telling me about PC-BSD. I go 'yeah, +I've heard of that'. + +0:19:15.010,0:19:19.020 +he goes 'we're running that at home, dude, it's cool' + +0:19:19.020,0:19:26.020 +so I went into the room and got some swag +for him and he was so happy + +0:19:29.350,0:19:33.530 +we've got our own users tool +we've got a WiFi tool + +0:19:33.530,0:19:35.170 +%uh + +0:19:35.170,0:19:39.540 +we did we do our own user management tool +over the KDE one? + +0:19:39.540,0:19:42.090 +>> CHRIS: it was the only sensible option + +0:19:42.090,0:19:42.910 +>> MATT: yeah + +0:19:42.910,0:19:47.410 +you know we're trying to get closer involved +with the KDE guys + +0:19:47.410,0:19:50.300 +so that they test a little more on FreeBSD + +0:19:50.300,0:19:53.010 +because we've discovered some of their stuff +just doesn't compile + +0:19:53.010,0:19:54.630 +or work properly + +0:19:54.630,0:19:57.760 +and we have to create BSD-specific tools for them + +0:19:57.760,0:20:02.080 +it would be nicer if we had a little +more collaboration there + +0:20:02.080,0:20:04.810 +there's your online update manager where I can +go + +0:20:04.810,0:20:06.620 +and manually check for updates + +0:20:06.620,0:20:10.090 +and install them + +0:20:10.090,0:20:13.910 +and I've got my %uh + +0:20:13.910,0:20:15.409 +my network settings + +0:20:15.409,0:20:19.799 +which pulls up the WiFi tool which is really +easy to use. I don't think it'll pull up + +0:20:19.799,0:20:21.039 +because my Mac + +0:20:21.039,0:20:23.170 +doesn't translate it + +0:20:23.170,0:20:26.750 +but it's pretty darn easy to use + +0:20:26.750,0:20:30.190 +actually, this is the old one huh? + +0:20:30.190,0:20:37.190 +we've got the new one upstairs +it's even better + +0:20:42.230,0:20:49.230 +almost done + +0:20:58.450,0:21:01.760 +ah yes, so we didn't see this yet +this is the X window configurator + +0:21:01.760,0:21:05.690 +and so this is really cool if you've +got a supported 3D graphics card + +0:21:05.690,0:21:06.970 +you can go ahead + +0:21:06.970,0:21:08.280 +and select it here + +0:21:08.280,0:21:11.220 +and be running Beryl right at startup + +0:21:11.220,0:21:13.160 +so when we do trade shows + +0:21:13.160,0:21:15.280 +which is part of what I do + +0:21:15.280,0:21:17.100 +to evangelize FreeBSD + +0:21:17.100,0:21:20.000 +we bring along a 32 inch LCD + +0:21:20.000,0:21:25.850 +and we throw PC-BSD up there with Beryl +running in it, and boy people just go +nuts over it + +0:21:25.850,0:21:27.160 +over those windows + +0:21:27.160,0:21:29.070 +you know, those squares, Beryl stuff + +0:21:29.070,0:21:32.380 +I'm over it already, but + +0:21:32.380,0:21:34.090 +and there's the boot screen + +0:21:34.090,0:21:36.940 +and the update manager +I've already covered the update + +0:21:36.940,0:21:39.600 +there's looking for a new PBI update + +0:21:39.600,0:21:43.530 +this was the test you were running for the FireFox +one before you rolled it out + +0:21:43.530,0:21:47.420 +so I was testing it and grabbing screenshots + +0:21:47.420,0:21:51.660 +and then it installed it and then +it told me it was finished + +0:21:51.660,0:21:53.500 +sure enough it fired right up +and worked like a charm + +0:21:53.500,0:21:55.040 +so, very cool + +0:21:55.040,0:21:57.460 +we just did this + +0:21:57.460,0:22:00.530 +in fact + +0:22:00.530,0:22:03.010 +the GIMP and we removed it + +0:22:03.010,0:22:06.370 +so, how can you help? + +0:22:06.370,0:22:07.580 +well you could grab a CD + +0:22:07.580,0:22:10.190 +today and install PC-BSD + +0:22:10.190,0:22:14.240 +and certainly one of the biggest contributions +any user can do + +0:22:14.240,0:22:15.980 +is reporting bugs back to us + +0:22:15.980,0:22:17.580 +I noticed, I don't know, last month + +0:22:17.580,0:22:23.810 +I think people had reported just a really small +list of bugs, but you troll the forums and +they're complaining in there + +0:22:23.810,0:22:26.900 +so you go on and say, please report the bug +where we will see it + +0:22:26.900,0:22:28.429 +because we can't be + +0:22:28.429,0:22:31.810 +trolling ten thousand posts every day + +0:22:31.810,0:22:35.540 +that would be one way you could help +documentation and translations + +0:22:35.540,0:22:37.370 +it would be nice if + +0:22:37.370,0:22:37.980 +%uh + +0:22:37.980,0:22:41.490 +I don't know, who did our Japanese translation? + +0:22:41.490,0:22:42.239 +we don't even know + +0:22:42.239,0:22:47.320 +it would be cool if a Japanese native speaker +could take a look at it and say +'hey, there are some errors here' + +0:22:47.320,0:22:48.010 +%uh + +0:22:48.010,0:22:49.050 +maybe it was a + +0:22:49.050,0:22:54.030 +%uh a British or an American guy who speaks Japanese +and maybe he made some mistakes + +0:22:54.030,0:22:57.700 +%uh some CDs out, download them, tell your friends + +0:22:57.700,0:22:58.999 +get your Mom on it + +0:22:58.999,0:23:01.890 +and prepare for all the phone calls + +0:23:01.890,0:23:03.320 +and of course evangelize + +0:23:03.320,0:23:08.299 +if you can do C, Qt, or Shell programming +we can always use help there + +0:23:08.299,0:23:10.340 +and you can just email the folk + +0:23:10.340,0:23:12.169 +or go to the forums + +0:23:12.169,0:23:15.059 +or the PC-BSD web site and see how to contact us + +0:23:15.059,0:23:19.530 +and there we have a core team list there very +similar to FreeBSD + +0:23:19.530,0:23:24.540 +so you can email the core team of PC-BSD +and everyone will see that + +0:23:24.540,0:23:27.460 +alright, so that's PC-BSD in a nutshell + +0:23:27.460,0:23:36.000 +does anyone have any questions 'cause we have Chris in the room + +0:23:36.000,0:23:35.450 +yes? + +0:23:35.450,0:23:39.750 +>> AUDIENCE MEMBER: like, so %uh +[unclear] + +0:23:39.750,0:23:43.890 +most of the computer is like a desktop +you know, it's an old computer + +0:23:43.890,0:23:48.890 +so, %uh, does this system have a +driver for the, like, the sound card + +0:23:48.890,0:23:54.090 +>> MATT: well this is FreeBSD 6.3 so +yeah, absolutely, FreeBSD + +0:23:54.090,0:23:56.070 +supports the older hardware really really well + +0:23:56.070,0:23:59.559 +from my experience + +0:23:59.559,0:24:02.190 +any other questions? + +0:24:02.190,0:24:06.299 +we have the FreeBSD table upstairs + +0:24:06.299,0:24:12.640 +so stop on by before I drink too much sake! + +0:24:12.640,0:24:17.770 +what's the latest on BSD stats? +I mean, there's always a certain +percentage of people + +0:24:17.770,0:24:19.250 +that turn off the reporting + +0:24:19.250,0:24:22.260 +for some weird reason + +0:24:22.260,0:24:24.650 +last month we had ten thousand installs + +0:24:24.650,0:24:26.859 +so you know - how many of those keep? + +0:24:26.859,0:24:30.859 +it's really hard to say, right +Yes? + +0:24:30.859,0:24:36.120 +>> AUDIENCE MEMBER: Matt, two questions +I'm from China + +0:24:36.120,0:24:40.220 +does FreeBSD or PC-BSD support Chinese? + +0:24:40.220,0:24:42.750 +>> CHRIS: we've had people translate into Chinese, yes + +0:24:42.750,0:24:55.360 +>> AUDIENCE MEMBER: we are interested in this project +because we've have seen that some consumers +we've tried to persuade them to use PC-BSD instead + +0:24:55.360,0:24:59.320 +>> MATT: this, this is great, we'd love to help you out +especially convincing Linux consumers + +0:24:59.320,0:25:02.130 +>> AUDIENCE MEMBER: can we, can we join this team? + +0:25:02.130,0:25:03.990 +>> MATT: yes absolutely + +0:25:03.990,0:25:05.199 +we would love to have you + +0:25:05.199,0:25:15.670 +>> AUDIENCE MEMBER: finally, can +PC-BSD have an alternative login like KDM? + +0:25:15.670,0:25:21.490 +>> MATT: yeah, it already does, it already does +so on the install, you just uncheck auto-login + +0:25:21.490,0:25:23.440 +and then it'll pull up with a customized KDM + +0:25:23.440,0:25:30.130 +>> AUDIENCE MEMBER: final question, +so does PC-BSD one disk? + +0:25:30.130,0:25:33.600 +>> MATT: yeah, I just install off +of this one disk + +0:25:33.600,0:25:36.820 +the second disk contains some optional + +0:25:36.820,0:25:41.400 +yes, some optional components as well as +some language translations + +0:25:41.400,0:25:43.860 +but yeah, if you want to change the language + +0:25:43.860,0:25:47.190 +during the install you would need the second disk + +0:25:47.190,0:25:49.160 +>> AUDIENCE MEMBER: can we contact +you after the conference? + +0:25:49.160,0:25:50.550 +>> MATT: absolutely and I would love it + +0:25:50.550,0:25:55.650 +we just had a school, an entire school district +in France switch over to PC-BSD so %uh + +0:25:55.650,0:25:58.900 +we're doing a case study + +0:25:58.900,0:25:59.539 +so we can throw that up on the FreeBSD site + +0:25:59.539,0:26:06.100 +they were ecstatic + +0:26:06.100,0:26:08.830 +yeah, they were absolutely ecstatic +and were all like, how can we help? + +0:26:08.830,0:26:13.160 +so all the servers in the city in France +were already on FreeBSD + +0:26:13.160,0:26:17.400 +so when the school district contacted +them to talk about + +0:26:17.400,0:26:20.980 +upgrading their Windows installs +they said why don't we do a test run + +0:26:20.980,0:26:25.960 +with the PC-BSD and see how you guys like it +and they said that the teachers and students loved it + +0:26:25.960,0:26:27.250 +so they rolled it out + +0:26:27.250,0:26:29.400 +and they rolled it out without having any issues + +0:26:29.400,0:26:34.179 +%uh we asked them, you know, I sent them a +bunch of questions and they responded +you know, what kind of issues did you have + +0:26:34.179,0:26:36.160 +how did you solve them? + +0:26:36.160,0:26:37.799 +how do you do upgrades + +0:26:37.799,0:26:39.980 +and so on and so on + +0:26:39.980,0:26:46.980 +the iXsystems marketing girl is +synthesing that into a case study that +we'll release shortly + +0:26:48.480,0:26:50.750 +absolutely + +0:26:50.750,0:26:51.809 +yes, Jason? + +0:26:51.809,0:26:56.760 +>> JASON: are there things like, %uh +remote home directories + +0:26:56.760,0:26:01.980 +to take your school example are +so students login to one PC and + +0:26:01.980,0:26:03.120 +they have the same home directory? + +0:26:03.120,0:26:05.080 +>> AUDIENCE MEMBER: like Terminal Server? + +0:26:05.080,0:26:10.150 +>> JASON: well, not necessarily a Terminal Server +but more, I don't know, let's go down the Windows path + +0:26:10.150,0:26:16.320 +what do they call it? they call it a Domain server +where you can bounce your home directories + +0:26:16.320,0:26:17.010 +>> AUDIENCE MEMBER: because it's all just FreeBSD + +0:26:17.10,0:27:22.290 +under the hood so you could +>> MATT: yeah, we don't have any +graphical tools right now for that + +0:27:22.290,0:27:23.860 +but any FreeBSD sys admin + +0:27:23.860,0:27:25.770 +could make it sit up and beg + +0:27:25.770,0:27:27.570 +what we are working on + +0:27:27.570,0:27:30.320 +right now some cool PXE stuff + +0:27:30.320,0:27:32.500 +so you'll be able to + +0:27:32.500,0:27:34.360 +boot off of a live CD + +0:27:34.360,0:27:36.630 +and then install tons of machines + +0:27:36.630,0:27:41.549 +from that one installation +so that's pretty cool + +0:27:41.549,0:27:43.950 +any other questions + +0:27:43.950,0:27:46.780 +>> AUDIENCE MEMBER: when are you +planning the next PC-BSD release? + +0:27:46.780,0:27:57.070 +>> CHRIS: 1.6 we're aiming for the end +of the summer to base the first one +off FreeBSD 7 + +0:27:57.070,0:28:03.680 +ideally, we'd like to use ZFS as well + +0:28:03.680,0:28:06.460 +>> MATT: KDE4 has a few bucks + +0:28:06.460,0:28:10.780 +so I think we should probably +wait for 4.1 + +0:28:10.780,0:28:17.780 +I've been reporting tons of them + +0:28:22.010,0:28:24.510 +[unclear question] + +0:28:24.510,0:28:27.950 +yes and PC-BSD 1.6 will %uh + +0:28:27.950,0:28:29.450 +include support for ZFS + +0:28:29.450,0:28:32.020 +with the front-end to it + +0:28:32.020,0:28:33.130 +and I don't want to put you on the spot + +0:28:33.130,0:28:35.530 +and %uh jails + +0:28:35.530,0:28:39.700 +I don't want to put you on the spot +that'd be cruel + +0:28:39.700,0:28:42.300 +okay well, come see us at the +table thank you very much" diff --git a/en_US.ISO8859-1/captions/2008/nycbsdcon/dixon-bsdvsgpl.sbv b/en_US.ISO8859-1/captions/2008/nycbsdcon/dixon-bsdvsgpl.sbv new file mode 100644 index 0000000000..4b35c0ab2e --- /dev/null +++ b/en_US.ISO8859-1/captions/2008/nycbsdcon/dixon-bsdvsgpl.sbv @@ -0,0 +1,932 @@ +0:00:02.199,0:00:08.939 +I + +0:00:08.939,0:00:11.849 +they said + +0:00:11.849,0:00:14.669 +welcome everybody welcome to BSD v. GPL + +0:00:14.669,0:00:19.259 +personaly i'd like to thank the organizers +and the volunteers for making the New York City BSDCon + +0:00:19.259,0:00:20.609 +possible + +0:00:20.609,0:00:23.579 +and my employer OmniTI + +0:00:23.579,0:00:24.830 +for sponsoring my talk + +0:00:24.830,0:00:26.789 +I had to get that in + +0:00:26.789,0:00:29.169 +a couple years ago I was invited to give a talk here + +0:00:29.169,0:00:33.990 +that was quite quite a lot of fun to give and seemed +be well received + +0:00:33.990,0:00:37.110 +fortunately they were fool hardy enough to invite me +back again + +0:00:37.110,0:00:42.460 +but as we know sequels really measure up so +the official title of this is + +0:00:42.460,0:00:44.890 +not the sequel to "BSD is Dying" + +0:00:44.890,0:00:51.790 +but don't worry it's just as short as the first one + +0:00:51.790,0:00:54.530 +BSD v. GSL the great debate + +0:00:54.530,0:00:57.990 +the ongoing conflict between users of the BSD + +0:00:57.990,0:01:00.880 +and GPL licenses + +0:01:00.880,0:01:02.290 +well what is the debate + +0:01:02.290,0:01:03.430 +do you mean like + +0:01:03.430,0:01:04.799 +gosh Jason + +0:01:04.799,0:01:07.000 +I didn't know there was a debate between + +0:01:07.000,0:01:10.990 +the BSD and GPL licenses + +0:01:10.990,0:01:13.130 +or like duh Jason + +0:01:13.130,0:01:14.220 +everyone knows that + +0:01:14.220,0:01:15.869 +BSD has already won + +0:01:20.220,0:01:22.360 +well for the sake of argument + +0:01:22.360,0:01:25.250 +let's assume that we're talking about the former + +0:01:25.250,0:01:27.480 +so back to the debate + +0:01:27.480,0:01:30.680 +which license is more free + +0:01:30.680,0:01:33.430 +it depends a lot on how you define freedom + +0:01:33.430,0:01:37.000 +is it a tool used to guarantee the freedom of the code + +0:01:37.000,0:01:38.560 +for everyone + +0:01:38.560,0:01:39.900 +or is it a mechanism + +0:01:39.900,0:01:42.620 +providing select freedoms + +0:01:42.620,0:01:49.620 +to a select group of users + +0:01:52.420,0:01:56.980 +I'll let you make that judgment for yourselfs + +0:01:56.980,0:01:59.540 +but first let's step back in time a moment + +0:01:59.540,0:02:01.570 +look at the origins of the BSD + +0:02:01.570,0:02:03.440 +and GPL licenses + +0:02:03.440,0:02:10.440 +which should help us better understand their +goals and philosophies + +0:02:12.420,0:02:15.079 +I know boo + +0:02:15.079,0:02:20.049 +even though work on the ancestors of UNIX +dates back to nineteen seventy four + +0:02:20.049,0:02:25.069 +it took roughly fifteen years to get to +a point where these licenses became a necessity + +0:02:25.069,0:02:26.959 +back in nineteen eighty five + +0:02:26.959,0:02:28.299 +Richard Stallman + +0:02:28.299,0:02:30.909 +a member of the staff at MIT + +0:02:30.909,0:02:31.840 +had a dispute + +0:02:31.840,0:02:34.769 +with the company named UniPress + +0:02:34.769,0:02:37.419 +another developer named James Gosling + +0:02:37.419,0:02:42.699 +developed the first version of Emacs written in C + +0:02:43.759,0:02:48.809 +after Stallman had incorporated parts Goslings Emacs +into Gnu Emacs + +0:02:48.809,0:02:53.079 +Gosling sold the rights to UniPress + +0:02:53.079,0:02:55.719 +UniPress theatened Stallman + +0:02:55.719,0:02:58.569 +against further redistribution of the Gosling bits + +0:02:58.569,0:03:01.169 +so he rewote the effected portions + +0:03:01.169,0:03:04.819 +and released a new version under a copyleft license + +0:03:04.819,0:03:08.449 +call Emacs General Public License + +0:03:08.449,0:03:10.220 +the June nineteen eighty eight issue + +0:03:10.220,0:03:12.159 +of the GNU bulletion + +0:03:12.159,0:03:15.669 +first mentioned that GNU general public +license + +0:03:15.669,0:03:18.449 +but even his earlest announcement foreshadowed + +0:03:18.449,0:03:25.449 +an underlying theme that forms the great debate + +0:03:27.009,0:03:29.599 +switching gears + +0:03:29.599,0:03:31.449 +switching gears we find ourselves + +0:03:31.449,0:03:33.609 +in the summer of 1989 + +0:03:33.609,0:03:36.010 +soaking up the summer rays in the computer +lab + +0:03:36.010,0:03:38.889 +of CSRG of Cal Berkeley + +0:03:38.889,0:03:41.019 +the Berkeley Software Distribution + +0:03:41.019,0:03:44.859 +already had thousands of copies distributed to +users + +0:03:44.859,0:03:46.769 +because the releases included the source code + +0:03:46.769,0:03:48.519 +AT&T Unix + +0:03:48.519,0:03:50.799 +as well as Berkeley's improvements + +0:03:50.799,0:03:52.480 +users were required to purchase + +0:03:52.480,0:03:55.619 +an AT&T source license + +0:03:55.619,0:03:57.079 +demand had built up + +0:03:57.079,0:03:59.139 +for a version of BSD Unix + +0:03:59.139,0:04:02.650 +that didn't require an AT&T source license + +0:04:02.650,0:04:05.269 +so BSD developers were able to put together a +release + +0:04:05.269,0:04:07.039 +free of AT&T code + +0:04:07.039,0:04:10.609 +that included the BSD originated TCP/IP networking + +0:04:10.609,0:04:13.979 +stack and it's supported utilities + +0:04:13.979,0:04:18.339 +this version was officially known as Networking Release 1 +or NET/1 + +0:04:18.339,0:04:20.359 +and was released under a liberal license + +0:04:20.359,0:04:22.839 +were the user could redistribute the code + +0:04:22.839,0:04:24.189 +modified or unmodified + +0:04:24.189,0:04:26.449 +source or binary form + +0:04:26.449,0:04:30.169 +with no accounting of roylaties to Berkeley + +0:04:30.169,0:04:32.069 +even with these liberal terms + +0:04:32.069,0:04:35.470 +hundreds of organizations step forward to paid +the one thousand dollar fee + +0:04:35.470,0:04:41.590 +for copies of the official NET/1 media + +0:04:41.590,0:04:45.989 +by 1991 the free software community had +considerable momentum + +0:04:45.989,0:04:48.239 +with the success of the GNU + +0:04:48.239,0:04:50.709 +compiler collection of userland tools + +0:04:50.709,0:04:53.059 +berkeley's own networking release 2 + +0:04:53.059,0:04:56.180 +GPLv2 and the LGPL + +0:04:56.180,0:04:58.419 +and the linux kernel + +0:04:59.639,0:05:02.130 +a fledgling knockoff of MINIX + +0:05:02.130,0:05:03.400 +created by a student + +0:05:03.400,0:05:08.219 +at the University of Helsinki + +0:05:08.219,0:05:12.879 +come 1992 the BSD community +was buckling up for a wild right + +0:05:12.879,0:05:15.080 +Unix Systems Laboratories + +0:05:15.080,0:05:17.709 +a subsidiary of AT&T + +0:05:17.709,0:05:20.050 +filed suit against BSDI + +0:05:20.050,0:05:22.840 +claiming that their derivative of Berkeley's NET/2 + +0:05:22.840,0:05:25.919 +infringed on the official the AT&T copyrights + +0:05:25.919,0:05:27.460 +by 1994 + +0:05:27.460,0:05:28.920 +the lawsuit was settled + +0:05:28.920,0:05:32.520 +largely in favor of BSDI and Cal Berkeley + +0:05:32.520,0:05:38.819 +but the impact of BSD adoption was +substantial + +0:05:38.819,0:05:41.149 +hello + +0:05:41.149,0:05:45.660 +new users gravitated to the Linux and +GNU alternative in droves + +0:05:45.660,0:05:48.469 +resulting in increased acceptance + +0:05:48.469,0:05:54.490 +of Stallman's GPL license and new manifesto + +0:05:54.490,0:05:57.910 +with a little bit of history under our belt +we can begin to take a deeper look at the + +0:05:57.910,0:05:59.180 +goals of each license + +0:05:59.180,0:06:02.479 +and the means it uses + +0:06:02.479,0:06:06.699 +and I means it uses to reach those ends + +0:06:06.699,0:06:09.909 +certainly these two licenses share +some common goals + +0:06:09.909,0:06:12.250 +they're both OSI approved licenses + +0:06:12.250,0:06:13.310 +suitable + +0:06:13.310,0:06:16.870 +for a variety of software projects + +0:06:16.870,0:06:20.459 +both licenses require attribution to remain intact + +0:06:20.459,0:06:23.770 +typically in the form of a copyright notice +within the license + +0:06:23.770,0:06:27.560 +every developer reserves the right to point +a completed works and take credit for their + +0:06:27.560,0:06:29.509 +efforts + +0:06:29.509,0:06:33.529 +admittedly some may choose not too + +0:06:33.529,0:06:37.439 +we don't have any MySQL developers in here do we + +0:06:37.439,0:06:41.729 +both licenses + +0:06:41.729,0:06:43.999 +but licenses convey freedoms to the user + +0:06:43.999,0:06:47.399 +to use the code as he or she sees fit + +0:06:47.399,0:06:50.059 +to read and learn from the code + +0:06:50.059,0:06:52.059 +use the resulting binaries + +0:06:52.059,0:06:54.470 +to develop your own applications + +0:06:54.470,0:06:59.960 +and stick it to your favorite three letter agencies + +0:06:59.960,0:07:02.619 +both licenses allow users + +0:07:02.619,0:07:04.500 +to redistribute + +0:07:04.500,0:07:06.979 +source code in binary forms + +0:07:06.979,0:07:08.930 +provided certain conditions are met + +0:07:08.930,0:07:11.949 +we'll come back to this in a few moments + +0:07:11.949,0:07:14.969 +both licenses provide some basic disclaimers + +0:07:14.969,0:07:17.699 +as a layer of protection for the developer + +0:07:17.699,0:07:21.159 +against claims of liability + +0:07:21.159,0:07:22.949 +for any direct + +0:07:22.949,0:07:24.539 +indirect + +0:07:24.539,0:07:26.550 +incidental + +0:07:26.550,0:07:27.999 +exemplary + +0:07:27.999,0:07:30.659 +thats a really hard one to get a picture of + +0:07:30.659,0:07:37.289 +or consequential damages + +0:07:37.289,0:07:39.070 +including but not limited to + +0:07:39.070,0:07:46.070 +procurement of substitute goods and services + +0:07:46.939,0:07:48.060 +loss of use, + +0:07:55.789,0:08:02.789 +loss of data, + +0:08:04.990,0:08:09.710 +lost profits, + +0:08:09.710,0:08:16.710 +or business interuption + +0:08:17.119,0:08:20.519 +I'd like to say that I did this slide before this +week + +0:08:20.519,0:08:24.449 +anyone who knows me knows that's not true + +0:08:24.449,0:08:28.599 +as we've seen both BSD and GPL licences share +some common freedoms + +0:08:28.599,0:08:31.839 +but as we know the devil's in the details + +0:08:31.839,0:08:35.600 +I'd actually like to ask for audience participation +in the following slides + +0:08:35.600,0:08:36.719 +this is my own experiment + +0:08:36.719,0:08:41.110 +to help gadge the communities familiarity with +the licenses + +0:08:41.110,0:08:42.870 +I'm going to read off a phrase + +0:08:42.870,0:08:45.990 +and I'd like you all to tell me which license +it is applicable too + +0:08:45.990,0:08:49.120 +k lets begin + +0:08:49.120,0:08:50.029 +freedom for everyone + +0:08:50.029,0:08:51.619 +even businesses + +0:08:51.619,0:08:54.810 +BSD + +0:08:54.810,0:08:57.060 +anti-capitalist socialism + +0:08:57.060,0:09:00.560 +GPL + +0:09:00.560,0:09:02.210 +thank you + +0:09:02.210,0:09:04.540 +that's correct + +0:09:04.540,0:09:08.190 +easy compliance + +0:09:08.190,0:09:09.529 +force code + +0:09:09.529,0:09:10.700 +to remain open + +0:09:10.700,0:09:13.190 +GDL + +0:09:13.190,0:09:15.230 +world domination + +0:09:15.230,0:09:18.180 +BSD + +0:09:18.180,0:09:19.690 +perhaps FreeBSD + +0:09:22.080,0:09:25.420 +I'll give you half credit + +0:09:25.420,0:09:29.960 +shut up and hack + +0:09:29.960,0:09:31.220 +brief and simple + +0:09:31.220,0:09:33.620 +BSD + +0:09:33.620,0:09:40.190 +complex and obtuse + +0:09:44.390,0:09:48.800 +this is not the ASB graph this is +something someone pointed to online + +0:09:48.800,0:09:55.800 +it represents the word count in each +version of the BSD and GPL licenses + +0:10:00.030,0:10:01.790 +yes and I think that that's true + +0:10:01.790,0:10:06.270 +the the for and third up losses + +0:10:06.270,0:10:11.340 +it was over the issue it before + +0:10:11.340,0:10:15.120 +take that as you will + +0:10:15.120,0:10:17.080 +and I mentioned previously that both include + +0:10:17.080,0:10:19.260 +some conditions regarding distribution + +0:10:19.260,0:10:23.370 +this in particular is where you see a real diversion +philosophies + +0:10:23.370,0:10:26.439 +the GPL license includes a number of +terms and conditions + +0:10:26.439,0:10:28.880 +intent on restricting the sale, use + +0:10:28.880,0:10:31.130 +and redistribution of the effect it works + +0:10:31.130,0:10:34.190 +let's look at the examples + +0:10:34.190,0:10:34.689 +first + +0:10:34.689,0:10:37.610 +your aloowed to copy and distribute the program source code + +0:10:37.610,0:10:39.090 +so long as you include + +0:10:39.090,0:10:40.920 +the copyright notice + +0:10:40.920,0:10:43.040 +the disclaimer of warrenty + +0:10:43.040,0:10:46.460 +and any notices that refer to the license + +0:10:46.460,0:10:52.230 +I wasn't sure that anybody would get this + +0:10:52.230,0:10:55.840 +you may also charge a fee for packaging or + +0:10:55.840,0:10:57.690 +transferal of the work + +0:10:57.690,0:11:02.750 +or offer warrenty protection for a fee + +0:11:02.750,0:11:07.330 +but you may not sell the software + +0:11:07.330,0:11:10.279 +you can copy or distribute any derivative works + +0:11:10.279,0:11:13.010 +providing you meet the following conditions + +0:11:13.010,0:11:17.860 +any modified files most prominently displayed +notice that the files were changed + +0:11:17.860,0:11:23.160 +it must be a license as a whole and no charge +to any thrid parties + +0:11:23.160,0:11:26.740 +and if the program reads commands interactivly when run + +0:11:26.740,0:11:28.800 +it must display an announcement + +0:11:28.800,0:11:35.800 +including a copyright notice and disclaimer +of warrenty + +0:11:37.570,0:11:40.860 +you may copy or distribute the code in binary form + +0:11:40.860,0:11:43.140 +provided you meet these conditions + +0:11:43.140,0:11:46.190 +you include the corresponding source code + +0:11:46.190,0:11:49.860 +a written offer + +0:11:49.860,0:11:52.380 +valid for at least three years + +0:11:52.380,0:11:57.720 +to provide any third party with a copy of the +source code + +0:11:57.720,0:12:04.720 +for a fee not to exceed the cost of distributing the source code + +0:12:06.120,0:12:09.690 +there are other restrictions as well we have covered +the most intrusive ones + +0:12:09.690,0:12:16.690 +the other is a mass of while the religious +right + +0:12:17.680,0:12:21.500 +to be fair the FCC licenses and without its +own restrictions + +0:12:21.500,0:12:26.620 +most of the issue mainstream the is the distributions +such as previously majesty + +0:12:26.620,0:12:31.950 +and ratified is the up for further to cosby +is the lessons for nafta + +0:12:31.950,0:12:34.420 +the only issue brought it goes one step further + +0:12:34.420,0:12:38.600 +using the issue like a modified highest the +license with no extra cost + +0:12:38.600,0:12:42.980 +but again even these have a number of his +first reactions use our freedoms + +0:12:42.980,0:12:44.500 +here's an exhaustive lives + +0:12:44.500,0:12:51.500 +of these restrictions + +0:12:52.270,0:12:58.040 +activists loyal there + +0:12:58.040,0:13:02.640 +no more free license is the easier it is to +comply + +0:13:02.640,0:13:05.190 +conversely the more complex a license becomes + +0:13:05.190,0:13:10.240 +the more likely it is that violations her + +0:13:10.240,0:13:13.250 +the saw for freedom law center + +0:13:13.250,0:13:20.150 +offers legal representation + +0:13:20.150,0:13:23.220 +the free and open source offer projects + +0:13:23.220,0:13:28.590 +the web site lead you to believe that they +defend the rights of all streets are process + +0:13:28.590,0:13:30.830 +but would review the press releases + +0:13:30.830,0:13:35.560 +shows that every single case it's a has been +released + +0:13:35.560,0:13:38.040 +in fact many of you may remember last year + +0:13:38.040,0:13:41.570 +when will it makes their story developers + +0:13:41.570,0:13:43.519 +realizing straight floaters + +0:13:43.519,0:13:46.840 +obviously if they're struggling the jeep you +know + +0:13:46.840,0:13:48.250 +what you saw + +0:13:48.250,0:13:52.860 +what they also removed rates copyright points +that the case + +0:13:52.860,0:13:55.020 +the software freedom also stepped in + +0:13:55.020,0:13:58.570 +in convince limits developers to correct the +rich states + +0:13:58.570,0:14:02.570 +but not until a storm of controversy was unleashed + +0:14:02.570,0:14:06.700 +causing hardship among the lyrics and the +it's the communities + +0:14:06.700,0:14:10.170 +the press releases carefully worded + +0:14:10.170,0:14:14.980 +to make it safe + +0:14:14.980,0:14:16.749 +to make it sound like they were betting + +0:14:16.749,0:14:20.360 +you can be as the license for compatibility + +0:14:20.360,0:14:25.420 +rather than explaining how the limits developers +infringed on race right + +0:14:25.420,0:14:31.830 +does anyone else in also + +0:14:31.830,0:14:35.030 +because copyright a softer also complex + +0:14:35.030,0:14:38.830 +the number of missus claus up we're going +to be as easy going as this is + +0:14:38.830,0:14:40.640 +i'd like to conclude my top + +0:14:40.640,0:14:43.740 +but they're not few days + +0:14:43.740,0:14:45.010 +this is number one + +0:14:45.010,0:14:47.390 +did you get a license is more free + +0:14:47.390,0:14:49.610 +although places restrictions on users + +0:14:49.610,0:14:51.540 +the integrity of freedom + +0:14:51.540,0:14:54.420 +possibly forcing uses to publish their changes + +0:14:54.420,0:14:59.390 +results in a greater level of freedom + +0:14:59.390,0:15:01.880 +the restrictions always means less free + +0:15:01.880,0:15:03.579 +what did the original developer + +0:15:03.579,0:15:04.620 +born in danger + +0:15:04.620,0:15:08.710 +did you kill always places additional restrictions +on the economy + +0:15:08.710,0:15:09.910 +modification + +0:15:09.910,0:15:12.950 +redistribution effect it works + +0:15:12.950,0:15:14.520 +that number two + +0:15:14.520,0:15:16.830 +the issue through to be still + +0:15:16.830,0:15:20.350 +he also claim that because the company can +take care + +0:15:20.350,0:15:22.110 +make the changes proprietary + +0:15:22.110,0:15:23.439 +that somehow lessons + +0:15:23.439,0:15:29.000 +the freemen's of the original up + +0:15:29.000,0:15:31.450 +this is completed on a ship + +0:15:31.450,0:15:35.060 +the result was always this tree is a ritual +developers she said + +0:15:35.060,0:15:41.130 +no evil corporation to subvert the rights +of other users to the the issue is + +0:15:41.130,0:15:42.600 +number three + +0:15:42.600,0:15:45.060 +so Richard Stallman is a socialist + +0:15:45.060,0:15:48.670 +intent on destroying commercial software companies + +0:15:48.670,0:15:49.830 +yes + +0:15:49.830,0:15:53.780 +I + +0:15:53.780,0:15:56.310 +Richard Stallman actually congress + +0:15:56.310,0:16:00.380 +intent on destroying commercial software companies + +0:16:00.380,0:16:03.920 +socialism actually from a state ownership +it means of production + +0:16:03.920,0:16:05.090 +rather + +0:16:05.090,0:16:07.870 +Stallman believes in the class of state fourteen +days of all the rules diff --git a/en_US.ISO8859-1/captions/2009/asiabsdcon/rao-kernellocking-1.sbv b/en_US.ISO8859-1/captions/2009/asiabsdcon/rao-kernellocking-1.sbv new file mode 100644 index 0000000000..3bc98ee9aa --- /dev/null +++ b/en_US.ISO8859-1/captions/2009/asiabsdcon/rao-kernellocking-1.sbv @@ -0,0 +1,2457 @@ +0:00:00.000,0:00:02.740 +My name is Attilio Rao and + +0:00:02.740,0:00:05.960 +I think that we are in time for the presentation + +0:00:05.960,0:00:10.870 +I want to ask sorry for my English because it's not really British English but I will + +0:00:10.870,0:00:12.480 +try to make this + + +0:00:12.480,0:00:16.359 +a little bit uncomfortable + +0:00:16.359,0:00:21.300 +Better? + +0:00:21.300,0:00:24.609 +Ok.Thank you.So we are going to speak about the + + +0:00:24.609,0:00:28.639 +the locking infrastructure in the FreeBSD kernel +which + +0:00:28.639,0:00:33.440 +is a bit interesting topic because + +0:00:33.440,0:00:38.890 +Its going to be with time very widely discussed on our mailing list not only + +0:00:38.890,0:00:43.100 +from developer's perspective but even from user's perspective. + +0:00:43.100,0:00:49.470 +and we will see why later + +0:00:49.470,0:00:52.990 +In this presentation we will specifically see what + +0:00:52.990,0:00:55.100 +was the situation + +0:00:55.100,0:00:57.010 +of the first + +0:00:57.010,0:00:59.150 +FreeBSD implementations + +0:00:59.150,0:01:01.120 +and what changed from that + +0:01:01.120,0:01:06.690 +what specifically what's called the SMPng era + +0:01:06.690,0:01:07.639 +and what + +0:01:07.639,0:01:10.500 +we had prior that + +0:01:10.500,0:01:12.780 + we are going to discuss + +0:01:12.780,0:01:13.579 +specifically + +0:01:13.579,0:01:19.160 +locking primitives that has been introduced with time until now + +0:01:19.160,0:01:20.910 + and + +0:01:20.910,0:01:24.730 +problems linked to + +0:01:24.730,0:01:27.620 +parellelism in general and how we solve that in + +0:01:27.620,0:01:30.950 +the FreeBSD kernel + +0:01:30.950,0:01:36.200 +You can see a table of content a little bit more detailed as + +0:01:36.200,0:01:39.850 +listing precisely what we + +0:01:39.850,0:01:43.210 +some problems like + +0:01:43.210,0:01:46.159 +Priority Inheritance + +0:01:46.159,0:01:53.159 +and Adaptive Spinning that we are going to discuss fruitfullly. + +0:01:53.370,0:01:58.890 +Mostly until FreeBSD 4.x + +0:01:58.890,0:02:00.830 +We had already moved to multitasking. + +0:02:00.830,0:02:05.210 +so the slide is a little bit confusing but + multitasking and preemptive system + +0:02:05.210,0:02:06.360 +since + +0:02:06.360,0:02:10.379 +that transition was not very + +0:02:10.379,0:02:14.180 +was not very difficult to implement in such systems + because + +0:02:14.180,0:02:17.479 +if you can see then our uniprocessor machine + +0:02:17.479,0:02:18.929 +you can get that + +0:02:18.929,0:02:20.019 +well + +0:02:20.019,0:02:24.029 +the sequential execution was +just + +0:02:24.029,0:02:25.699 +stopped by + +0:02:25.699,0:02:26.309 +preemption + +0:02:26.309,0:02:29.400 +and by arrival of interrupts + +0:02:29.400,0:02:33.969 +so you should adjustment in consistency of data structures + +0:02:33.969,0:02:36.289 +about these two issues + +0:02:36.289,0:02:37.079 +more precisely + +0:02:37.079,0:02:39.079 +we were handling + +0:02:39.079,0:02:41.779 +the interrupts and transitions through + +0:02:41.779,0:02:43.370 +a mechanism + +0:02:43.370,0:02:45.779 +called SPL + +0:02:45.779,0:02:50.769 +and for kernel threads, threads running in the kernel we were disabling + +0:02:50.769,0:02:51.379 +preemption + +0:02:51.379,0:02:53.019 +in order to avoid + +0:02:53.019,0:02:55.809 +the corruption of the data structure + +0:02:55.809,0:02:57.519 +This approach while was + +0:02:57.519,0:03:00.629 +pretty good on uniprocessor machines + +0:03:00.629,0:03:02.269 +was actually + +0:03:02.269,0:03:04.270 +impredictable for + +0:03:04.270,0:03:06.219 +the SMP environments + +0:03:06.219,0:03:10.199 +more precisely because we had more coures that + +0:03:10.199,0:03:12.959 +was running thread per time + +0:03:12.959,0:03:13.909 +and so + +0:03:13.909,0:03:14.980 +parallel + +0:03:14.980,0:03:19.309 +accesses to the data structures were possible + +0:03:19.309,0:03:21.290 +in order to + +0:03:21.290,0:03:22.469 +to avoid + +0:03:22.469,0:03:24.149 +big problems in the kernel + +0:03:24.149,0:03:25.799 +we have to just + +0:03:25.799,0:03:26.739 +allow + +0:03:26.739,0:03:28.989 +the entering of + +0:03:28.989,0:03:32.309 +one thread per time into kernel + +0:03:32.309,0:03:35.379 +while that was a pretty good approach + +0:03:35.379,0:03:39.049 +for workloads that were nearly user space + +0:03:39.049,0:03:40.969 +for work loads + +0:03:40.969,0:03:45.619 + requiring a lot of IO for example they were wateful because they wasn't + +0:03:45.619,0:03:47.839 +getting any advantage from the new + +0:03:47.839,0:03:49.819 +SMP architecture + +0:03:49.819,0:03:52.749 +like the parallelism was basically zero + +0:03:52.749,0:03:55.189 +at least in the kernel + +0:03:55.189,0:03:55.949 +in order + +0:03:55.949,0:04:00.650 +to fix that a new project was created +called SMP + +0:04:00.650,0:04:01.470 +New generation + + +0:04:01.470,0:04:05.169 +or NG + +0:04:05.169,0:04:07.309 +as you can see it from the slide + +0:04:07.309,0:04:10.329 +the entering in the kernel was preempted + +0:04:10.329,0:04:12.569 +by using Big Lock + +0:04:12.569,0:04:19.569 +called BKL basically + +0:04:23.199,0:04:28.109 +With FreeBSD 5.x we had the SMP new generation project + +0:04:28.109,0:04:30.110 +basically it was + +0:04:30.110,0:04:31.509 +a sanitization of the + +0:04:31.509,0:04:34.539 +of all + +0:04:34.539,0:04:40.039 +our kernel and the engineering over lot of mechanism inside our kernel. We could see it + +0:04:40.039,0:04:44.709 +FreeBSD 4.x and FreeBSD 5.x as mainly two different kernels + +0:04:44.709,0:04:45.550 +because of + +0:04:45.550,0:04:50.150 +substantial subsystem were rewritten and + +0:04:50.150,0:04:51.830 +were written with the + +0:04:51.830,0:04:56.949 +idea to use and implement a real parallelism in mind. + +0:04:56.949,0:05:02.610 + we can say that basically it was a major task a very big task + +0:05:02.610,0:05:04.029 +and that it required + +0:05:04.029,0:05:06.669 +a lot of years to be brought + +0:05:06.669,0:05:08.900 +in a good shape at least + +0:05:08.900,0:05:11.379 +In Italy, the people gave + +0:05:11.379,0:05:13.069 +a lot of + +0:05:13.069,0:05:16.350 +complaining about the + +0:05:16.350,0:05:20.430 +un-robustness of FreeBSD 5.x but + +0:05:20.430,0:05:22.249 +probably that's because they couldn't even + +0:05:22.249,0:05:28.929 +see that the changes were really really important and really huge + +0:05:28.929,0:05:34.490 +however for FreeBSD 5.x based this initial SMP system + +0:05:34.490,0:05:37.070 +inheriting from BSD/OS + +0:05:37.070,0:05:39.309 +that kindly + +0:05:39.309,0:05:42.699 +released this code above that + +0:05:42.699,0:05:44.009 +and the + + +0:05:44.009,0:05:46.579 +the process was break up in + +0:05:46.579,0:05:51.069 +some precise tasks at least in Italy + +0:05:51.069,0:05:55.429 +Mainly the first things was introducing in the kernel + +0:05:55.429,0:06:00.180 +new set of atomic instruction and locking primitives + +0:06:00.180,0:06:01.520 +Then introducing + +0:06:01.520,0:06:05.380 +an abstraction called interrupt threads that we are going to discuss + +0:06:05.380,0:06:06.929 +rather later but + +0:06:06.929,0:06:12.319 +it was basically restored completely the interrupt mechanism that was in the FreeBSD 4.x + +0:06:14.439,0:06:16.490 +the the BKL + +0:06:16.490,0:06:19.210 +lock was moved to a real + +0:06:19.210,0:06:20.679 +mutex called Giant + +0:06:20.679,0:06:23.180 +that still exists in our kernel + +0:06:23.180,0:06:26.660 +and they were introduced some threading primitives + +0:06:26.660,0:06:28.019 +the + +0:06:28.019,0:06:30.499 +like and and on and + +0:06:30.499,0:06:32.280 + threading primitives + +0:06:32.280,0:06:34.319 +called also KSE + +0:06:34.319,0:06:37.009 +which are actually never used in our kernel + +0:06:37.009,0:06:41.620 +and that being their being exit out in the past year + +0:06:41.620,0:06:43.409 +and the + +0:06:43.409,0:06:45.259 +slowly of the porting of + +0:06:45.259,0:06:50.459 +all the older subsystems to a finer locking was started + +0:06:50.459,0:06:55.919 +I have to say this task is not still completed, its still going on but + +0:06:55.919,0:06:58.889 +we are really good shape about that + +0:06:58.889,0:07:02.429 +just few subsystems remain which are still Giant protected + +0:07:02.429,0:07:05.939 +and with new release that we're going to ship this year, I think that we made + +0:07:05.939,0:07:10.220 +a very huge step forward in this direction + +0:07:12.319,0:07:18.599 +really the the SMPng has been considered closed around the end of + +0:07:18.599,0:07:20.600 +2007 + +0:07:20.600,0:07:22.579 +but the + +0:07:22.579,0:07:23.819 +the + +0:07:23.819,0:07:27.539 +the important parts where this initial moving + +0:07:27.539,0:07:32.669 +I rather thing that's not listed here but I can tell you is that + +0:07:32.669,0:07:38.279 +even that if Giant was preventing any parallelism initial parallelism + +0:07:38.279,0:07:43.219 +that were imported new kernel memory allocator that was + +0:07:43.219,0:07:45.009 +that I discovered + +0:07:45.009,0:07:48.439 +and the scheduler was move with a seperate lock + +0:07:48.439,0:07:50.449 +in order to + +0:07:50.449,0:07:52.080 +start getting some + +0:07:52.080,0:07:54.699 +a little bit of concurrency + +0:07:54.699,0:07:59.099 +a real concurrency + +0:07:59.099,0:08:01.520 +the + +0:08:01.520,0:08:06.280 +before to speak about FreeBSD specifics we can start digging in about + +0:08:06.280,0:08:08.219 +what kind of + +0:08:08.219,0:08:12.729 +of locking primitives you can find in our kernel. + +0:08:12.729,0:08:15.780 +from a more historical point of view + +0:08:15.780,0:08:19.710 +we have some versions of mutex which + +0:08:19.710,0:08:20.919 +I assume + +0:08:20.919,0:08:24.809 +people here knows about that but I'm going to give a little explanation + +0:08:24.809,0:08:26.449 +for people that doesn't know + +0:08:26.449,0:08:28.939 +a mutex is basically + +0:08:28.939,0:08:30.739 +a lock allowing to access + +0:08:30.739,0:08:36.700 +to some protected data's thread to just one thread per time + +0:08:36.700,0:08:38.150 +so if a thread + +0:08:38.150,0:08:39.690 +owns the lock, + +0:08:39.690,0:08:40.760 +owns the mutex + +0:08:40.760,0:08:42.539 +other threads + +0:08:42.539,0:08:44.039 +won't be able to + +0:08:44.039,0:08:46.090 +to access to this until + +0:08:46.090,0:08:48.730 +this lock is released + +0:08:48.730,0:08:50.430 +we offer even + +0:08:50.430,0:08:54.890 +some kind of locks called R/W lock Read/Write lock + +0:08:54.890,0:08:57.920 +which are basically a + +0:08:57.920,0:09:03.050 +locks that can be acquired in two different versions + +0:09:03.050,0:09:04.060 +one version + +0:09:04.060,0:09:07.980 +is the write lock which is the same as the mutex just one + +0:09:07.980,0:09:10.010 +in the protected part per time + +0:09:10.010,0:09:13.860 +and other one is the read mode which basically + +0:09:13.860,0:09:15.100 +allows + +0:09:15.100,0:09:18.410 +all the thread willing to acquire to read mode to + +0:09:18.410,0:09:23.699 +concurrently adjust to the structure but prevents the threads from + +0:09:23.699,0:09:25.390 +writing nto the protected path. + +0:09:25.390,0:09:28.890 +while the reader..while they are readers + +0:09:28.890,0:09:30.280 +then we have even + +0:09:30.280,0:09:33.030 +the locks called the Read Mostly + +0:09:33.030,0:09:37.570 +which are basically the same of Read/Write Locks but are + +0:09:37.570,0:09:42.500 +they have some optimization in order to make the Read + +0:09:42.500,0:09:44.180 +part be really fast + +0:09:44.180,0:09:46.930 +and to have like + +0:09:46.930,0:09:48.180 +zero overhead + +0:09:48.180,0:09:51.410 +zero overhead kind of lock + +0:09:51.410,0:09:53.350 +from the read path while + +0:09:53.350,0:09:55.590 +probably the write path is even + +0:09:55.590,0:09:59.210 +heavier than the other one but if you think about cases that + +0:09:59.210,0:10:01.710 +just + + +0:10:01.710,0:10:02.750 +where + +0:10:02.750,0:10:06.980 +there are a lot of reader chases and very few writer chases you can find that a + +0:10:06.980,0:10:08.220 +very useful + +0:10:08.220,0:10:11.070 +very useful primitive + +0:10:11.070,0:10:11.850 +then we have + +0:10:11.850,0:10:14.360 +some form of Wait channels + +0:10:14.360,0:10:16.030 +Wait channels + +0:10:16.030,0:10:17.140 +basically are what + +0:10:17.140,0:10:21.700 +generalizations of what other people con call like + +0:10:22.470,0:10:24.240 +condition variable and + +0:10:24.240,0:10:28.240 +they basically let that thread sleep + +0:10:28.240,0:10:30.870 +under some conditions that are + +0:10:30.870,0:10:35.200 +that are previously started with some + +0:10:35.200,0:10:36.610 +some variables + +0:10:36.610,0:10:37.150 +usually + +0:10:37.150,0:10:39.500 +having a Wait channel means that its + +0:10:39.500,0:10:45.080 +chases are controlled through another locking primitive like a mutex + +0:10:45.080,0:10:46.640 +or R/Wlock + +0:10:46.640,0:10:52.010 +and so often the Wait channel is associated to its + +0:10:52.010,0:10:53.620 +to its locking primitive + +0:10:53.620,0:11:00.140 +usually if you have no necessity to use a Wait channel without a primitive + +0:11:00.140,0:11:04.150 +a locking primitive you probably have bad code + +0:11:04.150,0:11:06.830 +but there are some edge cases + +0:11:06.830,0:11:09.660 +with that seem possible + +0:11:09.660,0:11:13.550 +As last thing FreeBSD sub primitive counting semaphore + +0:11:13.550,0:11:15.290 +even if thats considered not featured + +0:11:15.290,0:11:17.710 +as we are going to see I think they're going to see it and + +0:11:17.710,0:11:23.570 +its usage is pretty much discouraged + +0:11:23.570,0:11:28.320 +basically FreeBSD you can consider locking primative divided into three classes + +0:11:28.320,0:11:31.250 +three classes of + +0:11:31.250,0:11:32.450 +of locking + +0:11:32.450,0:11:34.090 +based mainly in + +0:11:34.090,0:11:35.600 +particular + +0:11:35.600,0:11:37.340 +from an outside perspective + +0:11:37.340,0:11:38.690 +based on the behavior + +0:11:38.690,0:11:42.680 +the contending threads as you regard of the lock + +0:11:42.680,0:11:48.100 +for example in case of a mutex you can can get that + +0:11:48.100,0:11:53.360 +spinning and blocking mutex do very different things about the contenders + +0:11:53.360,0:11:59.680 +as we are going to see more of this later + +0:11:59.680,0:12:03.410 +usually in the traditional literature, + +0:12:03.410,0:12:05.430 +there are just two + +0:12:05.430,0:12:07.280 +cases of the lock classes mainly + +0:12:07.280,0:12:08.620 +you will find the + +0:12:08.620,0:12:11.200 +spinning lock and the blocking lock + +0:12:11.200,0:12:14.370 +or what they called the sleeping lock + +0:12:14.370,0:12:16.670 +the I think that + +0:12:16.670,0:12:21.020 +as we're going to see why we have three types I think that things will be clear but + +0:12:21.020,0:12:27.100 +if you have any questions please ask us. Thats not a problem + +0:12:27.100,0:12:29.930 +Spinning primitives as I told you + +0:12:29.930,0:12:32.810 +allows the contesting thread to + +0:12:32.810,0:12:36.120 +to check the status of the lock periodically + +0:12:36.120,0:12:37.590 +and the + +0:12:37.590,0:12:40.420 +and they just do busy waiting around + +0:12:40.420,0:12:41.890 + the locking variable + +0:12:41.890,0:12:46.400 +as the spinning primitive FreeBSD just offers mutex + +0:12:46.400,0:12:50.689 +What are the problems linked with this kind of, with this class + +0:12:50.689,0:12:53.869 +of locks? Mainly its that CPU + +0:12:53.869,0:12:58.130 +remains busy without doing really nothing useful + +0:12:58.130,0:12:59.740 +it happens + +0:12:59.740,0:13:03.620 +that if several threads contest on the + +0:13:03.620,0:13:04.870 +on the locks + +0:13:04.870,0:13:08.210 +basically they share the same cache line where the lock is + +0:13:08.210,0:13:10.220 +where the lock is + +0:13:10.220,0:13:12.400 +that means that + +0:13:12.400,0:13:17.470 +contesting or sharing a cache line is a lot underlying activity + +0:13:17.470,0:13:20.150 +on a lot of architectures like for example + +0:13:20.150,0:13:23.660 +having a lot of snoop messages between CPUs + +0:13:23.660,0:13:26.450 +and some buses + +0:13:26.450,0:13:28.120 +some buses traffic + +0:13:28.120,0:13:31.980 +which means in a variety operations + +0:13:31.980,0:13:35.740 +and the last things even the most important you can note is that interrupts + +0:13:35.740,0:13:37.120 +are disabled + +0:13:37.120,0:13:39.330 +while spin locks are held + +0:13:39.330,0:13:40.810 +that was + +0:13:40.810,0:13:42.979 +that happens mainly because there are + +0:13:42.979,0:13:45.140 +there were identified in the past by some + +0:13:45.140,0:13:47.970 +kind of deadlocks possible + +0:13:47.970,0:13:50.180 +if you were going to lead + +0:13:50.180,0:13:51.710 +the spin locks + +0:13:51.710,0:13:55.900 +the interrupts enabled while holding a spin lock. In particular + +0:13:55.900,0:13:58.180 +you could find that there are + +0:13:58.180,0:14:02.530 +some problems with the interrupts angling good in the botom half that was + +0:14:02.530,0:14:05.040 +going to deadlock + +0:14:05.040,0:14:10.250 +Its not very simple to understand the thing so I've left out + +0:14:10.250,0:14:12.360 +but if you want to know + +0:14:12.360,0:14:15.990 +we could speak later probably + +0:14:17.820,0:14:21.320 +with spinning primitives we are even blocking primitives + +0:14:21.320,0:14:22.890 +blocking primitives + +0:14:25.260,0:14:26.860 +allows the + +0:14:26.860,0:14:28.440 +basically the contenders to be + +0:14:28.440,0:14:30.980 +descheduled from the runqueue + +0:14:30.980,0:14:35.790 +to be put on another kind of container + +0:14:35.790,0:14:38.000 +put on another kind of container + +0:14:38.000,0:14:40.489 +and basically + +0:14:40.489,0:14:41.399 +context switch immediately + +0:14:41.399,0:14:44.360 +immediately. + +0:14:44.360,0:14:49.440 +then we put again on runqueue of the scheduler just once the just when the owner + +0:14:49.440,0:14:51.570 + is going to release the lock + +0:14:51.570,0:14:53.260 +and it will be the owner + +0:14:53.260,0:14:56.930 +the owner that was going to + +0:14:56.930,0:15:00.310 +do all the operations about that + +0:15:00.310,0:15:05.550 +we have several primitives implemented as blocking primitives like mutexes + +0:15:05.550,0:15:10.470 +R/W locks and R-M locks + +0:15:11.430,0:15:13.140 +with + +0:15:13.140,0:15:16.890 +basically with + +0:15:16.890,0:15:21.780 +blocking primitives we have a lot of advantages over the spinning mutex + +0:15:21.780,0:15:24.650 +like having the contenders + +0:15:24.650,0:15:26.560 +that + +0:15:26.560,0:15:27.590 +that sleeps + +0:15:27.590,0:15:31.840 +or that blocks avoids CPU busyness + +0:15:31.840,0:15:34.660 +and mainly we can leave the + +0:15:34.660,0:15:37.150 +we can leave the + +0:15:37.150,0:15:42.040 +we can leave that basically the interrupts out + +0:15:42.040,0:15:45.760 +that happens mainly because the interrupts code is just allowed + +0:15:45.760,0:15:50.710 + at least the bottom of one is just allowed + +0:15:50.710,0:15:52.070 +to use spin locks + +0:15:52.070,0:15:56.049 + probably if it was going to use blocking primitives + +0:15:56.049,0:16:01.060 +we wouldnt have been able to disable interrupts here + +0:16:01.060,0:16:02.239 +There are however some + +0:16:02.239,0:16:04.790 +big drawbacks that as you will see + +0:16:04.790,0:16:07.210 +we handle in FreeBSD + +0:16:07.210,0:16:11.280 +in order to make the blobking primitives our + +0:16:11.280,0:16:13.540 +how could I tell + +0:16:13.540,0:16:16.440 +the first choice in terms of blocking + +0:16:16.440,0:16:19.690 +where the problem called Priority Inversion + +0:16:19.690,0:16:21.899 + and we have + +0:16:21.899,0:16:27.589 +the problem that context switches are very heavy in particular + +0:16:27.589,0:16:30.209 +on machines that FreeBSD uses as referral + +0:16:30.209,0:16:33.500 +like E38 and the MD64 + +0:16:33.500,0:16:37.940 +but as you're going to see we've used two techniques in order to + +0:16:37.940,0:16:40.020 +to cope with that + +0:16:42.020,0:16:45.830 +another thing is that while you cant + +0:16:45.830,0:16:47.920 +allow + +0:16:47.920,0:16:50.089 +context switches while having + +0:16:50.089,0:16:52.570 +while holding spin lock + +0:16:52.570,0:16:55.249 +it's obvious you cant + +0:16:55.249,0:16:59.580 +acquire a locking primitive while holding a spin lock + +0:16:59.580,0:17:02.110 +that's an important rule in FreeBSD + +0:17:02.110,0:17:06.089 +that sometimes its confused and often its not + +0:17:06.089,0:17:07.470 +observed + +0:17:07.470,0:17:09.929 +that leads to block refusal + +0:17:12.170,0:17:16.610 +usually you will always prefer a blocking primitive for a spin lock + +0:17:16.610,0:17:22.159 +if not in some very particular condition like what + +0:17:22.159,0:17:25.010 +Alrick said about the interrupt and even + +0:17:25.010,0:17:26.090 +about the + +0:17:28.160,0:17:30.570 +some parts that are very very short + +0:17:30.570,0:17:33.629 +we should have some example in the kernel even if I can + +0:17:33.629,0:17:35.390 +I can tell you one right now + +0:17:35.390,0:17:38.770 +I have no idea actually + +0:17:38.770,0:17:39.500 +so that + +0:17:39.500,0:17:43.740 +we're going to see the problemslinked with the blocking primitives the first one is + +0:17:43.740,0:17:45.679 +called Priority Inversion + +0:17:45.679,0:17:46.389 +basically + +0:17:46.389,0:17:49.130 +it could happen that like a thread A + +0:17:49.130,0:17:51.410 +which has a priority + +0:17:51.410,0:17:55.380 +owns a lock. call it L for example + +0:17:55.380,0:17:58.710 +then another thread with another priority than this one + +0:17:58.710,0:18:00.690 +locks on this lock + +0:18:00.690,0:18:03.299 +what happens is that the second thread + +0:18:03.299,0:18:04.120 +the thread B + +0:18:04.120,0:18:05.870 +for example + +0:18:05.870,0:18:08.920 +will need to wait for a lower priority thread + +0:18:08.920,0:18:13.070 +to finish its work load + +0:18:13.070,0:18:15.120 +we + +0:18:15.120,0:18:17.780 +solve this problem actually in the + +0:18:17.780,0:18:21.170 +kernel using a technique called priority propogation + +0:18:21.170,0:18:22.020 +basically + +0:18:22.020,0:18:24.620 +what happens is that priority of thread B + +0:18:25.760,0:18:27.880 +is lent to thread A + +0:18:27.880,0:18:31.460 +until it doesn't release the lock + +0:18:31.460,0:18:34.760 +of its directly implemented in the container + +0:18:34.760,0:18:36.180 +the turnstiles + +0:18:37.870,0:18:39.530 +while that could be done + +0:18:39.530,0:18:44.290 +even on the primitive it has been much convenient to use the container for + +0:18:44.290,0:18:45.190 +that + +0:18:45.190,0:18:45.990 +because + +0:18:45.990,0:18:52.990 +it was going to offer some advantage we are going to see right now + +0:18:53.030,0:18:54.240 +just note that + +0:18:54.240,0:18:56.090 +Read locks + +0:18:56.090,0:18:57.310 +cannot support + +0:18:57.310,0:19:03.430 +priority propogation fixes for read lock that happens because you'd like to + +0:19:03.430,0:19:07.290 +the turnstile should keep track of all the readers + +0:19:07.290,0:19:11.100 +and these would be very very expensive from + +0:19:11.100,0:19:12.880 +from a + +0:19:12.880,0:19:15.540 +from a point of view of the overhead + +0:19:15.540,0:19:19.800 +and even I think I've tried to do something in this regard and I + +0:19:19.800,0:19:24.050 +saw that there was some races that were trying to + +0:19:24.050,0:19:29.390 +acquire a spin lock as base even in fast path so it was a + +0:19:29.390,0:19:31.320 +an impredicable way + +0:19:31.320,0:19:32.380 +I will tell + +0:19:32.380,0:19:37.200 +at least for what we found so far + +0:19:37.200,0:19:37.630 +basically + +0:19:37.630,0:19:39.070 + what happens + +0:19:39.070,0:19:42.150 +about the priority propogation is that the + +0:19:42.150,0:19:44.830 +the threads and the turnstiles + +0:19:44.830,0:19:47.000 +are chained together + +0:19:47.000,0:19:48.350 +the thread + +0:19:48.350,0:19:50.970 +owns the a pointer + +0:19:50.970,0:19:53.710 +to wrench the turnstile is sleeping on + +0:19:53.710,0:19:58.540 +and the turnstile owns a pointer above + +0:19:58.540,0:20:00.549 +the owner of the lock + +0:20:00.549,0:20:04.620 +what happens is that for example in this case we have + +0:20:05.080,0:20:08.070 +a sleeper which is going to sleep on a turnstile + +0:20:08.070,0:20:08.990 +the first lock + +0:20:08.990,0:20:13.470 +which has a priority of one hundred and twenty eight + +0:20:14.120,0:20:15.520 +the turnstile + +0:20:15.520,0:20:18.370 +to the pointer + +0:20:18.370,0:20:20.570 +ts_owner knows which is its owner + +0:20:20.570,0:20:26.150 +and this owner has a priority of two hundred and fifty six + +0:20:26.150,0:20:31.120 +well as you know higher level, higher value means lower priority. so if this is +0:20:31.120,0:20:34.960 +a suitable pace for priority propogation + +0:20:34.960,0:20:40.820 +but what happens is that this owner is actually sleeping on another turnstile + +0:20:40.820,0:20:43.419 +and the other owner + +0:20:43.419,0:20:48.820 +of the second turnstile has always the same priority of its sleepers + +0:20:48.820,0:20:50.750 +so + +0:20:50.750,0:20:55.530 +just propogating priority to the first owner was just unuseful because the first + +0:20:55.530,0:20:56.340 +one + +0:20:56.340,0:20:57.320 +could + +0:20:57.320,0:20:58.760 +still + +0:20:58.760,0:21:00.580 + keep the chain to a + +0:21:00.580,0:21:04.820 +lower priority so it's was going to be propogated to the first one + +0:21:04.820,0:21:07.679 + actually running + +0:21:07.679,0:21:09.870 +owner of the chain + +0:21:09.870,0:21:14.670 +this is the situation after the propogation as you can see all of threads in the chain + +0:21:14.670,0:21:16.559 +has the same priority + +0:21:16.559,0:21:17.950 +either possible + +0:21:17.950,0:21:24.480 +in this case the one the last one arriving + +0:21:25.750,0:21:31.720 +there are question about that + +0:21:31.720,0:21:34.780 +no? + +0:21:34.780,0:21:36.760 +yeah when the + +0:21:36.760,0:21:39.720 +when the for example the third owner + +0:21:39.720,0:21:41.679 +the second owner there + +0:21:41.679,0:21:43.659 +when it goes to release the lock + +0:21:43.659,0:21:47.010 +it basically brings back the priority to the + +0:21:47.010,0:21:49.340 +to the + +0:21:49.340,0:21:52.490 +twenty hundred and sixty five to all the chains + +0:21:52.490,0:21:54.650 +he is responsible for + +0:21:54.650,0:22:01.179 +so it just happens at locking operation + +0:22:01.179,0:22:04.159 +and that is what we do about the Priority Inversion + +0:22:04.159,0:22:09.970 +inorder to fix instead the overhead given by the + +0:22:09.970,0:22:14.030 +big amount of context switch we use another technique called adaptive spinning + +0:22:14.030,0:22:16.030 +basically + +0:22:16.030,0:22:20.260 +as the context switch brings a lot of overhead + +0:22:22.310,0:22:26.090 +we prefer to not do + +0:22:26.090,0:22:27.770 +completely a context switch + +0:22:27.770,0:22:30.760 +in the case the lock owner is still running + +0:22:30.760,0:22:32.190 +on a runqueue + +0:22:32.190,0:22:38.340 +because there are very good chance that the owner is going to release the lock very early + +0:22:40.440,0:22:43.990 +that means that for example + +0:22:43.990,0:22:46.070 +we choose just to spin + +0:22:46.070,0:22:49.149 +in order to wait that the state of the + +0:22:49.149,0:22:52.240 +lock changed or the state of the owner + +0:22:52.240,0:22:57.660 +was going to change like the owner going to sleep on another turstile + +0:22:57.660,0:22:59.140 +and the + +0:22:59.140,0:23:03.270 +basically we, there have been very big measurement even in the + +0:23:03.270,0:23:07.510 +another operating system like solice that + +0:23:07.510,0:23:12.300 +where I think we brought in this approach the first time + +0:23:12.300,0:23:16.430 +that we're we're showing + +0:23:16.430,0:23:23.430 +a very big improvement in performance from this technique + +0:23:25.790,0:23:30.640 +apart from the two types of primitives, these are sleeping primitives + +0:23:30.640,0:23:36.120 +now there is a consideration we have to make about that + +0:23:36.120,0:23:38.110 +basically sleeping primitives + +0:23:38.110,0:23:42.320 +should be in theory just the + +0:23:42.320,0:23:44.340 +the wait channels + +0:23:44.340,0:23:49.170 +wait channels should have been the only one implemented using the + +0:23:49.170,0:23:50.630 +container called + +0:23:50.630,0:23:52.760 +sleepqueue + +0:23:52.760,0:23:53.910 +but + +0:23:53.910,0:23:56.170 +due to some legacy + +0:23:56.170,0:24:01.000 +the actually the sleepqueues were used to implement other kind of other + +0:24:01.000,0:24:03.290 +kinds of lock like the + +0:24:03.290,0:24:04.219 +lockmgr + +0:24:04.219,0:24:08.080 +and the sx locks and the + +0:24:08.080,0:24:11.100 +basically the + +0:24:11.100,0:24:13.679 +semaphore's condvars too + +0:24:13.679,0:24:16.010 +that has been this is + +0:24:16.010,0:24:18.809 +going to give some problems actually + +0:24:18.809,0:24:19.350 +because + +0:24:20.450,0:24:24.820 +as we're going to see + +0:24:24.820,0:24:26.889 +and as you can see on the line too + +0:24:26.889,0:24:27.929 +in the FreeBSD + +0:24:27.929,0:24:31.600 +while sleeping threads should not hold any kind of lock + +0:24:31.600,0:24:33.809 +neither blocking nor spinning + +0:24:33.809,0:24:36.770 +thats a simple thing to explain + +0:24:36.770,0:24:40.200 +we just want to enforce very + +0:24:40.200,0:24:43.490 +we just want to enforce + +0:24:43.490,0:24:46.060 +correct semantics of locking + +0:24:46.060,0:24:47.880 +so imagine to keep a lock + +0:24:47.880,0:24:50.190 +a blocking primitive while + +0:24:50.190,0:24:50.729 +sleeping + +0:24:50.729,0:24:53.010 +it's going to waste a lot of time + +0:24:53.010,0:24:56.530 +because all the contenders are going to + +0:24:56.530,0:24:58.760 +are going to start on the + +0:24:58.760,0:25:01.400 +lock owner which is sleeping + +0:25:01.400,0:25:03.120 +basically in fact what + +0:25:03.120,0:25:07.169 +as you should know condition variables do usually is to drop the lock + +0:25:07.169,0:25:11.070 +once it was passed to the primitives + +0:25:11.070,0:25:12.380 +in this case + +0:25:14.170,0:25:18.249 +basically we just dont allow that this means that's the + +0:25:18.249,0:25:23.160 +the same conditions happens even for other kinds of lock + +0:25:23.160,0:25:25.540 +lockmgr and the sx lock + +0:25:25.540,0:25:26.860 +so you cant hold + +0:25:26.860,0:25:29.410 +a mutex for example + +0:25:29.410,0:25:33.640 +of blocking mutex an R/W lock while trying to acquire + +0:25:33.640,0:25:38.559 +a lockmgr and sx + +0:25:38.559,0:25:41.850 +this is going to create some problems because + +0:25:41.850,0:25:46.830 +in some parts that is unavoidable so you have to drop the lock for example and try + +0:25:46.830,0:25:48.190 +to acquire + +0:25:48.190,0:25:49.770 +the other primitive + +0:25:49.770,0:25:51.320 +which is going to + +0:25:53.400,0:25:59.110 +and so can create some raisee problems + +0:26:00.130,0:26:04.779 +as the sleepqueues are born just to serve wait channels + +0:26:04.779,0:26:09.190 + they don't track owner too so they dont care about priority propogation and priority inversion problem + +0:26:09.190,0:26:14.430 +just because sleepqueues entirely should not have work + +0:26:14.430,0:26:20.150 +so for example lockmgr and sx have not priority propogation + +0:26:20.150,0:26:22.360 +systems and the + +0:26:22.360,0:26:29.360 +so they are discouraged to be used even for this thing mainly + +0:26:31.590,0:26:34.930 +sure + +0:26:36.780,0:26:39.000 +it's you mean why it's not + +0:26:39.000,0:26:41.790 +why doesnt blocking primitives exist yeah? + +0:26:41.790,0:26:44.250 +so imagine that for example the + +0:26:44.250,0:26:45.570 +you have a wait channel + +0:26:45.570,0:26:47.679 +condvar a condition variable + +0:26:47.679,0:26:50.950 +or M sleep + +0:26:50.950,0:26:52.090 +M sleep + +0:26:52.090,0:26:54.910 +the primitive that allows you to sleep on + +0:26:54.910,0:26:57.850 +a condition variable for example + +0:26:57.850,0:26:58.870 +however + +0:27:00.510,0:27:02.270 +the you are + +0:27:02.270,0:27:03.350 +using the blocking + +0:27:03.350,0:27:06.930 +the using the turnstile you will go to a + +0:27:06.930,0:27:12.110 +always the mechanism of priority propogation and priority inversion handling.Its + +0:27:12.110,0:27:13.760 +not very + +0:27:13.760,0:27:14.970 +it's pretty + +0:27:14.970,0:27:17.320 +it's not a simple operation + +0:27:17.320,0:27:20.219 +it acquires even some kind of spin locks + +0:27:20.219,0:27:22.650 + in order to avoid some raises + +0:27:22.650,0:27:23.340 +and so + +0:27:23.340,0:27:24.289 +it + +0:27:24.289,0:27:26.590 +so it has an overhead + +0:27:26.590,0:27:31.770 +if you do in this case it will be not to be useful it will be completely unuseful to have + +0:27:31.770,0:27:34.159 +a mechanism like that so + +0:27:34.159,0:27:37.410 +in theory if you just would have used + +0:27:37.410,0:27:41.320 +a sleeping the sleepqueue for wait channels + +0:27:41.320,0:27:42.990 +you are to add + +0:27:42.990,0:27:46.640 +bigperformance boost than just using the turnstile + +0:27:46.640,0:27:49.330 +for the same problem + +0:27:49.330,0:27:51.310 +in theory + +0:27:51.310,0:27:54.780 +but what happened is that other locks are implementedo + +0:27:54.780,0:27:55.839 +using this sleepqueue + +0:27:55.839,0:27:58.070 +that should have not be happened + +0:27:58.070,0:27:59.260 +on the principle + +0:27:59.260,0:28:02.960 +really I'm not sure who introduced the sx lock + +0:28:02.960,0:28:04.440 +I'm actually not sure + +0:28:04.440,0:28:06.280 +and even the lockmgr + +0:28:06.280,0:28:09.870 +but + +0:28:09.870,0:28:12.340 +however + +0:28:12.340,0:28:17.669 +as you could have seen before the three containers create a heirarchy that + +0:28:17.669,0:28:20.090 +should not be broken like + +0:28:20.090,0:28:21.639 +you have spinqueues + +0:28:21.639,0:28:26.900 +you have spin locks you have blocking primitives and sleeping primitives and + +0:28:26.900,0:28:31.470 +you cannot acquire you cannot mix them there are precise rules like + +0:28:31.470,0:28:33.710 +on the top the sleeping primitive + +0:28:33.710,0:28:37.690 +in the mid the blocking primitive and in the end the spinning primitive + +0:28:38.900,0:28:44.440 +the main choice will be to use blocking primitives always + +0:28:44.440,0:28:48.240 +because as you can see we handled a lot of problem that they have + +0:28:48.240,0:28:49.659 +and the practice + +0:28:49.659,0:28:52.229 +they have proven to be very + +0:28:52.229,0:28:53.799 +very helpful + +0:28:53.799,0:28:54.999 +but sometimes + +0:28:56.789,0:28:58.790 +some nasty conditions can happen + +0:28:58.790,0:29:02.900 +for example one of the most widespread is the + +0:29:02.900,0:29:06.350 +using a mallok with a flag M_WAITOK + +0:29:06.350,0:29:11.240 +in FreeBSD that means that if the allocator is pretty busy or going to + +0:29:11.240,0:29:12.680 + to sleep + +0:29:12.680,0:29:15.760 +in order to retreive your memory + +0:29:15.760,0:29:17.890 +and if you do with a lock hold + +0:29:17.890,0:29:22.080 +you're going to violate one of our rules and its not + +0:29:22.080,0:29:23.440 +possible + +0:29:23.440,0:29:25.320 +another one is just we just + +0:29:25.320,0:29:28.299 +said before like call a sleeping lock while + +0:29:28.299,0:29:32.090 +holding a blocking primitive + +0:29:33.390,0:29:37.530 +in the next example in the next I'm going to show you a way to + +0:29:37.530,0:29:41.140 +to handle for example the Mallock case + +0:29:41.140,0:29:42.520 +and similar + +0:29:42.520,0:29:45.000 +but the that usually + +0:29:46.830,0:29:47.620 +usually that + +0:29:47.620,0:29:49.980 +are not very common cases + +0:29:49.980,0:29:52.920 +at least for simple parts + +0:29:52.920,0:29:56.280 +you should even try to avoid the + +0:29:56.280,0:30:03.280 +the + +0:30:04.620,0:30:06.180 +yes + +0:30:06.180,0:30:07.050 +even in the + +0:30:07.050,0:30:09.120 +in the + +0:30:09.120,0:30:10.220 +wait channel + +0:30:10.220,0:30:14.530 +as in the FreeBSD you can differentiate between the condition variables and + +0:30:14.530,0:30:15.720 +Msleep + +0:30:15.720,0:30:17.510 + usually Msleep was + +0:30:17.510,0:30:22.210 +really Msleep was introduced as the first primitive + +0:30:22.210,0:30:26.190 +but it has an interface very very difficult to + +0:30:26.190,0:30:28.460 + to make saner and to understand + +0:30:28.460,0:30:30.470 +at least for + +0:30:30.470,0:30:31.220 +for people + +0:30:31.220,0:30:32.120 +which are + +0:30:32.120,0:30:34.960 +comfortable with + +0:30:34.960,0:30:39.260 +with interface of condition variable that we all saw but they are + +0:30:39.260,0:30:40.649 +newer primitive + +0:30:40.649,0:30:42.660 +mainly there is + +0:30:42.660,0:30:44.400 +so far the newer code + +0:30:44.400,0:30:46.960 +what you should do is just to + +0:30:46.960,0:30:49.000 +use condition variables + +0:30:49.000,0:30:50.659 +and not Msleep + +0:30:50.659,0:30:51.630 +basically + +0:30:51.630,0:30:56.220 +Msleep should be dropped off but they have avery nice feature which + +0:30:56.220,0:31:02.669 +is the the possibility to specify a wake up priority on the sleeping threads + +0:31:02.669,0:31:04.740 +once they are asleep + +0:31:04.740,0:31:07.470 +that condvar still doesnt + +0:31:07.470,0:31:12.430 +maybe if we could port these features to the condition variables we we will be able + +0:31:12.430,0:31:13.659 +to completely drop off Msleep + +0:31:13.659,0:31:18.529 +from the work arena + +0:31:18.529,0:31:20.450 +this is a + +0:31:20.450,0:31:25.580 +simple case that it's going to show a way to + +0:31:26.620,0:31:30.670 +a simple way to deal with the for example + +0:31:30.670,0:31:34.100 +condition I told before the Mallock willing to + +0:31:34.100,0:31:35.390 +to sleep + +0:31:35.390,0:31:38.260 +and the doing that while holding a lock + +0:31:38.260,0:31:45.070 +as you see we have some fake C as some members like flags + +0:31:45.070,0:31:47.659 +and an object called instructful + +0:31:47.659,0:31:49.940 +which needs to be allocated + +0:31:49.940,0:31:54.400 +and that they are protected by an internal lock + +0:31:54.400,0:31:58.810 +you imagine that for example the fake C create + +0:31:58.810,0:32:02.269 +holds lock of the object and does some things + +0:32:02.269,0:32:04.460 +which are not important + +0:32:04.460,0:32:07.650 +then in the end for example it's going to + +0:32:07.650,0:32:09.170 +to allocate + +0:32:09.170,0:32:14.110 +the FC object and that should be protected in + +0:32:14.110,0:32:16.470 +in anatomic part + +0:32:16.470,0:32:20.030 +something you can do is just to set the flag + +0:32:20.030,0:32:22.160 +for that + +0:32:22.160,0:32:22.730 +saying + +0:32:22.730,0:32:28.460 +the allocation is going to happen if you're adjust to this structure concurrently + +0:32:28.460,0:32:29.899 +just keep the allocation + +0:32:29.899,0:32:31.500 +and that's what we do + +0:32:31.500,0:32:32.919 +we check for this flag + +0:32:32.919,0:32:37.969 +and if its present it means that another thread is still + +0:32:37.969,0:32:40.149 +is already allocating and we just keep + +0:32:40.149,0:32:46.360 +so otherwise we set it and then we have locked the mutex + +0:32:46.360,0:32:49.100 +then we allocate the memory for the + +0:32:49.100,0:32:50.610 +for the object + +0:32:50.610,0:32:52.450 +acquire again the lock + +0:32:52.450,0:32:54.860 +and we simply have seen + +0:32:54.860,0:33:00.200 +please note that Ive used the temporary storage for that in order to make + +0:33:00.200,0:33:01.830 +some search on + +0:33:01.830,0:33:03.280 +like the MS + +0:33:03.280,0:33:04.180 +about the + +0:33:04.180,0:33:05.500 +the pointer + +0:33:05.500,0:33:10.700 +it was just a tricky note that you verify that really the structure was not + +0:33:10.700,0:33:14.330 +really allocated + +0:33:14.330,0:33:16.600 +and so that we can get some + +0:33:16.600,0:33:21.870 +kind of session about that + +0:33:22.640,0:33:26.340 +one of the biggest innovation that was brought to FreeBSD + +0:33:26.340,0:33:30.120 +about the locking primitive about the locking primitives + +0:33:30.120,0:33:33.770 +are the interrupts that + +0:33:34.640,0:33:36.850 +mainly + +0:33:36.850,0:33:40.820 +this is pretty simple to explain maybe + +0:33:40.820,0:33:44.070 +As the top half remains basically the same + +0:33:44.070,0:33:49.790 +and was going to handle the ISR for the interrupt line for example + +0:33:49.790,0:33:54.330 +the bottom half changed set and running the interrupts + +0:33:54.330,0:33:58.700 +handler is solid on that line as it was traditionally happened + +0:33:58.700,0:34:02.140 +it was going just to schedule a thread + +0:34:02.140,0:34:04.980 +that was going to run the + +0:34:04.980,0:34:06.940 +the interrupt handler in a + +0:34:06.940,0:34:12.389 +--- context and not the kind of --it was going to happen + +0:34:12.389,0:34:15.509 +traditionally in a lot of unique system + +0:34:16.699,0:34:23.179 +this has the big advantage that in using your own context you can + +0:34:23.179,0:34:24.429 +basically + +0:34:24.990,0:34:29.889 +you're not forced to use spin locks and you can do a lot of other fancy things + +0:34:29.889,0:34:32.209 +this necesity came over because + +0:34:32.209,0:34:33.149 +often + +0:34:33.149,0:34:38.529 +interrupts handlers needs to adjust to some + +0:34:38.529,0:34:42.589 +needs to adjust to some subsystem locks and the + +0:34:42.589,0:34:45.799 +as we were going to use blocking ---around + +0:34:45.799,0:34:50.379 +we had the necessity to support the + +0:34:50.379,0:34:52.589 +the locking of the + +0:34:52.589,0:34:57.119 +the possibilities of wide mutex actually + +0:34:57.559,0:35:01.759 +A similar thing was implemented using taskqueues + +0:35:01.759,0:35:02.879 +previously + +0:35:02.879,0:35:04.010 +and the sometimes it + +0:35:04.010,0:35:05.740 +I think I saw a lenux too + +0:35:05.740,0:35:08.439 +using taskqueues maybe + +0:35:08.439,0:35:10.029 +but the + +0:35:10.029,0:35:14.709 +it was basically something similar but not exactly in this way + +0:35:14.709,0:35:16.809 +a actually FreeBSD + +0:35:16.809,0:35:20.559 + from the release seven + +0:35:20.559,0:35:22.579 +the interrupt threads + +0:35:22.579,0:35:24.659 +are this model is a little bit changed + +0:35:24.659,0:35:26.499 +in order to include the + +0:35:26.499,0:35:29.739 +a new mechanism called the filtering + +0:35:29.739,0:35:36.249 +we have interrupt filters that basically if set then directly + +0:35:36.249,0:35:39.809 +directly + +0:35:39.809,0:35:40.879 +schedule the thread + +0:35:40.879,0:35:43.209 +linked to the parked line + +0:35:43.209,0:35:46.619 +they just check for + +0:35:46.619,0:35:50.939 +they just let run some new thing in the kernel or context + +0:35:50.939,0:35:52.449 +that will decide if + +0:35:52.449,0:35:56.709 + handle directly to requests or just schedule the kernel + +0:35:56.709,0:35:59.739 +it's like if you have the old bottom handler + +0:35:59.739,0:36:04.529 +that add the possibility to register a handler + +0:36:04.529,0:36:08.869 +still running in interrupt context and at the same time + +0:36:08.869,0:36:12.009 +decide if scheduled or not + +0:36:12.009,0:36:14.499 +so that it's no + +0:36:14.499,0:36:18.579 +no more madatory + +0:36:18.579,0:36:22.919 +So I think that the first part is going to finish so if you have some questions we can + +0:36:22.919,0:36:23.430 +handle + +0:36:23.430,0:36:28.699 + right now + +0:36:28.699,0:36:35.699 +this should be material for the second part actually + +0:36:45.279,0:36:48.529 +a new bus for example + +0:36:48.529,0:36:51.259 +some + +0:36:51.259,0:36:55.769 +some drivers that kind of a frequently used I'm not sure but which ones but all + +0:36:55.769,0:37:00.049 +the big ones are compared to finer locking + +0:37:00.049,0:37:03.109 +%um + +0:37:03.109,0:37:07.479 +actually the problem is not which parts are under Giant + +0:37:07.479,0:37:08.530 +well how we could + +0:37:08.530,0:37:12.380 +optimize the locking of some subsystems because + +0:37:12.380,0:37:15.079 +for example we have to virtual memory + +0:37:15.079,0:37:17.910 +which is not on the Giant but its + +0:37:17.910,0:37:19.719 +not locate + +0:37:19.719,0:37:24.400 +optimally and it's going to bring a lot of contention + +0:37:24.400,0:37:26.230 +so + +0:37:26.230,0:37:30.329 +it's not under Giant but it should be optimized + +0:37:30.329,0:37:37.329 + because the parts under Giant are very tiny.New bus for example + +0:37:37.599,0:37:44.599 +some parts relating to the VFS on the mounting +but yet a very short parts + +0:37:44.979,0:37:51.979 +I'm not sure about others + +0:37:57.479,0:37:59.170 +sorry + +0:38:02.069,0:38:08.549 +well usually it should be moved completely but + +0:38:08.549,0:38:11.019 +yes + +0:38:11.019,0:38:12.539 +it could + +0:38:32.909,0:38:34.809 +okay although + +0:38:34.809,0:38:38.289 +in the kernel we have a basically + +0:38:38.289,0:38:39.450 +%um + +0:38:39.450,0:38:43.019 +as you should know we already imported the trays for example + +0:38:43.019,0:38:47.839 +and I have wondered, I have submitted by developed + +0:38:47.839,0:38:48.669 +my country + +0:38:48.669,0:38:51.479 +called ---some patches that brings the + +0:38:51.479,0:38:54.689 +the ----- directly in our locking + +0:38:54.689,0:38:55.699 +in order to + +0:38:55.699,0:38:58.890 +allow it to be tracked with the trace. + +0:38:58.890,0:39:02.009 +which is very nice but it's still not completed + +0:39:02.009,0:39:03.310 +we are reviewing + +0:39:03.310,0:39:08.309 +above that we have a very the other useful tool called the lock profiling + +0:39:08.309,0:39:12.039 +that has been very helpful in the past in order to + +0:39:12.039,0:39:14.110 +find the most contended lock + +0:39:14.110,0:39:17.469 +and the to try to propose them to finer locking + +0:39:17.469,0:39:20.589 +so at least for the kernel we have such mechanism + +0:39:20.589,0:39:22.719 +I'm not sure what should + +0:39:22.719,0:39:26.640 +have been the user space.I'm sure we've not something similar + +0:39:26.640,0:39:28.310 +but maybe other systems + +0:39:28.310,0:39:29.469 +have + +0:39:29.469,0:39:30.749 +similar tools + +0:39:30.749,0:39:36.039 +I don't know I just know FreeBSD so + +0:39:58.479,0:39:59.220 +not sure + +0:39:59.220,0:39:59.919 +would you repeat + +0:39:59.919,0:40:03.879 + some voice please. No I cant hear + +0:40:03.879,0:40:05.509 +It seems to me that + +0:40:05.509,0:40:08.269 +you don't you have to do all the work that you do with locking + +0:40:08.269,0:40:11.469 +well if you're not on SMP right? + +0:40:11.469,0:40:13.029 +well no + +0:40:13.029,0:40:15.259 +it's not right because the + +0:40:15.259,0:40:20.210 +you have to protect even against some mechanism like preemption + +0:40:20.210,0:40:25.989 +which is going to be tricky.It is dfferent implemented than FreeBSD 4.x so + +0:40:25.989,0:40:28.909 +it's going to be with preemption its like + +0:40:28.909,0:40:30.099 +from + +0:40:30.099,0:40:34.479 +it's like if you have a real SMP system from our technical point of view + +0:40:34.479,0:40:35.809 +so you have to handle + +0:40:35.809,0:40:38.339 +problems typical of that + +0:40:38.339,0:40:43.249 +really in the kernel we have other kind of synchronization like atomics + +0:40:43.249,0:40:45.500 +I don't, I should have had + +0:40:45.500,0:40:50.609 +a slide about that but it disappeared so I can tell you by voice + +0:40:50.609,0:40:55.170 +its well like we have the possibility to use atomic instruction in the + +0:40:55.170,0:40:57.369 +in FreeBSD kernel directly + +0:40:57.369,0:40:59.249 +but the + +0:40:59.249,0:41:03.119 +to use even memory bytes linked with them + +0:41:03.119,0:41:08.869 +the only pitfall is that you cannot really trust about the + +0:41:08.869,0:41:10.469 +cash coherency + +0:41:10.469,0:41:14.339 +because as long as it's Im be specific you can just + +0:41:14.339,0:41:16.989 +you can just be trust about + +0:41:16.989,0:41:21.879 +what happens in your CPU where use the atomic and where to use the memory byte + +0:41:21.879,0:41:26.349 + you cannot make assumptions about the what happens about if other CPUs + +0:41:26.349,0:41:29.289 +can see your modifiers or not + +0:41:29.289,0:41:31.640 +and if the cache can handle that + +0:41:31.640,0:41:37.119 +we have a specific primitives in order to for example disable preemption + +0:41:37.119,0:41:39.379 +which are the critical sections + +0:41:39.379,0:41:42.179 +critical entry and critical exit + +0:41:42.179,0:41:45.309 +that what you call them you are not to + +0:41:45.309,0:41:48.219 +the preemption is simply allowed + +0:41:48.219,0:41:54.749 +it's that's a very fast primitive so there is not much overhead + +0:41:54.749,0:41:56.049 +so there's not much overhead + +0:41:56.049,0:42:00.679 +we also have a way to disable interrupt which is unofficial.I will tell + +0:42:00.679,0:42:03.079 +that + +0:42:03.079,0:42:07.720 +because you can do that in machine dependant way + +0:42:07.720,0:42:10.619 +with a spin lock entry and spin lock exit + +0:42:10.619,0:42:14.989 + and then + +0:42:14.989,0:42:16.049 +yeah that you can + +0:42:16.049,0:42:17.389 +even disable + +0:42:17.389,0:42:19.479 +some thread migration + +0:42:19.479,0:42:22.940 + using skid primitives + +0:42:22.940,0:42:25.319 +that are very useful + +0:42:25.319,0:42:29.779 +when you are going to adjust for example to per-CPU datas + +0:42:29.779,0:42:33.270 +and you have several chases and you don't want the CPU migrate + +0:42:33.270,0:42:34.200 +from that + +0:42:34.200,0:42:36.619 +thread migrate from that CPU + +0:42:36.619,0:42:38.729 +because you could read different + +0:42:38.729,0:42:45.369 +values from different CPU then + +0:42:45.369,0:42:46.479 +I'm not sure + +0:42:46.479,0:42:52.079 +if there is something else okay + +0:42:52.079,0:42:57.229 +questions? no? + +0:42:57.229,0:42:58.189 +so i'll see you later" diff --git a/en_US.ISO8859-1/captions/2009/dcbsdcon/davis-isolatingcluster.sbv b/en_US.ISO8859-1/captions/2009/dcbsdcon/davis-isolatingcluster.sbv new file mode 100644 index 0000000000..b05e61955b --- /dev/null +++ b/en_US.ISO8859-1/captions/2009/dcbsdcon/davis-isolatingcluster.sbv @@ -0,0 +1,3912 @@ +0:00:15.749,0:00:18.960 +I do apologize for the + +0:00:18.960,0:00:22.130 +for the Euro BSD Con slide. I've redone the + +0:00:22.130,0:00:23.890 +title page and redone the + +0:00:23.890,0:00:27.380 +and made some changes to the slides +and they didn't make it for approval + +0:00:27.380,0:00:33.130 +by this afternoon so + +0:00:33.130,0:00:34.640 + okay so + +0:00:34.640,0:00:36.390 +I'm gonna be talking about + +0:00:36.390,0:00:38.430 +doing + +0:00:38.430,0:00:42.889 +about isolating jobs for performance and predictability +in clusters + +0:00:42.889,0:00:43.970 +before I get into that + +0:00:43.970,0:00:46.010 +I want to talk a little bit about + +0:00:46.010,0:00:47.229 +who we are and + +0:00:47.229,0:00:49.520 +what our problem space is like because that + +0:00:49.520,0:00:54.760 +%uh a dictates that that %uh has an effect +are solutions base + +0:00:54.760,0:00:57.079 +I work for the aerospace corporation + +0:00:57.079,0:00:58.609 +%uh we we work + +0:00:58.609,0:01:02.480 + we operate a federally-funded +research and development center + +0:01:02.480,0:01:05.400 +%uh in the area national security space + +0:01:05.400,0:01:09.310 +and in particular we work with the air force +space and missile command + +0:01:09.310,0:01:13.090 +and with that the national reconnaissance +office + +0:01:13.090,0:01:16.670 + and our engineers support a wide variety + +0:01:16.670,0:01:20.550 +of activities within that area + +0:01:20.550,0:01:21.830 +so we have + +0:01:21.830,0:01:23.740 +a bit over fourteen hundred to correct + +0:01:23.740,0:01:25.860 +sorry twenty four hundred engineers + +0:01:25.860,0:01:28.820 + in virtually every discipline we have + +0:01:28.820,0:01:33.520 +as you would expect we have our rocket scientists + we have people who build satellites + +0:01:33.520,0:01:37.439 +we have people who build sensors that go on +satellites people who study this sort of things + +0:01:37.439,0:01:38.130 +that you + +0:01:38.130,0:01:39.590 +see when you + +0:01:39.590,0:01:40.819 +use those sensors + +0:01:40.819,0:01:42.040 +that sort of thing + +0:01:42.040,0:01:44.180 + we also have civil engineers and + +0:01:44.180,0:01:45.680 +electronic engineers + +0:01:45.680,0:01:46.649 +and process + +0:01:46.649,0:01:49.170 +computer process people + +0:01:49.170,0:01:53.120 +so we literally do everything related to space +and all sorts of things that you might not + +0:01:53.120,0:01:55.270 +expect to be related to space + +0:01:55.270,0:01:58.820 +because we also for instance help build ground +systems since satellites arent very useful if + +0:01:58.820,0:02:00.680 +there isn't anything to talk to them + +0:02:00.680,0:02:02.540 +%um + +0:02:02.540,0:02:04.090 +and these engineers + +0:02:04.090,0:02:07.420 +since they're solving all these different problems we have + +0:02:07.420,0:02:11.499 +engineering applications in you know +virtually every size you can think of + +0:02:11.499,0:02:15.539 +ranging from you know little spreadsheet things that +you might not think of as an engineering + +0:02:15.539,0:02:17.229 +application but they are + +0:02:17.229,0:02:22.249 +to Matlab programs or want to see code + +0:02:22.249,0:02:23.960 +or one of traditional parallel for us + +0:02:23.960,0:02:25.159 +serial code + +0:02:25.159,0:02:26.049 +and then + +0:02:26.049,0:02:30.949 +large parallel applications either in house +or genetic algorithms and that sort + +0:02:30.949,0:02:31.769 +of thing + +0:02:31.769,0:02:32.900 +or traditional + +0:02:32.900,0:02:34.749 +the classic parallel code + +0:02:34.749,0:02:37.599 +like you work around a crater or something material simulation + +0:02:37.599,0:02:40.119 +%uh or %uh + +0:02:40.119,0:02:41.459 +or that or food flow + +0:02:41.459,0:02:43.869 +or that sort of thing + +0:02:43.869,0:02:44.240 +so + +0:02:44.240,0:02:46.349 +so we have this big application space + +0:02:46.349,0:02:49.029 +just want to give a little introduction to that because +it + +0:02:49.029,0:02:51.529 +does come back and influence what we + +0:02:51.529,0:02:55.999 +the sort of solutions we work at + +0:02:55.999,0:03:00.499 +so the rest of the talk Im gonna talk about oops + +0:03:00.499,0:03:05.259 +we skipped a slide, There we are. Thats a little better + +0:03:05.259,0:03:08.940 +what I'm interested in is I do high +performance computing + +0:03:08.940,0:03:10.109 +at company + +0:03:10.109,0:03:13.949 +and I provide high performance computing resources +to our users + +0:03:13.949,0:03:19.949 +as part of my role in our technical +computing services organization + +0:03:19.949,0:03:20.370 +so + +0:03:20.370,0:03:23.120 +our primary resource at this point is + +0:03:23.120,0:03:25.429 + the fellowship cluster + +0:03:25.429,0:03:26.540 +it's a for the + +0:03:26.540,0:03:29.569 +named for the fellowship the ring + +0:03:29.569,0:03:30.449 +it's the + +0:03:30.449,0:03:32.520 +we're gonna wrap some nodes + +0:03:32.520,0:03:33.930 +wrap the core systems + +0:03:33.930,0:03:35.909 +%uh over here there's a + +0:03:35.909,0:03:39.659 +Cisco a large Cisco switch. Actually today +there are around two sixty five oh nines if + +0:03:39.659,0:03:40.899 +you assess them + +0:03:40.899,0:03:46.149 +and because we couldnt get the core density otherwise + +0:03:46.149,0:03:50.219 +and primarily the Gigabit Ethernet system runs +FreeBSD currently 6.0 because we havent upgraded + +0:03:50.219,0:03:51.089 +it yet + +0:03:51.089,0:03:55.639 +planning to move to probably to 7.1 +maybe slightly past 7.1 + +0:03:55.639,0:04:01.029 +%uh if we want to get the latest initial APM changes in + +0:04:01.029,0:04:05.900 +we use the Sun Grid Engine scheduler was one of +the two main options for open source + +0:04:05.900,0:04:08.949 +resource managers on cluster the other one being +that + +0:04:08.949,0:04:09.959 +the %uh + +0:04:09.959,0:04:11.499 +Torp + +0:04:11.499,0:04:15.939 +and now the combination from cluster resources + +0:04:15.939,0:04:17.389 +so we also have + +0:04:17.389,0:04:18.079 + that's actually + +0:04:18.079,0:04:22.090 +40 TB thats really the raw number on a sun thumper and + +0:04:22.090,0:04:23.219 +and + +0:04:23.219,0:04:26.290 +that thirty two usable once you start using ---- two + +0:04:26.290,0:04:30.939 +since you might actually like to have your data +should a disk fail + +0:04:30.939,0:04:32.969 +and with today's discs drade + +0:04:32.969,0:04:34.009 +grade five + +0:04:34.009,0:04:35.249 +doesn't really cut it + +0:04:35.249,0:04:37.379 +%um + +0:04:37.379,0:04:40.220 +we also have some other resources coming on but Im going to be + +0:04:40.220,0:04:43.530 +two smaller clusters unfortunately probably running Sun x and + +0:04:43.530,0:04:45.900 + some SMPs but + +0:04:45.900,0:04:49.990 +Im going to be concentrating here on the work we're +doing on our other + +0:04:49.990,0:04:54.259 +our FreeBSD based cluster + +0:04:54.259,0:04:55.060 +first of all + +0:04:55.060,0:04:59.410 +first of all I want to talk about why we want to +share resources. Should be fairly obvious + +0:04:59.410,0:05:00.610 +but I'll talk about it in a little bit + +0:05:00.610,0:05:04.900 +and then what goes wrong when you start sharing resources + +0:05:04.900,0:05:08.449 +after that I'll talk about some different solutions +to those problems + +0:05:08.449,0:05:09.759 +and + +0:05:09.759,0:05:13.399 +some fairly trivial experiments that we've done +so far in terms of the it's enghancing the schedule or + +0:05:13.399,0:05:15.860 +using operating system features + +0:05:15.860,0:05:17.730 +so you mitigate those problems + +0:05:17.730,0:05:19.349 +%um + +0:05:19.349,0:05:20.110 +and %uh + +0:05:20.110,0:05:25.110 +then conclude with some future work + +0:05:25.110,0:05:29.289 +obviously if you have a resource the size +of the size of our cluster fourteen hundred + +0:05:29.289,0:05:30.970 +cores roughly + +0:05:30.970,0:05:32.819 +you probably want to share it unless you + +0:05:32.819,0:05:35.080 +purpose built it for a single application + +0:05:35.080,0:05:37.340 + you're going to want to have your users + +0:05:37.340,0:05:39.440 +sharing it + +0:05:39.440,0:05:42.909 +and you don't want to just say you know you get on Monday + +0:05:42.909,0:05:45.330 +probably not going to be a very effective +option + +0:05:45.330,0:05:49.270 +especially not when we have as many uses we +do + +0:05:49.270,0:05:53.849 +we also can't just afford to buy another one +every time a user shows up + +0:05:53.849,0:05:54.959 +so one of our + +0:05:54.959,0:05:57.339 +senior VPs said a while back + +0:05:57.339,0:05:57.969 +you know + +0:05:57.969,0:06:02.349 +we could probably afford to buy just about +anything we could need once + +0:06:02.349,0:06:03.800 + we can't just + +0:06:03.800,0:06:06.359 +buy ten of them though + +0:06:06.359,0:06:08.939 +if you really really needed it + +0:06:08.939,0:06:09.680 +dropping + +0:06:09.680,0:06:11.460 +small numbers of millions of dollars on + +0:06:11.460,0:06:13.349 +computing resources wouldnt be + +0:06:13.349,0:06:15.039 +impossible + +0:06:15.039,0:06:20.829 +but we can't go to you know just have every engineer +who wants one just call Dell and say ship me ten racks + +0:06:20.829,0:06:24.030 +it's not going to work + +0:06:24.030,0:06:25.580 +and the other thing is that we cant + +0:06:25.580,0:06:28.360 +we need to also provide quick turnaround + +0:06:28.360,0:06:29.390 +for some users + +0:06:29.390,0:06:33.229 +so we can't have one user hogging the system and +hogging it until they are done + +0:06:33.229,0:06:34.720 +because we have some users + +0:06:34.720,0:06:37.099 +n then the next one can run + +0:06:37.099,0:06:40.949 +because we have some users who'll +come in and say well I need to run + +0:06:40.949,0:06:43.159 +for three months + +0:06:43.159,0:06:43.690 +and + +0:06:43.690,0:06:46.810 +we've had users come in and literally run + +0:06:46.810,0:06:49.740 +pretty much using the entire system for three months + +0:06:49.740,0:06:53.839 +well so we've had to provide some ability for other +users to still get their work done + +0:06:53.839,0:06:58.300 +so we can't just.. so we do have to have some share + +0:06:58.300,0:07:00.619 +however when you start to share any resource + +0:07:00.619,0:07:01.610 +like this + +0:07:01.610,0:07:03.509 +you start getting contention + +0:07:03.509,0:07:06.300 +users need the same thing at the same time + +0:07:06.300,0:07:09.700 +and so they fight back and forth for it and they +can't get what they want + +0:07:09.700,0:07:11.639 +so you have to balance them a bit + +0:07:11.639,0:07:12.999 +%um + +0:07:12.999,0:07:14.529 +you know also + +0:07:14.529,0:07:17.869 +%uh some jobs lie when they + +0:07:17.869,0:07:20.870 +request resources and they actually need +more than they ask for + +0:07:20.870,0:07:23.279 +which can cause problems + +0:07:23.279,0:07:27.229 +so we schedule them. We say you're going to fit +here fine and they run off and use + +0:07:27.229,0:07:28.580 +more than they said + +0:07:28.580,0:07:31.000 +and if we don't have a mechanism to constrain +them + +0:07:31.000,0:07:32.389 +we have problems + +0:07:32.389,0:07:34.270 +%uh likewise + +0:07:34.270,0:07:37.109 +once these users start to contend + +0:07:37.109,0:07:39.029 +that doesn't just result in + +0:07:39.029,0:07:40.439 +the jobs taking + +0:07:40.439,0:07:43.360 +taking longer in terms of wall clock time + +0:07:43.360,0:07:44.659 +because they are extremely slow + +0:07:44.659,0:07:48.430 +but there's overhead related to that contention +because they get swapped out due to to that pressure on + +0:07:48.430,0:07:49.219 +on + +0:07:49.219,0:07:51.509 +on various systems + +0:07:51.509,0:07:52.550 +if you really + +0:07:52.550,0:07:57.039 +for instance run put of memory then you go into +swap and you end up wasting all your cycles + +0:07:57.039,0:07:58.710 +pulling junk in and out of disc + +0:07:58.710,0:08:00.830 +wasting your bandwidth on that + +0:08:00.830,0:08:03.530 +so there are + +0:08:03.530,0:08:04.219 +resource + +0:08:04.219,0:08:08.139 +there are resource cost to the contention not merely + +0:08:08.139,0:08:11.979 +a delay in returning results + +0:08:11.979,0:08:16.590 +so now I'm going to switch gears and start talk so I'm +going to talk a little bit about different + +0:08:16.590,0:08:18.270 +solutions to these + + +0:08:18.270,0:08:20.610 +to the + +0:08:20.610,0:08:22.339 +these contention issues + +0:08:22.339,0:08:23.710 +and %uh + +0:08:23.710,0:08:27.840 +%uh and and look at different ways of solving the +problem.most of these are things that have + +0:08:27.840,0:08:29.440 +already been done + +0:08:29.440,0:08:30.620 +but I just want to talk about + +0:08:30.620,0:08:32.990 +the different ways and then + +0:08:32.990,0:08:35.710 +evaluate them in our context + +0:08:35.710,0:08:38.119 +so a classic solution to the problem is + +0:08:38.119,0:08:39.280 +Gang scheduling + +0:08:39.280,0:08:44.139 + it's basically conventional Unex process +context switching + +0:08:44.139,0:08:46.560 +written really big + +0:08:46.560,0:08:50.339 +you what you do is you have your parallel +job thats running + +0:08:50.339,0:08:51.390 +on a system + +0:08:51.390,0:08:52.839 +and it runs for a while + +0:08:52.839,0:08:57.920 +and then after a certain amount of time you basically +shove it all you kick it off of all the nodes + +0:08:57.920,0:08:59.940 +and let the next one come in + +0:08:59.940,0:09:04.030 +and typically when people do this they do it on +on the order of hours because the context switch + +0:09:04.030,0:09:09.270 +time is extremely large is extremely high + +0:09:09.270,0:09:10.639 +for example + +0:09:10.639,0:09:14.530 +because it's not just like swapping a process +internet. You suddenly have to co-ordinate + +0:09:14.530,0:09:17.470 +the this context which across to all your processes + +0:09:17.470,0:09:19.280 +if you're running say + +0:09:19.280,0:09:21.190 +MPI over TCP + +0:09:21.190,0:09:25.910 +you actually need to tear down the TCP sessions +because you can't just have TCP timers sitting + +0:09:25.910,0:09:26.570 +around + +0:09:26.570,0:09:28.260 +or that sort of thing so + +0:09:28.260,0:09:29.950 +there there's a there's a lot of overhead + +0:09:29.950,0:09:34.340 +associated with this.You take a long context switch + +0:09:34.340,0:09:36.820 +if all of your infrastructure supports this + +0:09:36.820,0:09:39.420 + it's fairly effective + +0:09:39.420,0:09:43.300 +and it does allow jobs to avoid interfering +with each other which is nice + +0:09:43.300,0:09:46.100 +so you can't you don't have issues + +0:09:46.100,0:09:47.689 +because you're typically allocating + +0:09:47.689,0:09:50.950 +whole swaps of the system + +0:09:50.950,0:09:53.390 +and for properly written applications + +0:09:55.000,0:09:59.690 +partial results can be returned which for some of +our users is really important where you're doing a + +0:09:59.690,0:10:00.490 +refinement + +0:10:00.490,0:10:04.350 +users would want to look at the results and +say okay + +0:10:04.350,0:10:06.130 +you know is this just going off into the weeds + +0:10:06.130,0:10:10.860 +or does it look like it's actually converging on +some sort of useful solution + +0:10:10.860,0:10:13.980 +as they don't want to just wait till the end + +0:10:13.980,0:10:19.270 +down side of course is that this context +switches costs are very high + +0:10:19.270,0:10:22.460 +and most importantly there's really a lack +of useful implementations + +0:10:22.460,0:10:25.340 +a number of platforms have implemented this in the past + +0:10:25.340,0:10:29.840 +but in practice on modern clusters which are +built on commodity hardware + +0:10:29.840,0:10:32.340 +with you know + +0:10:32.340,0:10:35.530 +communication libraries written on standard protocols + +0:10:35.530,0:10:37.050 +the tools just arent there + +0:10:37.050,0:10:39.100 +and so + +0:10:39.100,0:10:40.860 +it's not very practical + +0:10:40.860,0:10:44.010 +also it doesn't really make a lot of sense with small jobs + +0:10:44.010,0:10:47.789 +and one of the things that we found is we have users who have + +0:10:47.789,0:10:50.860 +embarrassingly parallel problems for they need to look at + +0:10:50.860,0:10:53.450 +you know twenty thousand studies + +0:10:53.450,0:10:57.400 +and they could write something that looked more like a +conventional parallel application where they + +0:10:57.400,0:11:01.930 +you know wrote a schedule and set up an MPI a Message Pasting Interface + +0:11:01.930,0:11:05.400 +and handed out tasks to pieces of their job and then you +could do this + +0:11:05.400,0:11:09.280 +but then they would be running a schedule and they would +probably do a bad job of it turns out it's actually + +0:11:09.280,0:11:10.820 +fairly difficult to do right + +0:11:10.820,0:11:13.740 +even a trivial case + +0:11:13.740,0:11:16.189 +and so what they do instead is they just select twenty + +0:11:16.189,0:11:18.730 +twenty thousand jobs to great and say okay + +0:11:18.730,0:11:21.330 +whatever I'll deal with it + +0:11:21.330,0:11:23.140 +earlier versions that might have been a problem + +0:11:23.140,0:11:24.730 +current versions of the code + +0:11:24.730,0:11:27.060 + handle easily a million jobs that + +0:11:27.060,0:11:29.370 +so not really a big deal + +0:11:29.370,0:11:31.610 +but those sort of users wouldn't fit well + +0:11:31.610,0:11:34.190 +into the gang scheduled environment + +0:11:34.190,0:11:35.690 +at least not in a + +0:11:35.690,0:11:39.149 +conventional gang scheduled environment where +you do gang scheduling on the regularity of + +0:11:39.149,0:11:40.940 +jobs + +0:11:40.940,0:11:44.140 +so from that perspective it wouldnt work very well + +0:11:44.140,0:11:48.380 +if you have all the pieces in place and you are +doing a big parallel applications it in fact + +0:11:48.380,0:11:53.770 +an extremely effective approach + +0:11:53.770,0:11:56.290 +another option which is sort of related + +0:11:56.290,0:11:57.420 +it's in fact + +0:11:57.420,0:12:00.079 +take taking an even course with regularity + +0:12:00.079,0:12:04.360 +is single application or single project +clusters or sub-clusters + +0:12:04.360,0:12:07.590 +%uh for instance this is used some national labs + +0:12:07.590,0:12:11.910 +where you're given a cycle allocation for a +year based on your grant proposals + +0:12:11.910,0:12:14.779 +and what your cycle allocation actually comes to you as is + +0:12:14.779,0:12:16.580 +here's your cluster + +0:12:16.580,0:12:17.489 +here's a fun-ed + +0:12:17.489,0:12:19.840 +here's this chunk of notes. they're yours. Go to it + +0:12:19.840,0:12:21.930 +Install your own OS. Whatever you want + +0:12:21.930,0:12:25.580 +it's yours + +0:12:25.580,0:12:30.310 +and then and at a sort of finer scale there's things such as + +0:12:30.310,0:12:31.800 +you could use Emulab + +0:12:31.800,0:12:36.300 +which is the network emulation system but also does a less install and configuration + +0:12:36.300,0:12:39.300 +so you could do dynamic allocation that way + +0:12:39.300,0:12:40.540 +Sun's + +0:12:40.540,0:12:44.040 +Project Hedeby now actually I think it's +called service domain manager + +0:12:44.040,0:12:46.500 + is the product size version + +0:12:46.500,0:12:50.010 +or some Clusters on Demand + +0:12:50.010,0:12:54.450 +they were actually talking about web hosting clusters but + +0:12:54.450,0:12:57.780 +things that allow rapid deployment unless you +do that a little + +0:12:57.780,0:12:59.510 +little + +0:12:59.510,0:13:02.810 +a more granular level than the + +0:13:02.810,0:13:05.580 +the allocate them once a year approach + +0:13:05.580,0:13:07.720 +none the less + +0:13:07.720,0:13:11.220 +lets you give people whole clusters to work with + +0:13:11.220,0:13:12.920 +nice one nice thing about it is + +0:13:12.920,0:13:15.450 +the at the isolation between the processes + +0:13:15.450,0:13:16.890 +is complete + + +0:13:16.890,0:13:20.800 +so you dont have to worry about users stomping on each other. +Its their own system. they can trash it all they + +0:13:20.800,0:13:22.230 +want + +0:13:22.230,0:13:24.709 +if they flood the network or they + +0:13:24.709,0:13:26.180 +run the nodes into swap + +0:13:26.180,0:13:28.480 +well that's their problem + +0:13:28.480,0:13:32.120 +but it also has the advantage that you can tailor the the images + +0:13:32.120,0:13:36.980 +on the nodes of the operative systems to +meet the exact needs of the application + +0:13:36.980,0:13:40.560 +down side of course is its course +theres a system whichgranularity. in our environment that doesn't work + +0:13:40.560,0:13:41.500 +very well + +0:13:41.500,0:13:46.800 + since we do have all of these all these different types of jobs + +0:13:46.800,0:13:51.710 +context switches are also pretty expensive. certainly on the order of minutes + +0:13:51.710,0:13:54.690 +Emulab typically claim something like ten minutes + +0:13:54.690,0:13:57.970 +there are some systems out there + +0:13:57.970,0:14:03.320 +for instance if you use I think its open boot that +they're calling it today. It used to be 1xBIOS + +0:14:03.320,0:14:06.790 +where you can actually deploy a system in + +0:14:06.790,0:14:08.700 +tens of seconds + +0:14:08.700,0:14:11.520 +mostly by getting rid of all that junk the BIOS writers wrote + +0:14:11.520,0:14:12.890 +and + +0:14:12.890,0:14:17.770 +the OS speed is pretty fast if you dont have all +that stuff to waylay you not + +0:14:17.770,0:14:19.940 +but in practice on sort of + +0:14:19.940,0:14:21.660 +on the shelf + +0:14:21.660,0:14:24.400 +the context switches time quite high + +0:14:24.400,0:14:26.930 +users of course can interfere with themselves + +0:14:26.930,0:14:29.200 +you can argue it's not a problem but + +0:14:29.200,0:14:31.660 +ideally you would like to prevent +that + +0:14:31.660,0:14:35.350 +one of the things that I have to deal with +is that my users are + +0:14:35.350,0:14:37.830 +almost universally + +0:14:37.830,0:14:40.410 +not trained as computer scientists are programmes + +0:14:40.410,0:14:42.550 +you know there's there's really no domain +area + +0:14:42.550,0:14:44.780 +they're really good in that area + +0:14:44.780,0:14:48.389 +their concepts of the way hardware works in the +way software works + +0:14:48.389,0:14:55.389 +dont match reality in many cases + +0:15:01.269,0:15:02.830 +its pretty rare in practice + +0:15:02.830,0:15:06.700 +well I've heard one one lab that does it significantly + +0:15:06.700,0:15:09.839 +but it's like they do it on sort of a yearly +allocation basis + +0:15:09.839,0:15:12.790 +and throw the hardware away after two or three years + +0:15:12.790,0:15:15.999 +and you do typically have some sort of the deployment + +0:15:15.999,0:15:18.340 +system in place + +0:15:18.340,0:15:20.680 +or in most types of cases actually + +0:15:20.680,0:15:22.359 +usually your application comes with + +0:15:22.359,0:15:26.500 +and here's what we're going to spend on this many people + +0:15:26.500,0:15:27.730 +on this project so this is + +0:15:27.730,0:15:34.730 +big resource allocation + +0:15:36.000,0:15:39.780 +and %uh yet and I guess one other issue with this is there's no real easy + +0:15:39.780,0:15:43.320 +way to capture on underutilized resources +for example + +0:15:43.320,0:15:44.389 +if you have + +0:15:44.389,0:15:49.190 +an application which you know say single-threaded +and uses a ton of memory + +0:15:49.190,0:15:51.210 +on and is running on a machine + +0:15:51.210,0:15:55.040 +the machines we're buying these days are eight core so + +0:15:55.040,0:16:00.040 +thats wasting a lot of CPU cycles you're just +generating a lot of heat doing nothing + +0:16:00.040,0:16:03.890 +so ideally you would like a scheduler that +said okay so you're using + +0:16:03.890,0:16:08.040 +using eight or seven of the eight Gigabytes of +RAM but we've got these jobs + +0:16:08.040,0:16:10.080 +sitting here that + +0:16:10.080,0:16:11.560 +need next to know need + +0:16:11.560,0:16:15.910 +a hundred megabytes so we swap seven of +those in along with the big job + +0:16:15.910,0:16:18.580 +and backfill and in this + +0:16:18.580,0:16:19.600 +mechanism there's no + +0:16:19.600,0:16:21.810 +there's no good way to do that + +0:16:21.810,0:16:26.820 +obviously if the users have that application +next they can do it themselves + +0:16:26.820,0:16:30.510 +it's not something where we can be easily +bring in + +0:16:30.510,0:16:35.090 +bring in more jobs and have a mix to +take advantage of the different + +0:16:35.090,0:16:37.300 +resources + +0:16:37.300,0:16:39.940 +a related approach is to + +0:16:39.940,0:16:43.950 +to install virtualization software on the +equipment and this is this is + +0:16:43.950,0:16:44.980 +a + +0:16:44.980,0:16:46.379 +this is the essence of + +0:16:46.379,0:16:49.800 +what Cloud computing is at the moment + +0:16:49.800,0:16:53.520 +it's Amazon providing Zen + +0:16:53.520,0:16:55.129 +Zen hosting for + +0:16:55.129,0:16:56.769 +relatively arbitrary yet + +0:16:56.769,0:16:59.710 +OS images + +0:16:59.710,0:17:02.720 +it does have advantage that allows rapid deployment + +0:17:02.720,0:17:06.510 +in theory if your application is scaleable provides for + +0:17:06.510,0:17:08.259 +extremely high scaleability + +0:17:08.259,0:17:10.110 +particularly if you + +0:17:10.110,0:17:14.470 +arent us and therefore can possibly somebody else's hardware + +0:17:14.470,0:17:16.520 +in in our application's face thats + +0:17:16.520,0:17:18.790 +not very practical so + +0:17:18.790,0:17:20.360 +we can't do that + +0:17:20.360,0:17:20.870 +and + +0:17:20.870,0:17:23.790 +it also has the advantage that you can run + +0:17:23.790,0:17:26.470 +you can have people with their own image in there + +0:17:26.470,0:17:30.000 +which is tightly resource constrained but you +can run more than one of them on it. but no for instance + +0:17:30.000,0:17:31.170 +you can give + +0:17:31.170,0:17:32.730 +one job + +0:17:32.730,0:17:35.489 +four cores and another job two cores another + +0:17:35.489,0:17:37.500 +you know and have a couple single core + +0:17:37.500,0:17:38.860 +jobs in theory + +0:17:38.860,0:17:43.340 +you can get fairly strong isolation there +obviously there are shared resources underneath + +0:17:43.340,0:17:44.710 +and you + +0:17:44.710,0:17:45.570 +probably can't + +0:17:45.570,0:17:48.370 +afford to completely isolate say network bandwidth + +0:17:48.370,0:17:49.520 +at the bottom layer + +0:17:49.520,0:17:51.580 +you can be some but + +0:17:51.580,0:17:56.170 +if you go overboard you can spend all your time on accounting + +0:17:56.170,0:17:58.830 +you also can again + +0:17:58.830,0:18:01.410 +tailor the images to the job + +0:18:01.410,0:18:05.030 +and in this environment actually you can +do that even more strongly than that + +0:18:05.030,0:18:07.030 +the sub-cluster approach + +0:18:07.030,0:18:09.860 +in that you can often do run + +0:18:09.860,0:18:16.360 +a five-year-old operating system or ten-year-old +operating system if you're using full virtualization + +0:18:16.360,0:18:19.030 +and that can allow + +0:18:19.030,0:18:23.820 +allow obsolete core with your baseline core which is +important in our space because + +0:18:23.820,0:18:27.390 +the average program runs ten years or more + +0:18:27.390,0:18:30.860 +our average project runs ten years or more + +0:18:30.860,0:18:32.530 +and as a result + +0:18:32.530,0:18:36.010 +you might have to go rerun this program that was written + +0:18:36.010,0:18:37.320 +way back on + +0:18:37.320,0:18:40.550 +some ancient version of windows or whatever + +0:18:40.550,0:18:41.890 +it also does provide + +0:18:41.890,0:18:43.840 +the ability to recover resources + +0:18:43.840,0:18:45.290 +as I was talking about before + +0:18:45.290,0:18:49.530 +%uh but you can't do easily with sub-clusters because you cant just slip + +0:18:49.530,0:18:50.360 +another image + +0:18:50.360,0:18:52.910 +on the on there and say are you can use anything and + +0:18:52.910,0:18:56.730 +you know get that image ideal priority essentially + +0:18:56.730,0:19:00.480 +down side of course is that it is in complete +isolation and that there is a shared + +0:19:00.480,0:19:02.340 +hardware + +0:19:02.340,0:19:06.490 +you're not likely to find I don't think +any the virtualization systems out there + +0:19:06.490,0:19:08.890 +right now + +0:19:08.890,0:19:09.890 +virtualize + +0:19:09.890,0:19:11.470 +your segment of + +0:19:11.470,0:19:13.540 +memory bandwidth + +0:19:13.540,0:19:15.159 +or your segment + +0:19:15.159,0:19:16.390 +of cache + +0:19:16.390,0:19:18.390 +of cache base + +0:19:18.390,0:19:24.809 +so users can in fact interfere with themselves and each other in this +environment + +0:19:24.809,0:19:25.589 +it's also + +0:19:25.589,0:19:30.479 +%uh not really efficient for small jobs from the cost of running an + entire arrest for every + +0:19:30.479,0:19:33.020 +job is fairly high + +0:19:33.020,0:19:34.020 +even with + +0:19:34.020,0:19:34.710 +relatively light + +0:19:34.710,0:19:38.250 +%uh you know it's like OS is you're still looking + +0:19:38.250,0:19:40.900 +couple hundred megabytes in practice + +0:19:40.900,0:19:46.240 +once you get everything up and running unless you get totally stripped +down + +0:19:46.240,0:19:47.230 +and %uh + +0:19:47.230,0:19:49.460 +theres significant overhead + +0:19:49.460,0:19:52.240 +theres CPU slowdown typically in the + +0:19:52.240,0:19:55.360 +you know typical estimates are in the twenty +percent range + +0:19:55.360,0:20:00.450 +numbers really range from fifty percent to +five percent depending on what exactly you're doing + +0:20:00.450,0:20:02.100 +possibly even lower + +0:20:02.100,0:20:04.830 +or higher + +0:20:04.830,0:20:05.870 +and and just + +0:20:05.870,0:20:09.920 +you know the overhead because you have the whole OS there's a lot of a lot + +0:20:09.920,0:20:11.420 +of duplicate + +0:20:11.420,0:20:13.320 +stuff + +0:20:13.320,0:20:15.010 +the various vendors + +0:20:15.010,0:20:17.090 +have their answers they claim you know we can + +0:20:17.090,0:20:21.430 +we can merge that and say oh you're running the same kernel so we'll keep your memory + +0:20:21.430,0:20:24.120 +we use the same memory but + +0:20:24.120,0:20:25.220 +at some level + +0:20:25.220,0:20:29.309 +it's all going to get duplicated + +0:20:29.309,0:20:30.590 +a related option + +0:20:30.590,0:20:34.820 +comes from sort of the the internet havesting +industry which is to use virtual private + +0:20:34.820,0:20:38.130 +which is the technology from virtual private servers + +0:20:38.130,0:20:42.110 +the example that everyone here is probably familiar with is jails where + +0:20:42.110,0:20:44.130 +you can provide + +0:20:44.130,0:20:46.720 +your own file system root + +0:20:46.720,0:20:49.060 +your network interface + +0:20:49.060,0:20:50.620 +and what not + +0:20:50.620,0:20:51.500 +and + +0:20:51.500,0:20:53.129 +the nice thing about this is + +0:20:53.129,0:20:56.210 +that unlike full virtualization + +0:20:56.210,0:20:58.680 +the overhead is very small + +0:20:58.680,0:21:01.030 +basically costs you + + +0:21:01.030,0:21:02.820 +an entry in your process table + +0:21:02.820,0:21:05.570 +or an entry in few structures + +0:21:05.570,0:21:08.760 +there's some extra tests in their kernel but otherwise + +0:21:10.220,0:21:14.900 +there's there's not a huge overhead for virtualization you don't need +an extra kernel for every + +0:21:14.900,0:21:15.460 +image + +0:21:15.460,0:21:18.390 +so you get you get the difference here +between + +0:21:18.390,0:21:21.620 +be able to run maybe + +0:21:21.620,0:21:25.250 +you might be able to squeeze two hundred VMR images onto a machine + +0:21:25.250,0:21:29.620 +VMR people say no no don't do that but we have machines that are running + +0:21:29.620,0:21:30.509 +nearly that many + +0:21:33.720,0:21:34.790 +they're what + +0:21:34.790,0:21:38.289 +on the other hand there are people out there on thousands of + +0:21:38.289,0:21:40.730 +virtual hosts + +0:21:40.730,0:21:43.170 +using this technique at a single machine so + +0:21:43.170,0:21:45.200 +big difference in resource use + +0:21:45.200,0:21:46.400 +on especially with light + +0:21:46.400,0:21:48.070 +in the lightly loaded use + +0:21:48.070,0:21:52.400 +in our environment we're looking more running a very small number of them but still + +0:21:52.400,0:21:55.880 +that overhead is significant + +0:21:55.880,0:21:59.440 +you still do have some ability to tailor the + +0:21:59.440,0:22:01.670 +images to jobs needs + +0:22:01.670,0:22:03.309 +you could have a + +0:22:03.309,0:22:05.400 +custom root that for instance you could be running + +0:22:05.400,0:22:07.380 +FreeBSD x6 in one + +0:22:07.380,0:22:08.650 +in one + +0:22:08.650,0:22:11.040 +virtual server and seven in another + +0:22:11.040,0:22:15.090 +you have to be running of course seven kernel or eight kernel to make +that work + +0:22:15.090,0:22:16.330 +but it allows you to do that + +0:22:16.330,0:22:18.500 +we also in principle can do + +0:22:18.500,0:22:23.080 +evil things like our sixty four-bit kernel and then thirty two bit +user spaces because + +0:22:23.080,0:22:26.400 +say you have applications that you can't find the source to do anymore + +0:22:26.400,0:22:31.830 +or wide worst wide worries you don't +have the source to any more + +0:22:31.830,0:22:32.990 +an answer + +0:22:32.990,0:22:34.150 +interesting things there + +0:22:34.150,0:22:36.680 +and the other nice thing is since you're + +0:22:36.680,0:22:39.629 +you're doing a very lightweight and incomplete +virtualization + +0:22:39.629,0:22:43.269 +you don't have to virtualize things you don't +care about so you dont have the overhead of + +0:22:43.269,0:22:45.520 +virtualizing everything + +0:22:45.520,0:22:48.070 +downsides of course are incomplete isolation + +0:22:48.070,0:22:50.690 +you are running processes that on the same kernel + +0:22:50.690,0:22:52.770 +and they can interfere with each other + +0:22:52.770,0:22:55.320 +and there's dubious flexibility obviously + +0:22:55.320,0:22:57.900 +I don't think anybody + +0:22:57.900,0:23:01.850 +should have the ability to run Windows in a jail + +0:23:01.850,0:23:02.860 +theres some + +0:23:02.860,0:23:04.960 +Net BSD peak of support but + +0:23:04.960,0:23:10.510 +and I dont think it's really gotten to that point + +0:23:10.510,0:23:12.420 +one one final area + +0:23:12.420,0:23:14.350 +that sort of diverges from this + +0:23:14.350,0:23:16.159 +is the classic + +0:23:16.159,0:23:18.400 +Unix solution to the problem + +0:23:18.400,0:23:20.580 +on this on single + +0:23:20.580,0:23:22.070 +in a single machines + +0:23:22.070,0:23:22.800 +which is + +0:23:22.800,0:23:28.950 +to use existing resource limits and resource partitioning techniques + +0:23:28.950,0:23:33.430 +you know for example all Unix like our Unix systems have to process +resource limits + +0:23:33.430,0:23:36.240 +a resource and typically + +0:23:36.240,0:23:36.999 +schedule a + + +0:23:38.340,0:23:41.510 +cluster schedulers support the common ones + +0:23:41.510,0:23:43.150 +so you can set a + +0:23:43.150,0:23:47.230 +memory limit on your process or a CPU time limit on your process + +0:23:47.230,0:23:49.830 +and the schedulers typically provide + +0:23:49.830,0:23:51.350 +at least + +0:23:51.350,0:23:54.740 +lot of support for + +0:23:54.740,0:23:56.850 +the limits on + +0:23:56.850,0:24:01.900 +a given set of process. thats part of the job + +0:24:01.900,0:24:02.850 +also the most + +0:24:02.850,0:24:05.640 +you know there are a number of forms of research +partitioning that + +0:24:05.640,0:24:07.170 +are available as + +0:24:07.170,0:24:08.100 +the in that + +0:24:08.100,0:24:09.700 +as the standard feature + +0:24:09.700,0:24:12.000 +on so memory discs are one of them so + +0:24:12.000,0:24:16.800 +if you want to create a file system space it's +limited in size. Create a memory disc + +0:24:16.800,0:24:17.969 +and back it + +0:24:17.969,0:24:21.130 +and back it with a --- file + +0:24:21.130,0:24:22.520 +Quotas another mechanism + +0:24:22.520,0:24:24.570 +of partitioning that + +0:24:24.570,0:24:26.330 +disc use + +0:24:26.330,0:24:30.330 +and then there are techniques like CPU affinities that you can walk +processes to it + +0:24:30.330,0:24:32.010 +a single process + +0:24:32.010,0:24:34.540 +processor a set of processors + +0:24:34.540,0:24:39.310 +and so they can't interfere with each other +with processes running on other processes + + +0:24:39.310,0:24:44.280 +the nice thing about this first is that you're using existing +facilities so you dont have to rewrite + +0:24:44.280,0:24:46.170 +also new features + +0:24:46.170,0:24:49.590 +for each application + +0:24:49.590,0:24:52.790 +and they tend to integrate well with existing schedulers +in many cases + +0:24:52.790,0:24:55.940 +parts of them are already implemented + +0:24:55.940,0:24:59.650 +and in fact the experiments that we'll talk about this later are using +this type of + +0:24:59.650,0:25:02.160 +technique + +0:25:02.160,0:25:02.830 +cons are of course + +0:25:02.830,0:25:04.850 +incomplete isolation again + +0:25:04.850,0:25:08.270 +and theres typically no unified framework + +0:25:08.270,0:25:12.310 +for the concept of a job when it comes to the center process + +0:25:12.310,0:25:16.710 +yeah there there are a number of data structures within the kernel for +instance the session + +0:25:16.710,0:25:18.120 +which + +0:25:18.120,0:25:19.499 +certain aggregate processes + +0:25:19.499,0:25:20.990 +but there isnt one + +0:25:20.990,0:25:22.230 +in + +0:25:22.230,0:25:24.800 +in BVSD at this point + +0:25:24.800,0:25:29.020 +allows you to place resource limits on this in a way that you can process + +0:25:29.020,0:25:32.520 +--- did have support like that + +0:25:32.520,0:25:34.160 +where they have a job ID + +0:25:34.160,0:25:36.210 +and there could be a job limit + +0:25:36.210,0:25:38.280 + and slurs projects + +0:25:38.280,0:25:41.320 +pursue similar not not quite the same + +0:25:41.320,0:25:43.149 +processes or part of a project but + +0:25:43.149,0:25:46.770 +it's not quite the same inherited relationship + +0:25:47.720,0:25:49.500 +and typically + +0:25:49.500,0:25:50.900 +there arent + +0:25:50.900,0:25:55.390 +limits on things like badwidth. there was + +0:25:55.390,0:25:56.430 +a sort of a + +0:25:56.430,0:25:58.350 +bandwidth limiting + +0:25:58.350,0:26:00.630 +nice type interface + +0:26:00.630,0:26:01.950 +on that I saw + +0:26:01.950,0:26:03.720 +first thing as a research project + +0:26:03.720,0:26:07.150 +many years ago I think it was that stage + +0:26:07.150,0:26:09.880 +where you could say this process can have + +0:26:09.880,0:26:11.580 +you know five megabits + +0:26:11.580,0:26:12.530 +or or whatever + +0:26:12.530,0:26:14.380 +but I haven't really seen anything take off + +0:26:14.380,0:26:16.940 +that would be a pretty neat thing to have + +0:26:16.940,0:26:19.309 +but one other exception there + +0:26:19.309,0:26:22.230 +is on Irex again + +0:26:22.230,0:26:28.210 +the XFS file system supported guaranteed data rates on file handles +you could say + +0:26:28.210,0:26:30.140 +you know if you would say I need + +0:26:30.140,0:26:32.940 +ten megabits read or ten megabits write + +0:26:32.940,0:26:34.029 +whatever you say + +0:26:34.029,0:26:35.529 +okay and go + +0:26:35.529,0:26:39.279 +and and then you could read and write and +it would do evil things file system + +0:26:39.279,0:26:40.600 +in some cases + +0:26:40.600,0:26:43.940 +all to making sure that you could get that terrific data rate + +0:26:43.940,0:26:44.900 +by + +0:26:44.900,0:26:49.710 +by keeping the file + + +0:26:49.710,0:26:53.620 +so now Im going to talk about what we've done + +0:26:53.620,0:26:59.510 + what we needed was the solution to handle +a wide range of job types + +0:26:59.510,0:27:01.570 +so all the options we looked at for instance + +0:27:01.570,0:27:04.990 +single application clusters of +project clusters + +0:27:04.990,0:27:11.990 +I think that the isolation they +provide is essentially unparalleled + +0:27:12.590,0:27:16.630 +and in our environment we probably have to +virtualize in order to be + +0:27:16.630,0:27:18.179 +efficient in terms of + +0:27:18.179,0:27:22.060 +being able to handle our job nix and what not to handle +the fact that our users + +0:27:22.060,0:27:23.740 +tend to have + +0:27:23.740,0:27:27.730 +spikes in their in their use + +0:27:27.730,0:27:32.799 +on a on a large scale so for instance we get those that show and say +that they need to run for a month + +0:27:32.799,0:27:33.780 +on and then + +0:27:33.780,0:27:38.460 +some indeterminate number of months later +they'll do it again + +0:27:38.460,0:27:40.840 +for that sort of quick + +0:27:40.840,0:27:41.480 +demands + +0:27:42.240,0:27:44.850 +we really need the virtuals something +virtualized + +0:27:44.850,0:27:47.120 +and then we got to pay the price of %uh + +0:27:47.120,0:27:48.380 +of the overhead + +0:27:48.380,0:27:51.590 +and again it doesn't handle small jobs and that is a + +0:27:51.590,0:27:54.050 +large portion of our job nix and + +0:27:54.050,0:27:55.180 +of that + +0:27:55.180,0:27:58.070 +quarter million or something jobs line + +0:27:58.070,0:27:59.700 +on our cluster + +0:27:59.700,0:28:02.490 +%uh I would guess that + +0:28:02.490,0:28:04.730 +more than half of those were submitted + +0:28:04.730,0:28:05.890 +in + +0:28:05.890,0:28:09.660 +batches of more than ten thousand + +0:28:09.660,0:28:11.400 +so they'll just pop up + +0:28:11.400,0:28:14.030 +the other method to have looked at + +0:28:14.030,0:28:14.800 +are up + +0:28:14.800,0:28:16.750 +were using resource limits + +0:28:16.750,0:28:19.060 +the nice thing of course is they're achievable +with + +0:28:19.060,0:28:21.429 +they acheive useful isolation + +0:28:21.429,0:28:26.289 +and the inflexible with under existing functionality or small +extension so that's what we think + +0:28:26.289,0:28:27.230 +concentrating on + +0:28:27.230,0:28:29.740 +also been doing some thinking about + +0:28:29.740,0:28:31.809 +could we use the techniques there + +0:28:31.809,0:28:33.940 +and combine them with jails + +0:28:33.940,0:28:36.170 +or related features + +0:28:36.170,0:28:40.019 +it may be bulking up jails to be more like ------ + +0:28:40.019,0:28:44.150 +or containers I think they're calling this +week + +0:28:44.150,0:28:44.840 +and + +0:28:44.840,0:28:46.770 +so we're looking that + +0:28:46.770,0:28:50.840 +to be able to provide + + +0:28:50.840,0:28:54.250 +to to to be able to provide pretty user operating environments + +0:28:54.250,0:28:59.200 +potentially isolating users from operating suffrance as we upgrade the kernel + +0:28:59.200,0:29:03.469 +and users can continue using it all the +images they don't have time to rebuild their + +0:29:03.469,0:29:04.330 +application in + +0:29:04.330,0:29:09.970 +and handle the updates in libraries and what not + +0:29:09.970,0:29:13.840 +they also have potential to provide strong isolation for security +purposes + +0:29:13.840,0:29:18.740 +%uh which could be useful in the future + +0:29:18.740,0:29:20.159 +we do think that that + +0:29:20.159,0:29:24.040 +of of its to of these of these mechanisms the nice thing is that +resource limit + +0:29:24.040,0:29:26.150 +the resource limits and partitioning scheme + +0:29:26.150,0:29:29.860 +as well as virtual private service are very +similar imitation requirements + +0:29:29.860,0:29:33.090 + set up a fair bit more expensive + +0:29:33.090,0:29:34.620 +in the VPS case + + +0:29:34.620,0:29:38.780 +while nonetheless they're fairly similar + +0:29:38.780,0:29:42.610 +what we've been doing is we've taken Sun Grid Engine we've taken Sun +Grid Engine + +0:29:42.610,0:29:46.880 +and we were originally intended to actually +extend Sun Grid Engine and modify its demands + +0:29:46.880,0:29:48.480 +to do work + +0:29:48.480,0:29:51.150 +on what we ended up doing instead is realize +that well + +0:29:51.150,0:29:54.910 +we can actually starts flying alternate program +to run instead of the shepherd + +0:29:54.910,0:29:57.990 +The shepherd is the process + +0:29:57.990,0:30:00.580 +that starts all + +0:30:00.580,0:30:02.250 +starts the the script that + +0:30:02.250,0:30:03.380 +can reach job + +0:30:03.380,0:30:04.920 +on a given node + +0:30:04.920,0:30:08.559 + it collects usage and forwards signals to the +children + +0:30:08.559,0:30:12.620 +and also is responsible for starting remote + components + +0:30:12.620,0:30:14.560 +so shepherd is started and then + +0:30:14.560,0:30:17.640 + traditionally in seperate engine it starts out + +0:30:17.640,0:30:19.910 +it's own --the event + +0:30:19.910,0:30:20.800 +and + +0:30:20.800,0:30:22.010 +jobs connect over + +0:30:22.010,0:30:23.670 +these days that for their own + +0:30:23.670,0:30:25.870 +you're a mechanism which is + +0:30:25.870,0:30:26.950 +secure + +0:30:26.950,0:30:28.000 +not using the + +0:30:28.840,0:30:30.530 +arch old code + +0:30:30.530,0:30:31.920 +on + +0:30:35.370,0:30:37.970 +so what we've done is we've implemented a rapid script + +0:30:37.970,0:30:40.139 +which allows a pre-command hook + +0:30:40.139,0:30:42.559 +to run before the shepherd starts + +0:30:42.559,0:30:47.170 +the command rapper send before we send shepherd before we can run like the N program + +0:30:47.170,0:30:49.150 +or the week and why not + +0:30:49.150,0:30:50.430 +troops or whatever + +0:30:50.430,0:30:54.040 + to set up the environment that it runs in or CPU + +0:30:54.040,0:30:56.600 +that will show later + +0:30:56.600,0:30:58.750 +on and I first met her for cleanup + +0:30:58.750,0:31:03.940 +simply move because I felt like it + +0:31:03.940,0:31:07.830 +the first thing we implemented is memory backs temporary directors. the motivation for + +0:31:07.830,0:31:08.700 +this + +0:31:08.700,0:31:09.640 +is that + +0:31:09.640,0:31:12.180 +we've had problems for users will you know + +0:31:12.180,0:31:15.510 +run slash ten on the nodes + +0:31:15.510,0:31:19.059 +where we have the figures is that they do have discs + +0:31:19.059,0:31:22.960 +and most of the disc is available as slash ten + +0:31:22.960,0:31:25.049 +we had some cases + +0:31:25.049,0:31:27.840 +particularly early on where users would fill the discs and not complete it + +0:31:27.840,0:31:32.300 +their job would crash and they wold forget to add clean up code or whatever + +0:31:32.300,0:31:35.100 +other jobs would fail strangely + +0:31:35.100,0:31:39.029 +you might expect that you just get a you would get a nice error message + +0:31:39.029,0:31:42.040 +being programmers + +0:31:42.040,0:31:42.909 +people would not + +0:31:42.909,0:31:44.630 +handle very correctly + +0:31:44.630,0:31:47.380 +now of course you have issues like for instance + +0:31:47.380,0:31:49.600 +the PDL library + +0:31:49.600,0:31:52.600 +unexpectedly fails and reports a completely strange error + +0:31:52.600,0:31:54.759 +if it can't create a file to have + +0:31:54.759,0:32:01.669 +because it needs to create an extra file in itself + +0:32:01.669,0:32:03.360 +so what we've done here + +0:32:03.360,0:32:08.059 +is it turns out that Sun Grid Engine actually creates a temporary +directory often the + +0:32:08.059,0:32:11.730 +typically but you can change +that + +0:32:11.730,0:32:14.490 +I think it's that's a + +0:32:14.490,0:32:15.370 +location + +0:32:15.370,0:32:17.499 +we educated most of all users now + +0:32:17.499,0:32:21.360 +to use that location correctly and values +that very cool + +0:32:21.360,0:32:23.279 +they treat their files understand her + +0:32:23.279,0:32:24.950 +and then when the job exits + +0:32:24.950,0:32:26.569 +the Grid Engine deletes the temp dir + +0:32:26.569,0:32:28.510 +and that all gets cleaned up + +0:32:28.510,0:32:32.720 +the problem of course being that of multiples +also warning on the same note same time + +0:32:32.720,0:32:35.290 +one of them could still fill temp + +0:32:35.290,0:32:38.759 +so the solution was pretty simple +we created a + +0:32:38.759,0:32:41.420 +rapper script at the beginning of the job + +0:32:41.420,0:32:42.760 +creates a + +0:32:42.760,0:32:43.940 +a + +0:32:43.940,0:32:47.260 +memory file to swap back to MB file system + +0:32:47.260,0:32:50.790 +of a user requestable size of the default + +0:32:50.790,0:32:53.310 +and + +0:32:53.310,0:32:56.520 +this has a number of advantages the biggest one of course is that + +0:32:56.520,0:32:58.320 +it's their fixed size so we get + +0:32:58.320,0:32:59.449 +you know + +0:32:59.449,0:33:01.000 + the user gets + +0:33:01.000,0:33:03.420 + what they asked for + +0:33:03.420,0:33:05.930 +and once they run of space, they run out of space well + +0:33:05.930,0:33:09.300 +and too bad they ran out of space + +0:33:09.300,0:33:12.760 +they should have asked for more + +0:33:12.760,0:33:16.350 +the other + +0:33:16.350,0:33:18.770 +the other advantage is the side-effect that + +0:33:18.770,0:33:21.619 +now that we're running swap back memory files systems for temp + +0:33:21.619,0:33:24.560 +the users who only use a fairly small amount of temp + +0:33:24.560,0:33:28.190 + should see vastly improved performance +because they're running in memory + +0:33:28.190,0:33:32.980 +rather than writing to disc + +0:33:32.980,0:33:34.690 +quick example + +0:33:34.690,0:33:38.270 +we've a little job script herel + +0:33:38.270,0:33:39.830 +prints chapter and + +0:33:39.830,0:33:41.950 +prints the + +0:33:41.950,0:33:43.080 +amount of space + +0:33:43.080,0:33:46.210 +we consider job request saying that we want + +0:33:46.210,0:33:51.539 +this is what we want hundred megabytes of +temp space + +0:33:51.539,0:33:53.580 +the same that's why if this + +0:33:53.580,0:33:55.230 +so the program doesn't + +0:33:55.230,0:33:57.620 +so the program ends at the end of it + +0:33:57.620,0:33:58.709 +%uh for doing it + +0:33:58.709,0:34:00.510 +heres a live demo + +0:34:00.510,0:34:01.840 +all and then + +0:34:01.840,0:34:03.389 +you look at the output + +0:34:03.389,0:34:04.280 +you can see it + +0:34:04.280,0:34:07.549 +does in fact it creates a memory file system + +0:34:07.549,0:34:10.449 +I attempted to do as great code + +0:34:10.449,0:34:13.409 +having a variable space that + +0:34:13.409,0:34:15.839 +having a variable space that is roughly what user asked for + +0:34:15.839,0:34:17.089 +the version I had + +0:34:17.089,0:34:20.739 +when I was attempting this was not entirely +accurate + +0:34:20.739,0:34:24.710 +trying to guess what all the +USFS overhead would be + +0:34:24.710,0:34:25.889 +as the result was + +0:34:25.889,0:34:28.399 +%uh not quite consistent + +0:34:30.790,0:34:33.899 +I couldn't figure out easy function so + +0:34:33.899,0:34:39.589 +it does a better job than it did to start with + +0:34:39.589,0:34:40.600 +sometimes however + +0:34:40.600,0:34:42.329 +today that that's a good case + +0:34:42.329,0:34:43.550 +we're coming to it + +0:34:43.550,0:34:45.359 +deployed pretty soon + +0:34:45.359,0:34:47.159 +it's pretty easily + +0:34:47.159,0:34:48.570 +well sometimes it's not enough + +0:34:48.570,0:34:51.390 +the biggest issue is that they were badly designed programs all + +0:34:51.390,0:34:52.720 +all over the world + +0:34:52.720,0:34:54.919 +don't you step to like they're supposed to + +0:34:54.919,0:34:59.319 +in fact + +0:35:10.099,0:35:12.759 +so there are all these applications + +0:35:12.759,0:35:17.979 +there are a lot about patience still that need +ten because they'll still need start up + +0:35:17.979,0:35:19.230 +that sort of thing + +0:35:19.230,0:35:20.809 +so + +0:35:20.809,0:35:22.599 +all + +0:35:22.599,0:35:25.829 +so we have problems with these + +0:35:25.829,0:35:26.290 +realistically + +0:35:26.290,0:35:27.799 +we cant change all of them + +0:35:27.799,0:35:30.019 +it's just not going to happen + +0:35:30.019,0:35:31.950 +so we still have a lot of people + +0:35:31.950,0:35:34.509 +running out resources + +0:35:34.509,0:35:35.819 +%uh so we probably + +0:35:35.819,0:35:37.489 +feel that + + +0:35:37.489,0:35:41.240 +was general solution is right a per job slash temp + +0:35:41.240,0:35:44.880 +the first was that her from the files system +at its best + +0:35:44.880,0:35:47.119 +we think there is some ways to me that + +0:35:47.119,0:35:52.539 +and so he said okay let's give it a shot + +0:35:52.539,0:35:56.969 +just to inter these concepts for people who are unfamiliar with him + +0:35:56.969,0:36:00.280 +offering someone services recent ones that +contained rules + +0:36:00.280,0:36:02.389 +Richard Senator long time + +0:36:02.389,0:36:05.549 +angeles half past be different for different +processes + +0:36:05.549,0:36:06.969 +for example + +0:36:06.969,0:36:08.689 +you create the files + +0:36:08.689,0:36:10.069 +on create + +0:36:10.069,0:36:12.459 +they ask someone whose contents are + +0:36:12.459,0:36:18.329 +this veritable which has a the fall not shells +fell the fall diet + +0:36:18.329,0:36:18.990 +and you + +0:36:18.990,0:36:24.949 +he didn't get different results with different +variables that + +0:36:24.949,0:36:27.170 +what about her the implementation we've got + +0:36:27.170,0:36:32.389 +it's drive from grateful and mission was to +the data structures are gonna call + +0:36:32.389,0:36:33.869 +authorities a number of changes + +0:36:33.869,0:36:39.649 +the biggest one is that the two the concept +of us troops and returned them entirely around + +0:36:39.649,0:36:40.409 +he added that + +0:36:40.409,0:36:45.329 +in trade as one of his assistants the which +is over overridden by the users scope and by a + +0:36:45.329,0:36:47.259 +progressive scope + +0:36:49.819,0:36:53.449 +problem with that is if you + +0:36:53.449,0:36:56.099 +only think about say the systems scope + +0:36:56.099,0:36:57.079 +and + +0:36:57.079,0:36:59.459 +you decide you want to do something clever like have + +0:36:59.459,0:37:02.219 +root file system which +%uh + +0:37:02.219,0:37:06.109 +were slashed with points the different things +for different %uh + +0:37:06.109,0:37:08.249 +different architectures + +0:37:08.249,0:37:11.849 +were seriously until the users come along +and + +0:37:11.849,0:37:14.189 +they're upset there are variable + +0:37:14.189,0:37:15.629 +up for you + +0:37:15.629,0:37:18.900 +if you have CSX like the program and you don't +defensively + +0:37:18.900,0:37:22.319 +and you don't implement correctly + +0:37:22.319,0:37:24.900 +the obvious that they sat obviously you would + +0:37:24.900,0:37:28.599 +Richard riordan often that I believe they +did + +0:37:28.599,0:37:31.700 +there's a whole class of problems where + +0:37:31.700,0:37:33.449 +it's easy to screw up + +0:37:33.449,0:37:36.219 +add and do something wrong there + +0:37:36.219,0:37:37.270 +so by + +0:37:37.270,0:37:38.509 +reversing the order + +0:37:38.509,0:37:41.849 +we can't we can reduce the risks + +0:37:41.849,0:37:43.329 +at the moment we don't + +0:37:43.329,0:37:44.309 +haven't you sir + +0:37:44.309,0:37:47.530 +I just don't like the idea of the users scope +to be honest + +0:37:47.530,0:37:50.900 +from being they see you dont have to have +poor user + +0:37:50.900,0:37:55.509 +that just sits around forever +you can never guard elected accepted the strain + +0:37:55.509,0:37:57.059 +of late + +0:37:57.059,0:37:59.489 +just doesn't seem like a great idea to me + +0:37:59.489,0:38:00.700 +on it + +0:38:00.700,0:38:04.609 +just hasn't been implemented + +0:38:04.609,0:38:09.809 +because it wasn't entirely clear as to what the semantics should be + +0:38:09.809,0:38:11.010 +i also + +0:38:11.010,0:38:14.719 +I also added default variable support variable +also shell style + +0:38:14.719,0:38:16.999 +for + +0:38:16.999,0:38:19.169 +to some extent undoes the scope + +0:38:19.169,0:38:20.870 +the scope change + +0:38:20.870,0:38:21.779 +in that + +0:38:21.779,0:38:24.749 +the default variable becomes a system scope + +0:38:24.749,0:38:26.540 +which is overridden by everything + +0:38:26.540,0:38:30.890 +but there are cases where we need to do that +in particular who wants implement their + +0:38:30.890,0:38:33.380 +slashed temp which varies + +0:38:33.380,0:38:36.240 +we have to do something like this because the temp needs to work + +0:38:36.240,0:38:37.209 +if + +0:38:37.209,0:38:42.059 +if we don't have the job values set + +0:38:42.059,0:38:45.829 +I also decided to use + +0:38:45.829,0:38:49.839 +percent instead of dollar signs to avoid +confusion over shell variables because these + +0:38:49.839,0:38:50.379 +are + +0:38:50.379,0:38:52.620 +a separate namespace in the kernel + +0:38:52.620,0:38:56.669 +can't do it in a nice to do all the evaluation in the +user space + +0:38:56.669,0:38:59.269 +it's classic vulnerability + +0:38:59.269,0:39:02.739 +the that in the database for instance + +0:39:02.739,0:39:08.109 +and when I was in the past and avoid confusion +with but yet that's and worthy and for the + +0:39:08.109,0:39:09.819 +now BST implementation + +0:39:09.819,0:39:11.019 +first is not allowed + +0:39:11.019,0:39:14.879 +he's a reduced rate of one set of core values + +0:39:14.879,0:39:17.019 +that will be enough ballots for + +0:39:17.019,0:39:20.359 +on I don't have any automated variables such +as the %uh + +0:39:20.359,0:39:25.789 +the process is not you which is universally +sat and that he is the implementation war + +0:39:25.789,0:39:26.750 +on + +0:39:26.750,0:39:28.039 +aids + +0:39:28.039,0:39:32.579 +hey I do i'd be very foolish they also have + +0:39:32.579,0:39:34.909 +and currently and it allows Senator + +0:39:34.909,0:39:40.880 +setting about using other processes yourself +in your own insurance + +0:39:40.880,0:39:42.699 +that may change but it's a + +0:39:42.699,0:39:47.339 +one of my goals here is because they were +subtle ways to make no mistakes and + +0:39:47.339,0:39:48.930 +cause securities undergo days + +0:39:48.930,0:39:52.479 +I I've attempted to storm the features that +down to the point where you + +0:39:52.479,0:39:54.909 +at some reasonable chance of not + +0:39:54.909,0:39:56.339 +doing that + +0:39:56.339,0:40:03.339 +if you start building systems on the for + +0:40:04.419,0:40:06.909 +the final area that we've worked on + +0:40:06.909,0:40:09.499 +he is moving away from the final system states + +0:40:09.499,0:40:12.559 +and the NCR see these sets of them + +0:40:12.559,0:40:16.379 +chuck roberts indictment alleges a out for +them out + +0:40:16.379,0:40:20.699 +people may I see he said functionality which +allows you to + +0:40:20.699,0:40:23.489 +but it also seems the issues that have been +set + +0:40:23.489,0:40:24.879 +the affinity of that + +0:40:24.879,0:40:26.269 +see he said + +0:40:26.269,0:40:29.189 +political observers also stopped in on this + +0:40:29.189,0:40:33.059 +and on a CD set was it was stuffed in the +one that was created by this + +0:40:33.059,0:40:37.269 +in an apparent + +0:40:37.269,0:40:38.619 +so what I hear + +0:40:38.619,0:40:40.740 +it's here unless you can figure issue + +0:40:40.740,0:40:42.769 +every day has won so what + +0:40:42.769,0:40:44.429 +first you here + +0:40:44.429,0:40:48.639 +there there are a number of other ways you +can configure basically us lost is something + +0:40:48.639,0:40:50.019 +in jobs and money + +0:40:50.019,0:40:56.719 +%uh federal jobs crosses what's going to happen +would be more than one swatch + +0:40:56.719,0:41:01.359 +pop quizzes making the applications where +persons who tends to spend a fair bit of time + +0:41:01.359,0:41:02.380 +waiting for Iran + +0:41:02.380,0:41:06.209 +you are looking at more than one slot per CPU so two slots per + +0:41:06.209,0:41:08.089 +CPU is not uncommon + +0:41:08.089,0:41:10.869 +but probably the most common configuration +and the one that + +0:41:10.869,0:41:13.719 +you get out of the box is you just install a Grid Engine + +0:41:13.719,0:41:16.739 + wants free CPU + +0:41:16.739,0:41:19.830 +and that's how that's how we run because we +want users to have + +0:41:19.830,0:41:23.699 +that whole CPU for whatever they want to do with +it + +0:41:23.699,0:41:26.130 +So drums are allocated one or more slots + +0:41:26.130,0:41:27.599 +if they're + +0:41:27.599,0:41:33.189 +depending on whether they're sequential or parallel jobs +and how many they ask for + +0:41:33.189,0:41:37.239 +but there is but this is just a convection +there's no actual connection between slots + +0:41:37.239,0:41:39.119 +and CPUs + +0:41:39.119,0:41:40.829 +it's quite possible to + +0:41:40.829,0:41:42.819 +submit a non-parallel job + +0:41:42.819,0:41:45.019 +that goes often spawns a zillion threads + +0:41:45.019,0:41:48.369 +and sucks up the whole system + +0:41:48.369,0:41:50.800 +in some early versions of grid engine + +0:41:50.800,0:41:53.569 +there actually was up + +0:41:53.569,0:41:55.729 +support for tying slots + +0:41:55.729,0:41:58.669 +for CPU to set it up that +way + +0:41:58.669,0:42:02.979 +there is a sensible implementation for Iraq's +and then things got weirder and weirder is + +0:42:02.979,0:42:06.010 +people try to implement it on other platforms +which had + +0:42:06.010,0:42:07.030 +vastly different + +0:42:07.030,0:42:09.839 +CPU binding semantics + +0:42:09.839,0:42:12.359 +and at this point in time we broke it + +0:42:12.359,0:42:14.959 +on every platform as far as I can tell + +0:42:14.959,0:42:18.759 +also the we decided okay we've got this rapper +let's see what we can do + +0:42:18.759,0:42:21.009 +on in terms of making things work +0:42:21.009,0:42:21.659 +certainly + +0:42:21.659,0:42:27.119 +we now have have the rapper store allocations in the final system + +0:42:27.119,0:42:31.239 +%uh three and nineteen ninety percent that +allocation I've read them + +0:42:31.239,0:42:33.369 +well we try to do years + +0:42:33.369,0:42:34.690 +find the best fit + +0:42:34.690,0:42:35.779 +fitting set of + +0:42:35.779,0:42:39.539 +adjacent cores + +0:42:39.539,0:42:42.329 +and then if that doesn't work we take orders +to repeat + +0:42:42.329,0:42:43.519 +%um + +0:42:43.519,0:42:45.180 +and until we fix + +0:42:45.180,0:42:47.300 +or until we've got enough slots + +0:42:47.300,0:42:50.800 +the goal is to minimize the fragments we haven't +done any analysis + +0:42:50.800,0:42:52.269 +to determine whether that's actually + +0:42:52.269,0:42:55.179 +an appropriate algorithm + +0:42:55.179,0:42:56.289 +off hand it seems + +0:42:56.289,0:43:00.519 +you'd find another part of a privilege + +0:43:00.519,0:43:02.810 +should forties lay down their arms the US +is + +0:43:02.810,0:43:09.649 +on turns out the FreeBSD, CPU API +and the last one + +0:43:09.649,0:43:12.519 +differ only in very small details + +0:43:12.519,0:43:13.599 +on that + +0:43:13.599,0:43:15.479 +essentially exactly + +0:43:15.479,0:43:17.569 +identical which is + +0:43:17.569,0:43:20.489 +correct in terms pretty semantically + +0:43:20.489,0:43:24.869 +so I think it is of interest to demonstrate +the effectiveness of cebu said they also happen + +0:43:24.869,0:43:27.019 +to demonstrate the %uh + +0:43:27.019,0:43:28.089 +the %uh + +0:43:28.089,0:43:29.359 +rather they probably have + +0:43:29.359,0:43:33.319 +the relevance + +0:43:33.319,0:43:35.229 +it's all of the young box + +0:43:35.229,0:43:36.629 +%um + +0:43:36.629,0:43:38.289 +the and %uh + +0:43:38.289,0:43:40.749 +%uh someone clearly is that + +0:43:40.749,0:43:43.239 +John dalton and backward it's not + +0:43:43.239,0:43:46.640 +our CD set it up for me + +0:43:46.640,0:43:49.039 +shortly before they released + +0:43:49.039,0:43:53.450 +well it's usually is supposed to be shortly +before + +0:43:53.450,0:43:55.579 +and that in essence six two + +0:43:55.579,0:43:59.739 +I will use the simple intervention or so and +greens + +0:43:59.739,0:44:02.519 +program were tested + +0:44:02.519,0:44:03.349 +for instance it + +0:44:03.349,0:44:05.360 +this any play for place + +0:44:05.360,0:44:08.069 +the queen so they can capture each other + +0:44:08.069,0:44:09.289 +on the board + +0:44:09.289,0:44:11.039 +%um + +0:44:11.039,0:44:13.680 +so it's a it's a simple symbol of benchmark + +0:44:13.680,0:44:18.800 +%uh that we ran a a small version of the problem +is our as a measure to man the man to generate + +0:44:18.800,0:44:19.599 +one of the + +0:44:19.599,0:44:24.439 +greta we're close and that we have much longer + +0:44:24.439,0:44:28.149 +some results + +0:44:28.149,0:44:30.129 +so for baseline do it for us + +0:44:30.129,0:44:33.170 +I think the most interesting thing is to do +a slot blot + +0:44:33.170,0:44:34.279 +you see this + +0:44:34.279,0:44:36.410 +some very it's not really very high + +0:44:36.410,0:44:38.979 +not surprising that doesn't really do anything + +0:44:38.979,0:44:40.979 +on accept socks see here + +0:44:40.979,0:44:41.729 +so on + +0:44:41.729,0:44:45.229 +going on what's going on + +0:44:45.229,0:44:50.029 +they don't think in this case for about seven +to one processes and a single + +0:44:50.029,0:44:52.789 +a single assassin process morning + +0:44:52.789,0:44:55.160 +this is the slogans wait wait + +0:44:55.160,0:44:55.890 +and %uh + +0:44:55.890,0:44:58.389 +the standard deviation goes up a bit + +0:44:58.389,0:45:00.829 +to live with a deviation from baseline + +0:45:00.829,0:45:03.659 + the obvious explanation is this + +0:45:03.659,0:45:07.339 +you know were just context switch is a bit +more + +0:45:07.339,0:45:08.840 +and %uh and %uh + +0:45:08.840,0:45:10.349 +because we don't have + +0:45:10.349,0:45:12.410 +CPUs that are doing nothing at all + +0:45:12.410,0:45:15.559 +on this is there some extra load from the system +as well + +0:45:15.559,0:45:20.049 +since the kernel have to run and not contests +have to run + +0:45:20.049,0:45:23.150 +you know if this is a story about maybe a +deposition story + +0:45:23.150,0:45:26.579 +we have people across this is what would some +couples the year + +0:45:26.579,0:45:28.879 +you know we try to run a marathon process + +0:45:28.879,0:45:30.639 +we see a + +0:45:30.639,0:45:32.739 +substantial performance the trees + +0:45:32.739,0:45:35.570 +you know abandon the interest rates that's +a + +0:45:35.570,0:45:37.289 +see if any + +0:45:37.289,0:45:40.140 +the trees + +0:45:40.140,0:45:43.220 +we fired up because the views that + +0:45:43.220,0:45:44.249 +when I saw it + +0:45:44.249,0:45:46.190 +the interesting thing here is to see it + +0:45:46.190,0:45:49.429 +we didn't know statistically significant difference + +0:45:49.429,0:45:52.819 +not between the baseline news with with a + +0:45:52.819,0:45:56.539 +southern cross is it for you see you sense +we don't see this very + +0:45:56.539,0:45:58.520 +which is nice to know that this is it + +0:45:58.520,0:45:59.509 +that's it + +0:45:59.509,0:46:02.869 +we have to we have to see a slight performance +improvement + +0:46:02.869,0:46:04.179 +and %uh + +0:46:04.179,0:46:05.579 +we %uh + +0:46:05.579,0:46:07.589 +we see a reduction in various + +0:46:07.589,0:46:11.569 +hans this issue he says action program performance +even lot of love it + +0:46:11.569,0:46:13.510 +and then you see a vote in the case + +0:46:13.510,0:46:15.589 +it's it's a it's the same + +0:46:15.589,0:46:20.319 +for the opposite the other on process is a +stop on others he years + +0:46:20.319,0:46:22.820 +one interesting side note actually is there + +0:46:22.820,0:46:26.719 +where is the what I was doing some tests early +on + +0:46:26.719,0:46:27.869 +we actually saw + +0:46:27.869,0:46:32.359 +the training base line in the base and a seat +in the senate he just fired off with the original + +0:46:32.359,0:46:33.869 +either of them which + +0:46:33.869,0:46:34.540 +greta + +0:46:34.540,0:46:36.489 +grass seed use your own + +0:46:36.489,0:46:39.339 +he's also if a performance decline + +0:46:39.339,0:46:42.319 +because there's a lot of stuff that ends up +running on see user + +0:46:42.319,0:46:43.819 +which %uh + +0:46:43.819,0:46:45.100 +what led to the year + +0:46:45.100,0:46:49.890 +the for a conservationist you want to allocate +from the large numbers down + +0:46:49.890,0:46:50.569 +so the issue + +0:46:50.569,0:46:55.069 +you see used to not learning the random process +is that it's gone forever we're getting all + +0:46:55.069,0:46:57.880 +the interruptions some architecture's + +0:46:57.880,0:47:02.199 +and it's a way to force the road project + +0:47:02.199,0:47:04.029 +so some conclusions + +0:47:04.029,0:47:07.530 +all I think we have useful prefer concept +of going to be deploying + +0:47:07.530,0:47:09.880 +I was certainly the man with the %uh + +0:47:09.880,0:47:11.000 +memories are seeing + +0:47:11.000,0:47:13.329 +once we have to bring it to you at seven what + +0:47:13.329,0:47:15.959 +definitely be going to see few sets up as +well + +0:47:15.959,0:47:16.849 +so it's a + +0:47:16.849,0:47:18.509 +both includes performance + +0:47:18.509,0:47:22.009 +in the contentious in the and contentious + +0:47:22.009,0:47:26.299 +we would like in the future to do some work +with a personal private superstar + +0:47:26.299,0:47:28.979 +in particular and the really interesting + +0:47:28.979,0:47:30.759 +you know when different + +0:47:30.759,0:47:32.540 +different previous the persons in jails + +0:47:32.540,0:47:37.660 +for to run up for instance up one several +sentences in jail since eleven cents lost + +0:47:37.660,0:47:40.649 +on our allies assistance + +0:47:40.649,0:47:43.240 +there could actually be some really interesting +things there + +0:47:43.240,0:47:45.759 +on in that process which one + +0:47:45.759,0:47:50.989 +we think is actually teach reason except occasions +it's never going to happen only if one takes + +0:47:50.989,0:47:53.069 +we are simply there's another example work + +0:47:53.069,0:47:56.269 +%uh also goes to Tsvangirai who recently + +0:47:56.269,0:48:01.039 +and what if the lights on the same farm workers +seen if we don't have time to implement + +0:48:01.039,0:48:04.900 +all in basic matrix multiplication + +0:48:04.900,0:48:07.230 +relative to current with with current because + +0:48:07.230,0:48:08.549 +%um + +0:48:08.549,0:48:11.849 +previously supervision functionality + +0:48:11.849,0:48:14.499 +that's been reduced the number of GOP entries + +0:48:14.499,0:48:16.150 +on anonymity a stable + +0:48:16.150,0:48:17.229 +and sir + +0:48:17.229,0:48:21.109 +that sort of thing can apply even to allow +me to sing population + +0:48:21.109,0:48:23.969 +a previous the summer winds there + +0:48:23.969,0:48:26.309 +on + +0:48:26.309,0:48:27.579 +Michael what did that work + +0:48:27.579,0:48:30.859 +on the whole on the planet who is leading +uses proliferates + +0:48:30.859,0:48:32.620 +one of the issues we've had is that + +0:48:32.620,0:48:34.019 +you will need to win in the fall + +0:48:34.019,0:48:38.399 +we have reasons to depend on all sorts of +a also to libraries immediate which + +0:48:38.399,0:48:41.380 +you know the vendors like to have them to +do + +0:48:41.380,0:48:44.640 +stupid eight API briefing change is fairly +regularly said + +0:48:44.640,0:48:48.380 +it be nice for users if we can get all the +benefits to cooperate + +0:48:48.380,0:48:51.699 +and you wouldn't have taken operate at their +leisure + +0:48:51.699,0:48:54.459 +so we're hoping to be that in future as well + +0:48:54.459,0:48:57.809 +all would like to see more women Sunday with +type resources + +0:48:57.809,0:48:59.219 +%um + +0:48:59.219,0:49:01.199 +for instance a limiting the amount of + +0:49:01.199,0:49:02.910 +%um + +0:49:02.910,0:49:05.649 +it's it's really what you want you know it's +like you know + +0:49:05.649,0:49:10.279 +all but it's her if you want a place of for +a limit on networking with by a particular + +0:49:10.279,0:49:11.819 +process + +0:49:11.819,0:49:16.979 +all our store almost forced to resign one +and ask how do you classify that traffic with + +0:49:16.979,0:49:17.649 +that + +0:49:17.649,0:49:21.259 +after going to change the current somehow +taking that + +0:49:21.259,0:49:23.799 +it's an interesting challenge + +0:49:23.799,0:49:28.309 +we'd also like to see it could be needed some +you implement something like blacks the irish + +0:49:28.309,0:49:30.089 +job when you're out there + +0:49:30.089,0:49:34.099 +so I was scheduled to just hang out at processes +as part of a job + +0:49:34.099,0:49:36.309 +%uh currently + +0:49:36.309,0:49:38.939 +I've heard it uses a clever but people past + +0:49:38.939,0:49:40.010 +were they out + +0:49:40.010,0:49:42.509 +an extra boost to the process + +0:49:42.509,0:49:44.819 +and they just outrageous troops there + +0:49:44.819,0:49:48.209 +available so they get inherited in the users +can drop them said + +0:49:48.209,0:49:51.889 +thousand track process that it's about what +happened with the correct limits on the number + +0:49:51.889,0:49:57.499 +of groups that can become a real problem + +0:49:57.499,0:49:59.529 +actually for raising questions + +0:49:59.529,0:49:59.980 +argumentative + +0:49:59.980,0:50:01.119 +one quick point + +0:50:01.119,0:50:05.100 +the think it's not interesting you live in +the area and if you're looking for + +0:50:05.100,0:50:06.430 +looking for jobs + +0:50:06.430,0:50:09.780 +we are trying to hire few people it's difficult +to hire good + +0:50:09.780,0:50:13.069 +we do have some some openings and you're looking +for + +0:50:13.069,0:50:17.409 +PSD people in general system ads +people + +0:50:17.409,0:50:24.409 +so questions + +0:50:38.419,0:50:40.989 +%um + +0:50:40.989,0:50:45.719 +I would would expect that to happen +but it's not something that attempted to test + +0:50:45.719,0:50:50.570 +what I would really like is a topology allocator + +0:50:50.570,0:50:53.179 +so that you can request that you know I want + +0:50:53.179,0:50:56.229 +I want to share cache or I don't want to share cache + +0:50:56.229,0:51:00.170 +I want to share memory band width or i want to not share memory band width + +0:51:00.170,0:51:02.459 +actually open MPI of three + +0:51:02.459,0:51:08.469 +on the 1x side have a topology where a rapper for that CPU + +0:51:08.469,0:51:10.159 +functionality + +0:51:10.159,0:51:12.249 +makes it something called %uh + +0:51:12.249,0:51:14.139 +the PLAP + +0:51:14.139,0:51:15.259 +portable lenux + +0:51:15.259,0:51:16.519 +%um + +0:51:16.519,0:51:19.599 +CPU allocator.Is that what +it's actually been + +0:51:19.599,0:51:21.959 +what it would act the act on his part + +0:51:21.959,0:51:25.400 +in essence they have to work around the fact +that there were three standard + +0:51:25.400,0:51:27.809 +there there were three different + +0:51:27.809,0:51:31.759 +currently the eyes for the same siskel + +0:51:31.759,0:51:38.759 +first the EU or takes to get all the letters +to get himself some have used the same number + +0:51:38.769,0:51:44.969 +they're completely about some of these guys +were for routine + +0:51:44.969,0:51:48.749 +when you first saw the application it calls +and this is called a test of figure out which + +0:51:48.749,0:51:50.579 +one it is + +0:51:50.579,0:51:52.719 +whitewater reserve returns depending on what + +0:51:52.719,0:51:56.139 +are you missing and completely evil + +0:51:56.139,0:52:00.859 +I think people support the heat very the eye +and we should have their lives were that + +0:52:00.859,0:52:05.650 +we don't need to do that job because they +didn't make that mistake + +0:52:05.650,0:52:12.650 +so I would like to see it of all its universal +particular + +0:52:30.710,0:52:32.529 +yes larry King + +0:52:32.529,0:52:37.180 +the trick is to do you want to be leaving +it's he's he's a limited application + +0:52:37.180,0:52:38.869 +then with + +0:52:38.869,0:52:39.500 +on + +0:52:39.500,0:52:42.269 +there's no easy limit application + +0:52:42.269,0:52:44.329 +the defense more difficult when you have to + +0:52:44.329,0:52:45.430 +the if you're + +0:52:45.430,0:52:49.759 +the new faces shared between application traffic + +0:52:49.759,0:52:50.880 +%um + +0:52:50.880,0:52:53.049 +they had a fast + +0:52:53.049,0:52:57.399 +getting classifying that is going to treat +you have to take you what the data for particular + +0:52:57.399,0:53:04.399 +to to trace that down through the current +interim I said certainly do + +0:53:12.069,0:53:15.499 +I I am + +0:53:15.499,0:53:18.389 +I I have talked contemplating doing just that + +0:53:18.389,0:53:22.059 +or in fact %uh the other thing we consider +doing + +0:53:22.059,0:53:24.829 +morris a research project that is not practical + +0:53:24.829,0:53:26.719 +jane would be actually help + +0:53:26.719,0:53:28.619 +with would be + +0:53:28.619,0:53:30.029 +independent he lands + +0:53:30.029,0:53:31.839 +because then we could do + +0:53:31.839,0:53:32.459 +things like + +0:53:32.459,0:53:35.489 +the peace process of the lambs' the couldn't +even + +0:53:35.489,0:53:37.979 +sheer at the internet where + +0:53:37.979,0:53:41.259 +once the images in place for instance we will +be able to do that + +0:53:41.259,0:53:45.049 +and that say you know you've got your cases +jurors whatever + +0:53:45.049,0:53:46.479 +on but then we can't win it + +0:53:46.479,0:53:49.959 +we cannot we could raise when did that occur +on can also have + +0:53:49.959,0:53:54.729 +we got of physically isolated we got a lot +of this would have been at work as well + +0:53:54.729,0:53:57.589 +with some analysts which is we can actually +raped women + +0:53:57.589,0:54:04.589 +in this without the switch as well + +0:54:19.939,0:54:22.369 +Bob so that's the first question + +0:54:22.369,0:54:26.190 +we get more and more visible sensitivity cannot +miss foster's + +0:54:26.190,0:54:27.639 +last night the oscars + +0:54:27.639,0:54:28.709 +I'm sorry + +0:54:28.709,0:54:30.460 +we've avoided that problem I think + +0:54:30.460,0:54:32.299 +not allowing it + +0:54:32.299,0:54:34.929 +but is it is a real issue + +0:54:34.929,0:54:36.939 +she's not one we've had to do it + +0:54:36.939,0:54:39.559 +%um + +0:54:39.559,0:54:42.109 +in practice was stuff that sensitive + +0:54:42.109,0:54:43.059 +%um + +0:54:43.059,0:54:47.579 +has handling requirements that you can't touch +the same program without a strong said + +0:54:47.579,0:54:49.859 +you need a mystery + +0:54:49.859,0:54:51.739 +ridiculously aggressive + +0:54:51.739,0:54:53.770 +you need a very close to a new letter to the +end + +0:54:53.770,0:54:57.240 +a ridiculous noted aging process that you +never moved over there + +0:54:57.240,0:55:00.959 +so if I were to do that until we get rid of +that test + +0:55:00.959,0:55:01.389 +just + +0:55:01.389,0:55:02.400 +the witness list + +0:55:02.400,0:55:04.910 +don't get rid of my number-one failure case +of + +0:55:04.910,0:55:07.839 +that would be pretty good but + +0:55:07.839,0:55:09.419 +but havent done that + +0:55:09.419,0:55:10.609 +on + +0:55:10.609,0:55:13.819 +so we we've had occasional problems of NFS overloading + + +0:55:13.819,0:55:15.679 +we haven't had real problem + +0:55:15.679,0:55:19.279 +we're all local network is fairly tightly +contained so we haven't had problems with + +0:55:19.279,0:55:20.539 +things + +0:55:20.539,0:55:21.819 +%uh with + +0:55:21.819,0:55:26.039 +you know the server going down for extended +periods and causing everything to hang + +0:55:26.039,0:55:27.819 +it's been more an issue of + +0:55:27.819,0:55:33.189 +I mean there there isn't there's a problem +that Panache is described as in cast + +0:55:33.189,0:55:36.109 +you can take out any NFS + +0:55:36.109,0:55:40.809 +I mean we have the BLueDrat guys come in and the +PGA this stuff multiple ten-gate I said + +0:55:40.809,0:55:42.049 +you know I've got + +0:55:42.049,0:55:46.779 +to do this and they said can we not try this with all your cluster + +0:55:46.779,0:55:47.950 +because if you got + +0:55:47.950,0:55:49.370 +three hundred and fifty + +0:55:49.370,0:55:52.599 +gigabit ethernet interface is going into +the system + +0:55:52.599,0:55:56.589 +even ten gig you can saturate pre-turbulate + +0:55:56.589,0:55:57.120 +so that level + +0:55:57.120,0:55:58.930 +there's an inherent problem + +0:55:58.930,0:56:01.969 +on we need to handle that kind of band width +we've + +0:56:01.969,0:56:04.459 +got to get it a parallel file system + +0:56:04.459,0:56:06.069 +get a cluster + +0:56:06.069,0:56:12.289 +before doing streaming stuff we could go file some loners + +0:56:12.289,0:56:14.949 +anyone else? + +0:56:14.949,0:56:15.429 +thank you diff --git a/en_US.ISO8859-1/captions/2009/dcbsdcon/dixon-bsdisstilldying.sbv b/en_US.ISO8859-1/captions/2009/dcbsdcon/dixon-bsdisstilldying.sbv new file mode 100644 index 0000000000..baf87e0de7 --- /dev/null +++ b/en_US.ISO8859-1/captions/2009/dcbsdcon/dixon-bsdisstilldying.sbv @@ -0,0 +1,1484 @@ +0:00:00.460,0:00:04.370 +BSD is Still Dying + +0:00:04.370,0:00:06.220 +Welcome to BSD is still Dying + +0:00:06.220,0:00:07.790 +It’s not quite dead yet + +0:00:07.790,0:00:09.409 +but we're getting there + +0:00:09.409,0:00:09.980 +and %uh + +0:00:09.980,0:00:16.980 +Well, welcome to the closing of the DCBSDCon +2009 + +0:00:17.240,0:00:19.330 +So, what is BSD? + +0:00:19.330,0:00:22.299 +Well, BSD is the derivative of UNIX + +0:00:22.299,0:00:23.879 +Okay, so what is UNIX? + +0:00:23.879,0:00:26.749 +UNIX is an operating system + +0:00:26.749,0:00:29.259 +What’s an operating system? + +0:00:29.259,0:00:32.560 +Operating system is the soul of the computer + +0:00:32.560,0:00:34.000 +But, what’s a computer? + +0:00:34.000,0:00:37.469 +A computer is a tool, it’s basically a glorified calculator + +0:00:37.469,0:00:42.160 +that enables users to accomplish tasks better + +0:00:42.160,0:00:44.060 +So, what is a user? + +0:00:44.060,0:00:46.190 +A user is someone who operates a computer + +0:00:46.190,0:00:49.580 +It tends to stand up right sort of like me + +0:00:49.580,0:00:51.940 +and Bob(?) + +0:00:51.940,0:00:52.840 +So, who am I? + +0:00:52.840,0:00:54.610 +My name is Jason Dixon + +0:00:54.610,0:00:57.080 +First and foremost, I’m a SysAdmin + +0:00:57.080,0:00:59.500 +I like to work on networks and firewalls + +0:00:59.500,0:01:02.870 +I like to tweak + +0:01:02.870,0:01:03.630 +no, yes + +0:01:03.630,0:01:05.650 +I'm a programmer, sort of + +0:01:05.650,0:01:07.190 +I enjoy programming with Perl + +0:01:08.450,0:01:10.340 +and Apache Web servers + +0:01:10.340,0:01:12.229 +I'm a consultant here + +0:01:12.229,0:01:15.159 +I'm an employee + +0:01:15.159,0:01:16.840 +there + +0:01:16.840,0:01:18.880 +.. + +0:01:18.880,0:01:19.819 +I'm sorry, [xx] meets the eye + +0:01:19.819,0:01:25.430 +I + +0:01:25.430,0:01:30.229 +[xx] + +0:01:30.229,0:01:32.909 +[xx] + +0:01:32.909,0:01:33.870 +And I'm a lover + +0:01:33.870,0:01:36.240 +of BSD + +0:01:36.240,0:01:38.360 +Okay, but why am I here? + +0:01:38.360,0:01:39.880 +To talk about why + +0:01:39.880,0:01:43.830 +BSD is dying + +0:01:43.830,0:01:45.270 +sex + +0:01:45.270,0:01:47.710 +and greed + +0:01:47.710,0:01:50.210 +It really has nothing to do with either of this + +0:01:50.210,0:01:53.170 +But if I told you licensing is a + +0:01:53.170,0:01:55.750 +[xx] + +0:01:55.750,0:01:58.160 +To quickly summarize, what is BSD? + +0:01:58.160,0:01:59.290 +What is UNIX? + +0:01:59.290,0:02:00.800 +What is an operating system? + +0:02:00.800,0:02:02.310 +What is a computer? + +0:02:02.310,0:02:04.929 +A computer is a device that computes + +0:02:04.929,0:02:08.429 +especially, a programmable electronic machine +that performs high speed mathematical + +0:02:08.429,0:02:10.180 +or logical operations + +0:02:10.180,0:02:13.869 +or that assembles, stores, correlates, or otherwise processes +information + +0:02:13.869,0:02:15.189 +This is a computer + +0:02:15.189,0:02:17.029 +This is a small computer + +0:02:17.029,0:02:18.899 +This is a big computer + +0:02:18.899,0:02:20.169 +This is a big + +0:02:20.169,0:02:23.479 +fake(?) computer + +0:02:23.479,0:02:26.889 +And this is a really old computer + +0:02:26.889,0:02:28.909 +Well, what does a computer really do? + +0:02:28.909,0:02:30.670 +It helps us write documents + +0:02:30.670,0:02:32.509 +For example, [xx] fun pages + +0:02:32.509,0:02:33.859 +I + +0:02:33.859,0:02:36.159 +[xx] shopping lists + +0:02:36.159,0:02:38.569 +Computers can even delete documents on the + +0:02:38.569,0:02:41.459 +fly + +0:02:41.459,0:02:43.340 +We can write emails + +0:02:43.340,0:02:44.370 +surf the Web + +0:02:44.370,0:02:46.539 +watch movies + +0:02:46.539,0:02:48.889 +movies + +0:02:48.889,0:02:50.489 +listen to our favorite music + +0:02:50.489,0:02:54.279 +and even play games + +0:02:54.279,0:02:56.839 +But how does the computer do all these things? + +0:02:56.839,0:02:59.609 +Let's start by taking text [xx] source code + +0:02:59.609,0:03:03.439 +and using the [xx] to translate it into binary +machine language + +0:03:03.439,0:03:05.119 +That’s the foundation + +0:03:05.119,0:03:06.310 +for the kernel + +0:03:06.310,0:03:06.910 +libraries + +0:03:06.910,0:03:09.100 +and userland application, otherwise known as + +0:03:09.100,0:03:14.609 +an operating system + +0:03:14.609,0:03:15.709 +like BSD. So you ask + +0:03:15.709,0:03:17.649 +What is a kernel? + +0:03:17.649,0:03:19.549 +It's a wonderful thing + +0:03:19.549,0:03:21.209 +[xx] for the management + +0:03:21.209,0:03:23.219 +or processes memory + +0:03:23.219,0:03:28.009 +and peripheral devices + +0:03:28.009,0:03:30.730 +and by extension, allows us to do [xx] stuff +like + +0:03:30.730,0:03:31.620 +networking + +0:03:31.620,0:03:33.319 +provide better security + +0:03:33.319,0:03:36.930 +work with disks and file systems, create user interfaces + +0:03:36.930,0:03:39.900 +interactive userland applications, allow us +to do things like + +0:03:39.900,0:03:43.249 +write documents, read emails, surf the Web, +watch movies + +0:03:43.249,0:03:44.639 +listen to music + +0:03:44.639,0:03:45.219 +play games + +0:03:45.219,0:03:51.879 +and much, much more + +0:03:51.879,0:03:54.619 +In summary + +0:03:54.619,0:03:58.359 +BSD is a UNIX-derived operating system that enables +users to harness the power of + +0:03:58.359,0:04:01.049 +the computer and process information better + +0:04:01.049,0:04:05.239 +It uses a combination of processes, memories, +and peripheral devices, and by extension + +0:04:05.239,0:04:09.379 +We can perform networking, enforce security, +read from and write to storage devices and interface visually to applications + +0:04:09.379,0:04:10.329 +like text editors + +0:04:10.329,0:04:17.329 +mail clients, Web browsers, multimedia players, and games + +0:04:18.239,0:04:19.560 +I'd like to look back + +0:04:19.560,0:04:22.350 +on the history of UNIX for a few minutes + +0:04:22.350,0:04:24.039 +Now, to be honest + +0:04:24.039,0:04:27.460 +Kirk McKusick just trumped everything I had +from my talk + +0:04:27.460,0:04:32.620 +But, this can be a repeat, but it's still pretty good(?) + +0:04:32.620,0:04:34.770 +So, in the beginning + +0:04:34.770,0:04:37.439 +And yes, there was life before UNIX + +0:04:37.439,0:04:39.559 +We got the Holy Trinity + +0:04:39.559,0:04:43.439 +MIT, Bell Labs and GE teamed to create +a system called Multics + +0:04:43.439,0:04:46.999 +the Multiplexed Information and Computing Service + +0:04:46.999,0:04:50.709 +We have a huge GE650 mainframe Multics + +0:04:50.709,0:04:56.550 +You can tell the engineers from the [xx] bosses + +0:04:56.550,0:04:58.800 +and so, it was a huge success + +0:04:58.800,0:05:01.399 +We don’t run Multics on a laptop, mainframes, of course + +0:05:01.399,0:05:04.959 +I should know, it was a commercial failure + +0:05:04.959,0:05:08.749 +[xx] is a computer scientist at Bell Labs +named Ken Thompson + +0:05:08.749,0:05:12.110 +He’d worked on the Multics project and was inspired +by interactive computing + +0:05:12.110,0:05:13.239 +It provided + +0:05:13.239,0:05:15.500 +Unfortunately, scrapping the Multics project + +0:05:15.500,0:05:19.240 +(but they no longer have a system that question +for serious work life) +[xx] known they had a system of which to perform +serious work like + +0:05:19.240,0:05:21.220 +Space Travel + +0:05:21.220,0:05:23.329 +a space simulation game + +0:05:23.329,0:05:25.319 +[xx] from Multics GE + +0:05:25.319,0:05:27.590 +650 mainframe + +0:05:27.590,0:05:29.260 +With Dennis Ritchie’s assistance + +0:05:29.260,0:05:30.960 +[xx] experience with the Multics project + +0:05:30.960,0:05:34.439 +they all craft(?) together an operating system +[xx] language + +0:05:34.439,0:05:36.569 +[xx] PDP-7 + +0:05:36.569,0:05:40.059 +This is capable of interactive +computing with the terminal + +0:05:40.059,0:05:43.979 +rather than just being punched +keycards + +0:05:43.979,0:05:45.909 +They termed that the UNIX systems + +0:05:45.909,0:05:49.630 +UNIX system, short for Uniplexed Information +and Computing System + +0:05:49.630,0:05:50.609 +a play + +0:05:50.609,0:05:53.409 +on the Multics name + +0:05:53.409,0:05:59.169 +it's [xx] a number of users + +0:05:59.169,0:06:06.050 +Did I miss something? + +0:06:06.050,0:06:13.050 +it's for a number of users + +0:06:15.020,0:06:19.879 +talk about anticlimactic files + +0:06:19.879,0:06:21.120 +By 1970 + +0:06:21.120,0:06:23.680 +it officially became known as U-N-I-X + +0:06:23.680,0:06:27.800 +probably [xx] + +0:06:27.800,0:06:31.020 +it would be really good [xx] + +0:06:31.020,0:06:34.409 +By 1971, the UNIX System was officially +put into production use + +0:06:34.409,0:06:36.330 +and imported to the PDP-11 + +0:06:36.330,0:06:37.989 +It was capable of text processing + +0:06:37.989,0:06:42.240 +for the purpose of filing patents + +0:06:42.240,0:06:44.789 +I was trying to visualize software patents + +0:06:44.789,0:06:51.789 +and tack it [xx] and probably the closest thing to [xx] + +0:06:52.729,0:06:56.669 +By 1973 + +0:06:56.669,0:06:59.929 +they rewrote UNIX + +0:06:59.929,0:07:01.950 +in a programming language created by Dennis Ritchie + +0:07:01.950,0:07:04.569 +The C programming language evolved from the B language + +0:07:04.569,0:07:07.759 +adding data type [xx] structures + +0:07:07.759,0:07:10.680 +Based in a 1958 antitrust +Case, AT&T had been + +0:07:10.680,0:07:12.520 +forbidden to enter the computer business + +0:07:12.520,0:07:14.799 +UNIX could not be turned into a product + +0:07:14.799,0:07:16.529 +so they shift [xx] + +0:07:16.529,0:07:20.489 +and just pass the source code to anyone who ask + +0:07:20.489,0:07:22.789 +Since the source code is readily available + +0:07:22.789,0:07:25.490 +and ran in a portable computer language + +0:07:25.490,0:07:29.349 +universities and research labs worldwide were able to run +UNIX on their own systems + +0:07:29.349,0:07:31.049 +By 1974 + +0:07:31.049,0:07:32.740 +Professor Bob Fabry + +0:07:32.740,0:07:34.750 +at the University of Cal Berkeley + +0:07:34.750,0:07:36.859 +purchased a copy of UNIX for $99 + +0:07:36.859,0:07:38.990 +for their own PDP-11 + +0:07:38.990,0:07:40.659 +By 1977 + +0:07:40.659,0:07:43.579 +Bill Jolitz, a graduate student of Cal Berkeley) + +0:07:43.579,0:07:46.330 +distributed(?) the Berkeley Software Distribution + +0:07:46.330,0:07:47.259 +otherwise known as + +0:07:47.259,0:07:49.029 +1BSD + +0:07:49.029,0:07:50.759 +It included a Pascal compiler + +0:07:50.759,0:07:52.460 +the ex Editor + +0:07:52.460,0:07:53.619 +vi ex Editor + +0:07:53.619,0:07:54.900 +By 1978 + +0:07:54.900,0:07:57.159 +2BSD had been released + +0:07:57.159,0:07:59.139 +which added the vi editor + +0:07:59.139,0:08:03.149 +and the C shell + +0:08:03.149,0:08:05.459 +I'm definitely not old school [xx] editor + +0:08:05.459,0:08:08.419 +[xx] for the win + +0:08:08.419,0:08:09.789 +1979 + +0:08:09.789,0:08:12.110 +3BSD was released + +0:08:12.110,0:08:13.879 +a support for the VAX platform + +0:08:13.879,0:08:16.599 +it was coined as Virtual VAX for VMUNIX + +0:08:16.599,0:08:20.349 +based on the new virtual memory implementation +specifically for the VAX,UNIX/32V + +0:08:20.349,0:08:23.059 +Computer Berkeley + +0:08:23.059,0:08:24.830 +Based on the success of 3BSD + +0:08:24.830,0:08:28.769 +the Defense Advanced Research Project Agency (DARPA) + +0:08:28.769,0:08:33.080 +[xx] Berkeley CSRG with the contract enhanced UNIX + +0:08:33.080,0:08:35.810 +for VLSI the project + +0:08:35.810,0:08:37.550 +these enhancements were + +0:08:37.550,0:08:38.570 +eventually released + +0:08:38.570,0:08:40.080 +as 4BSD + +0:08:40.080,0:08:42.300 +which [xx] job control for the C shell + +0:08:42.300,0:08:45.580 +delivermail, the predecessor to sendmail + +0:08:45.580,0:08:48.140 +the curses [xx] + +0:08:48.140,0:08:51.240 +and reliable signals + +0:08:51.240,0:08:55.960 +Unfortunately, 4BSD was criticized for bad +performance on the VAX platform + +0:08:55.960,0:08:59.950 +so [xx] went back to work [xx] kernel +to outperform [xx] VAX + +0:08:59.950,0:09:02.900 +eventually releasing 4.1BSD for these performance + +0:09:02.900,0:09:04.030 +fixes + +0:09:04.030,0:09:05.410 +Two years later + +0:09:05.410,0:09:07.940 +4.2BSD was released + +0:09:07.940,0:09:10.650 +incorporating a TCP/IP stack BBN Technologies + +0:09:10.650,0:09:13.360 +also, a contractor for DARPA + +0:09:13.360,0:09:17.090 +as well as the Berkeley Fast File System + +0:09:17.090,0:09:21.510 +written by a dapper young man by the name of Kirk McKusick + +0:09:21.510,0:09:24.500 +He's also kind enough to give us the original BSD mascot + +0:09:24.500,0:09:25.940 +In 1986 + +0:09:25.940,0:09:28.410 +4.3BSD was unleashed + +0:09:28.410,0:09:31.300 +incorporating numerous performance improvements + +0:09:31.300,0:09:34.870 +including a non-BBN version of the BSD TCP/IP stack, +which is found to be superior by + +0:09:34.870,0:09:36.370 +DARPA + +0:09:36.370,0:09:37.290 +1998 + +0:09:37.290,0:09:40.030 +the next version of BSD was released + +0:09:40.030,0:09:45.460 +4.3BSD-Tahoe, so named for the short-lived +Power 6/32 “Tahoe” platform + +0:09:45.460,0:09:49.960 +This was an initial attempt of moving away +from the VAX platform + +0:09:49.960,0:09:53.800 +Although the Tahoe was unsuccessful, it helped introduce machine +in a [xx] that will improve + +0:09:53.800,0:10:00.720 +[xx] BSD future credibility + +0:10:00.720,0:10:04.670 +Up to this point, all versions of BSD +included proprietary AT&T code + +0:10:04.670,0:10:07.550 +[xx] require license for AT&T for their use + +0:10:07.550,0:10:10.190 +These licenses have become prohibitively expensive + +0:10:10.190,0:10:13.260 +[xx] the demand for AT&T free version of UNIX + +0:10:13.260,0:10:15.130 +This led to the release of Net/1 + +0:10:15.130,0:10:17.740 +a BSD license distribution of the FreeBSD + +0:10:17.740,0:10:18.440 +The free + +0:10:18.440,0:10:20.230 +FreeBSD networking code + +0:10:20.230,0:10:23.340 +and uncovered by any AT&T [xx] licenses + +0:10:23.340,0:10:24.319 +The next year + +0:10:24.319,0:10:25.810 +4.3BSD-Reno + +0:10:25.810,0:10:26.930 +was released + +0:10:26.930,0:10:28.610 +The name suggesting + +0:10:28.610,0:10:32.360 +a gamble [xx] production use + +0:10:32.360,0:10:35.560 +regardless, it included the MACH virtual memory +system + +0:10:35.560,0:10:37.030 +Sun-compatible NFS + +0:10:37.030,0:10:40.280 +and continued the movement towards a POSIX compliance + +0:10:40.280,0:10:44.980 +It became apparent that the AT&T code was a +Hassle [xx] CSRG + +0:10:44.980,0:10:48.060 +virtually all of the utilities and code that was +stolen from AT&T + +0:10:48.060,0:10:49.720 +Then, in the middle of 1991 + +0:10:49.720,0:10:51.459 +Net/2 was released + +0:10:51.459,0:10:56.480 +A nearly complete BSD UNIX system that was freely +distributable under the BSD license + +0:10:56.480,0:11:00.320 +Net/2 was the basis for two separate parts +of BSD to the Intel + +0:11:00.320,0:11:02.100 +8386 architecture + +0:11:02.100,0:11:05.670 +Bill Jolitz started the 386BSD +distribution + +0:11:05.670,0:11:06.590 +[xx] + +0:11:06.590,0:11:11.000 +became the basis for 3BSD and NetBSD +projects + +0:11:11.000,0:11:13.430 +Berkeley Software Design BSDI + +0:11:13.430,0:11:15.199 +came out with BSD/386 + +0:11:15.199,0:11:22.199 +A proprietary version later renamed as BSD +OS + +0:11:22.560,0:11:23.520 +In 1992 + +0:11:23.520,0:11:25.970 +a wholly-owned subsidiary of AT&T + +0:11:25.970,0:11:27.800 +UNIX System Laboratories + +0:11:27.800,0:11:30.310 +also known as USL + +0:11:30.310,0:11:34.860 +filed a suit against BSDI claiming that AT&T’s +proprietary code which included + +0:11:34.860,0:11:37.420 +the BSDI UNIX products + +0:11:37.420,0:11:41.640 +The injunction was filed asking for BSDI +to discontinue their use of advertising + +0:11:41.640,0:11:43.380 +[xx] use of the UNIX name + +0:11:43.380,0:11:45.780 +own by AT&T and supposedly aimed + +0:11:45.780,0:11:47.510 +to confuse unsuspecting + +0:11:47.510,0:11:47.880 +users + +0:11:47.880,0:11:49.860 +for virus + +0:11:49.860,0:11:53.620 +Also for the fact that the BSD OS sold +for $995 + +0:11:53.620,0:11:57.090 +a 99% discount of AT&T UNIX + +0:11:57.090,0:12:00.400 +which [xx] out for $100 to +$2,000 + +0:12:00.400,0:12:07.140 +You can see why the average consumer might +get confused + +0:12:07.140,0:12:10.720 +After extensive analysis, it was +determined that the BSD OS + +0:12:10.720,0:12:14.960 +was little more than Net/2, probably six files from Bill +Jolitz’s 386BSD + +0:12:14.960,0:12:17.000 +The judge presiding over the case + +0:12:17.000,0:12:18.430 +denied the injunction + +0:12:18.430,0:12:20.750 +forcing USL [xx] complaint + +0:12:20.750,0:12:22.280 +to recent copyrights + +0:12:22.280,0:12:24.920 +and a possibility of the loss of trade secrets + +0:12:24.920,0:12:28.580 +The judge also recommended that the case be +heard at a state court before filing in a federal + +0:12:28.580,0:12:29.450 +court + +0:12:29.450,0:12:31.460 +So, taking this hint from the judge + +0:12:31.460,0:12:33.430 +they ran out to California + +0:12:33.430,0:12:34.110 +refiled + +0:12:34.110,0:12:38.260 +as quickly as possible the University of Cal Berkeley + +0:12:38.260,0:12:40.800 +take action to prevent + +0:12:40.800,0:12:44.260 +resulted if USL wanted to take any action against + +0:12:44.260,0:12:45.940 +[xx] will be forced to do so in California + +0:12:45.940,0:12:49.860 +rather than their home state of New Jersey + +0:12:49.860,0:12:51.720 +Soon after the filing in a state court + +0:12:51.720,0:12:55.380 +USL has bought(?) from AT&T by Novell + +0:12:55.380,0:12:58.760 +By 1994, a settlement had been reached in private + +0:12:58.760,0:13:01.100 +[xx] details of the settlement were not known + +0:13:01.100,0:13:03.920 +until 2004 when a California +public records law + +0:13:03.920,0:13:06.570 +allow the details to be released + +0:13:06.570,0:13:10.070 +[xx] published on the Groklaw site + +0:13:10.070,0:13:11.810 +Details of the settlement included + +0:13:11.810,0:13:13.870 +I think we've covered this on + +0:13:13.870,0:13:16.640 +Chri’s talk + +0:13:16.640,0:13:17.580 +there were those +[xx] + +0:13:17.580,0:13:23.740 +In hindsight, this confirmed our suspicions +Of USL’s fate + +0:13:23.740,0:13:25.800 +In June of 1994 + +0:13:25.800,0:13:27.930 +the 2BSD distributions were released + +0:13:27.930,0:13:29.509 +4.4BSD-Encumbered + +0:13:29.509,0:13:31.650 +was a version of BSD with AT&T code + +0:13:31.650,0:13:34.800 +It was only available to AT&T licensees + +0:13:34.800,0:13:38.510 +4.4BSD-Lite was also released which +now contained AT&T code + +0:13:38.510,0:13:42.120 +They become a new basis for the [xx] for the +previously [xx] + +0:13:42.120,0:13:46.890 +Trees +[xx] + +0:13:46.890,0:13:49.430 +As I mentioned. 4.4BSD-Lite + +0:13:49.430,0:13:53.530 +became the new baseline from which all the BSD +distributions will be based + +0:13:53.530,0:13:56.150 +FreeBSD, one of the first + +0:13:56.150,0:14:03.110 +and probably the most popular of the modern BSD +distributions + +0:14:03.110,0:14:07.130 +has a story [xx] focused on the Intel +8386 platform + +0:14:07.130,0:14:12.370 +and [xx] Windows markets + +0:14:12.370,0:14:13.570 +Nevertheless + +0:14:13.570,0:14:18.020 +FreeBSD has been [xx] platforms including +Intel x86, Itanium + + +0:14:18.020,0:14:18.680 +AMD64 + +0:14:18.680,0:14:20.360 +DEC Alpha + +0:14:20.360,0:14:23.130 +PowerPC, and Sun UltraSPARC + +0:14:23.130,0:14:27.420 +So, the more popular features include +application jails, access controls + +0:14:27.420,0:14:30.340 +excellent networking, SMP performance + +0:14:30.340,0:14:35.330 +and more recently, [xx] Solaris ZFS and dtrace + +0:14:35.330,0:14:36.190 +NetBSD + +0:14:36.190,0:14:40.200 +which was also richly derived from Jolitz’s [xx] +BSD + +0:14:40.200,0:14:44.710 +is typically known for portability through a wide range of +[xx] platforms + +0:14:44.710,0:14:51.060 +NetBSD has been [xx] to over 50 hardware platforms + +0:14:51.060,0:14:58.060 +and kitchen appliances + +0:15:00.110,0:15:04.020 +OpenBSD + +0:15:04.020,0:15:08.070 +no bias at all + +0:15:08.070,0:15:10.840 +OpenBSD is yet another modern BSD derivative + +0:15:10.840,0:15:13.140 +originally [xx] from NetBSD 1.0 + +0:15:13.140,0:15:14.769 +for being highly secure + +0:15:14.769,0:15:16.910 +with an emphasis on code correctness + +0:15:16.910,0:15:21.160 +proper documentation and truly open in free source + +0:15:21.160,0:15:23.600 +the mantra is secure by default + +0:15:23.600,0:15:28.220 +This philosophy is influenced in countless other free +proprietary operating systems which now + +0:15:28.220,0:15:29.950 +the example + +0:15:29.950,0:15:34.220 +Here are some of the more popular platforms +[xx] BSD has been imported to + +0:15:34.220,0:15:37.380 +And although you may see numerous security enhancements + +0:15:37.380,0:15:41.790 +to make efforts to integrate these changes into the [xx] system +[xx] transfer of technologies + +0:15:41.790,0:15:46.470 +The philosophy has been proven less effective +[xx] technology such as SELinux + +0:15:46.470,0:15:50.290 +[xx] is placed on the Systems Administrator + +0:15:50.290,0:15:53.590 +and is quite often disabled + +0:15:53.590,0:15:55.680 +Be generous + +0:15:55.680,0:15:57.710 +You see + +0:15:57.710,0:15:58.940 +You see + +0:15:58.940,0:16:05.940 +some of the features of OpenBSD + +0:16:08.460,0:16:09.710 +[xx] + +0:16:09.710,0:16:14.760 +[xx] + +0:16:14.760,0:16:20.720 +[xx] + +0:16:20.720,0:16:26.450 +They have consistent release schedule of new releases +available around the first of May and November + +0:16:26.450,0:16:27.510 +DragonFly + +0:16:27.510,0:16:31.850 +another BSD, started by Matt Dillon in 2003 + +0:16:31.850,0:16:34.500 +It's a logical continuation of FreeBSD +4.8 + +0:16:34.500,0:16:38.780 +He started the foot-dragging 5BSD project +when his vision for threading an SMP conflicted + +0:16:38.780,0:16:41.610 +(with the other developers working on FreeBSD +file + +0:16:41.610,0:16:45.540 +They continue work on SMP revamp +which, I think, is probably complete by now + +0:16:45.540,0:16:47.369 +as well as the other lightweight + +0:16:47.369,0:16:49.100 +kernel threads implementation + +0:16:49.100,0:16:53.550 +(well Martin balsam focused on spring jenner +jericho street supportive of the coral and) +[xx] focused on generating +[xx] support the kernel + +0:16:53.550,0:16:56.700 +and features like the [xx] + +0:16:56.700,0:16:58.599 +MAC OS X is an operating system + +0:16:58.599,0:16:59.459 +sold by Apple + +0:16:59.459,0:17:06.010 +which runs on both PowerPC and Intel platforms) + +0:17:06.010,0:17:08.210 +hecklers + +0:17:08.210,0:17:12.709 +BSD is the heart of MacOS X, +a full capable BSD UNIX derivative + +0:17:12.709,0:17:16.510 +the enhancement was [xx] to make OS X + +0:17:16.510,0:17:19.270 +MAC OS + +0:17:20.890,0:17:23.150 +a consumer-friendly operating system + +0:17:23.150,0:17:25.540 +Unfortunately, it's still somewhat of a hybrid + +0:17:25.540,0:17:30.320 +I + +0:17:30.320,0:17:32.630 +yeah + +0:17:32.630,0:17:34.530 +Merging the MAC kernel with NeXTSTEP + +0:17:34.530,0:17:41.350 +while it's very useful in +FreeBSD + +0:17:41.350,0:17:43.740 +It's about the BSD distributions including + +0:17:43.740,0:17:45.780 +364 units from DEC + +0:17:45.780,0:17:52.780 +then Compaq, then HP + +0:17:54.710,0:17:58.490 +Now, we've covered some of the history behind BSD. +I like to address the real shortcomings + +0:17:58.490,0:18:00.960 +of modern day BSD + +0:18:00.960,0:18:3.980 +So, why is BSD dying? + +0:18:03.980,0:18:05.890 +That's why we're here to uncover + +0:18:05.890,0:18:08.650 +First and foremost + +0:18:08.650,0:18:13.570 +because IDC says so + +0:18:13.570,0:18:15.290 +Market share is an all-time low + +0:18:15.290,0:18:17.830 +under 1% + +0:18:17.830,0:18:18.970 +And of course + +0:18:18.970,0:18:20.010 +Netcraft + +0:18:20.010,0:18:24.170 +confirms these findings + +0:18:24.170,0:18:26.309 +BSD came in last place + +0:18:26.309,0:18:28.240 +in a SysAdmin networking test + +0:18:28.240,0:18:35.240 +I don’t recall which of the BSDs were tested, +but it's probably safe to assume they all finished [xx] last + +0:18:37.570,0:18:41.190 +Market leaders and pundits have predicted that +open-source software can't make money + +0:18:41.190,0:18:45.320 +If we continue to give away free software, how +are we finance our developers? + +0:18:45.320,0:18:51.290 +[xx] the forecasting [xx] the spiraling into bankruptcy + +0:18:51.290,0:18:52.669 +I + +0:18:52.669,0:18:53.690 +yeah + +0:18:53.690,0:19:00.690 +It's a little dated, but man, you just got to love this guy + +0:19:00.700,0:19:05.470 +BSD + +0:19:05.470,0:19:06.520 +As a community + +0:19:06.520,0:19:10.500 +BSD has had a surprising inability to +adapt + +0:19:10.500,0:19:12.140 +As we can see by this graph + +0:19:12.140,0:19:14.930 +I + +0:19:14.930,0:19:19.630 +I + +0:19:19.630,0:19:24.940 +As we can see by this graph, the number of +pages served per hour + +0:19:24.940,0:19:27.030 +[xx] far outnumber those + +0:19:27.030,0:19:29.640 +on Linux and all the BSDs + +0:19:29.640,0:19:34.750 +Linux actually degenerates some [xx] we +have to presume because of [xx] + +0:19:34.750,0:19:40.520 +renaming their files ASP + +0:19:40.520,0:19:43.560 +Of course, a significant loss of talent + +0:19:43.560,0:19:46.310 +FreeBSD has lost 93% of their core +developers + +0:19:46.310,0:19:50.370 +This is certainly the [xx] to jump ship [xx] + +0:19:50.370,0:19:54.850 +Unfortunately, since none of the BSDs [xx] code, +they usually have to start from scratch all over again + +0:19:54.850,0:19:55.970 +Tracing the project’s roots + +0:19:55.970,0:19:59.880 +from 4.5BSD-lite + +0:19:59.880,0:20:04.890 +Fortunately, not all is lost + +0:20:04.890,0:20:11.890 +There's still a handful of very small companies still using +BSD today + +0:20:17.140,0:20:23.559 +And you probably haven’t heard most of these + +0:20:23.559,0:20:28.480 +This is my employer + +0:20:28.480,0:20:31.510 + +0:20:31.510,0:20:32.990 +Hopefully, the success + +0:20:32.990,0:20:37.230 +yes, the [xx] had told me +I had to put that on there + +0:20:37.230,0:20:44.230 +[xx] the success as seen [xx] seen in the industry’s +Will [xx] open in the mainstream + +0:20:45.050,0:20:47.230 +Looking forward, there's a number of challenges ahead + +0:20:47.230,0:20:50.960 +ahead of us to ensure the BSD survives +generations of future geeks + +0:20:50.960,0:20:53.540 +The challenges aren’t simply of the technological +nature + +0:20:53.540,0:20:57.750 +but included a number of political and legal +obstacles as well + +0:20:57.750,0:21:03.240 +First and foremost, virtualization is on everybody’s +radar + +0:21:03.240,0:21:04.400 +[xx] + +0:21:04.400,0:21:09.650 +[xx] to optimize our resource running BSD + +0:21:09.650,0:21:13.820 +All this means I can run over 100,000 [xx] of NetBSD +on a server four gigs of + +0:21:13.820,0:21:14.690 +memory + +0:21:14.690,0:21:18.520 +It also means I have to hire 1,000 +NetBSD systems administrators + +0:21:18.520,0:21:20.360 + +0:21:20.360,0:21:23.940 +Not only is this a miserable return of investment +I don’t think there are over a thousand NetBSD users out there + +0:21:23.940,0:21:30.280 + +0:21:30.280,0:21:33.350 +We're looking for more advanced file systems + +0:21:33.350,0:21:38.970 +to handle the current performance limits +associated with high capacity multi-tier + +0:21:38.970,0:21:43.850 +[xx] Solaris CFS have already been completed +[xx] previous BSD-7 and MAC) + +0:21:43.850,0:21:46.460 +OS X + +0:21:46.460,0:21:48.799 +They have blobs, NDA’s + +0:21:48.799,0:21:51.100 +Closed documentation can now go hand in hand + +0:21:51.100,0:21:55.550 +It's a clerical challenge that threatens +[xx] hardware availability to BSD developers + +0:21:55.550,0:21:56.809 +and endusers + +0:21:56.809,0:21:59.370 +Projects like Linux, and even FreeBSD + +0:21:59.370,0:22:00.830 +had casually signed + +0:22:00.830,0:22:04.770 +agreements and NDA’s to accept none free binary +drivers into the source stream + +0:22:04.770,0:22:07.320 +These arrangements worked against the spirit +of free software + +0:22:07.320,0:22:08.920 +and open-source + +0:22:08.920,0:22:10.940 +software, all for the purposes of short-term gains + +0:22:10.940,0:22:13.690 +by 3D [xx] and rotating desktops) + +0:22:13.690,0:22:18.510 +That’s coming the guys [xx] serving in the next room + +0:22:18.510,0:22:21.220 +Within the last year or so, legal challenges have surfaced + +0:22:21.220,0:22:24.720 +Linux developers have blatantly stolen BSD license code + +0:22:24.720,0:22:27.270 +replacing the license notification with GPL + +0:22:27.270,0:22:29.700 +and any of their copyrights when no work was done + +0:22:29.700,0:22:33.419 +Beyond the legal ramifications of these actions, +this demonstrate a lack of respect + +0:22:33.419,0:22:38.050 +in cooperation for their free +software peers + +0:22:38.050,0:22:39.649 +diversity in the BSD is a healthy trend + +0:22:39.649,0:22:41.620 +it promotes new features and competition + +0:22:41.620,0:22:43.220 +in the intellectual market + +0:22:43.220,0:22:47.169 +resulting in better systems for all of us + +0:22:47.169,0:22:49.740 +In the end, diversity can bring unity throughout our community + +0:22:49.740,0:22:50.519 +and with unity + +0:22:50.519,0:22:53.020 +a common goal + +0:22:53.020,0:22:54.290 +The end diff --git a/en_US.ISO8859-1/captions/2009/dcbsdcon/mckusick-historyofbsd.sbv b/en_US.ISO8859-1/captions/2009/dcbsdcon/mckusick-historyofbsd.sbv new file mode 100644 index 0000000000..8b93823c28 --- /dev/null +++ b/en_US.ISO8859-1/captions/2009/dcbsdcon/mckusick-historyofbsd.sbv @@ -0,0 +1,3908 @@ +0:00:00.570,0:00:03.510 +Anyways it is my great honour and privilege + +0:00:03.510,0:00:10.510 +to introduce Kirk McKusick uh! Which is +give a talk our history of BSD + +0:00:13.570,0:00:17.170 +So this is actually a three-hour lecture + +0:00:17.170,0:00:19.949 +I but I don't have three hours so there is + +0:00:19.949,0:00:23.659 +uh is this where I allow audience participation + +0:00:23.659,0:00:27.099 +The lecture is really sort of three +main parts + +0:00:27.099,0:00:30.460 +and what Ill do is + +0:00:30.460,0:00:34.210 +describe the three main parts. Then everyone +will get to vote on which part they find the + +0:00:34.210,0:00:35.490 +most interesting + +0:00:35.490,0:00:36.510 +Then what I.. I wont.. + +0:00:36.510,0:00:41.309 +I used to just do just that part and Ive got a +lot of negative feedback about that's all + +0:00:41.309,0:00:43.819 +So Ill do those two parts don't get elected in + +0:00:43.819,0:00:47.210 +I'll just do quickly and then I'll concentrate +on the the one + +0:00:47.210,0:00:48.010 +piece that + +0:00:48.010,0:00:49.720 +seems to be the most popular + +0:00:49.720,0:00:51.360 +so the history of BSD + +0:00:51.360,0:00:55.750 +This is BSD at Berkeley is supposed +to one to shout into that + +0:00:55.750,0:00:57.050 +open source world + +0:00:57.050,0:01:01.810 +So the first period of time is what I call +the Bill Joy era + +0:01:01.810,0:01:06.360 +And this is where Bill Joy is starting up BSD + and it's really getting a rolling and + +0:01:06.360,0:01:07.180 +so it ends when he + +0:01:07.180,0:01:08.370 +departs +0:01:08.370,0:01:09.610 +and +0:01:09.610,0:01:13.080 +Sponsers uh.. starts up this company +called Sun Microsystems which you may have + +0:01:13.080,0:01:14.680 +heard of + +0:01:14.680,0:01:19.250 +Uh.. Then the second part is one of the heyday +of BSD. So this is the period of time + +0:01:19.250,0:01:20.930 +where everybody is + +0:01:20.930,0:01:24.290 +running is forty two and for three and so on + +0:01:24.290,0:01:24.960 +and + +0:01:24.960,0:01:28.830 +going accurate time we were being funded +by DARPA which + +0:01:28.830,0:01:31.020 +deploy a microphone somewhere in this room + +0:01:31.020,0:01:32.950 +Uh at any rate + +0:01:32.950,0:01:35.160 +there was some very interesting + +0:01:35.160,0:01:36.889 +Uh.. battles that went on + +0:01:36.889,0:01:38.630 +Uh.. about + +0:01:38.630,0:01:41.110 +exactly how to see or to be implemented + +0:01:41.110,0:01:46.180 +and a lot of this at that history all +that all percolates out + +0:01:46.180,0:01:48.000 +and so that's sort of the middle period and + +0:01:48.000,0:01:54.000 +and then the end period is where we have +gotten religion and it decided that + +0:01:54.000,0:01:58.130 +you know maybe BSD isnt going be at +Berkeley forever and if we want there to be + +0:01:58.130,0:01:58.960 +any + +0:01:58.960,0:02:00.430 +possibility with the continue + +0:02:00.430,0:02:04.530 +Uh.. we better figure out how to get it outside +the university to people that + +0:02:04.530,0:02:08.489 +other than to the people that are licensed +by AT&T Bell Laboratories. + +0:02:08.489,0:02:09.970 +And so it's the whole story out + +0:02:09.970,0:02:13.859 +how we open-sourced it and fought a one lawsuit +about that and + +0:02:13.859,0:02:15.189 +so on and so forth. + +0:02:15.189,0:02:19.049 +So here is the three main parts.. The Bill Joy .. that the middle +PCP + +0:02:19.049,0:02:20.519 +building years + +0:02:20.519,0:02:20.940 +and + +0:02:20.940,0:02:24.099 +the lawsuit years at the end. + +0:02:24.099,0:02:28.879 +So how many people are most interested in hearing +the Bill joy years + +0:02:28.879,0:02:33.999 +one two three four five six seven eight nine +ten eleven twelve thirteen + +0:02:33.999,0:02:35.199 +fourteen fifteen + +0:02:35.199,0:02:39.559 +Sixteen. Ok.. how many people are interested +in hearing through the middle era + +0:02:39.559,0:02:44.579 +one two three four five six seven eight nine +ten eleven twelve thirteen fourteen fifteen + +0:02:44.579,0:02:45.320 +Sixteen.. All right.. + +0:02:45.320,0:02:46.959 +Seventeen eighteen ninety twenty twenty one + +0:02:46.959,0:02:49.959 +twenty five OK and how many people want to hear +the open + +0:02:49.959,0:02:51.079 +source + +0:02:51.079,0:02:53.159 +Lawsuits et cetera + +0:02:53.159,0:02:57.120 +Laugh.... Okkk + +0:02:57.120,0:02:58.969 +Do you see this is why I ask + +0:02:58.969,0:03:01.139 +This is usual it's interesting 'cause I get + +0:03:01.139,0:03:03.499 +different different groups votes for different +things + +0:03:03.499,0:03:06.799 +When I was in Japan they all want to hear +about the lawsuit.. you know.. thats all they + +0:03:06.799,0:03:09.229 +have those in Japan. + +0:03:09.229,0:03:11.669 +Yeah that story is quite well known here + +0:03:11.669,0:03:14.639 +Infact the so interesting tidbit is that + +0:03:14.639,0:03:16.459 +uh threw up + +0:03:16.459,0:03:20.289 +Freedom of Information Act California Freedom +of Information Act + +0:03:20.289,0:03:23.489 +uh we were able to get the uh + +0:03:23.489,0:03:27.819 +decision on that lawsuit unsealed so that +the actual text could get out which was + +0:03:27.819,0:03:30.319 +sort of interesting reading. + +0:03:30.319,0:03:34.349 +Okay, for those of you who are interested in +a section that I'm not doing + +0:03:34.349,0:03:38.050 +I advice to buy them to make a shameless plug for +my DVD. + +0:03:38.050,0:03:41.709 +This is the entire three-and-a-half hour version +of this lecture + +0:03:41.709,0:03:45.949 +I wish I could conveniently have a box up your surf +if it's worth twenty dollars to you could + +0:03:45.949,0:03:52.219 +got one of these and walk away with it. + +0:03:52.219,0:03:57.119 +Ok. So uh unlike most of my talks and I dont actually have a +laptop with slides on it and I just have these + +0:03:57.119,0:03:59.529 +handwritten notes + +0:03:59.529,0:04:01.680 +and the top line of my handwritten note now it says + +0:04:01.680,0:04:06.190 +the first four pages of this were written +on the Indian Pacific train on way to Greater Perth + +0:04:06.190,0:04:12.629 +in January 1986 because I had +to give a lecture on the day I arrived. + +0:04:12.629,0:04:14.219 +So I have been uh told + +0:04:14.219,0:04:18.100 + that this was what my talk was going +to be and I was unprepared so + +0:04:18.100,0:04:23.259 +it's what a bouncy up and down the handwriting +but it was the early history + +0:04:23.259,0:04:24.840 +so anyway + +0:04:24.840,0:04:25.880 +we're just gonna zip + +0:04:25.880,0:04:27.029 +quickly through + +0:04:27.029,0:04:29.750 +what BSD started at Berkeley + +0:04:29.750,0:04:31.150 +Uh by Bill Joy + +0:04:31.150,0:04:34.120 +uh Bill Joy was a graduate student I +happen to + +0:04:34.120,0:04:38.050 +share the same adviser with him, we were both +working on programming languages. + +0:04:38.050,0:04:41.610 +Uh you might find that hard to believe considering +that he did the seashell but + +0:04:41.610,0:04:44.539 +Uh.. laugh +0:04:44.539,0:04:48.000 +at any rate up that's what we're doing and + +0:04:48.000,0:04:50.740 +So I shared an office with him and along with + +0:04:50.740,0:04:53.720 +three other graduate students uh + +0:04:53.720,0:04:56.120 +and + +0:04:56.120,0:05:00.060 +So uh I sort of was sitting there watching a lot of +this unfolding and in fact Bill was very good + +0:05:00.060,0:05:03.080 +at getting other people enthusiastic about +doing work. + +0:05:03.080,0:05:06.849 +There was his typical comment was.. how hard +can it be, you could just knock this off in + +0:05:06.849,0:05:08.869 +a couple hours + +0:05:08.869,0:05:12.709 +and usually you could knock off one thing +but of course that with can you sucked + +0:05:12.709,0:05:16.060 +in and you know you end up writing the entire pascal +interpreter of + +0:05:16.060,0:05:18.039 +the back and or something + +0:05:18.039,0:05:20.249 +At any rate + +0:05:20.249,0:05:24.749 +Bill started doing initially just utilities +so was the seashell + +0:05:24.749,0:05:27.439 +uh that ex editor later + +0:05:27.439,0:05:29.750 +it also became the vi editor + +0:05:29.750,0:05:31.490 +the pascal system + +0:05:31.490,0:05:37.080 +and say start packaging these things out and +sending out tapes to people and + +0:05:37.080,0:05:38.050 +then when + +0:05:38.050,0:05:39.930 +Ken Thompson who would + +0:05:39.930,0:05:42.099 +in an Alumnus of Berkeley + +0:05:42.099,0:05:45.610 +on without for one year's sabbatical and brought +UNIX with him + +0:05:45.610,0:05:49.909 +when he left Bill sort of took over running +the system what we would call assistant administrator + +0:05:49.909,0:05:51.729 +today + +0:05:51.729,0:05:55.660 +Uh at any rate this got him interested in this the system +There were patches that were coming back + +0:05:55.660,0:05:58.550 +from AT&T he was putting those in and he was then + +0:05:58.550,0:06:01.590 +making some other sending them back to AT&T +and he + +0:06:01.590,0:06:01.930 +and + +0:06:01.930,0:06:03.729 + so + +0:06:03.729,0:06:07.770 +over time there were bigger and bigger bits of +the system and finally got tired of trying + +0:06:07.770,0:06:08.370 +to + +0:06:08.370,0:06:10.909 +track what other people's UNIX is look +like + +0:06:10.909,0:06:11.729 +and so + +0:06:11.729,0:06:16.569 +uh with the advance of 3 BSD the +third distribution + +0:06:16.569,0:06:18.560 +He just put together an entire package + +0:06:18.560,0:06:22.310 +uh and this was actually to the VAX rather +than the PDP-11 which is what we've started + +0:06:22.310,0:06:23.080 +on + +0:06:23.080,0:06:25.069 +all the two dot + +0:06:25.069,0:06:27.310 +versions of the of BSD + +0:06:27.310,0:06:30.889 +nicely worked on by the other people not by +the Bill especially not the stuff where + +0:06:30.889,0:06:33.050 +the kernel begins to become involved + +0:06:33.050,0:06:35.300 +2.9, 10, 11 et cetera + +0:06:35.300,0:06:37.569 +uh was actually done by + +0:06:37.569,0:06:41.090 +other people many years uh later so 2.11 + +0:06:41.090,0:06:43.279 +does not preceed 3.0 + +0:06:43.279,0:06:48.550 +Uh 2.11 is you know contemporary +with 4.4 + +0:06:48.550,0:06:51.100 +Uh at any rate uh + +0:06:51.100,0:06:52.800 +once we got the + +0:06:52.800,0:06:54.790 +uh the BSD + +0:06:54.790,0:06:57.360 +distribution out into the world + +0:06:57.360,0:07:00.870 +Uh it started getting used by a lot of people +because it + +0:07:00.870,0:07:01.589 +really + +0:07:01.589,0:07:05.000 +was a big step forward over what AT&T were +shipping + +0:07:05.000,0:07:06.660 +in particular + +0:07:06.660,0:07:09.529 +uh as we start getting things like + +0:07:09.529,0:07:11.450 +the TCP/IP networking + +0:07:11.450,0:07:15.770 +Uh So you had a choice you can run a AT& T +version which had UUCP as networking + +0:07:15.770,0:07:19.149 +or you could run BSD that had sockets et + cetera + +0:07:19.149,0:07:20.610 +you could run + +0:07:20.610,0:07:24.090 +AT&T's version which had a file system that +could utilize 3 to 5 percent of the bandwidth + +0:07:24.090,0:07:24.979 +of the disk + +0:07:24.979,0:07:28.810 +or you could run BSD which could utilize 50 percent +of the bandwidth of the disk + +0:07:28.810,0:07:30.519 +et cetera. + +0:07:30.519,0:07:34.939 +So many people would buy the AT&T license + and immediately drop Berkeley to add UNIX + +0:07:34.939,0:07:37.180 +on to it. + +0:07:37.180,0:07:41.879 +Uh consequently it was getting wide distribution +atleast in the university environments + +0:07:41.879,0:07:44.139 +Uh + +0:07:44.139,0:07:46.979 +And uh starting this way he got into the commercial +environments + +0:07:46.979,0:07:51.249 +The commercial environments were not large because +commercial environments just werent using UNIX + +0:07:51.249,0:07:52.679 +at that time + +0:07:52.679,0:07:54.679 +for the most part + +0:07:54.679,0:07:58.090 +Uh at any rate + +0:07:58.090,0:07:59.290 +Bill got it + +0:07:59.290,0:08:00.349 +Uh well + +0:08:00.349,0:08:04.299 +with 3 BSD coming out uh + +0:08:04.299,0:08:06.340 +The DARPA which is + +0:08:06.340,0:08:07.540 +essentially + +0:08:07.540,0:08:10.160 +put out research contracts for + +0:08:10.160,0:08:12.629 +all the different branches of the military + +0:08:12.629,0:08:13.780 +Uh was + +0:08:13.780,0:08:15.960 +had a huge number of contracts + +0:08:15.960,0:08:19.850 +uh with lots of different people being stuff +related computers all have different computers + +0:08:19.850,0:08:22.939 +and operating systems and maybe using some different +languages + +0:08:22.939,0:08:24.969 +and so if one group would do something + +0:08:24.969,0:08:27.309 +they just couldn't really pass that + +0:08:27.309,0:08:30.739 +over take some other group that might want +to utilize it because this was written in + +0:08:30.739,0:08:33.390 +FORTRAN and that was written in Pearl 1 + +0:08:33.390,0:08:35.830 +You know this was running on + +0:08:35.830,0:08:40.830 +you know some deck machine that was running on some +IBM's hardware et cetera. + +0:08:40.830,0:08:47.220 +and so DARPA decided that they +really wanted to consolidate + +0:08:47.220,0:08:50.280 +on hardware and operating system + +0:08:50.280,0:08:55.000 +and then how people work there and so there's a +better chance if they would be able to co-operate. + +0:08:55.000,0:08:57.640 +So they put out a a sort of + +0:08:57.640,0:08:59.829 +call sends a request for + +0:08:59.829,0:09:02.430 +proposals or whatever they called in those +days + +0:09:02.430,0:09:04.290 +and + +0:09:04.290,0:09:05.520 +essentially there were two + +0:09:05.520,0:09:08.500 +uh main contending things that came in + +0:09:08.500,0:09:10.029 +one was the VAX + +0:09:10.029,0:09:11.320 +running VMX + +0:09:11.320,0:09:14.300 +and the other one was the VAX running + +0:09:14.300,0:09:16.800 +Berkeley UNIX + +0:09:16.800,0:09:18.150 +and of course + +0:09:18.150,0:09:21.650 +the uh the people that were promoting the +VAX running VMS + +0:09:21.650,0:09:27.220 +were arguing like uh you know this is a real +vendor supported operating system and + +0:09:27.220,0:09:32.059 +you know that this not just some flaky students +at Berkeley that are doing it + +0:09:32.059,0:09:34.530 +then you know therefore there's sort of the operating +system + +0:09:34.530,0:09:39.500 +And that really put the final nail +on the coffin + +0:09:39.500,0:09:43.410 +Uh this time they encased and came up with a set of +benchmarks + +0:09:43.410,0:09:47.370 +Uh what we would call micro benchmarks +today so how fast can you do get a PID + +0:09:47.370,0:09:49.970 +and how fast you could pipe thing + +0:09:49.970,0:09:54.180 +When you pack it back and forth to see +up that certain context which is so on. + +0:09:54.180,0:09:55.410 +and + +0:09:55.410,0:09:56.710 +uh + +0:09:56.710,0:10:00.910 +showed that we must have did these things much faster +than BSD did. + +0:10:00.910,0:10:01.760 +and this of course + +0:10:01.760,0:10:03.680 +made Bill go ballistic + +0:10:03.680,0:10:04.360 +and + +0:10:04.360,0:10:07.780 +Uh besides saying that they were as stupid as benchmarks + he'd ever since have zipped that's the way + +0:10:07.780,0:10:11.070 +they're gonna decide then we'll just go +make this thing run fast + +0:10:11.070,0:10:14.920 +and it was sort of an interesting story about +other half and what + +0:10:14.920,0:10:19.980 +he got uh the BSD to run those benchmarks as fast +as the VMS + +0:10:19.980,0:10:21.450 +and + +0:10:21.450,0:10:22.170 +so + +0:10:22.170,0:10:24.900 +the upshot of it was that DARPA + +0:10:24.900,0:10:28.740 +was still sort of sitting on the fence +and Bills comment was well VMS will stock + +0:10:28.740,0:10:30.380 +with VMX forever + +0:10:30.380,0:10:34.490 +and you know may be some other computer might +be interesting down the road + +0:10:34.490,0:10:40.700 +and BSD is set up so that it's much more portable +uh you know it's tie. UNIX, in general is + +0:10:40.700,0:10:44.750 +the portable across different architectures we've +already proved it by running it for any cost too + +0:10:44.750,0:10:45.910 +so therefore + +0:10:45.910,0:10:48.010 +it's proved point + +0:10:48.010,0:10:49.040 +so + +0:10:49.040,0:10:51.850 +at any rate DARPA agreed to that + +0:10:51.850,0:10:54.780 +and they put in some initial money to Berkeley + +0:10:54.780,0:10:57.230 +and the initial things that Berkley was +really just + +0:10:57.230,0:11:00.720 +uh to come up with + +0:11:00.720,0:11:02.500 +Uh well we've put out 3 BSD + +0:11:02.500,0:11:05.950 +which had a lot of functionality but the +performance wasn't so great + +0:11:05.950,0:11:09.330 +and so the that first point was to get +4 BSD out + +0:11:09.330,0:11:10.790 +which was going to happen + +0:11:10.790,0:11:12.870 +sort of that + +0:11:12.870,0:11:14.960 +that additional functionality they really felt +they needed + +0:11:14.960,0:11:16.459 +uh which would just is of the + +0:11:16.459,0:11:17.930 +really small tweeks + +0:11:17.930,0:11:22.510 +Uh things like the ability to automatically +reboot after crash + +0:11:22.510,0:11:23.880 +Uh Job control + +0:11:23.880,0:11:29.540 +uh PRAMs list which was created by a bunch +of their uh folks, deliver mail, smoke in air + +0:11:29.540,0:11:32.810 + that was the uh the predecessor to +send mail + +0:11:32.810,0:11:36.820 +except without a configuration file you just +complied and check the way he needs things + +0:11:36.820,0:11:37.700 +that were + +0:11:37.700,0:11:39.720 +uh at any rate + +0:11:39.720,0:11:41.260 +Uh that came out + +0:11:41.260,0:11:42.800 +and that 4.1 + +0:11:42.800,0:11:44.550 +so it nail that down cleaned it up + +0:11:44.550,0:11:46.590 +got the performance decent + +0:11:46.590,0:11:49.680 +and that fulfilled the first DARPA contract. + +0:11:49.680,0:11:52.450 +DARPA had put in something like + +0:11:52.450,0:11:56.390 +you know three-quarters of a million dollars or something +and boomed topped this huge pay back from it + +0:11:56.390,0:11:56.960 +and + +0:11:56.960,0:11:58.380 +people seem to be happy with it + +0:11:58.380,0:12:02.880 +so they decided that maybe that the funky graduate students +at Berkeley could do stuff + +0:12:02.880,0:12:05.640 +and so then they put in that of + +0:12:05.640,0:12:07.310 +considerably larger + +0:12:07.310,0:12:10.710 +contract it was I don't know what three or +five million dollars + +0:12:10.710,0:12:12.260 +on this was a longer term + +0:12:12.260,0:12:13.520 +he knows he wouldve at + +0:12:13.520,0:12:15.870 + those times he would get + +0:12:15.870,0:12:17.839 +for example a three-year contract + +0:12:17.839,0:12:20.320 +of which is sort of unheard of in lot of a + +0:12:20.320,0:12:22.440 +research stuff today. + +0:12:22.440,0:12:25.630 +So you could really sort of spent some time +figuring out what you can do and do it + +0:12:25.630,0:12:28.789 +for eighteen months to two years before you had +to start working about + +0:12:28.789,0:12:31.030 +the new assignment et cetera. + +0:12:31.030,0:12:33.180 +Uh in any way + +0:12:33.180,0:12:34.930 +the grand goals for this + +0:12:34.930,0:12:37.300 +Big hunt that DARPA gave + +0:12:37.300,0:12:38.800 +was + +0:12:38.800,0:12:41.470 +Uh there were four pieces. One was + +0:12:41.470,0:12:42.779 +to get a + +0:12:42.779,0:12:43.880 +real networking end + +0:12:43.880,0:12:46.220 +uh at that time + +0:12:46.220,0:12:47.980 +Uh there was just NCP + +0:12:47.980,0:12:49.680 +Uh which was that + +0:12:49.680,0:12:54.240 +Network Control Protocol predecessor to TCP/IP + +0:12:54.240,0:12:57.100 +Uh a good file system in their + +0:12:57.100,0:13:01.490 +Uh restart able signal so you could actually +use them to drive work as opposed to just + +0:13:01.490,0:13:03.440 +terminating programs + +0:13:03.440,0:13:05.600 +and a new virtual memory system + +0:13:05.600,0:13:07.390 +Uh that wasn't + +0:13:07.390,0:13:09.810 +well essentially that allowed shared memory + +0:13:09.810,0:13:11.530 +uh that the one + +0:13:11.530,0:13:13.640 +the predecessor + +0:13:13.640,0:13:15.880 +that we were running BSD at that +time + +0:13:15.880,0:13:19.810 +shared the text space that would be only the part +of your program but all the rest + +0:13:19.810,0:13:21.580 +could not shared + +0:13:21.580,0:13:23.810 +So therere the four deliverables. + +0:13:23.810,0:13:25.980 +now for the kind of networking + +0:13:25.980,0:13:28.000 +the students really trust us. + +0:13:28.000,0:13:30.879 +So they decided that what they were going +to do + +0:13:30.879,0:13:33.720 +was divide into two parts. + +0:13:33.720,0:13:35.300 +So they were going to get + +0:13:35.300,0:13:37.630 + Bolt, Beranek and Newman BBN + +0:13:37.630,0:13:42.360 +we are going to actually write the TCP/IP protocols +stuff + +0:13:42.360,0:13:44.709 +and What Berkley was supposed to do + +0:13:44.709,0:13:46.589 +was designed the + +0:13:46.589,0:13:47.550 +programming interface + +0:13:47.550,0:13:49.040 +uh so + +0:13:49.040,0:13:53.130 +the socket connect except so on + +0:13:53.130,0:13:55.639 +and then VB9 was to deliver + +0:13:55.639,0:14:00.159 +the thing plug-in underneath that interface +and Berkley was to integrated into + +0:14:00.159,0:14:02.150 +BSD and uh + +0:14:02.150,0:14:04.570 +they would ship it. + +0:14:04.570,0:14:05.239 +and + +0:14:05.239,0:14:08.090 +Uh so + +0:14:08.090,0:14:11.240 +this all started on and VB9 gets to work + +0:14:11.240,0:14:13.090 +and + +0:14:13.090,0:14:15.750 +Joy gets to work on the interface and + +0:14:15.750,0:14:18.620 +uh so + +0:14:18.620,0:14:22.610 +eventually what happens is that Bill has sort +of the framework of this stuff. And Bill being + +0:14:22.610,0:14:23.960 +Bill uh + +0:14:23.960,0:14:26.580 +he he work rather quickly + +0:14:26.580,0:14:29.880 +Uh someone wants to ask you to compare myself +to build Joy and I said look + +0:14:29.880,0:14:33.980 +there's really nothing that he's done that +I couldn't do but the problem is + +0:14:33.980,0:14:36.810 +what he gets done in a year would take me +a decade + +0:14:36.810,0:14:39.430 +because he was the master of figuring out. + +0:14:39.430,0:14:40.730 +I'm here + +0:14:40.730,0:14:42.039 +that's where I want to get to + +0:14:42.039,0:14:44.740 +what is absolutely shortest half from here +to there + +0:14:44.740,0:14:45.810 +and he would do it + +0:14:45.810,0:14:47.339 +and you end up with code + +0:14:47.339,0:14:48.260 +that worked but + +0:14:48.260,0:14:51.699 +that was not maintainable, expendable, changeable. + +0:14:51.699,0:14:56.130 +Anybody looked at the old version of the VI +understand what Im talking about. + +0:14:56.130,0:14:56.860 +so + +0:14:56.860,0:14:58.430 +So some of the rest of us you know + +0:14:58.430,0:14:59.709 +like to have stuff that like + +0:14:59.709,0:15:02.910 +you could go back and do something with later +but that takes longer + +0:15:02.910,0:15:03.689 +In any way + +0:15:03.689,0:15:06.300 +Bill figured out this socket interface the first + +0:15:06.300,0:15:07.930 +draft away it anyway + +0:15:07.930,0:15:12.140 +and he's got it and now he needs the newer call protocols +to put underneath it + +0:15:12.140,0:15:15.480 +so he goes to to + +0:15:15.480,0:15:16.399 +Uh Rob Gurvitz who was + +0:15:16.399,0:15:18.259 +leading the program of + +0:15:18.259,0:15:20.429 +development of TCP/IP at PBN + +0:15:20.429,0:15:23.660 +and says hey I got a socket interface and I need something to put +in here + +0:15:23.660,0:15:28.140 +and Rob says well we're not really done yet +but we have we have a slight prototype code + +0:15:28.140,0:15:29.990 +and you know you could put that in it + +0:15:29.990,0:15:31.319 +see how that works in + +0:15:31.319,0:15:33.100 +you know then well iterate hopefully the race + +0:15:33.100,0:15:35.810 +up so that we can work in parallel of you. + +0:15:35.810,0:15:38.410 +So Bill gets the code and + +0:15:38.410,0:15:40.420 +he puts it in + +0:15:40.420,0:15:40.859 +and + +0:15:40.859,0:15:42.979 +cranks it up and gets a kind of working + +0:15:42.979,0:15:43.820 +and + +0:15:43.820,0:15:45.580 +then decides to run + +0:15:45.580,0:15:49.110 +Some full test between machines to see +how well it works. + +0:15:49.110,0:15:55.150 +And were running at the at that time our development +machines development inputs were + +0:15:55.150,0:15:55.820 +VAC 750s +0:15:55.820,0:15:59.640 +so that was the big honk at VAC 780 +that ran entire NIP + +0:15:59.640,0:16:04.600 +and then there were these much smaller things +that were merely the size of a large + +0:16:04.600,0:16:05.769 +uh dishwasher + +0:16:05.769,0:16:11.839 +uh its bigger than a dishwasher more like +washing machine yes +its over range at any rate and used about + +0:16:11.839,0:16:13.660 +as much power as the same. + +0:16:13.660,0:16:16.260 +Uh with all the burners turned ON assuming its electric + +0:16:16.260,0:16:23.080 +uh anyway and the VAC 750 only +ran at 0.7 NIPs + +0:16:23.080,0:16:25.480 +Uh and so + +0:16:25.480,0:16:28.920 +Bill needs to be overrun sometimes back and forth +between these machines + +0:16:28.920,0:16:31.080 +and of course we don't have + +0:16:31.080,0:16:34.900 +anything like FTP or TELNET +anything that, which is not written yet + +0:16:34.900,0:16:36.890 +and so he just + +0:16:36.890,0:16:38.369 +being Bill as well + +0:16:38.369,0:16:39.699 +hack up something + +0:16:39.699,0:16:43.309 +uh which was all logged in RCP et cetera. + +0:16:43.309,0:16:47.820 +which was really just the design so that he +could test running between these two machines + +0:16:47.820,0:16:48.710 +Uh + +0:16:48.710,0:16:52.130 +This is the legacy that we end up with and +why the protocol number or anything like + +0:16:52.130,0:16:57.820 +that in the header you know its just a +test program no one is gonna actually use this + +0:16:57.820,0:17:02.020 +so the uh upshot of all this that + +0:17:02.020,0:17:03.840 +Bill fires up + +0:17:03.840,0:17:05.530 +RCP and + +0:17:05.530,0:17:08.230 +to copy a file between machines + +0:17:08.230,0:17:08.730 +and + +0:17:08.730,0:17:15.300 +and it copies a file at about 50..6 or +60 kilobits per second + +0:17:15.300,0:17:17.520 +and at that point the CPU is PEC + +0:17:17.520,0:17:19.799 +on the VAX 750 + +0:17:19.799,0:17:20.810 +And it you know + +0:17:20.810,0:17:24.810 +there's plenty of bandwidth still available +because we had 10 megabit Ethernet at that point + +0:17:24.810,0:17:28.519 +we had the new 10 megabit Ethernet because mostly + it was 3 megabit but + +0:17:28.519,0:17:31.059 +we had a 10 which was like really cool stuff. + +0:17:31.059,0:17:32.260 +uh and + +0:17:32.260,0:17:37.090 +uh so he talks to her rocker you know this thing runs +and girl was his attitude + +0:17:37.090,0:17:39.280 +as well the backbone + +0:17:39.280,0:17:44.540 +of the only the 56 K so as long as +we can saturate that what's the problem + +0:17:44.540,0:17:48.710 +and Bills attitude is well know I need to be + all you know get closer to the bandwidth of + +0:17:48.710,0:17:51.450 +the network between these local machines + +0:17:51.450,0:17:56.430 +And so of course Bill being Bill +dives in and see why this is the case that it's not running very + +0:17:56.430,0:17:57.440 +fast. + +0:17:57.440,0:17:58.500 +Infact DRAPA + +0:17:58.500,0:17:59.880 + wrote to + +0:17:59.880,0:18:01.680 +to some of this stuff to figure out + +0:18:01.680,0:18:06.320 +what was not running quickly. We already +had it for usual of our programs thats what we + +0:18:06.320,0:18:09.170 +put it into the kernel + +0:18:09.170,0:18:12.160 +And said to say some of these tools are still like +considered + +0:18:12.160,0:18:17.050 +like leading tools but are not leading tools but +still used tools today + +0:18:17.050,0:18:22.800 +Uh which isn't to say bad that it's just that +seems like in twenty years you could do better + +0:18:22.800,0:18:23.690 +at any rate + +0:18:23.690,0:18:27.490 +he quickly discovers that way the TCP/IP +code is written + +0:18:27.490,0:18:30.000 +is its this elaborate state machine + +0:18:30.000,0:18:32.970 +and there's the state transitions that you +go through and + +0:18:32.970,0:18:36.919 +each time we want to a state transition the +you jump through an indirect + +0:18:36.919,0:18:39.090 +this and that and other things and + +0:18:39.090,0:18:42.520 +and he goes well you know this is just great +and efficient and you know + +0:18:42.520,0:18:46.620 +we could just take this machine turned +into a giant switch statement + +0:18:46.620,0:18:49.860 +And you know if you're going to see a red area +know to get from here to there are going to + +0:18:49.860,0:18:53.070 +need to switch states buttoning ON and thats that + +0:18:53.070,0:18:54.380 +We'll just do that + +0:18:54.380,0:18:57.770 +And so he starts crank on his way +and + +0:18:57.770,0:18:59.230 +pounding out all of that + +0:18:59.230,0:19:01.680 +the structure and getting a down + +0:19:01.680,0:19:04.110 +of what it looks like today + +0:19:04.110,0:19:06.200 +And uh + +0:19:06.200,0:19:09.390 +that's the spin cleaned up considerably over +the years but + +0:19:09.390,0:19:12.280 +Uh so he cranks it down to this and + +0:19:12.280,0:19:14.770 +gets up to the point where + +0:19:14.770,0:19:19.420 +he can at least saturate that 3 megabit wire +we cant saturate the + +0:19:19.420,0:19:21.960 +10 megabit wire but it's because the + +0:19:21.960,0:19:24.640 +actually Ethernet controllers are which themselves are + +0:19:24.640,0:19:26.740 +not able to uh + +0:19:26.740,0:19:30.310 +really run at 10 megabits uh + +0:19:30.310,0:19:33.330 +these were you know prototype early release +stuff and so + +0:19:33.330,0:19:35.020 +we get it upto + +0:19:35.020,0:19:36.960 +somewhere around four + +0:19:36.960,0:19:39.370 +megabit uh + +0:19:39.370,0:19:43.370 +But we're only looking at 40% CPU time +so in theory if we could have run at wires + +0:19:43.370,0:19:46.809 +speed we got up to 10 megabit +and for other reasons it wouldn't have + +0:19:46.809,0:19:47.350 +but + +0:19:47.350,0:19:53.760 +the point is we were like uh a way way over of magnitude +ahead of where we started from later of + +0:19:53.760,0:19:56.350 +and so + +0:19:56.350,0:19:59.480 +Uh there's a lot of pressure from a lot of the + +0:19:59.480,0:20:02.950 +uh the people that have been that have 4.1 + +0:20:02.950,0:20:04.640 +that they want to + +0:20:04.640,0:20:07.320 +try out this networking stuff + +0:20:07.320,0:20:10.080 +and so uh + +0:20:10.080,0:20:11.870 +this code + +0:20:11.870,0:20:13.210 +as such as it was + +0:20:13.210,0:20:17.280 +uh got released as something that we call +4.1a + +0:20:17.280,0:20:20.680 +Uh so 4.1a was sort of the + +0:20:20.680,0:20:22.559 +first intermediate release of + +0:20:22.559,0:20:26.020 +Uh between 4.1 and 4.2 + +0:20:26.020,0:20:28.860 +and this got surprisingly large + +0:20:28.860,0:20:32.890 +larger distribution then we have really planned +on because you know you give it to one person + +0:20:32.890,0:20:38.010 +and then someone else hears about it. May want it +then someone else wants it nahnah..... +0:20:38.010,0:20:42.030 +It's not like today and where you just put it up + just anonymous FTP. This was still + +0:20:42.030,0:20:45.340 +put it on a tape Mule tape somebody big + +0:20:45.340,0:20:46.640 +nine track tapes + +0:20:46.640,0:20:48.110 +uh + +0:20:48.110,0:20:52.900 +And uh if you'd want lucky enough to have a high +tape drive 6250 + +0:20:52.900,0:20:55.090 +Which for inchtape drive then you had to get + +0:20:55.090,0:20:56.430 +the 1604 + +0:20:56.430,0:20:59.360 + tapes together all out there + +0:20:59.360,0:21:01.660 +Uh at any rate + +0:21:01.660,0:21:03.820 +This this code goes out + +0:21:03.820,0:21:04.299 +Uh and uh + +0:21:04.299,0:21:10.039 +measured about this time Sam Leffler +somebody's name probably know in contemporary + +0:21:10.039,0:21:11.800 +stuff having to do with wireless + +0:21:11.800,0:21:14.000 +Uh I actually joined + +0:21:14.000,0:21:15.710 +the Source Forge group there + +0:21:15.710,0:21:17.340 +uh and + +0:21:17.340,0:21:19.530 +he begins to help build with + +0:21:19.530,0:21:21.669 +a rewrite of the networking interface + +0:21:21.669,0:21:23.020 +on in particular + +0:21:23.020,0:21:27.210 +uh Bill hadn't quiet hard on to the +fact that + +0:21:27.210,0:21:30.320 +uh When you may want to have multiple connections + +0:21:30.320,0:21:33.970 +concurrently on a particular port +so if you're running some TP + +0:21:33.970,0:21:35.720 +server of HTTP + +0:21:35.720,0:21:38.640 +Uh the way he had written had a except + +0:21:38.640,0:21:42.780 +that the that the what we think of surround view of +the socket today was actually connected + +0:21:42.780,0:21:46.350 +and so that nobody else to talk to you on +that particular port to you. When youre + +0:21:46.350,0:21:48.170 +finished with that conversation and then + +0:21:48.170,0:21:51.520 +that the sock would become free again and you +couldve the next connection. + +0:21:51.520,0:21:52.759 +he says like that no + +0:21:52.759,0:21:54.389 +no we got it like +0:21:54.389,0:21:56.500 +to have more than one at a time + +0:21:56.500,0:22:01.510 +and so that's when the whole listen and the +rendezvous socket works up returns to a new + +0:22:01.510,0:22:03.670 +connected socket and so on + +0:22:03.670,0:22:06.960 +uh and so at any rate + +0:22:06.960,0:22:08.900 +this this all it gets done + +0:22:08.900,0:22:10.970 +And uh + +0:22:10.970,0:22:15.930 +I've gotten ropped into doing the files system +which is another part of an earlier story. + +0:22:15.930,0:22:20.130 +uh and so the file system is sort of the +point where we're ready to inflicted I mean + +0:22:20.130,0:22:22.120 +let other users try it + +0:22:22.120,0:22:24.480 +and we have revise networking + +0:22:24.480,0:22:25.600 +and so this + +0:22:25.600,0:22:26.460 +Uh came up + +0:22:26.460,0:22:28.960 +as a release called 4.1b + +0:22:28.960,0:22:33.390 +and then fairly quickly thereafter the signal +stuff got cleaned up + +0:22:33.390,0:22:36.670 +so 4.1b got replaced to 4.1c + +0:22:36.670,0:22:40.450 +Uh remember there were four things that we need +to get done so A was the networking B was the file system + +0:22:40.450,0:22:42.309 +and C was the signalling and + +0:22:42.309,0:22:43.920 +D was supposed to be + +0:22:43.920,0:22:47.880 +the virtual memory which of course did happen +for another six or seven years + +0:22:47.880,0:22:49.190 +Uh but + +0:22:49.190,0:22:52.390 +at any rate 4.1c was another one +of these ones + +0:22:52.390,0:22:54.600 +which was an enormously picked up + +0:22:54.600,0:22:56.300 +very wide distribution + +0:22:56.300,0:22:57.860 +uh and in + +0:22:57.860,0:23:01.110 +in the meantime + +0:23:01.110,0:23:05.640 +Uh you know the folks at the PBNF continues to +work and developing and + +0:23:05.640,0:23:06.570 +polishing their code + +0:23:06.570,0:23:09.720 +uh but Bill had asked another copy +of it so + +0:23:09.720,0:23:12.570 +they hadnt delivered it because it +wasnt done yet any way + +0:23:12.570,0:23:15.370 +uh and theyre just get it done + +0:23:15.370,0:23:18.450 +then give it to Berkley we could put it +in this + +0:23:18.450,0:23:20.790 +preparation for shipping 4.2 + +0:23:20.790,0:23:24.090 +Well 4.1c got out there + +0:23:24.090,0:23:25.610 +and uh + +0:23:25.610,0:23:29.760 +Bill then was setting about to work on +the VM stuff + +0:23:29.760,0:23:34.260 +When one day going to his office to +Chat with him. He says, Come into my office and you know I + +0:23:34.260,0:23:37.840 + I am + +0:23:37.840,0:23:41.870 +involved with these other folks down at Stanford +and they have some hardware + +0:23:41.870,0:23:42.860 +and they want + +0:23:42.860,0:23:45.020 +me to put some Berkley UNIX on it + +0:23:45.020,0:23:46.020 +and you know + +0:23:46.020,0:23:48.770 +starts selling these things as workstations + +0:23:48.770,0:23:49.889 +and uh + +0:23:49.889,0:23:52.500 +Im going I gonna go + +0:23:52.500,0:23:56.430 +I am going down there and you know that one of the +founding members of this and you should come + +0:23:56.430,0:23:57.230 +along + +0:23:57.230,0:23:58.750 +uh because you know + +0:23:58.750,0:24:03.989 +you being a single digit +employee at workstations. + +0:24:03.989,0:24:06.559 +Youll get huge stock option +its gonna be really exciting and you know + +0:24:06.559,0:24:11.450 +I at that time I had already completed my degree +in business from + +0:24:11.450,0:24:12.580 +Uh University California so + +0:24:12.580,0:24:14.710 +I knew something about business + +0:24:14.710,0:24:18.460 +in fact I've taken an entrepreneurial course +so I knew a lot about starting up companies + +0:24:18.460,0:24:19.830 +and selling back. + +0:24:19.830,0:24:23.230 +And so I said, Well you know Bill + +0:24:23.230,0:24:27.750 +this all sounds good what you know what what +is thats going to make you so successful. + +0:24:27.750,0:24:31.480 +He says, All those facts that you know we're +not going to vendor lock in and you know it's going + +0:24:31.480,0:24:33.190 +to be running the UNIX and + +0:24:33.190,0:24:38.000 +if SUN doesnt do it right they +can always go to another vendor and can get UNIX from now. + +0:24:38.000,0:24:40.730 +and uh thats that's going to be the story + +0:24:40.730,0:24:43.320 +And I said, Well you know this is well and good but + +0:24:43.320,0:24:46.120 +let me explain about workstations. + +0:24:46.120,0:24:50.860 +Uh workstations is about the install base of applications +as people buy workstations because the applications + +0:24:50.860,0:24:52.700 + run on it + +0:24:52.700,0:24:56.810 +And uh you know this is company APPOLO that had + +0:24:56.810,0:25:01.220 +a 4 years head start on SUN and they've +got a lock on the applications + +0:25:01.220,0:25:04.860 +and people arent going to switch the SUN because +they need the applications that run on APPOLO. + +0:25:04.860,0:25:06.690 +So you know + +0:25:06.690,0:25:09.400 +I'm sure it's going to be interesting but +uh + +0:25:09.400,0:25:13.909 +you know I'm only a few months from finishing +my PHD in I know I got a SUN with you that + +0:25:13.909,0:25:15.249 +won't happen + +0:25:15.249,0:25:18.390 +So Im gonna my PhD we'll see how SUN is doing in + +0:25:18.390,0:25:20.660 +over six or eight months and you know + +0:25:20.660,0:25:22.130 +Well evaluate + +0:25:22.130,0:25:26.640 +well of course it took me in and fourteen +fifteen months to finish my PhD and of + +0:25:26.640,0:25:27.740 +course by that time + +0:25:27.740,0:25:31.590 +SUN had burst onto the scene and didn't +have anything interesting stock options + +0:25:31.590,0:25:33.910 +left + +0:25:33.910,0:25:39.410 +So uh if you want it buy it's about business perhaps +I'm not the one to ask + +0:25:39.410,0:25:41.410 +at any rate + +0:25:41.410,0:25:44.420 +Bill left without doing to the VM system + +0:25:44.420,0:25:49.500 +Uh and he took 4.1c with him to get it +ported on to the the SUN hardware. + +0:25:49.500,0:25:51.300 +and + +0:25:51.300,0:25:54.240 +with this sort uh of hit + +0:25:54.240,0:25:55.910 +on the the main developer + +0:25:55.910,0:26:00.510 +uh we decided that rather than trying up to get +someone else up to speed in getting to the + +0:26:00.510,0:26:02.710 +VM done and so on. + +0:26:02.710,0:26:05.520 +We would instead + +0:26:05.520,0:26:09.370 +just declare the first three pieces of this +4 piece contract + +0:26:09.370,0:26:11.580 +uh work released + +0:26:11.580,0:26:14.020 +and so + +0:26:14.020,0:26:15.720 +uh around this time + +0:26:15.720,0:26:18.050 +this be back June of 1983 + +0:26:18.050,0:26:20.620 +Uh Micheal Karl joins the group + +0:26:20.620,0:26:22.590 +essentially coming in uh + +0:26:22.590,0:26:25.500 +to replace in into the position that was + +0:26:25.500,0:26:27.119 +up formerly held by + +0:26:27.119,0:26:29.330 +Sam ..Sam uh out by + +0:26:29.330,0:26:32.000 +Bill as that sort of + +0:26:32.000,0:26:33.250 +Head of lead programmer + +0:26:33.250,0:26:36.510 +uh and + +0:26:36.510,0:26:40.490 +because Sam also Sam had also considered taking +that position but ultimately decided that + +0:26:40.490,0:26:44.610 +uh he he didn't want to do that he was gonna +to go off to do graphic and + +0:26:44.610,0:26:46.990 + looks as well + +0:26:46.990,0:26:48.830 +and so + +0:26:48.830,0:26:51.030 +the uh + +0:26:51.030,0:26:52.980 +the upshot of this was that + +0:26:52.980,0:26:55.429 +4.2 got released + +0:26:55.429,0:26:57.920 +in about August of 83 + +0:26:57.920,0:27:01.549 +And uh + +0:27:01.549,0:27:05.640 +therere uh about 1000 copies around remember for each +copy that goes out thats like an institution gets + +0:27:05.640,0:27:07.429 +a copy and you know + +0:27:07.429,0:27:10.760 +spread across all those machines et cetera. + +0:27:10.760,0:27:14.470 +Uh there's another way of looking at it is you can +look at the number of licenses that AT&T + +0:27:14.470,0:27:16.400 + had outstanding + +0:27:16.400,0:27:21.420 +and how many licenses that Berkley had + +0:27:21.420,0:27:24.310 +at the time we had shipped a thousand, there were +about twelve hundred + +0:27:24.310,0:27:27.450 +uh AT&T licenses. So it says that + +0:27:27.450,0:27:29.149 +fair number of people were running + +0:27:29.149,0:27:35.310 +Uh now at this point just for context AT&T is +releasing system 5 release one + +0:27:35.310,0:27:37.380 +So that is short of where they are. + +0:27:37.380,0:27:39.290 +Uh They had taken their file system and + +0:27:39.290,0:27:44.010 +up the block size from 512 bytes + to 1 K + +0:27:44.010,0:27:46.750 +Uh at any rate + +0:27:46.750,0:27:47.870 +PBNN + +0:27:47.870,0:27:50.650 +was very upset that their TCP/IP + +0:27:50.650,0:27:53.340 +had not been incorporated. + +0:27:53.340,0:27:57.650 +and here we released it but it was this like +prototype early version had been bastardize + +0:27:57.650,0:27:59.100 +by Bill + +0:27:59.100,0:28:04.290 +and like you know this is it that you know +DARPA paid and now you know have the shiny wonderful + +0:28:04.290,0:28:05.990 +TCP/IP stack. + +0:28:05.990,0:28:09.150 +You should be putting this in + +0:28:09.150,0:28:10.369 +and so + +0:28:10.369,0:28:15.310 +they began agitating with DARPA and saying +what you know Blah Blah got this happened + +0:28:15.310,0:28:16.280 +and so + +0:28:16.280,0:28:20.950 +uh Michael Karl being very deliberative +what a person he is. + +0:28:20.950,0:28:23.330 +Says Ok well + +0:28:23.330,0:28:27.780 +I'll take the TCP/IP code from BBN and I'll +take the TCP code from + +0:28:27.780,0:28:28.870 +that we have in 4.2 + +0:28:28.870,0:28:31.550 +and will evaluate both of them + +0:28:31.550,0:28:38.430 +and the evaluation involves you know +performance and maintenance abilities + +0:28:38.430,0:28:41.980 +and functionality and all these various +necessary things + +0:28:41.980,0:28:42.779 +and + +0:28:42.779,0:28:47.370 +He says, Well you know the TCP/IP the idea comes +from BBN and has some really good ideas + +0:28:47.370,0:28:49.039 +and we don't have so we'll just + +0:28:49.039,0:28:51.330 +Pick those up and drop it in our distribution + +0:28:51.330,0:28:55.510 +Uh but we don't really feel that their + +0:28:55.510,0:28:57.950 +code is uh you know + +0:28:57.950,0:29:01.090 +is the good place to start from. There are lot of people +this point + +0:29:01.090,0:29:05.730 +that have worked with the prickly code +and they know that code is based on huge amount of enhancements + +0:29:05.730,0:29:07.570 +on it. + +0:29:07.570,0:29:10.880 +Uh This is wonderful book that's been written +all about it + +0:29:10.880,0:29:13.049 +So it is very well documented + +0:29:13.049,0:29:13.500 +and + +0:29:13.500,0:29:17.960 +it just doesn't seem to have really make sense +to + +0:29:17.960,0:29:22.460 +reset back to here and then have to get all +of those improvements changes et cetera. + +0:29:22.460,0:29:23.809 +Uh into this + +0:29:23.809,0:29:25.280 +particular new version. + +0:29:25.280,0:29:27.150 +uh it it + +0:29:27.150,0:29:30.440 +I finally graduated in 84 and during +the project + +0:29:30.440,0:29:33.260 +uh foolishly that taking over as + +0:29:33.260,0:29:34.170 +it's technically but + +0:29:34.170,0:29:35.419 +is the lead + +0:29:35.419,0:29:38.960 +which meant that I got to deal with all of +the political polls showed while + +0:29:38.960,0:29:41.870 +Mike got to do lots of programming + +0:29:41.870,0:29:45.140 +So let that be a lesson to you that if you think +that they're trying to sweet talk + +0:29:45.140,0:29:50.409 +you and How great it is to be in-charge +of things. + +0:29:50.409,0:29:50.960 +Uh + +0:29:50.960,0:29:53.080 +At any rate we + +0:29:53.080,0:29:56.680 +continue debating back and forth about this +stuff with BBN. + +0:29:56.680,0:30:00.180 +We kept taking stuff putting it into +And look you know that we've taken that we've weve taken that + +0:30:00.180,0:30:03.720 +No No. You have to take the code +You got to take the code. + +0:30:03.720,0:30:05.440 +So at any rate 4.3 + +0:30:05.440,0:30:07.750 +was really designed + +0:30:07.750,0:30:12.340 +much like 4.1 in the sense of consolidating +what we already had. So throughout the eh + +0:30:12.340,0:30:17.350 +the even-numbered releases from Berkeley had +a lot of functionality and odd releases + +0:30:17.350,0:30:19.930 +tried to really get the performance up + +0:30:19.930,0:30:24.170 +Uh we we're always striving to make sure that +things worked when the released up, but we didn't + +0:30:24.170,0:30:24.830 +always + +0:30:24.830,0:30:27.169 +strive to make sure they work optimally. + +0:30:27.169,0:30:30.090 +Uh figuring that it was better to sort of +tune things + +0:30:30.090,0:30:33.090 +after the fact that once you had a electrode working +in right way. + +0:30:33.090,0:30:37.510 +So anyway we made an announcement +of 4.3 + +0:30:37.510,0:30:39.379 +Uh actually at the June 1985 + +0:30:39.379,0:30:43.510 +it's the fact that two years basically have +gone by here. + +0:30:43.510,0:30:44.510 + and + +0:30:44.510,0:30:45.620 +uh + +0:30:45.620,0:30:46.470 +BBN + +0:30:46.470,0:30:50.620 +you know immediately objects that we have +not taken their code + +0:30:50.620,0:30:51.110 +and + +0:30:51.110,0:30:52.970 +uh they go to DARPA + +0:30:52.970,0:30:57.940 +and they said, You paid BBN +how many millions of dollars, more than you paid to + +0:30:57.940,0:30:59.719 +Berkley to do this code + +0:30:59.719,0:31:00.730 +you gonna uh you know +0:31:00.730,0:31:02.900 +and now they're not taking it + +0:31:02.900,0:31:04.330 +and and you know + +0:31:04.330,0:31:06.360 +you got look like ninnies or something. I +don't know + +0:31:06.360,0:31:08.390 +I dont know what theyve +said I wasn't there + +0:31:08.390,0:31:10.179 +But at any rate + +0:31:10.179,0:31:13.280 +DARPA + +0:31:13.280,0:31:16.600 +representative shows up in my office + +0:31:16.600,0:31:18.220 +and + +0:31:18.220,0:31:19.240 +Uh very calmly + +0:31:19.240,0:31:21.090 +uh explains that. + +0:31:21.090,0:31:22.100 +You know + +0:31:22.100,0:31:27.940 +the agreement was that BBN&N was going to +TCP/IP code and + +0:31:27.940,0:31:30.240 +Uh Berkley was to do the interface + +0:31:30.240,0:31:31.070 +and + +0:31:31.070,0:31:33.049 +This is this is the way the world and + +0:31:33.049,0:31:36.769 +you know we appreciate that you know youve +done with others stuff and it was it was great work + +0:31:36.769,0:31:38.670 +but + +0:31:38.670,0:31:42.460 +they are funding us and this is the way it is. + +0:31:42.460,0:31:44.120 +And my quarrel is like + +0:31:44.120,0:31:48.370 +uh you know arguing with them and look, you +know we've taken this and weve + +0:31:48.370,0:31:52.160 +put over here, all the reasons that I gave earlier + +0:31:52.160,0:31:55.410 +and you know the guy finally at the end just +says, + +0:31:55.410,0:31:57.679 + Not . You don't understand. + +0:31:57.679,0:31:58.549 +we're in charge + +0:31:58.549,0:32:02.500 +this is what you doing. Goodbye! + +0:32:02.500,0:32:04.960 +And you know + +0:32:04.960,0:32:08.000 +it just doesn't work very well at Berkeley +just come in and say, + +0:32:08.000,0:32:10.730 +This is what you're going to do. + +0:32:10.730,0:32:13.000 +The free speech movement comes to mind. + +0:32:13.000,0:32:15.970 +Uh so + +0:32:15.970,0:32:17.970 +You know Mike just go particularly BANANAS. + +0:32:17.970,0:32:21.190 +This like , Aggarh + +0:32:21.190,0:32:23.370 +and then fires off + +0:32:23.370,0:32:26.570 +unannounced to me fires off the thing +that the DARPA uh + +0:32:26.570,0:32:30.050 +which if I'd been would keep in shut +because I would not let them blame SUN + +0:32:30.050,0:32:31.690 +At any rate + +0:32:31.690,0:32:35.350 +I mean I only see it because DARPA guy +sent it back to me and says you need to keep + +0:32:35.350,0:32:38.720 +your staff under control. + +0:32:38.720,0:32:39.650 +Uh so + +0:32:39.650,0:32:44.130 +at any rate um you know +0:32:44.130,0:32:46.090 +Mike just says, I'm not doing it. + +0:32:46.090,0:32:49.510 +They can't make me, I'll just leave. + +0:32:49.510,0:32:51.640 + You know so you know + +0:32:51.640,0:32:56.450 +You tell them that they can either this way or +they can have that way or we wont release it. Um + +0:32:56.450,0:32:57.619 +Agrh . um + +0:32:57.619,0:33:00.330 +Come on theres got to be some compromise +we can do here. +0:33:00.330,0:33:01.730 +So + +0:33:01.730,0:33:05.360 +Finally uh I say, Okay lets I + +0:33:05.360,0:33:07.820 +I propose that we have a break-off. + +0:33:07.820,0:33:13.150 +So we will uh we we had already incorporated +into the paper there was big switch to compile one + +0:33:13.150,0:33:14.190 + way or the other + +0:33:14.190,0:33:15.609 +So my first proposal was + +0:33:15.609,0:33:16.910 +we will just ship it + +0:33:16.910,0:33:20.820 +with a compile time options. +So you can compile for VBN code +or it may compile powerful Berkley code. + +0:33:20.820,0:33:21.640 +Even BBN + +0:33:21.640,0:33:24.799 +Even DARPA could see +this was a no-win + +0:33:24.799,0:33:27.950 +because youll end up with some +people running one some people running the + +0:33:27.950,0:33:30.549 +other, this wont work smoothly. + +0:33:30.549,0:33:31.539 + This was just not + +0:33:31.539,0:33:33.550 +where they want it to go. + +0:33:33.550,0:33:35.409 +and so we said, Okay well + +0:33:35.409,0:33:38.480 +switch is there it's really easy to compile +with two different systems. + +0:33:38.480,0:33:41.830 +Lets have a break-off + +0:33:41.830,0:33:43.649 +you know well have compiler + +0:33:43.649,0:33:46.549 +with one and compiler with other one and have bunch of tests + +0:33:46.549,0:33:49.059 +and you know will decide which one is better. + +0:33:49.059,0:33:50.610 +Well of course we're not going to let + +0:33:50.610,0:33:55.429 +Berkley do the testing because therell be biased +for Berkley. Were not going to let BBN to do the testing + +0:33:55.429,0:33:57.640 +because theyll be biased for BB&N. + +0:33:57.640,0:33:59.720 +so weve to find a + +0:33:59.720,0:34:01.210 +uh + +0:34:01.210,0:34:03.480 +a third party and + +0:34:03.480,0:34:08.440 +uh the third party that DARPA actually proposes +is Col Mike Moose at the Ballistics + +0:34:08.440,0:34:10.979 +Research Laboratory, just up the road from +here and + +0:34:10.979,0:34:12.929 +Uh yeah + +0:34:12.929,0:34:14.509 +little known to + +0:34:14.509,0:34:18.659 +most of the people is that we actually knew Mike Moose +very well because we worked a lot with him. + +0:34:18.659,0:34:19.910 +working on TCP/IP + +0:34:19.910,0:34:23.659 +so + +0:34:23.659,0:34:25.059 +were like uhm yah its ok with us. + +0:34:25.059,0:34:27.869 +And you know he was in military therere + +0:34:27.869,0:34:31.139 +chain of command and BBN mock +new ones so + +0:34:31.139,0:34:33.589 +theyre fine with it + +0:34:33.589,0:34:34.370 +So ok great so + +0:34:34.370,0:34:36.469 +you know off we ship that tape and + +0:34:36.469,0:34:40.749 +He does not tell us what test he is gonna do +so neither side gets to + +0:34:40.749,0:34:44.720 +do anything other than make some suggestions +general suggestions so + +0:34:44.720,0:34:48.779 +Of course we suggest the 3 port performance +test could be important + +0:34:48.779,0:34:51.659 +and the BBN suggests that + +0:34:51.659,0:34:55.320 +uh how well it works in that phase of large +package + +0:34:55.320,0:34:57.049 +great losses is important + +0:34:57.049,0:35:01.430 +because of course the military you know will drop +a nuclear bomb on Chicago we want to make sure + +0:35:01.430,0:35:03.589 +we make outstand through that + +0:35:03.589,0:35:07.320 +At any rate Mike Moose sets up these the systems +and + +0:35:07.320,0:35:09.150 +runs a bunch of tests + +0:35:09.150,0:35:12.179 +and of course without talking to either side + +0:35:12.179,0:35:16.950 +and so we're sort of pens and needles +because you know several weeks goes by + +0:35:16.950,0:35:18.550 +and finally duh + +0:35:18.550,0:35:19.589 +his report lands. + +0:35:19.589,0:35:23.029 +You know humh comes in the mail here's his report + +0:35:23.029,0:35:26.269 +And the first thing youd do is +to flip to the last page to see what the conclusion + +0:35:26.269,0:35:29.479 +is but he knew that's what we're going to +do so he hid the conclusion somewhere in the + +0:35:29.479,0:35:31.660 +middle + +0:35:31.660,0:35:33.369 +what's the need to read the entire thing + +0:35:33.369,0:35:36.229 +but it's a it's a very interesting week because + +0:35:36.229,0:35:39.910 +in terms of three protocols +we expected we did better + +0:35:39.910,0:35:40.840 +Uh what + +0:35:40.840,0:35:44.929 +that he introduced a thing where it would +randomly drop about every + +0:35:44.929,0:35:46.769 +fourth package + +0:35:46.769,0:35:47.799 +and + +0:35:47.799,0:35:49.390 +under this scenario + +0:35:49.390,0:35:50.919 +he says uh + +0:35:50.919,0:35:52.420 +the BBN code + +0:35:52.420,0:35:56.820 +Jumps out to an early start getting a lot +more data through interface of hardware + +0:35:56.820,0:35:59.039 +packet loss uh + +0:35:59.039,0:36:03.099 +but but the Berkley part managers to catch up +while the BB&N machine reboots. + +0:36:03.099,0:36:07.719 +Uh + +0:36:07.719,0:36:12.109 +Ultimately resulting in the conclusion +that even in the phase of the packet + +0:36:12.109,0:36:14.739 +loss, Berkley code seems to + +0:36:14.739,0:36:16.349 +do a little better. + +0:36:16.349,0:36:17.739 +And uh so his + +0:36:17.739,0:36:22.119 +conclusion is that it should continue to ship +with the BSD code. + +0:36:22.119,0:36:25.519 +Uh so with this result in hand + +0:36:25.519,0:36:28.749 +we are authorized by DARPA to do the release + +0:36:28.749,0:36:32.189 +and we finally managed to release + +0:36:32.189,0:36:35.829 +4.3 in June of + +0:36:35.829,0:36:38.420 +1986 + +0:36:38.420,0:36:39.599 +so + +0:36:39.599,0:36:41.519 +at this point uh + +0:36:41.519,0:36:42.199 +At this point + +0:36:42.199,0:36:45.389 +the new TCP code is gone out + +0:36:45.389,0:36:47.189 +were you know that + +0:36:47.189,0:36:52.289 +The revised BSD TCP/IP code is gone out + +0:36:52.289,0:36:54.689 +[with or without subnets] +With or without uh..subnets + +0:36:54.689,0:36:56.150 +uh that had + +0:36:56.150,0:36:59.809 +well the the class ABC type of subnets +not + +0:36:59.809,0:37:01.319 +not the uh + +0:37:01.319,0:37:02.910 +the sider stuff + +0:37:02.910,0:37:09.910 +sider came along + +0:37:16.329,0:37:17.299 +okay + +0:37:17.299,0:37:19.409 +uh at any rate uh + +0:37:19.409,0:37:23.719 +now we start accelerating the story hope +that again because I have to finish in + +0:37:23.719,0:37:25.559 +eight minutes + +0:37:25.559,0:37:27.540 +so at this point + +0:37:27.540,0:37:28.260 +Keith Bostic + +0:37:28.260,0:37:29.209 +comes to + +0:37:29.209,0:37:30.469 +comes to Berkeley + +0:37:30.469,0:37:34.059 +uh he had been at excise which some of you +may know + +0:37:34.059,0:37:37.329 +and this is in the fall of 1986 so + +0:37:37.329,0:37:40.219 +we've released 4.3 were just +getting cranked up + +0:37:40.219,0:37:43.049 +to work on 4.4 uh + +0:37:43.049,0:37:45.259 +remember that VM thing + +0:37:45.259,0:37:48.039 +and I think by this time this is other little + +0:37:48.039,0:37:50.329 +protocol MFS you mayve heard of it + +0:37:50.329,0:37:54.839 +and it was felt that we ought to have that +as well + +0:37:54.839,0:37:57.150 +uh so bring Keith to pass the protocol + +0:37:57.150,0:37:59.560 +Uh one of his of requirements for coming + +0:37:59.560,0:38:03.420 +is that we have to agree to allow him to finish +that port of + +0:38:03.420,0:38:06.539 +2.11 to the PDP 11 ah + +0:38:06.539,0:38:08.489 +and uh you know + +0:38:08.489,0:38:12.470 +it's it's a real challenge to get a kernel which +adapt the port where it was about + +0:38:12.470,0:38:17.239 +just under 200 kilobyte of text +to fit on to a machine that has a maximum + +0:38:17.239,0:38:20.299 +amount of program space of 64 kilobytes + +0:38:20.299,0:38:23.450 + because what you have to do is to +break and overlays + +0:38:23.450,0:38:27.789 +and then you map in the overly and + the granularity of mapping is 4k + +0:38:27.789,0:38:28.530 +okay + +0:38:28.530,0:38:33.079 +So there's only eight pieces that you +can remap uh + +0:38:33.079,0:38:37.519 +Mike can I added to it is thats +incredibly challenging problem and it's + +0:38:37.519,0:38:41.469 +probably you know intellectually very stimulating +but there's no way how we would do that. + +0:38:41.469,0:38:43.139 +Well if you want to do it you know + +0:38:43.139,0:38:45.849 +by all means you can do that in your +spare time + +0:38:45.849,0:38:48.309 +uh there is only three jobs you need to do + +0:38:48.309,0:38:51.540 +uh one is uh you need to be the answer the +phone + +0:38:51.540,0:38:55.409 +uh first thing other than distribution like +technical questions + +0:38:55.409,0:38:58.149 +and answer the incoming the mail stream for technical +questions. + +0:38:58.149,0:39:03.900 +Uh do some programming in in in your spare +time you can work on the PDP 11 + +0:39:03.900,0:39:04.850 +and know he was + +0:39:04.850,0:39:08.669 +had no other support for the phones and the +email + +0:39:08.669,0:39:09.460 +and we had + +0:39:09.460,0:39:12.209 +you know by this time about two thousand + +0:39:12.209,0:39:14.749 +uh distributions + +0:39:14.749,0:39:19.730 +So at any rate Keith did in fact finally +finished the PDP 11. + +0:39:19.730,0:39:20.779 +Uh port uh + +0:39:20.779,0:39:23.029 +got 2.11 out of the door + +0:39:23.029,0:39:26.960 +and we have a party to celebrate the fact +that he had completed this. + +0:39:26.960,0:39:29.850 +and part of the party was opening the window + +0:39:29.850,0:39:31.240 +on the fourth floor + +0:39:31.240,0:39:33.519 +of our uh building + +0:39:33.519,0:39:38.849 +and taking his PDP 11 and +ceremoniously dropping out of the window to the cement + +0:39:38.849,0:39:40.979 +sidewalks four stories below + +0:39:40.979,0:39:43.100 +to insure that he would never + +0:39:43.100,0:39:44.749 +work on it again. + +0:39:44.749,0:39:50.409 +This was him doing this not us. + +0:39:50.409,0:39:51.849 +At any rate uh + +0:39:51.849,0:39:53.930 +Keith of course being in the front line + +0:39:53.930,0:39:55.919 +uh support + +0:39:55.919,0:39:59.199 +Uh had a lot of feedback from various users uh + +0:39:59.199,0:40:00.509 +in particular + +0:40:00.509,0:40:03.559 +there were a number of vendors that work + +0:40:03.559,0:40:05.159 +trying to build + +0:40:05.159,0:40:05.930 +uh +0:40:05.930,0:40:08.560 +Network cards you could plug into a PC + +0:40:08.560,0:40:10.999 +that would provide TCP/IP network + +0:40:10.999,0:40:16.180 +and of course the PC it was way too feeble +to actually run the code in the PC itself + +0:40:16.180,0:40:17.539 +so they had a outward + + +0:40:17.539,0:40:18.799 +processor on the card + +0:40:18.799,0:40:22.039 +and theyd put the TCP/IP in there +that would sort of look + +0:40:22.039,0:40:23.200 +uh like a + +0:40:23.200,0:40:26.779 +file or some kind of device on +that PC and + +0:40:26.779,0:40:28.900 +so they would just open the uh + +0:40:28.900,0:40:30.709 +get connected to the network + +0:40:30.709,0:40:34.939 +and So of course they wanted TCP/IP +code from Berkeley along with the important + +0:40:34.939,0:40:38.210 +utilities like O or log in and + +0:40:38.210,0:40:38.659 +RCP uh + +0:40:38.659,0:40:42.709 +by that time FTP and TELNET uh + +0:40:42.709,0:40:45.189 +and + +0:40:45.189,0:40:48.909 +they didn't our standard thing was +to buy the distribution and take whatever + +0:40:48.909,0:40:50.100 +they need to get out of it. + +0:40:50.100,0:40:52.769 +But that meant they had to get an AT&T license. + +0:40:52.769,0:40:57.680 +By the late 80s the AT&T +licenses or up to a quarter million dollars + +0:40:57.680,0:40:59.379 +and when you're in a little + +0:40:59.379,0:41:03.939 +commodity PC business a quarter million dollar +cost just to get the surfer before you do + +0:41:03.939,0:41:04.619 +anything with it + +0:41:04.619,0:41:06.669 +was just a non-starter for them. + +0:41:06.669,0:41:09.850 +And so we're getting requests for just + +0:41:09.850,0:41:13.199 +that TCP/IP code and associated utilities. + +0:41:13.199,0:41:17.479 +clearly since it all been developed in Berkley +was none of that in the original version 7 UNIX + +0:41:17.479,0:41:19.910 +which started from uh + +0:41:19.910,0:41:22.099 +AT&T couldnt place any claim to that. + +0:41:22.099,0:41:26.829 +And so we talked to the lawyers at Berkeley and +theyre like this How can we make money on this? uh + +0:41:26.829,0:41:29.249 +because that's what they were paid to say and + +0:41:29.249,0:41:33.089 +so we pointed out to them that It had been developed +under a government contract and we showed them + +0:41:33.089,0:41:36.630 +the government contract saying that you know +this stuff done in this contract has to be + +0:41:36.630,0:41:37.229 +made + +0:41:37.229,0:41:40.769 +available to the public at reasonable cost or +whatever it said + +0:41:40.769,0:41:41.380 +uh + +0:41:41.380,0:41:42.100 +so + +0:41:42.100,0:41:45.669 +it didnt really say that but be convinced the +lawyers that it did + +0:41:45.669,0:41:46.359 +and so + +0:41:46.359,0:41:48.369 +They agreed to lead us to pull out + +0:41:48.369,0:41:50.889 +this chunk of Berkley code and we + +0:41:50.889,0:41:51.859 +released that + +0:41:51.859,0:41:52.579 +under + +0:41:52.579,0:41:54.980 +this new license that the lawyers coped up + +0:41:54.980,0:41:59.299 +uh which is the one that we today think of +as the BSD license. + +0:41:59.299,0:42:03.569 +And of course our goal was to the absolutely +as + +0:42:03.569,0:42:07.979 +free as you know free to give away as we could possibly could +the really only thing you had to do was to give the credit to + +0:42:07.979,0:42:08.789 +Berkeley. + +0:42:08.789,0:42:12.140 +Uh so this was networking release one. + +0:42:12.140,0:42:14.019 +and uh it was a wild success + +0:42:14.019,0:42:17.519 +uh the amazing thing was that we sold at 100 +we could + +0:42:17.519,0:42:20.249 +Buy it from Berkeley for 1000 dollars + +0:42:20.249,0:42:24.169 +or you could just downloaded it off your net +for free + +0:42:24.169,0:42:28.209 +and yet a thousand people chose to pay a 1000 +dollars at Berkeley to get it. + +0:42:28.209,0:42:30.220 +and the only difference was that you got + +0:42:30.220,0:42:33.660 +to take a 9 track tape which many people didn't +have any way of reading + +0:42:33.660,0:42:34.100 +and + +0:42:34.100,0:42:35.719 +hey uh + +0:42:35.719,0:42:39.869 +that a piece of paper signed by Berkley +lawyers + +0:42:39.869,0:42:42.289 +essentially saying Yes you can use this + +0:42:42.289,0:42:44.939 +and that's what they were paying a 1000 dollars for + +0:42:44.939,0:42:48.239 +and the lawyers of course got paid something +like a hundred dollars for every one of + +0:42:48.239,0:42:50.790 +these that they signed because of the overhead + +0:42:50.790,0:42:51.650 +et cetera + +0:42:51.650,0:42:54.349 +they thought this was the cat's pyjamas + +0:42:54.349,0:42:57.499 +you know maybe the campus was making a lot of +money but their office was confined to offer just to Thank + +0:42:57.499,0:42:59.009 +you very much! + +0:42:59.009,0:43:01.409 +so + +0:43:01.409,0:43:03.420 +Uh at any rate oh + +0:43:03.420,0:43:04.440 +Network one was + +0:43:04.440,0:43:06.059 +widely successful + +0:43:06.059,0:43:08.309 +and so there was pressure as Keith + +0:43:08.309,0:43:12.369 +would come inside uh someone today want to want to +be I or somebody wants this is or somebody + +0:43:12.369,0:43:13.589 +wants that + +0:43:13.589,0:43:16.289 +and you know we really ought to do this and uh + +0:43:16.289,0:43:19.440 +you know how about figuring out other parts of the +kernel. + +0:43:19.440,0:43:20.559 +Mike and I would + +0:43:20.559,0:43:24.929 +Michael and I would look at the each other we +look at Keith its like yah yah all right. + +0:43:24.929,0:43:28.989 +Uh what you figured out you know I mean +You know all entire of the C library and all + +0:43:28.989,0:43:29.819 +this and that +0:43:29.819,0:43:31.760 +so we got to the conferences and + +0:43:31.760,0:43:32.859 +Keith is + +0:43:32.859,0:43:36.809 +putting up you know these huge list of utilities +youll rewrite this utility and get their name + +0:43:36.809,0:43:37.729 +in lights. + +0:43:37.729,0:43:40.869 +and uh you know and uh + +0:43:40.869,0:43:43.939 +So he would get this stuff send too along + +0:43:43.939,0:43:48.529 +and he would almost always just take a +bash and Keiths normal form which + +0:43:48.529,0:43:54.139 +usually get look a lot like what he got but +he started from cleans so that + +0:43:54.139,0:43:55.119 +Was uh + +0:43:55.119,0:43:59.229 +you know good and you know its sort of easy + you know that first someone sensing cat in. + +0:43:59.229,0:44:01.450 +you know someone like really go ho hog and + +0:44:01.450,0:44:04.849 +does that all options to allow uh + +0:44:04.849,0:44:07.629 +this is in the days before the new software +you know + +0:44:07.629,0:44:10.119 +lots of options for in vote + +0:44:10.119,0:44:13.079 +Uh and then + +0:44:13.079,0:44:15.239 +one day uh + +0:44:15.239,0:44:16.759 +Keith comes into to report + +0:44:16.759,0:44:18.470 +that somebody is redoing + +0:44:18.470,0:44:20.449 +the T graph family + +0:44:20.449,0:44:24.099 +and this is when Mike and I look at each +sort of grope + +0:44:24.099,0:44:28.190 +because the next question out of +Keiths mouth is Hows that kernel coming guys? + +0:44:28.190,0:44:32.299 +We can't go to the useless conferences and say rewrite +the kernel to get your name in lights you know + +0:44:32.299,0:44:36.400 +so we actually have to like go through and +figure out what's + +0:44:36.400,0:44:39.130 +Berkley and what's not Berkeley and + +0:44:39.130,0:44:40.499 +segregate stuff out uh + +0:44:40.499,0:44:43.459 +A long story but eventually we get + +0:44:43.459,0:44:45.599 +something we feel that was clean + +0:44:45.599,0:44:47.929 +and uh so + +0:44:47.929,0:44:51.699 +we go back to the lawyers and say well +cut the update to the networking tape + +0:44:51.699,0:44:52.989 +that we'd like to do + +0:44:52.989,0:44:59.079 +uh that uh we're going to call it networking +release 2 not 2 and + +0:44:59.079,0:45:00.470 +so actually + +0:45:00.470,0:45:04.599 +we do sort of come on and explain it is practically +the entire system and + +0:45:04.599,0:45:07.759 +lots of stuff happens but they finally sign +off on it + +0:45:07.759,0:45:09.469 +and we put it out there + +0:45:09.469,0:45:14.519 +and it's out there for good half a year before +BSD i + +0:45:14.519,0:45:17.499 +was in the missing bits and start telling +it + +0:45:17.499,0:45:19.650 +they you know + +0:45:19.650,0:45:21.439 +engineers are + +0:45:21.439,0:45:23.230 +not really that good at + +0:45:23.230,0:45:25.419 +marketing and so + +0:45:25.419,0:45:28.859 +in particular they think that the sort of +cutesy see things like + +0:45:28.859,0:45:32.729 +having your phone number being 1800 +its UNIX + +0:45:32.729,0:45:38.759 +and adds to save a ninety nine percent of the +price of a AT&T charges for the source code + +0:45:38.759,0:45:40.439 +Uh are cute + +0:45:40.439,0:45:41.670 +Got it really irritates + +0:45:41.670,0:45:44.759 +the piss out of AT&T + +0:45:44.759,0:45:46.270 +And so they sue us + +0:45:46.270,0:45:49.319 +imagine that why they sue BSD I + +0:45:49.319,0:45:53.509 +and BSD i is what added only the 6 +file and you know all the rest came from Berkley. + +0:45:53.509,0:45:57.129 +So we would be happy to talk about any of +the six files youd like to + +0:45:57.129,0:45:59.530 +uh but otherwise you know forget it + +0:45:59.530,0:46:01.009 +and the judge agrees + +0:46:01.009,0:46:04.539 +because they don't have any problem about +those six files and so + +0:46:04.539,0:46:07.769 +there's no choice but to sue the University + +0:46:07.769,0:46:09.259 +and uh + +0:46:09.259,0:46:11.160 +thats another long story for which I did + +0:46:11.160,0:46:12.629 +become an expert witness + +0:46:12.629,0:46:14.390 +which is actually proven + +0:46:14.390,0:46:18.809 +to be of relatively lucrative thing because +being an expert witness + +0:46:18.809,0:46:20.039 +it's something that you + +0:46:20.039,0:46:24.669 +it's really hard to break into that is no +one wants to hire someone hasn't done it before + +0:46:24.669,0:46:28.359 +top and it turns out that it really didn't +matter how much you know, do + +0:46:28.359,0:46:33.829 +you have a PhD because if you have PhD by definition +you must know you're talking about + +0:46:33.829,0:46:34.859 +himself + +0:46:34.859,0:46:36.219 +and number two is + +0:46:36.219,0:46:40.839 +How you speak in public then you know especially +people harassing you look at ten thousand + +0:46:40.839,0:46:41.830 +students you know + +0:46:41.830,0:46:45.309 +there's not a question that a graduate student +has never asked me but Id deal with it + +0:46:45.309,0:46:51.309 +and lawyer is not up to a graduate student. + +0:46:51.309,0:46:53.219 +So I did pretty well that score too + +0:46:53.219,0:46:55.129 +uh and so + +0:46:55.129,0:46:57.599 +because I that they were forced to use me + in this case. + +0:46:57.599,0:47:00.080 +And but I did ran after then + +0:47:00.080,0:47:03.879 +the lawyer the outside lawyer that the Berkeley +got me to do some other stuff and + +0:47:03.879,0:47:08.800 +one thing leads to another so today I am uh +an expert in the + +0:47:08.800,0:47:12.989 +network complaints vs SUN microsystem +lawsuit vs ZFS lawsuit + +0:47:12.989,0:47:14.439 +which is uh + +0:47:14.439,0:47:17.739 +an interesting story that you can ask over +here + +0:47:17.739,0:47:18.679 +At any rate + +0:47:18.679,0:47:21.629 +we eventually win the the lawsuit + +0:47:21.629,0:47:26.089 +And uh and we don't win it be subtle the +lawsuit excuse me I have to use proper terminology + +0:47:26.089,0:47:27.029 +Here. + +0:47:27.029,0:47:28.750 +We have to agree + +0:47:28.750,0:47:32.999 +that networking released 2 did in fact have +things in it that infringed + +0:47:32.999,0:47:34.490 +uh and + +0:47:34.490,0:47:35.999 +however + +0:47:35.999,0:47:40.329 +Uh the things that infringe were sufficiently +minor that was some + +0:47:40.329,0:47:40.730 +changes + +0:47:40.730,0:47:42.039 +uh we can + +0:47:42.039,0:47:43.370 +get something else clean + +0:47:43.370,0:47:45.959 + we agreed that we will re-release + +0:47:45.959,0:47:50.890 +we'd do new release that has all these +changes made in it as 4.4 BSD lite. + +0:47:50.890,0:47:54.449 +So its 4.4 BSD which is the whole +thing complete with AT&T stuff and release + +0:47:54.449,0:47:56.059 +AT&T licenses + +0:47:56.059,0:47:57.580 +and for 4.4 BSD lite + +0:47:57.580,0:48:00.979 + which is got the stuff AT&T stuffs stripped out +of it. + +0:48:00.979,0:48:04.869 +And AT&T really thought theyd won big-time +because all these people had been using the + +0:48:04.869,0:48:05.509 +Net 2. + +0:48:05.509,0:48:08.909 +They're going to be forced to throw out +because were not allowed to tell what the changes + +0:48:08.909,0:48:09.640 +were + +0:48:09.640,0:48:12.729 +and have to start all over again + +0:48:12.729,0:48:14.929 +from lite + +0:48:14.929,0:48:19.339 +And the Bill Joy in particular has never did that +and so that's why this distribution got hammered + +0:48:19.339,0:48:22.150 +FreeBSD had to do it + +0:48:22.150,0:48:24.329 +not the BSD folks had to do it uh + +0:48:24.329,0:48:26.289 +and it was a huge amount of work but they +did it + +0:48:26.289,0:48:30.489 +and it was great because we got two more years +worth of our technology out + +0:48:30.489,0:48:32.959 +at any rate + +0:48:32.959,0:48:36.289 +we actually did one more incremental sort of the +release to + +0:48:36.289,0:48:39.510 +I would just fix some of the most egregious +bugs + +0:48:39.510,0:48:43.309 +thats what all happened was we got all these people +who bought this BSD lite licenses + +0:48:43.309,0:48:46.519 +Because they wanted to pay the +thousand dollars to get the piece of paper + +0:48:46.519,0:48:46.940 +saying + +0:48:46.940,0:48:48.899 +and this is fine too + +0:48:48.899,0:48:49.640 +and uh + +0:48:49.640,0:48:52.819 +so we had a chunk of money and so Keith +and I started to do the work + +0:48:52.819,0:48:54.689 +through that money + +0:48:54.689,0:48:56.329 +and it's just spend it + +0:48:56.329,0:48:58.640 +essentially doing bug fixes and stuff on lite + +0:48:58.640,0:49:00.839 +And finally released when the money ran out then + +0:49:00.839,0:49:03.769 +we left the university and that was the last release + +0:49:03.769,0:49:05.679 +Uh from from Berkley + +0:49:05.679,0:49:12.679 +and that is the end of my story +So any questions? +Uh .. Yes + + +0:49:20.069,0:49:21.929 +Sorry.. uh VM system + + +0:49:21.929,0:49:25.359 +VM system yup. + +0:49:25.359,0:49:30.579 +So the VM system was actually sort of an interesting +story uh.. + +0:49:30.579,0:49:33.759 +Actually both VM and NFS were interesting +Stories. + +0:49:33.759,0:49:36.890 +The VM system uh + +0:49:36.890,0:49:41.160 +one of our attitudes was that why always come +up with a good idea when you can steal a better + +0:49:41.160,0:49:42.759 +one and + +0:49:42.759,0:49:46.369 +you know were just allowed to write +a VM system this uh + +0:49:46.369,0:49:47.579 +you know that BSD folks + +0:49:47.579,0:49:48.599 +clearly know + +0:49:48.599,0:49:49.530 +and + +0:49:49.530,0:49:50.889 +uh so + +0:49:50.889,0:49:54.539 +we looked around to see if there's anything +that we could use + +0:49:54.539,0:49:57.969 +rather than have to interpret it from the scratch +ourselves. and + +0:49:57.969,0:50:01.689 + evaluated two different things one +was + +0:50:01.689,0:50:04.759 +uh the the stuff that SUN had done + +0:50:04.759,0:50:10.109 +and the other was the stuff that had been done at +Carnegie Mellon under the Marks then and + +0:50:10.109,0:50:14.069 +Uh they've done a whole lot of very good work +on the VM system we didn't really agree with + +0:50:14.069,0:50:18.179 +the microphone kernel principle but we liked +the VM system + +0:50:18.179,0:50:21.589 +and so we talked to SUN about potentially + +0:50:21.589,0:50:23.479 +Contributing uh + +0:50:23.479,0:50:27.159 +chunk of their VM system we talked to them +about contributing a chunk of their + +0:50:27.159,0:50:27.949 +VFS stuff. +0:50:27.949,0:50:31.649 +We knew that we couldnt get the NFS from them but + +0:50:31.649,0:50:32.659 +they they you know + +0:50:32.659,0:50:36.809 +the folks at SUN is sorts of an engineering were too +enthusiastic about it and percolated it all the way + +0:50:36.809,0:50:39.459 +up to the Scott McNealy, he was the CEO at that +time + +0:50:39.459,0:50:42.979 +and even Scott McNealy was pretty enthusiastic about +giving the stuff to Berkley. + +0:50:42.979,0:50:45.239 +you know as long as SUN would get credit for +it + +0:50:45.239,0:50:47.140 +And uh + +0:50:47.140,0:50:51.549 +so we've really got to the point where it +is needed to be signed off by the the SUN + +0:50:51.549,0:50:52.429 +lawyers + +0:50:52.429,0:50:58.919 +and sent off to them and after that SUN +lawyers came back and said, + +0:50:58.919,0:51:03.199 + Well you know your stock holders could accuse you of giving +away company property + +0:51:03.199,0:51:07.309 +and and sue you in court and we really don't +recommend that you do this. + +0:51:07.309,0:51:08.380 +and so + +0:51:08.380,0:51:09.999 +that the very last minute it was .. + +0:51:09.999,0:51:12.779 +We couldn't get any of that stuff from SUN. + +0:51:12.779,0:51:14.699 +So we ended up + +0:51:14.699,0:51:17.859 +adopting this stuff from Mark +the Mark VM system + +0:51:17.859,0:51:18.699 +and + +0:51:18.699,0:51:22.369 +so it fell to myself to take that and + +0:51:22.369,0:51:24.669 +figure out essentially how to plug it in. + +0:51:24.669,0:51:27.140 +apart of all of this the old VM system + +0:51:27.140,0:51:32.259 +was very portable provided you are pouring +into something that looked a lot like a VAX. + +0:51:32.259,0:51:37.429 +because it came down to uh you know the that the +basic page data structure was VAX + +0:51:37.429,0:51:38.979 +page data structure. + +0:51:38.979,0:51:40.590 +and so when people will ported it. + +0:51:40.590,0:51:44.440 +If they have something with the page data structure +to look different they wouldn't take it from + +0:51:44.440,0:51:48.360 +the VAX and would shuffle the bits around and put +it in the right order and then dropped it into + +0:51:48.360,0:51:50.829 +the whatever that architecture was + +0:51:50.829,0:51:55.509 +a page table. So it worked pretty well for porting +into the motorola uh because the + +0:51:55.509,0:51:57.959 +NMU network was a lot like the VAX +uh + +0:51:57.959,0:52:01.899 +and some other machines but in other ways it +just really didn't cut it at all + +0:52:01.899,0:52:06.130 +whereas the VM had been done by +the the folks at the CMU. + +0:52:06.130,0:52:09.389 +In Mark was uh you know + +0:52:09.389,0:52:11.059 +very elegantly designed + +0:52:11.059,0:52:13.150 +uh and in fact + +0:52:13.150,0:52:14.740 +it had in some ways + +0:52:14.740,0:52:18.980 +too many bells and whistles and too many options +and because a lot of sort of dead end so they + +0:52:18.980,0:52:21.189 +just sort of left hanging there + +0:52:21.189,0:52:22.589 +Uh but uh + +0:52:22.589,0:52:26.200 +the structure itself was nicely done about it was +about 90% machine independent code + +0:52:26.200,0:52:28.459 +10 % machine dependent code so that + +0:52:28.459,0:52:30.119 +a well defined interface. + +0:52:30.119,0:52:32.239 +the PMAC interface. + +0:52:32.239,0:52:34.359 +got that put in and of course we + +0:52:34.359,0:52:40.019 +did onto the the Berkeley mmap Interface +rather than what they did + +0:52:40.019,0:52:42.700 +with uh ports and so on + +0:52:42.700,0:52:43.160 +Uh and + +0:52:43.160,0:52:48.210 +So I got back together sort in the that +the 4 3 phase I didnt talk about there were + +0:52:48.210,0:52:51.130 +two releases between 4.3 and 4.4 +4.3 + +0:52:51.130,0:52:53.789 +Taha and 4.3 Reno and + +0:52:53.789,0:52:56.689 +the first of them having to the VM and the + +0:52:56.689,0:52:58.099 +second one having the NFS in it. + +0:52:58.099,0:52:59.759 +And uh + +0:52:59.759,0:53:01.499 +so + +0:53:01.499,0:53:05.529 +backup put out and then luckily people would feel +sort of look at it and said well you know + +0:53:05.529,0:53:07.709 +this really needs like a lot of work + +0:53:07.709,0:53:11.699 +so Michael, University Utah did a tremendous +amount of the clean up in it. + +0:53:11.699,0:53:16.489 +and then later actually after we release +lite BSD in agreement + +0:53:16.489,0:53:18.719 +Uh did a whole lot more work on it. + +0:53:18.719,0:53:22.069 +really were just throwing tons of shit out +that shouldnt have been at first place. + +0:53:22.069,0:53:26.559 +I had this whole notion of an external user +driven page which was nice but the cost + +0:53:26.559,0:53:30.129 +of all the extra context switching other +things that would happen on page was sort of + +0:53:30.129,0:53:32.549 +a killer uh + +0:53:32.549,0:53:39.479 +And so at any rate that's that's how the the +VM system got in there. + +0:53:39.479,0:53:40.420 +Any more questions? + +0:53:40.420,0:53:45.349 +See I can take like five minutes to answer +one question. + +0:53:45.349,0:53:45.599 +All right ! Well Thank You very much. diff --git a/en_US.ISO8859-1/captions/README b/en_US.ISO8859-1/captions/README new file mode 100644 index 0000000000..b7bf9ce68c --- /dev/null +++ b/en_US.ISO8859-1/captions/README @@ -0,0 +1,15 @@ +$FreeBSD$ + +This directory holds closed captions for audio and video recordings of +technical presentations about FreeBSD. + +The expected hierarchy is : + +doc/${LANG}/captions/${YEAR}/${CONFERENCE}/${TALK} + +e.g. + +doc/en_US.ISO8859-1/captions/2009/asiabsdcon/mckusick-kernelinternals.sbv + +Individual captions are currently stored as YouTube 'sbv' files which +are simple ASCII text files with timing and transcription information.