import repository from arizona
[raven.git] / webpage / cleantut.html
1 <html>
2 <head>
3 <title> Stork Project </title>
4 <LINK href="stork.css" rel="stylesheet" type="text/css">
5
6 </head>
7
8 <body>
9
10
11 <div class="display" align="left">
12
13
14                 <td valign="top">
15                 <table class="info" cellpadding="0" width="700" >
16                 <tr  height="75"><td colspan="3"></td></tr>
17                 <tr valign="top" align="left">
18
19                         <td>
20                                 <table cellpadding="8" id="content" class="content">
21                                         <tr>
22                                                 <td>
23
24                                                         <h1><a name = "top"></a>Stork Tutorial</h1>
25                                                         For a simple walkthrough of using Stork to manage package installation on a PlanetLab
26                                                         system, use the following instructions:
27                                                         <p><a href = "#install">1. Install Stork and Storkutil</a>
28                                                         <br><a href = "#tarball">2. Create a package</a>
29                                                         <br><a href = "#genpair">3. Create an identity</a>
30                                                         <br><a href = "#config">4. Modify the Stork configuration</a>
31                                                         <br><a href = "#tpfile">5. Modify your Trusted Packages File</a>
32                                                         <br><a href = "#maninstall">6. Set packages for installation using Pacman, or install manually</a>
33                                                         <br><a href = "#upload">7. Upload files to the Stork repository</a>
34                                                         <p><img src="blkfade.gif" width="100%"></img>
35                                                         <br>
36
37                                                         <a name = "#install"><h3>1. Installing Stork and Storkutil</h3></a>
38                                                         To use Stork on your slice, you must run the Stork initscript from PlanetLab. Log into
39                                                         PlanetLab, choose a slice, and select <b>Add New Attribute</b>. Select
40                                                         <b>pl_initscript</b>, and then select <b>arizona_stork2</b>.
41                                                         <p>After finishing installation, download the <a href="downloads/stork-enduser-tar-latest.tar.gz">Stork Enduser tarball</a> and
42                                                         unpack it to obtain the Stork enduser utilities. This tarball also includes a sample stork
43                                                         configuration file, named sample-stork.conf which will need to be <a href="
44                                                         #config">modified</a> in order
45                                                         to successfully use the stork utilities.
46                                                         <p><img src="blkfade.gif" width="100%"></img>
47                                                         <br>
48
49                                                         <a name = "tarball"><h3>2. Creating a Package</h3></a>
50                                                         <p>Stork currently supports two types of packages: RPMs and tarballs.
51                                                         Stork will install RPM packages according to the package scripts. RPMs
52                                                         are generally suggested for complex installations, or where the package needs
53                                                         to be installed outside of /root. However, for simple packages, it can be more
54                                                         efficient to use a simple tarball package.
55
56                                                         <p>Creating a tarball package can be a great way to distribute your own
57                                                         applications. With Stork, you can easily distribute your software to every
58                                                         node that needs it. Stork also allows you to execute scripts before and
59                                                         after you install or uninstall, making tarballs even more powerful.
60                                                         Including scripts in your tarball is very simple. Just package
61                                                         '.preinstall', '.postinstall', '.preremove', and/or '.postremove' scripts
62                                                         along with your own files. Stork will automatically execute the script when
63                                                         the user decides to install or uninstall your package. The scripts will run
64                                                         from the same directory the tarball is unpacked (typically '/root/' or what
65                                                         ever is defined to be the home directory). Note that these scripts are
66                                                         completely optional.
67                                                         <p>For example, consider a simple directory called <i>directory</i> full
68                                                         of dummy files, and also four scripts, <i>.preinstall,
69                                                         .postinstall, .preremove, .postremove</i>. We can create a
70                                                         simple zipped tarball by running the command:
71
72                                                         <br><code><b>  tar -czf <i>mypackage.tar.gz</i> directory ... </b></code>
73
74                                                         <br>Where "mypackage" is the name of the tarball to be created, and the
75                                                         directory is the directory we wish to make into a package. Instead
76                                                         directories, files can also be used.
77                                                         <p>Upon installing this package with
78                                                         Stork, Stork will automatically run .preinstall, and will run
79                                                         .postinstall after installation. This allows the package creator to
80                                                         include scripts to confirm that installation is going to occur, and confirm
81                                                         that it worked. Likewise, when the package is uninstalled, the preremove and
82                                                         postremove scripts work in the same way.
83                                                         <p><img src="blkfade.gif" width="100%"></img>
84                                                         <br>
85
86                                                         <a name="genpair"><h3>3. Creating an Identity</h3></a>
87                                                         <p>To start using Stork's enduser utilities, you will first need to
88                                                         create an identity. You can do this using the 'genpair' function. This
89                                                         will create a private key and a public key, which will identify you and the
90                                                         software you wish to distribute. The privatekey is used to sign the files
91                                                         that Stork uses to install your software, so be sure to keep it safe. The
92                                                         public key is distributed to all your nodes (and all others that wish to use
93                                                         your software). Note that Stork will generate an error if you attempt to use
94                                                         it before creating a pair of keys and setting it as your default identity.
95                                                         <p>To create your keys, just run the command:
96
97                                                         <br><code><b>./storkutil.py genpair USERNAME</b></code>
98                                                         <br>where USERNAME is any username you desire. The username exsits
99                                                         so that users can recognize each other without having to base interaction
100                                                         solely on public and private keys. After running the genpair command,
101                                                         Stork will ask you if you want this to be your default identity; if this is
102                                                         your first time using Stork, it is best to set this key as your default. If
103                                                         you've already created keys and would like to use those, you can use the
104                                                         'storkutil.py setdefault USERNAME' command.
105
106 <p><table width=400 height=150 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
107 <td width="100%" align="left" valign="top" bgcolor="teal">
108 <span style="color: white">
109 <code><b>./storkutil.py genpair foo</b>
110 <br>Generated foo.privatekey...
111 <br>Extracted foo.publickey...
112 <br>Would you like to make 'foo' your default identity? (y/n) > <b>y</b>
113 <br>'foo' is now your default identity
114 <br>
115 <br><b>./storkutil.py genpair foo</b>
116 <br>Error: Key files for 'foo' already exist
117 </code>
118 </span></td></tr></table>
119
120 <p>or, if you have already created foo.privatekey and foo.publickey:
121
122 <p><table width=400 height=60 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
123 <td width="100%" align="left" valign="top" bgcolor="teal">
124 <span style="color: white">
125 <code><b>./storkutil.py setdefault foo</b>
126 <br>Previous Identity: someguy
127 <br>'foo' is now your default identity</code>
128 </span></td></tr></table>
129                                                         <p>Be sure that the privatekey and publickey are in your current directory
130                                                         while using storkutil. The publickey can be distributed to all your nodes
131                                                         through the <a href="https://stork-repository.cs.arizona.edu">upload website</a>.
132                                                         Also, note that your public key is embedded in your TPFile filename, such that
133                                                         the filename should be <i>*publickey*Username.tpfile</i>, without the *.
134                                                         <p><img src="blkfade.gif" width="100%"></img>
135                                                         <br>
136                                                         <a name = "config"><h3>4. Configuration File</h3></a>
137                                                         You will need to change the stork configuration file included in the
138                                                         Storkutil tarball to match the
139                                                         identity you created. Simply open up the file which should be named
140                                                         <i>sample-stork.conf</i> and change the 'username = ' line
141                                                         to match the username you chose for your keys. You will also need to modify the
142                                                         'publickeyfile = ' line, to match the location of your publickey. By default, this should be
143                                                         "publickeyfile = /usr/local/stork/var/keys/<username>.publickey", where <username> is the
144                                                         name you generated your keys for.
145                                                         You will need to distribute this
146                                                         configuration file to all your
147                                                         nodes through the <a href="https://stork-repository.cs.arizona.edu">upload website</a>.
148                                                         <i><p>Note!</i> The stork configuration file is different from the storkutil
149                                                         configuration file. The <code>setdefault</code> command will appropriately change
150                                                         the storkutil configuration file, but you will still need to change and distribute
151                                                         the Stork configuration file manually.
152                                                         <p><img src="blkfade.gif" width="100%"></img>
153                                                         <br>
154                                                         <a name = "tpfile"><h3>5. Trusted Packages File</h3></a>
155                                                         The Trusted Packages File (TPFILE) defines which files you trust to be autmoatically
156                                                         downloaded and install. This helps your nodes stay secure and autonomous, as files will be
157                                                         verified through a set of hash codes. Even if
158                                                         two files share the same name, your nodes will only recieve the one that
159                                                         matches the hash obtained when the Trusted Packages File was created
160                                                         (preventing your nodes from downloading maliciously modified packages in the
161                                                         repository). You are also allowed to trust other users, allowing your nodes
162                                                         to accept packages from those users exclusively. This will let your nodes
163                                                         download and install applications created by other users. Note that adding packages to
164                                                         your TPFILE will not download the packages; it just notifies Stork which packages
165                                                         you trust to be downloaded.
166
167                                                         <p><b>Adding a package to your tpfile:</b>
168                                                         <br>To add a package to your tpfile, use the command
169                                                         <br><b><code>./storkutil.py
170                                                         addfile FILE [FILE]...</b></code>
171                                                         <br>where FILE is the location of the package you would
172                                                         like to trust. If the tpfile does not exist, storkutil will create one for
173                                                         you. Storkutil will look at the file, create a hash, and write it to your
174                                                         tpfile. If you <a href="https://stork-repository.cs.arizona.edu">upload</a>
175                                                         the package to the repository and run
176                                                         stork to install the package, your node will select your specific file and
177                                                         install it.
178                                                                 <p>For example, we will add packages 'package1' and 'package2'.
179 <p><table width=400 height=60 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
180 <td width="100%" align="left" valign="top" bgcolor="teal">
181 <span style="color: white">
182                                                                 <code><b>./storkutil.py addfile package1.tar.gz package2.tar.gz</b>
183                                                                 <br>No tpfile found for foo. Generating new tpfile.
184                                                                 <br>File 'foo.tpfile' successfully signed
185                                                                 <br>File 'foo.tpfile' has been copied to 'foo.~~~.tpfile'
186                                                                 <br>Unembeded trusted packages file 'foo.tpfile' removed.</code>
187 </span></td></tr></table>
188
189
190                                                                 <br>This script will run whenever Stork fails to find a trusted package
191                                                                 file for the default user. It will then continue to the regular package adding script:
192
193 <p><table width=400 height=60 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
194 <td width="100%" align="left" valign="top" bgcolor="teal">
195 <span style="color: white">
196                                                                 <code>Using trustedpackages file '~~~~.tpfile'
197                                                                 <br>Successfully added file 'package1.tar.gz'.
198                                                                 <br>Successfully added file 'package2.tar.gz'.
199                                                                 <br>File '~~~.tpfile' successfully signed</code>
200
201 </span></td></tr></table>
202                                                                 <br>Storkutil will automatically sign your trusted packages file using
203                                                                 your key. Also note that your TPFile will be named as your public key followed by
204                                                                 your username, with the .tpfile extension.
205
206
207                                                         <p><b>Viewing your tpfile:</b>
208                                                                 <br>Afterwards, we can look at the trusted packages file by running:
209 <p><table width=400 height=60 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
210 <td width="100%" align="left" valign="top" bgcolor="teal">
211 <span style="color: white">
212                                                                 <code><b>./storkutil.py view tpfile</b>
213                                                                 <br>Viewing: foo.~~.tpfile
214                                                                 <br><br>TRUSTED PACKAGES FILE
215                                                                 <br>    ALLOW package1.tar.gz
216                                                                 <br>    ALLOW package2.tar.gz</code>
217
218 </span></td></tr></table>
219
220
221                                                                 <br>We can confirm that the packages were added to our trusted packages
222                                                                 file in this way.
223
224                                                         <p><b>Removing a package from the tpfile:</b>
225                                                         <br>You can remove a file from your TPFile, just use the 'removefile' command.
226                                                         <p><code><b> ./storkutil.py removefile package1.tar.gz</b>
227
228                                                         <p><b>Adding and removing trusted users:</b>
229                                                         <br>Before adding a user to your TPFile, you will need to obtain their
230                                                         publickey.
231                                                         <br>To trust users, you can use the <code>adduser</code> command:
232                                                         <br><code><b>storkutil adduser USER USER.publickey (allow|deny|any) AFFECTEDPACKAGES</b></code>
233                                                         <br>Where USER is the name of the user you want to trust, followed by the
234                                                         public key that user has made available. AFFECTEDPACKAGES is a pattern
235                                                         that defines the packages you want to accept. If the tpfile does not exist,
236                                                         storkutil will create one for you.
237                                                         <p>Examples:
238                                                         <br><b><code>storkutil.py adduser USER1 USER1.publickey allow user-4-2.tar.gz
239                                                         <br>storkutil.py adduser USER2 USER2.publickey deny java\*
240                                                         <br>storkutil.py adduser PlanetLab PlanetLab.publickey allow \*
241                                                         <br>storkutil.py adduser USER3</b></code>
242
243                                                         <p>The first example shows how to accept a single package from the user
244                                                         USER1. The second example will deny all files from USER2 that have the
245                                                         pattern java*. The third example shows how to accept all packages from the
246                                                         user PlanetLab (be sure to shell escape the * character instead of using * by itself).
247                                                         Note that this is not necessary; the PlanetLab node is <b>trusted by default</b> when
248                                                         a tpfile is created to allow default PlanetLab files to be installed,
249                                                         and the PlanetLab publickey is included in the enduser tarball.
250                                                         You can again use the <code>view tpfile</code>
251                                                         command to see the contents of your TPFILE. You can also remove users using
252                                                         the <code>removeuser</code> command.
253
254                                                                 <p><b><code>./storkutil.py removeuser USER2</code></b>
255
256                                                         <p>When you are done, you can upload the trusted packages file to your nodes
257                                                         through the <a href="https://stork-repository.cs.arizona.edu">upload
258                                                         website.</a>
259                                                         <p><img src="blkfade.gif" width="100%"></img>
260                                                         <br>
261                                                         <a name = "pacman"><h3>6b. Pacman</h3></a>
262                                                         Stork's package manager (pacman) can help you manage your packages
263                                                         across all your nodes. Rather than installing or uninstalling packages on
264                                                         each of your nodes individually, you can assign your nodes to groups and
265                                                         instruct stork to install or remove packages on those groups. Pacman
266                                                         is therefore efficient to use when a package needs to be installed on
267                                                         a large number of nodes. Pacman also allows packages to be automatically
268                                                         updated when the package creator updates them in the repository; this
269                                                         allows Stork to work near autonomously when set up for use.
270                                                         Pacman uses two files; groups and packages. Using storkutil, you can create these two
271                                                         files and upload them. Stork will then distribute these files to your nodes
272                                                         and perform the tasks you assigned.
273
274                                                         <p><b>Pacman Groups</b>
275                                                         <br>Your groups.pacman file allows you organize your nodes into separate
276                                                         groups. You are allowed to include a node into a group or exclude a node
277                                                         from a group. You can also include, exclude, or intersect groups with each
278                                                         other. To define our groups, we can use:
279                                                         <p><code><b>./storkutil pacgroups (include|exclude|intersect|remove) group NAME [NAME...]</b></code>
280                                                         <br>where <code>group</code> is the name of the group to be affected, and
281                                                         <code>NAME</code> are the names of the nodes or groups to be included/excluded/removed.
282                                                         <p>We start by adding a node to a group. If a pacman group file for your
283                                                         username does not exist, then storkutil will create a new one.
284
285
286 <p><table width=400 height=200 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
287 <td width="100%" align="left" valign="top" bgcolor="teal">
288 <span style="color: white">
289 <code><b>./storkutil.py pacgroups include myGroup foo1.node foo2.node</b>
290 <br>Using groups file: ./foo.~~~.groups.pacman
291 <br>File Not Found, Creating New File
292 <br>File './foo.~~~.groups.pacman' successfully signed</code>
293 <p><code><b>./storkutil.py view groups</b>
294 <br>Viewing: foo.~~~.groups.pacman
295 <br>
296 <br>PACMAN GROUP FILE
297     <br>    GROUP:  myGroup
298          <br>       INCLUDE:        foo1.node
299              <br>   INCLUDE:        foo2.node
300 </code></span></td></tr></table>
301
302
303
304                                                         <p>Storkutil will automatically sign the file using your default private key
305                                                         file, which you created when you <a href="#genpair">generated your first key pairs</a>. If
306                                                         you want to use a username that is not your default, you can use the <code>--username</code>
307                                                         option to work on a different file.
308                                                         <br>We can continue adding more data
309                                                         into our group definitions. For instance, if we want myGroup to include all the
310                                                         nodes in the group oldGroup, except for the node badNode which is a member of oldGroup,
311                                                         we can run the following commands:
312                                                         <p><b><code>./storkutil.py pacgroups include myGroup oldGroup
313                                                         <br>./storkutil.py pacgroups exclude myGroup badNode</b></code>
314                                                         <br>This means that installing a package to myGroup will install it to every node
315                                                         included in oldGroup except for badNode.
316                                                         <p>We can remove groups or nodes from a group by simply using the <code>remove</code>
317                                                         command, as such:
318                                                         <p><code><b>./storkutil.py pacgroups remove myGroup</b>
319
320
321                                                         <p>If you want to
322                                                         view the contents of the group file, you can use the <code>view groups</code> command in
323                                                         storkutil, but you will first need to add a node to a group to generate a group file.
324                                                         <br>When you are done, you will need to upload the groups.pacman file to the Stork repository.
325
326                                                         <b><p>Pacman Packages</b>
327                                                         <br>The packages.pacman file allows you to manage which packages to
328                                                         install on which nodes. With the pacman packages file, you can tell a node
329                                                         or group (as defined by your <a href = "#pacman">groups.pacman file</a>) to install or
330                                                         uninstall a package. To create and modify our packages file, we will use:
331
332                                                         <p><table width=400 height=50 cellspacing=0 cellpadding=5 border=5 bordercolor="orange"><tr>
333                                                         <td width="100%" align="left" valign="top" bgcolor="teal">
334                                                         <span style="color: white">
335                                                         <code><b><code><b>./storkutil.py pacpackages (node|group|all) NAME (install|remove|upgrade) PACKAGE</b></code>
336 </span></td></tr></table>
337
338
339                                                         <p>We can start by assigning a node to install a package. Be sure your
340                                                         filename follows the pattern <name>.packages.pacman, where the name is the
341                                                         one your configured in your <a href="#config">Stork configuration file</a>. This will aid
342                                                         your nodes in finding the correct group file to use. If the file you
343                                                         specified does not exist, then storkutil will create a new one.
344
345                                                         <p><b><code>./storkutil.py pacpackages node planetlab.foo.edu install gnuchess</code></b>
346                                                         <br>This will assign the planetlab.foo.edu node to install gnuchess.
347                                                         <p>We can also assign groups to install or remove packages:
348
349                                                         <br><b><code>./storkutil.py pacpackages group myGroup install lynx man gnuchess</b></code>
350                                                         <br>This command will tell myGroup to install the <i>lynx, man, and gnuchess</i> packages
351                                                         in all the nodes in the group.
352
353                                                         <p>We can also instruct Pacman to install packages on multiple nodes without grouping them,
354                                                         using the <code>all</code> command:
355                                                         <br><b><code>./storkutil.py pacpackages all install gnuchess</b></code>
356                                                         <br>Note that in this example, the <code>group</code> and <code>node</code> commands are replaced
357                                                         with <code>all</code> command, and also that there is no group or node name required. This command
358                                                         will instruct Stork to install gnuchess on every node to which the packages.pacman file is distributed.
359
360                                                         <p>Using the <code>upgrade</code> command, Stork will also instruct all nodes in a group
361                                                         to update a preexisting package:
362                                                         <br><code><b>./storkutil.py pacpackages group myGroup upgrade gnuchess</b></code>
363                                                         <br>This will instruct all myGroup nodes to upgrade gnuchess when necessary.
364
365                                                         <p>Storkutil will automatically sign the file using your default private key
366                                                         file, which you created when you <a href = "#genpair">generated</a> your first key pairs.
367                                                         If you want to work using a different set of keys and files, you can use the
368                                                         <code>--username</code> option.
369                                                         You can view the contents of your package file using the <code>view packages</code>
370                                                         command. When you are done,
371                                                         <a href="https://stork-repository.cs.arizona.edu">
372                                                         upload your Pacman config file to the Stork repository.</a>
373                                                         <p><img src="blkfade.gif" width="100%"></img>
374                                                         <br>
375                                                         <a name = "maninstall"><h3>6a. Manual Install</h3></a>
376                                                         To manually install a package on a node, simply run the command <code>Stork</code>
377                                                         followed by the package name, from root. For instance:
378                                                         <br><b><code>./stork.py gnuchess</b></code>
379                                                         <br>will install gnuchess on the node on which it is run. To install gnuchess
380                                                         on three separate nodes, then, the user would need to log into each node
381                                                         separately and run <code>Stork gnuchess</code> on that node. This is an efficient way
382                                                         to install packages when the package only needs to be installed on a single or small
383                                                         number of nodes.
384                                                         <p><img src="blkfade.gif" width="100%"></img>
385                                                         <br>
386                                                         <a name="upload"><h3>7. Uploading to the Repository</h3></a>
387                                                         The Stork repository upload site is located at:
388                                                         <a href="https://stork-repository.cs.arizona.edu">https://stork-repository.cs.arizona.edu</a> .
389                                                         From this site, you can upload <a href="#tarball">packages</a>,
390                                                          your <a href="#genpair">public keys</a>, your
391                                                         <a href="#config">Stork configuration file</a>,
392                                                         <a href="#tpfile">trusted packages files</a>, and <a href="#pacman">pacman files
393                                                         (both groups.pacman and packages.pacman files)</a>. All of these files will need
394                                                         to be uploaded to the repository in order for Stork to work correctly with your
395                                                         nodes.
396
397
398
399                                                 </td>
400                                         </tr>
401                                 </table>
402                         </td>
403                 </tr>
404
405        </table>
406        </td>
407
408      </tr>
409      <tr>
410         <td></td>
411         <td>
412         </td>
413      </tr>
414
415 </table>
416
417 </div>
418
419
420 <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
421 </script>
422 <script type="text/javascript">
423 _uacct = "UA-1868232-1";
424 urchinTracker();
425 </script>
426 </body>
427 </html>