import repository from arizona
[raven.git] / webpage / storkutil.html
1 <html>
2 <head>
3 <title> Stork Project </title>
4 <LINK href="stork.css" rel="stylesheet" type="text/css">
5
6 </head>
7
8 <body>
9
10
11 <div style="position: relative; top: 15px;" class="display" align="center">
12         <table border="0">
13         <tr>
14                 <td width="170" valign="top">
15
16                                 <br/>
17                                 <table cellpadding="3" width="170" id="links" class="links">
18                                         <tr>
19                                                 <td align="right">
20
21                                                         <ul class="links">
22
23                                                                 <a href = "index.html"><h3>Stork</h3></a>
24                                                                 <li class="links"><a href="util.html">Using Storkutil</a></i>
25                                                                 <li class="links"><a href="storkclient.html">Using Stork Client</a></li>
26                                                                 <li class="links"><a href="storkutil.html">Storkutil Runthrough</a></li>
27                                                                 <li class="links"><a href="arch.html">The Stork Architecture</a></li>
28                                                                 <li class="links"><a href="filelist.html">Stork File List</a></li>
29                                                                 <li class="links"><a href="about.html">About Us</a></li>
30                                                                 <li class="links"><a href="apps.html">Related Links</a></li>
31                                                                 <li class="links"><a href="http://appmanager.berkeley.intel-research.net/plcontrol/apps.php?appid=1029">
32                                                                 PlanetLab Slice Status</a></li>
33                                                                 <li class="links"><a href="http://nr06.cs.arizona.edu/testphp/upload.php?">Stork Repository
34                                                                 </a></li>
35                                                                 <li class="links"><a href="mailto:stork@cs.arizona.edu">Contact Stork</a></link>
36                                                                 <br><p><li class="links"><a href="http://www.arizona.edu">University of Arizona</a></li>
37                                                                 <li class="links"><a href="http://www.planet-lab.org">PlanetLab</a></li>
38
39
40
41                                                         </ul>
42
43                                                  </td>
44                                         </tr>
45                                 </table>
46                                 <br/>
47                 </td>
48
49                 <td valign="top">
50                 <table class="info" cellpadding="0" width="700" >
51                 <tr  height="75"><td colspan="3"><img style="margin-left: -0px;" src="images/stork-header.png" alt="stork logo"/></td></tr>
52                 <tr  bgcolor="#444444" class="headerrow" width="100%" height="2">
53
54                         <th colspan="2"> Storkutil Runthrough </th>
55
56                 </tr>
57                 <tr valign="top" align="left">
58
59                         <td>
60                                 <table cellpadding="8" id="content" class="content">
61                                         <tr>
62                                                 <td>
63
64                                                         <h1><a name = "top">Using Stork</a></h1>
65                                                         For a simple rundown of using Stork to manage package installation on a PlanetLab
66                                                         system, use the following instructions:
67                                                         <p><a href = "#install">1. Install Stork and Storkutil</a>
68                                                         <br><a href = "#tarball">2. Create a package</a>
69                                                         <br><a href = "#genpair">3. Create an identity</a>
70                                                         <br><a href = "#config">4. Modify the Stork configuration</a>
71                                                         <br><a href = "#tpfile">5. Modify your Trusted Packages File</a>
72                                                         <br><a href = "#maninstall">6. Install the packages manually or with Pacman</a>
73                                                         <br><a href = "#upload">7. Upload files to the Stork repository</a>
74                                                         <p><img src="blkfade.gif" width="100%"></img>
75                                                         <br>
76
77                                                         <a name = "#install"><h3>1. Installing Stork and Storkutil</h3></a>
78                                                         To use Stork on your slice, you must run the Stork initscript from PlanetLab. Log into
79                                                         PlanetLab, choose a slice, and select <b>Add New Attribute</b>. Select
80                                                         <b>pl_initscript</b>, and then select <b>arizona_stork2</b>.
81                                                         <p>After finishing installation, download the <a href="downloads/stork-enduser-tar-latest.tar.gz">Stork Enduser tarball</a> and
82                                                         unpack it to obtain the Stork enduser utilities. This tarball also includes a sample stork
83                                                         configuration file, named sample-stork.conf which will need to be <a href="
84                                                         #config">modified</a> in order
85                                                         to successfully use the stork utilities.
86                                                         <p><img src="blkfade.gif" width="100%"></img>
87                                                         <br>
88
89                                                         <a name = "tarball"><h3>2. Creating a Package</h3></a>
90                                                         <p>Stork currently supports two types of packages: RPMs and tarballs.
91                                                         Stork will install RPM packages according to the package scripts. RPMs
92                                                         are generally suggested for complex installations, or where the package needs
93                                                         to be installed outside of /root. However, for simple packages, it can be more
94                                                         efficient to use a simple tarball package.
95
96                                                         <p>Creating a tarball package can be a great way to distribute your own
97                                                         applications. With Stork, you can easily distribute your software to every
98                                                         node that needs it. Stork also allows you to execute scripts before and
99                                                         after you install or uninstall, making tarballs even more powerful.
100                                                         Including scripts in your tarball is very simple. Just package
101                                                         '.preinstall', '.postinstall', '.preremove', and/or '.postremove' scripts
102                                                         along with your own files. Stork will automatically execute the script when
103                                                         the user decides to install or uninstall your package. The scripts will run
104                                                         from the same directory the tarball is unpacked (typically '/root/' or what
105                                                         ever is defined to be the home directory). Note that these scripts are
106                                                         completely optional.
107                                                         <p>For example, consider a simple directory called <i>directory</i> full
108                                                         of dummy files, and also four scripts, <i>.preinstall,
109                                                         .postinstall, .preremove, .postremove</i>. We can create a
110                                                         simple zipped tarball by running the command:
111
112                                                         <br><code><b>  tar -czf <i>mypackage.tar.gz</i> directory ... </b></code>
113
114                                                         <br>Where "mypackage" is the name of the tarball to be created, and the
115                                                         directory is the directory we wish to make into a package. Instead
116                                                         directories, files can also be used.
117                                                         <p>Upon installing this package with
118                                                         Stork, Stork will automatically run .preinstall, and will run
119                                                         .postinstall after installation. This allows the package creator to
120                                                         include scripts to confirm that installation is going to occur, and confirm
121                                                         that it worked. Likewise, when the package is uninstalled, the preremove and
122                                                         postremove scripts work in the same way.
123                                                         <p><img src="blkfade.gif" width="100%"></img>
124                                                         <br>
125
126                                                         <a name="genpair"><h3>3. Creating an Identity</h3></a>
127                                                         <p>To start using Stork's enduser utilities, you will first need to
128                                                         create an identity. You can do this using the 'genpair' function. This
129                                                         will create a private key and a public key, which will identify you and the
130                                                         software you wish to distribute. The privatekey is used to sign the files
131                                                         that Stork uses to install your software, so be sure to keep it safe. The
132                                                         public key is distributed to all your nodes (and all others that wish to use
133                                                         your software). Note that Stork will generate an error if you attempt to use
134                                                         it before creating a pair of keys and setting it as your default identity.
135                                                         <p>To create your keys, just run the command:
136
137                                                         <br><code><b>./storkutil.py genpair USERNAME</b></code>
138                                                         <br>where USERNAME is any username you desire. The username exsits
139                                                         so that users can recognize each other without having to base interaction
140                                                         solely on public and private keys. After running the genpair command,
141                                                         Stork will ask you if you want this to be your default identity; if this is
142                                                         your first time using Stork, it is best to set this key as your default. If
143                                                         you've already created keys and would like to use those, you can use the
144                                                         'storkutil.py setdefault USERNAME' command.
145                                                                 <p><code><b>./storkutil.py genpair foo</b>
146                                                                 <br><i>Would you like to make 'foo' your default identity?</i> y</code>
147
148                                                                 <p>or, if you have already created me.privatekey and me.publickey:
149
150                                                                 <p><code><b> ./storkutil.py setdefault me</b>
151                                                                 <br><i>Previous Identity: foo
152                                                                 <br>'me' is now your default identity.</i></code>
153                                                         <p>Be sure that the privatekey and publickey are in your current directory
154                                                         while using storkutil. The publickey can be distributed to all your nodes
155                                                         through the <a href="http://nr06.cs.arizona.edu/testphp/upload.php">upload website</a>.
156                                                         Also, note that your public key is embedded in your TPFile filename, such that
157                                                         the filename should be <i>*publickey*Username.tpfile</i>, without the *.
158                                                         <p><img src="blkfade.gif" width="100%"></img>
159                                                         <br>
160                                                         <a name = "config"><h3>4. Configuration File</h3></a>
161                                                         You will need to change the stork configuration file included in the
162                                                         Storkutil tarball to match the
163                                                         identity you created. Simply open up the file which should be named
164                                                         <i>sample-stork.conf</i> and change the 'username = ' line
165                                                         to match the username you chose for your keys. You will also need to modify the
166                                                         'publickeyfile = ' line, to match the location of your publickey. By default, this should be
167                                                         "publickeyfile = /usr/local/stork/var/keys/<username>.publickey", where <username> is the
168                                                         name you generated your keys for.
169                                                         You will need to distribute this
170                                                         configuration file to all your
171                                                         nodes through the <a href="http://nr06.cs.arizona.edu/testphp/upload.php">upload website</a>.
172                                                         <i><p>Note!</i> The stork configuration file is different from the storkutil
173                                                         configuration file. The <code>setdefault</code> command will appropriately change
174                                                         the storkutil configuration file, but you will still need to change and distribute
175                                                         the Stork configuration file manually.
176                                                         <p><img src="blkfade.gif" width="100%"></img>
177                                                         <br>
178                                                         <a name = "tpfile"><h3>5. Trusted Packages File</h3></a>
179                                                         The Trusted Packages File (TPFILE) defines which files you trust to be autmoatically
180                                                         downloaded and install. This helps your nodes stay secure and autonomous, as files will be
181                                                         verified through a set of hash codes. Even if
182                                                         two files share the same name, your nodes will only recieve the one that
183                                                         matches the hash obtained when the Trusted Packages File was created
184                                                         (preventing your nodes from downloading maliciously modified packages in the
185                                                         repository). You are also allowed to trust other users, allowing your nodes
186                                                         to accept packages from those users exclusively. This will let your nodes
187                                                         download and install applications created by other users. Note that adding packages to
188                                                         your TPFILE will not download the packages; it just notifies Stork which packages
189                                                         you trust to be downloaded.
190
191                                                         <p><b>Adding a package to your tpfile:</b>
192                                                         <br>To add a package to your tpfile, use the command
193                                                         <br><b><code>./storkutil.py
194                                                         addfile FILE [FILE]...</b></code>
195                                                         <br>where FILE is the location of the package you would
196                                                         like to trust. If the tpfile does not exist, storkutil will create one for
197                                                         you. Storkutil will look at the file, create a hash, and write it to your
198                                                         tpfile. If you <a href="http://nr06.cs.arizona.edu/testphp/upload.php">upload</a>
199                                                         the package to the repository and run
200                                                         stork to install the package, your node will select your specific file and
201                                                         install it.
202                                                                 <p>For example, we will add packages 'package1' and 'package2'.
203                                                                 <p><code><b>./storkutil.py addfile package1.tar.gz package2.tar.gz</b>
204                                                                 <br>No tpfile found for foo. Generating new tpfile.
205                                                                 <br>File 'foo.tpfile' successfully signed
206                                                                 <br>File 'foo.tpfile' has been copied to 'foo.~~~.tpfile'
207                                                                 <br>Unembeded trusted packages file 'foo.tpfile' removed.</code>
208                                                                 <br>This script will run whenever Stork fails to find a trusted package
209                                                                 file for the default user. It will then continue to the regular package adding script:
210
211                                                                 <p><code>Using trustedpackages file '~~~~.tpfile'
212                                                                 <br>Successfully added file 'package1.tar.gz'.
213                                                                 <br>Successfully added file 'package2.tar.gz'.
214                                                                 <br>File '~~~.tpfile' successfully signed</code>
215                                                                 <br>Storkutil will automatically sign your trusted packages file using
216                                                                 your key. Also note that your TPFile will be named as your public key followed by
217                                                                 your username, with the .tpfile extension.
218
219                                                         <p><b>Viewing your tpfile:</b>
220                                                                 <br>Afterwards, we can look at the trusted packages file by running:
221                                                                 <code><b>./storkutil.py view tpfile</b>
222                                                                 <br>Viewing: foo.~~.tpfile
223                                                                 <br><br>TRUSTED PACKAGES FILE
224                                                                 <br>    ALLOW package1.tar.gz
225                                                                 <br>    ALLOW package2.tar.gz</code>
226                                                                 <br>We can confirm that the packages were added to our trusted packages
227                                                                 file in this way.
228
229                                                         <p><b>Removing a package from the tpfile:</b>
230                                                         <br>You can remove a file from your TPFile, just use the 'removefile' command.
231                                                         <p><code><b> ./storkutil.py removefile package1.tar.gz</b>
232
233                                                         <p><b>Adding and removing trusted users:</b>
234                                                         <br>Before adding a user to your TPFile, you will need to obtain their
235                                                         publickey. The user you wish to add needs to upload their publickey to the
236                                                         Stork repository, which you can then download.
237                                                         <br>To trust users, you can use the <code>adduser</code> command:
238                                                         <br><code><b>storkutil adduser USER USER.pubkey (allow|deny|any) AFFECTEDPACKAGES</b></code>
239                                                         <br>Where USER is the name of the user you want to trust, followed by the
240                                                         public key that user has made available. AFFECTEDPACKAGES is a pattern
241                                                         that defines the packages you want to accept. If the tpfile does not exist,
242                                                         storkutil will create one for you.
243                                                         <p>Examples:
244                                                         <br><b><code>storkutil.py adduser USER1 USER1.publickey allow user-4-2.tar.gz
245                                                         <br>storkutil.py adduser USER2 USER2.publickey deny java\*
246                                                         <br>storkutil.py adduser PlanetLab PlanetLab.publickey allow \*
247                                                         <br>storkutil.py adduser USER3</b></code>
248
249                                                         <p>The first example shows how to accept a single package from the user
250                                                         USER1. The second example will deny all files from USER2 that have the
251                                                         pattern java*. The third example shows how to accept all packages from the
252                                                         user PlanetLab (be sure to shell escape the * character instead of using * by itself).
253                                                         Note that this is not necessary; the PlanetLab node is <b>trusted by default</b> when
254                                                         a tpfile is created to allow default PlanetLab files to be installed,
255                                                         and the PlanetLab publickey is included in the enduser tarball.
256                                                         You can again use the <code>view tpfile</code>
257                                                         command to see the contents of your TPFILE. You can also remove users using
258                                                         the <code>removeuser</code> command.
259
260                                                                 <p><code>./storkutil.py removeuser USER2</code>
261
262                                                         <p>When you are done, you can upload the trusted packages file to your nodes
263                                                         through the <a href="http://nr06.cs.arizona.edu/testphp/upload.php">upload
264                                                         website.</a>
265                                                         <p><img src="blkfade.gif" width="100%"></img>
266                                                         <br>
267                                                         <a name = "maninstall"><h3>6a. Manual Install</h3>
268                                                         To manually install a package on a node, simply run the command <code>Stork</code>
269                                                         followed by the package name, from root. For instance:
270                                                         <br><b><code>./stork.py gnuchess</b></code>
271                                                         <br>will install gnuchess on the node on which it is run. To install gnuchess
272                                                         on three separate nodes, then, the user would need to log into each node
273                                                         separately and run <code>Stork gnuchess</code> on that node. This is an efficient way
274                                                         to install packages when the package only needs to be installed on a single or small
275                                                         number of nodes.
276                                                         <p><img src="blkfade.gif" width="100%"></img>
277                                                         <br>
278                                                         <a name = "pacman"><h3>6b. Pacman</h3></a>
279                                                         Stork's package manager (pacman) can help you manage your packages
280                                                         across all your nodes. Rather than installing or uninstalling packages on
281                                                         each of your nodes individually, you can assign your nodes to groups and
282                                                         instruct stork to install or remove packages on those groups. Pacman
283                                                         is therefore efficient to use when a package needs to be installed on
284                                                         a large number of nodes. Pacman also allows packages to be automatically
285                                                         updated when the package creator updates them in the repository; this
286                                                         allows Stork to work near autonomously when set up for use.
287                                                         Pacman uses two files; groups and packages. Using storkutil, you can create these two
288                                                         files and upload them. Stork will then distribute these files to your nodes
289                                                         and perform the tasks you assigned.
290
291                                                         <p><b>Pacman Groups</b>
292                                                         <br>Your groups.pacman file allows you organize your nodes into separate
293                                                         groups. You are allowed to include a node into a group or exclude a node
294                                                         from a group. You can also include, exclude, or intersect groups with each
295                                                         other. To define our groups, we can use:
296                                                         <p><code><b>./storkutil pacgroups (include|exclude|intersect|remove) group NAME [NAME...]</b></code>
297                                                         <br>where <code>group</code> is the name of the group to be affected, and
298                                                         <code>NAME</code> are the names of the nodes or groups to be included/excluded/removed.
299                                                         <p>We start by adding a node to a group. If a pacman group file for your
300                                                         username does not exist, then storkutil will create a new one.
301                                                         <p><code><b>storkutil pacgroups include myGroup planetlab.foo.edu</b></code>
302                                                         <p>Storkutil will automatically sign the file using your default private key
303                                                         file, which you created when you <a href="#genpair">generated your first key pairs</a>. If
304                                                         you want to use a username that is not your default, you can use the <code>--username</code>
305                                                         option to work on a different file.
306                                                         <br>We can continue adding more data
307                                                         into our group definitions. For instance, if we want myGroup to include all the
308                                                         nodes in the group oldGroup, except for the node badNode which is a member of oldGroup,
309                                                         we can run the following commands:
310                                                         <p><b><code>./storkutil.py pacgroups include myGroup oldGroup
311                                                         <br>./storkutil.py pacgroups exclude myGroup badNode</b></code>
312                                                         <br>This means that installing a package to myGroup will install it to every node
313                                                         included in oldGroup except for badNode.
314                                                         <p>We can remove groups or nodes from a group by simply using the <code>remove</code>
315                                                         command, as such:
316                                                         <p><code><b>./storkutil.py pacgroups remove myGroup</b>
317
318
319                                                         <p>If you want to
320                                                         view the contents of the group file, you can use the <code>view groups</code> command in
321                                                         storkutil, but you will first need to add a node to a group to generate a group file.
322                                                         <br>When you are done, you will need to upload the groups.pacman file to the Stork repository.
323
324                                                         <b><p>Pacman Packages</b>
325                                                         <br>The packages.pacman file allows you to manage which packages to
326                                                         install on which nodes. With the pacman packages file, you can tell a node
327                                                         or group (as defined by your <a href = "#pacman">groups.pacman file</a>) to install or
328                                                         uninstall a package. To create and modify our packages file, we will use:
329                                                         <br><code><b>./storkutil.py pacpackages (node|group|all) NAME (install|remove|upgrade) PACKAGE [PACKAGE]</b></code>
330
331                                                         <p>We can start by assigning a node to install a package. Be sure your
332                                                         filename follows the pattern <name>.packages.pacman, where the name is the
333                                                         one your configured in your <a href="#config">Stork configuration file</a>. This will aid
334                                                         your nodes in finding the correct group file to use. If the file you
335                                                         specified does not exist, then storkutil will create a new one.
336
337                                                         <p><b><code>./storkutil.py pacpackages node planetlab.foo.edu install gnuchess</code></b>
338                                                         <br>This will assign the planetlab.foo.edu node to install gnuchess.
339                                                         <p>We can also assign groups to install or remove packages:
340
341                                                         <br><b><code>./storkutil.py pacpackages group myGroup install lynx man gnuchess</b></code>
342                                                         <br>This command will tell myGroup to install the <i>lynx, man, and gnuchess</i> packages
343                                                         in all the nodes in the group.
344
345                                                         <p>We can also instruct Pacman to install packages on multiple nodes without grouping them,
346                                                         using the <code>all</code> command:
347                                                         <br><b><code>./storkutil.py pacpackages all install gnuchess</b></code>
348                                                         <br>Note that in this example, the <code>group</code> and <code>node</code> commands are replaced
349                                                         with <code>all</code> command, and also that there is no group or node name required. This command
350                                                         will instruct Stork to install gnuchess on every node to which the packages.pacman file is distributed.
351
352                                                         <p>Using the <code>upgrade</code> command, Stork will also instruct all nodes in a group
353                                                         to update a preexisting package:
354                                                         <br><code><b>./storkutil.py pacpackages group myGroup upgrade gnuchess</b></code>
355                                                         <br>This will instruct all myGroup nodes to upgrade gnuchess when necessary.
356
357                                                         <p>Storkutil will automatically sign the file using your default private key
358                                                         file, which you created when you <a href = "#genpair">generated</a> your first key pairs.
359                                                         If you want to work using a different set of keys and files, you can use the
360                                                         <code>--username</code> option.
361                                                         You can view the contents of your package file using the <code>view packages</code>
362                                                         command. When you are done,
363                                                         <a href="http://nr06.cs.arizona.edu/testphp/upload.php?type=pacman">
364                                                         upload your Pacman config file to the Stork repository.</a>
365                                                         <p><img src="blkfade.gif" width="100%"></img>
366                                                         <br>
367                                                         <a name="upload"><h3>7. Uploading to the Repository</h3></a>
368                                                         The Stork repository upload site is located at:
369                                                         <a href="http://nr06.cs.arizona.edu/testphp/upload.php">http://nr06.cs.arizona.edu/testphp/upload.php</a> .
370                                                         From this site, you can upload <a href="#tarball">packages</a>,
371                                                          your <a href="#genpair">public keys</a>, your
372                                                         <a href="#config">Stork configuration file</a>,
373                                                         <a href="#tpfile">trusted packages files</a>, and <a href="#pacman">pacman files
374                                                         (both groups.pacman and packages.pacman files)</a>. All of these files will need
375                                                         to be uploaded to the repository in order for Stork to work correctly with your
376                                                         nodes.
377
378                                                         <center><a href = "index.html">Home</a>     <a href = "#top">Top</a></center>
379
380
381                                                 </td>
382                                         </tr>
383                                 </table>
384                         </td>
385                 </tr>
386
387        </table>
388        </td>
389
390      </tr>
391      <tr>
392         <td></td>
393         <td>
394                 <a href="http://www.planet-lab.org"><img style="border: 0px; border-style: none;" src="images/powered_by_pl_grey.png" alt="powered by planetlab"></a>
395
396                 <a href="http://www.cs.arizona.edu"><img style="position:relative; left: 20px; border: 0px; border-style: none;" src="images/template_logo_small_grey.png" alt="University of Arizona, Computer Science logo"></a>
397         </td>
398      </tr>
399
400 </table>
401
402 </div>
403
404
405 <script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
406 </script>
407 <script type="text/javascript">
408 _uacct = "UA-1868232-1";
409 urchinTracker();
410 </script>
411 </body>
412 </html>