Lecture 9

Putting it all to good use: getting info from the web

Here's the ren script that was posed in the latest homework.

#!/bin/bash
for i
do
  newname=`echo "$i" | tr " " "."`
  mv "$i" $newname
done

In this class, we will make a series of incremental modifications to this file, to obtain a much better script. First, let us trap the case when there are no arguments given. In this case, we want to print an error message to this effect, and exit the script. So, here is version 2 of the script.
```
#!/bin/bash
if [ $# -eq 0 ]
then
   echo "ren: No arguments given." 
   exit 2
fi
for i
do
  newname=`echo "$i" | tr " " "."`
  mv "$i" $newname
done
```
There are several new features of this script. First, note the use of an if conditional to test for the value of $# (this variable gives the number of command line arguments). We check if this is zero (note that the check for equality is -eq) and if so, print a message. Second observe that we exit with an error code of 2 (this is a number cooked up by us; the convention is to use 0 for normal exits and some non-zero number for abnormal exits). Finally, the spaces around the [ and ] are important!
The above code is rather ugly. For instance, if we change the name of the script from ren to rename, then the above error message (inside the echo) must be changed. One choice is to replace ren with $0. Recall that $0 by default gives the name of the command that was invoked on the command line. So we replace the offending line with:
```
echo "$0: No arguments given." 
```
But even this is not so elegant. If we cd into some other directory and run the script from there, via some long path, e.g., as somelongpathnamehere/ren (without any arguments), then the message will be printed as:
```
somelongpathnamehere/ren: No arguments given.
```
Ideally, we want only
```
ren: No arguments given.
```
The command basename does exactly this. It takes a long pathname and chops off all but the end stuff (which is usually the name of the command). Do a man basename to learn more. So, now our script becomes:
```
#!/bin/bash
if [ $# -eq 0 ]
then
   echo "`basename $0`: No arguments given." 
   exit 2
fi
for i
do
  newname=`echo "$i" | tr " " "."`
  mv "$i" $newname
done
```
Note the use of the ` quotes to execute the basename commmand.

Now, do we want the error messages to go to the same "stdout" stream as the legitimate output? Probably not. In UNIX we also have the "stderror" stream that is designed specifically for this situation. You will still see the error message printed to the screen, but if you were to pipe "|" the output into some other program, only the stdout will get piped, not stderror. Which is nice: we don't want the error messages to go where the legitimate output is supposed to. The 1>&2 essentially means "redirect whatever used to be in stream 1 onto stream 2". Stream 1, by default is the standard output, and stream 2 is the standard error. You will see the difference between such redirection and non-redirection, if you try to redirect the output of the script at the command prompt, e.g.,
```
ren > output
```
In the old script, the error messages will go to the file output. In the new script, the error messages will be printed onto the screen (this is because the > symbol only redirects standard output, not standard error).
```
#!/bin/bash
if [ $# -eq 0 ]
then
   echo "`basename $0`: No arguments given."  1>&2
   exit 2
fi
for i
do
  newname=`echo "$i" | tr " " "."`
  mv "$i" $newname
done
```

Other Useful Bash Features

Try using bash with:
```
#!/bin/bash -xv
```
This option prints commands as they are executed by the script and will give you good insight into how bash works and is also useful during debugging.

The $? is a variable that returns the status of the last command executed. For instance,

bashprompt>ls > out
bashprompt>echo $?
0
bashprompt>li
li: Command not found
bashprompt>echo $?
1

Retreiving Stockquotes from the Web

Suppose you want to automatically get a list of your favorite quotes from the web. Perhaps, you want to do it every minute and then plot a graph. You can go to yahoo finance site and type in your company's symbol in the box next to "Get quote". Suppose the company you are interested in is Microsoft, whose ticker symbol is MSFT . By pressing GO next to the box, you will be redirected to a page with all (and perhaps more than) the info you need. Of course, you can do it manually each time. But what if you are away? What if you want the price checked every minute and, if it rises above a certain threshold, you want an email sent to you with a subject line "Sell NOW!". Well, you can pay your broker to do that, or you can use UNIX.
First, observe that the URL to obtain the quote for Microsof is http://finance.yahoo.com/q?s=MSFT . You can guess that if you are interested in any other company, all you need to do is to append a sticker symbol to the "stem" http://finance.yahoo.com/q?s= , e.g. http://finance.yahoo.com/q?s=GOOG for google.
Ok, so what? How do we access this website from within our shell script? For this, you have wget command. Do a man wget to learn more ( wget -h for a list of options). Once you have read the man page, you will have realized that this is what you want to do:
```
  
	wget "http://finance.yahoo.com/q?s=MSFT"  -o /dev/null -O page   
	
```
So, wget takes the "URL" as its argument, sends all of it technical messages to /dev/null (which is "nowhere": you don't need those anyways), and fetches the page to file page .

Now we can look inside the retreived page. Not the prettiest of sights, is it? Just a few lines are below.


#screen,#masthead{text-align:center;margin:0;width:752px;}
#screen.xpand,#masthead.xpand{width:100%;}
#screen.xpand #yfncsubtit{width:100%;}
#masthead{padding-bottom:12px;}
#leftcol,#rightcol{margin:0;}
#leftcol{width:155px;float:left;}
#rightcol{width:585px;}
#content{width:752px;}
#footer {clear:both;}

Of course, you don't need all that grbage: what you are looking for are the line that contains Last Trade: and 27.21 next to it. The trouble is, the HTML page is not human-readable. Open it in vi and you'll see why: it has very long lines without breaks. In fact, we are looking for line 5.

The challange now is to exatract the price 27.21 . Note that simply searching for <td class= or some other non-specific text won't help as there are probably lots of those symbols in this HTML page. Here is an awk line that does the trick:
```
 
awk 'BEGIN{FS = ">Last Trade:</td><td class=\"yfnc_tabledata1\"><big><b>"  }{if (NR == 5) print substr($2,1,5) }' page 
```
Note that:
- We redefined FS so that 27.21 is in the beginning of a new field.
- We use substr() function: you already saw it in use in HW3. In this particular case, it takes a substring of length 5, beginning at position 1 from field 2. You can check that this points to 27.21 if applied to the field that now begins with 27.21
- As is alsways the case, there are many other ways to achieve the same result. The above is not necessarily the most elegant or robust way.
Here is the complete bash script:
```
 
#!/bin/bash


# the script below "downloads" the webpage and parses it
# for the information we need.
# the search patterns below work only for stocks (like MSFT or GOOG),
# not mutual funds  and other traded items

# The line below defines the URL and the query string
prefix="http://finance.yahoo.com/q?s="

for i
do
        url=$prefix$i
        wget $url -o /dev/null -O page
        val=`awk 'BEGIN{FS = ">Last Trade:</td><td class=\"yfnc_tabledata1\"><big><b>"  }{if (NR == 5) print substr($2,1,5) }' page`
        echo "The stock of" "(" $i ") is currently priced at $"$val
done
```
Suppose you want the script to instruct you to buy the stock if its price is below $100. Well, one problem is that bash can only operate with integer numbers. But here is a trick: save your output in a file, and make a call to gawk from within your script. To test this idea, add the following to the script above (inside the do loop):
```
 

    rm data.out
    echo $val >> data.out
    /usr/bin/gawk '{ { if ($1 > 100 ) print "SELL!" } {if ($1 < 100 ) print "BUY!" }  }' < data.out
```