Link checker

Write a Java program to:

1. Take a URL as a command-line argument.
2. Examine all the pages it links to, and display broken links.
3. Pages it links to can be found by searching for: <a .. href
   • HTML 4.01
     • list of tags
     • <A definition

4. See Parsing XML / HTML

5. We will define a "broken" link as any link with a HTTP return code other than 200, or a link that times out.
6. For timeout settings try here.

7. Produce an output file listing in a nice 3-column table:
   1. All broken links (return code other than 200).
   2. Their HTTP return codes.
   3. Comment on why we do not get return code 200. Is the link really broken? Or just moved?
      For every link that your program claims is broken, test the link manually in a browser. If you can view the link, explain why your program thinks it is broken.

8. I suggest the output file should be a web page that you can browse (offline) and click on the allegedly broken links to check them in the browser.
9. Do not bother listing the links that worked (return code 200).

Test on these URLs:

Your final output should demonstrate your program working on these URLs:

http://humphrysfamilytree.com/links.html
http://humphrysfamilytree.com/sources.html

Hint: Get your program working on smaller pages first, before testing it on larger pages. These pages are huge!

For 100 percent, check these:

1. Relative links, like:
   <a href="subdir/file.html">
   <a href="../index.html">

2. Links to a location within a page, like:
   <a href="#location">
   <a href="file.html#location">
   Check if the destination location (marked by name= or id=) exists on that page.

3. href links to files that are not web pages, like:
   <a href="pic.jpg">
   Check if file exists.

4. Embedded image src links, like:
   <img src="pic.jpg">
   Check if image file exists.

5. Forms (check the ACTION= link)

Ignore these:

1. You may ignore href followed by (common):
   mailto
   or (rare):
   ftp, telnet, news, gopher.

Ignore all links to Google

1. Google doesn't allow scripting of search results.

2. Your program can ignore all Google searches of the form:

     http://www.google.DOMAIN/search?ARGS
     e.g. biscuits
   

   These searches never break anyway. If this search link once worked (i.e. is formatted correctly), it will always work.

3. Ignore all links to Google's directory:

     http://www.google.DOMAIN/Top/PATH
     e.g. Biscuits
   

   I need to delete all these.

4. In fact, ignore all links to Google.

To hand up:

What to hand up (Include a printout of the output when run on the URLs above.)