Add tool for bots to download a web directory.

scripts/slave/download_from_url.py

+#!/usr/bin/python

[cmp, 2012/05/16 17:53:19]

there's a general problem with this approach that makes it hard to reproduce
builds

for example, we cannot go back and say "rebuild the earlier version" and get the
same result since the contents of the files at the given URL may have changed.

i don't think this is right.  i think instead you should find a way to check
these dependencies in somewhere and reference them from a deps file.  how big
are they?  are they all binaries?  how often do they change?  do they change
from build to build?

[chrisphan1, 2012/05/16 19:22:51]

The files:

top_million_urls 20megs

minidump_stackwalk 1meg
http://code.google.com/searchframe#OAMlx_jo-ck/src/breakpad/src/processor/min...

crash_report 4megs
http://code.google.com/searchframe#OAMlx_jo-ck/src/breakpad/src/tools/mac/cra...

minidump_stackwalk and crash_report just parse the dump files.  They don't
change from build to build.  So I was thinking these are the files that I can
build on one machine and distribute to the rest of the test machines.

[cmp, 2012/05/16 20:23:19]

Okay, as for the size of the files, I don't think it will be a problem.  Have
you tried gzip compressing these files to see how much that saves us?  And if
the data needs to be internal, we can check them into an internal repo.  Once
they're in a repo, we can reference them from the DEPS file and then we won't
need this script.  (Any new updates will be pulled via the update_scripts step.)

[chrisphan1, 2012/05/16 20:45:42]

gzip saves about 50%.  It doesn't need to be internal.  Where should I put them?

+# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Use of this source code is governed by a BSD-style license that can be
+# found in the LICENSE file.
+
+"""Download all files and directories from a URL.
+
+Provided a URL, this script will download files and folders with 'href'
+attribute as the name and the URL as the base path.
+
+This was only tested on Apache web directory.
+"""
+
+import optparse
+import os
+import re
+import sys
+import urllib
+import urllib2
+
+
+def IsFile(url):
+  """Check if URL is a file type.
+
+  If 'Etag' is in the header response, it must be a file type.
+  """
+  try:
+    response = urllib2.urlopen(url)
+    if 'Etag' in response.info():
+      return True
+  except urllib2.URLError:
+    pass
+  return False
+  
+  
+def Download(base_url, target):
+  """Recursively download files and directories.
+
+  Returns:
+    True if succeeded downloading all items, false otherwise.
+  """
+  # Get list of files in this url path.
+  try:
+    response = urllib2.urlopen(base_url)
+  except urllib2.URLError:
+    print 'URL does not exist : ' + base_url
+    return False
+  html = response.read()
+  item_list = re.findall(r'href="([^"]*)"', html)
+
+  # Download all files.
+  for item in item_list:
+    if item.startswith('/'):
+      continue
+    url = base_url + item
+    if item.endswith('/'):
+      dest = os.path.join(target, urllib.unquote(item))
+      if not os.path.isdir(dest):
+        os.makedirs(dest)
+        os.chmod(dest, 0755)
+      if not Download(url, dest):
+        return False
+    elif IsFile(url):
+      dest = os.path.join(target, urllib.unquote(item))
+      print 'Downloading %s to %s' % (url, dest)
+      urllib.urlretrieve(url, dest)
+      os.chmod(dest, 0755)
+  return True
+
+
+def main():
+  """Download files from |url| to |target_dir|."""
+  option_parser = optparse.OptionParser()
+
+  option_parser.add_option('--url',
+                           help='URL where to find the build to extract.  '
+                                'If ommited, default URL will be used.')

[cmp, 2012/05/16 17:53:19]

ommited -> omitted

later you wrote:
  if not options.url:
    ...
    return 1

so if it's omitted, it looks like the script will exit.  that's fine if it's the
case, just drop the "if omitted" line from the help doc.

[chrisphan1, 2012/05/16 19:22:51]

Done.

+  target_dir = os.getcwd()
+  option_parser.add_option('--target-dir', default=target_dir,
+                           help='Build target to archive (Release)')
+  options, _ = option_parser.parse_args()
+
+  if not options.url:
+    print 'No url provided.'
+    return 1
+  base_url = options.url
+  if not base_url.endswith('/'):
+    base_url += '/'
+
+  if not os.path.isdir(options.target_dir):
+    os.makedirs(options.target_dir)
+  os.chmod(options.target_dir, 0755)
+
+  success = Download(base_url, options.target_dir)
+  if success:
+    return 0
+  return 1
+
+
+if '__main__' == __name__:
+  sys.exit(main())