Add tool for bots to download a web directory.

Add tool for bots to download a web directory.

This script allow buildbot slaves to download dependency files before it run tests.  For example 30+ chromebots will retrieve the list of urls and binaries that are not to be checked in the repository.

TEST=None
BUG=None

[chrisphan, 2012-05-16 00:06:13]

I use to do this using rsync before moving to buildbot.  I'm not sure if there's
a better way to do this.

[cmp, 2012-05-16 17:53:19]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
scripts/slave/download_from_url.py:1: #!/usr/bin/python
there's a general problem with this approach that makes it hard to reproduce
builds

for example, we cannot go back and say "rebuild the earlier version" and get the
same result since the contents of the files at the given URL may have changed.

i don't think this is right.  i think instead you should find a way to check
these dependencies in somewhere and reference them from a deps file.  how big
are they?  are they all binaries?  how often do they change?  do they change
from build to build?

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
scripts/slave/download_from_url.py:77: 'If ommited, default URL will be used.')
ommited -> omitted

later you wrote:
  if not options.url:
    ...
    return 1

so if it's omitted, it looks like the script will exit.  that's fine if it's the
case, just drop the "if omitted" line from the help doc.

[chrisphan1, 2012-05-16 19:22:51]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
scripts/slave/download_from_url.py:1: #!/usr/bin/python
On 2012/05/16 17:53:19, cmp wrote:
> there's a general problem with this approach that makes it hard to reproduce
> builds
> 
> for example, we cannot go back and say "rebuild the earlier version" and get
the
> same result since the contents of the files at the given URL may have changed.
> 
> i don't think this is right.  i think instead you should find a way to check
> these dependencies in somewhere and reference them from a deps file.  how big
> are they?  are they all binaries?  how often do they change?  do they change
> from build to build?

The files:

top_million_urls 20megs

minidump_stackwalk 1meg
http://code.google.com/searchframe#OAMlx_jo-ck/src/breakpad/src/processor/min...

crash_report 4megs
http://code.google.com/searchframe#OAMlx_jo-ck/src/breakpad/src/tools/mac/cra...

minidump_stackwalk and crash_report just parse the dump files.  They don't
change from build to build.  So I was thinking these are the files that I can
build on one machine and distribute to the rest of the test machines.

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
scripts/slave/download_from_url.py:77: 'If ommited, default URL will be used.')
On 2012/05/16 17:53:19, cmp wrote:
> ommited -> omitted
> 
> later you wrote:
>   if not options.url:
>     ...
>     return 1
> 
> so if it's omitted, it looks like the script will exit.  that's fine if it's
the
> case, just drop the "if omitted" line from the help doc.

Done.

[cmp, 2012-05-16 20:23:19]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
scripts/slave/download_from_url.py:1: #!/usr/bin/python
Okay, as for the size of the files, I don't think it will be a problem.  Have
you tried gzip compressing these files to see how much that saves us?  And if
the data needs to be internal, we can check them into an internal repo.  Once
they're in a repo, we can reference them from the DEPS file and then we won't
need this script.  (Any new updates will be pulled via the update_scripts step.)

[chrisphan1, 2012-05-16 20:45:38]

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
File scripts/slave/download_from_url.py (right):

https://chromiumcodereview.appspot.com/10391151/diff/1/scripts/slave/download...
scripts/slave/download_from_url.py:1: #!/usr/bin/python
On 2012/05/16 20:23:19, cmp wrote:
> Okay, as for the size of the files, I don't think it will be a problem.  Have
> you tried gzip compressing these files to see how much that saves us?  And if
> the data needs to be internal, we can check them into an internal repo.  Once
> they're in a repo, we can reference them from the DEPS file and then we won't
> need this script.  (Any new updates will be pulled via the update_scripts
step.)

gzip saves about 50%.  It doesn't need to be internal.  Where should I put them?

[cmp, 2012-05-16 20:48:42]

In that case this could go into chrome/trunk/tools/chromebot.  WDYT?

[chrisphan1, 2012-05-16 20:53:10]

On 2012/05/16 20:48:42, cmp wrote:
> In that case this could go into chrome/trunk/tools/chromebot.  WDYT?

Actually awhile ago nirnimesh told me not to check these in.  But it would nice
to check them in somewhere.

[cmp, 2012-05-16 20:55:30]

+nirnimesh, was there a specific reason not to check them in besides the binary
size?

[chrisphan1, 2012-05-16 21:01:26]

This is for the files:
top_million_urls
minidump_stackwalk (process linux crash dumps)
crash_report (process mac crash dumps)


On Wed, May 16, 2012 at 1:55 PM, <cmp@chromium.org> wrote:

> +nirnimesh, was there a specific reason not to check them in besides the
> binary
> size?
>
>
https://chromiumcodereview.**appspot.com/10391151/<https://chromiumcodereview...
>

[Nirnimesh, 2012-05-16 21:16:25]

Chris, all these 3 files (top_million, minidump_stackwalk, crash_report) should
be checked in.