Extensions Docs Server: Doc conversion script

Extensions Docs Server: Doc conversion script

This script will convert static files from the old documentation
system to the new one.

To run from the server2/ directory try:
converter.py ../static/ ../../api templates/articles/ templates/intros/ templates/public/ static/images/ -r

BUG=131095

[cduvall, 2012-07-26 21:56:13]

I didn't add the generated docs to the CL, but I generated them all locally, and
all the ones I looked at looked good.

[not at google - send to devlin, 2012-07-30 16:56:35]

This is super cool.

We should do a couple more things here:
1. add/remove templates for when APIs are added or moved. An example is the
appWindow API which has moved to app.window. The script should detect that.
 - later on, we should make this a PRESUBMIT check.
2. add this script to build.py, so that when people build the old docs the new
ones are automatically updated.

Perhaps something to aim for: if you delete templates/public, this script will
perfectly re-create it. That may be unreasonable (e.g. we have some of our own
cool things which we don't want to override), but it should at least work for
what you'd expect it to.

Or to put it another way: let's deploy the server after this patch goes in, but
assume that nobody will actually maintain the new docs alongside the old. We are
used to running build.py so let that keep them in sync for us.

---

Something else I've noticed is that the table of contents needs to go below the
header in the articles. I remember you having a change that did this; we may
have to revisit it, I think :\

[not at google - send to devlin, 2012-07-30 16:56:35]

This is super cool.

We should do a couple more things here:
1. add/remove templates for when APIs are added or moved. An example is the
appWindow API which has moved to app.window. The script should detect that.
 - later on, we should make this a PRESUBMIT check.
2. add this script to build.py, so that when people build the old docs the new
ones are automatically updated.

Perhaps something to aim for: if you delete templates/public, this script will
perfectly re-create it. That may be unreasonable (e.g. we have some of our own
cool things which we don't want to override), but it should at least work for
what you'd expect it to.

Or to put it another way: let's deploy the server after this patch goes in, but
assume that nobody will actually maintain the new docs alongside the old. We are
used to running build.py so let that keep them in sync for us.

---

Something else I've noticed is that the table of contents needs to go below the
header in the articles. I remember you having a change that did this; we may
have to revisit it, I think :\

[not at google - send to devlin, 2012-07-30 17:05:32]

Also, http://localhost:8080/devguide.html looks awful (but issues like that we
can fix later, I don't think it's the fault of the conversion script).

[not at google - send to devlin, 2012-07-31 06:23:28]

A couple more things to add:
3. Remove all those silly comments, like <!-- BEGIN AUTHORED CONTENT -->. The
DOM should be pretty clean now so it'll just be distracting.
4. Perhaps do that <a blah> -> $ref:bah conversion I was talking about. Better
to do that here than dynamically in the server? Not sure. But that's another
thing we can make a presubmit check, I think better that than let people submit
typo-prone code. Change cargo-culting habits and all that.

[cduvall, 2012-07-31 20:22:35]

Here is what I've done and why I didn't do some things:

1. It might not be possible to distinguish between API names like
appWindow/app.window because they have the same unix name, so the name of the
JSON file is the same. I made the script handle situations where a file with
static content changes names, because the files with static content do not use
pure unix names, so they are distinguishable.

2. Script has been added to build.py

3. Comments are removed

4. <a blah> -> $ref:blah conversion is not done yet because some of the ref
handling will change in the devtools patch.

[not at google - send to devlin, 2012-07-31 22:25:46]

Just noticed a couple of things.

Also as I said on IRC, there's the camel casing issue.

I will have a more careful look at the output tomorrow.

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
File chrome/common/extensions/docs/build/build.py (right):

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
chrome/common/extensions/docs/build/build.py:186: 
comment, explain what this does, why we do it, when it will go away, etc.

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
File chrome/common/extensions/docs/server2/converter.py (right):

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
chrome/common/extensions/docs/server2/converter.py:44: return
('{{+partials.standard_api api:apis.%s intro:intros.%s}}' %
some APIs don't have intros, like fileSystem

that said, it looks like the script hasn't picked up the existence of the
fileSystem API. I think you need to look in chrome/common/extensions/api too,
cross reference with the static docs.

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
chrome/common/extensions/docs/server2/converter.py:60: contents =
re.sub(r'<!--.*-->', r'', contents)
I don't think we want to delete *all* comments. Just the ones that look like
<!-- BEGIN --> or <!-- END -->

[cduvall, 2012-08-01 01:40:50]

I fixed the things you commented on, and the double files (eg. browserAction and
browser_action). I didn't have much time to look it over carefully, but it
looked good as far as I could tell.

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
File chrome/common/extensions/docs/build/build.py (right):

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
chrome/common/extensions/docs/build/build.py:186: 
On 2012/07/31 22:25:46, kalman wrote:
> comment, explain what this does, why we do it, when it will go away, etc.

Done.

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
File chrome/common/extensions/docs/server2/converter.py (right):

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
chrome/common/extensions/docs/server2/converter.py:44: return
('{{+partials.standard_api api:apis.%s intro:intros.%s}}' %
On 2012/07/31 22:25:46, kalman wrote:
> some APIs don't have intros, like fileSystem
> 
> that said, it looks like the script hasn't picked up the existence of the
> fileSystem API. I think you need to look in chrome/common/extensions/api too,
> cross reference with the static docs.

Done.

http://codereview.chromium.org/10832042/diff/3004/chrome/common/extensions/do...
chrome/common/extensions/docs/server2/converter.py:60: contents =
re.sub(r'<!--.*-->', r'', contents)
On 2012/07/31 22:25:46, kalman wrote:
> I don't think we want to delete *all* comments. Just the ones that look like
> <!-- BEGIN --> or <!-- END -->

Done.

[not at google - send to devlin, 2012-08-01 20:41:04]

I went through this today and found some issues with the docs CSS, but I left my
notes at work unfortunately. Most were old problems and some I fixed (didn't
check in), but I can't really remember.

Anyway, a couple more comments.

And something that we definitely need to fix is that TOC thing, to remove the
first <h1> element then use it as the title. Sorry, I know you already wrote
code for this earlier, but I think we do need it (also now I think it's done
with an HTML parser where previously it was done with regex?).

http://codereview.chromium.org/10832042/diff/7005/chrome/common/extensions/do...
File chrome/common/extensions/docs/build/build.py (right):

http://codereview.chromium.org/10832042/diff/7005/chrome/common/extensions/do...
chrome/common/extensions/docs/build/build.py:194: Popen([os.path.join(_base_dir,
'server2', 'converter.py'),
paths here need to be fixed so that it can be run from src/

http://codereview.chromium.org/10832042/diff/12300/chrome/common/extensions/d...
File chrome/common/extensions/docs/server2/converter.py (right):

http://codereview.chromium.org/10832042/diff/12300/chrome/common/extensions/d...
chrome/common/extensions/docs/server2/converter.py:30: # names of the JSON files
do not give enough information on the actual API name.
seems like most of these you could figure out based on camelcasing the
underscores, with the exception of experimental (programmable).

with two exception, and experimental.input.virtualKeyboard and input.ime.

http://codereview.chromium.org/10832042/diff/12300/chrome/common/extensions/d...
File chrome/common/extensions/docs/server2/templates/articles/about_apps.html
(left):

http://codereview.chromium.org/10832042/diff/12300/chrome/common/extensions/d...
chrome/common/extensions/docs/server2/templates/articles/about_apps.html:11: 
what's with these blank line removals? just curious?

http://codereview.chromium.org/10832042/diff/12300/chrome/common/extensions/d...
File chrome/common/extensions/docs/server2/templates/articles/app_identity.html
(right):

http://codereview.chromium.org/10832042/diff/12300/chrome/common/extensions/d...
chrome/common/extensions/docs/server2/templates/articles/app_identity.html:3:
Web authentication protocols utilize HTTP features,
random file which is getting detected as an "add" where it should be a copy.

Sadly I don't think that git-svn knows that it's a copy, or how to communicated
that to svn.

Perhaps it's worth looking quickly into whether you can
1. get an svn checkout going
2. in the script, when you copy do an "svn copy" instead.

This initial svn-mode thing should be a once off. Perhaps you could add it as a
flag to the script.

Scripts like this are complicated :\ sorry.

[cduvall, 2012-08-02 00:54:06]

Fixed all the comments except for the svn stuff. I'm in the process of getting a
checkout set up.

https://chromiumcodereview.appspot.com/10832042/diff/7005/chrome/common/exten...
File chrome/common/extensions/docs/build/build.py (right):

https://chromiumcodereview.appspot.com/10832042/diff/7005/chrome/common/exten...
chrome/common/extensions/docs/build/build.py:194: Popen([os.path.join(_base_dir,
'server2', 'converter.py'),
On 2012/08/01 20:41:04, kalman wrote:
> paths here need to be fixed so that it can be run from src/

Done.

https://chromiumcodereview.appspot.com/10832042/diff/12300/chrome/common/exte...
File chrome/common/extensions/docs/server2/converter.py (right):

https://chromiumcodereview.appspot.com/10832042/diff/12300/chrome/common/exte...
chrome/common/extensions/docs/server2/converter.py:30: # names of the JSON files
do not give enough information on the actual API name.
On 2012/08/01 20:41:04, kalman wrote:
> seems like most of these you could figure out based on camelcasing the
> underscores, with the exception of experimental (programmable).
> 
> with two exception, and experimental.input.virtualKeyboard and input.ime.

Done.

https://chromiumcodereview.appspot.com/10832042/diff/12300/chrome/common/exte...
File chrome/common/extensions/docs/server2/templates/articles/about_apps.html
(left):

https://chromiumcodereview.appspot.com/10832042/diff/12300/chrome/common/exte...
chrome/common/extensions/docs/server2/templates/articles/about_apps.html:11: 
On 2012/08/01 20:41:04, kalman wrote:
> what's with these blank line removals? just curious?

I did it because removing some of the stuff at the beginning of the docs left a
few blank lines, but I could just use strip() instead.

https://chromiumcodereview.appspot.com/10832042/diff/4301/chrome/common/exten...
File chrome/common/extensions/docs/server2/intro_data_source.py (right):

https://chromiumcodereview.appspot.com/10832042/diff/4301/chrome/common/exten...
chrome/common/extensions/docs/server2/intro_data_source.py:64: intro =
re.sub(re.compile('<h1[^>.]*?>.*?</h1>', flags=re.DOTALL),
Used re here because there's not an easy way to remove things with HTMLParser.
I'm pretty sure my re handles all the cases though.

[cduvall, 2012-08-03 01:50:46]

This has now moved to https://chromiumcodereview.appspot.com/10834130/

[cduvall, 2012-08-03 01:51:22]

(couldn't figure out how to change the issue number with SVN)

[not at google - send to devlin, 2012-08-06 07:16:08]

So should this be closed then?