Devserver: only populate data during cron jobs, meaning all data from instances

Devserver: only populate data during cron jobs, meaning all data from instances
are served out of caches. Add a PersistentObjectStore to make that work.

BUG=226625

[not at google - send to devlin, 2013-04-16 13:20:48]

Hey Clark,

This is a WIP insofar as it's done, except I can't get PersistentObjectStore to
work. Perhaps you can have a look there and see what I'm doing wrong.

I can't seem to get my brain around the datastore APIs. I have the set()ing work
(they show up in the dev server console), though having to call wait() is a pain
and possibly a quirk of the dev server. Neither the get()ting nor the
delete()ing is working, though.

After this patch we'll be free of both svn.chromium.org and omahaproxy failures!

[方觉(Fang Jue), 2013-04-16 14:26:49]

drive-by. I was wondering where PersistentDataStore was in the first patchset:)
Glad to see this change (and it could possibly help http://crbug.com/161658).

https://chromiumcodereview.appspot.com/14218004/diff/2001/chrome/common/exten...
File chrome/common/extensions/docs/server2/persistent_object_store.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/2001/chrome/common/exten...
chrome/common/extensions/docs/server2/persistent_object_store.py:48: item =
_Item(key_=self._ToItemKeyPath(k),
you probably need to use key_name=... here so that
db.get_async(object_store._ToItemKey(k)) will work. (and key_ may or may not be
necessary)
See also
https://developers.google.com/appengine/docs/python/datastore/modelclass#Model
and related docs about keys of entities.

https://chromiumcodereview.appspot.com/14218004/diff/2001/chrome/common/exten...
chrome/common/extensions/docs/server2/persistent_object_store.py:52:
future.wait()
why not use synchronous api here?

[cduvall, 2013-04-17 01:12:11]

I upploaded a patchset that works with datastore. It had minimal changes to
persistent_object_store.py.

I think the .wait() will be unnecessary on App Engine. For some reason,
dev_appserver only finds things in datastore after restarting. So it won't
synchronize datastore until you kill and restart, and then it should work.

[not at google - send to devlin, 2013-04-17 18:00:50]

Thanks for the help, good to go now.

(and I hackily ran the persistent unit test inside the dev server environment
and it passes)

https://chromiumcodereview.appspot.com/14218004/diff/2001/chrome/common/exten...
File chrome/common/extensions/docs/server2/persistent_object_store.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/2001/chrome/common/exten...
chrome/common/extensions/docs/server2/persistent_object_store.py:52:
future.wait()
On 2013/04/16 14:26:49, 方觉(Fang Jue) wrote:
> why not use synchronous api here?

So that while each of these is wait()ing the other put_async()s are running. In
any case, I've put this behind an IsDevServer flag.

[not at google - send to devlin, 2013-04-17 18:08:08]

sorry upload failed on the patch, not quite ready...

[not at google - send to devlin, 2013-04-17 18:10:37]

ok PTAL

[cduvall, 2013-04-17 23:30:37]

lgtm with comments

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/cache_chain_object_store.py (right):

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/cache_chain_object_store.py:1: # Copyright
2013 The Chromium Authors. All rights reserved.
I like this class

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/handler.py (right):

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/handler.py:23: ALWAYS_ONLINE = False
So this will have to be changed manually before uploading or using
dev_appserver?

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/integration_test.py (left):

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/integration_test.py:45: def
_TestSamplesLocales(self, sample_path, failures):
This isn't testing locales any more, is that ok?

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/integration_test.py (right):

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/integration_test.py:27: with open('%s/%s'
% (path, filename), 'r') as f:
use os.path.join here?

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/integration_test.py:59: # This is
reaaaaaly rough since usually these will be tiny templates
nit: two 'l's in reaaaaaly :)

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/persistent_object_store.py (right):

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/persistent_object_store.py:11: def log(s):
take out

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/persistent_object_store.py:25: #if
future.get_result() is None:
take out

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
File chrome/common/extensions/docs/server2/test_object_store.py (right):

https://codereview.chromium.org/14218004/diff/16001/chrome/common/extensions/...
chrome/common/extensions/docs/server2/test_object_store.py:46: '''Returns a
tuple (success, error). Use in tests like:
This is cool

[not at google - send to devlin, 2013-04-18 04:05:41]

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
File chrome/common/extensions/docs/server2/cache_chain_object_store.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/cache_chain_object_store.py:1: # Copyright
2013 The Chromium Authors. All rights reserved.
On 2013/04/17 23:30:37, cduvall wrote:
> I like this class

:)

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
File chrome/common/extensions/docs/server2/handler.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/handler.py:23: ALWAYS_ONLINE = False
On 2013/04/17 23:30:37, cduvall wrote:
> So this will have to be changed manually before uploading or using
> dev_appserver?

Nope. Adding a comment here to explain the following:

What I'm going for is appengine instances never needing to call out to SVN,
since that has been flaky recently. If this is False we can catch errors in that
area ahead of time, and also, not bother asking SVN when we know it's going to
be a 404 anyway (e.g. when requesting a bogus URL). Only crons get to be online,
and they're allowed to fail.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
File chrome/common/extensions/docs/server2/integration_test.py (left):

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/integration_test.py:45: def
_TestSamplesLocales(self, sample_path, failures):
On 2013/04/17 23:30:37, cduvall wrote:
> This isn't testing locales any more, is that ok?

Ah, thanks. Added it back.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
File chrome/common/extensions/docs/server2/integration_test.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/integration_test.py:27: with open('%s/%s'
% (path, filename), 'r') as f:
On 2013/04/17 23:30:37, cduvall wrote:
> use os.path.join here?

Done.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/integration_test.py:59: # This is
reaaaaaly rough since usually these will be tiny templates
On 2013/04/17 23:30:37, cduvall wrote:
> nit: two 'l's in reaaaaaly :)

haha.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
File chrome/common/extensions/docs/server2/persistent_object_store.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/persistent_object_store.py:11: def log(s):
On 2013/04/17 23:30:37, cduvall wrote:
> take out

Done.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/persistent_object_store.py:25: #if
future.get_result() is None:
On 2013/04/17 23:30:37, cduvall wrote:
> take out

Done.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/persistent_object_store.py:25: #if
future.get_result() is None:
On 2013/04/17 23:30:37, cduvall wrote:
> take out

Done.

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
File chrome/common/extensions/docs/server2/test_object_store.py (right):

https://chromiumcodereview.appspot.com/14218004/diff/16001/chrome/common/exte...
chrome/common/extensions/docs/server2/test_object_store.py:46: '''Returns a
tuple (success, error). Use in tests like:
On 2013/04/17 23:30:37, cduvall wrote:
> This is cool

Yah proud of that little pythonism. I'll try to remember to do the same to
test_file_system after this.

[commit-bot: I haz the power, 2013-04-18 04:06:38]

CQ is trying da patch. Follow status at
https://chromium-status.appspot.com/cq/kalman@chromium.org/14218004/21001

[commit-bot: I haz the power, 2013-04-18 04:06:49]

Presubmit check for 14218004-21001 failed and returned exit status 1.

INFO:root:Found 31 file(s).

Traceback (most recent call last):
  File "docs/server2/integration_test.py", line 6, in <module>
    from handler import Handler
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/handler.py",
line 13, in <module>
    from server_instance import ServerInstance
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/server_instance.py",
line 9, in <module>
    from api_data_source import APIDataSource
  File
"/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/api_data_source.py",
line 10, in <module>
    import third_party.json_schema_compiler.json_parse as json_parse
ImportError: No module named third_party.json_schema_compiler.json_parse
testGetBranchNumberForChannelName (__main__.BranchUtilityTest) ... ok
testSplitChannelNameFromPath (__main__.BranchUtilityTest) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK
testLevels (__main__.SamplesDataSourceTest) ... ok
testSelected (__main__.SamplesDataSourceTest) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.007s

OK
testCheckAndReset (__main__.TestObjectStoreTest) ... ok
testEmpty (__main__.TestObjectStoreTest) ... ok
testNonEmpty (__main__.TestObjectStoreTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.003s

OK
testListDir (__main__.SubversionFileSystemTest) ... ok
testReadFiles (__main__.SubversionFileSystemTest) ... ok
testStat (__main__.SubversionFileSystemTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.107s

OK
testFilterSamples (__main__.SamplesDataSourceTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.007s

OK
testFactoryWithBranch (__main__.ObjectStoreCreatorTest) ... ok
testIllegalIinput (__main__.ObjectStoreCreatorTest) ... ok
testVanilla (__main__.ObjectStoreCreatorTest) ... ok
testWithCategory (__main__.ObjectStoreCreatorTest) ... ok
testWithVersion (__main__.ObjectStoreCreatorTest) ... ok
testWithVersionAndCategory (__main__.ObjectStoreCreatorTest) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.001s

OK
testListDir (__main__.LocalFileSystemTest) ... ok
testReadFiles (__main__.LocalFileSystemTest) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.002s

OK
testCaching (__main__.CachingFileSystemTest) ... ok
testListDir (__main__.CachingFileSystemTest) ... ok
testReadFiles (__main__.CachingFileSystemTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK
testCreateZip (__main__.ExampleZipperTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.005s

OK
testNamespaceIsolation (__main__.PersistentObjectStoreTest) ... ok
testPersistence (__main__.PersistentObjectStoreTest) ... ok

----------------------------------------------------------------------
Ran 2 tests in 0.001s

OK
testDel (__main__.CacheChainObjectStoreTest) ... ok
testGetFromAllLayers (__main__.CacheChainObjectStoreTest) ... ok
testGetFromFirstLayer (__main__.CacheChainObjectStoreTest) ... ok
testGetFromSecondLayer (__main__.CacheChainObjectStoreTest) ... ok
testGetFromThirdLayer (__main__.CacheChainObjectStoreTest) ... ok
testNotFound (__main__.CacheChainObjectStoreTest) ... ok
testPartiallyCachedInMemory (__main__.CacheChainObjectStoreTest) ... ok
testSet (__main__.CacheChainObjectStoreTest) ... ok

----------------------------------------------------------------------
Ran 8 tests in 0.003s

OK
testApps (__main__.APIListDataSourceTest) ... ok
testExperimentalApps (__main__.APIListDataSourceTest) ... ok
testExtensions (__main__.APIListDataSourceTest) ... ok

----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK
testKeyGeneration (__main__.GithubFileSystemTest) ... ok
testList (__main__.GithubFileSystemTest) ... ok
testRead (__main__.GithubFileSystemTest) ... ok
testStat (__main__.GithubFileSystemTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.097s

OK
testCheckAndReset (__main__.TestFileSystemTest) ... ok
testEmptyFileSystem (__main__.TestFileSystemTest) ... ok
testMoveTo (__main__.TestFileSystemTest) ... ok
testNonemptyFileNotFoundErrors (__main__.TestFileSystemTest) ... ok
testNonemptySuccess (__main__.TestFileSystemTest) ... ok
testStat (__main__.TestFileSystemTest) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.002s

OK
testPartials (__main__.TemplateDataSourceTest) ...
ERROR:root:SamplesDataSource.Factory was never set in APIDataSource.Factory.
ERROR:root:ReferenceResolver.Factory was never set in APIDataSource.Factory.
ok
testRender (__main__.TemplateDataSourceTest) ... ok
testSimple (__main__.TemplateDataSourceTest) ... WARNING:root:Template junk.html
in . not found: ./junk.html
ok

----------------------------------------------------------------------
Ran 3 tests in 0.020s

OK
testCaching (__main__.CompiledFileSystemTest) ... ok
testFailures (__main__.CompiledFileSystemTest) ... ok
testIdentityFromFile (__main__.CompiledFileSystemTest) ... ok
testIdentityFromFileListing (__main__.CompiledFileSystemTest) ... ok
testIdentityNamespace (__main__.CompiledFileSystemTest) ... ok
testPopulateFromFile (__main__.CompiledFileSystemTest) ... ok
testPopulateNamespace (__main__.CompiledFileSystemTest) ... ok

----------------------------------------------------------------------
Ran 7 tests in 0.003s

OK
testGetLink (__main__.APIDataSourceTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.005s

OK
testApps (__main__.PathCanonicalizerTest) ... ok
testExtensions (__main__.PathCanonicalizerTest) ... ok
testNeither (__main__.PathCanonicalizerTest) ... ok
testStatic (__main__.PathCanonicalizerTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK
testCreateId (__main__.APIDataSourceTest) ... ok
testFormatDescription (__main__.APIDataSourceTest) ... ok
testFormatValue (__main__.APIDataSourceTest) ... ok
testRemoveNoDocs (__main__.APIDataSourceTest) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.011s

OK
testIntro (__main__.IntroDataSourceTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.004s

OK
Running presubmit commit checks ...
Running /b/commit-queue/workdir/chromium/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/chrome/PRESUBMIT.py
Running /b/commit-queue/workdir/chromium/chrome/common/extensions/PRESUBMIT.py
Running
/b/commit-queue/workdir/chromium/chrome/common/extensions/docs/server2/PRESUBMIT.py
Running ./branch_utility_test.py
Running ./sidenav_data_source_test.py
Running ./test_object_store_test.py
Running ./subversion_file_system_test.py
Running ./samples_data_source_test.py
Running ./object_store_creator_test.py
Running ./local_file_system_test.py
Running ./caching_file_system_test.py
Running ./example_zipper_test.py
Running ./persistent_object_store_test.py
Running ./cache_chain_object_store_test.py
Running ./api_list_data_source_test.py
Running ./github_file_system_test.py
Running ./test_file_system_test.py
Running ./template_data_source_test.py
Running ./compiled_file_system_test.py
Running ./reference_resolver_test.py
Running ./path_canonicalizer_test.py
Running ./api_data_source_test.py
Running ./intro_data_source_test.py

** Presubmit Warnings **
_VERSION of chrome/common/extensions/docs/server2/api_data_source.py needs to be
incremented.

_VERSION of chrome/common/extensions/docs/server2/subversion_file_system.py
needs to be incremented.

_VERSION of chrome/common/extensions/docs/server2/template_data_source.py needs
to be incremented.

** Presubmit ERRORS **
IntegrationTest failed!

Presubmit checks took 4.1s to calculate.

Was the presubmit check useful? Please send feedback & hate mail to
maruel@chromium.org!