Interface update, per-method Contexts.

Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithoutTransaction(c), obj0)
  taskqueue.Add(datastore.WithoutTransaction(c), task)
  prod.AEContext(datastore.WithoutTransaction(c))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

patch from issue 2302743002 at patchset 1 (http://crrev.com/2302743002#ps1)

Committed: https://github.com/luci/gae/commit/3715a27a534b6e3896e23662e229fa8caaeb8a60

[dnj, 2016-09-01 05:19:30]

Description was changed from

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
schanges. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in datstore
and taskqueue services to use Contexts directly. This is somewhat at
odds with the *NoTxn-style function calls. The datastore Transaction
interface is something that has been desired for a while, so supporting
nil Transactions should be a good starting point without over-committing
(e.g., actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)

- Methods generating datastore keys now must use a KeyContext instance.
  This isn't expected to be a problem, since most external packages
  should not be using direct key generation, but should, instead, be
  using the datstore-bound versions of those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.
==========

to

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
schanges. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)
  taskqueue.Add(datastore.WithTransaction(c, nil), task)
  prod.AEContext(datastore.WithTransaction(c, nil))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

Implement info.ServiceAccount() on devserver via info.AccessToken(...).

R=iannucci@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2220193002

Update cloud package.

Fixes #54.

BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2217063003

Remove empty namespace assertion from impl/memory.

This was fixed upstream in:
 
https://github.com/golang/appengine/commit/5a9e2b406e4e87cd8a88d1881a31b1248d...

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2219673002

GAE CL for luci-go common refactor

R=nodir@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/2192003003 .

Add an option to install just Cloud Datastore.

TBR=iannucci@chromium.org
BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2179843004

cq.cfg: use luci builders

do not use buildbot builder
update luci bucket name

R=estaab@chromium.org, dnj@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2179543003

Add lightning talk that I didn't gave at Gophercon '16


Add cloud datastore implementation.

Fixes #52.

BUG=

Review-Url: https://codereview.chromium.org/1957953002

datastore: Update AllocateIDs to take keys.

Update the Datastore's AllocateIDs call to accept variadic arguments similar
to Get/Put/Delete/Exists. Structs can be supplied to populate their keys
directly, and keys or key slices can be supplied to directly allocate keys.

This adds some Key helper functions:
- SameKind, which asserts that keys reference the same kind.
- Partial, which returns a partial key given a source key.
- WithID, which returns a derivative key with the specified ID.
- NewIncompleteKeys, which allocates a slice of incomplete keys given a kind.

BUG=

Review-Url: https://codereview.chromium.org/2007123002

Refactor multiarg, split MGS/PLS.

Refactor the multiarg construction method to handle the various cases
inline, rather than defining and invoking methods for specific
scenarios. This enables PropertyLoadSaver-based methods to be handled
separately from MetaGetterSetter-based methods, which, in turn, enables
the two to be implemented independently.

A bit of boilerplate is removed and the odd case of MGS/PLS Map/Chan fits
better. Additionally, a few double-reflection code paths are optimized
along the way.

BUG=None

Review-Url: https://codereview.chromium.org/2048933004

make swarming trybuilders non-experimental

R=iannucci@chromium.org, dnj@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2083193006

cq.cfg: enable experimental swarming builders

R=estaab@chromium.org, vadimsh@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2090433002

Add datastore dumper tool.

This allows you to dump the datastore in whole or part, with or without special
formatting directives.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2037433002

Add TrimmedAppID to info service.

Also refactor info service to share common 'augmented' methods in
the service/info package itself, ala service/datastore.

Split out from CL: https://chromiumcodereview.appspot.com/1537883002

R=dnj@chromium.org
BUG=550684

Review-Url: https://codereview.chromium.org/2062613002

Add DefaultVersionHostname to impl/memory/info.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2028423002

Fix typo in pls.go docs


Allow uint16 and uint32 in MakeKey.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036543003

Add GetNoTxn to txnBuf as a hack.

I really don't like this but short of fixing #23, I don't think there's much we
can do.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036593002

Change gae to apache 2 license.

Update AUTHORS, CONTRIBUTORS, and LICENSE.
Add apache 2 license check to PRESUBMIT.py.

Ran:
  git sed 's/Chromium Authors/LUCI Authors/'
  git sed 's/governed by a BSD-style license that can be/governed under the
Apache License, Version 2.0/'
  git sed 's/found in the LICENSE file/that can be found in the LICENSE file/'
  git checkout HEAD -- $(git status -sb | grep '^ T' | cut -c 4-)
  git checkout origin/master -- $(git diff origin/master --name-only | grep
third_party)

BUG=

Review-Url: https://codereview.chromium.org/2021753002

Change codereview server to be consistent with other luci projects.

BUG=

Review-Url: https://codereview.chromium.org/2020933002

dscache: Reduce log level of memcache failures.

Warning-level failures quickly pollute the AppEngine log space. This
change converts some of the more common warnings to Debug-level logs so
the Warning space remains reserved for higher attention calls.

Also add a warning about lock eviction poisoning to THERE BE DRAGONS
section of the doc.

BUG=

Review-Url: https://codereview.chromium.org/1987353002

datastore: variadic Get, Put, Exists, Delete.

Fixes #51.

Update Get, Put, Exists, and Delete to be variadic. This will make the
interface more usable and get rid of boilerplate in client code.

Get, Put, Exists, and Delete now accept variadic arguments. Any one of
these arguments can be an argument that was previously valid or
the *Multi version of those functions. This effectively obsoletes the *Multi
versions, but they are left in for backwards compatiblility with the stated
intention of removing them at a later date.

Get/Put/Exists/Delete will now return a single error in the one-argument
case or a MultiError in a multiple-argument case. Note that now that single
argument case can take a slice type, the single error may itself be a
MultiError if the single argument is a slice.

Review-Url: https://codereview.chromium.org/2011773002

filter/txnBuf: Use fully-qualified App ID.

Use the fully-qualified app ID when creating the in-memory datastore for
transaction buffer. Previously, transaction buffer would use the
standard App ID, while production datastore would use the
fully-qualified App ID, causing key incompatibility errors in
production.

Cleanup impl/memory's NewDatastore function to not install superfluous
services into the Context. This draws a cleaner line between
impl/memory's dual purposes as both a rich testing enviornment and a
production-grade in-memory datastore.

In addition, have impl/memory's info service differentiate between the
two. This causes tests for filter/txnBuf to properly fail lacking this
fix, enabling regression protection.

To support this, impl/memory's Info service will now parse the supplied
App ID. If it contains a "~" character, it is split into fully-qualified
and regular AppID. Otherwise, the two will be the same.

BUG=

Review-Url: https://codereview.chromium.org/1929423002

Travis: Increase system specs, limit concurrent.

Increase the Travis system specs and limit the number of concurrent
tests to try and avoid the signal-kill error that we're seeing in Travis
ATM.

TBR=iannucci@chromium.org
BUG=None

Review URL: https://codereview.chromium.org/1922583002

pcg: Bump test time to 20 minutes.

Travis is currently timing out.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1918713003

Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1917513002

impl/memory: datastore and memcache namespacing

Update GetNamespace() to return an additional boolean, which will be
true if the namespace was set and false if it was not. The AppEngine SDK
defaults to having no namespace installed ("", false), but will allow
the user to install any namespace. This allows a user to differentiate
between no namespace and an empty namespace ("", true).

Make impl/memory's datastore query operation to error if the namespace has
been set to the empty namespace (""). This reflects observed production
behavior.

BUG=

Review URL: https://codereview.chromium.org/1916463004

Bump max_duration for CI tests to 480s

Hopefully Fixes #48.
Fix memory corruption bug in impl/memory

When you `SetCollection` in gkvlite, it invalidates the old
collection, but keeps the Items of the previous collection. The way
that impl/memory was using SetCollection was more along the lines of
GetOrCreate, which could cause corruption or other issues if two
codepaths did SetCollection simultaneously.

This fixes the corruption bug by removing SetCollection from the
memStore interface and replaces it with a GetOrCreateCollection,
which checks for the collection, and only calls SetCollection if it
doesn't exist. As a bonus, this seems to give a slight performance
speedup over the previous broken code.

Additionally, this forces all uses of VisitItemsAscend to use
a Snapshot, to avoid accidental modification of a Collection while
its being iterated over, which can lead to the situation discussed
above, or other nasty errors.

Finally, this adds an optional 'gkvlite tracer' functionality
which came in (extremely) handy for finding this bug. When given
a folder, it will emit go-code-like invocations of the functions run
on every collection, and includes enough information to reveal which
functions are being called on the root collection v. snapshots, and
if there are any overlapping VisitItemsAscend invocations, etc.

R=dnj@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1911263002

Travis: Build against Go1.6.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1911293002

Remove race from query finalization.

BUG=

Review URL: https://codereview.chromium.org/1910293002

datastore: Fix AddIndexes with existing namespaces

AddIndexes previously only added an index for the current namespace.
Since indexes are global, they should be added for all existing
namespaces.

BUG=

Review URL: https://codereview.chromium.org/1894403002

service/taskqueue: Add NewPOSTTask, remove NewTask

Add NewPOSTTask, which mirrors the method of the same name in
AppEngine's Go task queue SDK.

Remove NewTask from the taskqueue Interface, since it was basically
useless.

BUG=

Review URL: https://codereview.chromium.org/1890983004

Add __namespace__ metadata query support to impl/memory.

Ref: https://cloud.google.com/datastore/docs/concepts/metadataqueries

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1892343002

Add Info Testable interface, implement for memory.

This adds a Testable interface to the Info service which allows it to
install Info parameters in Contexts.

Add support for setting version ID and request ID to impl/memory Info
service.

BUG=

Review URL: https://codereview.chromium.org/1871943003

Add PopulateKey method to datastore.

Add PopulateKey, a method that, given an object, fills in its key
fields.

Fixes #44.

BUG=

Review URL: https://codereview.chromium.org/1867963003

Fix missing/broken ds.Stop handling in raw DS.

Several places were missed when patching "ds.Stop" to being correctly filtred
at the top interface rather than the raw interface.

At the top-level, GetAll and Count both did not filter the Stop error.
At the filter level, several filters were ignoring error values from
their callbacks.

Also fixes an issue in count filter where the ds.Stop error was being
incorrectly filtered at the RawInterface level.

Add datastore iterating query filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1846123002

Add datastore iterating query filter.

BUG=

Review URL: https://codereview.chromium.org/1843313004

Don't filter "Stop" error in raw interface.

The Stop error is a high-level error used by the datastore service to
indicate that an iterative operation should stop rather. It should
appear to the user as a "nil" error.

However, currently the datastore raw implementations are filtering Stop
and converting it to "nil". This prevents other filters from
identifying Stop events from their lower level callbacks.

Update raw datastore methods to not filter Stop, and update their tests
to accept Stop as a valid successful result.

BUG=

Review URL: https://codereview.chromium.org/1844183003

Make ExistsMulti return a BoolList

BoolList has some convenience functions (e.g. to check if Any() or All() are
set.). It's compatible with current uses of ExistsMulti.

R=dnj@chromium.org, estaab@chromium.org, nodir@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1816413002

txnbuf: Add write count budget, size budget.

Add a "maximum number of datastore writes" budget to the transaction
buffer. Rework the size budget to remove preemptive child transaction
failures and directly derive budget from the parent.

BUG=

Review URL: https://codereview.chromium.org/1776833002

Add wrappers for the module module

BUG=570879

Review URL: https://codereview.chromium.org/1772943003

Add a MustNamespace that panics if the namespace wasn't valid

BUG=

Review URL: https://codereview.chromium.org/1766593002

Auto-project on distinct inequality fields.

BUG=

Review URL: https://codereview.chromium.org/1745703002

Add header text to proto-gae tool

In order to appease our presubmit scripts, default to the Chromium Authors
copyright header.

R=dnj@chromium.org, estaab@chromium.org, hinoka@chromium.org,
martiniss@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1718123003

dscache: Only log unusual memcache errors.

BUG=None
TEST=None
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1695833002

Use ShouldResemble instead of ShouldResembleV.

Now that goconvey incorporates our ShouldResembleV method directly, we
can go back to using ShouldResemble instead of hacking in
ShouldResembleV!

BUG=None

Review URL: https://codereview.chromium.org/1679093002

Add support for uint8, uint16 and uint32 in luci/gae

This has come up for like 3 models in DM so far, and I'm tired of having this
be a stupid "gotcha!". I was working around it with some PropertyConverter, but
with this change it will just be obvious what the heck is going on.

They all map to positive int64 values (which should make indexes obvious), and
they include out-of-bounds checks on deserialization.

I excluded uint64, since there's no way to properly encode it in the native
datastore representation which still makes any sense for indexes.

I also added some (apparently?) missing documentation in PLS which explicitly
lays out what field types are allowed in a pls-managed struct.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1667403003

Fixes #8: Seed indexes from index.yml

- Add `ParseIndexYAML` and `FindAndParseIndexYAML` functions
  to service/datastore
- Add YAML marshalling and unmarshalling for IndexDefinition and IndexColumn
- Add tests for the changes
- Imports: additional package: gopkg.in/yaml.v2

https://github.com/luci/gae/issues/8

Closes #8

R=dnj@chromium.org, estaab@chromium.org, iannucci@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1574353004

impl/prod: Expose Remote API scopes.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1632943003

Add UseBackground.

Fixes #39.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1630833002

Add tool to generate PropertyConverter implementations for proto messages.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1625773002

Fix logger for filter api change in luci-go.

API change will occur: https://chromiumcodereview.appspot.com/1622553005/

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1621993003

Fix fast string comparison bug.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1620873004

Add a test to ensure []byte => BlobKey PLS works.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1616033004

Encode nil datastore keys as PTNull.

This fixes a panic in serialize when it is asked to serialize a nil
*ds.Key, causing it to dereference its fields. Instead, a *ds.Key is
serialized as a PTNull entry like datastore.

Add some tests to assert proper conversion and serialization.

BUG=

Review URL: https://codereview.chromium.org/1576153003

Add tests for binary_tools functions

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1576353002

Add dscache.AlwaysFilterRDS function.

It's like dscache.FilterRDS, except it doesn't use IsGloballyEnabled. Intended
to be used by callers that know better when to apply (or not) the filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1579853002

Fix GAE <-> GAE RemoteAPI

Urlfetch.Get needs the user context, not the aeCtx.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1578943002

Make UseRemote work for dev_appserver too.

Now interacting with remote_api is as simple as:

    ```go
    ctx := context.Background()
    if err := prod.UseRemote(&ctx, "localhost:8080", nil); err != nil {
      panic(err)
    }
    // ctx is usable with luci/gae
    ```

Also changes the context-derivation function signature to be safer, since the
client's context will always be valid, and can be used for things, such as
logging even in the error handling case.

This also makes UseRemote work when used FROM appengine to another appengine
app.

Fixes #36.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1557223002

impl/memory: Fix time serialization encoding.

In a PropertyMap, time is serialized as an integer type. However, for
queries, time is serialized as a time type. This causes all queries on
time fields to select any encoded time properties, since PTTime > PTInt.

This patch causes queries to serialize PTTime properties as PTInts to
match the indexes.

Fixes #34.

Review URL: https://codereview.chromium.org/1550903002

Replace spammy log message in dscache with comment.

Fixes luci/gae#35.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/35

Review URL: https://codereview.chromium.org/1556193002

Add remote_api support

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/33

Review URL: https://codereview.chromium.org/1549223002

impl/memory: Add "PULL" as valid taskqueue method.

BUG=

Review URL: https://codereview.chromium.org/1550863002

Use prod logger's user context to read logging level

TBR=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/32

Review URL: https://codereview.chromium.org/1542263003

Make setting a Memcache Item with a nil Value OK

This conforms to the expectations that if you Set with a nil Value, it's the
same as setting to []byte{}... One less trap :)

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/31

Review URL: https://codereview.chromium.org/1541183002

Fix doc bug in pls.go


Implement Mail service.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/20

Review URL: https://codereview.chromium.org/1525923002

impl/prod: Fix logging field output.

By using the AppEngine context, logging calls would not have access to
logging fields that were set in subsequent contexts (most of them).
Retain both the AppEngine context (for logging calls) and the logger
context (for up-to-date logging configuration), and use the latter for
field extraction.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1523343002

Add filters for user service.

This brings the user service up to parity with the rest.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1530533002

Panic when non-runtime-handleable type errors occur

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/28

Review URL: https://codereview.chromium.org/1525873002

Fix nil-deref panic from Put.

We were using key before checking the error.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/26

Review URL: https://codereview.chromium.org/1523943002

Return error on Get(*Key).

Before it would end up returning ErrInvalidKey, which is silly :).

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org,
martiniss@chromium.org
BUG=https://github.com/luci/gae/issues/27

Review URL: https://codereview.chromium.org/1530503003

Clean up callback interfaces.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/17

Review URL: https://codereview.chromium.org/1521823003

Add User Testing interface to impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/24

Review URL: https://codereview.chromium.org/1522783002

Add logging service documentation.

Just points to github.com/luci/luci-go/common/logging, but explains how that
works and meshes with the rest of the service definitions.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1526453002

Handle unexpected entity fields gracefully.

Introduces a new field type 'extra', which allows full control over what to do
with the extra properties, instead of having all operations return unhelpful
conversion error messages.

This allows black-holes, schema upgrades and 'expando' style mixed
structured/unstructured objects. Please see the tests for usage and examples.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/13

Review URL: https://codereview.chromium.org/1525453002

Clean up PropertyLoadSaver interface.

This removes GetMetaDefault and Problem from the interfaces, and makes GetPLS
errors into panics, on the assumption that if you pass it a bad datatype, you
can't really do anything about it without recompiling, and failing fast is
better.

This also fixes a bug where doing any operations on an invalid (non datastore-
compatible) struct would return the unhelpful "unable to extract $kind" error.

R=dnj@chromium.org, hinoka@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1516173002

Transfer context deadline to AEContext.

It's needed to be able to tweak default RPC timeout (5 sec), in particularly
when using urlfetch service.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1515413002

Move gaelogging from luci/luci-go to luci/gae

Also makes impl/memory Use memlogger too for consistency. This change is
required to make luci-go logging work on appengine due to
3a50526e08194bd66487c7c03cccd2108c236468.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1498963003

Add API to allow you to get the non-transactional datastore or taskqueue.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1494223002

GAE Datastore - Fix derived meta types not being reflected properly

Stuff like "type FooBar string" were not allowed as metadata fields correctly.

BUG=

Review URL: https://codereview.chromium.org/1494803005

Fix User API (CurrentOAuth)

TBR=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1493183003

Add User service.

Doesn't add a memory implementation... it's easy enough to mock directly in the
test and doesn't have any complex systemic behaviors (that I know of) that are
necessary for a single mock implementation.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1500433002

Make appengine connection explicitly owned by impl/prod.

R=vadimsh@chromium.org, hinoka@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/1490243004 .

Update README.md

Typo :)
Fix races in txnBuf

Runs clean with `-cpu=2,4 -race` now. It's also a lot clearer to follow the
logic now. No more 'siblingLock'. At the time that each recursive transaction
starts it locks (in the normal way!). It then keeps a reference to the parent
layer with the locks disabled for its internal usage.

This refactor makes it so that all (interacting) operations at a given 'layer'
in a transaction are mutually exclusive. Previously only nested transactions
would be mutually exclusive.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1434873003

Add some race-detector tests

The impl/memory one is fine, but the txnBuf one is currently failing.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1437553002

Fix another race.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1430193002

Fix race in impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/16

Review URL: https://codereview.chromium.org/1411573014

Allow metadata fields to be PropertyConverters for symmetry.

This makes the easiest forms of parsed kinds/ids extremely simple to implement,
as well as composable. The MetaGetterSetter interface requires every top-level
struct to implement it, since allowing embedded implementations will get the
wrong value for the derived "$kind".

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1414043006

service/datastore: Call MGS GetAllMeta() in Save()

Fix a bug where a structPLS that uses Save(withMeta: true) will ignore
its object's MetaGetterSetter overrides and use struct-default.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1425283002

Decouple PLS from MGS.

Decouple PropertyLoadSaver from MetaGetterSetter. This allows
implementing structs to override either one of those interfaces without
depending on spooky the interaction between them implemented by
"structPLS".

"structPLS" no longer magic-detects if its type implements
MetaGetterSetter and defers to it. Instead, a struct that implements
MetaGetterSetter and wants to use "structPLS" cool meta defaults
can call GetPLS(this) to get a handle on that base functionality.

BUG=

Review URL: https://codereview.chromium.org/1427933002

Use the sort.Search api correctly.

I was using it again and I realized that I used it wrong here :(. Includes
regression test.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1409613002

Add ability to override just the metadata in a struct.

This is useful when the '$id' is actually multiple fields packed into a string,
but it's generally available for any metadata field.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/15

Review URL: https://codereview.chromium.org/1401533007

Add Exists and ExistsMulti convenience methods

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1398103003

Add VersionID to impl/memory info service implementation

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1403873002

Add Gob encoding functions for Key.

This is so that *Key can transparently be encoded in tumble mutation structs.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533020

Review URL: https://codereview.chromium.org/1397673003

Add transaction buffer filter.

This will be used for running multiple non-cooperative idempotent tasks inside
of tumble, which require seeing the previous tasks' effects on the entity group.

However, the effect of having transactions transparently buffer datastore
interactions until the final transaction commit is a generally useful alteration
of the current behavior.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309803004

Fix bugs in impl/memory queries.

This prevents built-in indexes from being selected when using ancestor and
non-__key__ inequality filters, and also fixes a bug where a multi-iterator on
builtin indexes using __key__ inequality with ancestor would return empty result
sets.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1380613002

Add a NewDatastore helper function.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379673002

Make impl/memory have an AppID.

This associates an immutable AppID with the datastoreData object (e.g. the thing
that actually holds the data). It gets rid of the hard-coded globalAppID hack.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1369353003

Improve the way "consistent" works.

This avoids the overhead of taking many many snapshots when the in-memory
datastore is in full-consistency mode.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379563004

Fix bugs in service/datastore

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1378613003

Add Key.HasAncestor()

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1377863004

Add the ability to disable special automatic entities in impl/memory

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1372473002

Add AutoIndex to datastore Testable interface.

This allows test writers to set Consistency(true) and AutoIndex(true), and
pretend like the datastore is a fully consistent database which always has
perfect indexes. Will be used for the recursive transaction manager to
maintained tiered automatic mini-datastores in memory.

Also add some more tests for queries in transactions, and YAML rendering for
*IndexDefinition.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021, https://github.com/luci/gae/issues/9

Review URL: https://codereview.chromium.org/1364333002

Add Consistent(bool) to Testing interface

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1366793002

Add missing Count api

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1367003002

Refactor serialization helpers from impl/memory -> service/datastore/serialize

These will be used to aid in joining query results for nested transactions.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1365743002

Add ability to estimate the size of a PropertyMap

This will be used to track in-flight transaction sizes.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1363063002

Fix minor bug in ParseIndexColumn

Also improve error message.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1359963003

Add Kind/StringID/IntID back to Key.

Rename Last -> LastTok.

luci-go change here: https://chromiumcodereview.appspot.com/1359813004

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1358063003

service/datastore: misc improvements in properties.go

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354343002

Add missing AllocateIDs API.

No idea why I missed it. Tweaked a bit to get rid of some redunant info.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1357343002

Add API compat warning :)


Fix memcache.Interface.Get to return a memcache.Item

This intentionally doesn't change GetMulti. Since GetMulti is used for batch
operations, it's actually more convenient to have a pipeline of actions (e.g.
AddMulti -> GetMulti). Otherwise you have to carry around a []Item and a
[]string and keep them both in sync (overhead, error prone).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/3

Review URL: https://codereview.chromium.org/1354333002

Make Get operations only serialize the bare minimum.

Now PropertyLoadSaver has GetAllMeta, which is like Save, except that it ONLY
returns the metadata. This avoids situations where zero-initialized structs
aren't expecting to be serialized for a nominally write-only operation (e.g.
Get).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/11

Review URL: https://codereview.chromium.org/1358743002

Fix zero time bug.

`time.Time{}` needs to be able to round-trip everywhere. Previously
various things were implicitly adding the Local or UTC timezones, even
when the time value was zero.

R=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/10

Review URL: https://codereview.chromium.org/1345043005

Refactor keys and queries in datastore service and implementation.   *
datastore.Query and datastore.FinalizedQuery are now first class objects     in
the service/datastore api.   * datastore.Key is similarly a first class object.
All luci/gae users will     need to interact with `*ds.Key` now. This object
gained all the superpowers     that previously existed in the dskey subpackage
of service/datastore.

The new Query objects are inspectable, and adjustable. Additionally, this
homogenizes they query generation between different implementations. The
normalization which normally occurs on the server-side is now done client-side.
This should ensure consistency between 'prod' and 'memory' implementations.

Additionally, the new FinalizedQuery object gains a GQL() method which should
(hopefully) allow trivial debugging by printing a query. You should be able to
copy and paste the generated GQL* into the appengine dashboard and see what
your request was querying for.

* With the exception of Cursors, which currently have no textual format in the
GQL schema.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533024

Review URL: https://codereview.chromium.org/1355783002

Add simplest service/urlfetch implementation.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354083002

Allow PRESUBMIT run pcg on trybuilder.

R=vadimsh@chromium.org
BUG=529936

Review URL: https://codereview.chromium.org/1345303002

Make SetTransactionRetryCount actually work.

It used to store the state in ephemeral dsImpl object that is minted with each
call to datastore.Get(...). Store it in more durable dataStoreData instead.

R=iannucci@chromium.org
BUG=https://github.com/luci/gae/issues/2

Review URL: https://codereview.chromium.org/1347843002

Add IsDevAppServer implementation to impl/memory.

Returns 'true', since impl/memory is sort of a dev sandbox too.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1347813003

Use int64 to store time.Time internally in memory implementation.

Uint64 causes overflow when storing zero time.Time{} (that due to some Go
stubborn weirdness is represented by negative timestamp).

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334043003

Make PRESUBMIT.py run checks via 'pcg' if it is available in PATH.

R=maruel@chromium.org, iannucci@chromium.org
BUG=chromium:529936

Review URL: https://codereview.chromium.org/1342673002

Add emulation of transaction retries to memory impl of RunInTransaction.

Available via Testable().SetTransactionRetryCount().

BUG=https://github.com/luci/gae/issues/2
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1335083002

Switch to external stringset implementation.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1312433013

Add pcg to ignore impl/prod

This should fix the travis redness.

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334833002 .

Implement projection queries correctly.

* Add Project and ForIndex to Property in service/datastore
* Remove PTBoolFalse/PTBoolTrue silliness (now just PTBool)
* Remove ByteSlice silliness (now []byte behaves exactly like string w.r.t.
  indexing. If you don't want it indexed, put `gae:",noindex"` on it.)
* Make indexes index 'underlying' types. e.g. PTTime is not distinguished from
  PTInt in the index, nor are the various byte sequence types.
* Make impl/prod correctly extract data from internal datastoreIndex objects
  when doing projection queries. This allows doing a projection into
  []PropertyMap. Since all data is bussed internally as PropertyMap (instead of
  crazy PropertyLoadSaver implementations), this is required to make projection
  queries work. I'm pretty confident that this won't break, however, as it's all
  derived from protobufs which are extremely unlikely to change.

  To make this better, we'd have to implement the protobuf protocol directly in
  luci/gae instead of routing through the SDK. Not as absurd as it sounds, but
  definitely out of scope for this CL :).
* Include extremely basic test for impl/prod using aetest. This test takes 12s
  to run on my laptop. *sigh*.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/1

Review URL: https://codereview.chromium.org/1336443003

More minor formatting of README.md


Fix formatting in README.md


impl/memory: Implement Queries

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1302813003

filter/count: Fix alignment on 32-bit systems.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1306373002 .

datastore: Add DecodeCursor to [Raw]Interface.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309933003 .

Update README to use CQ.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1301833005

Update codereview.settings for CQ.

Also add infra-reviers+luci-gae@ to CC list.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1286973004 .

Add Chrome infrastructure CQ config.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1298323004 .

prod: fix timezone conversion from AppEngine layer

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1298423005 .

Fix miscellaneous prod bugs.

- Fix bug where "nil" Parent field was being translated into non-nil
  empty key.
- Use fully qualified AppID for key AppID in checkfilter.
- Use a context-installed singleton to probe AppEngine runtime parameters.
- Bump Task field count to match gae/AppEngine count.
- Fix UTC time check when Locations are the same, but have different
  pointers.
- Pass PropertyLoadSaver value array (rather than source) in
  prod/PutMulti :)
- Use struct name as default even when it implements PropertyLoadSaver.

R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1289323002 .

Fix key encoding to be null terminated

This will allow the use of automatic indicies to fulfil basic ancestor queries.
Otherwise keys with many components sort after keys with few components (instead
of lexicographically per token)

R=dnj@google.com, tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286093007 .

Add single-collection gkvlite iterator.

Multi-collection iterator is up next.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1300493002 .

Split off serialization and key functions to their own packages.

This makes the service/datastore package smaller and more single-purpose.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1292913002 .

impl/memory: Make queries self-validate

This moves validation and normalization logic into the query construction
process itself. This makes it so that queries are either valid and normalized,
or they contain an error (and query building 'sticks' on the first error
encountered).

This replaces the previous
build-a-questionable-query-and-then-try-to-make-it-sensible
logic (which was copied from dev_appserver for the most part).

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286943002 .

Add testable interface for datastore.

Includes the ability to manipulate the 'eventual consistency' state of the
datastore, as well as the ability to add compound indicies.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1285703002 .

Make LazyMultiError constructed via function

Followup to https://codereview.chromium.org/1249933002/

Counterpart to https://codereview.chromium.org/1284323002/

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1291813003 .

Refactor: Rename some files.

R=tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1281173002 .

Make obtaining the Testable interface actually part of the interface.

Reduces the number of free casts, and will actually work with the upcoming
Testable interface for datastore, which uses a checkfilter (and so casting
won't work there anyway).

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1282673002 .

Get rid of awkward proto argument to Interface.Run

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1279703003 .

A transparent cache for datastore, backed by memcache.

Check doc.go for details, but:
  * allows cache disablement on a per-model/Key basis
  * allows cache expiration on a per-model basis
  * efficient (non-gob) encoding scheme
  * gzip for large entities
  * allows randomized sharding for Gets
    * allows good throughput for hot items
    * shards may be determined on a per-Key basis
  * caches negative hits

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269113005 .

Re-add metadata passthrough on Get operations

It got lost in translation during the RawInterface conversion. This will be used
to e.g. allow SomeStruct to set a default dscache.enable metadatum when doing
e.g. Interface.GetMulti([]SomeStruct{...}).

R=dnj@chromium.org, dnj@google.com, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270113002 .

Use time.Time for internal memcache memory implementation

Followup to https://chromiumcodereview.appspot.com/1270063003/

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1262173003 .

Make the rest of the services have a similar raw/user interface structure.

Also implement missing functionality from taskqueue/memcache in the memory
implementation.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063003 .

Add 'user friendly' datastore API.

This is a thin veneer on top of datastore.RawInterface, and may be used
with any filters, any backend, and it may be used interchangably and
inderleved with datastore.RawInterface.

In particular, this adds a convention for embedding keys inside of
loadable/savable objects.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259593005 .

Rename rawdatastore -> datastore

Also rename some of the .go files in the service to prep for
https://codereview.chromium.org/1259593005/

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063002 .

Fix obviously broken error handling.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269593002 .

Make rawdatastore API safer for writing filters.

This removes Get/Put/Delete, since they always end up being implemented
as wrappers around the Multi versions anyway, it just caused unnecessary noise.
They'll come back with the 'user friendly' datastore interface, but there's no
reason to have them at every layer of the cake.

It moves the *Multi methods to be callback-based, which makes interacting with
them easier (since you don't have to do lots of slice allocations).

It introduces 'checkFilter' to rawdatastore which guarantees that filters and
backend implementations don't need to do lots of redundant checks to sanitize
their inputs.

It adds a new API to the Info service to retrieve the current namespace.

R=dnj@google.com, estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1253263002 .

Add Toggle for rawdatastore.

This will be used to control cacheability for entities, and allows setting a
default value in the tag without ambiguity (since the 'zero-value' for bools is
false, a simple bool wouldn't provide a way to override defaulted-true field).

R=dnj@google.com, vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259463002 .

Update README.md formatting


Add PRESUBMIT.py


Minor doc improvements

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1241423003 .

Add LICENSE


Revert "Enable coveralls.io"

This reverts commit fc93f78cf1d04d25fb3f52dd8040dba88649494e.

Enable coveralls.io


Remove most error code from luci/gae

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1249863004 .

Improve docs

Simplify godoc by making some functions methods
Rename 'filters' to 'filter' to be more consistent with 'impl' and 'service'.

R=estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1247283003 .

Refactor a bit.

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1243323002 .

Switch to other coveralls badge


Force timezone to be non-UTC when testing non-UTC


Fix gae.DSTransactionOptions


Minor change for Travis :)


Add README.md


Add .travis.yml


Add codereview.settings


Fix imports and tests

Run goimports over everything for good measure as well.

Fix DSKey encoding.

Specifically, IntID's always sort before StringID's. This is according to
https://cloud.google.com/appengine/docs/go/datastore/queries#Go_Key_filters

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1231863018

Simplify memlock and make it less racy.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1233413002

Rename poorly named files.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240123003

Improve memory implementation of gae to provide the full RawDatastore interface.

Still no query support yet, working on that next, but this enables *Multi apis.
Also do a few other code cleanups.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1242043005

Add LazyMultiError.

This is essentially a MultiError, except that it knows how big it should get
if it's ever assigned an actual error, and it only does its allocation lazily.
This is useful for implementing e.g. PutMulti and other Multi-style methods.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234493005

Add NewDSKeyFromEncoded convenience function.

Also placate golint :).

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1238573004

Convert 'Testable' interface into an optional filter.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227203004

Add "filters/count" filter.

This filter allows you to track the number of times a give API has been
invoked, as well as the number of errors that the API has returned. Will mostly
be used for testing.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1239103003

Add filters for gae library.

Filters allow users to add their own wrappers around the underlying gae
implementation. They can do anything that an implementation could do, except
that their backend is another gae library interface, instead of the SDK or
memory implementations. Filters are also necessary (instead of just stacking
implementations with SetXXFactory functions), due to transactions, which change
the context object.

They will be used to implement (at least):
  * caching (hitting memcache before datastore)
  * transaction buffering (giving a self-consistent view of the datastore
    within a transaction, buffering all writes to the end of the transaction
    to make transaction sizes smaller, etc.)
  * featureBreaker (for tests, breaking certain APIs with certain errors)
  * counter (for tests, counting the number of times an API was hit)

It may also be used for:
  * ACL enforcement (rejecting access to datastore items the user can't get,
    filtering queries, etc.)

Thi CL also implements a counter filter as a demo (which will also be used for
testing other filter implementations to ensure that the filter being tested).

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222983002

Some miscellaneous fixes

  * Fix DSKeyIncomplete to check for nil
  * Fix docstring in memory/context.go
  * Make GlobalInfo support BrokenFeatures
  * Fix memory TaskQueue to fill in the default queue name, per SDK spec.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234343006

Simplify taskqueue interface.

The removed complexity was a holdover from the initial versions of this library.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1235563007

Change RawDatastore to do less reflection.

Followup CL will provide the 'magic' reflection-full Datastore interface.

This change makes filters easier to implement, none of the layers of the stack
are required to do reflection, instead of the previous design where all of the
layers were forced to do reflection.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227183003

Move dummy service implementations into their own package.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1226063003

Reland: Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

Original broke because endpoints needed to be moved to a version which relies
on the new SDK.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240573002

Revert "Refactor current GAE abstraction library to be free of the SDK*"

This reverts commit a91c4ac2149f8bbe3711c4c662b6d500c9a8f799.

CQ didn't validate that everything builds. Consequently, note everything builds
:D.

TBR=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1230303003

Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222903002

Use SetFinalizer to avoid leaking Snapshots.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1223263006

Roll infra/go/src/github.com/luci/luci-go/ to
b85c6dd748361e8244122faac44b5beefcd2db37.

$ git log cd33aabb0..b85c6dd74 --date=short --format='%ad %ae %s'
2015-07-07 robbie Rename funnybase and add more serializations.
2015-07-07 yyanagisawa Reference implementation of LinkToPrimaryHandler.

R=pgervais@chromium.org, agable@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1229503007 .

A client/server helper wrapper for endpoints in Go.

The epservice directory/main program generates a appengine app which links all
the available endpoints implementations by finding packages which implement
RegisterEndpointsService. It then runs this with dev_appserver.py.

Subpackages of epservice are supposed to be one-folder-per-service.

The epclient directory/main program/go-generate invocation runs the epservice
server, and then runs the golang client library generator to generate go client
libraries for all the services it finds.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153473008

Non racy tests for memlock.

Bonus, no more time.Sleep, and the tests are 100% faster (e.g. they take 0.04s
instead of 0.08s :D )

R=dnj@chromium.org, maruel@chromium.org, vadimsh@chromium.org
BUG=496436

Review URL: https://codereview.chromium.org/1159173004

Add context-aware "time" library wrapper.

TEST=unittest
R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1154213012

A simple memcache lock for appengine.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/986553002

Last fix for 32 bit tests?

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159623005

Fix tests to run on 32 bits.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159343002

Fix overflow on 32bit which causes tests to fail on 32bit.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1162813002

Add initial Query generation, correctness checking and index generation.

The ability to execute a query will come in a following CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1160253002

Simplify memory package interface.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1143053004

Simple memory testing for gae/wrapper

Provides partial implementations of Memcache, TaskQueue and Datastore.

Most missing things I just didn't care to implement... Datastore queries are
conspicuously absent, and I'll be doing those in a separate CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1152383003

Better attempt at an appengine wrapper.

Includes implementation using old-style appengine, and will be able to be
expanded to include support for new-style appengine SDK. I also have a partial
in-memory testing implementation in the wings which will be a followup CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1151473003

go: infra/libs/* now live in luci-go.

Companion commit that moved them there:
https://github.com/luci/luci-go/commit/15984f36e8e3a78d9893369993ea3b999e919d26

R=iannucci@chromium.org,maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153883002

cipd: Skip test on Windows for now.

The code compiles, but unit tests don't work because Windows handles symlinks
and other file system stuff differently.

R=iannucci@chromium.org
BUG=487483

Review URL: https://codereview.chromium.org/1133353002

Fix meta lib to be correctly compilable.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/998153002

Thin library for interacting with the GAE datastore metadata entities.

Only implements the one function I need right now, but can be extended for
others as needed.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/985583003

Simple GAE/goon Context wrapper

This is useful for restricting an API to implement 'datastore without queries'
or 'key-only context', etc.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/981003003
==========

[dnj, 2016-09-01 05:27:25]

Description was changed from

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
schanges. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)
  taskqueue.Add(datastore.WithTransaction(c, nil), task)
  prod.AEContext(datastore.WithTransaction(c, nil))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

Implement info.ServiceAccount() on devserver via info.AccessToken(...).

R=iannucci@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2220193002

Update cloud package.

Fixes #54.

BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2217063003

Remove empty namespace assertion from impl/memory.

This was fixed upstream in:
 
https://github.com/golang/appengine/commit/5a9e2b406e4e87cd8a88d1881a31b1248d...

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2219673002

GAE CL for luci-go common refactor

R=nodir@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/2192003003 .

Add an option to install just Cloud Datastore.

TBR=iannucci@chromium.org
BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2179843004

cq.cfg: use luci builders

do not use buildbot builder
update luci bucket name

R=estaab@chromium.org, dnj@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2179543003

Add lightning talk that I didn't gave at Gophercon '16


Add cloud datastore implementation.

Fixes #52.

BUG=

Review-Url: https://codereview.chromium.org/1957953002

datastore: Update AllocateIDs to take keys.

Update the Datastore's AllocateIDs call to accept variadic arguments similar
to Get/Put/Delete/Exists. Structs can be supplied to populate their keys
directly, and keys or key slices can be supplied to directly allocate keys.

This adds some Key helper functions:
- SameKind, which asserts that keys reference the same kind.
- Partial, which returns a partial key given a source key.
- WithID, which returns a derivative key with the specified ID.
- NewIncompleteKeys, which allocates a slice of incomplete keys given a kind.

BUG=

Review-Url: https://codereview.chromium.org/2007123002

Refactor multiarg, split MGS/PLS.

Refactor the multiarg construction method to handle the various cases
inline, rather than defining and invoking methods for specific
scenarios. This enables PropertyLoadSaver-based methods to be handled
separately from MetaGetterSetter-based methods, which, in turn, enables
the two to be implemented independently.

A bit of boilerplate is removed and the odd case of MGS/PLS Map/Chan fits
better. Additionally, a few double-reflection code paths are optimized
along the way.

BUG=None

Review-Url: https://codereview.chromium.org/2048933004

make swarming trybuilders non-experimental

R=iannucci@chromium.org, dnj@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2083193006

cq.cfg: enable experimental swarming builders

R=estaab@chromium.org, vadimsh@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2090433002

Add datastore dumper tool.

This allows you to dump the datastore in whole or part, with or without special
formatting directives.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2037433002

Add TrimmedAppID to info service.

Also refactor info service to share common 'augmented' methods in
the service/info package itself, ala service/datastore.

Split out from CL: https://chromiumcodereview.appspot.com/1537883002

R=dnj@chromium.org
BUG=550684

Review-Url: https://codereview.chromium.org/2062613002

Add DefaultVersionHostname to impl/memory/info.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2028423002

Fix typo in pls.go docs


Allow uint16 and uint32 in MakeKey.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036543003

Add GetNoTxn to txnBuf as a hack.

I really don't like this but short of fixing #23, I don't think there's much we
can do.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036593002

Change gae to apache 2 license.

Update AUTHORS, CONTRIBUTORS, and LICENSE.
Add apache 2 license check to PRESUBMIT.py.

Ran:
  git sed 's/Chromium Authors/LUCI Authors/'
  git sed 's/governed by a BSD-style license that can be/governed under the
Apache License, Version 2.0/'
  git sed 's/found in the LICENSE file/that can be found in the LICENSE file/'
  git checkout HEAD -- $(git status -sb | grep '^ T' | cut -c 4-)
  git checkout origin/master -- $(git diff origin/master --name-only | grep
third_party)

BUG=

Review-Url: https://codereview.chromium.org/2021753002

Change codereview server to be consistent with other luci projects.

BUG=

Review-Url: https://codereview.chromium.org/2020933002

dscache: Reduce log level of memcache failures.

Warning-level failures quickly pollute the AppEngine log space. This
change converts some of the more common warnings to Debug-level logs so
the Warning space remains reserved for higher attention calls.

Also add a warning about lock eviction poisoning to THERE BE DRAGONS
section of the doc.

BUG=

Review-Url: https://codereview.chromium.org/1987353002

datastore: variadic Get, Put, Exists, Delete.

Fixes #51.

Update Get, Put, Exists, and Delete to be variadic. This will make the
interface more usable and get rid of boilerplate in client code.

Get, Put, Exists, and Delete now accept variadic arguments. Any one of
these arguments can be an argument that was previously valid or
the *Multi version of those functions. This effectively obsoletes the *Multi
versions, but they are left in for backwards compatiblility with the stated
intention of removing them at a later date.

Get/Put/Exists/Delete will now return a single error in the one-argument
case or a MultiError in a multiple-argument case. Note that now that single
argument case can take a slice type, the single error may itself be a
MultiError if the single argument is a slice.

Review-Url: https://codereview.chromium.org/2011773002

filter/txnBuf: Use fully-qualified App ID.

Use the fully-qualified app ID when creating the in-memory datastore for
transaction buffer. Previously, transaction buffer would use the
standard App ID, while production datastore would use the
fully-qualified App ID, causing key incompatibility errors in
production.

Cleanup impl/memory's NewDatastore function to not install superfluous
services into the Context. This draws a cleaner line between
impl/memory's dual purposes as both a rich testing enviornment and a
production-grade in-memory datastore.

In addition, have impl/memory's info service differentiate between the
two. This causes tests for filter/txnBuf to properly fail lacking this
fix, enabling regression protection.

To support this, impl/memory's Info service will now parse the supplied
App ID. If it contains a "~" character, it is split into fully-qualified
and regular AppID. Otherwise, the two will be the same.

BUG=

Review-Url: https://codereview.chromium.org/1929423002

Travis: Increase system specs, limit concurrent.

Increase the Travis system specs and limit the number of concurrent
tests to try and avoid the signal-kill error that we're seeing in Travis
ATM.

TBR=iannucci@chromium.org
BUG=None

Review URL: https://codereview.chromium.org/1922583002

pcg: Bump test time to 20 minutes.

Travis is currently timing out.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1918713003

Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1917513002

impl/memory: datastore and memcache namespacing

Update GetNamespace() to return an additional boolean, which will be
true if the namespace was set and false if it was not. The AppEngine SDK
defaults to having no namespace installed ("", false), but will allow
the user to install any namespace. This allows a user to differentiate
between no namespace and an empty namespace ("", true).

Make impl/memory's datastore query operation to error if the namespace has
been set to the empty namespace (""). This reflects observed production
behavior.

BUG=

Review URL: https://codereview.chromium.org/1916463004

Bump max_duration for CI tests to 480s

Hopefully Fixes #48.
Fix memory corruption bug in impl/memory

When you `SetCollection` in gkvlite, it invalidates the old
collection, but keeps the Items of the previous collection. The way
that impl/memory was using SetCollection was more along the lines of
GetOrCreate, which could cause corruption or other issues if two
codepaths did SetCollection simultaneously.

This fixes the corruption bug by removing SetCollection from the
memStore interface and replaces it with a GetOrCreateCollection,
which checks for the collection, and only calls SetCollection if it
doesn't exist. As a bonus, this seems to give a slight performance
speedup over the previous broken code.

Additionally, this forces all uses of VisitItemsAscend to use
a Snapshot, to avoid accidental modification of a Collection while
its being iterated over, which can lead to the situation discussed
above, or other nasty errors.

Finally, this adds an optional 'gkvlite tracer' functionality
which came in (extremely) handy for finding this bug. When given
a folder, it will emit go-code-like invocations of the functions run
on every collection, and includes enough information to reveal which
functions are being called on the root collection v. snapshots, and
if there are any overlapping VisitItemsAscend invocations, etc.

R=dnj@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1911263002

Travis: Build against Go1.6.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1911293002

Remove race from query finalization.

BUG=

Review URL: https://codereview.chromium.org/1910293002

datastore: Fix AddIndexes with existing namespaces

AddIndexes previously only added an index for the current namespace.
Since indexes are global, they should be added for all existing
namespaces.

BUG=

Review URL: https://codereview.chromium.org/1894403002

service/taskqueue: Add NewPOSTTask, remove NewTask

Add NewPOSTTask, which mirrors the method of the same name in
AppEngine's Go task queue SDK.

Remove NewTask from the taskqueue Interface, since it was basically
useless.

BUG=

Review URL: https://codereview.chromium.org/1890983004

Add __namespace__ metadata query support to impl/memory.

Ref: https://cloud.google.com/datastore/docs/concepts/metadataqueries

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1892343002

Add Info Testable interface, implement for memory.

This adds a Testable interface to the Info service which allows it to
install Info parameters in Contexts.

Add support for setting version ID and request ID to impl/memory Info
service.

BUG=

Review URL: https://codereview.chromium.org/1871943003

Add PopulateKey method to datastore.

Add PopulateKey, a method that, given an object, fills in its key
fields.

Fixes #44.

BUG=

Review URL: https://codereview.chromium.org/1867963003

Fix missing/broken ds.Stop handling in raw DS.

Several places were missed when patching "ds.Stop" to being correctly filtred
at the top interface rather than the raw interface.

At the top-level, GetAll and Count both did not filter the Stop error.
At the filter level, several filters were ignoring error values from
their callbacks.

Also fixes an issue in count filter where the ds.Stop error was being
incorrectly filtered at the RawInterface level.

Add datastore iterating query filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1846123002

Add datastore iterating query filter.

BUG=

Review URL: https://codereview.chromium.org/1843313004

Don't filter "Stop" error in raw interface.

The Stop error is a high-level error used by the datastore service to
indicate that an iterative operation should stop rather. It should
appear to the user as a "nil" error.

However, currently the datastore raw implementations are filtering Stop
and converting it to "nil". This prevents other filters from
identifying Stop events from their lower level callbacks.

Update raw datastore methods to not filter Stop, and update their tests
to accept Stop as a valid successful result.

BUG=

Review URL: https://codereview.chromium.org/1844183003

Make ExistsMulti return a BoolList

BoolList has some convenience functions (e.g. to check if Any() or All() are
set.). It's compatible with current uses of ExistsMulti.

R=dnj@chromium.org, estaab@chromium.org, nodir@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1816413002

txnbuf: Add write count budget, size budget.

Add a "maximum number of datastore writes" budget to the transaction
buffer. Rework the size budget to remove preemptive child transaction
failures and directly derive budget from the parent.

BUG=

Review URL: https://codereview.chromium.org/1776833002

Add wrappers for the module module

BUG=570879

Review URL: https://codereview.chromium.org/1772943003

Add a MustNamespace that panics if the namespace wasn't valid

BUG=

Review URL: https://codereview.chromium.org/1766593002

Auto-project on distinct inequality fields.

BUG=

Review URL: https://codereview.chromium.org/1745703002

Add header text to proto-gae tool

In order to appease our presubmit scripts, default to the Chromium Authors
copyright header.

R=dnj@chromium.org, estaab@chromium.org, hinoka@chromium.org,
martiniss@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1718123003

dscache: Only log unusual memcache errors.

BUG=None
TEST=None
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1695833002

Use ShouldResemble instead of ShouldResembleV.

Now that goconvey incorporates our ShouldResembleV method directly, we
can go back to using ShouldResemble instead of hacking in
ShouldResembleV!

BUG=None

Review URL: https://codereview.chromium.org/1679093002

Add support for uint8, uint16 and uint32 in luci/gae

This has come up for like 3 models in DM so far, and I'm tired of having this
be a stupid "gotcha!". I was working around it with some PropertyConverter, but
with this change it will just be obvious what the heck is going on.

They all map to positive int64 values (which should make indexes obvious), and
they include out-of-bounds checks on deserialization.

I excluded uint64, since there's no way to properly encode it in the native
datastore representation which still makes any sense for indexes.

I also added some (apparently?) missing documentation in PLS which explicitly
lays out what field types are allowed in a pls-managed struct.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1667403003

Fixes #8: Seed indexes from index.yml

- Add `ParseIndexYAML` and `FindAndParseIndexYAML` functions
  to service/datastore
- Add YAML marshalling and unmarshalling for IndexDefinition and IndexColumn
- Add tests for the changes
- Imports: additional package: gopkg.in/yaml.v2

https://github.com/luci/gae/issues/8

Closes #8

R=dnj@chromium.org, estaab@chromium.org, iannucci@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1574353004

impl/prod: Expose Remote API scopes.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1632943003

Add UseBackground.

Fixes #39.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1630833002

Add tool to generate PropertyConverter implementations for proto messages.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1625773002

Fix logger for filter api change in luci-go.

API change will occur: https://chromiumcodereview.appspot.com/1622553005/

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1621993003

Fix fast string comparison bug.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1620873004

Add a test to ensure []byte => BlobKey PLS works.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1616033004

Encode nil datastore keys as PTNull.

This fixes a panic in serialize when it is asked to serialize a nil
*ds.Key, causing it to dereference its fields. Instead, a *ds.Key is
serialized as a PTNull entry like datastore.

Add some tests to assert proper conversion and serialization.

BUG=

Review URL: https://codereview.chromium.org/1576153003

Add tests for binary_tools functions

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1576353002

Add dscache.AlwaysFilterRDS function.

It's like dscache.FilterRDS, except it doesn't use IsGloballyEnabled. Intended
to be used by callers that know better when to apply (or not) the filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1579853002

Fix GAE <-> GAE RemoteAPI

Urlfetch.Get needs the user context, not the aeCtx.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1578943002

Make UseRemote work for dev_appserver too.

Now interacting with remote_api is as simple as:

    ```go
    ctx := context.Background()
    if err := prod.UseRemote(&ctx, "localhost:8080", nil); err != nil {
      panic(err)
    }
    // ctx is usable with luci/gae
    ```

Also changes the context-derivation function signature to be safer, since the
client's context will always be valid, and can be used for things, such as
logging even in the error handling case.

This also makes UseRemote work when used FROM appengine to another appengine
app.

Fixes #36.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1557223002

impl/memory: Fix time serialization encoding.

In a PropertyMap, time is serialized as an integer type. However, for
queries, time is serialized as a time type. This causes all queries on
time fields to select any encoded time properties, since PTTime > PTInt.

This patch causes queries to serialize PTTime properties as PTInts to
match the indexes.

Fixes #34.

Review URL: https://codereview.chromium.org/1550903002

Replace spammy log message in dscache with comment.

Fixes luci/gae#35.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/35

Review URL: https://codereview.chromium.org/1556193002

Add remote_api support

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/33

Review URL: https://codereview.chromium.org/1549223002

impl/memory: Add "PULL" as valid taskqueue method.

BUG=

Review URL: https://codereview.chromium.org/1550863002

Use prod logger's user context to read logging level

TBR=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/32

Review URL: https://codereview.chromium.org/1542263003

Make setting a Memcache Item with a nil Value OK

This conforms to the expectations that if you Set with a nil Value, it's the
same as setting to []byte{}... One less trap :)

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/31

Review URL: https://codereview.chromium.org/1541183002

Fix doc bug in pls.go


Implement Mail service.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/20

Review URL: https://codereview.chromium.org/1525923002

impl/prod: Fix logging field output.

By using the AppEngine context, logging calls would not have access to
logging fields that were set in subsequent contexts (most of them).
Retain both the AppEngine context (for logging calls) and the logger
context (for up-to-date logging configuration), and use the latter for
field extraction.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1523343002

Add filters for user service.

This brings the user service up to parity with the rest.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1530533002

Panic when non-runtime-handleable type errors occur

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/28

Review URL: https://codereview.chromium.org/1525873002

Fix nil-deref panic from Put.

We were using key before checking the error.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/26

Review URL: https://codereview.chromium.org/1523943002

Return error on Get(*Key).

Before it would end up returning ErrInvalidKey, which is silly :).

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org,
martiniss@chromium.org
BUG=https://github.com/luci/gae/issues/27

Review URL: https://codereview.chromium.org/1530503003

Clean up callback interfaces.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/17

Review URL: https://codereview.chromium.org/1521823003

Add User Testing interface to impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/24

Review URL: https://codereview.chromium.org/1522783002

Add logging service documentation.

Just points to github.com/luci/luci-go/common/logging, but explains how that
works and meshes with the rest of the service definitions.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1526453002

Handle unexpected entity fields gracefully.

Introduces a new field type 'extra', which allows full control over what to do
with the extra properties, instead of having all operations return unhelpful
conversion error messages.

This allows black-holes, schema upgrades and 'expando' style mixed
structured/unstructured objects. Please see the tests for usage and examples.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/13

Review URL: https://codereview.chromium.org/1525453002

Clean up PropertyLoadSaver interface.

This removes GetMetaDefault and Problem from the interfaces, and makes GetPLS
errors into panics, on the assumption that if you pass it a bad datatype, you
can't really do anything about it without recompiling, and failing fast is
better.

This also fixes a bug where doing any operations on an invalid (non datastore-
compatible) struct would return the unhelpful "unable to extract $kind" error.

R=dnj@chromium.org, hinoka@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1516173002

Transfer context deadline to AEContext.

It's needed to be able to tweak default RPC timeout (5 sec), in particularly
when using urlfetch service.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1515413002

Move gaelogging from luci/luci-go to luci/gae

Also makes impl/memory Use memlogger too for consistency. This change is
required to make luci-go logging work on appengine due to
3a50526e08194bd66487c7c03cccd2108c236468.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1498963003

Add API to allow you to get the non-transactional datastore or taskqueue.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1494223002

GAE Datastore - Fix derived meta types not being reflected properly

Stuff like "type FooBar string" were not allowed as metadata fields correctly.

BUG=

Review URL: https://codereview.chromium.org/1494803005

Fix User API (CurrentOAuth)

TBR=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1493183003

Add User service.

Doesn't add a memory implementation... it's easy enough to mock directly in the
test and doesn't have any complex systemic behaviors (that I know of) that are
necessary for a single mock implementation.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1500433002

Make appengine connection explicitly owned by impl/prod.

R=vadimsh@chromium.org, hinoka@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/1490243004 .

Update README.md

Typo :)
Fix races in txnBuf

Runs clean with `-cpu=2,4 -race` now. It's also a lot clearer to follow the
logic now. No more 'siblingLock'. At the time that each recursive transaction
starts it locks (in the normal way!). It then keeps a reference to the parent
layer with the locks disabled for its internal usage.

This refactor makes it so that all (interacting) operations at a given 'layer'
in a transaction are mutually exclusive. Previously only nested transactions
would be mutually exclusive.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1434873003

Add some race-detector tests

The impl/memory one is fine, but the txnBuf one is currently failing.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1437553002

Fix another race.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1430193002

Fix race in impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/16

Review URL: https://codereview.chromium.org/1411573014

Allow metadata fields to be PropertyConverters for symmetry.

This makes the easiest forms of parsed kinds/ids extremely simple to implement,
as well as composable. The MetaGetterSetter interface requires every top-level
struct to implement it, since allowing embedded implementations will get the
wrong value for the derived "$kind".

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1414043006

service/datastore: Call MGS GetAllMeta() in Save()

Fix a bug where a structPLS that uses Save(withMeta: true) will ignore
its object's MetaGetterSetter overrides and use struct-default.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1425283002

Decouple PLS from MGS.

Decouple PropertyLoadSaver from MetaGetterSetter. This allows
implementing structs to override either one of those interfaces without
depending on spooky the interaction between them implemented by
"structPLS".

"structPLS" no longer magic-detects if its type implements
MetaGetterSetter and defers to it. Instead, a struct that implements
MetaGetterSetter and wants to use "structPLS" cool meta defaults
can call GetPLS(this) to get a handle on that base functionality.

BUG=

Review URL: https://codereview.chromium.org/1427933002

Use the sort.Search api correctly.

I was using it again and I realized that I used it wrong here :(. Includes
regression test.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1409613002

Add ability to override just the metadata in a struct.

This is useful when the '$id' is actually multiple fields packed into a string,
but it's generally available for any metadata field.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/15

Review URL: https://codereview.chromium.org/1401533007

Add Exists and ExistsMulti convenience methods

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1398103003

Add VersionID to impl/memory info service implementation

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1403873002

Add Gob encoding functions for Key.

This is so that *Key can transparently be encoded in tumble mutation structs.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533020

Review URL: https://codereview.chromium.org/1397673003

Add transaction buffer filter.

This will be used for running multiple non-cooperative idempotent tasks inside
of tumble, which require seeing the previous tasks' effects on the entity group.

However, the effect of having transactions transparently buffer datastore
interactions until the final transaction commit is a generally useful alteration
of the current behavior.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309803004

Fix bugs in impl/memory queries.

This prevents built-in indexes from being selected when using ancestor and
non-__key__ inequality filters, and also fixes a bug where a multi-iterator on
builtin indexes using __key__ inequality with ancestor would return empty result
sets.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1380613002

Add a NewDatastore helper function.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379673002

Make impl/memory have an AppID.

This associates an immutable AppID with the datastoreData object (e.g. the thing
that actually holds the data). It gets rid of the hard-coded globalAppID hack.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1369353003

Improve the way "consistent" works.

This avoids the overhead of taking many many snapshots when the in-memory
datastore is in full-consistency mode.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379563004

Fix bugs in service/datastore

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1378613003

Add Key.HasAncestor()

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1377863004

Add the ability to disable special automatic entities in impl/memory

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1372473002

Add AutoIndex to datastore Testable interface.

This allows test writers to set Consistency(true) and AutoIndex(true), and
pretend like the datastore is a fully consistent database which always has
perfect indexes. Will be used for the recursive transaction manager to
maintained tiered automatic mini-datastores in memory.

Also add some more tests for queries in transactions, and YAML rendering for
*IndexDefinition.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021, https://github.com/luci/gae/issues/9

Review URL: https://codereview.chromium.org/1364333002

Add Consistent(bool) to Testing interface

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1366793002

Add missing Count api

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1367003002

Refactor serialization helpers from impl/memory -> service/datastore/serialize

These will be used to aid in joining query results for nested transactions.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1365743002

Add ability to estimate the size of a PropertyMap

This will be used to track in-flight transaction sizes.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1363063002

Fix minor bug in ParseIndexColumn

Also improve error message.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1359963003

Add Kind/StringID/IntID back to Key.

Rename Last -> LastTok.

luci-go change here: https://chromiumcodereview.appspot.com/1359813004

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1358063003

service/datastore: misc improvements in properties.go

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354343002

Add missing AllocateIDs API.

No idea why I missed it. Tweaked a bit to get rid of some redunant info.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1357343002

Add API compat warning :)


Fix memcache.Interface.Get to return a memcache.Item

This intentionally doesn't change GetMulti. Since GetMulti is used for batch
operations, it's actually more convenient to have a pipeline of actions (e.g.
AddMulti -> GetMulti). Otherwise you have to carry around a []Item and a
[]string and keep them both in sync (overhead, error prone).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/3

Review URL: https://codereview.chromium.org/1354333002

Make Get operations only serialize the bare minimum.

Now PropertyLoadSaver has GetAllMeta, which is like Save, except that it ONLY
returns the metadata. This avoids situations where zero-initialized structs
aren't expecting to be serialized for a nominally write-only operation (e.g.
Get).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/11

Review URL: https://codereview.chromium.org/1358743002

Fix zero time bug.

`time.Time{}` needs to be able to round-trip everywhere. Previously
various things were implicitly adding the Local or UTC timezones, even
when the time value was zero.

R=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/10

Review URL: https://codereview.chromium.org/1345043005

Refactor keys and queries in datastore service and implementation.   *
datastore.Query and datastore.FinalizedQuery are now first class objects     in
the service/datastore api.   * datastore.Key is similarly a first class object.
All luci/gae users will     need to interact with `*ds.Key` now. This object
gained all the superpowers     that previously existed in the dskey subpackage
of service/datastore.

The new Query objects are inspectable, and adjustable. Additionally, this
homogenizes they query generation between different implementations. The
normalization which normally occurs on the server-side is now done client-side.
This should ensure consistency between 'prod' and 'memory' implementations.

Additionally, the new FinalizedQuery object gains a GQL() method which should
(hopefully) allow trivial debugging by printing a query. You should be able to
copy and paste the generated GQL* into the appengine dashboard and see what
your request was querying for.

* With the exception of Cursors, which currently have no textual format in the
GQL schema.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533024

Review URL: https://codereview.chromium.org/1355783002

Add simplest service/urlfetch implementation.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354083002

Allow PRESUBMIT run pcg on trybuilder.

R=vadimsh@chromium.org
BUG=529936

Review URL: https://codereview.chromium.org/1345303002

Make SetTransactionRetryCount actually work.

It used to store the state in ephemeral dsImpl object that is minted with each
call to datastore.Get(...). Store it in more durable dataStoreData instead.

R=iannucci@chromium.org
BUG=https://github.com/luci/gae/issues/2

Review URL: https://codereview.chromium.org/1347843002

Add IsDevAppServer implementation to impl/memory.

Returns 'true', since impl/memory is sort of a dev sandbox too.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1347813003

Use int64 to store time.Time internally in memory implementation.

Uint64 causes overflow when storing zero time.Time{} (that due to some Go
stubborn weirdness is represented by negative timestamp).

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334043003

Make PRESUBMIT.py run checks via 'pcg' if it is available in PATH.

R=maruel@chromium.org, iannucci@chromium.org
BUG=chromium:529936

Review URL: https://codereview.chromium.org/1342673002

Add emulation of transaction retries to memory impl of RunInTransaction.

Available via Testable().SetTransactionRetryCount().

BUG=https://github.com/luci/gae/issues/2
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1335083002

Switch to external stringset implementation.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1312433013

Add pcg to ignore impl/prod

This should fix the travis redness.

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334833002 .

Implement projection queries correctly.

* Add Project and ForIndex to Property in service/datastore
* Remove PTBoolFalse/PTBoolTrue silliness (now just PTBool)
* Remove ByteSlice silliness (now []byte behaves exactly like string w.r.t.
  indexing. If you don't want it indexed, put `gae:",noindex"` on it.)
* Make indexes index 'underlying' types. e.g. PTTime is not distinguished from
  PTInt in the index, nor are the various byte sequence types.
* Make impl/prod correctly extract data from internal datastoreIndex objects
  when doing projection queries. This allows doing a projection into
  []PropertyMap. Since all data is bussed internally as PropertyMap (instead of
  crazy PropertyLoadSaver implementations), this is required to make projection
  queries work. I'm pretty confident that this won't break, however, as it's all
  derived from protobufs which are extremely unlikely to change.

  To make this better, we'd have to implement the protobuf protocol directly in
  luci/gae instead of routing through the SDK. Not as absurd as it sounds, but
  definitely out of scope for this CL :).
* Include extremely basic test for impl/prod using aetest. This test takes 12s
  to run on my laptop. *sigh*.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/1

Review URL: https://codereview.chromium.org/1336443003

More minor formatting of README.md


Fix formatting in README.md


impl/memory: Implement Queries

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1302813003

filter/count: Fix alignment on 32-bit systems.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1306373002 .

datastore: Add DecodeCursor to [Raw]Interface.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309933003 .

Update README to use CQ.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1301833005

Update codereview.settings for CQ.

Also add infra-reviers+luci-gae@ to CC list.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1286973004 .

Add Chrome infrastructure CQ config.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1298323004 .

prod: fix timezone conversion from AppEngine layer

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1298423005 .

Fix miscellaneous prod bugs.

- Fix bug where "nil" Parent field was being translated into non-nil
  empty key.
- Use fully qualified AppID for key AppID in checkfilter.
- Use a context-installed singleton to probe AppEngine runtime parameters.
- Bump Task field count to match gae/AppEngine count.
- Fix UTC time check when Locations are the same, but have different
  pointers.
- Pass PropertyLoadSaver value array (rather than source) in
  prod/PutMulti :)
- Use struct name as default even when it implements PropertyLoadSaver.

R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1289323002 .

Fix key encoding to be null terminated

This will allow the use of automatic indicies to fulfil basic ancestor queries.
Otherwise keys with many components sort after keys with few components (instead
of lexicographically per token)

R=dnj@google.com, tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286093007 .

Add single-collection gkvlite iterator.

Multi-collection iterator is up next.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1300493002 .

Split off serialization and key functions to their own packages.

This makes the service/datastore package smaller and more single-purpose.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1292913002 .

impl/memory: Make queries self-validate

This moves validation and normalization logic into the query construction
process itself. This makes it so that queries are either valid and normalized,
or they contain an error (and query building 'sticks' on the first error
encountered).

This replaces the previous
build-a-questionable-query-and-then-try-to-make-it-sensible
logic (which was copied from dev_appserver for the most part).

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286943002 .

Add testable interface for datastore.

Includes the ability to manipulate the 'eventual consistency' state of the
datastore, as well as the ability to add compound indicies.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1285703002 .

Make LazyMultiError constructed via function

Followup to https://codereview.chromium.org/1249933002/

Counterpart to https://codereview.chromium.org/1284323002/

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1291813003 .

Refactor: Rename some files.

R=tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1281173002 .

Make obtaining the Testable interface actually part of the interface.

Reduces the number of free casts, and will actually work with the upcoming
Testable interface for datastore, which uses a checkfilter (and so casting
won't work there anyway).

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1282673002 .

Get rid of awkward proto argument to Interface.Run

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1279703003 .

A transparent cache for datastore, backed by memcache.

Check doc.go for details, but:
  * allows cache disablement on a per-model/Key basis
  * allows cache expiration on a per-model basis
  * efficient (non-gob) encoding scheme
  * gzip for large entities
  * allows randomized sharding for Gets
    * allows good throughput for hot items
    * shards may be determined on a per-Key basis
  * caches negative hits

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269113005 .

Re-add metadata passthrough on Get operations

It got lost in translation during the RawInterface conversion. This will be used
to e.g. allow SomeStruct to set a default dscache.enable metadatum when doing
e.g. Interface.GetMulti([]SomeStruct{...}).

R=dnj@chromium.org, dnj@google.com, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270113002 .

Use time.Time for internal memcache memory implementation

Followup to https://chromiumcodereview.appspot.com/1270063003/

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1262173003 .

Make the rest of the services have a similar raw/user interface structure.

Also implement missing functionality from taskqueue/memcache in the memory
implementation.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063003 .

Add 'user friendly' datastore API.

This is a thin veneer on top of datastore.RawInterface, and may be used
with any filters, any backend, and it may be used interchangably and
inderleved with datastore.RawInterface.

In particular, this adds a convention for embedding keys inside of
loadable/savable objects.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259593005 .

Rename rawdatastore -> datastore

Also rename some of the .go files in the service to prep for
https://codereview.chromium.org/1259593005/

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063002 .

Fix obviously broken error handling.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269593002 .

Make rawdatastore API safer for writing filters.

This removes Get/Put/Delete, since they always end up being implemented
as wrappers around the Multi versions anyway, it just caused unnecessary noise.
They'll come back with the 'user friendly' datastore interface, but there's no
reason to have them at every layer of the cake.

It moves the *Multi methods to be callback-based, which makes interacting with
them easier (since you don't have to do lots of slice allocations).

It introduces 'checkFilter' to rawdatastore which guarantees that filters and
backend implementations don't need to do lots of redundant checks to sanitize
their inputs.

It adds a new API to the Info service to retrieve the current namespace.

R=dnj@google.com, estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1253263002 .

Add Toggle for rawdatastore.

This will be used to control cacheability for entities, and allows setting a
default value in the tag without ambiguity (since the 'zero-value' for bools is
false, a simple bool wouldn't provide a way to override defaulted-true field).

R=dnj@google.com, vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259463002 .

Update README.md formatting


Add PRESUBMIT.py


Minor doc improvements

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1241423003 .

Add LICENSE


Revert "Enable coveralls.io"

This reverts commit fc93f78cf1d04d25fb3f52dd8040dba88649494e.

Enable coveralls.io


Remove most error code from luci/gae

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1249863004 .

Improve docs

Simplify godoc by making some functions methods
Rename 'filters' to 'filter' to be more consistent with 'impl' and 'service'.

R=estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1247283003 .

Refactor a bit.

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1243323002 .

Switch to other coveralls badge


Force timezone to be non-UTC when testing non-UTC


Fix gae.DSTransactionOptions


Minor change for Travis :)


Add README.md


Add .travis.yml


Add codereview.settings


Fix imports and tests

Run goimports over everything for good measure as well.

Fix DSKey encoding.

Specifically, IntID's always sort before StringID's. This is according to
https://cloud.google.com/appengine/docs/go/datastore/queries#Go_Key_filters

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1231863018

Simplify memlock and make it less racy.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1233413002

Rename poorly named files.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240123003

Improve memory implementation of gae to provide the full RawDatastore interface.

Still no query support yet, working on that next, but this enables *Multi apis.
Also do a few other code cleanups.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1242043005

Add LazyMultiError.

This is essentially a MultiError, except that it knows how big it should get
if it's ever assigned an actual error, and it only does its allocation lazily.
This is useful for implementing e.g. PutMulti and other Multi-style methods.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234493005

Add NewDSKeyFromEncoded convenience function.

Also placate golint :).

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1238573004

Convert 'Testable' interface into an optional filter.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227203004

Add "filters/count" filter.

This filter allows you to track the number of times a give API has been
invoked, as well as the number of errors that the API has returned. Will mostly
be used for testing.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1239103003

Add filters for gae library.

Filters allow users to add their own wrappers around the underlying gae
implementation. They can do anything that an implementation could do, except
that their backend is another gae library interface, instead of the SDK or
memory implementations. Filters are also necessary (instead of just stacking
implementations with SetXXFactory functions), due to transactions, which change
the context object.

They will be used to implement (at least):
  * caching (hitting memcache before datastore)
  * transaction buffering (giving a self-consistent view of the datastore
    within a transaction, buffering all writes to the end of the transaction
    to make transaction sizes smaller, etc.)
  * featureBreaker (for tests, breaking certain APIs with certain errors)
  * counter (for tests, counting the number of times an API was hit)

It may also be used for:
  * ACL enforcement (rejecting access to datastore items the user can't get,
    filtering queries, etc.)

Thi CL also implements a counter filter as a demo (which will also be used for
testing other filter implementations to ensure that the filter being tested).

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222983002

Some miscellaneous fixes

  * Fix DSKeyIncomplete to check for nil
  * Fix docstring in memory/context.go
  * Make GlobalInfo support BrokenFeatures
  * Fix memory TaskQueue to fill in the default queue name, per SDK spec.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234343006

Simplify taskqueue interface.

The removed complexity was a holdover from the initial versions of this library.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1235563007

Change RawDatastore to do less reflection.

Followup CL will provide the 'magic' reflection-full Datastore interface.

This change makes filters easier to implement, none of the layers of the stack
are required to do reflection, instead of the previous design where all of the
layers were forced to do reflection.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227183003

Move dummy service implementations into their own package.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1226063003

Reland: Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

Original broke because endpoints needed to be moved to a version which relies
on the new SDK.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240573002

Revert "Refactor current GAE abstraction library to be free of the SDK*"

This reverts commit a91c4ac2149f8bbe3711c4c662b6d500c9a8f799.

CQ didn't validate that everything builds. Consequently, note everything builds
:D.

TBR=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1230303003

Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222903002

Use SetFinalizer to avoid leaking Snapshots.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1223263006

Roll infra/go/src/github.com/luci/luci-go/ to
b85c6dd748361e8244122faac44b5beefcd2db37.

$ git log cd33aabb0..b85c6dd74 --date=short --format='%ad %ae %s'
2015-07-07 robbie Rename funnybase and add more serializations.
2015-07-07 yyanagisawa Reference implementation of LinkToPrimaryHandler.

R=pgervais@chromium.org, agable@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1229503007 .

A client/server helper wrapper for endpoints in Go.

The epservice directory/main program generates a appengine app which links all
the available endpoints implementations by finding packages which implement
RegisterEndpointsService. It then runs this with dev_appserver.py.

Subpackages of epservice are supposed to be one-folder-per-service.

The epclient directory/main program/go-generate invocation runs the epservice
server, and then runs the golang client library generator to generate go client
libraries for all the services it finds.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153473008

Non racy tests for memlock.

Bonus, no more time.Sleep, and the tests are 100% faster (e.g. they take 0.04s
instead of 0.08s :D )

R=dnj@chromium.org, maruel@chromium.org, vadimsh@chromium.org
BUG=496436

Review URL: https://codereview.chromium.org/1159173004

Add context-aware "time" library wrapper.

TEST=unittest
R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1154213012

A simple memcache lock for appengine.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/986553002

Last fix for 32 bit tests?

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159623005

Fix tests to run on 32 bits.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159343002

Fix overflow on 32bit which causes tests to fail on 32bit.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1162813002

Add initial Query generation, correctness checking and index generation.

The ability to execute a query will come in a following CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1160253002

Simplify memory package interface.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1143053004

Simple memory testing for gae/wrapper

Provides partial implementations of Memcache, TaskQueue and Datastore.

Most missing things I just didn't care to implement... Datastore queries are
conspicuously absent, and I'll be doing those in a separate CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1152383003

Better attempt at an appengine wrapper.

Includes implementation using old-style appengine, and will be able to be
expanded to include support for new-style appengine SDK. I also have a partial
in-memory testing implementation in the wings which will be a followup CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1151473003

go: infra/libs/* now live in luci-go.

Companion commit that moved them there:
https://github.com/luci/luci-go/commit/15984f36e8e3a78d9893369993ea3b999e919d26

R=iannucci@chromium.org,maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153883002

cipd: Skip test on Windows for now.

The code compiles, but unit tests don't work because Windows handles symlinks
and other file system stuff differently.

R=iannucci@chromium.org
BUG=487483

Review URL: https://codereview.chromium.org/1133353002

Fix meta lib to be correctly compilable.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/998153002

Thin library for interacting with the GAE datastore metadata entities.

Only implements the one function I need right now, but can be extended for
others as needed.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/985583003

Simple GAE/goon Context wrapper

This is useful for restricting an API to implement 'datastore without queries'
or 'key-only context', etc.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/981003003
==========

to

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)
  taskqueue.Add(datastore.WithTransaction(c, nil), task)
  prod.AEContext(datastore.WithTransaction(c, nil))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

Implement info.ServiceAccount() on devserver via info.AccessToken(...).

R=iannucci@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2220193002

Update cloud package.

Fixes #54.

BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2217063003

Remove empty namespace assertion from impl/memory.

This was fixed upstream in:
 
https://github.com/golang/appengine/commit/5a9e2b406e4e87cd8a88d1881a31b1248d...

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2219673002

GAE CL for luci-go common refactor

R=nodir@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/2192003003 .

Add an option to install just Cloud Datastore.

TBR=iannucci@chromium.org
BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2179843004

cq.cfg: use luci builders

do not use buildbot builder
update luci bucket name

R=estaab@chromium.org, dnj@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2179543003

Add lightning talk that I didn't gave at Gophercon '16


Add cloud datastore implementation.

Fixes #52.

BUG=

Review-Url: https://codereview.chromium.org/1957953002

datastore: Update AllocateIDs to take keys.

Update the Datastore's AllocateIDs call to accept variadic arguments similar
to Get/Put/Delete/Exists. Structs can be supplied to populate their keys
directly, and keys or key slices can be supplied to directly allocate keys.

This adds some Key helper functions:
- SameKind, which asserts that keys reference the same kind.
- Partial, which returns a partial key given a source key.
- WithID, which returns a derivative key with the specified ID.
- NewIncompleteKeys, which allocates a slice of incomplete keys given a kind.

BUG=

Review-Url: https://codereview.chromium.org/2007123002

Refactor multiarg, split MGS/PLS.

Refactor the multiarg construction method to handle the various cases
inline, rather than defining and invoking methods for specific
scenarios. This enables PropertyLoadSaver-based methods to be handled
separately from MetaGetterSetter-based methods, which, in turn, enables
the two to be implemented independently.

A bit of boilerplate is removed and the odd case of MGS/PLS Map/Chan fits
better. Additionally, a few double-reflection code paths are optimized
along the way.

BUG=None

Review-Url: https://codereview.chromium.org/2048933004

make swarming trybuilders non-experimental

R=iannucci@chromium.org, dnj@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2083193006

cq.cfg: enable experimental swarming builders

R=estaab@chromium.org, vadimsh@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2090433002

Add datastore dumper tool.

This allows you to dump the datastore in whole or part, with or without special
formatting directives.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2037433002

Add TrimmedAppID to info service.

Also refactor info service to share common 'augmented' methods in
the service/info package itself, ala service/datastore.

Split out from CL: https://chromiumcodereview.appspot.com/1537883002

R=dnj@chromium.org
BUG=550684

Review-Url: https://codereview.chromium.org/2062613002

Add DefaultVersionHostname to impl/memory/info.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2028423002

Fix typo in pls.go docs


Allow uint16 and uint32 in MakeKey.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036543003

Add GetNoTxn to txnBuf as a hack.

I really don't like this but short of fixing #23, I don't think there's much we
can do.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036593002

Change gae to apache 2 license.

Update AUTHORS, CONTRIBUTORS, and LICENSE.
Add apache 2 license check to PRESUBMIT.py.

Ran:
  git sed 's/Chromium Authors/LUCI Authors/'
  git sed 's/governed by a BSD-style license that can be/governed under the
Apache License, Version 2.0/'
  git sed 's/found in the LICENSE file/that can be found in the LICENSE file/'
  git checkout HEAD -- $(git status -sb | grep '^ T' | cut -c 4-)
  git checkout origin/master -- $(git diff origin/master --name-only | grep
third_party)

BUG=

Review-Url: https://codereview.chromium.org/2021753002

Change codereview server to be consistent with other luci projects.

BUG=

Review-Url: https://codereview.chromium.org/2020933002

dscache: Reduce log level of memcache failures.

Warning-level failures quickly pollute the AppEngine log space. This
change converts some of the more common warnings to Debug-level logs so
the Warning space remains reserved for higher attention calls.

Also add a warning about lock eviction poisoning to THERE BE DRAGONS
section of the doc.

BUG=

Review-Url: https://codereview.chromium.org/1987353002

datastore: variadic Get, Put, Exists, Delete.

Fixes #51.

Update Get, Put, Exists, and Delete to be variadic. This will make the
interface more usable and get rid of boilerplate in client code.

Get, Put, Exists, and Delete now accept variadic arguments. Any one of
these arguments can be an argument that was previously valid or
the *Multi version of those functions. This effectively obsoletes the *Multi
versions, but they are left in for backwards compatiblility with the stated
intention of removing them at a later date.

Get/Put/Exists/Delete will now return a single error in the one-argument
case or a MultiError in a multiple-argument case. Note that now that single
argument case can take a slice type, the single error may itself be a
MultiError if the single argument is a slice.

Review-Url: https://codereview.chromium.org/2011773002

filter/txnBuf: Use fully-qualified App ID.

Use the fully-qualified app ID when creating the in-memory datastore for
transaction buffer. Previously, transaction buffer would use the
standard App ID, while production datastore would use the
fully-qualified App ID, causing key incompatibility errors in
production.

Cleanup impl/memory's NewDatastore function to not install superfluous
services into the Context. This draws a cleaner line between
impl/memory's dual purposes as both a rich testing enviornment and a
production-grade in-memory datastore.

In addition, have impl/memory's info service differentiate between the
two. This causes tests for filter/txnBuf to properly fail lacking this
fix, enabling regression protection.

To support this, impl/memory's Info service will now parse the supplied
App ID. If it contains a "~" character, it is split into fully-qualified
and regular AppID. Otherwise, the two will be the same.

BUG=

Review-Url: https://codereview.chromium.org/1929423002

Travis: Increase system specs, limit concurrent.

Increase the Travis system specs and limit the number of concurrent
tests to try and avoid the signal-kill error that we're seeing in Travis
ATM.

TBR=iannucci@chromium.org
BUG=None

Review URL: https://codereview.chromium.org/1922583002

pcg: Bump test time to 20 minutes.

Travis is currently timing out.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1918713003

Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1917513002

impl/memory: datastore and memcache namespacing

Update GetNamespace() to return an additional boolean, which will be
true if the namespace was set and false if it was not. The AppEngine SDK
defaults to having no namespace installed ("", false), but will allow
the user to install any namespace. This allows a user to differentiate
between no namespace and an empty namespace ("", true).

Make impl/memory's datastore query operation to error if the namespace has
been set to the empty namespace (""). This reflects observed production
behavior.

BUG=

Review URL: https://codereview.chromium.org/1916463004

Bump max_duration for CI tests to 480s

Hopefully Fixes #48.
Fix memory corruption bug in impl/memory

When you `SetCollection` in gkvlite, it invalidates the old
collection, but keeps the Items of the previous collection. The way
that impl/memory was using SetCollection was more along the lines of
GetOrCreate, which could cause corruption or other issues if two
codepaths did SetCollection simultaneously.

This fixes the corruption bug by removing SetCollection from the
memStore interface and replaces it with a GetOrCreateCollection,
which checks for the collection, and only calls SetCollection if it
doesn't exist. As a bonus, this seems to give a slight performance
speedup over the previous broken code.

Additionally, this forces all uses of VisitItemsAscend to use
a Snapshot, to avoid accidental modification of a Collection while
its being iterated over, which can lead to the situation discussed
above, or other nasty errors.

Finally, this adds an optional 'gkvlite tracer' functionality
which came in (extremely) handy for finding this bug. When given
a folder, it will emit go-code-like invocations of the functions run
on every collection, and includes enough information to reveal which
functions are being called on the root collection v. snapshots, and
if there are any overlapping VisitItemsAscend invocations, etc.

R=dnj@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1911263002

Travis: Build against Go1.6.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1911293002

Remove race from query finalization.

BUG=

Review URL: https://codereview.chromium.org/1910293002

datastore: Fix AddIndexes with existing namespaces

AddIndexes previously only added an index for the current namespace.
Since indexes are global, they should be added for all existing
namespaces.

BUG=

Review URL: https://codereview.chromium.org/1894403002

service/taskqueue: Add NewPOSTTask, remove NewTask

Add NewPOSTTask, which mirrors the method of the same name in
AppEngine's Go task queue SDK.

Remove NewTask from the taskqueue Interface, since it was basically
useless.

BUG=

Review URL: https://codereview.chromium.org/1890983004

Add __namespace__ metadata query support to impl/memory.

Ref: https://cloud.google.com/datastore/docs/concepts/metadataqueries

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1892343002

Add Info Testable interface, implement for memory.

This adds a Testable interface to the Info service which allows it to
install Info parameters in Contexts.

Add support for setting version ID and request ID to impl/memory Info
service.

BUG=

Review URL: https://codereview.chromium.org/1871943003

Add PopulateKey method to datastore.

Add PopulateKey, a method that, given an object, fills in its key
fields.

Fixes #44.

BUG=

Review URL: https://codereview.chromium.org/1867963003

Fix missing/broken ds.Stop handling in raw DS.

Several places were missed when patching "ds.Stop" to being correctly filtred
at the top interface rather than the raw interface.

At the top-level, GetAll and Count both did not filter the Stop error.
At the filter level, several filters were ignoring error values from
their callbacks.

Also fixes an issue in count filter where the ds.Stop error was being
incorrectly filtered at the RawInterface level.

Add datastore iterating query filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1846123002

Add datastore iterating query filter.

BUG=

Review URL: https://codereview.chromium.org/1843313004

Don't filter "Stop" error in raw interface.

The Stop error is a high-level error used by the datastore service to
indicate that an iterative operation should stop rather. It should
appear to the user as a "nil" error.

However, currently the datastore raw implementations are filtering Stop
and converting it to "nil". This prevents other filters from
identifying Stop events from their lower level callbacks.

Update raw datastore methods to not filter Stop, and update their tests
to accept Stop as a valid successful result.

BUG=

Review URL: https://codereview.chromium.org/1844183003

Make ExistsMulti return a BoolList

BoolList has some convenience functions (e.g. to check if Any() or All() are
set.). It's compatible with current uses of ExistsMulti.

R=dnj@chromium.org, estaab@chromium.org, nodir@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1816413002

txnbuf: Add write count budget, size budget.

Add a "maximum number of datastore writes" budget to the transaction
buffer. Rework the size budget to remove preemptive child transaction
failures and directly derive budget from the parent.

BUG=

Review URL: https://codereview.chromium.org/1776833002

Add wrappers for the module module

BUG=570879

Review URL: https://codereview.chromium.org/1772943003

Add a MustNamespace that panics if the namespace wasn't valid

BUG=

Review URL: https://codereview.chromium.org/1766593002

Auto-project on distinct inequality fields.

BUG=

Review URL: https://codereview.chromium.org/1745703002

Add header text to proto-gae tool

In order to appease our presubmit scripts, default to the Chromium Authors
copyright header.

R=dnj@chromium.org, estaab@chromium.org, hinoka@chromium.org,
martiniss@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1718123003

dscache: Only log unusual memcache errors.

BUG=None
TEST=None
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1695833002

Use ShouldResemble instead of ShouldResembleV.

Now that goconvey incorporates our ShouldResembleV method directly, we
can go back to using ShouldResemble instead of hacking in
ShouldResembleV!

BUG=None

Review URL: https://codereview.chromium.org/1679093002

Add support for uint8, uint16 and uint32 in luci/gae

This has come up for like 3 models in DM so far, and I'm tired of having this
be a stupid "gotcha!". I was working around it with some PropertyConverter, but
with this change it will just be obvious what the heck is going on.

They all map to positive int64 values (which should make indexes obvious), and
they include out-of-bounds checks on deserialization.

I excluded uint64, since there's no way to properly encode it in the native
datastore representation which still makes any sense for indexes.

I also added some (apparently?) missing documentation in PLS which explicitly
lays out what field types are allowed in a pls-managed struct.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1667403003

Fixes #8: Seed indexes from index.yml

- Add `ParseIndexYAML` and `FindAndParseIndexYAML` functions
  to service/datastore
- Add YAML marshalling and unmarshalling for IndexDefinition and IndexColumn
- Add tests for the changes
- Imports: additional package: gopkg.in/yaml.v2

https://github.com/luci/gae/issues/8

Closes #8

R=dnj@chromium.org, estaab@chromium.org, iannucci@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1574353004

impl/prod: Expose Remote API scopes.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1632943003

Add UseBackground.

Fixes #39.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1630833002

Add tool to generate PropertyConverter implementations for proto messages.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1625773002

Fix logger for filter api change in luci-go.

API change will occur: https://chromiumcodereview.appspot.com/1622553005/

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1621993003

Fix fast string comparison bug.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1620873004

Add a test to ensure []byte => BlobKey PLS works.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1616033004

Encode nil datastore keys as PTNull.

This fixes a panic in serialize when it is asked to serialize a nil
*ds.Key, causing it to dereference its fields. Instead, a *ds.Key is
serialized as a PTNull entry like datastore.

Add some tests to assert proper conversion and serialization.

BUG=

Review URL: https://codereview.chromium.org/1576153003

Add tests for binary_tools functions

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1576353002

Add dscache.AlwaysFilterRDS function.

It's like dscache.FilterRDS, except it doesn't use IsGloballyEnabled. Intended
to be used by callers that know better when to apply (or not) the filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1579853002

Fix GAE <-> GAE RemoteAPI

Urlfetch.Get needs the user context, not the aeCtx.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1578943002

Make UseRemote work for dev_appserver too.

Now interacting with remote_api is as simple as:

    ```go
    ctx := context.Background()
    if err := prod.UseRemote(&ctx, "localhost:8080", nil); err != nil {
      panic(err)
    }
    // ctx is usable with luci/gae
    ```

Also changes the context-derivation function signature to be safer, since the
client's context will always be valid, and can be used for things, such as
logging even in the error handling case.

This also makes UseRemote work when used FROM appengine to another appengine
app.

Fixes #36.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1557223002

impl/memory: Fix time serialization encoding.

In a PropertyMap, time is serialized as an integer type. However, for
queries, time is serialized as a time type. This causes all queries on
time fields to select any encoded time properties, since PTTime > PTInt.

This patch causes queries to serialize PTTime properties as PTInts to
match the indexes.

Fixes #34.

Review URL: https://codereview.chromium.org/1550903002

Replace spammy log message in dscache with comment.

Fixes luci/gae#35.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/35

Review URL: https://codereview.chromium.org/1556193002

Add remote_api support

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/33

Review URL: https://codereview.chromium.org/1549223002

impl/memory: Add "PULL" as valid taskqueue method.

BUG=

Review URL: https://codereview.chromium.org/1550863002

Use prod logger's user context to read logging level

TBR=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/32

Review URL: https://codereview.chromium.org/1542263003

Make setting a Memcache Item with a nil Value OK

This conforms to the expectations that if you Set with a nil Value, it's the
same as setting to []byte{}... One less trap :)

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/31

Review URL: https://codereview.chromium.org/1541183002

Fix doc bug in pls.go


Implement Mail service.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/20

Review URL: https://codereview.chromium.org/1525923002

impl/prod: Fix logging field output.

By using the AppEngine context, logging calls would not have access to
logging fields that were set in subsequent contexts (most of them).
Retain both the AppEngine context (for logging calls) and the logger
context (for up-to-date logging configuration), and use the latter for
field extraction.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1523343002

Add filters for user service.

This brings the user service up to parity with the rest.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1530533002

Panic when non-runtime-handleable type errors occur

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/28

Review URL: https://codereview.chromium.org/1525873002

Fix nil-deref panic from Put.

We were using key before checking the error.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/26

Review URL: https://codereview.chromium.org/1523943002

Return error on Get(*Key).

Before it would end up returning ErrInvalidKey, which is silly :).

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org,
martiniss@chromium.org
BUG=https://github.com/luci/gae/issues/27

Review URL: https://codereview.chromium.org/1530503003

Clean up callback interfaces.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/17

Review URL: https://codereview.chromium.org/1521823003

Add User Testing interface to impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/24

Review URL: https://codereview.chromium.org/1522783002

Add logging service documentation.

Just points to github.com/luci/luci-go/common/logging, but explains how that
works and meshes with the rest of the service definitions.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1526453002

Handle unexpected entity fields gracefully.

Introduces a new field type 'extra', which allows full control over what to do
with the extra properties, instead of having all operations return unhelpful
conversion error messages.

This allows black-holes, schema upgrades and 'expando' style mixed
structured/unstructured objects. Please see the tests for usage and examples.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/13

Review URL: https://codereview.chromium.org/1525453002

Clean up PropertyLoadSaver interface.

This removes GetMetaDefault and Problem from the interfaces, and makes GetPLS
errors into panics, on the assumption that if you pass it a bad datatype, you
can't really do anything about it without recompiling, and failing fast is
better.

This also fixes a bug where doing any operations on an invalid (non datastore-
compatible) struct would return the unhelpful "unable to extract $kind" error.

R=dnj@chromium.org, hinoka@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1516173002

Transfer context deadline to AEContext.

It's needed to be able to tweak default RPC timeout (5 sec), in particularly
when using urlfetch service.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1515413002

Move gaelogging from luci/luci-go to luci/gae

Also makes impl/memory Use memlogger too for consistency. This change is
required to make luci-go logging work on appengine due to
3a50526e08194bd66487c7c03cccd2108c236468.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1498963003

Add API to allow you to get the non-transactional datastore or taskqueue.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1494223002

GAE Datastore - Fix derived meta types not being reflected properly

Stuff like "type FooBar string" were not allowed as metadata fields correctly.

BUG=

Review URL: https://codereview.chromium.org/1494803005

Fix User API (CurrentOAuth)

TBR=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1493183003

Add User service.

Doesn't add a memory implementation... it's easy enough to mock directly in the
test and doesn't have any complex systemic behaviors (that I know of) that are
necessary for a single mock implementation.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1500433002

Make appengine connection explicitly owned by impl/prod.

R=vadimsh@chromium.org, hinoka@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/1490243004 .

Update README.md

Typo :)
Fix races in txnBuf

Runs clean with `-cpu=2,4 -race` now. It's also a lot clearer to follow the
logic now. No more 'siblingLock'. At the time that each recursive transaction
starts it locks (in the normal way!). It then keeps a reference to the parent
layer with the locks disabled for its internal usage.

This refactor makes it so that all (interacting) operations at a given 'layer'
in a transaction are mutually exclusive. Previously only nested transactions
would be mutually exclusive.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1434873003

Add some race-detector tests

The impl/memory one is fine, but the txnBuf one is currently failing.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1437553002

Fix another race.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1430193002

Fix race in impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/16

Review URL: https://codereview.chromium.org/1411573014

Allow metadata fields to be PropertyConverters for symmetry.

This makes the easiest forms of parsed kinds/ids extremely simple to implement,
as well as composable. The MetaGetterSetter interface requires every top-level
struct to implement it, since allowing embedded implementations will get the
wrong value for the derived "$kind".

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1414043006

service/datastore: Call MGS GetAllMeta() in Save()

Fix a bug where a structPLS that uses Save(withMeta: true) will ignore
its object's MetaGetterSetter overrides and use struct-default.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1425283002

Decouple PLS from MGS.

Decouple PropertyLoadSaver from MetaGetterSetter. This allows
implementing structs to override either one of those interfaces without
depending on spooky the interaction between them implemented by
"structPLS".

"structPLS" no longer magic-detects if its type implements
MetaGetterSetter and defers to it. Instead, a struct that implements
MetaGetterSetter and wants to use "structPLS" cool meta defaults
can call GetPLS(this) to get a handle on that base functionality.

BUG=

Review URL: https://codereview.chromium.org/1427933002

Use the sort.Search api correctly.

I was using it again and I realized that I used it wrong here :(. Includes
regression test.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1409613002

Add ability to override just the metadata in a struct.

This is useful when the '$id' is actually multiple fields packed into a string,
but it's generally available for any metadata field.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/15

Review URL: https://codereview.chromium.org/1401533007

Add Exists and ExistsMulti convenience methods

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1398103003

Add VersionID to impl/memory info service implementation

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1403873002

Add Gob encoding functions for Key.

This is so that *Key can transparently be encoded in tumble mutation structs.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533020

Review URL: https://codereview.chromium.org/1397673003

Add transaction buffer filter.

This will be used for running multiple non-cooperative idempotent tasks inside
of tumble, which require seeing the previous tasks' effects on the entity group.

However, the effect of having transactions transparently buffer datastore
interactions until the final transaction commit is a generally useful alteration
of the current behavior.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309803004

Fix bugs in impl/memory queries.

This prevents built-in indexes from being selected when using ancestor and
non-__key__ inequality filters, and also fixes a bug where a multi-iterator on
builtin indexes using __key__ inequality with ancestor would return empty result
sets.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1380613002

Add a NewDatastore helper function.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379673002

Make impl/memory have an AppID.

This associates an immutable AppID with the datastoreData object (e.g. the thing
that actually holds the data). It gets rid of the hard-coded globalAppID hack.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1369353003

Improve the way "consistent" works.

This avoids the overhead of taking many many snapshots when the in-memory
datastore is in full-consistency mode.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379563004

Fix bugs in service/datastore

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1378613003

Add Key.HasAncestor()

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1377863004

Add the ability to disable special automatic entities in impl/memory

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1372473002

Add AutoIndex to datastore Testable interface.

This allows test writers to set Consistency(true) and AutoIndex(true), and
pretend like the datastore is a fully consistent database which always has
perfect indexes. Will be used for the recursive transaction manager to
maintained tiered automatic mini-datastores in memory.

Also add some more tests for queries in transactions, and YAML rendering for
*IndexDefinition.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021, https://github.com/luci/gae/issues/9

Review URL: https://codereview.chromium.org/1364333002

Add Consistent(bool) to Testing interface

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1366793002

Add missing Count api

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1367003002

Refactor serialization helpers from impl/memory -> service/datastore/serialize

These will be used to aid in joining query results for nested transactions.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1365743002

Add ability to estimate the size of a PropertyMap

This will be used to track in-flight transaction sizes.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1363063002

Fix minor bug in ParseIndexColumn

Also improve error message.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1359963003

Add Kind/StringID/IntID back to Key.

Rename Last -> LastTok.

luci-go change here: https://chromiumcodereview.appspot.com/1359813004

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1358063003

service/datastore: misc improvements in properties.go

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354343002

Add missing AllocateIDs API.

No idea why I missed it. Tweaked a bit to get rid of some redunant info.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1357343002

Add API compat warning :)


Fix memcache.Interface.Get to return a memcache.Item

This intentionally doesn't change GetMulti. Since GetMulti is used for batch
operations, it's actually more convenient to have a pipeline of actions (e.g.
AddMulti -> GetMulti). Otherwise you have to carry around a []Item and a
[]string and keep them both in sync (overhead, error prone).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/3

Review URL: https://codereview.chromium.org/1354333002

Make Get operations only serialize the bare minimum.

Now PropertyLoadSaver has GetAllMeta, which is like Save, except that it ONLY
returns the metadata. This avoids situations where zero-initialized structs
aren't expecting to be serialized for a nominally write-only operation (e.g.
Get).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/11

Review URL: https://codereview.chromium.org/1358743002

Fix zero time bug.

`time.Time{}` needs to be able to round-trip everywhere. Previously
various things were implicitly adding the Local or UTC timezones, even
when the time value was zero.

R=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/10

Review URL: https://codereview.chromium.org/1345043005

Refactor keys and queries in datastore service and implementation.   *
datastore.Query and datastore.FinalizedQuery are now first class objects     in
the service/datastore api.   * datastore.Key is similarly a first class object.
All luci/gae users will     need to interact with `*ds.Key` now. This object
gained all the superpowers     that previously existed in the dskey subpackage
of service/datastore.

The new Query objects are inspectable, and adjustable. Additionally, this
homogenizes they query generation between different implementations. The
normalization which normally occurs on the server-side is now done client-side.
This should ensure consistency between 'prod' and 'memory' implementations.

Additionally, the new FinalizedQuery object gains a GQL() method which should
(hopefully) allow trivial debugging by printing a query. You should be able to
copy and paste the generated GQL* into the appengine dashboard and see what
your request was querying for.

* With the exception of Cursors, which currently have no textual format in the
GQL schema.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533024

Review URL: https://codereview.chromium.org/1355783002

Add simplest service/urlfetch implementation.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354083002

Allow PRESUBMIT run pcg on trybuilder.

R=vadimsh@chromium.org
BUG=529936

Review URL: https://codereview.chromium.org/1345303002

Make SetTransactionRetryCount actually work.

It used to store the state in ephemeral dsImpl object that is minted with each
call to datastore.Get(...). Store it in more durable dataStoreData instead.

R=iannucci@chromium.org
BUG=https://github.com/luci/gae/issues/2

Review URL: https://codereview.chromium.org/1347843002

Add IsDevAppServer implementation to impl/memory.

Returns 'true', since impl/memory is sort of a dev sandbox too.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1347813003

Use int64 to store time.Time internally in memory implementation.

Uint64 causes overflow when storing zero time.Time{} (that due to some Go
stubborn weirdness is represented by negative timestamp).

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334043003

Make PRESUBMIT.py run checks via 'pcg' if it is available in PATH.

R=maruel@chromium.org, iannucci@chromium.org
BUG=chromium:529936

Review URL: https://codereview.chromium.org/1342673002

Add emulation of transaction retries to memory impl of RunInTransaction.

Available via Testable().SetTransactionRetryCount().

BUG=https://github.com/luci/gae/issues/2
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1335083002

Switch to external stringset implementation.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1312433013

Add pcg to ignore impl/prod

This should fix the travis redness.

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334833002 .

Implement projection queries correctly.

* Add Project and ForIndex to Property in service/datastore
* Remove PTBoolFalse/PTBoolTrue silliness (now just PTBool)
* Remove ByteSlice silliness (now []byte behaves exactly like string w.r.t.
  indexing. If you don't want it indexed, put `gae:",noindex"` on it.)
* Make indexes index 'underlying' types. e.g. PTTime is not distinguished from
  PTInt in the index, nor are the various byte sequence types.
* Make impl/prod correctly extract data from internal datastoreIndex objects
  when doing projection queries. This allows doing a projection into
  []PropertyMap. Since all data is bussed internally as PropertyMap (instead of
  crazy PropertyLoadSaver implementations), this is required to make projection
  queries work. I'm pretty confident that this won't break, however, as it's all
  derived from protobufs which are extremely unlikely to change.

  To make this better, we'd have to implement the protobuf protocol directly in
  luci/gae instead of routing through the SDK. Not as absurd as it sounds, but
  definitely out of scope for this CL :).
* Include extremely basic test for impl/prod using aetest. This test takes 12s
  to run on my laptop. *sigh*.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/1

Review URL: https://codereview.chromium.org/1336443003

More minor formatting of README.md


Fix formatting in README.md


impl/memory: Implement Queries

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1302813003

filter/count: Fix alignment on 32-bit systems.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1306373002 .

datastore: Add DecodeCursor to [Raw]Interface.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309933003 .

Update README to use CQ.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1301833005

Update codereview.settings for CQ.

Also add infra-reviers+luci-gae@ to CC list.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1286973004 .

Add Chrome infrastructure CQ config.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1298323004 .

prod: fix timezone conversion from AppEngine layer

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1298423005 .

Fix miscellaneous prod bugs.

- Fix bug where "nil" Parent field was being translated into non-nil
  empty key.
- Use fully qualified AppID for key AppID in checkfilter.
- Use a context-installed singleton to probe AppEngine runtime parameters.
- Bump Task field count to match gae/AppEngine count.
- Fix UTC time check when Locations are the same, but have different
  pointers.
- Pass PropertyLoadSaver value array (rather than source) in
  prod/PutMulti :)
- Use struct name as default even when it implements PropertyLoadSaver.

R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1289323002 .

Fix key encoding to be null terminated

This will allow the use of automatic indicies to fulfil basic ancestor queries.
Otherwise keys with many components sort after keys with few components (instead
of lexicographically per token)

R=dnj@google.com, tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286093007 .

Add single-collection gkvlite iterator.

Multi-collection iterator is up next.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1300493002 .

Split off serialization and key functions to their own packages.

This makes the service/datastore package smaller and more single-purpose.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1292913002 .

impl/memory: Make queries self-validate

This moves validation and normalization logic into the query construction
process itself. This makes it so that queries are either valid and normalized,
or they contain an error (and query building 'sticks' on the first error
encountered).

This replaces the previous
build-a-questionable-query-and-then-try-to-make-it-sensible
logic (which was copied from dev_appserver for the most part).

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286943002 .

Add testable interface for datastore.

Includes the ability to manipulate the 'eventual consistency' state of the
datastore, as well as the ability to add compound indicies.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1285703002 .

Make LazyMultiError constructed via function

Followup to https://codereview.chromium.org/1249933002/

Counterpart to https://codereview.chromium.org/1284323002/

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1291813003 .

Refactor: Rename some files.

R=tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1281173002 .

Make obtaining the Testable interface actually part of the interface.

Reduces the number of free casts, and will actually work with the upcoming
Testable interface for datastore, which uses a checkfilter (and so casting
won't work there anyway).

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1282673002 .

Get rid of awkward proto argument to Interface.Run

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1279703003 .

A transparent cache for datastore, backed by memcache.

Check doc.go for details, but:
  * allows cache disablement on a per-model/Key basis
  * allows cache expiration on a per-model basis
  * efficient (non-gob) encoding scheme
  * gzip for large entities
  * allows randomized sharding for Gets
    * allows good throughput for hot items
    * shards may be determined on a per-Key basis
  * caches negative hits

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269113005 .

Re-add metadata passthrough on Get operations

It got lost in translation during the RawInterface conversion. This will be used
to e.g. allow SomeStruct to set a default dscache.enable metadatum when doing
e.g. Interface.GetMulti([]SomeStruct{...}).

R=dnj@chromium.org, dnj@google.com, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270113002 .

Use time.Time for internal memcache memory implementation

Followup to https://chromiumcodereview.appspot.com/1270063003/

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1262173003 .

Make the rest of the services have a similar raw/user interface structure.

Also implement missing functionality from taskqueue/memcache in the memory
implementation.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063003 .

Add 'user friendly' datastore API.

This is a thin veneer on top of datastore.RawInterface, and may be used
with any filters, any backend, and it may be used interchangably and
inderleved with datastore.RawInterface.

In particular, this adds a convention for embedding keys inside of
loadable/savable objects.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259593005 .

Rename rawdatastore -> datastore

Also rename some of the .go files in the service to prep for
https://codereview.chromium.org/1259593005/

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063002 .

Fix obviously broken error handling.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269593002 .

Make rawdatastore API safer for writing filters.

This removes Get/Put/Delete, since they always end up being implemented
as wrappers around the Multi versions anyway, it just caused unnecessary noise.
They'll come back with the 'user friendly' datastore interface, but there's no
reason to have them at every layer of the cake.

It moves the *Multi methods to be callback-based, which makes interacting with
them easier (since you don't have to do lots of slice allocations).

It introduces 'checkFilter' to rawdatastore which guarantees that filters and
backend implementations don't need to do lots of redundant checks to sanitize
their inputs.

It adds a new API to the Info service to retrieve the current namespace.

R=dnj@google.com, estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1253263002 .

Add Toggle for rawdatastore.

This will be used to control cacheability for entities, and allows setting a
default value in the tag without ambiguity (since the 'zero-value' for bools is
false, a simple bool wouldn't provide a way to override defaulted-true field).

R=dnj@google.com, vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259463002 .

Update README.md formatting


Add PRESUBMIT.py


Minor doc improvements

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1241423003 .

Add LICENSE


Revert "Enable coveralls.io"

This reverts commit fc93f78cf1d04d25fb3f52dd8040dba88649494e.

Enable coveralls.io


Remove most error code from luci/gae

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1249863004 .

Improve docs

Simplify godoc by making some functions methods
Rename 'filters' to 'filter' to be more consistent with 'impl' and 'service'.

R=estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1247283003 .

Refactor a bit.

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1243323002 .

Switch to other coveralls badge


Force timezone to be non-UTC when testing non-UTC


Fix gae.DSTransactionOptions


Minor change for Travis :)


Add README.md


Add .travis.yml


Add codereview.settings


Fix imports and tests

Run goimports over everything for good measure as well.

Fix DSKey encoding.

Specifically, IntID's always sort before StringID's. This is according to
https://cloud.google.com/appengine/docs/go/datastore/queries#Go_Key_filters

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1231863018

Simplify memlock and make it less racy.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1233413002

Rename poorly named files.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240123003

Improve memory implementation of gae to provide the full RawDatastore interface.

Still no query support yet, working on that next, but this enables *Multi apis.
Also do a few other code cleanups.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1242043005

Add LazyMultiError.

This is essentially a MultiError, except that it knows how big it should get
if it's ever assigned an actual error, and it only does its allocation lazily.
This is useful for implementing e.g. PutMulti and other Multi-style methods.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234493005

Add NewDSKeyFromEncoded convenience function.

Also placate golint :).

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1238573004

Convert 'Testable' interface into an optional filter.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227203004

Add "filters/count" filter.

This filter allows you to track the number of times a give API has been
invoked, as well as the number of errors that the API has returned. Will mostly
be used for testing.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1239103003

Add filters for gae library.

Filters allow users to add their own wrappers around the underlying gae
implementation. They can do anything that an implementation could do, except
that their backend is another gae library interface, instead of the SDK or
memory implementations. Filters are also necessary (instead of just stacking
implementations with SetXXFactory functions), due to transactions, which change
the context object.

They will be used to implement (at least):
  * caching (hitting memcache before datastore)
  * transaction buffering (giving a self-consistent view of the datastore
    within a transaction, buffering all writes to the end of the transaction
    to make transaction sizes smaller, etc.)
  * featureBreaker (for tests, breaking certain APIs with certain errors)
  * counter (for tests, counting the number of times an API was hit)

It may also be used for:
  * ACL enforcement (rejecting access to datastore items the user can't get,
    filtering queries, etc.)

Thi CL also implements a counter filter as a demo (which will also be used for
testing other filter implementations to ensure that the filter being tested).

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222983002

Some miscellaneous fixes

  * Fix DSKeyIncomplete to check for nil
  * Fix docstring in memory/context.go
  * Make GlobalInfo support BrokenFeatures
  * Fix memory TaskQueue to fill in the default queue name, per SDK spec.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234343006

Simplify taskqueue interface.

The removed complexity was a holdover from the initial versions of this library.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1235563007

Change RawDatastore to do less reflection.

Followup CL will provide the 'magic' reflection-full Datastore interface.

This change makes filters easier to implement, none of the layers of the stack
are required to do reflection, instead of the previous design where all of the
layers were forced to do reflection.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227183003

Move dummy service implementations into their own package.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1226063003

Reland: Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

Original broke because endpoints needed to be moved to a version which relies
on the new SDK.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240573002

Revert "Refactor current GAE abstraction library to be free of the SDK*"

This reverts commit a91c4ac2149f8bbe3711c4c662b6d500c9a8f799.

CQ didn't validate that everything builds. Consequently, note everything builds
:D.

TBR=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1230303003

Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222903002

Use SetFinalizer to avoid leaking Snapshots.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1223263006

Roll infra/go/src/github.com/luci/luci-go/ to
b85c6dd748361e8244122faac44b5beefcd2db37.

$ git log cd33aabb0..b85c6dd74 --date=short --format='%ad %ae %s'
2015-07-07 robbie Rename funnybase and add more serializations.
2015-07-07 yyanagisawa Reference implementation of LinkToPrimaryHandler.

R=pgervais@chromium.org, agable@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1229503007 .

A client/server helper wrapper for endpoints in Go.

The epservice directory/main program generates a appengine app which links all
the available endpoints implementations by finding packages which implement
RegisterEndpointsService. It then runs this with dev_appserver.py.

Subpackages of epservice are supposed to be one-folder-per-service.

The epclient directory/main program/go-generate invocation runs the epservice
server, and then runs the golang client library generator to generate go client
libraries for all the services it finds.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153473008

Non racy tests for memlock.

Bonus, no more time.Sleep, and the tests are 100% faster (e.g. they take 0.04s
instead of 0.08s :D )

R=dnj@chromium.org, maruel@chromium.org, vadimsh@chromium.org
BUG=496436

Review URL: https://codereview.chromium.org/1159173004

Add context-aware "time" library wrapper.

TEST=unittest
R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1154213012

A simple memcache lock for appengine.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/986553002

Last fix for 32 bit tests?

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159623005

Fix tests to run on 32 bits.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159343002

Fix overflow on 32bit which causes tests to fail on 32bit.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1162813002

Add initial Query generation, correctness checking and index generation.

The ability to execute a query will come in a following CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1160253002

Simplify memory package interface.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1143053004

Simple memory testing for gae/wrapper

Provides partial implementations of Memcache, TaskQueue and Datastore.

Most missing things I just didn't care to implement... Datastore queries are
conspicuously absent, and I'll be doing those in a separate CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1152383003

Better attempt at an appengine wrapper.

Includes implementation using old-style appengine, and will be able to be
expanded to include support for new-style appengine SDK. I also have a partial
in-memory testing implementation in the wings which will be a followup CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1151473003

go: infra/libs/* now live in luci-go.

Companion commit that moved them there:
https://github.com/luci/luci-go/commit/15984f36e8e3a78d9893369993ea3b999e919d26

R=iannucci@chromium.org,maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153883002

cipd: Skip test on Windows for now.

The code compiles, but unit tests don't work because Windows handles symlinks
and other file system stuff differently.

R=iannucci@chromium.org
BUG=487483

Review URL: https://codereview.chromium.org/1133353002

Fix meta lib to be correctly compilable.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/998153002

Thin library for interacting with the GAE datastore metadata entities.

Only implements the one function I need right now, but can be extended for
others as needed.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/985583003

Simple GAE/goon Context wrapper

This is useful for restricting an API to implement 'datastore without queries'
or 'key-only context', etc.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/981003003
==========

[dnj, 2016-09-01 05:28:54]

Description was changed from

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)
  taskqueue.Add(datastore.WithTransaction(c, nil), task)
  prod.AEContext(datastore.WithTransaction(c, nil))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

Implement info.ServiceAccount() on devserver via info.AccessToken(...).

R=iannucci@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2220193002

Update cloud package.

Fixes #54.

BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2217063003

Remove empty namespace assertion from impl/memory.

This was fixed upstream in:
 
https://github.com/golang/appengine/commit/5a9e2b406e4e87cd8a88d1881a31b1248d...

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2219673002

GAE CL for luci-go common refactor

R=nodir@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/2192003003 .

Add an option to install just Cloud Datastore.

TBR=iannucci@chromium.org
BUG=None
TEST=None

Review-Url: https://codereview.chromium.org/2179843004

cq.cfg: use luci builders

do not use buildbot builder
update luci bucket name

R=estaab@chromium.org, dnj@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2179543003

Add lightning talk that I didn't gave at Gophercon '16


Add cloud datastore implementation.

Fixes #52.

BUG=

Review-Url: https://codereview.chromium.org/1957953002

datastore: Update AllocateIDs to take keys.

Update the Datastore's AllocateIDs call to accept variadic arguments similar
to Get/Put/Delete/Exists. Structs can be supplied to populate their keys
directly, and keys or key slices can be supplied to directly allocate keys.

This adds some Key helper functions:
- SameKind, which asserts that keys reference the same kind.
- Partial, which returns a partial key given a source key.
- WithID, which returns a derivative key with the specified ID.
- NewIncompleteKeys, which allocates a slice of incomplete keys given a kind.

BUG=

Review-Url: https://codereview.chromium.org/2007123002

Refactor multiarg, split MGS/PLS.

Refactor the multiarg construction method to handle the various cases
inline, rather than defining and invoking methods for specific
scenarios. This enables PropertyLoadSaver-based methods to be handled
separately from MetaGetterSetter-based methods, which, in turn, enables
the two to be implemented independently.

A bit of boilerplate is removed and the odd case of MGS/PLS Map/Chan fits
better. Additionally, a few double-reflection code paths are optimized
along the way.

BUG=None

Review-Url: https://codereview.chromium.org/2048933004

make swarming trybuilders non-experimental

R=iannucci@chromium.org, dnj@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2083193006

cq.cfg: enable experimental swarming builders

R=estaab@chromium.org, vadimsh@chromium.org
BUG=622093

Review-Url: https://codereview.chromium.org/2090433002

Add datastore dumper tool.

This allows you to dump the datastore in whole or part, with or without special
formatting directives.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2037433002

Add TrimmedAppID to info service.

Also refactor info service to share common 'augmented' methods in
the service/info package itself, ala service/datastore.

Split out from CL: https://chromiumcodereview.appspot.com/1537883002

R=dnj@chromium.org
BUG=550684

Review-Url: https://codereview.chromium.org/2062613002

Add DefaultVersionHostname to impl/memory/info.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2028423002

Fix typo in pls.go docs


Allow uint16 and uint32 in MakeKey.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036543003

Add GetNoTxn to txnBuf as a hack.

I really don't like this but short of fixing #23, I don't think there's much we
can do.

R=dnj@chromium.org, nodir@chromium.org
BUG=

Review-Url: https://codereview.chromium.org/2036593002

Change gae to apache 2 license.

Update AUTHORS, CONTRIBUTORS, and LICENSE.
Add apache 2 license check to PRESUBMIT.py.

Ran:
  git sed 's/Chromium Authors/LUCI Authors/'
  git sed 's/governed by a BSD-style license that can be/governed under the
Apache License, Version 2.0/'
  git sed 's/found in the LICENSE file/that can be found in the LICENSE file/'
  git checkout HEAD -- $(git status -sb | grep '^ T' | cut -c 4-)
  git checkout origin/master -- $(git diff origin/master --name-only | grep
third_party)

BUG=

Review-Url: https://codereview.chromium.org/2021753002

Change codereview server to be consistent with other luci projects.

BUG=

Review-Url: https://codereview.chromium.org/2020933002

dscache: Reduce log level of memcache failures.

Warning-level failures quickly pollute the AppEngine log space. This
change converts some of the more common warnings to Debug-level logs so
the Warning space remains reserved for higher attention calls.

Also add a warning about lock eviction poisoning to THERE BE DRAGONS
section of the doc.

BUG=

Review-Url: https://codereview.chromium.org/1987353002

datastore: variadic Get, Put, Exists, Delete.

Fixes #51.

Update Get, Put, Exists, and Delete to be variadic. This will make the
interface more usable and get rid of boilerplate in client code.

Get, Put, Exists, and Delete now accept variadic arguments. Any one of
these arguments can be an argument that was previously valid or
the *Multi version of those functions. This effectively obsoletes the *Multi
versions, but they are left in for backwards compatiblility with the stated
intention of removing them at a later date.

Get/Put/Exists/Delete will now return a single error in the one-argument
case or a MultiError in a multiple-argument case. Note that now that single
argument case can take a slice type, the single error may itself be a
MultiError if the single argument is a slice.

Review-Url: https://codereview.chromium.org/2011773002

filter/txnBuf: Use fully-qualified App ID.

Use the fully-qualified app ID when creating the in-memory datastore for
transaction buffer. Previously, transaction buffer would use the
standard App ID, while production datastore would use the
fully-qualified App ID, causing key incompatibility errors in
production.

Cleanup impl/memory's NewDatastore function to not install superfluous
services into the Context. This draws a cleaner line between
impl/memory's dual purposes as both a rich testing enviornment and a
production-grade in-memory datastore.

In addition, have impl/memory's info service differentiate between the
two. This causes tests for filter/txnBuf to properly fail lacking this
fix, enabling regression protection.

To support this, impl/memory's Info service will now parse the supplied
App ID. If it contains a "~" character, it is split into fully-qualified
and regular AppID. Otherwise, the two will be the same.

BUG=

Review-Url: https://codereview.chromium.org/1929423002

Travis: Increase system specs, limit concurrent.

Increase the Travis system specs and limit the number of concurrent
tests to try and avoid the signal-kill error that we're seeing in Travis
ATM.

TBR=iannucci@chromium.org
BUG=None

Review URL: https://codereview.chromium.org/1922583002

pcg: Bump test time to 20 minutes.

Travis is currently timing out.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1918713003

Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1917513002

impl/memory: datastore and memcache namespacing

Update GetNamespace() to return an additional boolean, which will be
true if the namespace was set and false if it was not. The AppEngine SDK
defaults to having no namespace installed ("", false), but will allow
the user to install any namespace. This allows a user to differentiate
between no namespace and an empty namespace ("", true).

Make impl/memory's datastore query operation to error if the namespace has
been set to the empty namespace (""). This reflects observed production
behavior.

BUG=

Review URL: https://codereview.chromium.org/1916463004

Bump max_duration for CI tests to 480s

Hopefully Fixes #48.
Fix memory corruption bug in impl/memory

When you `SetCollection` in gkvlite, it invalidates the old
collection, but keeps the Items of the previous collection. The way
that impl/memory was using SetCollection was more along the lines of
GetOrCreate, which could cause corruption or other issues if two
codepaths did SetCollection simultaneously.

This fixes the corruption bug by removing SetCollection from the
memStore interface and replaces it with a GetOrCreateCollection,
which checks for the collection, and only calls SetCollection if it
doesn't exist. As a bonus, this seems to give a slight performance
speedup over the previous broken code.

Additionally, this forces all uses of VisitItemsAscend to use
a Snapshot, to avoid accidental modification of a Collection while
its being iterated over, which can lead to the situation discussed
above, or other nasty errors.

Finally, this adds an optional 'gkvlite tracer' functionality
which came in (extremely) handy for finding this bug. When given
a folder, it will emit go-code-like invocations of the functions run
on every collection, and includes enough information to reveal which
functions are being called on the root collection v. snapshots, and
if there are any overlapping VisitItemsAscend invocations, etc.

R=dnj@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1911263002

Travis: Build against Go1.6.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1911293002

Remove race from query finalization.

BUG=

Review URL: https://codereview.chromium.org/1910293002

datastore: Fix AddIndexes with existing namespaces

AddIndexes previously only added an index for the current namespace.
Since indexes are global, they should be added for all existing
namespaces.

BUG=

Review URL: https://codereview.chromium.org/1894403002

service/taskqueue: Add NewPOSTTask, remove NewTask

Add NewPOSTTask, which mirrors the method of the same name in
AppEngine's Go task queue SDK.

Remove NewTask from the taskqueue Interface, since it was basically
useless.

BUG=

Review URL: https://codereview.chromium.org/1890983004

Add __namespace__ metadata query support to impl/memory.

Ref: https://cloud.google.com/datastore/docs/concepts/metadataqueries

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1892343002

Add Info Testable interface, implement for memory.

This adds a Testable interface to the Info service which allows it to
install Info parameters in Contexts.

Add support for setting version ID and request ID to impl/memory Info
service.

BUG=

Review URL: https://codereview.chromium.org/1871943003

Add PopulateKey method to datastore.

Add PopulateKey, a method that, given an object, fills in its key
fields.

Fixes #44.

BUG=

Review URL: https://codereview.chromium.org/1867963003

Fix missing/broken ds.Stop handling in raw DS.

Several places were missed when patching "ds.Stop" to being correctly filtred
at the top interface rather than the raw interface.

At the top-level, GetAll and Count both did not filter the Stop error.
At the filter level, several filters were ignoring error values from
their callbacks.

Also fixes an issue in count filter where the ds.Stop error was being
incorrectly filtered at the RawInterface level.

Add datastore iterating query filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1846123002

Add datastore iterating query filter.

BUG=

Review URL: https://codereview.chromium.org/1843313004

Don't filter "Stop" error in raw interface.

The Stop error is a high-level error used by the datastore service to
indicate that an iterative operation should stop rather. It should
appear to the user as a "nil" error.

However, currently the datastore raw implementations are filtering Stop
and converting it to "nil". This prevents other filters from
identifying Stop events from their lower level callbacks.

Update raw datastore methods to not filter Stop, and update their tests
to accept Stop as a valid successful result.

BUG=

Review URL: https://codereview.chromium.org/1844183003

Make ExistsMulti return a BoolList

BoolList has some convenience functions (e.g. to check if Any() or All() are
set.). It's compatible with current uses of ExistsMulti.

R=dnj@chromium.org, estaab@chromium.org, nodir@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1816413002

txnbuf: Add write count budget, size budget.

Add a "maximum number of datastore writes" budget to the transaction
buffer. Rework the size budget to remove preemptive child transaction
failures and directly derive budget from the parent.

BUG=

Review URL: https://codereview.chromium.org/1776833002

Add wrappers for the module module

BUG=570879

Review URL: https://codereview.chromium.org/1772943003

Add a MustNamespace that panics if the namespace wasn't valid

BUG=

Review URL: https://codereview.chromium.org/1766593002

Auto-project on distinct inequality fields.

BUG=

Review URL: https://codereview.chromium.org/1745703002

Add header text to proto-gae tool

In order to appease our presubmit scripts, default to the Chromium Authors
copyright header.

R=dnj@chromium.org, estaab@chromium.org, hinoka@chromium.org,
martiniss@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1718123003

dscache: Only log unusual memcache errors.

BUG=None
TEST=None
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1695833002

Use ShouldResemble instead of ShouldResembleV.

Now that goconvey incorporates our ShouldResembleV method directly, we
can go back to using ShouldResemble instead of hacking in
ShouldResembleV!

BUG=None

Review URL: https://codereview.chromium.org/1679093002

Add support for uint8, uint16 and uint32 in luci/gae

This has come up for like 3 models in DM so far, and I'm tired of having this
be a stupid "gotcha!". I was working around it with some PropertyConverter, but
with this change it will just be obvious what the heck is going on.

They all map to positive int64 values (which should make indexes obvious), and
they include out-of-bounds checks on deserialization.

I excluded uint64, since there's no way to properly encode it in the native
datastore representation which still makes any sense for indexes.

I also added some (apparently?) missing documentation in PLS which explicitly
lays out what field types are allowed in a pls-managed struct.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=550684

Review URL: https://codereview.chromium.org/1667403003

Fixes #8: Seed indexes from index.yml

- Add `ParseIndexYAML` and `FindAndParseIndexYAML` functions
  to service/datastore
- Add YAML marshalling and unmarshalling for IndexDefinition and IndexColumn
- Add tests for the changes
- Imports: additional package: gopkg.in/yaml.v2

https://github.com/luci/gae/issues/8

Closes #8

R=dnj@chromium.org, estaab@chromium.org, iannucci@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1574353004

impl/prod: Expose Remote API scopes.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1632943003

Add UseBackground.

Fixes #39.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1630833002

Add tool to generate PropertyConverter implementations for proto messages.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1625773002

Fix logger for filter api change in luci-go.

API change will occur: https://chromiumcodereview.appspot.com/1622553005/

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1621993003

Fix fast string comparison bug.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1620873004

Add a test to ensure []byte => BlobKey PLS works.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1616033004

Encode nil datastore keys as PTNull.

This fixes a panic in serialize when it is asked to serialize a nil
*ds.Key, causing it to dereference its fields. Instead, a *ds.Key is
serialized as a PTNull entry like datastore.

Add some tests to assert proper conversion and serialization.

BUG=

Review URL: https://codereview.chromium.org/1576153003

Add tests for binary_tools functions

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1576353002

Add dscache.AlwaysFilterRDS function.

It's like dscache.FilterRDS, except it doesn't use IsGloballyEnabled. Intended
to be used by callers that know better when to apply (or not) the filter.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1579853002

Fix GAE <-> GAE RemoteAPI

Urlfetch.Get needs the user context, not the aeCtx.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1578943002

Make UseRemote work for dev_appserver too.

Now interacting with remote_api is as simple as:

    ```go
    ctx := context.Background()
    if err := prod.UseRemote(&ctx, "localhost:8080", nil); err != nil {
      panic(err)
    }
    // ctx is usable with luci/gae
    ```

Also changes the context-derivation function signature to be safer, since the
client's context will always be valid, and can be used for things, such as
logging even in the error handling case.

This also makes UseRemote work when used FROM appengine to another appengine
app.

Fixes #36.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1557223002

impl/memory: Fix time serialization encoding.

In a PropertyMap, time is serialized as an integer type. However, for
queries, time is serialized as a time type. This causes all queries on
time fields to select any encoded time properties, since PTTime > PTInt.

This patch causes queries to serialize PTTime properties as PTInts to
match the indexes.

Fixes #34.

Review URL: https://codereview.chromium.org/1550903002

Replace spammy log message in dscache with comment.

Fixes luci/gae#35.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/35

Review URL: https://codereview.chromium.org/1556193002

Add remote_api support

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/33

Review URL: https://codereview.chromium.org/1549223002

impl/memory: Add "PULL" as valid taskqueue method.

BUG=

Review URL: https://codereview.chromium.org/1550863002

Use prod logger's user context to read logging level

TBR=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/32

Review URL: https://codereview.chromium.org/1542263003

Make setting a Memcache Item with a nil Value OK

This conforms to the expectations that if you Set with a nil Value, it's the
same as setting to []byte{}... One less trap :)

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/31

Review URL: https://codereview.chromium.org/1541183002

Fix doc bug in pls.go


Implement Mail service.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/20

Review URL: https://codereview.chromium.org/1525923002

impl/prod: Fix logging field output.

By using the AppEngine context, logging calls would not have access to
logging fields that were set in subsequent contexts (most of them).
Retain both the AppEngine context (for logging calls) and the logger
context (for up-to-date logging configuration), and use the latter for
field extraction.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1523343002

Add filters for user service.

This brings the user service up to parity with the rest.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1530533002

Panic when non-runtime-handleable type errors occur

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/28

Review URL: https://codereview.chromium.org/1525873002

Fix nil-deref panic from Put.

We were using key before checking the error.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/26

Review URL: https://codereview.chromium.org/1523943002

Return error on Get(*Key).

Before it would end up returning ErrInvalidKey, which is silly :).

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org,
martiniss@chromium.org
BUG=https://github.com/luci/gae/issues/27

Review URL: https://codereview.chromium.org/1530503003

Clean up callback interfaces.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/17

Review URL: https://codereview.chromium.org/1521823003

Add User Testing interface to impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/24

Review URL: https://codereview.chromium.org/1522783002

Add logging service documentation.

Just points to github.com/luci/luci-go/common/logging, but explains how that
works and meshes with the rest of the service definitions.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1526453002

Handle unexpected entity fields gracefully.

Introduces a new field type 'extra', which allows full control over what to do
with the extra properties, instead of having all operations return unhelpful
conversion error messages.

This allows black-holes, schema upgrades and 'expando' style mixed
structured/unstructured objects. Please see the tests for usage and examples.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/13

Review URL: https://codereview.chromium.org/1525453002

Clean up PropertyLoadSaver interface.

This removes GetMetaDefault and Problem from the interfaces, and makes GetPLS
errors into panics, on the assumption that if you pass it a bad datatype, you
can't really do anything about it without recompiling, and failing fast is
better.

This also fixes a bug where doing any operations on an invalid (non datastore-
compatible) struct would return the unhelpful "unable to extract $kind" error.

R=dnj@chromium.org, hinoka@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1516173002

Transfer context deadline to AEContext.

It's needed to be able to tweak default RPC timeout (5 sec), in particularly
when using urlfetch service.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1515413002

Move gaelogging from luci/luci-go to luci/gae

Also makes impl/memory Use memlogger too for consistency. This change is
required to make luci-go logging work on appengine due to
3a50526e08194bd66487c7c03cccd2108c236468.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/22

Review URL: https://codereview.chromium.org/1498963003

Add API to allow you to get the non-transactional datastore or taskqueue.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1494223002

GAE Datastore - Fix derived meta types not being reflected properly

Stuff like "type FooBar string" were not allowed as metadata fields correctly.

BUG=

Review URL: https://codereview.chromium.org/1494803005

Fix User API (CurrentOAuth)

TBR=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1493183003

Add User service.

Doesn't add a memory implementation... it's easy enough to mock directly in the
test and doesn't have any complex systemic behaviors (that I know of) that are
necessary for a single mock implementation.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/19

Review URL: https://codereview.chromium.org/1500433002

Make appengine connection explicitly owned by impl/prod.

R=vadimsh@chromium.org, hinoka@chromium.org
BUG=

Review URL: https://chromiumcodereview.appspot.com/1490243004 .

Update README.md

Typo :)
Fix races in txnBuf

Runs clean with `-cpu=2,4 -race` now. It's also a lot clearer to follow the
logic now. No more 'siblingLock'. At the time that each recursive transaction
starts it locks (in the normal way!). It then keeps a reference to the parent
layer with the locks disabled for its internal usage.

This refactor makes it so that all (interacting) operations at a given 'layer'
in a transaction are mutually exclusive. Previously only nested transactions
would be mutually exclusive.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1434873003

Add some race-detector tests

The impl/memory one is fine, but the txnBuf one is currently failing.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1437553002

Fix another race.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1430193002

Fix race in impl/memory.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/16

Review URL: https://codereview.chromium.org/1411573014

Allow metadata fields to be PropertyConverters for symmetry.

This makes the easiest forms of parsed kinds/ids extremely simple to implement,
as well as composable. The MetaGetterSetter interface requires every top-level
struct to implement it, since allowing embedded implementations will get the
wrong value for the derived "$kind".

R=dnj@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1414043006

service/datastore: Call MGS GetAllMeta() in Save()

Fix a bug where a structPLS that uses Save(withMeta: true) will ignore
its object's MetaGetterSetter overrides and use struct-default.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1425283002

Decouple PLS from MGS.

Decouple PropertyLoadSaver from MetaGetterSetter. This allows
implementing structs to override either one of those interfaces without
depending on spooky the interaction between them implemented by
"structPLS".

"structPLS" no longer magic-detects if its type implements
MetaGetterSetter and defers to it. Instead, a struct that implements
MetaGetterSetter and wants to use "structPLS" cool meta defaults
can call GetPLS(this) to get a handle on that base functionality.

BUG=

Review URL: https://codereview.chromium.org/1427933002

Use the sort.Search api correctly.

I was using it again and I realized that I used it wrong here :(. Includes
regression test.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1409613002

Add ability to override just the metadata in a struct.

This is useful when the '$id' is actually multiple fields packed into a string,
but it's generally available for any metadata field.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/15

Review URL: https://codereview.chromium.org/1401533007

Add Exists and ExistsMulti convenience methods

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1398103003

Add VersionID to impl/memory info service implementation

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1403873002

Add Gob encoding functions for Key.

This is so that *Key can transparently be encoded in tumble mutation structs.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533020

Review URL: https://codereview.chromium.org/1397673003

Add transaction buffer filter.

This will be used for running multiple non-cooperative idempotent tasks inside
of tumble, which require seeing the previous tasks' effects on the entity group.

However, the effect of having transactions transparently buffer datastore
interactions until the final transaction commit is a generally useful alteration
of the current behavior.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309803004

Fix bugs in impl/memory queries.

This prevents built-in indexes from being selected when using ancestor and
non-__key__ inequality filters, and also fixes a bug where a multi-iterator on
builtin indexes using __key__ inequality with ancestor would return empty result
sets.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1380613002

Add a NewDatastore helper function.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379673002

Make impl/memory have an AppID.

This associates an immutable AppID with the datastoreData object (e.g. the thing
that actually holds the data). It gets rid of the hard-coded globalAppID hack.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1369353003

Improve the way "consistent" works.

This avoids the overhead of taking many many snapshots when the in-memory
datastore is in full-consistency mode.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1379563004

Fix bugs in service/datastore

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1378613003

Add Key.HasAncestor()

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1377863004

Add the ability to disable special automatic entities in impl/memory

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1372473002

Add AutoIndex to datastore Testable interface.

This allows test writers to set Consistency(true) and AutoIndex(true), and
pretend like the datastore is a fully consistent database which always has
perfect indexes. Will be used for the recursive transaction manager to
maintained tiered automatic mini-datastores in memory.

Also add some more tests for queries in transactions, and YAML rendering for
*IndexDefinition.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021, https://github.com/luci/gae/issues/9

Review URL: https://codereview.chromium.org/1364333002

Add Consistent(bool) to Testing interface

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1366793002

Add missing Count api

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1367003002

Refactor serialization helpers from impl/memory -> service/datastore/serialize

These will be used to aid in joining query results for nested transactions.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1365743002

Add ability to estimate the size of a PropertyMap

This will be used to track in-flight transaction sizes.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533021

Review URL: https://codereview.chromium.org/1363063002

Fix minor bug in ParseIndexColumn

Also improve error message.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1359963003

Add Kind/StringID/IntID back to Key.

Rename Last -> LastTok.

luci-go change here: https://chromiumcodereview.appspot.com/1359813004

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1358063003

service/datastore: misc improvements in properties.go

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354343002

Add missing AllocateIDs API.

No idea why I missed it. Tweaked a bit to get rid of some redunant info.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1357343002

Add API compat warning :)


Fix memcache.Interface.Get to return a memcache.Item

This intentionally doesn't change GetMulti. Since GetMulti is used for batch
operations, it's actually more convenient to have a pipeline of actions (e.g.
AddMulti -> GetMulti). Otherwise you have to carry around a []Item and a
[]string and keep them both in sync (overhead, error prone).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/3

Review URL: https://codereview.chromium.org/1354333002

Make Get operations only serialize the bare minimum.

Now PropertyLoadSaver has GetAllMeta, which is like Save, except that it ONLY
returns the metadata. This avoids situations where zero-initialized structs
aren't expecting to be serialized for a nominally write-only operation (e.g.
Get).

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/11

Review URL: https://codereview.chromium.org/1358743002

Fix zero time bug.

`time.Time{}` needs to be able to round-trip everywhere. Previously
various things were implicitly adding the Local or UTC timezones, even
when the time value was zero.

R=vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/10

Review URL: https://codereview.chromium.org/1345043005

Refactor keys and queries in datastore service and implementation.   *
datastore.Query and datastore.FinalizedQuery are now first class objects     in
the service/datastore api.   * datastore.Key is similarly a first class object.
All luci/gae users will     need to interact with `*ds.Key` now. This object
gained all the superpowers     that previously existed in the dskey subpackage
of service/datastore.

The new Query objects are inspectable, and adjustable. Additionally, this
homogenizes they query generation between different implementations. The
normalization which normally occurs on the server-side is now done client-side.
This should ensure consistency between 'prod' and 'memory' implementations.

Additionally, the new FinalizedQuery object gains a GQL() method which should
(hopefully) allow trivial debugging by printing a query. You should be able to
copy and paste the generated GQL* into the appengine dashboard and see what
your request was querying for.

* With the exception of Cursors, which currently have no textual format in the
GQL schema.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=533024

Review URL: https://codereview.chromium.org/1355783002

Add simplest service/urlfetch implementation.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1354083002

Allow PRESUBMIT run pcg on trybuilder.

R=vadimsh@chromium.org
BUG=529936

Review URL: https://codereview.chromium.org/1345303002

Make SetTransactionRetryCount actually work.

It used to store the state in ephemeral dsImpl object that is minted with each
call to datastore.Get(...). Store it in more durable dataStoreData instead.

R=iannucci@chromium.org
BUG=https://github.com/luci/gae/issues/2

Review URL: https://codereview.chromium.org/1347843002

Add IsDevAppServer implementation to impl/memory.

Returns 'true', since impl/memory is sort of a dev sandbox too.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1347813003

Use int64 to store time.Time internally in memory implementation.

Uint64 causes overflow when storing zero time.Time{} (that due to some Go
stubborn weirdness is represented by negative timestamp).

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334043003

Make PRESUBMIT.py run checks via 'pcg' if it is available in PATH.

R=maruel@chromium.org, iannucci@chromium.org
BUG=chromium:529936

Review URL: https://codereview.chromium.org/1342673002

Add emulation of transaction retries to memory impl of RunInTransaction.

Available via Testable().SetTransactionRetryCount().

BUG=https://github.com/luci/gae/issues/2
R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1335083002

Switch to external stringset implementation.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1312433013

Add pcg to ignore impl/prod

This should fix the travis redness.

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1334833002 .

Implement projection queries correctly.

* Add Project and ForIndex to Property in service/datastore
* Remove PTBoolFalse/PTBoolTrue silliness (now just PTBool)
* Remove ByteSlice silliness (now []byte behaves exactly like string w.r.t.
  indexing. If you don't want it indexed, put `gae:",noindex"` on it.)
* Make indexes index 'underlying' types. e.g. PTTime is not distinguished from
  PTInt in the index, nor are the various byte sequence types.
* Make impl/prod correctly extract data from internal datastoreIndex objects
  when doing projection queries. This allows doing a projection into
  []PropertyMap. Since all data is bussed internally as PropertyMap (instead of
  crazy PropertyLoadSaver implementations), this is required to make projection
  queries work. I'm pretty confident that this won't break, however, as it's all
  derived from protobufs which are extremely unlikely to change.

  To make this better, we'd have to implement the protobuf protocol directly in
  luci/gae instead of routing through the SDK. Not as absurd as it sounds, but
  definitely out of scope for this CL :).
* Include extremely basic test for impl/prod using aetest. This test takes 12s
  to run on my laptop. *sigh*.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=https://github.com/luci/gae/issues/1

Review URL: https://codereview.chromium.org/1336443003

More minor formatting of README.md


Fix formatting in README.md


impl/memory: Implement Queries

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1302813003

filter/count: Fix alignment on 32-bit systems.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1306373002 .

datastore: Add DecodeCursor to [Raw]Interface.

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1309933003 .

Update README to use CQ.

TBR=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1301833005

Update codereview.settings for CQ.

Also add infra-reviers+luci-gae@ to CC list.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1286973004 .

Add Chrome infrastructure CQ config.

R=phajdan.jr@chromium.org, sergiyb@chromium.org
BUG=403494

Review URL: https://codereview.chromium.org/1298323004 .

prod: fix timezone conversion from AppEngine layer

R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1298423005 .

Fix miscellaneous prod bugs.

- Fix bug where "nil" Parent field was being translated into non-nil
  empty key.
- Use fully qualified AppID for key AppID in checkfilter.
- Use a context-installed singleton to probe AppEngine runtime parameters.
- Bump Task field count to match gae/AppEngine count.
- Fix UTC time check when Locations are the same, but have different
  pointers.
- Pass PropertyLoadSaver value array (rather than source) in
  prod/PutMulti :)
- Use struct name as default even when it implements PropertyLoadSaver.

R=iannucci@chromium.org

Review URL: https://codereview.chromium.org/1289323002 .

Fix key encoding to be null terminated

This will allow the use of automatic indicies to fulfil basic ancestor queries.
Otherwise keys with many components sort after keys with few components (instead
of lexicographically per token)

R=dnj@google.com, tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286093007 .

Add single-collection gkvlite iterator.

Multi-collection iterator is up next.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1300493002 .

Split off serialization and key functions to their own packages.

This makes the service/datastore package smaller and more single-purpose.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1292913002 .

impl/memory: Make queries self-validate

This moves validation and normalization logic into the query construction
process itself. This makes it so that queries are either valid and normalized,
or they contain an error (and query building 'sticks' on the first error
encountered).

This replaces the previous
build-a-questionable-query-and-then-try-to-make-it-sensible
logic (which was copied from dev_appserver for the most part).

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1286943002 .

Add testable interface for datastore.

Includes the ability to manipulate the 'eventual consistency' state of the
datastore, as well as the ability to add compound indicies.

R=dnj@google.com, dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1285703002 .

Make LazyMultiError constructed via function

Followup to https://codereview.chromium.org/1249933002/

Counterpart to https://codereview.chromium.org/1284323002/

TBR=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org,
tandrii@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1291813003 .

Refactor: Rename some files.

R=tandrii@chromium.org, dnj@chromium.org, estaab@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1281173002 .

Make obtaining the Testable interface actually part of the interface.

Reduces the number of free casts, and will actually work with the upcoming
Testable interface for datastore, which uses a checkfilter (and so casting
won't work there anyway).

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1282673002 .

Get rid of awkward proto argument to Interface.Run

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1279703003 .

A transparent cache for datastore, backed by memcache.

Check doc.go for details, but:
  * allows cache disablement on a per-model/Key basis
  * allows cache expiration on a per-model basis
  * efficient (non-gob) encoding scheme
  * gzip for large entities
  * allows randomized sharding for Gets
    * allows good throughput for hot items
    * shards may be determined on a per-Key basis
  * caches negative hits

R=dnj@chromium.org, vadimsh@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269113005 .

Re-add metadata passthrough on Get operations

It got lost in translation during the RawInterface conversion. This will be used
to e.g. allow SomeStruct to set a default dscache.enable metadatum when doing
e.g. Interface.GetMulti([]SomeStruct{...}).

R=dnj@chromium.org, dnj@google.com, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270113002 .

Use time.Time for internal memcache memory implementation

Followup to https://chromiumcodereview.appspot.com/1270063003/

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1262173003 .

Make the rest of the services have a similar raw/user interface structure.

Also implement missing functionality from taskqueue/memcache in the memory
implementation.

R=dnj@chromium.org, estaab@chromium.org, tandrii@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063003 .

Add 'user friendly' datastore API.

This is a thin veneer on top of datastore.RawInterface, and may be used
with any filters, any backend, and it may be used interchangably and
inderleved with datastore.RawInterface.

In particular, this adds a convention for embedding keys inside of
loadable/savable objects.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259593005 .

Rename rawdatastore -> datastore

Also rename some of the .go files in the service to prep for
https://codereview.chromium.org/1259593005/

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1270063002 .

Fix obviously broken error handling.

R=vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org,
tandrii@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1269593002 .

Make rawdatastore API safer for writing filters.

This removes Get/Put/Delete, since they always end up being implemented
as wrappers around the Multi versions anyway, it just caused unnecessary noise.
They'll come back with the 'user friendly' datastore interface, but there's no
reason to have them at every layer of the cake.

It moves the *Multi methods to be callback-based, which makes interacting with
them easier (since you don't have to do lots of slice allocations).

It introduces 'checkFilter' to rawdatastore which guarantees that filters and
backend implementations don't need to do lots of redundant checks to sanitize
their inputs.

It adds a new API to the Info service to retrieve the current namespace.

R=dnj@google.com, estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1253263002 .

Add Toggle for rawdatastore.

This will be used to control cacheability for entities, and allows setting a
default value in the tag without ambiguity (since the 'zero-value' for bools is
false, a simple bool wouldn't provide a way to override defaulted-true field).

R=dnj@google.com, vadimsh@chromium.org, dnj@chromium.org, estaab@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1259463002 .

Update README.md formatting


Add PRESUBMIT.py


Minor doc improvements

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1241423003 .

Add LICENSE


Revert "Enable coveralls.io"

This reverts commit fc93f78cf1d04d25fb3f52dd8040dba88649494e.

Enable coveralls.io


Remove most error code from luci/gae

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1249863004 .

Improve docs

Simplify godoc by making some functions methods
Rename 'filters' to 'filter' to be more consistent with 'impl' and 'service'.

R=estaab@chromium.org, dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1247283003 .

Refactor a bit.

TBR=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1243323002 .

Switch to other coveralls badge


Force timezone to be non-UTC when testing non-UTC


Fix gae.DSTransactionOptions


Minor change for Travis :)


Add README.md


Add .travis.yml


Add codereview.settings


Fix imports and tests

Run goimports over everything for good measure as well.

Fix DSKey encoding.

Specifically, IntID's always sort before StringID's. This is according to
https://cloud.google.com/appengine/docs/go/datastore/queries#Go_Key_filters

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1231863018

Simplify memlock and make it less racy.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1233413002

Rename poorly named files.

R=dnj@chromium.org, estaab@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240123003

Improve memory implementation of gae to provide the full RawDatastore interface.

Still no query support yet, working on that next, but this enables *Multi apis.
Also do a few other code cleanups.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1242043005

Add LazyMultiError.

This is essentially a MultiError, except that it knows how big it should get
if it's ever assigned an actual error, and it only does its allocation lazily.
This is useful for implementing e.g. PutMulti and other Multi-style methods.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234493005

Add NewDSKeyFromEncoded convenience function.

Also placate golint :).

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1238573004

Convert 'Testable' interface into an optional filter.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227203004

Add "filters/count" filter.

This filter allows you to track the number of times a give API has been
invoked, as well as the number of errors that the API has returned. Will mostly
be used for testing.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1239103003

Add filters for gae library.

Filters allow users to add their own wrappers around the underlying gae
implementation. They can do anything that an implementation could do, except
that their backend is another gae library interface, instead of the SDK or
memory implementations. Filters are also necessary (instead of just stacking
implementations with SetXXFactory functions), due to transactions, which change
the context object.

They will be used to implement (at least):
  * caching (hitting memcache before datastore)
  * transaction buffering (giving a self-consistent view of the datastore
    within a transaction, buffering all writes to the end of the transaction
    to make transaction sizes smaller, etc.)
  * featureBreaker (for tests, breaking certain APIs with certain errors)
  * counter (for tests, counting the number of times an API was hit)

It may also be used for:
  * ACL enforcement (rejecting access to datastore items the user can't get,
    filtering queries, etc.)

Thi CL also implements a counter filter as a demo (which will also be used for
testing other filter implementations to ensure that the filter being tested).

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222983002

Some miscellaneous fixes

  * Fix DSKeyIncomplete to check for nil
  * Fix docstring in memory/context.go
  * Make GlobalInfo support BrokenFeatures
  * Fix memory TaskQueue to fill in the default queue name, per SDK spec.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1234343006

Simplify taskqueue interface.

The removed complexity was a holdover from the initial versions of this library.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1235563007

Change RawDatastore to do less reflection.

Followup CL will provide the 'magic' reflection-full Datastore interface.

This change makes filters easier to implement, none of the layers of the stack
are required to do reflection, instead of the previous design where all of the
layers were forced to do reflection.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1227183003

Move dummy service implementations into their own package.

R=dnj@chromium.org, estaab@chromium.org, martiniss@chromium.org,
vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1226063003

Reland: Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

Original broke because endpoints needed to be moved to a version which relies
on the new SDK.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1240573002

Revert "Refactor current GAE abstraction library to be free of the SDK*"

This reverts commit a91c4ac2149f8bbe3711c4c662b6d500c9a8f799.

CQ didn't validate that everything builds. Consequently, note everything builds
:D.

TBR=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1230303003

Refactor current GAE abstraction library to be free of the SDK*

Also improves the struct serialization implementation over the SDK's:
  * 'non-struct' representation is now map[string][]values, instead of
    []PropertyList. This makes working with PropertyLoadSaver much easier.
  * Fields can implement a DSPropertyConverter interface to transparently be
    serialized to/from some other datastore-compatible type. This makes many
    uses of PropertyLoadSaver unnecessary.
  * Lots of reflection code moved out of structPLS.{Load,Save} into the creation
    of the struct codec (e.g. it happens once instead of on every load/save
now).
  * Struct codecs are directly accessible by users, allowing easier
    implementation of PropertyLoadSaver interface.
  * Struct codecs support 'special' tagged fields. This will be used for
    implementing an embedded-key Datastore interface without adding any extra
    reflection code.
  * DSProperty's now have strictly private fields. This ensures that every
    DSProperty instance contains only valid types for Value (since it's not
    possible to construct one with an incompatible Value now), which simplifies
    error checking code.
  * DSProperty does automatic upconversion of close-but-not-exact types (like
    MyString -> string, and int -> int64). This upconversion was previously only
    available for struct serialization.
  * DSProperty's now retain a sortable 'Type' field to avoid unnecessary
    additional reflection when serializing them (such as to memcache).
  * There is now a consistent, sortable binary serialization format for
    DSPropertyMap's, which will be used for the follow-on transparent memcache
    implementation. It's also used internally by the in-memory testing
    implementation of the datastore.

*I alias some error types from the new-SDK, but users of the abstraction no
longer need to import the SDK at all.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1222903002

Use SetFinalizer to avoid leaking Snapshots.

R=dnj@chromium.org, martiniss@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1223263006

Roll infra/go/src/github.com/luci/luci-go/ to
b85c6dd748361e8244122faac44b5beefcd2db37.

$ git log cd33aabb0..b85c6dd74 --date=short --format='%ad %ae %s'
2015-07-07 robbie Rename funnybase and add more serializations.
2015-07-07 yyanagisawa Reference implementation of LinkToPrimaryHandler.

R=pgervais@chromium.org, agable@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1229503007 .

A client/server helper wrapper for endpoints in Go.

The epservice directory/main program generates a appengine app which links all
the available endpoints implementations by finding packages which implement
RegisterEndpointsService. It then runs this with dev_appserver.py.

Subpackages of epservice are supposed to be one-folder-per-service.

The epclient directory/main program/go-generate invocation runs the epservice
server, and then runs the golang client library generator to generate go client
libraries for all the services it finds.

R=dnj@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153473008

Non racy tests for memlock.

Bonus, no more time.Sleep, and the tests are 100% faster (e.g. they take 0.04s
instead of 0.08s :D )

R=dnj@chromium.org, maruel@chromium.org, vadimsh@chromium.org
BUG=496436

Review URL: https://codereview.chromium.org/1159173004

Add context-aware "time" library wrapper.

TEST=unittest
R=iannucci@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1154213012

A simple memcache lock for appengine.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/986553002

Last fix for 32 bit tests?

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159623005

Fix tests to run on 32 bits.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1159343002

Fix overflow on 32bit which causes tests to fail on 32bit.

TBR=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1162813002

Add initial Query generation, correctness checking and index generation.

The ability to execute a query will come in a following CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1160253002

Simplify memory package interface.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1143053004

Simple memory testing for gae/wrapper

Provides partial implementations of Memcache, TaskQueue and Datastore.

Most missing things I just didn't care to implement... Datastore queries are
conspicuously absent, and I'll be doing those in a separate CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1152383003

Better attempt at an appengine wrapper.

Includes implementation using old-style appengine, and will be able to be
expanded to include support for new-style appengine SDK. I also have a partial
in-memory testing implementation in the wings which will be a followup CL.

R=maruel@chromium.org, vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1151473003

go: infra/libs/* now live in luci-go.

Companion commit that moved them there:
https://github.com/luci/luci-go/commit/15984f36e8e3a78d9893369993ea3b999e919d26

R=iannucci@chromium.org,maruel@chromium.org
BUG=

Review URL: https://codereview.chromium.org/1153883002

cipd: Skip test on Windows for now.

The code compiles, but unit tests don't work because Windows handles symlinks
and other file system stuff differently.

R=iannucci@chromium.org
BUG=487483

Review URL: https://codereview.chromium.org/1133353002

Fix meta lib to be correctly compilable.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/998153002

Thin library for interacting with the GAE datastore metadata entities.

Only implements the one function I need right now, but can be extended for
others as needed.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/985583003

Simple GAE/goon Context wrapper

This is useful for restricting an API to implement 'datastore without queries'
or 'key-only context', etc.

R=vadimsh@chromium.org
BUG=

Review URL: https://codereview.chromium.org/981003003
==========

to

==========

==========

[dnj, 2016-09-01 05:29:51]

Description was changed from

==========

==========

to

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)
  taskqueue.Add(datastore.WithTransaction(c, nil), task)
  prod.AEContext(datastore.WithTransaction(c, nil))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.
==========

[dnj, 2016-09-01 15:25:37]

dnj@chromium.org changed reviewers:
+ iannucci@chromium.org

[dnj, 2016-09-01 15:25:41]

This is more of a nice comfort food patch than something urgent. However, with
"luci/gae" gaining traction, the API changes might be better to stamp once and
early, so there's that.

PTAL!

https://codereview.chromium.org/2302743002/diff/1/filter/txnBuf/ds_txn.go
File filter/txnBuf/ds_txn.go (right):

https://codereview.chromium.org/2302743002/diff/1/filter/txnBuf/ds_txn.go#new...
filter/txnBuf/ds_txn.go:119: func (d *dsTxnBuf) CurrentTransaction()
ds.Transaction { return d.state.parentDS.CurrentTransaction() }
Perhaps in the future, we can synthesize transaction objects for the nested
transactions here. ATM we're just returning the underlying DS's transaction. I
was torn between implementing this right now and waiting until stronger support
for "WithTransaction" is implemented to do this, since it's not very useful
without that.

https://codereview.chromium.org/2302743002/diff/1/filter/txnBuf/ds_txn.go#new...
filter/txnBuf/ds_txn.go:126: return nil
oops, I'll make sure this returns parentDS's Testable in next patch set, don't
know how this got nil'd.

https://codereview.chromium.org/2302743002/diff/1/impl/cloud/datastore.go
File impl/cloud/datastore.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/cloud/datastore.go#new...
impl/cloud/datastore.go:120: var pmap ds.PropertyMap
I think this was a bug.

https://codereview.chromium.org/2302743002/diff/1/impl/memory/context.go
File impl/memory/context.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/memory/context.go#newc...
impl/memory/context.go:147: func inTxn(c context.Context) bool {
Ended up not needing this, will delete in next patch set.

https://codereview.chromium.org/2302743002/diff/1/impl/memory/datastore_test.go
File impl/memory/datastore_test.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/memory/datastore_test....
impl/memory/datastore_test.go:195: So(ds.CurrentTransaction(c), ShouldBeNil)
This test makes me happy.

https://codereview.chromium.org/2302743002/diff/1/impl/prod/context.go
File impl/prod/context.go (left):

https://codereview.chromium.org/2302743002/diff/1/impl/prod/context.go#oldcode32
impl/prod/context.go:32: prodContextKey      key
This is a dangerous/big change, only b/c it's hard to test this stuff live.
Basically, now instead of tracking the w/ and w/out Transaction Contexts
separately, we store one "prodState" object (see below). That object maintains a
"current" Context and the latest "No Transaction" Context, and sets "Current" to
"No Transaction" when WithTransaction is invoked with nil.

Info service namespace operations install a new prodState with both Contexts
updated to the new namespace.

Consequently, AEContextNoTxn need not exist; one just passes WithTransaction(c,
nil) to AEContext.

https://codereview.chromium.org/2302743002/diff/1/impl/prod/context.go
File impl/prod/context.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/prod/context.go#newcode99
impl/prod/context.go:99: aeCtx := AEContext(*inOutCtx)
As far as I can tell, the NoTxn part of the previous code was not important
here.

https://codereview.chromium.org/2302743002/diff/1/impl/prod/context.go#newcod...
impl/prod/context.go:168: func (ps *prodState) context(c context.Context)
context.Context {
We *should* probably do cancellation at some point. But that'd require its own
goroutine and junk so unless GAE main library supports it (which, I guess, we'll
never know since we don't offer it the opportunity to support it), this is fine.

https://codereview.chromium.org/2302743002/diff/1/impl/prod/everything_test.go
File impl/prod/everything_test.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/prod/everything_test.g...
impl/prod/everything_test.go:87: So(ds.CurrentTransaction(ctx), ShouldNotBeNil)
Added assertions that CurrentTransaction and WithTransaction work.

https://codereview.chromium.org/2302743002/diff/1/impl/prod/info.go
File impl/prod/info.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/prod/info.go#newcode71
impl/prod/info.go:71: c := g.usrCtx
Only interesting change here. As I mentioned earlier, this installs the
namespace in both prodState Contexts.

https://codereview.chromium.org/2302743002/diff/1/impl/prod/raw_datastore.go
File impl/prod/raw_datastore.go (right):

https://codereview.chromium.org/2302743002/diff/1/impl/prod/raw_datastore.go#...
impl/prod/raw_datastore.go:271: // Derive a prodState with this transaction
Context.
And here is the implementation.

https://codereview.chromium.org/2302743002/diff/1/service/datastore/context.go
File service/datastore/context.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/datastore/context.g...
service/datastore/context.go:92: return KeyContext{info.FullyQualifiedAppID(c),
info.GetNamespace(c)}
I'll replace this with a "RawInterface" Get so we don't have to do two lookups
in the next patch set.

https://codereview.chromium.org/2302743002/diff/1/service/datastore/datastore.go
File service/datastore/datastore.go (left):

https://codereview.chromium.org/2302743002/diff/1/service/datastore/datastore...
service/datastore/datastore.go:39: func (d *datastoreImpl) KeyForObjErr(src
interface{}) (*Key, error) {
These all moved into key.go.

https://codereview.chromium.org/2302743002/diff/1/service/datastore/datastore...
service/datastore/datastore.go:75: func PopulateKey(obj interface{}, key *Key)
bool {
This all moved into "interface.go".

https://codereview.chromium.org/2302743002/diff/1/service/datastore/interface.go
File service/datastore/interface.go (left):

https://codereview.chromium.org/2302743002/diff/1/service/datastore/interface...
service/datastore/interface.go:22: type Interface interface {
Now gone. This is done for all service/* packages.

https://codereview.chromium.org/2302743002/diff/1/service/datastore/interface.go
File service/datastore/interface.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/datastore/interface...
service/datastore/interface.go:94: func AllocateIDs(c context.Context, ent
...interface{}) error {
Now top-level! This is basically just copy/paste with "Raw(c)" in place of "ds"
at the call site.

https://codereview.chromium.org/2302743002/diff/1/service/datastore/key.go
File service/datastore/key.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/datastore/key.go#ne...
service/datastore/key.go:56: // KeyContext is the context in which a key is
generated.
Major change here is that KeyContext was introduced, and things were rewritten
in terms of KeyContext.

https://codereview.chromium.org/2302743002/diff/1/service/datastore/transacti...
File service/datastore/transaction.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/datastore/transacti...
service/datastore/transaction.go:14: type Transaction interface{}
ATM this is just an empty interface. At some point in the future we may want to
give it IDs or actual methods (commit, cancel, etc.). ATM it's 100%
implementation specific.

https://codereview.chromium.org/2302743002/diff/1/service/info/interface.go
File service/info/interface.go (left):

https://codereview.chromium.org/2302743002/diff/1/service/info/interface.go#o...
service/info/interface.go:47: type Interface interface {
Gone.

https://codereview.chromium.org/2302743002/diff/1/service/info/interface.go
File service/info/interface.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/info/interface.go#n...
service/info/interface.go:50: 
Top-level, transplant from wrapper.go.

https://codereview.chromium.org/2302743002/diff/1/service/info/support/namesp...
File service/info/support/namespace.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/info/support/namesp...
service/info/support/namespace.go:15: var validNamespace =
regexp.MustCompile(`^[0-9A-Za-z._-]{0,100}$`)
Added this package b/c both impl/memory and impl/cloud like to validate
namespace, but this doesn't seem appropriate to pollute the top-level package
with.

https://codereview.chromium.org/2302743002/diff/1/service/memcache/interface.go
File service/memcache/interface.go (left):

https://codereview.chromium.org/2302743002/diff/1/service/memcache/interface....
service/memcache/interface.go:13: type Interface interface {
Big change here is that I axed this Interface and collapsed the *Multi calls
into vararg calls.

https://codereview.chromium.org/2302743002/diff/1/service/memcache/interface.go
File service/memcache/interface.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/memcache/interface....
service/memcache/interface.go:90: if len(items) == 1 {
(This is the "single argument, single error" logic; you'll see this everywhere.)

https://codereview.chromium.org/2302743002/diff/1/service/taskqueue/interface.go
File service/taskqueue/interface.go (left):

https://codereview.chromium.org/2302743002/diff/1/service/taskqueue/interface...
service/taskqueue/interface.go:8: type Interface interface {
Interface now gone, *Multi collapsed into vararg.

https://codereview.chromium.org/2302743002/diff/1/service/user/interface.go
File service/user/interface.go (right):

https://codereview.chromium.org/2302743002/diff/1/service/user/interface.go#n...
service/user/interface.go:32: // signed in.
Comments! Basically copy/pasted from the real user service's godoc.

[viperphish, 2016-09-01 16:16:20]

Highlevel comment on CL description:

   key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

Can we make KeyContext a method? Otherwise golint will yell at us for not
specifying field names. Or have MakeRawKey("aid", "ns", ...)?

[dnj, 2016-09-01 16:17:44]

On 2016/09/01 16:16:20, viperphish wrote:
> Highlevel comment on CL description:
> 
>    key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")
> 
> Can we make KeyContext a method? Otherwise golint will yell at us for not
> specifying field names. Or have MakeRawKey("aid", "ns", ...)?

Yeah that's a good idea, will do in next PS.

[dnj, 2016-09-01 16:23:18]

On 2016/09/01 16:17:44, dnj wrote:
> On 2016/09/01 16:16:20, viperphish wrote:
> > Highlevel comment on CL description:
> > 
> >    key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")
> > 
> > Can we make KeyContext a method? Otherwise golint will yell at us for not
> > specifying field names. Or have MakeRawKey("aid", "ns", ...)?
> 
> Yeah that's a good idea, will do in next PS.

Hmm, I'm running `golint` and `go vet` and not seeing this warning. Are you
sure?

[iannucci, 2016-09-03 00:49:36]

On 2016/09/01 at 16:23:18, dnj wrote:
> On 2016/09/01 16:17:44, dnj wrote:
> > On 2016/09/01 16:16:20, viperphish wrote:
> > > Highlevel comment on CL description:
> > > 
> > >    key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")
> > > 
> > > Can we make KeyContext a method? Otherwise golint will yell at us for not
> > > specifying field names. Or have MakeRawKey("aid", "ns", ...)?
> > 
> > Yeah that's a good idea, will do in next PS.
> 
> Hmm, I'm running `golint` and `go vet` and not seeing this warning. Are you
sure?

From another package? It usually yells at me about not using labels for struct
fields when I'm using another packages' struct?

[dnj, 2016-09-03 00:53:52]

On 2016/09/03 00:49:36, iannucci wrote:
> On 2016/09/01 at 16:23:18, dnj wrote:
> > On 2016/09/01 16:17:44, dnj wrote:
> > > On 2016/09/01 16:16:20, viperphish wrote:
> > > > Highlevel comment on CL description:
> > > > 
> > > >    key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")
> > > > 
> > > > Can we make KeyContext a method? Otherwise golint will yell at us for
not
> > > > specifying field names. Or have MakeRawKey("aid", "ns", ...)?
> > > 
> > > Yeah that's a good idea, will do in next PS.
> > 
> > Hmm, I'm running `golint` and `go vet` and not seeing this warning. Are you
> sure?
> 
> From another package? It usually yells at me about not using labels for struct
> fields when I'm using another packages' struct?

"golint ./..." and "go vet ./..." don't yell at me for that. I have seen that
error before, though. Idunno

[iannucci, 2016-09-16 01:01:14]

lgtm.

Do you think we should use caching contexts when we derive new ones? It used to
be the case that we would do lookups into the context 1/function, but now we'll
do them 1/call. It's probably a microoptimization, but all that list walking
gives me the heebie jeebies. I guess it's maybe something we should simply
suggest that the user does, e.g.

c := longcontextderivation(...)
c = cacheContext(c)

?

https://chromiumcodereview.appspot.com/2302743002/diff/1/doc.go
File doc.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/doc.go#newcode180
doc.go:180: //     // don't pass ds/mc/etc. directly, pass the context instead.
this comment no longer makes sense

https://chromiumcodereview.appspot.com/2302743002/diff/1/doc.go#newcode201
doc.go:201: // So datastore.Raw would return the full stack. In code, this would
look
Nor does this one

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
File filter/featureBreaker/gi.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
filter/featureBreaker/gi.go:39: c, err = g.RawInterface.Namespace(namespace)
I think this is wrong now; this err was intentionally aliased before so that the
inner err was scoped to the inner function. Now it leaks and then gets
overwritten later.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
File filter/featureBreaker/mc.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
filter/featureBreaker/mc.go:42: })
hm... why was this transformation necessary?

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/contex...
File filter/txnBuf/context.go (left):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/contex...
filter/txnBuf/context.go:57: func GetNoTxn(c context.Context) ds.Interface {
it's important that escaping the transaction via `datastore.WithTransaction(c,
nil)` also escapes the transaction buffer too.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn.go
File filter/txnBuf/ds_txn.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn...
filter/txnBuf/ds_txn.go:119: func (d *dsTxnBuf) CurrentTransaction()
ds.Transaction { return d.state.parentDS.CurrentTransaction() }
On 2016/09/01 at 15:25:39, dnj wrote:
> Perhaps in the future, we can synthesize transaction objects for the nested
transactions here. ATM we're just returning the underlying DS's transaction. I
was torn between implementing this right now and waiting until stronger support
for "WithTransaction" is implemented to do this, since it's not very useful
without that.

hm good point. Yes I think waiting on this is a good call, but please document
it in the mean time (in doc.go)

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn...
filter/txnBuf/ds_txn.go:122: return d.state.parentDS.WithTransaction(nil)
doesn't this need to remove transaction stuff from the context.Context as well?
Basically what the GetNoTxn function used to do?

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/txnbuf...
File filter/txnBuf/txnbuf_test.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/txnbuf...
filter/txnBuf/txnbuf_test.go:97: root           =
ds.KeyContext{"something~else", ""}.MakeKey("Parent", 1)
I expect that golint hollers about this?

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/cloud/datastore.go
File impl/cloud/datastore.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/cloud/datastore...
impl/cloud/datastore.go:120: var pmap ds.PropertyMap
On 2016/09/01 at 15:25:39, dnj wrote:
> I think this was a bug.

K

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/cloud/datastore...
impl/cloud/datastore.go:255: func (bds *boundDatastore) WithTransaction(t
ds.Transaction) context.Context {
let's just make this WithoutTransaction() (in the package value too).

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/memory/context.go
File impl/memory/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/memory/context....
impl/memory/context.go:147: func inTxn(c context.Context) bool {
On 2016/09/01 at 15:25:39, dnj wrote:
> Ended up not needing this, will delete in next patch set.

sgtm

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/memory/datastor...
File impl/memory/datastore_test.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/memory/datastor...
impl/memory/datastore_test.go:195: So(ds.CurrentTransaction(c), ShouldBeNil)
On 2016/09/01 at 15:25:40, dnj wrote:
> This test makes me happy.

:D

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/prod/context.go
File impl/prod/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/prod/context.go...
impl/prod/context.go:31: prodStateKey  key
lets change these to 

string prodStateKey = "contains a *Type"


WithValue(&prodStateKey, ...)

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/prod/context.go...
impl/prod/context.go:168: func (ps *prodState) context(c context.Context)
context.Context {
On 2016/09/01 at 15:25:40, dnj wrote:
> We *should* probably do cancellation at some point. But that'd require its own
goroutine and junk so unless GAE main library supports it (which, I guess, we'll
never know since we don't offer it the opportunity to support it), this is fine.

Hm I think this method needs a docstring or something. It does say 'context' 5
times in the function declaration, but I'm still not sure what it's supposed to
do :)

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/co...
File service/datastore/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/co...
service/datastore/context.go:92: return KeyContext{info.FullyQualifiedAppID(c),
info.GetNamespace(c)}
On 2016/09/01 at 15:25:40, dnj wrote:
> I'll replace this with a "RawInterface" Get so we don't have to do two lookups
in the next patch set.

sgtm

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/tr...
File service/datastore/transaction.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/tr...
service/datastore/transaction.go:14: type Transaction interface{}
On 2016/09/01 at 15:25:40, dnj wrote:
> ATM this is just an empty interface. At some point in the future we may want
to give it IDs or actual methods (commit, cancel, etc.). ATM it's 100%
implementation specific.

discussed offline, but I think the only thing we'd want to expose would be query
operations like "Active() bool", "AffectedGroups() []*ds.Key", etc. The explicit
commit/cancel/etc. doesn't seem to be needed for anything that we can think of
right now.

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/info/support...
File service/info/support/namespace.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/info/support...
service/info/support/namespace.go:15: var validNamespace =
regexp.MustCompile(`^[0-9A-Za-z._-]{0,100}$`)
On 2016/09/01 at 15:25:40, dnj wrote:
> Added this package b/c both impl/memory and impl/cloud like to validate
namespace, but this doesn't seem appropriate to pollute the top-level package
with.

maybe it should move to service/datastore?

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/user/interfa...
File service/user/interface.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/user/interfa...
service/user/interface.go:32: // signed in.
On 2016/09/01 at 15:25:41, dnj wrote:
> Comments! Basically copy/pasted from the real user service's godoc.

sgtm, we should probably do this in more places in service/*. We've diverged a
lot and these are helpful for Ctrl-F on godoc.org.

[dnj, 2016-09-16 05:44:43]

PS2 was "how may wrong ways can I spell 'transaction'?"

https://chromiumcodereview.appspot.com/2302743002/diff/1/doc.go
File doc.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/doc.go#newcode180
doc.go:180: //     // don't pass ds/mc/etc. directly, pass the context instead.
On 2016/09/16 01:01:13, iannucci wrote:
> this comment no longer makes sense

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/doc.go#newcode201
doc.go:201: // So datastore.Raw would return the full stack. In code, this would
look
On 2016/09/16 01:01:13, iannucci wrote:
> Nor does this one

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
File filter/featureBreaker/gi.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
filter/featureBreaker/gi.go:39: c, err = g.RawInterface.Namespace(namespace)
On 2016/09/16 01:01:13, iannucci wrote:
> I think this is wrong now; this err was intentionally aliased before so that
the
> inner err was scoped to the inner function. Now it leaks and then gets
> overwritten later.

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
File filter/featureBreaker/mc.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/featureBreake...
filter/featureBreaker/mc.go:42: })
On 2016/09/16 01:01:13, iannucci wrote:
> hm... why was this transformation necessary?

Don't remember, un-did it.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/contex...
File filter/txnBuf/context.go (left):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/contex...
filter/txnBuf/context.go:57: func GetNoTxn(c context.Context) ds.Interface {
On 2016/09/16 01:01:13, iannucci wrote:
> it's important that escaping the transaction via `datastore.WithTransaction(c,
> nil)` also escapes the transaction buffer too.

I remember thinking about this, and for some reason I concluded that it was not
a problem, but it's been a few weeks and I don't remember specifically why. I'll
add the nilling back in.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn.go
File filter/txnBuf/ds_txn.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn...
filter/txnBuf/ds_txn.go:119: func (d *dsTxnBuf) CurrentTransaction()
ds.Transaction { return d.state.parentDS.CurrentTransaction() }
On 2016/09/16 01:01:13, iannucci wrote:
> On 2016/09/01 at 15:25:39, dnj wrote:
> > Perhaps in the future, we can synthesize transaction objects for the nested
> transactions here. ATM we're just returning the underlying DS's transaction. I
> was torn between implementing this right now and waiting until stronger
support
> for "WithTransaction" is implemented to do this, since it's not very useful
> without that.
> 
> hm good point. Yes I think waiting on this is a good call, but please document
> it in the mean time (in doc.go)

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn...
filter/txnBuf/ds_txn.go:122: return d.state.parentDS.WithTransaction(nil)
On 2016/09/16 01:01:13, iannucci wrote:
> doesn't this need to remove transaction stuff from the context.Context as
well?
> Basically what the GetNoTxn function used to do?

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/ds_txn...
filter/txnBuf/ds_txn.go:126: return nil
On 2016/09/01 15:25:39, dnj wrote:
> oops, I'll make sure this returns parentDS's Testable in next patch set, don't
> know how this got nil'd.

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/txnbuf...
File filter/txnBuf/txnbuf_test.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/filter/txnBuf/txnbuf...
filter/txnBuf/txnbuf_test.go:97: root           =
ds.KeyContext{"something~else", ""}.MakeKey("Parent", 1)
On 2016/09/16 01:01:13, iannucci wrote:
> I expect that golint hollers about this?

It doesn't :/ Maybe you should apply the patch and try. It's possible my golint
is messed up or something.

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/cloud/datastore.go
File impl/cloud/datastore.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/cloud/datastore...
impl/cloud/datastore.go:255: func (bds *boundDatastore) WithTransaction(t
ds.Transaction) context.Context {
On 2016/09/16 01:01:13, iannucci wrote:
> let's just make this WithoutTransaction() (in the package value too).

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/memory/context.go
File impl/memory/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/memory/context....
impl/memory/context.go:147: func inTxn(c context.Context) bool {
On 2016/09/16 01:01:13, iannucci wrote:
> On 2016/09/01 at 15:25:39, dnj wrote:
> > Ended up not needing this, will delete in next patch set.
> 
> sgtm

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/prod/context.go
File impl/prod/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/prod/context.go...
impl/prod/context.go:31: prodStateKey  key
On 2016/09/16 01:01:13, iannucci wrote:
> lets change these to 
> 
> string prodStateKey = "contains a *Type"
> 
> 
> WithValue(&prodStateKey, ...)
> 

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/impl/prod/context.go...
impl/prod/context.go:168: func (ps *prodState) context(c context.Context)
context.Context {
On 2016/09/16 01:01:13, iannucci wrote:
> On 2016/09/01 at 15:25:40, dnj wrote:
> > We *should* probably do cancellation at some point. But that'd require its
own
> goroutine and junk so unless GAE main library supports it (which, I guess,
we'll
> never know since we don't offer it the opportunity to support it), this is
fine.
> 
> Hm I think this method needs a docstring or something. It does say 'context' 5
> times in the function declaration, but I'm still not sure what it's supposed
to
> do :)

context context context context context context context context context context
context context context context context context context context

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/co...
File service/datastore/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/co...
service/datastore/context.go:92: return KeyContext{info.FullyQualifiedAppID(c),
info.GetNamespace(c)}
On 2016/09/16 01:01:13, iannucci wrote:
> On 2016/09/01 at 15:25:40, dnj wrote:
> > I'll replace this with a "RawInterface" Get so we don't have to do two
lookups
> in the next patch set.
> 
> sgtm

Done.

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/tr...
File service/datastore/transaction.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/datastore/tr...
service/datastore/transaction.go:14: type Transaction interface{}
On 2016/09/16 01:01:13, iannucci wrote:
> On 2016/09/01 at 15:25:40, dnj wrote:
> > ATM this is just an empty interface. At some point in the future we may want
> to give it IDs or actual methods (commit, cancel, etc.). ATM it's 100%
> implementation specific.
> 
> discussed offline, but I think the only thing we'd want to expose would be
query
> operations like "Active() bool", "AffectedGroups() []*ds.Key", etc. The
explicit
> commit/cancel/etc. doesn't seem to be needed for anything that we can think of
> right now.

Added some comment.

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/info/support...
File service/info/support/namespace.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/info/support...
service/info/support/namespace.go:15: var validNamespace =
regexp.MustCompile(`^[0-9A-Za-z._-]{0,100}$`)
On 2016/09/16 01:01:14, iannucci wrote:
> On 2016/09/01 at 15:25:40, dnj wrote:
> > Added this package b/c both impl/memory and impl/cloud like to validate
> namespace, but this doesn't seem appropriate to pollute the top-level package
> with.
> 
> maybe it should move to service/datastore?

I think namespace is a concept owned by "info" service, so it makes sense to
keep it here. Also this is sort of internal-ish without being named "internal",
same role though. I don't mind leaving it here.

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/user/interfa...
File service/user/interface.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/1/service/user/interfa...
service/user/interface.go:32: // signed in.
On 2016/09/16 01:01:14, iannucci wrote:
> On 2016/09/01 at 15:25:41, dnj wrote:
> > Comments! Basically copy/pasted from the real user service's godoc.
> 
> sgtm, we should probably do this in more places in service/*. We've diverged a
> lot and these are helpful for Ctrl-F on http://godoc.org.

Acknowledged.

[dnj, 2016-09-16 05:48:08]

Oh WRT context caching: I think it is probably still a microoptimization. The
CPU's caches will almost certainly optimize this lookup if it ever becomes close
to a dominant performance issue. That said, we do have "caching context" in
luci-go and we apply it in our production installation, so I think that's
probably fine. It'll hopefully reduce the depth from 20+ to 5+-then-map in
standard usage.

[iannucci, 2016-09-16 07:39:43]

lgtm

https://chromiumcodereview.appspot.com/2302743002/diff/20001/impl/prod/contex...
File impl/prod/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/20001/impl/prod/contex...
impl/prod/context.go:172: // cancellation from "c".
that's unfortunate :/. I think it would be good to address this now (or at least
file a bug/followup CL so we don't forget about it).

It looks like Context.Done() is supposed to return nil if the context cannot be
cancelled. That would at least allow us to only burn a goroutine in the event
that the parent context is even cancellable.

https://chromiumcodereview.appspot.com/2302743002/diff/20001/service/datastor...
File service/datastore/raw_interface.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/20001/service/datastor...
service/datastore/raw_interface.go:187: // WithoutTransaction returns a derived
Context without a transaction applied.
Note that it's safe to call this even when the context doesn't currently have a
transaction (and then let's make it return self in that event)

[dnj, 2016-09-16 16:08:35]

Description was changed from

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithTransaction(c, nil), obj0)
  taskqueue.Add(datastore.WithTransaction(c, nil), task)
  prod.AEContext(datastore.WithTransaction(c, nil))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.
==========

to

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithoutTransaction(c), obj0)
  taskqueue.Add(datastore.WithoutTransaction(c), task)
  prod.AEContext(datastore.WithoutTransaction(c))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

patch from issue 2302743002 at patchset 1 (http://crrev.com/2302743002#ps1)
==========

[dnj, 2016-09-16 16:09:21]

I'll plan to land this after the single/multi patch is in and adjusted.

https://chromiumcodereview.appspot.com/2302743002/diff/20001/impl/prod/contex...
File impl/prod/context.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/20001/impl/prod/contex...
impl/prod/context.go:172: // cancellation from "c".
On 2016/09/16 07:39:43, iannucci wrote:
> that's unfortunate :/. I think it would be good to address this now (or at
least
> file a bug/followup CL so we don't forget about it).
> 
> It looks like Context.Done() is supposed to return nil if the context cannot
be
> cancelled. That would at least allow us to only burn a goroutine in the event
> that the parent context is even cancellable.

This is consistent with pre-patch behavior, so I'm reluctant to add noise from
this change to this already huge patch, especially since such a change would be
API-compatible. Filed: https://github.com/luci/gae/issues/59

https://chromiumcodereview.appspot.com/2302743002/diff/20001/service/datastor...
File service/datastore/raw_interface.go (right):

https://chromiumcodereview.appspot.com/2302743002/diff/20001/service/datastor...
service/datastore/raw_interface.go:187: // WithoutTransaction returns a derived
Context without a transaction applied.
On 2016/09/16 07:39:43, iannucci wrote:
> Note that it's safe to call this even when the context doesn't currently have
a
> transaction (and then let's make it return self in that event)

Done. I also made the user-facing implementation of this automatically return
input Context if not in a transaction.

[dnj, 2016-09-18 16:12:11]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-09-18 16:12:12]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/2302743002/#ps80001
(title: "Rebase w/ multi change.")

[commit-bot: I haz the power, 2016-09-18 16:12:20]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-18 16:22:47]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2016-09-18 16:22:49]

Try jobs failed on following builders:
  Luci-GAE Presubmit on luci.infra.try (JOB_FAILED,
https://luci-milo.appspot.com/swarming/task/3156e39c8f45f010)

[dnj, 2016-09-19 02:12:27]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-09-19 02:12:28]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://codereview.chromium.org/2302743002/#ps100001
(title: "Lightning talk licenses.")

[commit-bot: I haz the power, 2016-09-19 02:12:37]

CQ is trying da patch. Follow status at
 
https://chromium-cq-status.appspot.com/v2/patch-status/codereview.chromium.or...

[commit-bot: I haz the power, 2016-09-19 02:19:07]

Description was changed from

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithoutTransaction(c), obj0)
  taskqueue.Add(datastore.WithoutTransaction(c), task)
  prod.AEContext(datastore.WithoutTransaction(c))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

patch from issue 2302743002 at patchset 1 (http://crrev.com/2302743002#ps1)
==========

to

==========
Interface update, per-method Contexts.

This is a very large patch that fixes #53. It carries some additional
changes. This is an API change, and will break virtually all users of
any "luci/gae" packages. The API update is trivial, though.

This patch:
- Remove user-facing "Interface" interfaces, when present, in favor of
  package-level methods that each individually accept a Context parameter.
- Remove package-level "Get" methods for services in favor of directly
  using package-level methods.
- Standardize all service/... packages' primary interfaces as
  "RawInterface" instead of "Interface".
- Add a datastore Transaction tracking interface, allowing a user to get
  or set the current datastore transaction in the Context. This replaces
  GetNoTxn-style methods in favor of setting a "nil" transaction which,
  as it happens, is the only implemented part of the interface.
- Introduce "KeyContext" to "services/datastore", which retains an AppID
  and Namespace tuple and uses them as a base to generate GAE keys.
  Rewrite several internal methods to use KeyContext.
- Rename "Testable" interface method to "GetTestable" since it's
  top-level now and it would conflict with the actual interface
  definition.
- Remove the boolean return value from the info service's "GetNamespace"
  now that the empty string equals the default namespace.
- Change *Multi "service/taskqueue" and "service/memcache" APIs to use
  singular vararg versions.

The transaction interface was added here (as opposed to another patch
set) because it is complementary to the API changes required in
datastore and taskqueue services to use Contexts directly. This is somewhat
at odds with the *NoTxn-style function calls, which have consequently been
removed in favor of setting a "nil" Transaction. The datastore Transaction
interface is something that has been desired for a while, so supporting nil
Transactions should be a good starting point without over-committing (e.g.,
actually hacking transaction manipulation into "impl/prod").

Migrating code to the new-style API should be fairly straightforward:
- Calls to "service/*" methods will no longer call "Get" to get an
  Interface instance; instead, use package-level calls and pass the
  Context as an extra parameter.

  BEFORE:
  ds := datastore.Get(c)
  if err := ds.Get(obj0, obj1, ...); err != nil { ... }

  NOW:
  if err := datastore.Get(c, obj0, obj1, ...); err != nil { ... }

- The "*NoTxn"-style calls are gone, replaced by installing a nil
  datastore Transaction.

  BEFORE:
  datastore.GetNoTxn(c).Put(obj0)
  taskqueue.GetNoTxn(c).Add(task)
  prod.AEContextNoTxn(c)

  NOW:
  datastore.Put(datastore.WithoutTransaction(c), obj0)
  taskqueue.Add(datastore.WithoutTransaction(c), task)
  prod.AEContext(datastore.WithoutTransaction(c))

- Methods generating full datastore keys (manually specifying app ID and
  namespace) now must use a KeyContext instance. This isn't expected to be
  a problem, since most external packages should not be using direct key
  generation, but should, instead, be using the datastore-bound versions of
  those methods.

  BEFORE:
  key := datastore.MakeKey("aid", "ns", "kind", "stringID")

  NOW:
  key := datastore.KeyContext{"aid", "ns"}.MakeKey("kind", "stringID")

- Methods using *Multi versions of public interfaces should use singular
  versions. Additonally, to support this, the queue name and task list
  parameter positions have been swapped.

  If the error return value is being examined for anything
  more than non-nil-ness, keep an eye out for the single-argument case, which
  returns the error directly instead of wrapping it in a MultiError.

  BEFORE:
  taskqueue.Get(c).AddMulti([]*taskqueue.Task{t0, t1, t2}, "queueName")

  NOW:
  taskqueue.Add(c, "queueName", t0, t1, t2)

BUG=None
TEST=local
  - All unit tests pass locally.
  - Ran "impl/cloud" tests successfully against emulator.
  - Ran `goapp test` on "impl/prod" successfully.

patch from issue 2302743002 at patchset 1 (http://crrev.com/2302743002#ps1)

Committed:
https://github.com/luci/gae/commit/3715a27a534b6e3896e23662e229fa8caaeb8a60
==========

[commit-bot: I haz the power, 2016-09-19 02:19:09]

Committed patchset #6 (id:100001) as
https://github.com/luci/gae/commit/3715a27a534b6e3896e23662e229fa8caaeb8a60