Add meta package to datastore.

Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=

Committed: https://github.com/luci/gae/commit/30cbeef9bced0f8dfcdbd1ed27faccc3ccced9a6

[dnj, 2016-04-23 02:17:10]

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
File service/datastore/meta/eg.go (right):

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
service/datastore/meta/eg.go:15: type EntityGroupMeta struct {
(This is verbatim transplant from luci-go)

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
File service/datastore/meta/namespaces.go (right):

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
service/datastore/meta/namespaces.go:19: func Namespaces(c context.Context)
([]string, error) {
(This is also verbatim transplant from luci-go) w/ method name change.

[iannucci, 2016-04-23 19:15:21]

generally lg, though please mention luci/gae#45 in the commit message too.

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
File service/datastore/meta/namespaces.go (right):

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
service/datastore/meta/namespaces.go:19: func Namespaces(c context.Context)
([]string, error) {
On 2016/04/23 02:17:10, dnj wrote:
> (This is also verbatim transplant from luci-go) w/ method name change.

I'm not comfortable making the default API be GetAll; it will only work with a
relatively small number of namespaces. A function taking `func (ns string)
error`, with the ability to return gae.Stop or a user error to stop, and nil to
continue would be much better, IMO. It could make sense to have a AllNamespaces
which does the allocate+GetAll though.

We'll probably also need to add NamespacesWithPrefix(c, "prefix-", func()) and
NamespacesBetween(c, "start", "end", func() ...). To that end, would it be
better to have a function that just returns the Query?

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
service/datastore/meta/namespaces.go:24: if err := datastore.Get(c).GetAll(q,
&namespaceKeys); err != nil {
Note: this query will ONLY run in the default (empty) namespace. I would doc
this in the help string for now, since the go SDK is broken and won't allow
setting the namespace "" unconditionally. This query also will not function from
inside a transaction :/

[dnj, 2016-04-24 00:08:38]

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
File service/datastore/meta/namespaces.go (right):

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
service/datastore/meta/namespaces.go:19: func Namespaces(c context.Context)
([]string, error) {
On 2016/04/23 19:15:21, iannucci wrote:
> On 2016/04/23 02:17:10, dnj wrote:
> > (This is also verbatim transplant from luci-go) w/ method name change.
> 
> I'm not comfortable making the default API be GetAll; it will only work with a
> relatively small number of namespaces. A function taking `func (ns string)
> error`, with the ability to return gae.Stop or a user error to stop, and nil
to
> continue would be much better, IMO. It could make sense to have a
AllNamespaces
> which does the allocate+GetAll though.
> 
> We'll probably also need to add NamespacesWithPrefix(c, "prefix-", func()) and
> NamespacesBetween(c, "start", "end", func() ...). To that end, would it be
> better to have a function that just returns the Query?

The problem with a query is that you still have to extract the namespaces out of
the keys.

I've restructured this around a callback, adding a collector class to help with
the "all" case. Moving forward, once #49 is resolved, we can add vararg
NamespaceOptions to the call to allow begin/end cursors. We can also update the
prefix function to use a begin cursor.

https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
service/datastore/meta/namespaces.go:24: if err := datastore.Get(c).GetAll(q,
&namespaceKeys); err != nil {
On 2016/04/23 19:15:21, iannucci wrote:
> Note: this query will ONLY run in the default (empty) namespace. I would doc
> this in the help string for now, since the go SDK is broken and won't allow
> setting the namespace "" unconditionally. This query also will not function
from
> inside a transaction :/

So on live Datastore, I can execute a GQL query for namespaces within any
namespace. I haven't tested transactional.

[iannucci, 2016-04-25 08:04:54]

On 2016/04/24 00:08:38, dnj wrote:
>
https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
> File service/datastore/meta/namespaces.go (right):
> 
>
https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
> service/datastore/meta/namespaces.go:19: func Namespaces(c context.Context)
> ([]string, error) {
> On 2016/04/23 19:15:21, iannucci wrote:
> > On 2016/04/23 02:17:10, dnj wrote:
> > > (This is also verbatim transplant from luci-go) w/ method name change.
> > 
> > I'm not comfortable making the default API be GetAll; it will only work with
a
> > relatively small number of namespaces. A function taking `func (ns string)
> > error`, with the ability to return gae.Stop or a user error to stop, and nil
> to
> > continue would be much better, IMO. It could make sense to have a
> AllNamespaces
> > which does the allocate+GetAll though.
> > 
> > We'll probably also need to add NamespacesWithPrefix(c, "prefix-", func())
and
> > NamespacesBetween(c, "start", "end", func() ...). To that end, would it be
> > better to have a function that just returns the Query?
> 
> The problem with a query is that you still have to extract the namespaces out
of
> the keys.
> 
> I've restructured this around a callback, adding a collector class to help
with
> the "all" case. Moving forward, once #49 is resolved, we can add vararg
> NamespaceOptions to the call to allow begin/end cursors. We can also update
the
> prefix function to use a begin cursor.
> 
>
https://chromiumcodereview.appspot.com/1917513002/diff/1/service/datastore/me...
> service/datastore/meta/namespaces.go:24: if err := datastore.Get(c).GetAll(q,
> &namespaceKeys); err != nil {
> On 2016/04/23 19:15:21, iannucci wrote:
> > Note: this query will ONLY run in the default (empty) namespace. I would doc
> > this in the help string for now, since the go SDK is broken and won't allow
> > setting the namespace "" unconditionally. This query also will not function
> from
> > inside a transaction :/
> 
> So on live Datastore, I can execute a GQL query for namespaces within any
> namespace. I haven't tested transactional.

Hm... that's unexpected. I think one of the libraries (I think python), always
resets the namespace back to "".

[iannucci, 2016-04-25 08:05:27]

lgtm, though please mention that this fixes luci/gae#45 in the commit message.

https://chromiumcodereview.appspot.com/1917513002/diff/20001/service/datastor...
File service/datastore/meta/namespaces.go (right):

https://chromiumcodereview.appspot.com/1917513002/diff/20001/service/datastor...
service/datastore/meta/namespaces.go:26: //	- The empty namespace will have
integer ID 1. We ignore this.
Out of curiosity... why? Why not call cb("")?

https://chromiumcodereview.appspot.com/1917513002/diff/20001/service/datastor...
service/datastore/meta/namespaces.go:44: func NamespacesWithPrefix(c
context.Context, p string, cb NamespacesCallback) error {
TODO: https://github.com/luci/gae/issues/49 : Actually support this in the
underlying query.

[dnj, 2016-04-25 15:23:56]

Description was changed from

==========
Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

R=iannucci@chromium.org
BUG=
==========

to

==========
Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=
==========

[dnj, 2016-04-25 15:24:17]

https://chromiumcodereview.appspot.com/1917513002/diff/20001/service/datastor...
File service/datastore/meta/namespaces.go (right):

https://chromiumcodereview.appspot.com/1917513002/diff/20001/service/datastor...
service/datastore/meta/namespaces.go:26: //	- The empty namespace will have
integer ID 1. We ignore this.
On 2016/04/25 08:05:27, iannucci wrote:
> Out of curiosity... why? Why not call cb("")?

The thought was that someone enumerating namespaces would not find the default
namespace to be useful. I'll update the comment and have it return "".

https://chromiumcodereview.appspot.com/1917513002/diff/20001/service/datastor...
service/datastore/meta/namespaces.go:44: func NamespacesWithPrefix(c
context.Context, p string, cb NamespacesCallback) error {
On 2016/04/25 08:05:27, iannucci wrote:
> TODO: https://github.com/luci/gae/issues/49 : Actually support this in the
> underlying query.

Done.

[dnj, 2016-04-25 15:24:22]

The CQ bit was checked by dnj@chromium.org

[dnj, 2016-04-25 15:24:23]

The patchset sent to the CQ was uploaded after l-g-t-m from
iannucci@chromium.org
Link to the patchset: https://chromiumcodereview.appspot.com/1917513002/#ps40001
(title: "Updated w/ comments.")

[commit-bot: I haz the power, 2016-04-25 15:24:33]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1917513002/40001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1917513002/40001

[commit-bot: I haz the power, 2016-04-25 15:26:46]

Description was changed from

==========
Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=
==========

to

==========
Add meta package to datastore.

Add the service/datastore/meta package, which currently implements:
- __entity_group__ meta.
- Namespaces, which lists all datastore namespaces.

Fixes luci/gae#45.

R=iannucci@chromium.org
BUG=

Committed:
https://github.com/luci/gae/commit/30cbeef9bced0f8dfcdbd1ed27faccc3ccced9a6
==========

[commit-bot: I haz the power, 2016-04-25 15:26:47]

Committed patchset #3 (id:40001) as
https://github.com/luci/gae/commit/30cbeef9bced0f8dfcdbd1ed27faccc3ccced9a6