Add "tumble" distributed transaction processing service for appengine.

Add "tumble" distributed transaction processing service for appengine.

R=dnj@chromium.org, estaab@chromium.org, maruel@chromium.org, tandrii@chromium.org, vadimsh@chromium.org
BUG=533020
Committed: https://github.com/luci/luci-go/commit/bac6f366b620c97222dc70c44de062751e82fa11

[iannucci, 2015-10-10 06:52:14]

ptal

[iannucci, 2015-10-10 17:51:53]

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/example_te...
File appengine/tumble/example_test.go (right):

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/example_te...
appengine/tumble/example_test.go:273: "successfully processed 72 mutations,
adding 0 more")
I bumped this over the batch size limit to show that it was doing correct
batching as well.

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/tumble.go
File appengine/tumble/tumble.go (right):

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/tumble.go#...
appengine/tumble/tumble.go:31: func enterTransactionInternal(c context.Context,
fromRoot *datastore.Key, f func(context.Context) ([]Mutation, error))
(map[uint64]struct{}, uint, error) {
split this up so that when we're in the process loop we can commit the new
mutations along with the originating mutation (in an inner transaction). This
should prevent the transaction from becoming too large and then failing to
commit the new mutations, causing the whole batch to fail. The original
buffering scheme was devised when we didn't have the ability to do nested
transactions.

[Vadim Sh., 2015-10-12 20:05:14]

Looks good.

Nits. Also store app version ID in mutation entities, as we discussed in IM.

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/config.go
File appengine/tumble/config.go (right):

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/config.go#...
appengine/tumble/config.go:153: r.GET(c.FireAllTasksURL(),
check X-AppEngine-Cron header is set to "true" as a precaution against
unprotected app.yaml handler.

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/config.go#...
appengine/tumble/config.go:158: r.POST(c.ProcessURLPattern(),
same here (X-AppEngine-QueueName header).

Perhaps we can write tiny middleware for httprouter to check this, sort of like
I did here but more "formal":
https://github.com/luci/luci-go/blob/master/appengine/cmd/cron/frontend/handl...

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/doc.go
File appengine/tumble/doc.go (right):

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/doc.go#new...
appengine/tumble/doc.go:88: // Optional Setup
strictly speaking, one also needs app.yaml routes with login: admin to protect
cron & task queue URL from external intruders

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/process.go
File appengine/tumble/process.go (right):

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/process.go...
appengine/tumble/process.go:66: if _, ok := r.Header["X-AppEngine-QueueName"];
!ok {
ah.. you do it here... still, consider a middleware, I expect we'll need it in
many places.

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/process.go...
appengine/tumble/process.go:92: if err != nil {
in luci-cron I use errors.IsTransient (and errors.WrapTransient) extensively for
distinguishing retriable and non-retrieable errors. Though I do a lot of URL
fetch calls to other services, that are primary source of "flaky" errors...

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/tumble.go
File appengine/tumble/tumble.go (right):

https://codereview.chromium.org/1395293002/diff/1/appengine/tumble/tumble.go#...
appengine/tumble/tumble.go:43: "$key": {datastore.MkPropertyNI(fromRoot)},
is it how hypothetical Exists(key) looks?

[iannucci, 2015-10-13 02:39:46]

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/con...
File appengine/tumble/config.go (right):

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/con...
appengine/tumble/config.go:153: r.GET(c.FireAllTasksURL(),
On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> check X-AppEngine-Cron header is set to "true" as a precaution against
unprotected app.yaml handler.

middlewaarrrereeerereerrere

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/con...
appengine/tumble/config.go:158: r.POST(c.ProcessURLPattern(),
On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> same here (X-AppEngine-QueueName header).
> 
> Perhaps we can write tiny middleware for httprouter to check this, sort of
like I did here but more "formal":
https://github.com/luci/luci-go/blob/master/appengine/cmd/cron/frontend/handl...

да

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/doc.go
File appengine/tumble/doc.go (right):

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/doc...
appengine/tumble/doc.go:88: // Optional Setup
On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> strictly speaking, one also needs app.yaml routes with login: admin to protect
cron & task queue URL from external intruders

right, I'll mention that (though the middleware helps)

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/pro...
File appengine/tumble/process.go (right):

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/pro...
appengine/tumble/process.go:92: if err != nil {
On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> in luci-cron I use errors.IsTransient (and errors.WrapTransient) extensively
for distinguishing retriable and non-retrieable errors. Though I do a lot of URL
fetch calls to other services, that are primary source of "flaky" errors...

The error here is purely advisory; cron and/or taskqueue will try it again at
some point soon anyway. Basically, everything in tumble will retry sometime (but
maybe not right away).

I suspect as we use it we'll find some particularly annoying errors which will
warrant an immediate retry.

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/tum...
File appengine/tumble/tumble.go (right):

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/tum...
appengine/tumble/tumble.go:43: "$key": {datastore.MkPropertyNI(fromRoot)},
On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> is it how hypothetical Exists(key) looks?

Yeah, it's kinda ugly. Do you think it's worth adding an Exists to
datastore.Interface?

[Vadim Sh., 2015-10-13 02:46:20]

lgtm

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/tum...
File appengine/tumble/tumble.go (right):

https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/tum...
appengine/tumble/tumble.go:43: "$key": {datastore.MkPropertyNI(fromRoot)},
On 2015/10/13 02:39:46, iannucci wrote:
> On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> > is it how hypothetical Exists(key) looks?
> 
> Yeah, it's kinda ugly. Do you think it's worth adding an Exists to
> datastore.Interface?

Yes. 

We actually needed this in isolate server (here
https://github.com/luci/luci-py/blob/master/appengine/isolate/handlers_endpoi...)
and weren't smart enough to implement over what Python SDK provides.

Or I misunderstood and it still fetches everything, just doesn't deserialize it?
Is there a way to fetch key only?

[iannucci, 2015-10-13 03:12:12]

On 2015/10/13 at 02:46:20, vadimsh wrote:
> lgtm
> 
>
https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/tum...
> File appengine/tumble/tumble.go (right):
> 
>
https://chromiumcodereview.appspot.com/1395293002/diff/1/appengine/tumble/tum...
> appengine/tumble/tumble.go:43: "$key": {datastore.MkPropertyNI(fromRoot)},
> On 2015/10/13 02:39:46, iannucci wrote:
> > On 2015/10/12 at 20:05:14, Vadim Sh. wrote:
> > > is it how hypothetical Exists(key) looks?
> > 
> > Yeah, it's kinda ugly. Do you think it's worth adding an Exists to
> > datastore.Interface?
> 
> Yes. 
> 
> We actually needed this in isolate server (here
https://github.com/luci/luci-py/blob/master/appengine/isolate/handlers_endpoi...)
and weren't smart enough to implement over what Python SDK provides.
> 
> Or I misunderstood and it still fetches everything, just doesn't deserialize
it? Is there a way to fetch key only?

It does fetch all the fields, but it doesn't do any of the struct loading magic
with them. They just get ignored. I can implement it even more smartly though in
Interface. I'll post a CL.

[iannucci, 2015-10-13 05:11:04]

The CQ bit was checked by iannucci@chromium.org

[iannucci, 2015-10-13 05:11:05]

The patchset sent to the CQ was uploaded after l-g-t-m from vadimsh@chromium.org
Link to the patchset: https://codereview.chromium.org/1395293002/#ps80001
(title: "use exists")

[commit-bot: I haz the power, 2015-10-13 05:11:14]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1395293002/80001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1395293002/80001

[commit-bot: I haz the power, 2015-10-13 05:11:54]

The CQ bit was unchecked by commit-bot@chromium.org

[commit-bot: I haz the power, 2015-10-13 05:11:55]

Try jobs failed on following builders:
  Luci-go Linux Trusty 64 Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Linux%20Trust...)
  Luci-go Mac Tester on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Mac%20Tester/...)
  Luci-go Presubmit on master.tryserver.infra (JOB_FAILED,
https://build.chromium.org/p/tryserver.infra/builders/Luci-go%20Presubmit/bui...)

[iannucci, 2015-10-13 21:40:07]

The CQ bit was checked by iannucci@chromium.org

[commit-bot: I haz the power, 2015-10-13 21:40:21]

CQ is trying da patch. Follow status at
 https://chromium-cq-status.appspot.com/patch-status/1395293002/80001
View timeline at
 https://chromium-cq-status.appspot.com/patch-timeline/1395293002/80001

[commit-bot: I haz the power, 2015-10-13 21:42:32]

Committed patchset #5 (id:80001) as
https://github.com/luci/luci-go/commit/bac6f366b620c97222dc70c44de062751e82fa11