When running steward process meets a 500 from service broker, doesn't re-attempt to update from broker

[davidchua]

Hi guys,

I'm not sure if this is an issue or more of a feature request.

When a service broker returns a 500 error on action:create ServiceClaim, kubectl get servicecatalogentry --namespace=steward returns an error:

error: unknown api group/version: steward.deis.io/v1, Kind=ServiceCatalogEntry

before the 500 error occurs, I am able to get my service catalog as per normal.

When I fixed the 500 error and pushed out the new service broker, is Steward expected to loop with itself to try and update with the updated service broker?

[krancour]

Currently, Steward's control loop watches the k8s event steam. So if the claim hasn't been modified since failure, it's never re-considered / no further attempt is made at satisfying the claim. I also believe the internal state machine treats failure as a terminal state, so even if the claim were ever reconsidered, there would be no transitioning to another state.

That is current behavior. I think it's somewhat evident that we should endeavor to handle this as k8s handles other failing things. A classic example would be container creation within a pod failing for some reason such as the kubelet being unable to pull the image. The kubelet neither gives up nor flails indefinitely. Instead it goes into a CrashLoopBackOff state wherein it tries again and again with the interval between attempts growing with each subsequent failure. I think the kubelet gives up eventually, but I have to double check that.

Making Steward more consistent with that behavior seems like it would correct the issue you have reported.

cc @arschles: thoughts?

[krancour]

I will add, however, that one thing that complicates this is that the reason for failure probably needs to be considered. While some failures are a clear indication that no service was provisioned, bound, etc., there are other errors where the state of the requested service cannot be inferred. In the OP, the error from the broker was a 500-- that's very non-specific. Service provisioning, for instance, may be on-going in the background. Compare this to a more specific failure, such as a 404, where Steward could safely infer that the broker didn't do anything. We could reasonably retry in the later case, but in the former, operator intervention is probably required to understand what state the requested service is actually in.

[davidchua]

Thanks @krancour.

That makes sense. Perhaps if there was a way to have some form of indication that the steward process is retrying (ie. in the 404 example above) and not retrying. like some form of 3PR that shows the state of the steward process.

[arschles]

I'm a little confused here, so please bear with me. @davidchua it looks like the ServiceCatalogEntry third party resource becomes missing, and that may be correlated to the 500 response code from the broker.

Issues with broker interaction on provision, bind, unbind, and deprovision (includes action: create and action: delete) operations, (like the 500 response code), should only show up in claims. Currently, steward should never delete or otherwise make missing a ServiceCatalogEntry.

Anything I missed here?