PubSub StreamingPull fails after 25 hours with no messages, does not recover

#### Environment details - OS: Windows & Linux - .NET version: 9.0.x - Package name and version: `Google.Cloud.PubSub.V1 3.28.0` This issue appears both on local development (Windows, Denmark) and on our GCP hosted VMs (europe-west4). The pubsub topic+subscriptions used are in europe-west4. #### Issue I'm tracking down an issue with our SubscriberClient will eventually fail with: ``` Unrecoverable error in streaming pull for client 1; aborting subscriber. ``` And: ``` Can't recover after reaching the consecutive error limit 100 on stream for client 1. The last errors were 10 errors with status DeadlineExceeded. at Google.Cloud.PubSub.V1.SubscriberClientImpl.SingleChannel.HandlePullMessageData(Task`1 moveNextTask) at Google.Cloud.PubSub.V1.SubscriberClientImpl.SingleChannel.StartAsync() at Google.Cloud.PubSub.V1.Tasks.ForwardingAwaiter.GetResult() at Google.Cloud.PubSub.V1.Tasks.Extensions.<>c__DisplayClass4_0.<<ConfigureAwaitHideErrors>g__Inner|0>d.MoveNext() ``` I've found two issues in this process, but I'll focus on the one above. I've added the other below, a quick comment on if I should follow up would be great. On to the issue at hand: Update: Forgot to mention that all code below is located in `SubscriberClientImpl.SingleChannel.cs`, in the `Google.Cloud.PubSub.V1` project. We have a subscription with spiky traffic. Sometimes we'll see thousands of messages, at other times crickets for hours or days. In another issue here, we discussed with @jskeet (#11793) where I had a loop around the SubscriberClient to handle errors which he didn't think was necessary. I agree, and I've now removed it, but occasionally we see this error above, where the client aborts permanently. It was likely always happening, but now its very apparent. In my debugging, I found that when we do a PullStream, we set the default timeout of 15 minutes. After this time, GCP PubSub (or the client, I don't actually know) stops the request, and ends on a `DeadlineExceeded` status. This DeadlineExceeded is recoverable, but it does not reset the error counter (the `_exceptions` list) - after this, the pull stream is retried. Over time, in 15 minute intervals, we gradually reach the `ConsecutiveFailureLimit` ~~`ConcurrentFailureLimit` (bad name? probably meant `ConsecutiveFailureLimit`)~~, which is a constant 100. `100 * 15 minutes` is 25 hours. Once this limit is reached, then the exception we see is thrown, and the client stops entirely. I think this limit is a mistake? Or at least, its not a mistake, but **counting DeadlineExceeded when there were 0 messages / its an expected timeout as errors is a mistake**. Its entirely expected to have these timeouts. Side note: We also have a python subscriber, in another team. I've asked the team and they see the same DeadlineExceeded we do every 15 minutes, but their client doesn't stop. I've searched the python pubsub client and didn't immediately find any 100-try limits, which indicates the Python library retries indefinitely. At the moment, as I'm writing, my client logs `Pull stream terminated with no messages, but after success assumption threshold time. Retrying with no backoff.` every two minutes, which calls `OnSuccess()`, which in turn resets the `_exceptions` list. This means that at the moment, I can never reach the 100 exceptions limit I've seen in my logs, but I think this is a temporary service issue, and once it goes away, the above can happen. #### Steps to reproduce 1. Create a client, with `ClientCount=1` for good measure. Enable debug logging. 2. Subscribe to a subscription 3. After 25 hours, the exception we had in the top, should be thrown * A lower StreamingPull timeout than 900s may expedite this #### Sidenote: The other issue, continuous reconnects As I enabled debug logging for the SubscriberClient, I got a lot of these: ``` Pull stream for client 1 terminated with no messages, but after success assumption threshold time. Retrying with no backoff. ``` I've narrowed the code down to a pull stream beginning, and then the server responding with an error after about 1-2 minutes. As this is above an internal threshold of 45s (`s_streamingPullSuccessThreshold`), its counted as a success. The comments in the code indicate its "expected" but I think it its not "expected every 2 minutes". This seems to create new connections / restart stuff, so I'm worried about resource usage (both locally and at GCP in general) wrt. lots of TCP connections. Should I make an issue on this? Seems like a server-side issue. Mike