retryable_error
Treat io::ErrorKind::Interrupt
as retryable error.
Supports retry make our users' lives easier:
Feature request: Custom retries for the s3 backend
While the reading/writing from/to s3, AWS occasionally returns errors that could be retried (at least 5xx?). Currently, in the databend, this will fail the whole execution of the statement (which may have been running for an extended time).
Most users may need this retry feature, like decompress
. Implementing it in OpenDAL will make users no bother, no backoff logic.
With the retry
feature enabled:
opendal = {version="0.5.2", features=["retry"]}
Users can configure the retry behavior easily:
let backoff = ExponentialBackoff::default(); let op = op.with_backoff(backoff);
All requests sent by op
will be automatically retried.
We will implement retry features via adding a new Layer
.
In the retry layer, we will support retrying all operations. To do our best to keep retrying read & write, we will implement RetryableReader
and RetryableWriter
, which will support retry while no actual IO happens.
Most operations are safe to retry, like list
, stat
, delete
and create
.
We will retry those operations via input backoff.
Retry IO operations are a bit complex because IO operations have side effects, especially for HTTP-based services like s3. We can't resume an operation during the reading process without sending new requests.
This proposal will do the best we can: retry the operation if no actual IO happens.
If we meet an internal error before reading/writing the user‘s buffer, it’s safe and cheap to retry it with precisely the same argument.
io::ErrorKind::Interrupt
errors.io::ErrorKind::Interrupt
kind if the error is retryable.As we return Writer
to users, there is no way for OpenDAL to get the input data again.
We need to implement retry logic for every operator function, and can't address the same problem:
Reader
/ Writer
can't be retired.None
read
and write
can't be retried during IO.None