Tag Archives: Google AppEngine

Cloud storage features wish list

This is a follow up to my previous article about DynamoDB shortcomings. Here I’m talking about solutions I’m familiar with: AWS DynamoDB, MS Azure Storage Tables, Google AppEngine Datastore. As far as I know there is no other solutions of comparable scale / maturity out there.

Transparent support for data compression.

All the storages impose some limit on item size or attribute size. So when storing large objects like texts or JSON it makes sense to compress them as one way to mitigate the limit. This looks like totaly user-land problem, however life would be much easier if one could specify content-encoding for each attribute. This way one could have let’s say data attribute and serialize it with or without compression based on the specific conditions or software version. Then one can dream of having this natively supported by official client libraries — your cloud is only as good as your client library. Just let user set  is_compressible optional flag for each attribute, and let library handle the rest, and do transparent decompression on read based on content-encoding. And as a third level support web management console should understand and support decompression too.

Transparent support for “overflow” storage

This may be familiar to those who knows how Postgres et al store data. While it is short it is stored as a single row record, once it becomes too large it “overflows” to separate table. I understand technical reasons of 400KB / 1MB limits and reasoning that “nobody will ever need more” ;)) However occasionally people do need more. In our use case 99.9% or 99.99% of data fits into the limit (with a good headroom), but then there is that 0.01–0.1% which does not, and it is a headache. Please, do transparent support for large objects. Make it slow, make it expensive, but do support large object.

Transparent support for encryption

Some people/businesses are really nervous (or conservative, or cautious in a good sense) about storing sensitive data in the cloud. DynamoDB does not even provide an encryption-at-rest guaranty. But many would want not only at-rest, but end-to-end encryption to. Similar to compression support described above, it would be great to be able to tag sensitive attributes as encrypted and provide AES key for encode and decode operation. Storage may also support storing key thumbprint together with encrypted attribute to simplify decryption and key rotation.