Thank you all for test driving ClouSE and submitting bug reports and suggestions. Keep them coming! That’s what pre-release software is for :-).
Note that automatic data upgrade from version 1.0b.1.x is not supported. To migrate data to version 1.0rc.2.0 (or later), the data needs to be exported using the
mysqldump utility, and then re-imported to a new cloud location.
Add support for automatic data upgrades starting from version 1.0.2.x (this version). Now newer versions of ClouSE are required to support old data formats. Automatic data upgrades from data version 1.0.1.x (v1.0 Beta) are not supported.
Make internal representation of
VARCHAR use 1 or 2 bytes to store string size. Previously a fixed size of 4 bytes was used.
Reduce the length of Weblob URLs for usability. The random salt is reduced from 256-bit to 128-bit.
Add support for the AUTO_INCREMENT table option in the
CREATE TABLE and
ALTER TABLE statements.
Commits 76c785, fe3882
Add configuration options to control on-disk cache for relational data. The clouse_local_data_path option can be used to specify the on-disk cache path. If the path is not specified, the cache files are created in the MySQL data directory with the clse_ prefix. The clouse_local_data_size option can be used to specify the maximum cache size. If the maximum on-disk cache size is not specified, the size is auto-calculated based on the available free disk space. To disable on-disk cache, the clouse_local_data_size option can be set to 0.
Commits 21e8fd, 34d810, 2a3cc6, 71bfe2
Implement persistent on-disk cache for relational data. The on-disk cache data persists across server restarts and machine reboots, it is crash-tolerant and self-healing. If the amount of data exceeds the maximum cache size, ClouSE tries to keep the most frequently used pages in the cache. With persistent on-disk cache in place, the amount of data reads from the cloud storage is drastically reduced. This removes the last Beta limitation: persistent on-disk cache of recently accessed relational data is not implemented.
Commits a6d57d, 48a51b, 921800
Implement automatic recovery from cloud. Previously ClouSE would go offline and wait for an explicit user action to start the recovery; automatic recovery was unsafe and could lead to data corruption in some cases. Now with automatic full history retention, automatic recovery from cloud can property recover data in all cases. The recoverlogfromcloud ClouSE admin command is still accepted (for backward compatibility) but doesn’t do anything.
Commits 942374, af1c8c
Implement point-in-time recovery. The new restoreto YYYY-MM-DD hh:mm:ss+tztz ClouSE admin command (usage example:
SET GLOBAL clouse_admin_command='restoreto 2014-01-15 17:03:34-0800';) can be used to restore to any point in time within the retention period with second granularity. The timezone offset must be explicitly specified to avoid confusion between the user’s and server’s timezones. The restoreto command is not destructive and its effects can be undone by recovering to the point in time just before the comand was issued. During point-in-time recovery operation the engine can be in the Restoring state, which is reflected in the
SHOW ENGINE CLOUSE STATUS; command.
Add clouse_retention_period configuration option. The option can be used to configure history retention period. The retention period can be configured from 0 to 744 hours (31 days). The default value is 24 hours.
Commits bd6983, acb33e, 932cb1, afe171, fa0ad7, 3640f4, 306417, 3858ab, c09af2
Implement automatic full history retention. Now the database can be restored to any point in time within the retention period. Full history retention is achieved with virtually no overhead during normal processing; the only effect is that log truncation and Weblob removal operations are deferred for the retention period.
Implement statistics collection for tables / indexes and provide the statistics to MySQL’s query optimizer. Previously ClouSE provided hardcoded statistics that did not represent the actual data and could lead to suboptimal query plans. This removes the following Beta limitation: the statistics for ClouSE tables are not implemented..
Add support for MySQL 5.5.32-34. Add support for Cloud Linux flavor of MySQL: 5.5.31-cll – 5.5.33-cll. Drop support for MySQL 5.5.19 – 5.5.23. If you need a ClouSE build for MySQL prior 5.5.24 please contact us.
Add clouse_max_weblob_download_size configuration option. The option can be used to specify the maximum size of Weblob content that can be retrieved via
SELECT statement. The maximum size can be configured from 1 KB to 1 GB, and the default size is 16 MB. Previously the maximum size was 16 MB and could not be configured by the user.
Accept broken certificate chains, as long as the certificate chain is correctly rooted in the trusted CA certificate store. The standard OpenSSL logic requires that the certificate chain reported by the server either ends with a self-signed certificate, or ends with a certificate an issuer for which could be found in the trusted CA certificate store. At the time of this release s3.amazonaws.com reported a broken certificate chain and the standard OpenSSL certificate validation failed. This is now fixed.
Previously the errors that were encountered on ClouSE startup would be logged into the MySQL’s error log only. Now in addtion to being logged into the error log, ClouSE startup errors are shown to the user who tries to access ClouSE tables.
Previously cache control directives for Weblobs were not set, which limited the effectiveness of proxy / CDN caches. Now Weblobs have the
max-age cache control directive set to allow the content to be cached for one year. The fix also affects WebStor, which now provides a way to set cache control directives.
Commits da5680, cb1c93
Add support for MySQL 5.5.31. Add support for Cloud Linux flavor of MySQL: 5.5.30-cll.
Add support for Percona Server 5.5.29-rel30.0. Percona Server is supported only on Linux. The handlerton library for Percona Server is ha_clouse-5.5.29-p30.0.so.
Add support for MySQL 5.5.30 and 5.6.10. Now ClouSE supports both MySQL 5.5.x and 5.6.x release series.
Add support for MySQL partitioning. Partition’s metadata is saved to the cloud storage and recovered on auto-discovery. Subpartitions are currently not supported. If you need ClouSE to work with subpartitions please contact us.
Add support for ClouSE on Microsoft Windows. ClouSE builds are now available both for Linux and Windows. The cloud data format is the same across all platforms (Windows, Linux, 32-bit, 64-bit).
Implement auto-discovery logic for missing tables when recovering from cloud storage. Auto-discovery has always worked for existing tables: the table’s .frm file gets transparently recovered from the cloud storage on first access. This change makes it so that the table’s .frm file is transparently removed if the table is missing in the cloud storage.
Previously under stress ClouSE could hang while accessing cloud storage. This is now fixed.
Add support for MySQL 5.5.29. Drop support for MySQL 5.5.17 – 5.5.18. If you need a ClouSE build for MySQL prior 5.5.19 please contact us.
Implement support for
DECIMAL fields in keys. This removes the following Beta limitation: DECIMAL fields cannot be used in keys.
Previously under stress ClouSE could run out of memory due to a memory leak. This is now fixed. The fix also affects WebStor.
Implement support for tables without a primary key. This removes the following Beta limitation: The table must have a primary key.
Previously NULL values were not properly stored if used in
FLOAT fields. This is now fixed.
Allow MySQL query cache to cache query results in auto-commit mode.
Previously the query optimizer could choose a full table scan even when it was much more efficient to use an index.
SELECT queries that have
WHERE clause with
IN (expr [, expr] ...) conditions were most likely to incur a performance hit. This is now fixed.
Implement support for
BLOB fields larger than 128 KB. Now
BLOB fields can have any size that is supported by MySQL. Small
BLOB fields continue to be stored in the record for efficiency (so there is virtually no penalty for using a
BLOB instead of a
VARCHAR(N)). The on-disk data format is backward compatible, and the previous versions of ClouSE would gracefully fail if they encounter BLOBs created by ClouSE v1.0b.1.5. This removes the following Beta limitation: BLOB fields are stored in the record thus are limited by the max record size (128 KB).
Make Weblob operations more resilient to intermittent cloud storage connection failures.
Commits 0a4889, 1723b4, 242b5c
WebStor: make WebStor cloud storage performance benchmark (wsperf) print correct statistics.
Commits 677878, 40e455
WebStor: clean up codebase, error messages and documentation to consistently use general cloud storage terminology instead of Amazon S3. The WebStor API now accepts 3 storage types:
WST_S3, WST_GCS, WST_WALRUS. The s3test utility is renamed into wscmd to better reflect its purpose.
Add support for Google Cloud Storage as a cloud storage provider. Now a Google Cloud Storage location can be specified in the clouse_cloud_data_url configuration option using the
gs:// protocol scheme, e.g.
Add support for MySQL 5.5.27 – 5.5.28. Drop support for MySQL 5.5.14 – 5.5.16. If you need a ClouSE build for MySQL prior 5.5.17 please contact us.
NULL support for all data types. The on-disk data format is backward compatible, and the previous versions of ClouSE would gracefully fail if they encounter tables with NULLable fields created by ClouSE v1.0b.1.4. This removes the following Beta limitation: Integer, date / time, decimal, float, bit, enum, and set fields must be declared NOT NULL.
Previously some SELECT statements (e.g. using index_merge w/intersect query execution plans) may mistakenly miss records. This is now fixed.
Add support for MySQL 5.5.22 – 5.5.25. Drop support for MySQL 5.5.8 – 5.5.13. If you need a ClouSE build for MySQL prior 5.5.14 please contact us.
Implement support for Weblob content type. Previously, all Weblobs were uploaded with the
application/octet-stream content type, which could result into Web Browsers’ showing the Weblob content incorrectly (e.g. a Web Browser could try to save an image file to disk instead of showing the image). Now the content type is set according the the Weblob name (using a name-to-content-type mapping derived from this document).
Commits 055291, 2413dc, c6fc3b, dee2c9, 683d40, 268705, ea5483, a38ba2
Implement miscelaneous performance optimizations.
Implement high-performance cloud storage access layer. The functionality is also released as WebStor – the high performance open source C++ library for Amazon S3.
Add support for MySQL 5.5.20 and 5.5.21.
Add support for specifying weblob names instead of just extensions. Ability to specify names is important for SEO, so now instead of ‘extension:jpg’ the whole name can be specified as ‘name:creative-thinking.jpg’. This is a breaking change.
Change versioning structure: split feature version into a major.minor pair, so instead of 1b.1.0 the new version is 1.0b.1.0.
Increase data page size from 64 KB to 256 KB to improve network bandwidth utilization on high-latency connections.
AUTO_INCREMENT semantics similar to InnoDB’s “interleaved” lock mode. Previously,
AUTO_INCREMENT was implemented as a global monotonically increasing counter; the existing values in the table were not taken into account. This could lead to auto-incremented values overriding existing values in the table. Now the auto-incremented values are picked to be higher than any existing value in the table.
Now weblobs must have the
LONGBLOB type. Allowing
BLOB type could lead to confusion when weblob streaming was used: the content was correctly uploaded in small chunks, but the result got truncated by the
SELECT statement on retrieval.
Previously under stress the buffer manager could excessively write pages to the cloud storage. This is now fixed.
Previously the transaction commit grouping logic was not effective in some cases. This is now fixed.
Implement checkpoint on shutdown. The checkpoint is not required for proper functioning (crash at any point is a fully supported scenario), but may help with restart performance.
Implement more aggressive checkpoint flushing. Previously checkpoints were completely lazy and didn’t flush the log to the durable storage.
Implement “sorted split” logic for B-trees. Now when a record is inserted into the end of a B-tree the page is split 90/10% instead of 50/50%. This is a standard B-tree optimization that was cut from the initial Beta release.
Previously under stress ClouSE could crash due to a double-delete problem in the lock manager. This is now fixed.
Previously lock escalation logic might not properly kick in, leading to excessive memory usage and potentially out-of-memory errors. This is now fixed.
Initial relase of ClouSE v1.0 Beta.