At thethings.io we use Cassandra as our master time series database. Some weeks ago we needed to upgrade our Cassandra version to a major release version in order to improve the QoS of our architecture.
This post aims to group all steps that we use for upgrade our Cassandra cluster to a newer version. There are a lot of documentation and sometimes it is a little bit annoying to look at different pages. In case that you use Cassandra for Internet of Things (or any other use case) we hope that this post will be useful.
This section describes how to upgrade Apache Cassandra with zero downtime if your cluster has a replication factor of at least 3.
Note:
- Be careful when you run these steps on production environments.
- For command simplicity I don’t include the user and password. Ex: nodetool drain -u username -pw password
Prerequisites
- Make sure you have an up-to-date daily backup. We use nodetool snapshot tool for that but there are other options as well like OpsCenter.
- Familiarize yourself with the changes and features in this new release.
- Remove all dead nodes.
- Do not run nodetool repair or other tasks.
Please check this link for more information.
Steps for upgrade cassandra version
1. Run nodetool drain before shutting down the existing Cassandra service. This flushes all memtables from the node to SSTables on disk. Cassandra stops listening for connections from the client and other nodes. Also prevents overcounts of counter data, and speeds up restart post-upgrade.
nodetool drain -h hostname
2. Stop cassandra services.
service cassandra stop service datastax-agent stop
3. Back up your cassandra configuration files from the old installation to safe place.
4. Update java version.
apt-get update apt-get install oracle-java8-set-default java -version
5. Install the new version of Apache Cassandra.
apt-get update apt-get install cassandra=3.0.9
If you are running Cassandra from a source you should download the latest tar.gz instead of using the package manager.
6. Configure the new product. Review, compare, merge and/or update any modifications you have previously made into the new configuration files for the new version (cassandra.yml, cassandra-env.sh, etc.). Take care with the configuration options and check version restrictions.
7. Start the cassandra services.
service cassandra start service datastax-agent start
Check the logs for warnings, errors, and exceptions.
tail -f /ttio/var/logs/cassandra/system.log # or path where you set your logs.
8. Run nodetool upgradesstables
nodetool upgradesstables
9. Check the logs for warnings, errors, and exceptions.
tail -f /ttio/var/logs/cassandra/system.log # or path where you set your logs.
10. Check the status of the cluster
nodetool -h hostname status
11. Repeat theses upgrade steps on each node in the cluster.
References
- Upgrading Apache Cassandra
- Cassandra High Performance Cookbook
- Cassandra The Definitive Guide
- Mastering Apache Cassandra
[Note: this article was first published on Adria Galin’s blog]
