Rebuilding bad bitcask partitions in Riak

We had one of our Riak nodes that in a couple weeks time started eating up all it's disk recently.

Here's what we noticed:

1. Two of the twenty or so partitions on the node were 5 to 10 x the average size of the other partitions. The average partitions size was between 20 and 30 GB, and yet 2 of the partitions were 160GB and 210GB.
2. The logs showed that we had run out of open files even though we have the riak user set for max_open_files at 100k.  As it turns out, during the hardware maint I had started riak from a sudo -i session which gave the shell the default 1024 max_open_files setting.
3. After restarting Riak with the correct max_open_files setting, we noticed a lot of 0 byte bitcask files which we removed, as well as some invalid bitcask hint files which we cleaned up.
4. Once all the invalid bitcask files were cleaned up, we realized that any merge process against the 2 large partitions always failed, implying there were some corrupt bitcask files, or the merge process was timing out.
5. Rather than rebuild the whole node, we decided just to rebuild the specific partitions.

Here's the process we used for rebuilding the specific partitions:

- Stop Riak

# riak stop

- Move the bad partitions elsewhere for backup purposes

# mv /{riak_datadir}/riak/bitcask/{partition} /{backup_dir}/

- Start Riak

# riak start

- Wait for riak_kv process to start

# riak-admin wait-for-service riak_kv riak@{riak_node_name}

- Attach to riak and start the repair process

# riak attach
(riak@{node}) 1> Partitions =  [{part 1},{part 2},...{part n}].
(riak@{node}) 2> [riak_kv_vnode:repair(P) || P <- Partitions].

Note: to quit the riak attach shell, use cntl-D, not cntl-C  (otherwise you will stop riak)

- Check status of the repair process

# riak-admin transfers

All in all, Riak recovers quite nicely, and it wasn't terribly difficult to find out what was going on.

On a side note, Basho does a great job if you have the benefit of using their support .

Just a DBA in a developers world...

Sometimes I wonder where this whole web development thing is going...

I was asked to setup some backup and recovery tools for the BerkeleyDB Java Edition (JE).

Hmm... DB... I could probably argue that DB should be dropped from the name if you add Java edition to it. I suppose maybe it's Oracle's plan to drop BerkeleyDB for the newer improved BerkeleyDB Java Edition (JE). Maybe that's why there won't be a BDB storage engine for MySQL beyond 5.1...

So after reading way to many pages of Oracle docs, I try just using a shell script with some fancy tar work and cross my fingers. That doesn't work, so I call on the Java developers to write a little copy tool using a jdb backup class buried somewhere way down deep in those java class libraries somewhere, plug that it to my script, and now we have backups.

All this to make sure we have instant DBA recovery for one of our Java applications, that uses Hibernate, that talks to Terracotta, that uses the BerkeleyDB JE. No complexity there...

Of course, the next thing they ask is does that mean we can restore to a point in time?

Well, that all depends. If we crash Terracotta crashes, and the filesystem is intact, the BerkeleyDB JE has the ability to roll it's logs forward and restore to the point in time of the last valid transaction at the time of the crash.

That won't do us much good, however, if we need to actually restore from a backup and then roll the logs forward, because BerkeleyDB is a append only write model: Meaning that the log and the data are one in the same, so I can't get copy's of the transaction logs on a periodic basis without getting the data as well. So, unless we plan on running backups every 15 minutes or so, we better not be expecting to restore to a point in time.

So the next question is (of course):

How are we supposed to use Terracotta as a highly recoverable persistent data store then?

My Answer (these are my opinions, as a DBA, of course):

Don't, if you expect your persistent data store to act like a relational database engine.... :)

But don't worry... it's the DBA's responsibility to make sure the data is all recoverable... right?

SQLServer: Transactional Replication Issues

After seeing a few friends at the SQLPass 2007 Summit in Colorado and discussing some of the finer 'features' of pier to pier transactional replication and large database systems (my friends work on the replication team at Microsoft), I quickly ran into one of my big grips with P2P replication and SQLServer 2005.

I had noted that being able to set up P2P from a 3rd party backup tool is essential with large databases (> 100GB) because creating a backup with SQLServer 2005 just takes way too long. The problem is, however, that SQLServer must know which LSN the backup was on when it was restored to the replicated server, so that any transactions that have occurred on the main database since the backup was taken can be correctly applied in the correct order.

The only way for SQLServer to get that is by reading the header in the backup file, which can't be done via a third party backup tool (SQLLightspeed, SQLBackup, etc, etc). My friends implied that if I knew the LSN, it could be done. I'm not quite there yet.

However, it wouldn't be that hard for Microsoft to allow a third party backup tool to call a procedure letting replication know what the current LSN is, thus fixing the problem. Hmm...

Maybe one day. For now, we'll have to deal with 5 hour backups and 2 hours restores in order to setup P2P transactional replication from backup, at least until SQLServer 2008 come around and do the backups for us with it's supposedly blazing fast compression, etc.

I'm not holding my breath....