build-web-application-with-golang/en/12.4.md

12.4 Backup and recovery

In this section, we'll discuss another aspect of application management: data backup and recovery on production servers. We often encounter situations where production servers don't behave as as we expect them to. Server network outages, hard drive malfunctions, operating system crashes and other similar events can cause databases to become unavailable. The need to recover from these types of events has led to the emergence of many cold standby/hot standby tools that can help to facilitate disaster recovery remotely. In this section, we'll explain how to backup deployed applications in addition to backing up and restoring any MySQL and Redis databases you might be using.

Application Backup

In most cluster environments, web applications do not need to be backed up since they are actually copies of code from our local development environment, or from a version control system. In many cases however, we need to backup data which has been supplied by the users of our site. For instance, when sites require users to upload files, we need to be able to backup any files that have been uploaded by users to our website. The current approach for providing this kind of redundancy is to utilize so-called cloud storage, where user files and other related resources are persisted into a highly available network of servers. If our system crashes, as long as user data has been persisted onto the cloud, we can at least be sure that no data will be lost.

But what about the cases where we did not backup our data to a cloud service, or where cloud storage was not an option? How do we backup data from our web applications then? Here, we describe a tool called rysnc, which can be commonly found on unix-like systems. Rsync is a tool which can be used to synchronize files residing on different systems, and a perfect use-case for this functionality is to keep our website backed up.

Note: Cwrsync is an implementation of rsync for the Windows environment

Rsync installation

rsync 's official website: http://rsync.samba.org/can get the latest version from the above source. Of course, because rsync is a very useful software, so many Linux distributions will include it, including the.

Package Installation

# sudo apt-get install rsync ; Note: debian, ubuntu and other online installation methods ;
# yum install rsync ; Note: Fedora, Redhat, CentOS and other online installation methods ;
# rpm -ivh rsync ; Note: Fedora, Redhat, CentOS and other rpm package installation methods ;

Other Linux distributions, please use the appropriate package management methods to install. Installing source packages

tar xvf rsync-xxx.tar.gz
cd rsync-xxx
./configure - prefix =/usr; make; make install 

Note: Before using source packages compiled and installed, you have to install gcc compiler tools such as job

Rsync Configure

rsync mainly in the following three configuration files rsyncd.conf( main configuration file ), rsyncd.secrets( password file ), rsyncd.motd(rysnc server information ).

Several documents about this configuration we can refer to the official website or other websites rsync introduction, here the server and client how to open

• Services client opens:

  # /usr/bin/rsync --daemon --config=/etc/rsyncd.conf

• daemon parameter approach is to run rsync in server mode. Join the rsync boot

  echo 'rsync - daemon' >> /etc/rc.d/rc.local

Set rsync password

echo 'Your Username: Your Password' > /etc/rsyncd.secrets
chmod 600 /etc/rsyncd.secrets

• Client synchronization:

Clients can use the following command to synchronize the files on the server:

rsync -avzP --delete --password-file=rsyncd.secrets username@192.168.145.5::www/var/rsync/backup

This command, briefly explain a few points:

1. -avzP is what the reader can use the -help Show
2. -delete for example A, deleted a file, the time synchronization, B will automatically delete the corresponding files
3. -Password-file client/etc/rsyncd.secrets set password, and server to /etc/rsyncd.secrets the password the same, so cron is running, you do not need the password
4. This command in the " User Name" for the service side of the /etc/rsyncd.secrets the user name
5. This command 192.168.145.5 as the IP address of the server
6. :: www, note the two: number, www as a server configuration file /etc/rsyncd.conf in [www], meaning that according to the service on the client /etc/rsyncd.conf to synchronize them [www] paragraph, a: number, when used according to the configuration file does not directly specify the directory synchronization.

In order to synchronize real-time, you can set the crontab, keeping rsync synchronization every minute, of course, users can also set the level of importance depending on the file type of synchronization frequency.

MySQL backup

MySQL database application is still the mainstream, the current MySQL backup in two ways: hot backup and cold backup, hot backup is currently mainly used master/slave mode (master/slave) mode is mainly used for database synchronization separate read and write, but also can be used for hot backup data ), on how to configure this information, we can find a lot. Cold backup data, then that is a certain delay, but you can guarantee that the time period before data integrity, such as may sometimes be caused by misuse of our loss of data, then the master/slave model is able to retrieve lost data, but through cold backup can partially restore the data.

Cold backup shell script is generally used to achieve regular backup of the database, and then rsync synchronization through the above described non-local one server room.

The following is a scheduled backup MySQL backup script, we use the mysqldump program, this command can be exported to a database file.

#!/bin/bash
# The following configuration information, modify their own
mysql_user="USER" #MySQL backup user
mysql_password="PASSWORD" # MySQL backup user's password
mysql_host="localhost"
mysql_port="3306"
mysql_charset="utf8" # MySQL coding
backup_db_arr=("db1" "db2") # To back up the database name, separated by spaces separated by a plurality of such("db1" "db2" "db3")
backup_location=/var/www/mysql # backup data storage location, please do not end with a "/", this can keep the default, the program will automatically create a folder
expire_backup_delete="ON" # delete outdated backups is turned OFF to ON ON to OFF
expire_days=3 # default expiration time for the three days the number of days, this is only valid when the expire_backup_delete open

# We do not need to modify the following start
backup_time=`date +%Y%m%d%H%M` # define detailed time backup
backup_Ymd=`date +%Y-%m-%d` # define the backup directory date time
backup_3ago=`date-d '3 days ago '+%Y-%m-%d` # 3 days before the date
backup_dir=$backup_location/$backup_Ymd # full path to the backup folder
welcome_msg="Welcome to use MySQL backup tools!" # greeting

# Determine whether to start MYSQL, mysql does not start the backup exit
mysql_ps=`ps-ef | grep mysql | wc-l`
mysql_listen=`netstat-an | grep LISTEN | grep $mysql_port | wc-l`
if [[$mysql_ps==0]-o [$mysql_listen==0]]; then
  echo "ERROR: MySQL is not running! backup stop!"
  exit
else
  echo $welcome_msg
fi

# Connect to mysql database, can not connect to the backup exit
mysql-h $mysql_host-P $mysql_port-u $mysql_user-p $mysql_password << end
use mysql;
select host, user from user where user='root' and host='localhost';
exit
end

flag=`echo $?`
if [$flag!="0"]; then
  echo "ERROR: Can't connect mysql server! backup stop!"
  exit
else
  echo "MySQL connect ok! Please wait......"
   # Judgment does not define the backup database, if you define a backup is started, otherwise exit the backup
  if ["$backup_db_arr"!=""]; then
	   # dbnames=$(cut-d ','-f1-5 $backup_database)
	   # echo "arr is(${backup_db_arr [@]})"
	  for dbname in ${backup_db_arr [@]}
	  do
		  echo "database $dbname backup start..."
		  `mkdir -p $backup_dir`
		  `mysqldump -h $mysql_host -P $mysql_port -u $mysql_user -p $mysql_password $dbname - default-character-set=$mysql_charset | gzip> $backup_dir/$dbname -$backup_time.sql.gz`
		  flag=`echo $?`
		  if [$flag=="0"]; then
			  echo "database $dbname success backup to $backup_dir/$dbname-$backup_time.sql.gz"
		  else
			  echo "database $dbname backup fail!"
		  fi
		  
	  done
  else
	  echo "ERROR: No database to backup! backup stop"
	  exit
  fi
   # If you open the delete expired backup, delete operation
  if ["$expire_backup_delete"=="ON" -a "$backup_location"!=""]; then
	  # `find $backup_location/-type d -o -type f -ctime + $expire_days-exec rm -rf {} \;`
	  `find $backup_location/ -type d -mtime + $expire_days | xargs rm -rf`
	  echo "Expired backup data delete complete!"
  fi
  echo "All database backup success! Thank you!"
  exit
fi

Modify shell script attributes:

chmod 600 /root/mysql_backup.sh
chmod +x /root/mysql_backup.sh

Set attributes, add the command crontab, we set up regular automatic backups every day 00:00, then the backup script directory/var/www/mysql directory is set to rsync synchronization.

00 00 *** /root/mysql_backup.sh

MySQL Recovery

Earlier MySQL backup into hot backup and cold backup, hot backup main purpose is to be able to recover in real time, such as an application server hard disk failure occurred, then we can modify the database configuration file read and write into slave so that you can minimize the time interrupt service.

But sometimes we need to perform a cold backup of the SQL data recovery, as with database backup, you can import through the command:

mysql -u username -p databse < backup.sql

You can see, export and import database data is fairly simple, but if you also need to manage permissions, or some other character set, it may be a little more complicated, but these can all be done through a number of commands.

Redis backup

Redis is our most used NoSQL, its backup is also divided into two kinds: hot backup and cold backup, Redis also supports master/slave mode, so our hot backup can be achieved in this way, we can refer to the corresponding configuration the official document profiles, quite simple. Here we introduce cold backup mode: Redis will actually timed inside the memory cache data saved to the database file inside, we just backed up the corresponding file can be, is to use rsync backup to a previously described non-local machine room can be achieved.

Redis recovery

Redis Recovery divided into hot and cold backup recovery backup and recovery, hot backup and recovery purposes and methods of recovery with MySQL, as long as the modified application of the corresponding database connection.

But sometimes we need to cold backup to recover data, Redis cold backup and recovery is actually just put the saved database file copy to Redis working directory, and then start Redis on it, Redis at boot time will be automatically loaded into the database file memory, the start speed of the database to determine the size of the file.

Summary

This section describes the application of part of our backup and recovery, that is, how to do disaster recovery, including file backup, database backup. Also introduced different systems using rsync file synchronization, MySQL database and Redis database backup and recovery, hope that through the introduction of this section, you can give as a developer of products for online disaster recovery program provides a reference solution.

Links

• Directory
• Previous section: Deployment
• Next section: Summary