Categories
Cloud Management and Projects Sotfware & Developers & DevOps Tools & How-Tos

Docker mysql master-slave replication with docker

mysql percona innodb

Mysql replication; a process to enable automatic copy of database(s) from one instance of MySQL to the other. In this case we will do master to slave replication, which is the most popular way to replicate MySQL. We also can create multiple slave servers to make replication with a single master. In this post, I use docker compose to create the replication on mysql. So the code tech of the post is mysql master-slave replication with docker and docker-compose. The host OS is using ubuntu 18.04 operating system with docker and docker compose for this lab experiment. You can use windows, Centos, Macos for your lab as docker abstracts you from most of the host OS. I assume you have already installed docker and docker compose on your machine.

As a side note, Mysql is very popular as a database system. This is because it’s open source, has a lot of support from the community and BIG companies. Then also using sql commands, mysql can create, query the database and build websites, application. For instance one of the most common uses is to build wordpress (WP) websites. By following this post you should be able to make your DB system much more resilient. Finally if you want to take something out of why MySQL is important; lowers the learning curve, it’s free and also can scale to very big systems with billions of rows.

Make file docker compose

The docker compose file makes it easy for us to set some variables on the container, making the environment able to connect with the container. In this case we use docker compose to run a multicontainer environment based on definition in yml file. Let’s create a docker compose file for replication as shown below.

version: '3'
services:
  mysql-master:
    image: percona:ps-8.0
    container_name: mysql-master
    restart: unless-stopped
    env_file: ./master/.env.master
    cap_add:
      - all
    volumes:
      - ./master/data:/var/lib/mysql
      - ./master/my.cnf:/etc/my.cnf
    environment:
      - TZ:${TZ}
      - MYSQL_USER:${MYSQL_USER}
      - MYSQL_PASSWORD:${MYSQL_PASSWORD}
      - MYSQL_ROOT_PASSWORD:${MYSQL_PASSWORD}
    networks:
      default:
        aliases:
          - mysql

  mysql-slave:
    image: percona:ps-8.0
    container_name: mysql-slave
    restart: unless-stopped
    env_file: ./slave/.env.slave
    cap_add:
      - all
    volumes:
      - ./slave/data:/var/lib/mysql
      - ./slave/my.cnf:/etc/my.cnf
    environment:
      - TZ:${TZ}
      - MYSQL_USER:${MYSQL_USER}
      - MYSQL_PASSWORD:${MYSQL_PASSWORD}
      - MYSQL_ROOT_PASSWORD:${MYSQL_ROOT_PASSWORD}
    networks:
      default:
        aliases:
          - mysql

In the docker compose file above we use a different environment, in the mysql master container we use an env file with the name .env.master. To differentiate, let’s create a master and slave folder to separate the configuration for each container. Create folder using command “mkdir”.

mkdir master && mkdir slave
create folder with mkdir

We have created a folder to separate the master and slave. We will create 2 files .env master and .env.slave which we will call for the environment in the docker compose file.

touch master/.env.master && touch slave/.env.slave
touch command

Configuration env file for mysql master-slave replication with docker

This env file contains variables very important for container creation in docker compose. Env makes it easy for us to store some information in a file. We will create two env files for master and slave. We edit the env file using “vi” you can use any other text editor on linux or the operating system you are using.

vi master/.env.master
### WORKSPACE #############################################
TZ=UTC

#MYSQL_DATABASE=master
MYSQL_USER=master
[email protected]
MYSQL_PORT=3306
MYSQL_ROOT_PASSWORD=Mastermaster123

Create env.slave too for slave.

vi slave/.env.slave
### WORKSPACE #############################################
TZ=UTC

#MYSQL_DATABASE=slave
MYSQL_USER=slave
[email protected]
MYSQL_PORT=3306
MYSQL_ROOT_PASSWORD=slaveslave123

Below is an explanation for some of the variables contained in the env file :
TZ is the timezone that we will apply to the container.
MYSQL_DATABASE is the name of the database that will be created automatically.
MYSQL_USER is the user used to authenticate into the database that has been created. Create strong password.
MYSQL_PASSWORD is the password of the user that has been created.
MYSQL_PORT is the port to run the mysql service.
MYSQL_ROOT_PASSWORD is the root user credentials to access all mysql databases, create a password combination of letters and symbols to be safe.

Create my.cnf file for master.

[mysqladmin]
user=master

[mysqld]
skip_name_resolve
explicit_defaults_for_timestamp
basedir=/opt/bitnami/mysql
port=3306
tmpdir=/opt/bitnami/mysql/tmp
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
max_allowed_packet=16M
bind_address=0.0.0.0
log_error=/opt/bitnami/mysql/logs/mysqld.log
character_set_server=utf8
collation_server=utf8_general_ci
plugin_dir=/opt/bitnami/mysql/lib/plugin
server-id=1
binlog_format=ROW
log-bin

[client]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
default_character_set=UTF8
plugin_dir=/opt/bitnami/mysql/lib/plugin

[manager]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
!include /opt/bitnami/mysql/conf/bitnami/my_custom.cnf

And my.cnf for slave too.

[mysqladmin]
user=master

[mysqld]
skip_name_resolve
explicit_defaults_for_timestamp
basedir=/opt/bitnami/mysql
port=3306
tmpdir=/opt/bitnami/mysql/tmp
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
max_allowed_packet=16M
bind_address=0.0.0.0
log_error=/opt/bitnami/mysql/logs/mysqld.log
character_set_server=utf8
collation_server=utf8_general_ci
plugin_dir=/opt/bitnami/mysql/lib/plugin
server-id=2
binlog_format=ROW
log-bin

[client]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
default_character_set=UTF8
plugin_dir=/opt/bitnami/mysql/lib/plugin

[manager]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
!include /opt/bitnami/mysql/conf/bitnami/my_custom.cnf
tree docker compose

In the picture above we will explain the functions of each folder, following function :
data folder to store all the data in the container into the host.
my.cnf file is useful for making configurations on mysql.

Building container for mysql master-slave replication with docker

For the docker compose configuration we have made, let’s start to build the mysql master and slave containers. Make sure you have everything ready, then build container using the command “docker-compose up -d”.

docker-compose up -d
docker compose up

The process of building the container is running, wait for the container to be created successfully. Then after the container is successfully created, we check the process with the command “docker-compose ps”.

docker-compose ps
docker compose process

Replication mysql master-slave replication with docker

We have the container running properly now, let’s start the replication master slave configuration.

We will enter the container to configure the replication with the mysql command. Enter to container with “docker-compose exec container bash”.

docker-compose exec mysql-master bash
docker compose exec command

Let’s login using the root user on mysql.

mysql -u root -p
mysql command root

Create a user on mysql for replications.

mysql> CREATE USER 'replication'@'%' IDENTIFIED WITH mysql_native_password BY 'Slaverepl123';
mysql create user

Create user replication access so that it can be created for mysql replications.

mysql> GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%';
grant replications mysql

We check whether the replication was successful on the master.

mysql> show grants for [email protected]'%';
show grant replication

We will know the log status on master, Then we see the binary log on mysql master with below command.

mysql> SHOW MASTER STATUS\G
show master status mysql

The configuration on the master is complete, we continue to make the configuration on the slave. Login into container using “docker-compose exec” command.

docker-compose exec mysql-slave bash
docker compose exec slave

Enter into mysql slave in order to run the mysql command.

mysql -u root -p
mysql command slave

Execute this SQL command to make mysql slave join to master.

CHANGE MASTER TO
MASTER_HOST='mysql-master',
MASTER_USER='replication',
MASTER_PASSWORD='Slaverepl123',
MASTER_LOG_FILE='87e8982d00d1-bin.000004',
MASTER_LOG_POS=349;

Looks good, the command to join to master from slave has been successful.

join master slave replication

Let’s start the slave on mysql.

START SLAVE;
start slave mysql

When all the steps have been done, make sure all steps are not missed, after that we try to see the status of replication on the slave.

mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: mysql-master
                  Master_User: replication
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: 87e8982d00d1-bin.000005
          Read_Master_Log_Pos: 156
               Relay_Log_File: ba7af6f52d85-relay-bin.000002
                Relay_Log_Pos: 331
        Relay_Master_Log_File: 87e8982d00d1-bin.000005
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 156
              Relay_Log_Space: 547
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 5166800b-f068-11eb-abf5-0242ac150002
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.01 sec)

Make sure all databases are running well, we will check on mysql master.

show database master

The master database is working fine, let’s check on the slave.

mysql slave show database

Next we will test to create a database on the master, let’s create a database to test replication is working properly.

mysql> create database replicate_db;
create database replicate test

Let’s check on the slave whether the database created on the master can be created automatically on the slave.

mysql> show databases;
check slave database show

MySQL master slave replication on the docker machine is now successfully set up. You can also find my docker config on my github.

Conclusion

In this post, we have finished configuring mysql master slave. Master slave on mysql we can use in production in building applications. We must be careful in setting up replication in production. Maybe someday you will need to add slaves or mysql nodes again. In that case you need to manually configure again to add slaves. This allows you to horizontally scale the readers. You can also backup mysql into file storage, sesamedisk provides various storage for business and personal needs. We secure your data encryption and point in time recovery. If you like this article, you please add our blog to your bookmarks. We have lots of tech articles for you to study and understand. I would encourage you to also read my post about webrtc jitsi.

Hits: 9

Categories
Entrepreneurship Management and Projects Sotfware & Developers & DevOps Tools & How-Tos

What is Agile Automation Framework?

Agile Automation Testing is a methodology for implementing automated tests in agile software development. The aim of agile, automated testing is to increase the effectiveness and efficiency of the software development lifecycle while preserving quality, timeliness, and resource utilization. As a result, implementing such a procedure needs extensive coordination and cooperation across teams.

In the past few years, ever since the agile approach was introduced with its creators demanding an end to the tedious and complex reality of the old waterfall model, the effect of the same can also be seen regarding Automation Testing.

A real example of a financial service organization that developed a large-scale agile capacity to meet its aspirational automation goals. The firm began implementing agile methods in phases, beginning with its software development teams. It then implemented agile across teams to facilitate collaboration and the sharing of best practices. Finally, the firm convinced the program’s leadership to make the method the default for all automation initiatives. Since the transformation, the firm has seen project delivery times decrease by about 30% and expenses decrease by 15% to 20% across six distinct business lines.

Agile Automation
Automation in Agile

Automation in Waterfall

Within the context of the waterfall, automation testing is typically feasible when the application is consistent and reliable. The requirement requires a significant amount of time and a team of highly skilled automation expert resources, and a significant amount of set-up costs in most instances. Automation Testing’s primary goal is to decrease expenses over time and maximizes. Moreover, it guarantees that no new problems occur due to current test cases.

How to do Agile Automation as a Methodology

By definition, the agile approach emphasizes eliminating time-consuming and tedious documentation to facilitate the implementation of unique and innovative ideas and for people to communicate freely to reduce the implementation of more exploratory ideas.

As a result, we may see a conflict between the core principles of agile methods and automated testing.

Why is Agile Automation Necessary for Testing?

Automation results in improved production and cost savings. Automation has grown so embedded in agile software development so that it’s impossible to imagine one without the other.

We discussed the primary reasons why automation is necessary for agile testing methodology:

Incremental Development: 

The shortest development cycle is the primary reason that requires automation in agile testing. Agile Scrum teams have few weeks to understand the requirements, modify the code, and test the revised code. If we perform all the testing manually, the time needed would force scrum masters to exceed the time spent on development. As a result, we will have to rush the testing process hence results in reducing the overall quality.

Continual Modifications: 

Agile projects do not operate under a complete set of criteria. The requirements evolve over time and often alter in response to changing client requirements, market developments, and end-user needs. While the agile method’s most advantageous characteristic is its rapid flexibility to change, this also means that testing must be flexible enough to keep up with the changes. Moreover, Automation provides testing with the required agility and enables it to react more quickly and adaptable to changes.

Continuous assessment: 

Agility demands frequent testing. The newly introduced code is covered by the tests and the code from earlier versions. This is to verify that no previously implemented functionality is damaged due to the newly introduced feature. This puts a great deal of strain on the testers and may harm the product’s quality. By automating specific tests, testers get more time for exploratory testing.

Gain immediate insight into the quality of your code: 

Automation testing enables you to rapidly test your code using a standard set of test scripts. This provides the software tester and developer with an early indication of the code’s reliability and allows them more time to respond if the code falls short of expectations.

Testing support functions: 

Automation in testing may be used to automate test script execution against code and data setup, test result validation, and test reporting. Agile development needs continuous code releases, which can be automated. This relieves testers of tedious, repeated duties, allowing them to concentrate on testing.

Regression testing: 

Automation enables testing to be performed indefinitely, allowing for a thorough study of the code. This is very beneficial when dealing with a restricted testing window and guaranteeing code quality.

Agile Automation Tools

As discussed, not all the tests should be automated in Agile. However, the automated testing tool used by Agile teams should cover as much of the testing scope as feasible.

To do this, the team must examine the following factors while choosing an automation tool, keeping in mind the Agile methodology’s nature:

  • The tool must be compatible with all operating systems on which the program expects to operate; it must also support various devices and browsers for parallel testing.
  • The technology’s learning curve should be minimal, allowing all QA teams to involve rapidly; the solution has comprehensive reporting and integration features.
  • Apart from these criteria, the team must consider other factors while determining the most suitable automation tool for their projects.

Some important tools that cover all the above-listed requirements are:

  • Selenium
  • Kobiton
  • TestProject
  • Ranorex
  • Eggplant
  • Subject7
  • LambdaTest
  • IBM Rational Functional Tester
  • Katalon Studio

Conclusion

The growing need for Agile applications by virtually every software development team highlights automation’s competitive advantage. Although the long-term benefits of automation are unknown, QA teams must design their automation methods from the beginning.

On an individual basis, each team and company must consider additional considerations while developing an Agile automated testing plan that maximizes the methodology’s advantages.

Hits: 15