Categories
Cloud Entrepreneurship General Topics and tips Management and Projects Sotfware & Developers & DevOps Storage

Cloud and DevOps: A match made in heaven

How the relationship between Cloud and DevOps affects the industry.

Development teams look for new tools, methods, and approaches for developing and delivering the most advanced technologies every day. For many of today’s most creative innovators, the cloud offers a scalable and adaptable route to success. It provides novel security, automation, and development options. Integrating DevOps into the cloud enables businesses to compete more effectively in a complex and ever-changing industry. Rather than reinventing DevOps, effective cloud integration requires implementing and adopting best engineering practices.

Often, this involves upgrading the core infrastructure and ecosystem in preparation for cloud redeployment. In this article, we’ll look at what DevOps is and how it works, as well as the connection between cloud and DevOps, cloud-based DevOps solutions, and the benefits of cloud-based DevOps solutions.

What is DevOps?

DevOps is a technique that companies use to manage the development and release of software. DevOps is a methodology for rapidly providing services and applications. The development approach combines software development and operations into a unified process emphasizing automation and efficiency.

The goal of DevOps is to have frequent incremental improvements rather than periodic significant releases. This method enables businesses to produce higher-quality software products more smoothly and effectively.

DevOps Best Practices
DevOps Best Practices

The primary benefits of DevOps in today’s environment are as follows:

  • Effective application release automation 
  • Infrastructure automation choices
  • CI/CD
  • Rapid Delivery with Agile frameworks
  • Rapid issue resolution

How DevOps works?

Development and operations teams do not divide themselves in the DevOps environment, as they would be in traditional development setups. In specific scenarios, these two teams combine to form a single unit where engineers may work throughout the entire development lifecycle, from development to deployment and operations.

Here, you may acquire a variety of talents unrelated to any particular role. Security mechanisms and quality assurance may become increasingly engaged in the application development lifecycle in some instances. When everyone on a DevOps team focuses on security, the team becomes a DevSecOps solution.

In a DevOps environment, teams use a variety of techniques to automate formerly laborious and slow operations. They use the most advanced technology available to accelerate the development of apps. For example, in DevOps, automating tasks like code deployment and infrastructure provisioning that previously needed assistance from other teams to improve the team’s pace.

What is the relationship between DevOps and Cloud?

Both cloud computing and DevOps have plenty of other advantages in the area of the agile enterprise. Cloud technology enables businesses to access an infinite number of features and solutions at their own pace. There is no limit to the amount of capability that a company may access through the cloud. Cloud technology enables rapid feature upgrades and enhancements in any setting.

Similarly, DevOps fosters an agile workplace for all parties involved. Both systems provide distinct advantages in terms of speed and production. However, when cloud and DevOps combine, their capabilities complement one another, resulting in an even more attractive solution.

The cloud enables centralized deployments and may include built-in DevOps assistance. For instance, if DevOps teams are required to build the components of a solution in a particular manner, the cloud’s sophisticated automation capabilities may help simplify and repeat the process.

Cloud-Based DevOps Tools

Nowadays, you can operate your complete DevOps stack in the cloud through cloud-managed DevOps solutions. We have discussed two of the most popular ones: Azure DevOps and AWS DevOps.

Azure Cloud and DevOps

Azure DevOps
Microsoft Azure

Microsoft’s integrated DevOps platform is Azure DevOps (previously known as Visual Studio Team System or VSTS). It allows you to manage the whole of your DevOps cycle via a single integrated interface. While Azure DevOps Services is a cloud-based DevOps solution that you may use as a SaaS (Software-as-a-Service) application, Azure DevOps Server is a self-hosted on-premises version of the same product.

Microsoft’s DevOps solution comprises several products, each of which addresses a unique step of your process. Azure Boards enable planning and project management. The Azure Pipeline is a continuous integration and delivery tool. Azure Repos provides cloud-hosted Git repositories, Azure Test Plans is a testing toolkit, and Azure Artifacts enable the creation, management, and deployment of packages.

However, you don’t need to utilize all of the tools included in Azure DevOps Services; you may alternatively subscribe to them separately. If you need more capabilities, the Visual Studio Marketplace has over 1,000 Azure DevOps extensions, including integrations, statistics, visualizations, and crash reporting.

AWS Cloud and DevOps

AWS & DevOps

AWS DevOps is a service offered by Amazon Web Services that consists of a collection of integrated DevOps tools for managing the entire software development lifecycle. While AWS is mainly utilized in the cloud, you can also run all the tools on-premises using AWS Outposts, which enable you to deploy any AWS infrastructure component on your in-house server.

In contrast to Azure DevOps Services, a PaaS (Platform-as-a-Service) solution, AWS is an IaaS (Infrastructure-as-a-Service) solution, which means it is connected to the underlying infrastructure. While packages may be deployed from Azure DevOps Platform to another environment, such as AWS, the opposite is not feasible. You can only deploy to AWS infrastructure through AWS DevOps, such as EC2 (Elastic Compute Cloud) or S3 (Simple Storage Service).

AWS DevOps toolset includes:

  • A continuous integration/continuous delivery service called AWS CodePipeline.
  • A managed service build tool called AWS CodeBuild.
  • A deployment automation tool called AWS CodeDeploy.
  • A platform for managing DevOps projects called AWS CodeStar.

In general, AWS DevOps is probably the most refined DevOps platform for existing or prospective Amazon Web Services customers.

Which one to choose between Azure and AWS?

The primary distinction between the Azure and AWS DevOps toolsets is how they integrate with their respective platforms. Both products, for obvious reasons, combine the appearance and feel of their different cloud platform’s user interfaces. AWS DevOps is much simpler to get started with, while the Azure DevOps suite is more integrated across the various Azure DevOps toolsets and has a considerably more extensive set of integrations with the whole Azure Marketplace.

Moreover, upon choosing, it all comes down to what your employer thinks. That is, whatever job you get is of primary importance. If the sole job available targets Azure systems, then focus on completing an Azure DevOps certification. On the other hand, the industry has shifted toward AWS. Many companies and hiring managers prefer individuals with an AWS-based DevOps certification, owing to AWS’s increasing market dominance and various other fundamental options that perform better when AWS systems are used.

Apart from that, it all comes down to your work needs and which one you believe is most advantageous for you since this is the only way to choose between the two and seek a source of DevOps certifications.

Advantages of cloud computing DevOps

Cloud solutions and DevOps complement one another well in an environment designed for agility and adaptability. When DevOps and cloud computing are integrated, they can significantly improve the software development lifecycle. Businesses that use DevOps in the cloud may improve software delivery performance by an average of 81 percent.

The following are the primary advantages of cloud-based DevOps:

Automation options based on the cloud

Automation is a critical component of DevOps efficiency. Numerous cloud platforms provide sophisticated automation capabilities for DevOps operations, such as continuous development and integration. These technologies provide uniformity and efficiency while requiring less human involvement.

Centralized platform

The cloud offers a centralized framework from which businesses can manage all aspects of their production workloads, including testing, deployment, monitoring, and operation. This enables you to keep track of everything in one location. When all of your DevOps information is in one place, it’s simpler to manage compliance and security. This way, you may get even more actionable insights and business information.

Scalable infrastructure

The cloud is the most cost-effective method to guarantee that you can scale up or down any infrastructure required without spending a fortune on equipment. As a result of this scalability, DevOps is a highly effective way to roll out new features, functionality, and possibilities as your company develops. You can mix cloud computing with DevOps agility to provide limitless development possibilities for your organization.

Agile development

The cloud can offer a variety of staging and testing servers, allowing DevOps teams to continue working while waiting for servers to become available. Utilizing DevOps across the cloud environment enables teams to experiment more freely since builds may occur more often. DevOps teams can rapidly provision servers that meet their requirements.

Reliability and stability

Since cloud providers emphasize availability and stability, they can manage and maintain all aspects of the platform. Instead of worrying about these problems, IT firms can concentrate on product development, which results in better product performance, user experience, and speed to market. The key to success, in this case, is selecting a cloud provider capable of delivering the appropriate degree of uptime for your company.

Conclusions

Cloud computing, on its whole, has risen in popularity over the last several years. Businesses of all sizes can discover how a cloud environment enables them to develop at their speed. Just as cloud communications continue to increase in popularity, cloud-based development and application management procedures become more attractive as your team realizes the full advantages of cloud-based DevOps.

Later, you’ll find that you’re using various DevOps techniques to increase the efficiency of your whole workforce. When this occurs, your team’s performance and efficiency are sure to soar.  There are no limits to what your DevOps team and the cloud can achieve with the proper cloud provider guiding and supporting you.

Hits: 2

Categories
Cloud Management and Projects Sotfware & Developers & DevOps Tools & How-Tos

MySQL master-slave replication with docker

a computer that is running mysql database for replication
MySQL Master Slave DB Replication with Docker and Docker-Compose

This post “MySQL master-slave replication with docker” was updated by: Syed Umar Bukhari on August 26, 2021

MySQL replication: a process to enable automatic copying of database(s) from one instance of MySQL to the other. In this case, we will look at master-slave replication– the most popular way to replicate SQL databases, specifically MySQL. We can create multiple slave servers for replication with a single master server. In this post, we use docker compose to create the replication on MySQL. Additionally, the code part of the post uses docker and docker-compose for MySQL master-slave replication The host OS runs the Ubuntu 18.04 OS with docker and docker compose for this experiment. You can use Windows, CentOS, Mac OS, etc. for your experiment as docker abstracts you from most of the host OS. Before beginning the process, it is assumed you have already installed docker and docker compose on your machine. If you haven’t, please do so before proceeding.

On a side note, let’s see a few reasons why MySQL is extremely popular as a database system. It is because it’s open source, has a lot of support from the community and the big tech companies. Similarly, by using SQL commands, MySQL can create, run, and query the database as well as build websites, application, it is quite efficient to use for the database designers. In addition, one of its most common uses is to build a wordpress (WP) websites.

By following this post you should be able to make your own database (DB) system much more resilient. Finally, if you want to take something out of why MySQL is important then know this— it lowers the learning curve, it’s free and it can scale to huge systems with billions of rows.

Create docker compose file

The docker compose file makes it easy for us to set some variables in the container, making the environment able to connect with the container. In addition, we use docker compose to run a multi-container environment based on the definitions in a YML file. After that, let’s create a docker compose file for replication as shown below:

version: '3'
services:
  mysql-master:
    image: percona:ps-8.0
    container_name: mysql-master
    restart: unless-stopped
    env_file: ./master/.env.master
    cap_add:
      - all
    volumes:
      - ./master/data:/var/lib/mysql
      - ./master/my.cnf:/etc/my.cnf
    environment:
      - TZ:${TZ}
      - MYSQL_USER:${MYSQL_USER}
      - MYSQL_PASSWORD:${MYSQL_PASSWORD}
      - MYSQL_ROOT_PASSWORD:${MYSQL_PASSWORD}
    networks:
      default:
        aliases:
          - mysql

  mysql-slave:
    image: percona:ps-8.0
    container_name: mysql-slave
    restart: unless-stopped
    env_file: ./slave/.env.slave
    cap_add:
      - all
    volumes:
      - ./slave/data:/var/lib/mysql
      - ./slave/my.cnf:/etc/my.cnf
    environment:
      - TZ:${TZ}
      - MYSQL_USER:${MYSQL_USER}
      - MYSQL_PASSWORD:${MYSQL_PASSWORD}
      - MYSQL_ROOT_PASSWORD:${MYSQL_ROOT_PASSWORD}
    networks:
      default:
        aliases:
          - mysql

In the docker compose file above, we use a different environment; in the MySQL master container we use an ENV file with the name .env.master. To differentiate, let’s create two folders: a master and slave folder– to separate the configuration for each container. Create folders using the command “mkdir”.

mkdir master && mkdir slave
crate master and slave directory

We have created a new folder to separate the master and slave files. We will create 2 new files next: .env.master. and .env.slave to use later.

touch master/.env.master && touch slave/.env.slave
two files env master and env slave created

Configuring the ENV file for MySQL master-slave replication with docker

This env file contains variables crucial for the container’s creation in docker compose. Additionally, the file makes it easy for us to store some information. We will create two env files for master and slave respectively. We edit the env file using “vi” command; you can use any text editor on Linux or Windows– such as Visual Studio Code or Atom.

vi master/.env.master
### WORKSPACE #############################################
TZ=UTC

#MYSQL_DATABASE=master
MYSQL_USER=master
[email protected]
MYSQL_PORT=3306
MYSQL_ROOT_PASSWORD=Mastermaster123

Create env dot slave file for slave server.

vi slave/.env.slave
### WORKSPACE #############################################
TZ=UTC

#MYSQL_DATABASE=slave
MYSQL_USER=slave
[email protected]
MYSQL_PORT=3306
MYSQL_ROOT_PASSWORD=slaveslave123

Below is an explanation for some of the variables contained in the env file to help us understand what role they perform.

Let’s look at the variables and their uses:


TZ is the time zone that will apply to the container.
MYSQL_DATABASE is the name of the database that will be created by itself.
MYSQL_USER is how the use enters into the database we create
MYSQL_PASSWORD is the password of the user that has been created. Creating a strong password is safe.
MYSQL_PORT is the port that runs the MySQL server.
MYSQL_ROOT_PASSWORD is the root user info to access all MySQL databases; create a password made of letters and symbols to be safe.

Create my.cnf file for master database.

[mysqladmin]
user=master
[mysqld]
skip_name_resolve
explicit_defaults_for_timestamp
basedir=/opt/bitnami/mysql
port=3306
tmpdir=/opt/bitnami/mysql/tmp
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
max_allowed_packet=16M
bind_address=0.0.0.0
log_error=/opt/bitnami/mysql/logs/mysqld.log
character_set_server=utf8
collation_server=utf8_general_ci
plugin_dir=/opt/bitnami/mysql/lib/plugin
server-id=1
binlog_format=ROW
log-bin

[client]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
default_character_set=UTF8
plugin_dir=/opt/bitnami/mysql/lib/plugin

[manager]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
!include /opt/bitnami/mysql/conf/bitnami/my_custom.cnf

Make a my.cnf file for slave server as well.

[mysqladmin]
user=master

[mysqld]
skip_name_resolve
explicit_defaults_for_timestamp
basedir=/opt/bitnami/mysql
port=3306
tmpdir=/opt/bitnami/mysql/tmp
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
max_allowed_packet=16M
bind_address=0.0.0.0
log_error=/opt/bitnami/mysql/logs/mysqld.log
character_set_server=utf8
collation_server=utf8_general_ci
plugin_dir=/opt/bitnami/mysql/lib/plugin
server-id=2
binlog_format=ROW
log-bin

[client]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
default_character_set=UTF8
plugin_dir=/opt/bitnami/mysql/lib/plugin

[manager]
port=3306
socket=/opt/bitnami/mysql/tmp/mysql.sock
pid_file=/opt/bitnami/mysql/tmp/mysqld.pid
!include /opt/bitnami/mysql/conf/bitnami/my_custom.cnf
tree docker compose to understand the usage

From the picture above, let’s understand the usage of the master and slave folders and their files.


Data folder stores all the data files inside the container in the host.
My.cnf file makes configurations easier on MySQL.

Building a container for MySQL master-slave replication with Docker

Let’s build the MySQL master and slave containers with the docker compose config settings. Make sure you have everything ready to build container using the command “docker-compose up -d”.

docker-compose up -d
docker compose up building container for replication

Wait for the process of building the container to be done successfully. After that, check the process with the command “docker-compose ps”.

docker-compose ps
docker compose process check status mysql

Replication of MySQL master-slave with Docker

Now that the container runs properly, let’s begin the replication process.

Enter the container with “docker-compose exec container bash”. This will configure the replication with the MySQL command.

docker-compose exec mysql-master bash
docker compose exec command master slave bash replication start

Let’s login using the root user that we made above on MySQL now.

mysql -u root -p
login to the master slave replication server mysql

Create a user on MySQL for more replications.

mysql> CREATE USER 'replication'@'%' IDENTIFIED WITH mysql_native_password BY 'Slaverepl123';
creating a new user mysql

Grant user replication access to allow creation for MySQL replications.

mysql> GRANT REPLICATION SLAVE ON *.* TO 'replication'@'%';
granting user replication mysql

Now, we examine if replication executed successfully or not.

mysql> show grants for [email protected]'%';
see the log status of users mysql

We will know the log status on master from this.

After that, we see the binary log of MySQL master with the following command:

mysql> SHOW MASTER STATUS\G
see the status of master server mysql

After that, the configuration on the master is complete and we continue to make the configuration on the slave. Login to container using “docker-compose exec” command.

docker-compose exec mysql-slave bash
logging in tto the containter mysql docker

After that, enter the MySQL slave server to run the following MySQL command.

mysql -u root -p
MySQL login

Execute this SQL command to push the MySQl slave part to join the master.

CHANGE MASTER TO
MASTER_HOST='mysql-master',
MASTER_USER='replication',
MASTER_PASSWORD='Slaverepl123',
MASTER_LOG_FILE='87e8982d00d1-bin.000004',
MASTER_LOG_POS=349;

The command to join to master from slave has successfully executed.

mysql change master

Let’s start the slave on mysql.

START SLAVE;
start slave mysql

After completion of all steps, recheck your work to ensure nothing was missed. Subsequently, check the status of replication on the slave server.

mysql> SHOW SLAVE STATUS\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: mysql-master
                  Master_User: replication
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: 87e8982d00d1-bin.000005
          Read_Master_Log_Pos: 156
               Relay_Log_File: ba7af6f52d85-relay-bin.000002
                Relay_Log_Pos: 331
        Relay_Master_Log_File: 87e8982d00d1-bin.000005
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 156
              Relay_Log_Space: 547
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 5166800b-f068-11eb-abf5-0242ac150002
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.01 sec)

Make sure all databases are running smoothly, check on MySQL master.

show database master ensure mysql database db is working

If the master database is working fine, check on the slave.

mysql slave show database

After that, we will test to create a database on the master.

Therefore, let’s create a database to test if replication is working properly.

mysql> create database replicate_db;
create database replicate test

Let’s check on the slave whether the database created on the master can be created automatically on the slave.

mysql> show databases;
check slave database show

MySQL master slave replication on the docker machine is now successfully set up.

Conclusion

In this post, we have finished configuring MySQL master slave. Master slave on MySQL can be used in production and building applications. However, we must be careful when setting up replication in production. In the case you need to add slave servers, you need to manually configure it. This allows you to horizontally scale the readers. Moreover, you can backup MySQL into file storage; Sesamedisk provides storage for business and personal needs— data encryption and point in time recovery.

Additionally, you can find the docker config on github.

If you like this article, please add our blog to your bookmarks. We have lots of tech articles for you to study and understand. Lastly, I would encourage you to read this post about webrtc jitsi.

Update by: Syed Umar Bukhari on August 26, 2021

Hits: 456