Cluster / Loadbalancing Services
- ✓HA Proxy Loadbalancing
- ✓MySQL Galera Cluster
- ✓Elasticsearch Cluster
- ✓Ceph Cluster
Customized cluster and load balancing solutions
Since the foundation of IP-Projects we offer not only standardized solutions from the field of hosting, but also individual cluster and load balancing solutions, which are tailored to the requirements of our customers. Together with you, we conduct a needs analysis and advise you on the construction of an individual cluster environment for the operation of your application.
For most projects the following topics are particularly relevant:
The optimal cluster for every application does not exist. The requirements for a cluster are as individual as the needs of each person and vary greatly depending on the application. In the following, we would like to present cluster variants with which we, as an Internet service provider, have been able to gather knowledge and experience in customer projects over the past few years.
Database Cluster
Classic database clusters
In the area of database clusters, there have been numerous further developments in recent years. A classic approach is to perform a separation between read and write queries to the database, since there tends to be more read queries than write queries. A good example of this behavior is an online store. There are more visitors on the store, who look at the products, than orders, which are executed via the online store.
Therefore, a MySQL master - master cluster for write queries with an arbitrary number of slave MySQL servers for read queries was set up. However, this setup was only economical above a certain amount of queries, as a larger number of database cluster nodes naturally leads to higher costs. Therefore, we do not recommend this approach to any of our customers, but prefer to build MySQL clusters using MySQL Galera replication.
MySQL Galera database cluster
The advantage of a MySQL Galera cluster in contrast to a classic master - slave cluster structure is that each of the individual MySQL servers can be addressed in read and write mode. Therefore, there is no need to separate read and write requests on the application side and each of the data nodes has a complete database table.
With a MySQL Galera Cluster we always recommend a load balancing, for example with HA-Proxy. Thereby we can flexibly distribute read and write requests to the Galera Cluster nodes without having to implement a load balancing by the customer application.
When using a Galera Cluster, it is important that the application is cluster capable. Deadlock behavior must be prevented, otherwise the Galera cluster can get out of sync. Therefore, larger writeable imports should be split into many smaller imports to prevent longer database locks.
Filecluster
A central data storage within a cluster environment is useful if your application needs information from a central data source, which must have the same data status on all servers of the cluster. In the case of an online store, for example, this could be product images or the website layout.
To ensure that all server systems in the cluster can access this data, we recommend a central storage system for more than 2 server systems, which uses a network file system to make the data available to the individual web servers. Changes in the database are thus transmitted to all server systems in real time.
Classic file cluster approach
A classical approach to solve the challenge of centralized data storage redundantly over two servers is DRBD (Distributed Replicated Block Device - network file system) as a synchronous data network between two servers. However, we no longer recommend this approach of a data cluster to any customer. The problem with two servers is always a so-called split-brain behavior. If the network connection between the servers is interrupted, both servers no longer know which of the servers is providing the current data status. You then have to intervene manually and perform a repair of the DRBD cluster, which is contrary to the idea of a highly available solution.
In recent years, we have therefore replaced all DRBD-based clusters with an object-oriented storage solution that works with an odd number of data nodes to prevent split-brain behavior.
Ceph File Cluster
The Open-Source Software Defined Storage "Ceph" allows to combine multiple storage devices - hard disks / SSDs - from different servers to an object-oriented storage. Depending on availability and data integrity requirements, a failure of a single data node or multiple data nodes can be backed up.
One of the biggest advantages with Ceph is that in small cluster installations, Ceph storage can be co-located on the compute nodes of a cluster and no additional servers are required. In addition, there are already ready-made software integrations, for example in the open-source tool Proxmox, which is frequently used by us.
We have been using Ceph very successfully for 5 years in various customer projects and have not had any data loss so far, even in clusters where several storage devices failed at the same time. Our technicians will be happy to advise you in the course of a quotation!
Loadbalancing
Load balancing is the distribution of large amounts of requests to several systems working in parallel. In addition to the firewall, a load balancer is usually one of the most important entry points into a cluster and therefore forms an essential part of the cluster. The use of load balancing makes sense if either the volume of requests can no longer be handled by one server system or if the cluster is to be flexibly expandable. A load balancer is also indispensable when setting up a high-availability computing cluster, because in addition to the load balancing function, it also checks the availability of the systems and automatically removes them from the load balancing in the event of a failure.
In the past, we relied on hardware and software load balancers from well-known manufacturers, but were often disappointed and therefore tested the open source solution HA-Proxy, successfully and still in use today.
HA-Proxy Loadbalancing
HA-Proxy is an open source load balancing software that can be installed on any common Linux operating system. In 2017, we used it for the first time in a customer project and were thrilled with the performance and the scope of administration options. In addition to standard load balancing methods such as round-robin (simple distribution of requests to a pool of servers), SSL offloading is also easily possible.
In order to ensure high availability of the load balancers, we use floating IPs within a V-LAN as load balancing IPs. Therefore, we recommend virtual IP addresses that are actively-passively distributed between two load balancers with the help of the Heartbeat service. It is a basic requirement that both load balancers have an identical configuration status so that load balancing continues to function in the same configuration in the event of a failover.
Network / Security
To ensure the security of the cluster, a firewall solution serves as the central entry point. The firewalls are always set up in active-passive mode with high availability. The firewall stack controls which traffic from the Internet is allowed to access the cluster environment or which services in the cluster and checks this traffic for network attacks.
A firewall is mandatory for every cluster setup in order to protect database, file and load balancing systems available in the network. It also provides a VPN gateway for administration access and external monitoring to monitor the server landscape.
If we have aroused your interest, please contact our technicians for a free consultation.