I strongly recommend that you first read, then understand, then understand, and only after you have the whole picture drawn in your head, start setting up.
Setting up multiple providers in RouterOS.
The main problem with setting up multiple providers is that we have to send the packet to the same gateway from which the traffic came through the provider. If we do not do this, then in most cases the operator will block traffic with the wrong source address.
If the operator gave us the ip address 1.1.1.2, then we must provide an exit through this provider with exactly this ip address, and not with any other.
Let’s not beat around the bush, but immediately start setting up.
First, let’s define the input data.
The dependence of the normal operation of the enterprise on access to the Internet today is becoming critical and interruptions in the work of the provider can lead to real losses or downtime of work processes, so many people connect backup channels of other service providers to use them for redundancy or load balancing. In this series of articles, we will consider working with several providers for Mikrotik equipment and today we will talk about various backup methods, as well as their advantages and disadvantages.
You can learn how to set up MikroTik from scratch or systematize existing knowledge in an advanced MikroTik administration course. The author of the course, a certified MikroTik trainer Dmitry Skoromnov, personally checks laboratory work and monitors the progress of each of his students. Three times more information than in the MTCNA vendor program, over 20 hours of practice and access forever.
In this article, we will not consider the basic settings of the router: Internet access, firewall rules, etc., we will assume that the reader owns these things and will pay attention only to ensuring the reservation of external channels.
This article is for RouterOS 6.x
Below is a diagram implemented in our laboratory showing the addresses and interfaces connected to two conditional providers, as well as their gateways.
For all the methods described below to work correctly, you will need to disable dynamic route addition, which can be enabled if you receive network settings from your ISP via DHCP or use PPPoE:
Important! During operations with route settings, you will briefly lose access to the Internet. Therefore, all these actions should be performed with physical access to the device.
Although you can do without it, just create an additional null route through the main gateway before setting it up and give it the smallest administrative distance. But remember that any wrong action of yours threatens to lose network access to the device from the external network.
Reservation based on route distance
The simplest, one might even say primitive way, in this case we add two routes with different administrative distances and turn on the gateway availability check for them.
To do this, go to IP — Routes and add a route to the first provider: Dst. Address — leave the default — 0.0.0.0/0, Gateway — specify the gateway of the first provider, in our case 192.168.3.1, enable the mechanism for checking the availability of the gateway — Check Gateway — ping, and set the administrative distance of the route — Distance — 1.
The same actions in the terminal:
/ip routeadd check-gateway=ping distance=1 gateway=192.168.3.1
Similarly, we add a route to the second provider, only specify the administrative distance Distance — 2. In this case, absolute numbers are not so important, if there are several routes to the same destination address, the one with the administrative distance less will work.
Or in the terminal:
/ip routeadd check-gateway=ping distance=2 gateway=192.168.233.2
After that, you will have internet and the route through the first provider will be active:
Now we simulate an accident, in our case we will simply disconnect the patch cord from the ether1 interface, after a short time the router will detect the gateway is unavailable and make the route inactive, after which the route through the second provider will automatically work and the traffic will go there.
After the ISP gateway becomes available again, the switchback will happen automatically.
Redundancy using recursive routing
The previous method has one serious drawback, it checks the availability of the provider’s gateway, but does not check the presence of the Internet behind this gateway, and you may find yourself in a situation where there is no Internet, but the switch to the backup channel did not happen. To avoid this situation, you should check the availability not of the provider’s gateway, but of external highly accessible nodes.
One of the methods that allows you to do this is called recursive routing, we discussed it in detail in a separate article, so here we will just show the configuration sequence without going into details.
Reserving channels in Mikrotik using recursive routing
First of all, let’s select two highly accessible nodes, in our case they will be 8.8.8.8 and 9.9.9.9, and write a route to each of them through the gateways of the first and second providers.
Go to IP — Routes and create a route: Dst. Address — 8.8.8.8 — node to check the first provider, Gateway — 192.168.3.1 — gateway of the first provider, Distance — 1, Scope — 10. Pay attention to the value of the area (Scope), if you leave the default value there — 30, then recursive routing will not work! Then we add a second similar route, but to node 9.9.9.9 through the gateway of the second provider.
The same is fast in the terminal:
/ip routeadd distance=1 dst-address=8.8.8.8/32 gateway=192.168.3.1 scope=10add distance=1 dst-address=9.9.9.9/32 gateway=192.168.233.2 scope=10
Now let’s add recursive routes for each of the providers with the following settings: Dst. Leave the default Address — 0.0.0.0/0Gateway — 8.8.8.8 — a highly accessible node for the first provider, Check Gateway — ping — specify the gateway availability check, Distance — 1. For the second provider, everything is the same, but Gateway — 9.9.9.9 and Distance — 2.
In the terminal, use the commands:
/ip routeadd check-gateway=ping distance=1 gateway=8.8.8.8add check-gateway=ping distance=2 gateway=9.9.9.9
We will not go into details of how it works now, there is a separate article for this, but the general meaning is that recursive routes eventually work through real provider gateways, but check the availability of not a provider gateway, but a highly accessible node for him. If the connection is lost, the route is deactivated and the next one, with a greater distance, starts working.
In the figure below, we can see that the active route switched to the backup link despite the fact that the gateway of the main provider is available:
When the operation of the first provider is restored, the reverse switching will occur automatically.
Netwatch Backup
/ip routeadd comment=ISP1 distance=1 gateway=192.168.3.1add comment=ISP2 distance=2 gateway=192.168.233.2
Then select a highly accessible host and add a route to it through the first provider:
Then go to Tools — Netwatch and add a new node there, on the Host tab, specify the address of the selected highly accessible node, change the polling interval if necessary.
Enter the following script on the Up tab:
Then on the Down tab:
Their meaning is extremely simple, if the node is available through the first provider, then we turn on the route with the ISP1 label, and disable the route with the ISP2 label, if it is not available, then we do exactly the opposite.
In the terminal, all this can be done quickly with the help of a «spell»:
To check, we block access through the first provider and make sure that the switch occurs:
Switching back will happen automatically after communication with the high-availability node is restored.
This article describes the most complete instruction on how to set up a MikroTik router for two providers.
Simultaneous connection to two Internet providers is used to organize a backup communication channel in case the connection with one of the providers is lost. In this case, the router will automatically switch to the second provider, and you can continue to work on the Internet. Connection to two providers is used in organizations where it is necessary to ensure constant access of employees to the Internet.
To provide a fault-tolerant Internet channel, you will need a router that supports multi-provider configuration. MikroTik routers are perfect for this task.
In the example, we will use the MikroTik RB951Ui-2HnD router.
A cable from the 1st provider is connected to the 1st port of the router, a cable from the 2nd provider is connected to the 2nd port, ports 3-5 and Wi-Fi are used to connect computers on the local network.
Reset default configuration
Using the Winbox program, reset the factory default configuration to configure the MikroTik router for two providers from scratch:
1. Open the menu System — Reset Configuration; 2. Check the box No Default Configuration; 3. Click Reset Configuration.
4. Next, click the Yes button.
1st WAN port setting
We configure the 1st port to dynamically receive network settings from the provider via DHCP.
Now we have received an IP address from the ISP, which is displayed in the IP Address column.
Let’s check that there is an Internet connection. Open the New Terminal menu and enter the command ping ya.ru. As you can see, there is a ping.
Setting up the 2nd WAN port
Let’s check that there is an Internet connection. Disconnect the cable of the first provider, open the New Terminal menu and enter the ping ya.ru command.
Pings are coming, so everything is set up correctly. You can connect the cable of the first provider back.
Configuring LAN ports 3-5 and Wi-Fi
LAN ports 3-5 will be combined with a Wi-Fi interface into a single local network to which computers will connect.
We combine LAN ports 3-5 into a switch
Repeat the same for the ether5 interface.
The letter S (Slave) will appear opposite the ether4 and ether5 ports.
Create a Bridge-local interface and combine LAN ports and Wi-Fi in it
To combine LAN ports 3-5 with Wi-Fi into one network, you need to create a bridge interface and add the master port of the ether3 switch and the Wi-Fi interface wlan1 to it.
Add the main port of the ether3 switch to bridge-local:
Set up the DHCP server of the local network.
In order for computers connected to the router to receive network settings automatically, configure the DHCP server:
1. Open the IP — DHCP Server menu and click the DHCP Setup button;
2. In the DHCP Server Interface list, select bridge-local and click Next;
3. Select a network for DHCP distribution in this window. Leave it unchanged and click the Next button;
4. In the next window, specify the gateway address. Click the Next button;
5. This window specifies the range of IP addresses that the DHCP server will distribute. Click the Next button;
6. Next, enter the DNS server addresses. Click the Next button;
7. Here you can set the IP address reservation time. Click the Next button;
8. DHCP server setting completed successfully. Click the OK button.
First enable Wi-Fi:
Create a password to connect to the MikroTik access point:
Configuring the Wi-Fi parameters of the MikroTik point:
In order for computers to get access to the Internet, it is necessary to configure NAT.
Add NAT rule for the first provider:
7. In the Action list, select masquaerade; 8. Click OK.
Add a NAT rule for the second provider:
5. In the Action list, select masquaerade; 6. Click OK.
Now the Internet should appear on the computers connected to the router. Check it.
Setting the switching of Internet channels between two providers
To configure the switching of Internet channels between two providers, we will use Routes and the built-in Netwatch utility.
We will have two routes through which Internet traffic can go. All traffic will go by default through the 1st provider.
If the connection with the 1st provider suddenly disappears, then we will activate the 2nd route, and all traffic will go through the 2nd provider.
As soon as the connection through the 1st provider is restored, we will deactivate the 2nd route, and all traffic will go through the 1st provider.
First, let’s delete the route through the first provider, which was created automatically, since we cannot edit its properties.
Now let’s change the route parameters of the second provider:
7. Select the route of the second provider by clicking on it with the left mouse button, and deactivate it by clicking the button with a red cross. After that, the route will turn gray.
Now add a route through the first provider:
The 3rd route will be needed so that the Google server by default pings only through the 1st provider.
7. In the Action list, select Drop; 8. Click the OK button.
Checking the Internet switching between two providers
Let’s check how switching between two providers works.
1. Open the IP — Routes menu. The route of the second provider should be gray, i.e. not active;
2. Disconnect the cable of the 1st provider from the router; 3. In Routes, the route of the second provider should be activated.
4. Check that the computers have Internet access. 5. Now we connect the cable of the first provider back. 6. In Routes, the route of the second provider should be deactivated.
7. Check that the computers have internet.
Setting up a MikroTik router for two providers works correctly. Now you can increase the ping interval of the Google server.
This completes the configuration of the Mikrotik router for two providers.
Letting everyone else out and balancing the load
The only thing left is to release everyone else on the Internet and try to distribute the load. I will say right away that we cannot distribute the load by batch, we are hindered by the fact that providers will not let traffic with other IP addresses through their network. Therefore, we will balance the connections. Yes, we will not get the total bandwidth per connection of all providers, but this is the best we can do in this case.
We will use ECMP to distribute the load among providers.
You probably already guessed that all the remaining traffic from your local hosts will go to the main table.
Let’s make an ECMP route considering that gateway 88.88.88.1 is 10Mbps and 44.44.44.1 is 5Mbps.
Packets will be sent in turn for each gateway, for every 3 packets, 2 packets will be sent to 88.88.88.1 and the remaining one packet to gateway 44.44.44.1.
Although this is a bit wrong, the fact is that the ECMP process has a separate cache table, and in order not to choose a route each time, the router for the src-address:port dst-address:port binding creates a hash and selects for of this hash is already a gateway according to the Round Robin scheme, thus the same gateway will be selected for one connection.
But not everything is so simple, every time you change the route, for example, add a gateway or the gateway dies, the ECMP cache is cleared, and every 10 minutes the cache is also cleared, hence the problem that another one can get out every 10 minutes Gateway. And again, a route will be selected for each unique hash.
The problem is that the 10-minute limit is a hard limit and we can’t remove or change it.
We will solve it using standard RouterOS tools.
Let’s think together, what do we know about the packet that got under the ECMP route? Of course, this package is in the main table.
The idea of this method is as follows. Let the router select a route using ECMP, mark such connections and let all subsequent packets in this connection bypass ECMP, thus we remove the problem for 10 minutes.
chain=postrouting — We work in a chain based on the fact that we have already chosen ECMP, you can also use the forward chain, but then the traffic that was generated by the router itself will not get through.
routing-table=main — The ECMP route is in the main table.
out-interface=ether1 — ECMP chose the gateway that lies through the first interface.
connection-state=new — We mark the connection, it doesn’t make sense to work with all packets.
action=mark-connection new-connection-mark=ECMP/ether1 — mark the connection.
Once again, we’ve let ECMP choose the direction, and we’ve actually labeled the connections based on the interface choice.
And now all subsequent packets in connections with this marking should be sent to the same interface, but not to the main table, since ECMP works there, we don’t need it anymore.
Do not forget that traffic looks from all sides, and we only need traffic from the local network, and since there can be many interfaces, it is best to filter out all traffic EXCEPT for the one that came from the provider’s interface.
It remains to configure NAT
That’s all, thank you for your attention.
Remote setting of the router, to a long journey.
Access for NAT
172.20.18.2 is an exchange server, we need to ensure that 25,80,443 ports are available through each provider.
We do NAT, as usual, without any frills.
And now let’s remember what we did with the very first rule in the mangle, we marked all connections that come through all providers! This marking will help us again.
We need to filter all traffic that comes from the local network and send it in the direction to which the connection is related to marking.
Let’s try to describe logically step by step how it will work. We will describe for one provider, for the rest according to the same scheme.
1. When the packet arrives at the IP address 88.88.88.2, we marked the connection to which the packet belongs.
2. Next in NAT, we changed the destination address, and the router will send it to the server.
3. Naturally, the server will send a packet in response, at least syn, ack, since this packet belongs to an existing connection, we will send this packet to the named routing table by the name of the connection.
NAT has already been done, the only thing left is to find the traffic that comes from the server.
Probably it makes no sense to describe, everything is very clear, at the input we marked the traffic from the providers, and when the packet got to the router input again, but the packets are already coming from the servers, we send such packets to the named routing table.
Many people will probably ask why not the ether1 interface is specified, and not to take and specify an explicitly local interface, for example ether4.
We are doing a universal configuration, if you explicitly specify any other interface, then if we have another NAT that goes to a different port, we will need to add another similar rule, but with a different interface. But we can even wrap traffic using NAT in general to external servers, for example 1.1.1.1, and then what? The only interface from where in the prerouting chain we should not expect this traffic is traffic from the provider itself and, moreover, from the one from which it came.
Everything works, now you can connect to your server using any IP address. Create a DNS A RR record and specify all three external IP addresses and always refer to this record.
mail.mycompany.ru A 88.88.88.2
mail.mycompany.ru A 99.99.99.2
mail.mycompany.ru A 44.44.44.2
Did such a thought cross your mind? After all, when the packet from the server leaves through the router, we will need to replace its src address with the address to which it went at the very beginning. But we didn’t do anything about it.
We don’t need to do anything if we remember that only the very first packet gets into the NAT rule, the one on the basis of which the new connection was created, which is why you usually see small values in the counters of the NAT rule.
NAT reverse translation will happen automatically, let’s take a look at the conntrack entry.
Let’s remove the superfluous from the output so that it doesn’t bother us.
0 SAC protocol=tcp src-address=5.19.245.3:57839 dst-address=88.88.88.2:80 reply-src-address=172.20.18.2:80 reply-dst-address=5.19.245.3:57839
And so when the packet got to the router and the connection was “just” created, the connection looked like this
0 SAC src-address=5.19.245.3:57839 dst-address=88.88.88.2:80
After the dst NAT procedure took place, the router added flags and, most importantly, to which address the IP address changes in all packets in this connection.
0 SAC d src-address=5.19.245.3:57839 dst-address=88.88.88.2:80 reply-src-address=172.20.18.2:80
When a packet left the router, the src NAT procedure works in the postrouting chains, but we do not have src NAT rules, we left the original value.
0 SAC d src-address=5.19.245.3:57839 dst-address=88.88.88.2:80 reply-src-address=172.20.18.2:80 reply-dst-address=5.19.245.3:57839
Please note that all these stages were done at the time of passing only the first package.
Now, if a packet arrives at a router and has src-address=5.19.245.3:57839 dst-address=88.88.88.2:80 in its headers, dst-address will change to 172.20.18.2:80.
If a packet arrives at a router with addresses in the headers src-address=172.20.18.2:80 dst-address=5.19.245.3:57839 src address will change to 88.88.88.2:80.
Thus, src NAT at this stage works automatically based on dst-NAT rules.
Marking connections from the provider
You probably remember, and if not, then it’s worth recalling that any packet that arrives at the router and gets into the prerouting chain and it doesn’t matter where it goes next forward or Input.
Our task is to mark with a certain brand all packets that come from providers and are intended for a certain prefix, this is necessary so that we can further determine at any stage of packet processing which provider this packet belongs to.
First, let’s set it up, and then we’ll analyze what we did.
Let’s analyze only the first rule, all the rest by analogy.
If the packet comes from the ether1 interface and the destination address is in the 88.88.88.0/29 network, and such a packet created a connection, then mark the connection with the Next-Hop/88.88.88.1 brand.
88.88.88.0/29 — why network and not IP? If you make rules for each IP address, the number of rules will grow, but the meaning will be the same.
connection-state=new — no point in trying to mark connections that are already established. As you remember, new is the package that created the connection, that is, it is only one, and in the case of all routeros packages, it will try to mark the connection each time. Which may already be marked.
Next-Hop/99.99.99.1 — why exactly? As you remember, our main task is to send traffic through the same gateway through which the traffic came, and with this name we clearly remind ourselves which gateway will be used to exit towards the provider. We could use something like ISP1, but since our first provider gives two prefixes on the same interface, we must somehow separate the traffic that comes to different prefixes.
And so this is our preparation, then we will start working directly with logic.
From now on, all traffic that comes from the provider will be marked, and then we will operate with this marking.
Exit from router
Although in fact we have already done a lot, but we have a few more cases ahead. Now we must make sure that the packets that the router itself generates (not in response) go through the desired provider and gateway.
When you “ping” from a router, build PPP* or IP tunnels from the router itself or send dns requests from the router itself. The difference with the previous case is that we sent the packet over an already established connection, but here the packet leaves the router and the connection is not marked.
It will be much easier here.
All traffic leaving the router and having an IP address matching the prefix will be sent to the named routing table.
That’s it, now you can build tunnels and everything will work for you, but only if the IP address is explicitly specified, for example in Ipsec or gre, ip tunnels and so on, where you can explicitly specify the Source address. And what to do with services where it is impossible to explicitly specify the IP address of the source? The answer to this question lies in the routing table. The pref-src parameter is responsible for which IP address will be selected if it is not explicitly set.
If you need to establish a connection from a local network with an address that is assigned on the provider’s interface, for example when using Hairpin-NAT, in the rules you must exclude BOGON networks from destination addresses.
In our example, if only for one IP address.
Once again, for understanding, we have two options for a developed plot, if the source IP address is explicitly specified and then we are not in danger and everything works great. The second option is when the IP address is not explicitly set, for example, you crush the l2tp-client, we have no way to specify the source address, but there is a destination address anyway. Then the router does the following, it finds the route in the main routing table, looks at which pref-src is specified and uses this ip address as the source address.
You remember that we created a fictitious route.
Let’s change it a little so that we define the IP address from which traffic will be generated.
The choice of IP address is up to you, choose the address that you think is less busy.
That’s all at this stage, now you can build various tunnels and establish connections not only by explicitly specifying the source address.
IP address setting
Set up IP addresses according to the settings provided by the provider.
I think it’s pointless to somehow supplement these settings.
Router Access
The first task that we need to solve is to organize the ability to manage the router through different providers. We should be able to connect to any IP address via ssh, winbox, and if you use RouterOS in the case of a VPN server. In general, organize the correct output of packets that will be sent from the router in response to incoming connections.
First we need to organize the correct routing tables. There cannot be two identical active routes in the same routing table at the same time, if the routes are the same, then the priority is selected from the distance value. In order to be able to use several identical routes in RouterOS, it is possible to create alternative (named) routing tables.
Let’s create the table data.
dst-address=0.0.0.0/0 — normal default route, nothing more.
gateway=88.88.88.1 — gateway, please note that the first provider gives us two prefixes with different gateways, and therefore we create three routes, for us, in fact, not two providers, but three.
routing-mark=Next-Hop/88.88.88.1 — this is the named routing table. It can have any name that is more convenient for you, my personal opinion, it is convenient to write down such a naming in the name of the table the gateway through which the packet will be sent if it uses this route. The connection and route marking names are just the same and can be completely different.
Did you notice what I wrote about tables and markings? The fact is that route marking is not a table, although in most cases it is. Try to answer the question why and what is the difference in the rules between the two filters in the firewall.
The fact is that routing-mark is a marking when you manually assigned a marking to a packet using the mangle, and routing-table is the routing table through which the packet left.
If the router does not find a suitable route in the named routing table, the router will continue to look for a suitable route in the default main table and such a packet can be filtered out like this routing-mark=custommark routing-table=main. We don’t need this behavior, if for some reason the route is not found, the packet should die on the router, and there is no point in sending it through another provider.
We need to redefine this behavior.
That is, the mark and the table are in fact the same thing, a packet that has a certain routing-mark and tries to find a route in the table with the same name.
The only thing left is to send the necessary traffic to the right brand.
The output chain, since we are only working with traffic that is generated in response to a connection request. We selected packets belonging only to certain connections and sent them to a named table.
Everything seems to be fine and everything should work, but it’s not, there is one small nuance in RouterOS that needs to be remembered and taken into account. The fact is that the router, before sending a packet to the output chain, looks into the routing table and looks for a suitable route for the destination address in the main table, there are only three types of services that can work outside of this behavior, these are ping, traceroute and ospf — this is required to work in VRF.
At the moment we do not have a default route in the main routing table, the question arises which gateway to specify as the default route, if we specify the first provider, then in the event of an interface failure, we will be left with nothing, we can immediately specify three gateways and specify priorities using distance, but then we must remember this setting and in case of a provider replacement or any change, edit the configuration at this moment, and then all you need to do is select the route and then redefine the direction, i.e. In fact, this is a logical case.
All interfaces have the ability to fall, there is one interface that is always in the UP state, but in RouterOS it is not possible to use it in the configuration — this is Loopback.
In RouterOS, this interface can be replaced with an empty bridge.
I often tell in courses why this is the name, and not just Loopback, perhaps in the near future support for the Loopback interface will be added to RouterOS, and if the developers lay down the same name, then problems may arise when updating the RouterOS version.
Now let’s create a route in the main table that is needed to overcome the route selection and get the packet into the output chain.
The maximum distance so that if we create a normal default route, and not a fictitious one, this route does not interfere with us.
I remind you that distance 255 is an administrative distance, which clearly indicates that this route is dead and does not participate in routing.
That’s it, at this stage you should be able to connect to the router at any IP address through any provider at the same time, you can “ping” and do all sorts of other indecencies with external IP addresses.
Issue from correct address
There are often situations when you need to hard-code that some internal host comes out only from a certain IP address.
For example, VoIP telephony, your telephony provider requires that you register only from a certain IP address, or the bank client on the chief accountant’s computer must connect from a certain address.
First, let’s set up the address lists to which we will add internal addresses of hosts that need to be sent through the desired provider and from a specific IP address.
Address 0.0.0.1, in order to create a sheet, but not bind to a real address.
The name of the sheet clearly indicates which IP address the source address needs to be changed to.
To implement this case, we need to take into account the peculiarity of the work of RouterOS.
The fact is that for traffic that passes through the router, we can change the routing table only in the prerouting chain, and of course, the outgoing interface is not known to us yet, it will be known only in the forward chain.
And we can’t just take and wrap all traffic from the internal host to a specific provider, since we also have a local network.
Let’s say if we add host 172.20.17.100 and say that all traffic is directed to the first provider, then what will happen to the traffic that goes to host 172.20.18.2? Naturally, it will be sent to the first provider.
To solve this problem, BOGON networks will help us, this is a list of networks that should never be used for publication between public ASs in the BGP protocol, in other words, these are all those networks that are used exclusively for local purposes.
Let’s create an address sheet with BOGON networks.
And then we will create rules in order to release hosts from a specific IP address.
If the packet in the source address is an address that is listed in the address list and the destination address does not fall under the BOGON lists (ie Internet traffic), send to the named table.
And the last nuance is to set up NAT.
Please note that not all traffic, but only the one that is in a certain marked route, this is necessary so that traffic between local networks does not fall under the NAT rule.
