Topics

How to implement fast failover in keepalived by track_bfd ?

xueguanyi225038@...
 

Hi,
 
I have some question when using Keepalived V2.0.16 to implement master/backup fast failover.
In keepalived documentation (below), track_bfd seems can only increase priority when bfd session turn to UP state and can only decrease priority when bfd session turn to DOWN state.
           The common practice is to use positive weights to count a
           limited number of good services so that the server with the highest count
           becomes master. Negative weights are better to count unexpected failures
           among a high number of interfaces, as it will not saturate even with high
           number of interfaces.
However, if I want to increase priority when bfd session get down, how can I do? 
The scenario happens when a BACKUP node detect bfd session fail, BACKUP node should increase its priority and change to MASTER before Master_Down_Interval timeout.
And this is my BACKUP node configuration currently(MASTER node priority is 120), bfd can operate normally but the priority was added 40 when bfd session UP rather then DOWN. So the failover time is still in 3~4 seconds.
Or can you provide me any track_bfd configure example to implement fast failover? Thank you very much! And sorry for my poor english.
 
! Configuration File for keepalived

global_defs {
   process_names keepalived_bfd
   bfd_process_name bfdp
   notification_email {
     acassen@...
     failover@...
     sysadmin@...
   }
   notification_email_from Alexandre.Cassen@...
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

bfd_instance bfdp {
     neighbor_ip 10.0.2.8
     source_ip 10.0.2.4
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.2.210
    }
    track_bfd{
         bfdp weight 40
    }
}
 
 
 
 
 
 
Best regards,
Allen

Quentin Armitage
 

On Tue, 2019-06-25 at 18:50 -0700, xueguanyi225038@... wrote:
Hi,
 
I have some question when using Keepalived V2.0.16 to implement master/backup fast failover.
In keepalived documentation (below), track_bfd seems can only increase priority when bfd session turn to UP state and can only decrease priority when bfd session turn to DOWN state.
           The common practice is to use positive weights to count a
           limited number of good services so that the server with the highest count
           becomes master. Negative weights are better to count unexpected failures
           among a high number of interfaces, as it will not saturate even with high
           number of interfaces.
However, if I want to increase priority when bfd session get down, how can I do? 
The scenario happens when a BACKUP node detect bfd session fail, BACKUP node should increase its priority and change to MASTER before Master_Down_Interval timeout.
And this is my BACKUP node configuration currently(MASTER node priority is 120), bfd can operate normally but the priority was added 40 when bfd session UP rather then DOWN. So the failover time is still in 3~4 seconds.
Or can you provide me any track_bfd configure example to implement fast failover? Thank you very much! And sorry for my poor english.
 
! Configuration File for keepalived

global_defs {
   process_names keepalived_bfd
   bfd_process_name bfdp
   notification_email {
     acassen@...
     failover@...
     sysadmin@...
   }
   notification_email_from Alexandre.Cassen@...
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

bfd_instance bfdp {
     neighbor_ip 10.0.2.8
     source_ip 10.0.2.4
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s3
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        10.0.2.210
    }
    track_bfd{
         bfdp weight 40
    }
}
 

I cannot see any way of achieving what you want at the moment. However, what we could do is add a "reverse" option to the weight statement, for example:
track_bfd {
bfpd weight 40 reverse
}
which would subtract 40 from the priority when the bfd instance was up.

track_bfd {
bfpd weight -40 reverse
}
would add 40 to the priority when the bfd instance was down.

Would this work for you?

Quentin Armitage

xueguanyi225038@...
 

I try to use "reverse",but it's not work.


Priority still add when the bfd instance was up.


Is that any wrong? thanks for reply!

Quentin Armitage
 

On Wed, 2019-06-26 at 19:20 -0700, xueguanyi225038@... wrote:
I try to use "reverse",but it's not work.


Priority still add when the bfd instance was up.


Is that any wrong? thanks for reply!

I think you misinterpreted what I wrote. I wrote that I couldn't see a way of achieving it at the moment, but that we could potentially add a "reverse" option, i.e. the code doesn't support it at the moment but the functionality could be added.

I asked in my previous email if that would work for you, i.e. if the functionality were added, would it solve your problem.

Attached is a patch (untested) for adding the "reverse" weight option for track_bfd, which applies to the HEAD of the master branch. It would be helpful if you could apply it and see if it works.

Quentin Armitage

xueguanyi225038@...
 

Oh, I misunderstand what you mean, thanks for your detail explanation! I believe it's work after adding the functionality. 
But I think that the main purpose of BFD is to provide VRRP to implement fast failover(3~4 seconds => sub 1 second) when session is down.
Keepalived implements BFD protocol but does not support the functionality to implement fast failover, it seems to be a pity.
Any way, thanks again for your detail sharing!


best regards,
Allen

Quentin Armitage
 

On Thu, 2019-06-27 at 19:38 -0700, xueguanyi225038@... wrote:
Oh, I misunderstand what you mean, thanks for your detail explanation! I believe it's work after adding the functionality. 
But I think that the main purpose of BFD is to provide VRRP to implement fast failover(3~4 seconds => sub 1 second) when session is down.
Keepalived implements BFD protocol but does not support the functionality to implement fast failover, it seems to be a pity.
Any way, thanks again for your detail sharing!


best regards,
Allen


Allen,

With VRRP version 2, as you say the fastest failover configurable was 3~4 seconds, since the time interval between adverts is specified in seconds. When VRRP version 3 was designed, they reduced the granularity of the advert interval to units of 1 centi-second (10 milliseconds), and so the fastest failover configurable with VRRPv3 is 30~40 milliseconds. The higher the priority of the VRRP instance, the faster within the 30~40 millisecond band the take over occurs, so if you had two systems, one using priority 254 and the other priority 253, the failover times will be 30.234ms and 30.468ms respectively.

Using BFD the takeover time can be reduced even further. Once BFD detects a failure, the VRRP instance can transition to fault state, which will cause the vrrp instance to send a priority 0 advert. This speeds up the takeover by removing the need to wait for three advert intervals, and so a priority 254 vrrp instance will take over in 0.234 milli-seconds, and a priority 253 instance will take over in 0.468 milli-seconds.

So, for fast take overs, use vrrp version 3, and specify and advert interval of 0.01 seconds. Also, use high vrrp priorities, since that reduces the extra fraction of 1 advert interval. Since BFD can detect failures even faster, you can use that to make VRRP instances transition to fault state, triggering an even faster takeover.

I hope that helps,

Quentin Armitage

Alexandre Cassen
 

Hello,

Or maybe we could define a new paradigm for track_bfd inside vrrp_instance defining a 'fault_reflect' option like :

           # BFD instances we monitor, value is added to effective priority.
           # <STRING> is the name of a BFD instance
           track_bfd {
               <STRING>
               <STRING>
               <STRING> weight <INTEGER: -253..253>
<STRING> fault_reflect ... }

if 'fault_reflect' is used and if tracked bfd_instance is down then vrrp_instance will transit into FAULT state until bfd_instance is UP again.

what's your opinion ?

regs,
Alexandre

Alexandre Cassen
 

outch collision, VRRPv3 with centisec is certainly a good option. mixing VRRP & BFD to force VRRP state transition can be considered. Even if VRRP is a first op segment routing, forcing VRRP state transition to FAULT state if upper Layer3 is DOWN via BFD can be cool.

regs,
Alexandre

On 28/06/2019 08:58, Quentin Armitage wrote:
On Thu, 2019-06-27 at 19:38 -0700, xueguanyi225038@... wrote:
Oh, I misunderstand what you mean, thanks for your detail explanation! I believe it's work after adding the functionality.
But I think that the main purpose of BFD is to provide VRRP to implement fast failover(3~4 seconds => sub 1 second) when session is down.
Keepalived implements BFD protocol but does not support the functionality to implement fast failover, it seems to be a pity.
Any way, thanks again for your detail sharing!


best regards,
Allen
Allen,
With VRRP version 2, as you say the fastest failover configurable was 3~4 seconds, since the time interval between adverts is specified in seconds. When VRRP version 3 was designed, they reduced the granularity of the advert interval to units of 1 centi-second (10 milliseconds), and so the fastest failover configurable with VRRPv3 is 30~40 milliseconds. The higher the priority of the VRRP instance, the faster within the 30~40 millisecond band the take over occurs, so if you had two systems, one using priority 254 and the other priority 253, the failover times will be 30.234ms and 30.468ms respectively.
Using BFD the takeover time can be reduced even further. Once BFD detects a failure, the VRRP instance can transition to fault state, which will cause the vrrp instance to send a priority 0 advert. This speeds up the takeover by removing the need to wait for three advert intervals, and so a priority 254 vrrp instance will take over in 0.234 milli-seconds, and a priority 253 instance will take over in 0.468 milli-seconds.
So, for fast take overs, use vrrp version 3, and specify and advert interval of 0.01 seconds. Also, use high vrrp priorities, since that reduces the extra fraction of 1 advert interval. Since BFD can detect failures even faster, you can use that to make VRRP instances transition to fault state, triggering an even faster takeover.
I hope that helps,
Quentin Armitage

xueguanyi225038@...
 

Hi Quentin Armitage,

But if we use BFD to trigger VRRP instance to transit to fault state and send a priority 0 advert, the network interface is already failed.
I think that BACKUP could not receive the priority 0 advert and will also wait for three advert intervals, how to implement fast failover?

best regards,
Allen

Quentin Armitage
 

But if we use BFD to trigger VRRP instance to transit to fault state and send a priority 0 advert, the network interface is already failed.

That depends on whether you use the same or a different network interface for BFD.

how to implement fast failover?

How fast do you want the failover to be? As I wrote earlier, a fraction over 30ms failover time can be achieved with VRRPv3; is it faster than that that you need?

Quentin Armitage

On Mon, 2019-07-01 at 00:57 -0700, xueguanyi225038@... wrote:
Hi Quentin Armitage,

But if we use BFD to trigger VRRP instance to transit to fault state and send a priority 0 advert, the network interface is already failed.
I think that BACKUP could not receive the priority 0 advert and will also wait for three advert intervals, how to implement fast failover?

best regards,
Allen

xueguanyi225038@...
 

Hi Quentin Armitage,

That depends on whether you use the same or a different network interface for BFD.
If we use the same network interface, is there any way to shorten failover time?

How fast do you want the failover to be? As I wrote earlier, a fraction over 30ms failover time can be achieved with VRRPv3; is it faster than that that you need?
I want to make failover time in sub 1 second both in VRRPv2 or v3, so failover time is already fast enough in VRRP v3 without BFD.
In VRRP v2, I want to use BFD to shorten failover time when MASTER is dead, or the network interface between MASTER and BACKUP is failed.


best regards,
Allen