Fall 2000: CS 771/871 Operating Systems

[ Home | Class Roster | Syllabus | Status | Glossary | Search | Course Notes]

Lecture 7 - Agreement Protocols

• In coordinating distributed processes, it is often necessary to reach agreement about some part of the global state.

• In the absence of failures, reaching agreement is relatively straightfoward, even if initially the processes disagree about some value in the state (take majority or largest or average). 

• Failures can be due to faulty communications or faulty processors

• With faulty communications is not possible for all processes to know that the other processes have reached agreement.

• With faulty processors (that still communicate), agreement is possible under certain conditions.

Communications Failures: Two Army Problem

• Two processes must agree to undertake a coordinated action
• Neither process may proceed unilaterally (agreement is necessary)
• Both processes must act OR neither process must act
• Communications is unreliable
• First Army says "act" but must wait for confirmation from second army.
• If second army receives message it confirms the action.
• Suppose first message is lost, second army does not know first wants to act therefore first army should not act (because it got no confirmation).
• Suppose the second message is lost, second army is ready to act, but the first army cannot without the confirmation. Hence the second army needs confirmation that the first army has received its confirmation. hence it cannot act
• If the first army receives the first confirmation, it must confirm it to verify to the second that it knows that the second is ready to act. But what if this message gets lost. Can the first army act?

It can be shown that agreement in the presence of an unreliable channel is impossible.

Model Assumptions

• There are n processes and at most m of them are faulty

• Processes communicate directly with each other using messages

• Receiving process knows the identity of sending process

• Communications is reliable 

• Communications is synchronous (all wait to act together)

• Three modes of failures

  • Crash fault

  • Omission fault

  • Malicious fault (aka byzantine )

  The first two are detected by synchronous communications (i.e. bounded message delays)

• Messages can be authenticated or not (non-authenticated are called oral)

• Complexity is measured in

  • Time

  • Message traffic

  • storage overhead

Classification  of Agreement Problems

• Byzantine: single sender, all non-faulty must agree on same value

• Consensus: All processes start with some value but must agree on a single value

• Interactive Consistency: Vector of values, each process contributes one element. All must agree on vector 

Relationships Among Agreement Problems

• byzantine: special case of interactive consistency, only one element is of interest

• interactive consistency: can run "n" byzantine agreement protocols

• consensus: can use interactive consistency then take majority in vector (choose default value if no majority)

• byzantine: is most primitive since others can be derived from it.

• byzantine: source process sends value to others then runs consensus algorithm.

Byzantine Agreement

• N processes which must all agree, initial value broadcast by a source processor
• Communications is reliable, synchronous (all process communicate in a round) and authenticated or not.
• Some processes may have byzantine faults
• AGREEMENT: All nonfaulty processors must agree on the same value
• VALIDITY: If the source processor is nonfaulty, then all nonfaulty processors must agree upon the value sent by the source initially
• With "n" processes, each processor will have "n-1" messages to make a decision

Impossibility of Agreement with Three Processors

Let p0, p1, p2 be the three processors, let p0 be the source. The message is one bit.

• Case 1: p0 is not faulty, assume p2 is faulty.
  • Suppose p0 broadcasts initial value 1.
  • p1 will rebroadcast 1 to p2.
  • p2 will broadcast 0 to p1.
  • p1 has conflicting information - has no basis for choosing p0's value
• Case 2: p0 is faulty.
  • Suppose p0 sends 1 to p1 and 0 to p2.
  • p2 sends 0 to p1 while p1 sends 1 to p2.
  • p1 again has conflicting info, just like case 1 - cannot choose
  • p2 following same algorithm as p1 - cannot choose

Figure: Impossibility of 3

Pease showed that it is impossible to reach consensus if the number of faulty processors (m) is greater than floor((n-1)/3) and the showed an algorithm requiring (m-1) rounds of message exchanges

BASICALLY: need a clear majority from non-faulty processes. When n=3 there are only two messages used to decide. If one or more processes faulty, cannot reach majority.

Agreement with Four Processors and One Faulty

• Each process (except P0) chooses the majority. since there are three messages, one bad one cannot change the outcome.
• Pease showed that cannot reach consensus if number of faulty processes exceeds floor((n-1)/3).
  • n = 3, m = 0
  • n = 4,5 or 6?, m = 1
  • n = 7, m = 2

Example Seven Processors, Two Faulty

Suppose P0 and P4 faulty:

Case 1: P0 sends a different value to all processors

They quickly agree that the value is unknown (no majority even with P4 malicious).

BUT could pick minimum or a default value (e.g. 0)

Case 2: P7F2

Case 2: P0 sends 1 to P1, P2 and P3 and 0 to P3, P4 and P5

• Let P4 transmit 1 to P1 P2 and P3 forcing majority to 1 and let it send 0 to P4 and P5, forcing majority to 0. No consensus (P4 is forcing different consensus within two distinct groups (those who initially received 0 and those who received 1).

Lamport-Shostak-Pease Algorithm

Define set of deciders - initially all but source

Source sends value to set of deciders

For 3m+1 or more processors execute m+1 rounds of message passing as follows:

• Set value to that received from source

• Remove yourself from decider set received

• Send value to set of deciders

• Await replies for each decider, send majority (if exists) to your source (if any)

Complexity is O(n**m).

Lamport-Shostak-Pease Algorithm: Example Case 2

From http://class.ee.iastate.edu/somani/cpre545/references.html 

Let (Send value source target) mean that "value" was sent from the source to the target
Let (VoteValue source target) be the value that all good processes agree was sent from source to target
NOTE: this value must be the same for all good processes!
NOTE: does not have to be the actual value sent by the source

Example 1: no faults

BGA (0): // what P0 sends to each processor
// P1, P2 and P3 get on value (v1) and the others get a difference value (v2)
(SEND v1 0 1)
(SEND v1 0 2) 
(SEND v1 0 3) 
(SEND v1 0 4) 
(SEND v1 0 5) 
(SEND v1 0 6) 

BGA (1) 
/* This is the value that 1 calculates */ 
(VOTE 
   (SEND v1 0 1) // actual value received by 1 from 0
   (VoteValue v0.2)// agreed value sent from 0 to 2
   (VoteValue v0.3)// agreed value sent from 0 to 3
   (VoteValue v0.4)// agreed value sent from 0 to 4
   (VoteValue v0.5)// agreed value sent from 0 to 5
   (VoteValue v0.6)// agreed value sent from 0 to 6

The remaining processes 2 through 6 calculate the value of their orders in a similar manner. 
Calculating "VoteValue"

BGA (2) 
// This is the value that 1 calculates 
(VOTE 
   (SEND v1 0 1) // what 1 did receive from 0
   (VOTE (SEND (SEND v1 0 2) 2 1)
 // 2 sends to 1 the value v1 sent by 0 to 2
      (SEND (SEND (SEND v1 0 2) 2 3) 3 1)
  // 3 sends to 1 the value 2 sent to 3 which it claimed was sent to it (2) by 0
      (SEND (SEND (SEND v1 0 2) 2 4) 4 1) 
      (SEND (SEND (SEND v1 0 2) 2 5) 5 1) 
      (SEND (SEND (SEND v1 0 2) 2 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 3) 3 1) 
      (SEND (SEND (SEND v1 0 3) 3 2) 2 1) 
      (SEND (SEND (SEND v1 0 3) 3 4) 4 1) 
      (SEND (SEND (SEND v1 0 3) 3 5) 5 1) 
      (SEND (SEND (SEND v1 0 3) 3 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 4) 4 1) 
      (SEND (SEND (SEND v1 0 4) 4 2) 2 1) 
      (SEND (SEND (SEND v1 0 4) 4 3) 3 1) 
      (SEND (SEND (SEND v1 0 4) 4 5) 5 1) 
      (SEND (SEND (SEND v1 0 4) 4 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 5) 5 1) 
      (SEND (SEND (SEND v1 0 5) 5 2) 2 1) 
      (SEND (SEND (SEND v1 0 5) 5 3) 3 1) 
      (SEND (SEND (SEND v1 0 5) 5 4) 4 1) 
      (SEND (SEND (SEND v1 0 5) 5 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 6) 6 1) 
      (SEND (SEND (SEND v1 0 6) 6 2) 2 1) 
      (SEND (SEND (SEND v1 0 6) 6 3) 3 1) 
      (SEND (SEND (SEND v1 0 6) 6 4) 4 1) 
      (SEND (SEND (SEND v1 0 6) 6 5) 5 1)))

// collapsing 

(VOTE 
   (v1) // what 1 did receive from 0
   (v1) // voted value on what 2 received from 0
   (v1) // voted value on what 3 received from 0
   (v1) // voted value on what 4 received from 0
   (v1) // voted value on what 5 received from 0
   (v1) // voted value on what 6 received from 0
// of course this is v1 and is correct

The remaining processes 2 through 6 calculate their orders in a similar manner. 

Case: One faulty process

NOTE: if one process faulty - it can tell each of the other processes a different value
Say P4 is faulty

BGA (0): // what P0 sends to each processor
// P1, P2 and P3 get on value (v1) and the others get a difference value (v2)
(SEND v1 0 1)
(SEND v1 0 2) 
(SEND v1 0 3) 
(SEND v1 0 4) 
(SEND v1 0 5) 
(SEND v1 0 6) 

BGA (1) 
/* This is the value that 1 calculates */ 
(VOTE 
   (SEND v1 0 1) // actual value received by 1 from 0
   (VoteValue v0.2)// agreed value sent from 0 to 2
   (VoteValue v0.3)// agreed value sent from 0 to 3
   (VoteValue v0.4)// agreed value sent from 0 to 4
   (VoteValue v0.5)// agreed value sent from 0 to 5
   (VoteValue v0.6)// agreed value sent from 0 to 6

The remaining processes 2 through 6 calculate the value of their orders in a similar manner. 
Calculating "VoteValue"

BGA (2) 
// This is the value that 1 calculates 
(VOTE 
   (SEND v1 0 1) // what 1 did receive from 0
   (VOTE (SEND (SEND v1 0 2) 2 1)
      (SEND (SEND (SEND v1 0 2) 2 3) 3 1)
      (SEND (SEND (SEND v2.4.1 0 2) 2 4) 4 1) 
// value of what P2 received from P0 as reported by faulty 4 to P1
      (SEND (SEND (SEND v1 0 2) 2 5) 5 1) 
      (SEND (SEND (SEND v1 0 2) 2 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 3) 3 1) 
      (SEND (SEND (SEND v1 0 3) 3 2) 2 1) 
      (SEND (SEND (SEND v3.4.1 0 3) 3 4) 4 1) 
      (SEND (SEND (SEND v1 0 3) 3 5) 5 1) 
      (SEND (SEND (SEND v1 0 3) 3 6) 6 1)) 
   (VOTE (SEND (SEND v4.1 0 4) 4 1) 
      (SEND (SEND (SEND v4.2 0 4) 4 2) 2 1) 
      (SEND (SEND (SEND v4.3 0 4) 4 3) 3 1) 
      (SEND (SEND (SEND v4.5 0 4) 4 5) 5 1) 
      (SEND (SEND (SEND v4.6 0 4) 4 6) 6 1)) 
// p4 can tell each other process a different value or can try to split the vote
// but all the other processes will report what they receive from P4 reliably to P1 (and they  report the value
// value to P2, P3, P5 and P6)
   (VOTE (SEND (SEND v1 0 5) 5 1) 
      (SEND (SEND (SEND v1 0 5) 5 2) 2 1) 
      (SEND (SEND (SEND v1 0 5) 5 3) 3 1) 
      (SEND (SEND (SEND v5.4.1 0 5) 5 4) 4 1) 
      (SEND (SEND (SEND v1 0 5) 5 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 6) 6 1) 
      (SEND (SEND (SEND v1 0 6) 6 2) 2 1) 
      (SEND (SEND (SEND v1 0 6) 6 3) 3 1) 
      (SEND (SEND (SEND v6.4.1 0 6) 6 4) 4 1) 
      (SEND (SEND (SEND v1 0 6) 6 5) 5 1)))

// collapsing 

(VOTE 
   (v1) // what 1 did receive from 0
   (v1) // voted value on what 2 received from 0
   (v1) // voted value on what 3 received from 0
   (vP4) // voted value on what 4 received from 0
   (v1) // voted value on what 5 received from 0
   (v1) // voted value on what 6 received from 0
)
// VP4 could be a majority value (if P4 sent a majority the same value)
// or the vote could be undecided where in vP4 = UNDECIDED value
// In this case, all good processes will decide that vP4 is UNDECIDED

VOTE will return v1 regardless of the value (or not) of vP4)

Case: two faulty processes (not the original source)

Say P4 and P5 are faulty

BGA (0): // what P0 sends to each processor
// P1, P2 and P3 get on value (v1) and the others get a difference value (v2)
(SEND v1 0 1)
(SEND v1 0 2) 
(SEND v1 0 3) 
(SEND v1 0 4) 
(SEND v1 0 5) 
(SEND v1 0 6) 

BGA (1) 
/* This is the value that 1 calculates */ 
(VOTE 
   (SEND v1 0 1) // actual value received by 1 from 0
   (VoteValue v0.2)// agreed value sent from 0 to 2
   (VoteValue v0.3)// agreed value sent from 0 to 3
   (VoteValue v0.4)// agreed value sent from 0 to 4
   (VoteValue v0.5)// agreed value sent from 0 to 5
   (VoteValue v0.6)// agreed value sent from 0 to 6

The remaining processes 2 through 6 calculate the value of their orders in a similar manner. 
Calculating "VoteValue"

BGA (2) 
// This is the value that 1 calculates 
(VOTE 
   (SEND v1 0 1) // what 1 did receive from 0
   (VOTE (SEND (SEND v1 0 2) 2 1)
      (SEND (SEND (SEND v1 0 2) 2 3) 3 1)
      (SEND (SEND (SEND v2.4.1 0 2) 2 4) 4 1) 
      (SEND (SEND (SEND v2.5.1 0 2) 2 5) 5 1) 
      (SEND (SEND (SEND v1 0 2) 2 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 3) 3 1) 
      (SEND (SEND (SEND v1 0 3) 3 2) 2 1) 
      (SEND (SEND (SEND v3.4.1 0 3) 3 4) 4 1) 
      (SEND (SEND (SEND v3.5.1 0 3) 3 5) 5 1) 
      (SEND (SEND (SEND v1 0 3) 3 6) 6 1)) 
   (VOTE (SEND (SEND v4.1 0 4) 4 1) 
      (SEND (SEND (SEND v4.2 0 4) 4 2) 2 1) 
      (SEND (SEND (SEND v4.3 0 4) 4 3) 3 1) 
      (SEND (SEND (SEND v4.5 0 4) 4 5) 5 1) 
      (SEND (SEND (SEND v4.6 0 4) 4 6) 6 1)) 
   (VOTE (SEND (SEND v5.1 0 5) 5 1) 
      (SEND (SEND (SEND v5.2 0 5) 5 2) 2 1) 
      (SEND (SEND (SEND v5.3 0 5) 5 3) 3 1) 
      (SEND (SEND (SEND v5.4 0 5) 5 4) 4 1) 
      (SEND (SEND (SEND v5.6 0 5) 5 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 6) 6 1) 
      (SEND (SEND (SEND v1 0 6) 6 2) 2 1) 
      (SEND (SEND (SEND v1 0 6) 6 3) 3 1) 
      (SEND (SEND (SEND v6.4.1 0 6) 6 4) 4 1) 
      (SEND (SEND (SEND v6.5.1 0 6) 6 5) 5 1)))

// collapsing 

(VOTE 
   (v1) // what 1 did receive from 0
   (v1) // voted value on what 2 received from 0 (majority good processors)
   (v1) // voted value on what 3 received from 0
   (vP4) // voted value on what 4 received from 0
   (vP5) // voted value on what 5 received from 0
   (v1) // voted value on what 6 received from 0
)
VOTE will return v1 regardless of the value (or not) of vP4 and vP5)

CASE: Source and one other faulty

BGA (0): // what P0 sends to each processor
// P1, P2 and P3 get on value (v1) and the others get a difference value (v2)
(SEND v1 0 1)
(SEND v1 0 2) 
(SEND v1 0 3) 
(SEND v2 0 4) 
(SEND v2 0 5) 
(SEND v2 0 6) 

BGA (1) 
/* This is the value that 1 calculates */ 
(VOTE 
   (SEND v1 0 1) // actual value received by 1 from 0
   (VoteValue v0.2)// agreed value sent from 0 to 2
   (VoteValue v0.3)// agreed value sent from 0 to 3
   (VoteValue v0.4)// agreed value sent from 0 to 4
   (VoteValue v0.5)// agreed value sent from 0 to 5
   (VoteValue v0.6)// agreed value sent from 0 to 6

The remaining processes 2 through 6 calculate the value of their orders in a similar manner. 

BGA (2) 
// This is the value that 1 calculates 
(VOTE 
   (SEND v1 0 1) // what 1 did receive from 0
   (VOTE (SEND (SEND v1 0 2) 2 1) 
      (SEND (SEND (SEND v1 0 2) 2 3) 3 1) 
      (SEND (SEND (SEND v2.4.1 0 2) 2 4) 4 1) 
      (SEND (SEND (SEND v1 0 2) 2 5) 5 1) 
      (SEND (SEND (SEND v1 0 2) 2 6) 6 1)) 
   (VOTE (SEND (SEND v1 0 3) 3 1) 
      (SEND (SEND (SEND v1 0 3) 3 2) 2 1) 
      (SEND (SEND (SEND v3.4.1 0 3) 3 4) 4 1) 
      (SEND (SEND (SEND v1 0 3) 3 5) 5 1) 
      (SEND (SEND (SEND v1 0 3) 3 6) 6 1)) 
   (VOTE (SEND (SEND v2.1 0 4) 4 1) 
      (SEND (SEND (SEND v1.2 0 4) 4 2) 2 1) 
      (SEND (SEND (SEND v1.3 0 4) 4 3) 3 1) 
      (SEND (SEND (SEND v2.5 0 4) 4 5) 5 1) 
      (SEND (SEND (SEND v2.6 0 4) 4 6) 6 1)) 
   (VOTE (SEND (SEND v2 0 5) 5 1) 
      (SEND (SEND (SEND v2 0 5) 5 2) 2 1) 
      (SEND (SEND (SEND v2 0 5) 5 3) 3 1) 
      (SEND (SEND (SEND v5.4.1 0 5) 5 4) 4 1) 
      (SEND (SEND (SEND v2 0 5) 5 6) 6 1)) 
   (VOTE (SEND (SEND v2 0 6) 6 1) 
      (SEND (SEND (SEND v2 0 6) 6 2) 2 1) 
      (SEND (SEND (SEND v2 0 6) 6 3) 3 1) 
      (SEND (SEND (SEND v6.4.1 0 6) 6 4) 4 1) 
      (SEND (SEND (SEND v2 0 6) 6 5) 5 1))) 

(VOTE 
   (v1) // what 1 did receive from 0
   (v1) // voted value on what 2 received from 0
   (v1) // voted value on what 3 received from 0
   (v7) // voted value on what 4 received from 0
   (v2) // voted value on what 5 received from 0
   (v2) // voted value on what 6 received from 0

v7 = v1, vote = v1
v7 = v2, vote = undecided value
v7 = not v1 and not v2 (could be the undecided value), vote = v1

The remaining processes 2 through 6 calculate their orders in a similar manner. 
For 2 and 3 - the voted values will be identical, its own cannot be outvoted

References

• http://www.cs.cornell.edu/cs614-sp98/notes/byzantine.html 

• http://class.ee.iastate.edu/somani/cpre545/references.html 

• http://www.nada.kth.se/kurser/kth/2D5340/wwwbook/wwwbook.html 

• http://catless.ncl.ac.uk/Risks/16.96.html 

Interactive Consistency Problem

All non-faulty processors must agree on a vector of values, each processor contributing one element of the vector.

VALIDITY: if Pi is non-faulty than the i-th element of the vector must be the initial value broadcast by Pi, otherwise it doesn't matter what value is agreed upon.

Fischer proved that with asynchronous processors and unbounded delays, no agreement if possible even if one processor is faulty (and even if fail-silent).

Example Figure 4-23: Original Algorithm

Step 1:

• P1 broadcasts "1" to others
• P2 broadcasts "2" to others
• P3 broadcasts "x", "y", "z" to others
• P4 broadcasts "4" to others

Step 2:

• P1 forms vector (1,2,x,4)
• P2 (1,2,y,4)
• P3 (a,b,c,d) for P1, (e,f,g,h) for P2, (i,j,k,l) for P4
• P4 (1,2,z,4)

Step 3: all broadcast this vector

Step 4: Each form majority in i-th location (or ? is no majority)
Books claims this results in agreement, but what if c = z, g = x, k = y

• P1 (1,2,z,4)
• P2 (1,2,x,4)
• P3 (-,-,x2,-) to P1, (-,-,y2,-) to P2, (-,-,z2,-) to P4
• P4 (1,2,y,4)

No agreement, go another round, send vector, but what if x2=x, y2=y and z2=z? (P3 keeps switching its vote to confuse the others.)

Example Figure 4-23: Modified Algorithm

Solution: Don't accept Pi's value from Pi more than once (force consistency of Pi's value from Pi to Pj. This in effect forces c=x=x2, g=y=y2, k=z=z2.

Step 1:

• P1 broadcasts "1" to others
• P2 broadcasts "2" to others
• P3 broadcasts "x", "y", "z" to others
• P4 broadcasts "4" to others

Step 2: (blanks denote no value need be sent)

• P1 forms vector ( ,2,x,4)
• P2 (1, ,y,4)
• P3 (a,b, ,d) for P1, (e,f, ,h) for P2, (i,j, ,l) for P4
• P4 (1,2,z, )

Step 3: all broadcast this vector

Step 4: Each form majority in i-th location (or ? is no majority) using value sent from Pi previously in i-th location of Pi's new vector
Two cases, x, y and z are different

• P1 (1,2,?,4)
• P2 (1,2,?,4)
• P3: who cares
• P4 (1,2,?,4)

x = y but not z (and symmetrical cases of two agree)

• P1 (1,2,x,4)
• P2 (1,2,x,4)
• P4 (1,2,x,4)

If x=y=z, then can't catch a consistent liar.

Notice this is four simultaneous runnings of the Byzantine agreement discussed earlier.

Copyright chris wild 1996.
For problems or questions regarding this web contact [Dr. Wild].
Last updated: August 29, 1996.