Skip to content

API object deletion unreliable if HA zone is not connected #10022

Description

@julianbrost

When deleting objects using the REST API while the nodes aren't connected in an HA zone results in the objects reappearing at a later time.

To Reproduce

  1. Start with an HA master setup with both running.
  2. Create a few objects using the API:
    for i in {1..100}; do
      curl --fail-with-body -sSku root:icinga -X PUT "https://localhost:5665/v1/objects/hosts/test-delete-$i" \
           --json '{"attrs": {"check_command":"dummy"}}'
    done
  3. Check /var/lib/icinga2/api/packages/_api/*/conf.d/hosts/test-delete-*.conf on both masters to verify the objects exist on both.
  4. Stop the second master.
  5. Delete the just created objects via API on the first master:
    for i in {1..100}; do
      curl --fail-with-body -sSku root:icinga -X DELETE "https://localhost:5665/v1/objects/hosts/test-delete-$i" \
           --json '{"cascade": true}'
    done
  6. Repeat step 3, there should be none remaining on the first master and all on the second as it's not running.
  7. Start the second master again.
  8. Wait a bit for the masters to sync and repeat step 3 again. This is where things start to go wrong: the objects disappeared from the second master but reappeared on the first.
  9. Restart the second master.
  10. Wait for them to sync and repeat step 3 again. Now the objects are also reappearing on the second master and the objects exist on both masters again.

Theory on what happens

When both masters reconnect, both generate messages to sync the other. The first master will replay the delete messages to the second and the second will send the object updates for the (from its perspective still existing) objects to the first master causing them to reappear.

Your Environment

Tested with the current master (9e31b8b).

Metadata

Metadata

Labels

area/apiREST APIarea/distributedDistributed monitoring (master, satellites, clients)bugSomething isn't working

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions