1. Event Grid ensures at least one-time delivery
To support scalability and low latency event delivery, event grids are design in such a way that it supports at least one-time delivery of the event. Which means it can deliver same event multiple time. Hence, your event handler must be designed in such a way that if it handles the same event multiple times, the outcome should be the same.
2. Order of events is not guaranteed
Order of events is not guaranteed. But if your business process demands for that then you must have some internal queues or weight logic on events.
3. Event processing should be quick (<10 seconds)
Events are light weight messages and they are also meant for highly scalable scenarios. Because event grids are not queue or stream like event hubs, events should be process in fraction of a second or sometimes in couple of seconds. This is also to stop event delivery retry if an event received ack is not sent back to event grid. (refer resiliency)
But if your event handler is a long running process or you want to process events in batch, then you should offload the events from event grid to some internal queue (storage queue, service bus queue, etc).
4. Enable filtering based on event types
The event handler should not listen/subscribe to all the events but should only be triggered if the event is relevant to the handler. This filter logic should be at event subscription level using event types or other properties from event metadata or payload.
5. Enable instrumentation
Events should be logged into an event store through-out the process. This is to ease troubleshooting but this is also useful if you want to implement event-replayability feature.
6. Resiliency
Event grids support resiliency but you have to configure it. Event grid supports retries. Suppose your event handler was not successfully able to process the event or it was not able to respond back within 30 seconds, then event grid retries to deliver that event. By default for maximum 24 hours the event grid will hold the message and will try for maximum 30 times to deliver this message. After that the message can be offloaded to a dead-letter queue (azure storage). All these settings are configurable from azure portal.
7. Endpoint validation
If your event handler is not a native azure service i.e. functions, logic apps, hybrid connection etc but a custom handler which you are registering as webhook with event grid, then you have implement endpoint validation logic with in your handler.
When you try to register a webhook type of event handler, the event grid will immediately send a validation type event to that endpoint. The handler needs to handle that event and make a callback request to the URL mentioned in event payload. Only then the handler will be successfully registered as a valid subscriber and will receive events further.
