[FIXED] Which Azure service to use for processing data from Event Hub?

Issue

I would appreciate some help picking out the best suited Azure services for my scenario - I am just beginning with Azure services and my knowledge is pretty limited.

I have data from multiple sources, and of different shapes, coming into an Event Hub. I need to subscribe to the events from the Event Hub and, based on their format, process them and ultimately save them into an SQL Database. All components - events consumers, the SQL Database - need to be hosted in the cloud.

How would I implement this in an "Azure Orientated Architecture"?

In an off cloud application, I would have competing consumers subscribing to the Event Hub. They would be some console applications or Windows services, and each would be processing the events asynchronously (this is further simplified by the event processing being idempotent).

Ideally, the Azure equivalent of the above consumers would scale up and down automatically, so I would like to not have to use VMs that host console applications (where I would need to keep an eye on the VM's resources myself). Scaling and deployment wise they would have to behave like App Services, however I'm under the impression that those are just for web applications. I've also briefly looked at Web Jobs, but those seem to be polling data at various intervals, whereas I need a proper event subscriber that the Event Hub pushes data into.

Any help will be greatly appreciated! Thank you.

Later Edit:

1. I've looked into Web Jobs and they do allow continuous processing, hence looks like they can be used as automatically scaling subscribers.
2. Ideally I would like to write the code for the subscribers in F#. C# is the other option if that is not available.

Solution

I'd say use a WebJob in combination with an EventProcessor. I wrote some demo code that can easily be transferred to a WebJob: https://github.com/DeHeerSoftware/SemanticLogging.EventHub/tree/master/SemanticLogging.EventHub.Processor

See https://azure.microsoft.com/en-us/documentation/articles/event-hubs-csharp-ephcs-getstarted/#receive-messages-with-eventprocessorhost for official documentation.

I've created a WebJob myself using this approach. Works like a charm.

Answered By - Peter Bons

Answer Checked By - Gilberto Lyons (PHPFixing Admin)