Saturday, May 9, 2020

Practical tips while working with Azure Functions



                      Practical tips while working with Azure Functions


1) You will probably need to create a lot of functions to solve a business problem. It is not wise to put every function inside one function app. It is better to categorize/segregate into different function apps, as a function app is a deployable unit. 


2) Unless you have a strong reason not to, its better to use a "consumption plan" as opposed to "app service plan". 

3) If your function app has functions with service bus trigger, make sure the following configuration is set - 
a) The function app is set to "always on" to "true". (This is not needed in consumption plan though).

b) When multiple messages come to the service bus queue/topic, multiple parallel instances of your functions are created (default 15). Make sure you throttle it appropriately. Otherwise your app service can have spikes in memory/CPU consumption and there can be impact in downstream systems (example - The downstream system cannot accept 15 simultaneous update and might timeout).  Below is the setting required in host.json to throttle -

  "extensions": {
    "serviceBus": {
      "messageHandlerOptions": {
        "autoComplete": false,
        "maxConcurrentCalls": "5"
      }
    }
  }
}

c) It is always good to have autoComplete in a service bus triggered function as "false". That way you can maintain transactions and Abandon/Deadletter/Complete as per the outcome/requirement. Below is some code snippet to do the same.



    public class ProductReceivedFunction
    {
        private readonly ProductContext productDbContext;
        private IConfiguration configuration;
        private IHttpService httpService;
        private ILogger logger;
        public ProductReceivedFunction(IConfiguration configuration,ProductContext productDbContext, IHttpService httpService)
        {
            this.productDbContext = productDbContext;
            this.configuration = configuration;
            this.httpService = httpService;
        }
        [FunctionName("ProductReceivedFunction")]
        public async Task Run([ServiceBusTrigger("productreceived", Connection = "sbqueueconnection-productreceived-listen")]string queueMessage, ILogger logger, MessageReceiver messageReceiver, string lockToken)
        {
            this.logger = logger;
            try
            {
                this.logger.LogInformation($"C# ServiceBus queue trigger function processed message: {queueMessage}");
//do your stuff
                await messageReceiver.CompleteAsync(lockToken);
             
            }
            catch (Exception ex)
            {
                this.logger.LogError(ex, "Error in processing");
                await messageReceiver.DeadLetterAsync(lockToken);
            }
        }
    }

d) Never inject IConfiguration in startup.cs of a function app. It is injected by default. Infact if you explicitly inject, it will mess up with autoComplete=false setting. It will try to complete the message (when actually you have already done it in code manually) and give a messagelocklost exception

e) Double check that in all paths (if, else, try, catch etc) you are completing/deadlettering/abandoning the message and that too only once. If its done twice, you will get a messagelocklost exception.


4) Set your queue TTL and lock duration appropriately. In my experience, the lock duration of  1 minute should be good for most scenarios. The TTL however depends on your design. If there is an upstream process that runs periodically and copies the state (as messages in a queue), the TTL of the queue should be <= the upstream process frequency, otherwise there can be duplicate messages. (In such scenarios, the Receiver should also be made idempotent).


5) Always integrate your Function app with App insights. Sample kusto queries for diagnosis -

traces | where operationName == '<FunctionName>'
exceptions | take 10
traces | where severityLevel == 3 // for Errors

6) Always perform load testing on your functions and monitor the app service plan metrics like - CPU percentage/Memory percentage.

7) Monitor your function app using Azure Monitor. You can monitor the deadletter queue, main queue (if 100 msgs are there for over 10 minutes). You can also monitor on the results of log analytics workspace query (point 5 above).





Saturday, April 1, 2017

Advanced concepts WCF listener for Service Bus


For basics of creating a WCF listener to a ServiceBus queue/topic, Pl refer -
WCF-ServiceBus Listener




1)      The default protocol used is NetMessaging (not AMQP). NetMessaging is a native protocol available in .NET only. AMQP is an OASIS standard and interoperable, meaning message sent by .NET publisher could by consumed by PHP/Python client.
2)      Unlike Service Bus, ActiveMQ offers a lot of protocols for message interchange like –
a)      Openwire (native protocol for activemq – default)
b)      Mqtt (light weight pub-sub protocol typically used for IoT)
c)      Amqp (cross platform, efficient)
d)  STOMP
3)      With the above approach, a new instance of WCF service is created each time, a message arrives the topic/queue. This was confirmed with the following code –
public interface IService
{       
        [OperationContract(IsOneWay = true, Action = "*"), ReceiveContextEnabled(ManualControl = true)]
        void MessageReader(Message message);
}
public class MyService: IService
{
        private string instanceId = "";
        public MyService()
        {
            instanceId = Guid.NewGuid().ToString();           
        }
        public void MessageReader(Message message)
        {
            var logEntry = new LogEntry { Message = string.Format("Service instance id is {0}",instanceId) };
            Utilities.WriteLogSimple(logEntry);
        }
}

4)      Once message arrives a topic/queue, a lock is applied. All topics/queues have a message lock timeout property, whose default value is 1 min and can be set to 5 mins as maximum value. The processing of the message has to be committed (or rolled back)  within this time or else the lock will be automatically released and the message will go back to the topic/queue, thus increasing the delivery count. Pl note that this redelivery will trigger the wcf processing again .The max delivery count can also be set. Once the message has been redelivered more than the “max delivery count”, it goes to “deadletter queue”.



  5)   With reference to the point above, the following situation can occur and should be handled carefully. Suppose a message arrives a topic, a wcf instance is created and it applies the lock on the message. The processing is message takes more than 1 minute (default lock timeout) and get redelivered to the topic/queue 10 times (default max delivery count). Ultimately all the 10 times, the message gets processed, thus causing inconsistent results. So make sure the message processing timeout (say database execution timeout or service call) is always less than message lock timeout. Also reduce the max delivery count from 10 (default) to a suitable value.




Thursday, March 16, 2017

Transient fault handling and retry strategy



Transient fault handling and retry strategy

While working with webservices in the middleware layer (say a subscriber to a service bus queue), it is wise to use a good retry strategy for transient fault handling for scenarios such as – connectivity issues, timeout issues etc. Retrying immediately will not help much as the chances of the resource being unavailable would be high. The code below explains ExponentialBackoff strategy. Exponentialbackoff retry strategy means - It ll retry after 2seconds, 4seconds, 8seconds.  

Explanation of ExponentialBackoff parameters –

var retryStrategy = new ExponentialBackoff(3, TimeSpan.FromSeconds(2),
                    TimeSpan.FromSeconds(20), TimeSpan.FromSeconds(1));
a)      first param - number of retries
b)      second param - minimum backoff time (say 2seconds). Meaning - time it should wait 2 seconds before the first RETRY or minimum time between any retries.
c)      third param - maximum backoff limit.  In this case - the third retry will happen after about ~ 2 + 4 + 8 = 14 seconds. The third param is there to put a limit on backoff period (which is 20 in this case).
d)      fourth param - delta to add some randomness, otherwise all clients would be retrying simultaenously.
in this case random value is 1 second. So 2,4,8 could be 1,5,7 etc.


Please find the code files and detailed steps below –


1)      Add a nuget package to EnterpriseLibrary.TransientFaultHandling.
2)      Create the following class (to specify errors/exceptions which should be retried) –
public class TransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
    {
        // add the list of exceptions, that you consider transient.
        public bool IsTransient(Exception ex)
        {
            if (ex is WebException)
                return true;
            if (ex is FakeTimeoutException)
                return true;
            return false;
        }
    }

3)      Create a FakeTimeoutException (for demo/testing purpose) –
public class FakeTimeoutException : Exception
    {
        public FakeTimeoutException(string msg) : base(msg)
        {

        }
    }

4)      Create the main class and method as shown below –

class Program
    {
        static void Main(string[] args)
        {
            var retryStrategy = new ExponentialBackoff(3, TimeSpan.FromSeconds(2),
                    TimeSpan.FromSeconds(20), TimeSpan.FromSeconds(1));
            TransientErrorDetectionStrategy errorDetectionStrategy = new TransientErrorDetectionStrategy();

            var retryPolicy = new RetryPolicy(errorDetectionStrategy,retryStrategy);

            retryPolicy.ExecuteAction(() =>
            ExecuteHTTPGET("https://microsoft.sharepoint.com")
            );

        }

        private static void ExecuteHTTPGET(string requestUri)
        {           
            Console.WriteLine(DateTime.Now);
            throw new FakeTimeoutException("fake timeout");
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(requestUri);
            request.KeepAlive = false;
            request.Method = "GET";

            HttpWebResponse webResponse = (HttpWebResponse)request.GetResponse();
            int requestStatus = (int)webResponse.StatusCode;
            webResponse.Close();
        }

5)     Run the program and see the timestamp in the console. Try by switching FakeTimeoutException with SomeOtherException and see the behavior.