Like many others, our applications have some master data. These data are not user-configurable/user-changeable, so we have not found the need to put them into the database. So far, we have put them in JSON files in our solution, and used the dotnet Configuration API ↗, and this has worked flawlessly.

However, every once in a while we have to update some of this master data - and with the configuration packed and deployed with the application, this requires a new deployment to production. We wanted a solution where we would change the master data, without needing to re-deploy the application.

Since we are hosted in Azure, we looked into Azure App Configuration ↗. However, Azure App Configuration has a size limit for each key-value pair of 10KB ↗. This is probably fine in most cases. Still, one of our master data records is a long list of locations (with a size of 57 KB) - and we could not find a good way to split this data up. This ruled out Azure App Configuration.

Another option is to host the configuration files on blob storage - like Azure Blob Storage ↗. However, Microsoft has not provided us with a configuration provider for Azure Blob Storage. Fortunately, the Configuration API is extensible ↗. This lead to us creating the NovoNordisk.Configuration.AzureBlob ↗ NuGet package.

Let’s dive into how to create your own configuration provider.

Configuration API#

You can think of the Configuration API / IConfiguration as a key-value pair collection. You can load data into it with configuration providers, and then bind configuration values to objects. Dotnet has a bunch of built-in configuration providers to get configuration values from files, environment variables, command-line arguments et cetera. Let’s go through an example using a JSON file.

{
  "Settings": {
    "SomeString": "Completely new string"
  }
}

We can then load this JSON file into IConfiguration using the JSON file configuration provider:

var builder = Host.CreateApplicationBuilder(args);
builder.Configuration.AddJsonFile("config.json");

Now we have the “Settings” key in IConfiguration, which we can then bind to an object.

public record Settings
{
    public string? SomeString { get; init; }
}

builder.Services.Configure<Settings>(builder.Configuration.GetSection("Settings"));

With the configuration values bound to the Settings object, we can now use dependency injection to use the values in our code:

public class SomeService(IOptions<Settings> settings)
{
    public string? SomeMethod()
    {
        return settings.Value.SomeString;
    }
}

Please note that we use IOptions<T> to inject the configuration objects here. You cannot just inject the Settings object, as it is only registered using the Options interfaces. IOptions<T> is not the only interface we can use, and we will make use of the others later on.

For our use case, we want to retrieve a file from Azure Blob Storage, and load the contents into IConfiguration. We can do that by making our own configuration provider.

Making the configuration provider#

Starting out, we need a configuration source. This will contain the settings needed for our configuration provider, so we will include properties like the URL to the blob, a polling interval et cetera.

using Azure.Core;
using Microsoft.Extensions.Configuration;

public class BlobJsonConfigurationSource : IConfigurationSource
{
    public required Uri BlobUrl { get; init; }

    public bool ReloadOnChange { get; init; } = false;

    public TimeSpan PollingInterval { get; init; } = TimeSpan.FromSeconds(30);

    public Action<Exception>? ExceptionHandler { get; init; }

    public TokenCredential? Credential { get; init; }

    public IConfigurationProvider Build(IConfigurationBuilder builder)
    {
        return new BlobJsonConfigurationProvider(this);
    }
}

Let’s cover the properties included:

• BlobUrl is the URL for the Azure Blob to load.
• ReloadOnChange is for turning on/off the periodic change detection.
• PollingInterval is how often we should check for changes.
• ExceptionHandler is a callback/action to be called when an exception occurs during a reload.
• Credential is the Azure credentials needed to access the blob, if not public.

When creating a configuration provider, there are several ways to do so. If we are working with JSON data, the easiest path is to inherit from the JsonConfigurationProvider - this will provide us with a method, where we can hand it a stream of JSON data, and it takes care of the rest. However, this also means that our configuration source needs to inherit from JsonConfigurationSource, which in turn inherits from FileConfigurationSource. This includes properties such as a file provider, file path et cetera. Properties we have no use for, as we are not loading from the local file system.

Another way is to implement the IConfigurationProvider interface directly.

public interface IConfigurationProvider
{
  bool TryGet(string key, out string? value);
  void Set(string key, string? value);
  IChangeToken GetReloadToken();
  void Load();
  IEnumerable<string> GetChildKeys(IEnumerable<string> earlierKeys, string? parentPath);
}

But there is a middle-ground between these approaches - we can inherit from the ConfigurationProvider base class. This base class takes care of the core behaviours of configuration providers - like handling change tokens. We just have to fill the Data property with our configuration data and call the OnReload() method to signal to the configuration system that the data has changed.

using Azure.Storage.Blobs;
using Microsoft.Extensions.Configuration;

public class BlobJsonConfigurationProvider(BlobJsonConfigurationSource source) : ConfigurationProvider
{
    public override void Load()
    {
        var client = new BlobClient(source.BlobUrl, source.Credential);
        using var stream = new MemoryStream();
        using var response = client.DownloadTo(stream);
        stream.Seek(0, SeekOrigin.Begin);

        if (!response.IsError)
        {
            Data = /* TODO: Parse JSON */

            OnReload();
        }
    }
}

After we download the JSON file from Azure Blob Storage, we have to parse it in order to hand it over to the configuration system. The Data property is just a dictionary of strings, and while we could do this by hand, there is an easier approach. The JsonConfigurationFileParser is part of the dotnet runtime. However, it is not public, hence we cannot access it directly. Fortunately for us, the dotnet runtime is open-source, and this file is licensed under the MIT license - so we can just go ahead and grab it ↗.

...
if (!response.IsError)
{
    Data = JsonConfigurationFileParser.Parse(stream);

    OnReload();
}
...

While we technically have what we need, with the configuration source and configuration provider, we might want to match the existing Configuration API (e.g. builder.AddJsonFile()). To do this, we can create an extension method to help with registering the configuration source and provider.

using Azure.Core;
using Microsoft.Extensions.Configuration;

public static class ConfigurationBuilderExtensions
{
    public static IConfigurationBuilder AddJsonBlob(this IConfigurationBuilder builder, Uri blobUrl, bool reloadOnChange = false, TimeSpan? pollingInterval = null, TokenCredential? tokenCredential = null, Action<Exception>? exceptionHandler = null)
    {
        return builder.Add(new BlobJsonConfigurationSource
        {
            BlobUrl = blobUrl,
            ReloadOnChange = reloadOnChange,
            PollingInterval = pollingInterval ?? TimeSpan.FromSeconds(30),
            Credential = tokenCredential,
            ExceptionHandler = exceptionHandler
        });
    }
}

With this method, we can add configuration files from Azure Blob Storage just like we would any other configuration file.

builder.Configuration.AddJsonFile("localfile.json");
builder.Configuration.AddJsonBlob(new Uri(blobUrl),
    reloadOnChange: true,
    pollingInterval: TimeSpan.FromSeconds(30),
    tokenCredential: new DefaultAzureCredential());

Reload configuration on change#

To make our configuration provider reloadable, we can periodically check whether the blob has been updated. One way to do this is to use a System.Threading.Timer to trigger some code to run at an interval.

public class BlobJsonConfigurationProvider : ConfigurationProvider, IDisposable, IAsyncDisposable
{
    private readonly BlobJsonConfigurationSource _source;
    private readonly Timer? _timer;
    ...

    public BlobJsonConfigurationProvider(BlobJsonConfigurationSource source)
    {
        _source = source;
        
        if (_source.ReloadOnChange)
        {
            _timer = new Timer(CheckForChanges, null, _source.PollingInterval, _source.PollingInterval); // We cover the CheckForChanges method in a minute
        }
    }

    ...
    
    public void Dispose()
    {
        _timer?.Dispose();
    }

    public async ValueTask DisposeAsync()
    {
        if (_timer != null) await _timer.DisposeAsync();
    }
}

The first parameter passed to the timer is the callback method to run (in this case CheckForChanges(), which we will cover in a minute). The second parameter is a state object, which we will not use. The third and fourth parameter is the delay before the timer fires for the first time, and at which interval we wish to trigger the callback.

Now that we have a setup for periodically checking for updates, we have to consider how we check for changes. We want to avoid downloading the entire file each time, as this could be costly on our Azure bill.

Azure Blob Storage has ETags (Entity Tags) on our files. An ETag is an identifier for a specific version of a file - this ETag changes every time the file changes. Using the ETag, we can detect if a blob has changed.

Let’s first update our Load() method to save the ETag when we download the file.

public class BlobJsonConfigurationProvider : ConfigurationProvider, IDisposable, IAsyncDisposable
{
    private string? _etag;
    ...
    public override void Load()
    {
        var client = new BlobClient(_source.BlobUrl, _source.Credential);
        using var stream = new MemoryStream();
        using var response = client.DownloadTo(stream);
        stream.Seek(0, SeekOrigin.Begin);

        if (!response.IsError)
        {
            Data = JsonConfigurationFileParser.Parse(stream);
            _etag = response.Headers.ETag.ToString()?.Trim('"');
            
            OnReload();
        }
    }
}

Then we can write our CheckForChanges() method. We will get the blob properties, and check the ETag. If the ETag does not match, we will trigger the Load() method.

private void CheckForChanges(object? state)
{
    try
    {
        var client = new BlobClient(_source.BlobUrl, _source.Credential);
        var props = client.GetProperties();
        if (props is null || !props.HasValue)
        {
            throw new RequestFailedException("Failed to retrieve blob properties");
        }

        if (_etag != props.Value.ETag.ToString().Trim('"'))
        {
            Load();
        }
    }
    catch (Exception e)
    {
        _source.ExceptionHandler?.Invoke(e);
    }
}

We wrap the code in a try/catch, as the System.Threading.Timer callback runs on a threadpool thread, and an uncaught exception might cause the timer to stop firing, or even cause the application to crash. In the catch block, we can then invoke the exception handler action from our configuration source, and this can then be used to log or handle errors gracefully.

You might have wondered why we save the ETag as a string (instead of using the ETag type) and trim away the double quote. This string manipulation is a workaround for observed inconsistencies in how Azure SDK methods return ETag values. Of the two methods we use (GetProperties() and DownloadTo()), one returns a quoted ETag and the other returns an unquoted ETag. This results in the ETag not matching, hence it will load the entire file on every check.

Using the updated configuration#

Since we were already using IOptions for our configuration, switching from a local file to an Azure Blob Storage file required minimal changes to our codebase. The primary thing to note is that IOptions is registered as a singleton, hence it never updates - so even though you enable reloading, the IOptions instance never gets the updated configuration.

For getting updated configuration, you have two options (pun intended): IOptionsSnapshot and IOptionsMonitor.

IOptionsSnapshot provides, as the name implies, a snapshot of the current data. This is useful for scenarios where your configuration should be recomputed on every request. This is registered as Scoped, which means you will not be able to inject it into a class registered as Singleton.

public class SomeService(IOptionsSnapshot<Settings> settings)
{
    public string? SomeMethod()
    {
        return settings.Value.SomeString;
    }
}

IOptionsSnapshot will probably be fine in most use cases, but if you need to inject into a Singleton, or you have some complicated processing of the configuration, that you wish to cache, IOptionsMonitor might be the solution for you. This is registered as a Singleton, hence you can inject it into any service lifetime. Like IOptionsSnapshot, IOptionsMonitor provides a property with the current value. But it also provides the possibility of registering an OnChange listener, that will be triggered when the configuration reloads.

public class SomeService
{
    private readonly IOptionsMonitor<Settings> _settings;

    public SomeService(IOptionsMonitor<Settings> settings)
    {
        _settings = settings;
        settings.OnChange((s) =>
        {
            Console.WriteLine("Config changed: SomeString={0}",
                s.SomeString);
        });
    }
    
    public string? SomeMethod()
    {
        return _settings.CurrentValue.SomeString;
    }
}

If you want to know more about the use of IOptions vs IOptionsSnapshot vs IOptionsMonitor, Andrew Lock has some ↗ great ↗ articles ↗ on this matter, that goes into a lot more details than I do here.

Conclusion#

Hopefully, this provides an overview of how you can implement your own configuration provider. We have been using this in production for a few months, without any issues. This version polls Azure Blob Storage to check whether or not the ETag has changed. There are other options for handling changes in Azure Blob Storage.

You can enable a change feed for the storage account, and then dive into the change feed, and react when the specific blob has been updated. When a blob has changed, it is reflected in the change feed within minutes.

If “within minutes” is not fast enough, you can also get Blob Storage events. These events are pushed onto Azure Event Grid in near-realtime and allow for reacting to changes much faster. However we do not currently use Azure Event Grid, and we did not find it feasible to do so, just for this.

We can make further improvements to the configuration provider: We could make it async; We could make sure we will not trigger multiple updates at once. This is outside the scope of this article, but you can always take a look at the code ↗ for the NovoNordisk.Configuration.AzureBlob ↗ NuGet package to see how we ended up doing it - most should look familiar after reading this article.

References#

• Morteza Mousavi - A Refreshable SQL Server Configuration Provider for .NET Core ↗
• qinezh - Microsoft.Extensions.Configuration.AzureBlob ↗
• MilestoneTG - extensions-configuration-s3 ↗