Fundamentals

The DataSeeder service expects a set of data and a required execution order as input, and it will come out with a succeeded of faulted seeding process result as output. Both these parameters are designed to be provided via IConfiguration (because we need this thing to be flexible).

The way this translates into algorithmic steps is: the service will discover a set of DataSeeders, sort them according to a given order, and notify about the execution result.

DataSeeding modules

Including a new seeding in our service is quite simple but should follow a set of conventions in order to keep its integrity. First, each bounded context should provide its own data seeding module. In most cases, the project structure would look like this:

Spares/ITsynch.Suite.Spares.DataSeeding
├── SparesDataSeeder
├── SparesDataSeedingModuleOptions
├── SparesDataSource
└── SparesDataSeedingModule

Note

The usage of Spares here is just an example and should be replaced by your actual service / bounded context.

Lets break this down:

The SparesDataSeeder simply contains the seeding logic. Take into account that we'll be doing message-based data seeding here so we should be okay by implementing one of the base classes provided by the module:

public class SparesDataSeeder : MessagingDataSeeder<CreateOrUpdateSpare, SpareUpdated>
{
    public SparesDataSeeder(IRequestClient requestClient, IDataSource<SpareUpdated> dataSource)
        : base(dataSource, requestClient)
    {
    }
}

The SparesDataSeedingModuleOptions allows us to customize the data we intend to seed in each case (configuration to the rescue!):

public class SparesDataSeedingModuleOptions
{
    public CreateOrUpdateSpare[] Spares { get; set; } = Array.Empty<CreateOrUpdateSpare>();
}

The SparesDataSource should provide the data to be seeded in the form of the Message Type:

public class SparesDataSource : DataSource<CreateOrUpdateSpare>
{
    private readonly IOptions<SparesDataSeedingModuleOptions> options;

    public SparesDataSource(
        IOptions<SparesDataSeedingModuleOptions> options)
    {
        this.options = options;
    }

    public override IEnumerable<CreateOrUpdateSpare> Provide()
    {
        return this.options.Value.Spares;
    }
}

Note that the data source injects an options class for providing the required data.

The SparesDataSeedingModule will setup the configuration for us:

public class SparesDataSeedingModule : SuiteModule
{
    public override void SetupModule(IModuleBuilder builder)
    {
        base.SetupModule(builder);
        builder.AddOptions<SparesDataSeedingModuleOptions>();
        builder.ContributeAppConfiguration((_, config) =>
        {
            config.AddJsonFile("spares-seeding.json", optional: true);
        });
    }
}

We want to read configuration values from a specific file which will ultimately populate our options class with the data we expect to seed. This allows us to have different seeding configurations.

Important

Please make sure that all required components are registered in the DI container.

From here, we just need to include our data seeding module in the DataSeederApplicationModule composition:

internal class DataSeederApplicationModule : SuiteApplicationModule
{
    public override void SetupModule(IModuleBuilder builder)
    {
        base.SetupModule(builder);
        builder.DependsOn<DataSeedingModule, DataSeedingModuleOptions>(
            opts =>
            {
                opts.RunDataSeedingOnStartup = false;
                opts.ConfigureMessaging();
            });
        builder.DependsOn<SparesDataSeedingModule>();
    }
}

Ordering seeders

It is possible to define an order for the data seeders execution. This order can be configured via the DataSeeder appsettings.json configuration file:

{
    "DataSeedingModuleOptions": {
        "Order": ["create-or-update-spare", "create-or-update-department"]
    },

    "AllowedHosts": "*"
}

The service will look for all ordered seeders and run them first. All the seeders that were left out of the ordering list will be executed last.

Naming seeders

A seeder name is required for ordering. This can be achieved by adding the NamedSeeder attribute:

[NamedSeeder("create-or-update-spare")]
public class SparesDataSeeder : MessagingDataSeeder<CreateOrUpdateSpare, SpareUpdated>
{
    public SparesDataSeeder(IRequestClient requestClient, IDataSource<SpareUpdated> dataSource)
        : base(dataSource, requestClient)
    {
    }
}

Note

Our current naming convention uses the message name being published as seeder name.

Creating deployment manifests

During the Kubernetes deployment, we want to provide the IConfiguration data for our seeders using k8s Config Maps. This will allow us to define the Config Map differently for each environment, essentially allowing us to seed different data for each environment.

Data to be seed is stored in JSON files directly in the environment we want to seed it, or if it is used in multiple environments, it is stored in a Kustomize Component that is shared between them.

Defining data to be seed

As an example, let's seed data for our Spares in the localdev/hub environment. In the deployment/k8s/environments/localdev/hub/data-seeder/data you'll find the data that is being seeded for localdev/hub. We need to add a new JSON file there from which we will use a Kustomize Config Map Generator to generate a Kubernetes Config Map.

Let's create a directory and JSON file spares/spares-seeding.json with the below contents:

{
    "SparesDataSeedingModuleOptions": {
        "Spares": [
            {
                "CorrelationId": "3d000000-ac11-0242-b225-08dc4111cd5a",
                "DisplayName": "DOSING PUMP GAMMA/X"
                // [..]
            }
        ]
    }
}

Next, we need to define our Config Map Generator in the Data component kustomization yaml, which is located at deployment/k8s/environments/localdev/hub/data-seeder/data/kustomization.yaml

Note

These paths can be extrapolated to any environment, since conventionally, they are the same.

Add an entry to the configMapGenerators array in the yaml file:

# [..]
configMapGenerator:
    # [..]
    - name: spares-seeding-config
      files:
          - './spares/spares-seeding.json'

When this gets build by kustomize, it will generate a Config Map named spares-seeding-config with our JSON content. Now this is great and all, but we've never told it to use the config map for anything.. so this would just create the config map, but it won't be referenced by any service.

Particularly, we need the Data Seeder to volume mount this JSON file so dotnet discovers it.. It's simpler than it sounds:

Patching the Data Seeder to use our Config Map

In our kubernetes application manifest, deployment/k8s/application/spares, we will add a Kustomize Component which includes a Kustomize Patch that will reference the Config Map in the Data Seeder when the Component is imported.

In the deployment/k8s/applications/spares, create a directory and yaml file for our component seeding/kustomization.yaml.

apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component

patches:
    - path: './config-patch.yaml'
      target:
          kind: Job
          name: data-seeder

Next, we need to define the config-patch.yaml file being referenced by the component right next to the kustomization.yaml file we've just created. This is where it gets tricky..

This file represents a JSON patch to be applied to a resource, the data-seeder Job. the path is the YAML path to modify, and the value is the new value the path will have in the target resource.

To sum up:

1. We need to reference our Config Map by its name, spares-seeding-config
2. The mountPath and subPath have the JSON name we defined in dotnet.

- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
      name: spares-seeding-config
      mountPath: /home/itsynch/spares-seeding.json
      subPath: spares-seeding.json

- op: add
  path: /spec/template/spec/volumes/-
  value:
      name: spares-seeding-config
      configMap:
          name: spares-seeding-config
          optional: true

That great, now we have our Component that will patch the Data Seeder. But we need to reference it in our localdev/hub environment so it gets used.

Our new component needs to be referenced in the environment's Data component, the one we modified earlier located at deployment/k8s/environments/localdev/hub/data-seeder/data/kustomization.yaml

# [..]
components:
    # [..]
    - '../../../../../applications/spares/seeding'

We recommend to test your kustomization before running the data seeder deployment so you can be sure everything is properly set. By running kustomize build --enable-alpha-plugins ./k8s/environments/localdev/hub/data-seeder you can check how the deployment chart will be generated. If everything is correct you should be able to see your new volume mount and your JSON seeding data included via the Config Map.

Running the data seeder

The data seeder is set to execute anytime you run the suite-installer for your deployment. Nonetheless, we may want to run it ourselves; either because of a transient failure that stopped the data seeder to successfully complete the first time or because we made some changes in a JSON file and need to apply them. In any case, this can be accomplished by running

suite-apply ./k8s/environments/localdev/hub/data-seeder

The data seeder is run as a kubernetes job, meaning you can check its status by running kubectl get jobs, you will get this output:

NAME          COMPLETIONS   DURATION   AGE
data-seeder   0/1           5s         5s

The job will schedule new pods for the data seeder until one of them succeeds to complete its task, or until it reaches a certain limit of failed attempts. A succeeded data seeding job will manifest as a 1/1 completion.

NAME          COMPLETIONS   DURATION   AGE
data-seeder   1/1           40s        94s