Skip to content

RFC: Custom fields v2

Problem statement

Our strong suite is (mostly about) customized solutions that are based on some common denominator across our multiple clients in the cruise line industry. However, sometimes said customizations cannot be refactored/fitted into a common domain concept since it's just too custom. This could lead to potentially diverging code bases and deliverables based on apps, clients or even departments within the same company.

In order to increase the degree of customization we offer our clients, our Suite 3.0 based services should support user definable custom fields (CF from here onward). Admin level users should be able to pick an entity owned by our applications (i.e. Inspections or Service Orders) and define an arbitrary number of CF, which should then be mostly indistinguishable from "native fields".

Duration

Deadline for this RFC is set to the second Scrum of Scrum meeting scheduled after officially opening the RFC.

Current state

  • AMOS offers some degree of customization with something akin to CF, its UserDef fields (e.g. userDefText1). The amount and type of fields per entity is fixed (i.e. you only get 4 userDefText fields for the Work Order entity).
  • Some applications like AIMS provide customizable labels both for application owned custom fields as well as for AMOS owned user def fields (pending validate with RM)

  • MPM has a more generic approach for managing CFs, which was built from the ground-up. Users can pick any entity and create new CFs for it. A Custom Field Definition allows the user to set:

    • name
    • type: one of string, CLOB, number, date, money, option
    • default value
    • attributes: one of required, hidden

    MPM custom fields are supported server side by adding an interface and performing some ORM level configuration for the entity that should "support" that feature.

    Client side, Custom Field Values are sent in a customFields property along with the entity payload, the data type being a dictionary keyed by Custom Field ID. They are then displayed based on a set of generic form control fields.

    Architecturally, they resemble an EAV schema, however there's one table for the definition, and one for the values. Do note no FK are possible for the value tables, due to its custom nature (we would need to dynamically target the FK constraint target table based on the value of another column)

Proposer

RAR - Ramiro Adolfo Rivera - Suite 3.0 Team

Detail

Suite 3.0 considerations

Some terms first:

  • Consumer Service: A service which owns/manages an entity that requires CF, and consumes the CF library to implement it. These are usually Backend Services
  • App service or BFF: a service which works as an API gateway or coordinates otherwise application specific business use cases e.g. AIMS, AIMS BFF.
  • Satellite service: a service in charge of performing some client or environment specific operation, which enriches or generates data which would be later ingested into the different Backend or App services e.g. some custom satellite service for a Crystal specific feature, a sync service between AMOS and Identity for AMOS users, etc.
  • Admin center: combination of frontend and backend service whose responsibility is to provide an administration point for a given environment to some administrative type of user.

By the very nature of the Suite 3.0 based apps, custom fields make a lot of sense. On a first discussion with LC and JMR we arrived at the conclusion that there shouldn't exists a separate service for handling all suite custom fields, but that it would rather act as a sort of add on or plugin, augmenting already existing or new consumer services.

Why? Because it makes sense :P. In all seriousness, consumer services should allow enriching their entities with Custom Fields, which the consumer service will barely acknowledge exists.

In a sense, the Assets service will just receive and hand off processing of Custom Field values addition/edition/deletion to the Custom Fields library. Why? Because a consumer service should contain only domain specific and client/app agnostic logic, and if something is a custom field, then how specific to the domain actually is that field? However, it's the perfect place for managing and storing the values, since it's in the heart of all events and processing sagas in the microservices network.

Both App or Satellite service are better suited for using and consuming Custom Fields (until maybe a later point in time where upgrading the Custom Field to Client Field or similar, would make sense, if a client specific satellite service is created). The reason for this is twofold: on one hand, frontend apps like the AIMS frontend, would ideally be aware of Custom Field Values existence, thus allowing to CRUD them alongside native values (so a satellite service can read/update them while doing some process, without requiring modifications to its frontend app). On the other hand, not all Custom Fields need be visible to the end user, they could be used as inter service communication only metadata, which would be readily available for all other services consuming the Domain Entity data.

The Admin center users will be able to browse all Custom Fields defined for any entity type, as well as create/update Custom Field definitions from it.

Server-side

Implementation would follow MPM's database architecture. Each implementing service would get two new tables on its store: Custom Fields and Custom Field Values. Each service would own said data for its entities. Adding a new Custom Field type should ideally be simple (not necessarily easy) and said process should adhere to the O in S.O.L.I.D..

Warning

Until the use case presents itself, no consumer service will be able to define and register their own Custom Field type.

This library could provide utilities to be used in the Admin center, so that it can automagically listen to Custom Field definitions changes

Client-side

Out of RFC scope

While out of scope for this RFC, ideally Custom Fields would be displayed and updated on the different frontend applications the same way a Native Field is. This is, for the end user, working with a Custom Field should feel no different than for example updating an Inspection's description.

Pitfalls to avoid

Not everything is a Custom Field, nor should it be. We don't want Custom Entities. Custom Entities are the worst version of the Anemic Domain anti-pattern I can conjure in my mind. Initially, Custom Fields should serve as a key value store for entity metadata specific to a client or environment.

Only if it makes sense in the context of a client, or if it gives us time for a proper implementation (i.e. our development speed cannot match the required velocity of feature delivery), should Custom Fields be suggested/introduced to the end user.

Proposal

Database Schema

In the future, Custom Fields library should provide database mapping utilities at 3 different level/points:

  1. Admin Center
  2. Backend Service
  3. Search Service (or any service with materialized views)

This section only addresses item (2).

Database diagram

3 tables will be added to any Consumer Service of Custom Fields:

Custom Fields Definition

Contains Custom Field definitions or metadata. Each Custom Field Definition belongs to a single entity type and can be identified either by its ID or (more likely to be used) its string key. e.g. itsynch.suite.users.replicated-to-amos

Custom Field Values

Custom Field Values will contain a row for every Entity-CustomFieldDefinition combination. It has FK to its respective Definition and Collection (see next section).

RawValue will contain some basic information about serialization version and the actual serialized value. e.g.

Text Only
1
2
3
4
{
    version: "1",
    value: "2021-06-07 12:42:13",
}

Additionally, a (de-normalized) CustomFieldType column will be available to be used as an Entity Framework Core discriminator column on TPH strategy.

Custom Field Values Collection

Custom Field Values Collection is basically a workaround for an issue MPM's CF implementation had. Basically, we could not have relational integrity enforced by Foreign Keys, since each Custom Field Value would "know" its target entity by a TargetEntityId property, and the target entity schema's had no changes.

On this implementation, each IEntityWithCustomFields implementor's backing database table will get a new column CustomFieldValueCollectionId, a FK pointing to its corresponding Custom Field Values Collection table row.

This way, queries will be more efficient and we get integrity checks where we need them the most.

HTTP presentation

Consumer services (and any other service TBH) implementing the Custom Fields library feature, will expose the following new HTTP endpoints (names provided for readability purposes, endpoints are REST like named, but Suite 3.0 auto wired HTTP App Services will most likely generate different names)

Definitions

GET /custom-fields

HTTP GET endpoint for retrieving all custom fields definition defined in this service.

Entities with Custom Fields

When an entity fulfils the rules for "implementing Custom Fields", then all GET HTTP calls to that entity endpoint (e.g. api/v1/inspections/getAll) should include in their payload the Custom Field Values for that entity..

SEE Out of scope

Messaging

Custom messages related to Custom Fields will not be raised, but rather Custom Fields related payload will be by default (could be configurable) including in all/some consumer service messages payload.

List of messages:

  • CustomFieldDefinitionUpdate: handled by consumer services, updates an existing Custom Field definition. If everything is OK, reply to the void with CustomFieldDefinitionsUpdated
  • CustomFieldDefinitionsUpdated: handled (initially) by Admin Center, to keep its state up to date regarding existing Custom Fields definitions through the Suite
  • CustomFieldLibraryInitialized: handled by Admin Center, sent by consumer services on initialization

Constraints/Considerations

Here follows a list of unordered assumptions for this RFC, please share any comments about them. They are not an exhaustive list by no means:

  • For the server-side implementation, only an EF Core (latest version) implementation will be written.
  • NH is out of scope.
  • When in need of JSON de/serialization, the same serialization tool as the rest of the suite will be used (be it System.Json or Newtonsoft.Json).
  • C# extension methods would be avoided as much as possible, but when required, attention will be payed in order to make them unit test friendly.
  • Unit test utilities might be provided if within time budget
  • Default implementation methods for interfaces might be leveraged if it makes sense (a.k.a. if its fun for me to do so)
  • Hooks/messages for intercepting/interacting with the processing of Custom Fields from the satellite/application services will be properly documented and shared when/if appropriate
  • Consumer service level configuration of Custom Fields should be kept to a single place (possible startup or ConfigureServices method)
  • Entity level configuration in order to support custom fields will (be tried to) be kept at a minimum.

Examples

Usage

All versions consume the following common symbols:

C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
// Non generic interface for cases where the type is a hindrance
interface ICustomFieldValue
{
    string CustomFieldId { get; set; }

    string CustomFieldValueId { get; set; }

    object? RawValue { get; set; }
}

interface ICustomFieldValue<T> extends ICustomFieldValue
{
    T? Value { get; set; }
}

interface ICustomFieldsManager
{
    string GetString(string customFieldId);

    // Other strong typed methods for "every" custom field type available
    // ....

    // Escape hatch
    ICustomFieldValue<T> GetValue<T>(string customFieldId);

    // Setter
    void SetValue(string customFieldId, object value);

    // Shortcut to setting null OR removing the custom field value tuple all together
    void ClearValue(string customFieldId);
}

The idea of the ICustomFieldsManager is to limit boilerplate code for handling common operations across entities with Custom Fields, as well as simplifying unit testing.

Approach 01 - IGNORE

Approach 01 leverages default interface implementations for providing the ICustomFieldsManager

C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
interface IEntityWithCustomFields
{
    // Implementable navigation property so EF core can map it.
    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }

    // Here we can access CustomFieldValues, but not vice versa
    public ICustomFieldsManager CustomFieldsManager =>
        new CustomFieldsManager(CustomFieldValues) as ICustomFieldsManager;
}

public class ServiceOrder : IEntityWithCustomFields, BaseSagaStateMachineInstance
{
    public ServiceOrder(Guid correlationId)
    {
        this.CorrelationId = correlationId;
    }

    public string? AdditionalInformation { get; private set; } = null;

    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }

    public void ChangeAdditionalInformation(string? newAdditionalInfo)
    {
        // Unfortunately, with this approach, we need explicit castings...

        ((IEntityWithCustomFields)this).CustomFieldsManager.SetValue(
            "itsynch.suite.service-orders.42",
            newAdditionalInfo);

        this.AdditionalInformation = newAdditionalInfo;
    }
}

Approach 02 - IGNORE

Approach 02 leverages extension methods instead. Right now I cannot think of a case where this could limit our unit tests, but that is usually the case with logic-heavy extension methods. Thus, the ICustomFieldsManager stays.

C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
interface IEntityWithCustomFields
{
    // Implementable navigation property so EF core can map it.
    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }
}

public static class IEntityWithCustomFieldsExtensions
{
    public static ICustomFieldsManager CustomFieldsManager(
        this entity IEntityWithCustomFields
    )
    {
        return new CustomFieldsManager(entity.CustomFieldValues);
    }
}

public class ServiceOrder : IEntityWithCustomFields, BaseSagaStateMachineInstance
{
    public ServiceOrder(Guid correlationId)
    {
        this.CorrelationId = correlationId;
    }

    public string? AdditionalInformation { get; private set; } = null;

    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }

    public void ChangeAdditionalInformation(string? newAdditionalInfo)
    {
        this.CustomFieldsManager.SetValue(
            "itsynch.suite.service-orders.42",
            newAdditionalInfo);

        this.AdditionalInformation = newAdditionalInfo;
    }
}

Approach 03 - IGNORE

Approach 03 leverages is basically Approach 01 sans the ICustomFieldsManager

C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
interface IEntityWithCustomFields
{
    // Implementable navigation property so EF core can map it.
    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }

    // Default implementation methods for all methods originally belonging
    // to ICustomFieldsManager
    string GetString(string customFieldId);

    // Other strong typed methods for "every" custom field type available
    // ....

    // Escape hatch
    ICustomFieldValue<T> GetValue<T>(string customFieldId);

    // Setter
    void SetValue(string customFieldId, object value);

    // Shortcut to setting null OR removing the custom field value tuple all together
    void ClearValue(string customFieldId);
}

public class ServiceOrder : IEntityWithCustomFields, BaseSagaStateMachineInstance
{
    public ServiceOrder(Guid correlationId)
    {
        this.CorrelationId = correlationId;
    }

    public string? AdditionalInformation { get; private set; } = null;

    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }

    public void ChangeAdditionalInformation(string? newAdditionalInfo)
    {
        // Unfortunately, with this approach, we need explicit castings...

        ((IEntityWithCustomFields)this).SetValue(
            "itsynch.suite.service-orders.42",
            newAdditionalInfo);

        this.AdditionalInformation = newAdditionalInfo;
    }
}

Approach 04

Approach 04 is just some extension method syntactic sugar on top of Approach 03.

C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
public static class IEntityWithCustomFieldsExtensions
{
    public static IEntityWithCustomFields WithCustomFields(
        this IEntityWithCustomFields entity
    )
    {
        return entity;
    }
}

public class ServiceOrder : IEntityWithCustomFields, BaseSagaStateMachineInstance
{
    public ServiceOrder(Guid correlationId)
    {
        this.CorrelationId = correlationId;
    }

    public string? AdditionalInformation { get; private set; } = null;

    public ICollection<ICustomFieldValue> CustomFieldValues { get; set; }

    public void ChangeAdditionalInformation(string? newAdditionalInfo)
    {
        this.WithCustomFields.SetValue(
            "itsynch.suite.service-orders.42",
            newAdditionalInfo);

        this.AdditionalInformation = newAdditionalInfo;
    }
}

Approach 05 (CHOSEN)

Entities will implement the IEntityWithCustomFields interface. A special repository/service will be injected in order to manipulate said entities, so that we can have DI capable symbols from the start (otherwise, we'd lose for example logging capabilities).

Additionally, said interface will only as a marker for Suite Decorators to take action on.

C#
1
2
3
4
interface IEntityWithCustomFields
{
    CustomFieldValuesCollection CustomFieldValuesCollection { get; set; }
}

By using Suite Decorators, tables for entities implementing IEntityWithCustomFields will get a new FK column to Custom Field Values Collection table (see database schema section).

Configuration

Module

Extension methods will be provided to add Sagas to the Admin Center and/or Consumer Services, in order to install the Library.

If further configuration is required, the IOptions<T> pattern will be used.

EF Core

Entities implementing the required interfaces will be automagically configured using Suite Decorators

Testing

This section covers how Consumer Services can test their Custom Fields relying flows, it doesn't contain info on how the Custom Fields Library will be tested.

With all the approaches presented before, testing arrangement boils down to setting whatever our test scenario demands on CustomFieldValues property, which is just an ICollection<ICustomFieldValue>.

Assertions are a bit more complicated. One untested idea could be doing something like:

C#
1
2
3
4
public class TestServiceOrder : ITestEntityWithCustomFields, ServiceOrder
{

}

This way, ITestEntityWithCustomFields could intercept calls that rely on building an ICustomFieldsManager and replace it with a more assertion-friendly version under the hood.

Saga utilities

Saga utilities are methods to be used within the normal saga of the Consumer Service entities. Sagas for managing Custom Field Definitions and Values are not described here.

Note that message interfaces inheritance is not something we want to do. So all saga utilities will either take a lambda function to determine from where to read/write the custom fields related information, or need to be specified where required

C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
internal static class ServiceOrderSagaStateMachineExtensions
{
    public static EventActivityBinder<ServiceOrder, TData> PublishServiceOrderUpdated<TData>(
        this EventActivityBinder<ServiceOrder, TData> builder)
        where TData : class
    {
        return builder.PublishAsync(ctx => ctx.Init<ServiceOrderUpdated>(new
        {
            ctx.Instance.CorrelationId,
            ctx.Instance.AdditionalInformation,
            CustomFields = ctx.Instance.WithCustomFields.ToCustomFieldsPayload()
        }));
    }
}
C#
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
// Proposed method
internal static class EntityWithCustomFieldsSagaStateMachineExtensions
{
    public static EventActivityBinder<TInstance, TData> UpdateCustomFieldValues<TInstance, TData>(
            this EventActivityBinder<TInstance, TData> binder,
            Func<BehaviorContext<TInstance, TData>, IDictionary<string, ICustomFieldValueUpdate>> customFieldsPayloadGetter)
            where TInstance : BaseSagaStateMachineInstance
            where TData : class
    {
        return binder.ExecuteServiceAsync(b =>
        {
            return b => b.OfType<ICustomFieldValuesSagaUtilitiesService>((service, ctx) =>
            {
                var customFieldValuesUpdatePayload = customFieldsPayloadGetter.Invoke(ctx);
                return service.UpdateEntityCustomFieldValuesFromUpdatePayload(
                    ctx.Instance,
                    customFieldValuesUpdatePayload);
            });
        });
    };
}

// Usage example

Initially(
    When(UpdateRequested)
        // ....
        .UpdateCustomFieldValues(ctx => ctx.Data.CustomFields)
        .PublishServiceOrderUpdated();

Out of scope

Here are some of the capabilities/features that will be out of scope for this RFC implementation:

  • Simplified querying capabilities for AppServices: if you want to get an IEntityWithCustomFields that actually contains its Custom Fields back from a repository query, you'd better manually add the required Specification or do a separate query (for testing purposes, or similar)
  • Utilities to listen/emit/etc messages and integrate with Custom Fields at the "Search Service Layer" (read: materialized views)
  • Providing message inheritance for messages that have Custom Fields related fields, for now no inheritance is provided. Lambdas are used as a manual workaround for developers. Pending investigation on MT side is required to determine if some message inheritance is possible.