Architecture

Overview

Datasource

Service Logging Library (SLL) is an API that allows the services to generate telemetry events based on common schema using server-side code. Part B Scenario of the Correlation Context will be used for associating events to scenarios and partners/customers/constituents.

The events generated by SLL could be consumed by different event processors such as Xpert, Xlens, and Asimov Cooker. Both Xpert and Xlens need to have their agent installed in the machine. They process the events locally and send the aggregated data to their respective data stores. Asimov cooker is an offline cooker (data processed outside the machine) that contains set of data pipelines that takes the raw events as input and produces cooked hourly streams. In Asimov pipeline, the events are sent to cosmos thru Cosmos Data Loader (CDL). Asimov cooker removes the duplicates, applying common schema and cluster them based on the event name and produces the hourly streams.

As Asimov Cooker allows us to handle bulk data (without any service restrictions), our Availability pipeline will be based on the SLL events cooked by Asimov Cooker. We will be also looking for integrating data from other sources such as Xlens, Application Insights, etc.

Data Processing

The data processing contains the following components.

1. Aggregate
2. Translate
3. Validate
4. Publish

Aggregate

Aggregation of the events is done in cosmos. It involves two steps.

1. Filter and Extract
2. Minute and Daily Aggregation

Required fields are extracted from Asimov Cooker View. Scenario ID and Partner ID are extracted from Correlation Context. If Correlation Context does not exist (for SLL version <5), We will be using ScScenario and ScPartner or Operation Name and Caller Name from Part C to calculate the Scenario ID and Partner ID, respectively. The extracted fields will be stored as a structured stream (RawEvents) under respective hour folder in cosmos. The minute level aggregation is done using the produced structure stream. The daily aggregation is done on the minute level aggregated stream. Both Aggregated streams are copied to the datastore.

Translate

In datastore, the aggregated data is joined with Scenario and Customers tables based on the scenario ID and Partner ID.

Validate

The data is validated (duplicates, invalid scenario Ids, etc), in datastore.

Publish

The validated data will be pivoted to Scenarios and published for the UI to consume.

Schema

Filtered fields

1. ScenarioID (from cC: ms.b.tel.scenario)
2. PartnerID (from cC: ms.b.te.partner)
3. ScScenario
4. ScPartner
5. OperationName
6. CallerName
7. Time
8. RequestType
9. Environment
10. RoleInstance
11. SLLVersion
12. Latency
13. RequestStatus
14. testHeader
15. quoteIsTest
16. isTest

Aggregated Stream

see Calculations for how it is calculated.

1. ScenarioID
2. PartnerID
3. Time
4. RequestType
5. Environment
6. TotalRequests
7. SuccessfulRequests
8. FailedRequests
9. AverageLatency
10. STID

Filters

These are the filters applied while collecting the SLL data from Asimov pipeline.

1. Data_BaseType = Ms.QoS.IncomingServiceRequest
2. CloudEnvironment - Excludes dev, int, ppe, non-prod, nonprod, sandbox, perf, test (exact match or contains prefixed with "-")
3. If cC is empty, scScenario
4. If cC and ScScenario is empty, OperationName from the list provided by the OMS team

5. Excluding the test traffic using testHeader, quoteIsTest and isTest fields. Considering the events which satisfy following string.IsNullOrEmpty(testHeader) && (string.IsNullOrEmpty(quoteIsTest) || quoteIsTest.ToLowerInvariant() != "true") && (string.IsNullOrEmpty(isTest) || isTest.ToLowerInvariant() != "true")

   WHERE (data_baseType == "Ms.Qos.IncomingServiceRequest" OR data_baseType == "IncomingServiceRequest")
       AND (
           string.IsNullOrEmpty(cloudEnvironment)
           OR (
               cloudEnvironment.ToLowerInvariant() != "dev" AND
               cloudEnvironment.ToLowerInvariant() != "int" AND
               cloudEnvironment.ToLowerInvariant() != "ppe" AND
               cloudEnvironment.ToLowerInvariant() != "non-prod" AND
               cloudEnvironment.ToLowerInvariant() != "nonprod" AND
               cloudEnvironment.ToLowerInvariant() != "sandbox" AND
               cloudEnvironment.ToLowerInvariant() != "perf" AND
               cloudEnvironment.ToLowerInvariant() != "test" AND
               !cloudEnvironment.ToLowerInvariant().Contains("-dev") AND
               !cloudEnvironment.ToLowerInvariant().Contains("-int") AND
               !cloudEnvironment.ToLowerInvariant().Contains("-ppe") AND
               !cloudEnvironment.ToLowerInvariant().Contains("-sandbox") AND
               !cloudEnvironment.ToLowerInvariant().Contains("-perf") AND
               !cloudEnvironment.ToLowerInvariant().Contains("-test")
           )
           OR (
               cloudEnvironment.ToLowerInvariant().Contains("prod") AND
               !cloudEnvironment.ToLowerInvariant().Contains("non-prod") AND
               !cloudEnvironment.ToLowerInvariant().Contains("nonprod")
           )
       )
       AND (
           string.IsNullOrEmpty(testHeader) &&
           (string.IsNullOrEmpty(quoteIsTest) || quoteIsTest.ToLowerInvariant() != "true") &&
           (string.IsNullOrEmpty(isTest) || isTest.ToLowerInvariant() != "true")
       )
   

Calculations

1. Calculating ScenarioId: The following priority is used to calculate the scenario Id: Ms.b.tel.ScenarioId -> ScScenario -> OperationName

2. Determining RequestStatus: if(RequestStatus <5) status=success else //(>=5 or empty) status =failure

3. Calculating Availability: Availability = 100*SuccessfulRequests/TotalRequests