1. Plugin Architecture Overview
A plugin consists of 2 parts:
-
A plugin manifest - defines what data a crawler will submit.
-
A crawler executable - crawls data and submits it to XL Impact.
1.1. Plugin Manifest
Plugin Manifest contains:
-
Plugin metadata - id, human readable name, version, etc..
-
Types of entities that a plugin is going to submit.
-
Human readable names for entities, if they differ from generic names. For example, the Git Annotated Tag is usually called Release in GitHub.
-
Relations between entities that this Plugin defines.
-
Additional filtering fields that can be used when configuring a project in UI.
-
Crawler filters to specify which data to include/exclude when crawling.
1.2. Relations
The relation definition is very important part of XL Impact, its business value is based on it’s ability to find related events coming from different data sources and aggregating them together. An example of a relation would be a commit message that mentions a ticket for which the change was made.
Since plugins operate separately, relations should be discovered between entities coming from different plugins as well, including cases when the plugin developers do not know each other. Like in the example above, the commit message comes from some version control plugin, while the ticket IDs come from a ticket management plugin.
Relations are built by putting together entity content and relation definitions from different plugin manifests. It is important to understand, that the relation definitions from all plugins are applied to every entity, so it is important to set the correct filters on your relation definition. Each relation definition can be incoming and outgoing (relations are directional), and some filters are set independently for incoming and outgoing relations, while others can be set to require similarity (e.g. entities on both sides should be from the same dataSourceId or from same pluginId).
When applying a relation definition to an entity, a set of 'secrets' is produced. Incoming and outgoing secrets are produced separately. Then secrets are matched against each other, and same secrets of opposite nature (incoming vs outgoing) are considered established relations.
The configuration is complex.
There are 2 basic types of relation endpoint types: precise and pattern-based. The precise relation endpoint is one that can be precisely identified using any sort of ID. For example, if somebody references a commit, which can be identified by sha hash, we can say that on the commit side of relation, we have a sure value for that sha hash. However, the other side of relation it depends. For example, Jenkins may mention a commit in its console output - in this case our only chance to discover a relation is to regexp the whole console output using a generic regular expression, that will match anything that looks like sha hash and select the hashes that match commits from our database.
It is also possible to configure "via" relations. For example, a pull request mentions a sha of a squash commit and also the sha of the head of pull request that was squashed. This should result in the building of a "parent-child" relation between two commits, but none of these commits knows directly about each other.
A relation manifest contains the following groups of properties:
-
Filters for the "incoming" and "outgoing" parts of relation - from and to, HalfRelationMatcherConfiguration
-
Filters for the combination of "incoming" and "outgoing" - see properties starting with requireSame… and requiredTimeRange
-
Details about what exact fields should be captured from entities and what patterns should be used. See a detailed description for each property in following links.
Inside the entity unified model document, the convention is to put all relation-related content in the "other.relations.*" subdoc.
Examples of relations in the plugin manifest:
{
"relations": {
"parent": { // parent commit
"type": "precise",
"requireSameDataSource": true, // only useful within same data source
"from": { // filters for outgoing relation side
"requireMe": true, // this relation is only applicable to commits coming from this plugin
"types": [
"COMMIT" // this relation is only applicable to COMMIT entities
]
},
"to": { // filters for incoming relation side
"requireMe": true, // this relation is only applicable to commits coming from this plugin
"types": [
"COMMIT" // this relation is only applicable to COMMIT entities
]
},
"relationType": "parentCommit", // type of relation
// the following two lines mean, that the other.relations.parents value on one side
// should match the ID value on another side. XL Impact allows either of them to be
// an array or scalar value. For an array - any one of the values should match.
"fromField": "other.relations.parents",
"toField": "id"
},
"mergeCommitReference": {
"type": "via", // this is the aforementioned via type
"requireSameDataSource": true, // only works for entities from same data source
"from": {
"requireMe": true, // only works for entities coming from this plugin
"types": [
"COMMIT" // only works for an entity of type COMMIT on the outgoing side
]
},
"to": {
"requireMe": true, // only works for entities coming from this plugin
"types": [
"COMMIT" // only works for an entity of type COMMIT on the incoming side
]
},
"relationType": "parentCommit",
"fromField": "id", // both commits are referenced by their IDs
"toField": "id",
"via": {
"requireMe": true, // the middle part of this relation should also come from this plugin
"types": [
"PULL_REQUEST" // the middle type of this relation is Pull Request
]
},
// the following two lines specify which fields of the middle part should match
// in this case the ID of one commit should match the other.relations.mergeCommitId
// of a pull request and the ID of another commit should match the
// other.relations.mergeCommitId of the same pull request
"viaFieldToMatchFrom": "other.relations.mergeCommitId",
"viaFieldToMatchTo": "other.relations.lastCommitId"
},
"commitReference": {
// this relation is to find mentions of commit shas in the Jenkins console output
"type": "pattern", // regular expression matching on Jenkins side
"from": {
"types": [
"BUILD" // only requirement to the outgoing side is that type is BUILD
]
},
"to": {
"requireMe": true, // on the incoming side, the commit should come from this plugin
"types": [
"COMMIT" // only commits are allowed on the incoming side
]
},
"relationType": "commitReference",
"regexp": "(^|[^0-9a-f])([0-9a-f]{40})($|[^0-9a-f])",
"windowSize": 42,
"beforeGroup": 1,
"afterGroup": -1,
"meaningfulGroup": 2,
// allowed values are FROM and TO, FROM means that the outgoing
// side of the relation is matched using regular expression
"regexpIsOn": "FROM",
"patternFields": [
"*" // all fields on the BUILD side will be scanned
],
// on the COMMIT side we give field with sha that should be matched
"preciseField": "other.relations.sha"
}
}
}2. Types of Entities
2.1. Domains
There are 4 domains:
-
BUILD - servers making CI, CD - Jenkins and XL Release, etc
-
SCM - code versioning systems - GitHub, Subversion, etc
-
TRACKER - Jira, ServiceNow, Zendesk
-
COMMON - for common items. Right now, this is used only for the FIELDS type
2.2. Build domain
The following items belong to the build domain.
2.2.1. BUILD
Build is an automated job that was completed. For example, a Jenkins Build or an XL Release task.
-
It can be successful or unsuccessful
-
It has a start time
-
It has a particular duration
-
In most cases it is related to the BUILD_TEMPLATE
-
Usually BUILD is building a particular COMMIT or making a COMMIT itself. Thus it can be related to COMMIT
2.3. SCM domain
The following items belong to the commit domain.
2.3.1. COMMIT
This is a commit - a piece of work done by engineers.
-
Has a date
-
Has an identifier
-
Has one or more parents (root commits may not have parents)
-
Is always successful
-
Has a commit message which can be used to build relations with e.g. Jira tickets
2.3.2. PULL_REQUEST
This is group of commits annotated with dates.
-
Has a date
-
Has a duration if it was closed, and has no duration if it was not closed
-
Can be 'squashed' in the end so that the history is lost. It is plugin developer responsibility to specify a relation from the merge (sqhashed) commit to the head of PR branch, and XL Impact provides way to do that, see examples.
2.4. TRACKER domain
The following item belongs to the tracker domain.
2.4.1. TICKET
This is version of an issue or task in the issue or task tracker. In Jira, this maps to an issue version. A crawler must submit all information about an issue in order to track state transitions and analyze the issue lifecycle.
-
Has a version ordinal number - integer starting with 0 (first version when the issue was created)
-
Has a status, assignee, and other fields, some of which are multiple-entry
-
Has timestamp - for the first version when the issue was created, for other versions - when edit appeared
2.5. COMMON domain
The following item belongs to the common domain.
2.5.1. FIELDS
These are custom fields defined in the data source application (e.g. Jira custom fields).
=== Unified Model Data from Crawler point of view
2.5.2. Artefacts
A crawler pushes buckets of data called artefacts to XLI. An artefact is a package that contains one or more entities of one of known types.
Each artefact has its own ID, and each entity has its own ID. The artefact ID must be unique among all other artefacts within the pluginId/dataSourceId space. The entity ID should be unique among all other entities within the pluginId/dataSourceId/entityType.
A crawler may push one or more of the same entity into multiple artefacts, in error. It is the responsibility of the crawler developer to implement a plan for this scenario.
Store only one entity per artefact, unless, entities in the data source system are grouped together into an atomic group. In this case, place all grouped entities into one artefact.
For example, XL Impact has an entity for Jira Issue Version (TICKET), this means that every separate revision of Jira Issue is a separate XL Impact entity. Jira gives all version in one block, as a changelog for the issue, it does not allow to search for revisions, and it only tracks changes for issues (using the updatedAt field). In this case, one artefact is one Jira Issue, and one entity is a revision of an issue.
2.5.3. Additional fields
XL Impact defines only a minimum amount of fields for each entity. All other fields that are exposed by the source data system and are useful for project definition are additional or custom fields.
Additional fields are fields that are known to plugin developer when plugin is being created. For example, additional fields for GitHub would be: organization name and repository name. In an Jira example, an additional field would be project key. For more information, see the custom fields section below.
Additional fields must be declared in the plugin manifest.
2.5.4. Custom fields
Custom fields are specific to a particular instance (installation) of data source system. For example, fields in Jira that can be individually configured by administrator.
A crawler must submit custom field definitions, such as: names, and types, to XL Impact before submitting any artefacts that contain these fields. XL Impact will reject artifacts with undefined custom fields.
Custom field definitions are submitted using the special FIELDS type.
Inside the UM JSON custom field have a structure similar to additional fields.
2.5.5. UI URLs
XL Impact gives the user direct URLs, where applicable, so that user can click and get to particular commit in GitHub, or job in Jenkins.
In order for this to work:
-
The plugin manifest must specify a URL template for each entity type.
-
When configuring the data source for a plugin, the user must enter the UI base URL for that installation of a data source system (e.g Jira). This ensures that only the base URL in the credentials needs to be changed if Jira or Jenkins is moved to another URL.
-
Each entity must give parts to put into a URL template - e.g. GitHub repository name, commit sha, etc.
2.5.6. Data versioning
Some data types cannot be crawled twice. For example, Jenkins builds are wiped automatically by Jenkins so that the data inside XL Impact can appear unique.
A Jenkins crawler version 2.0 may provide data in a different manner then a Jenkins crawler version 1.0. In some cases, the installer may wish to migrate data from a version 2.0 to version 1.0. A Jenkins crawler version 2.0 supports this but this feature is not implemented in XL Impact and most likely will not be implemented until it is requested. If you need this feature - ask.
2.5.7. Unified Model Document (UDM) structure
The UDM structure depends on document type, see the examples below.
The following are common variables for all types except FIELDS
- id - any unique string
- version - ordinal version number, integer, starting from 0. So timestamp should not be used as version.
- title - any human readable string
- startMs and finishMs - some entities like COMMIT only have startMs but not finishMs
- urlParts - this subdoc is described in the UI URLs section
- additionalFields and customFields similar subdocs which have fieldName: fieldValue pairs. See Additional Fields and Custom Fields for more details
- other - subdoc for everything else, according to current convention this only has only 2 reserved names:
- relations - subdoc for relations-related data. For more information, see the Relations section.
- raw - we recommend that this variable is used to the store data that the crawler gets from the source system. XL Impact does not use this data immediately, but it can be useful for (a) improvements and further development of XL Impact, (b) getting more data (see the Data Versioning section)
BUILD example
"JenkinsBuild.v1": { // JenkinsBuild stands for pluginId, v1 stands for plugin version
"BUILD.v1": [ // BUILD stands for entity type, v1 stands for Unified Model version
{
// After a period of time, Jenkins build numbers may restart from 1.
// To accommodate this a timestamp should be added to the id variable.
"id": "XL Insight/Releases/Release XLI/47-1514818073064",
"title": "XL Insight » Releases » Release XLI #47",
// This variable is used for the failure chart - to see which types fail more often then others
"buildType": "freestylebuild",
"startMs": 1514818073064,
"finishMs": 1514819648273,
// Authorized values are "succeeded", "failed", "running", "unknown", "skipped", "planned"
"status": "succeeded",
"urlParts": {
"path": "XL Insight/job/Releases/job/Release XLI/47"
},
"other": {
"relations": {
"buildId": "XL Insight/Releases/Release XLI/47",
"jobId": "XL Insight/Releases/Release XLI"
},
"raw": { // We recommend that this variable is used to the store data that the crawler gets from the source system.
"consoleOutput": "..."
...
}
}
}
]
}
}BUILD_TEMPLATE example
{
"JenkinsBuild.v1": {
"BUILD_TEMPLATE.v1": [
{
"id": "XL Insight/Releases/Release XLI",
"title": "XL Insight » Releases » Release XLI",
"urlParts": {
"path": "XL Insight/job/Releases/job/Release XLI"
}
}
]
}
}COMMIT example
{
"GithubPlugin.v1": {
"COMMIT.v1": [
{
"id": "xebialabs/xl-insight/36a7a5130eda29b91191331a3069d0b89b2fbe42",
"authorName": "Bob White",
"authorEmail": "bwhite@example.com",
"message": "ABC-123 remove temporary visualizations",
"startMs": 1517409877000,
"urlParts": {
"repository": "xebialabs/xl-insight",
"sha": "36a7a5130eda29b91191331a3069d0b89b2fbe42"
},
"other": {
"relations": {
"parents": [
"xebialabs/xl-insight/c851fe00894cf629b195ae8bf684973f8c107c6b",
"xebialabs/xl-insight/z098fe00894cf629b195ae8bf684973f8c107c8c"
],
"sha": "36a7a5130eda29b91191331a3069d0b89b2fbe42"
},
"changedLines": 7,
"raw": {
...
}
}
}
]
}
}PULL_REQUEST example
{
"GithubPlugin.v1": {
"PULL_REQUEST.v1": [
{
"id": "xebialabs/xl-insight/1110",
"title": "ABC-123 remove temporary visualizations",
"startMs": 1485876118000,
"finishMs": 1486058967000,
"state": "merged", // Authorized states are "merged", "open", "rejected", "declined"
"urlParts": {
"repository": "xebialabs/xl-insight",
"id": "1110"
},
"other": {
"relations": {
"commits": {
"to": [ // list all commits here
"xebialabs/xl-insight/93a0c0a882d1d8d12372c44aacf475db5bb51076",
"xebialabs/xl-insight/0057ceb09e8e2538a5271307e914eb58d67f9ee1",
"xebialabs/xl-insight/01770c6e4a9ea05e66ce4765460c7eb885e888cb"
]
},
"lastCommitId": "xebialabs/xl-insight/01770c6e4a9ea05e66ce4765460c7eb885e888cb",
"mergeCommitId": "xebialabs/xl-insight/3b6139768c3db92b12f69c7dbe56b5b184410e3e"
},
// you are free to use "other" as you wish
"commits": 3,
"description": "asdf",
"raw": {
...
}
}
}
]
}
}LABEL example
{
"GithubPlugin.v1": {
"LABEL.v1": [
{
"id": "xebialabs/xl-insight/fe3d490a17f6823d99911a21f5ef1f527a2af87c",
"startMs": 0,
"title": "xl-insight-7.2.0-alpha.2",
"urlParts": {
"repository": "xebialabs/xl-release",
"title": "xl-insight-7.2.0-alpha.2"
},
"other": {
"relations": {
"commit": {
"to": [
"xebialabs/xl-insight/01770c6e4a9ea05e66ce4765460c7eb885e888cb"
]
}
},
"raw": {
...
}
}
}
]
}
}TICKET example
{
"JiraPlugin.v1": {
"TICKET.v1": [
{
"id": "91743",
"version": 0,
"startMs": 1506553073203, // TICKET does not have finishMs
"title": "ABC-123: Remove temporary visualizations",
// used for reports - any string value is accepted
"ticketType": "Story",
// not used for reports - used for project definition, any string values are accepted
"status": "ToDo",
// not used for reports - any string value is accepted
"assignee": null,
// used for reports - any string value is accepted
"priority": "Major",
// not used for reports - arrays of string values are accepted
"labels": [],
// not used for reports - array of string values is accepted
"affectsVersions": [],
// used for reports - arrays of string values are accepted
"components": ["UI", "Aggregations"],
// the crawler must find field that is relevant. For data sources like Jira, which has story points as custom field use the technique described at link:index.html#_pluginfilter to allow the user to configure which fields are used for story points when user create data source credentials. So story point field will be specified along with login and password to Jira data source.
"storyPoints": null,
"additionalFields": {
"project_key": "ABC", // used in the project definition
},
"customFields": {
"customfield_12314": 5,
"customfield_10372": ["Development"]
},
"urlParts": {
"key": "AG-123"
},
"other": {
"relations": {
"issueKey": "AG-123"
}
}
},
{
"id": "91743",
"version": 1,
"startMs": 1506553173203,
"title": "ABC-123: Remove temporary visualizations",
"ticketType": "Story",
"status": "Progress",
"assignee": "Bob White",
"priority": "Major",
"labels": ["TECH-DEBT"],
"affectsVersions": ["XLI 1.1-beta.5", "XLI 2.0"],
"components": ["UI", "Aggregations"],
"storyPoints": 0.5,
"additionalFields": {
"project_key": "ABC",
},
"customFields": {
"customfield_12314": 5,
"customfield_10372": ["Development"]
},
"urlParts": {
"key": "ABC-123"
},
"other": {
"relations": {
"issueKey": "ABC-123"
}
}
},
{
"id": "91743",
"version": 2,
"startMs": 1506554073203,
"title": "ABC-123: Remove temporary visualizations",
"ticketType": "Story",
"status": "Done",
"assignee": null,
"priority": "Major",
"labels": ["TECH-DEBT"],
"affectsVersions": ["XLI 1.1-beta.5", "XLI 2.0"],
"components": ["UI", "Aggregations"],
"storyPoints": 4.5,
"additionalFields": {
"project_key": "ABC",
},
"customFields": {
"customfield_12314": 5,
"customfield_10372": ["Development"]
},
"urlParts": {
"key": "ABC-123"
},
"other": {
"relations": {
"issueKey": "ABC-123"
},
"raw": {
...
}
}
}
]
}
}FIELDS example
{
"JiraPlugin.v1": {
"FIELDS.v1": [ // only one element is allowed in this array
{
// no "id" is necessary for fields
"fields": {
"customfield_10372": {
"type": "multiSelect",
"title": {
"displayName": "Department"
}
},
"customfield_12314": {
"type": "long",
"title": {
"displayName": "Magic Long"
}
}
}
}
]
}
}3. Crawler operation
3.1. Message queue
A crawler can use the XL Impact message queue or can operate on its own.
If a crawler uses the XL Impact message queue it can be completely stateless - failures and restarts will not be an issue.
If a crawler does not use the XL Impact message queue - data loss from failures and restarts must be managed.
Credentials are only passed from XL Impact to a crawler if the crawler uses the XL Impact message queue.
3.2. Credentials for data source systems
By default, credentials are stored and encrypted in XL Impact. Optionally, passwords or API keys can be passed directly to a crawler that is running locally, without saving them in XLI.
Credentials stored in XL Impact are added to all messages delivered from the message queue to the crawler. Credentials are configured by customer through the Client-App.
3.3. Message-driven operation
This section is only applicable if a crawler uses the XL Impact message queue.
The Client-App starts the crawling process when it first sends a message to the queue. For example, this message may contain: "Please crawl the whole data source", and include a dataSourceId. When the crawler receives this message it must plan it’s work and push a message to the XL Impact message queue for each piece of planned work. After this is done - the crawler must send an acknowledge message.
Each message must give a clear definition of what must be crawled, and be small enough to be processed quickly. For example, a message may say: "Please crawl GitHub repo X". While this message is being processed, the crawler will send messages to XL Impact instructing it to: crawl commits of repo X, crawl pull requests of repo X, and crawl tags of repo X.
Calls related to message queue:
3.4. Incremental vs non-incremental
A crawler can operate in two modes:
- Incremental mode: A crawler stops when it determines that the remaining data has already been crawled.
- Non-incremental mode: A crawler crawls everything.
In order to use both modes together, a crawler must be able to determine the version of crawled object from its contents. In a worst case scenario, a hash of the whole contents (be careful of the key ordering in JSON files, etc) can be used. Most data sources provide a means to derive a version. For example, update date-time can be used for Jira (GitHub commits are unmodifiable by nature).
3.5. Deduping messages
In the following scenarios, crawling messages should not be sent back to the queue:
1. When a message is a duplicate. Skip duplicate messages by using the following message variables: 'runTimestamp', and 'object dedupe metadata' which contains an 'incrementalTimestamp' and 'nonIncrementalTimestamp'. If a message has an older or the same timestamp, then the appropriate timestamp is saved in dedupe metadata, and no new messages should be published.
2. When an object version in dedupe metadata is the same as what is received from the data source, this means that the object is already known to XL Impact. As a default, if incremental crawling is in progress, you should stop processing messages if the data is already known.
There are two ways to determine if an object is known or not:
1. When saving the artifact - the artifact will be saved but you will get 'isKnown' as a response, see link below.
2. Without saving the artifact - either by sending special request to check which artifacts are known or by fetching dedupe metadata altogether and comparing versions yourself, see links to both calls below.
Saving an artifact:
Checking which artifacts are known:
Getting dedupe metadata
3.6. Rate limiting considerations
Some data sources may use rate limiting, this means that you must make as few requests to data source system as possible. If a data source uses rate limiting, perform deduping and then retrieve object from data source.
- For mutable objects, it is not possible to perform complete deduping without getting data from the data source, partial deduping can be performed.
- For immutable objects (like commits) - it is possible to perform full deduping given that messages contains object IDs.
4. Resources
4.2. Crawler-resource
Crawler Resource
4.2.1. ackUMMessage
POST /api/crawler/ack-um-message
4.2.2. checkWhichArtefactsAreKnown
POST /api/crawler/check-which-artefacts-are-known
|
Caution
|
operation.deprecated |
4.2.3. checkWhichUMArtefactsAreKnown
POST /api/crawler/check-which-um-artefacts-are-known
4.2.4. getArtefactDedupeUMMetadata
POST /api/crawler/get-artefact-dedupe-um-metadata
4.2.5. pollUMMessage
POST /api/crawler/poll-um-message
4.2.6. pushData
POST /api/crawler/push-data
|
Caution
|
operation.deprecated |
4.2.7. pushMultipleUMData
POST /api/crawler/push-multiple-um-data
4.2.8. pushProbeUMStatus
POST /api/crawler/push-probe-um-status
4.2.9. pushUMData
POST /api/crawler/push-um-data
4.2.10. pushUMMessage
POST /api/crawler/push-um-message
4.4. Profile-info-resource
Profile Info Resource
4.5. User-jwt-controller
User JWT Controller
5. Definitions
5.1. AckCrawlerUMMessageRequestVM
| Name | Schema |
|---|---|
messageId |
string |
networkId |
string |
pluginId |
string |
queue |
enum (NORMAL, PRIORITY, RETRIES, DLQ, UNRECOGNIZED) |
tenantId |
string |
5.3. ArtefactDedupeUMMetadataRequestVM
| Name | Schema |
|---|---|
artefactId |
< string > array |
dataSourceId |
string |
dataSourceType |
enum (FIELDS, BUILD, BUILD_TEMPLATE, COMMIT, PULL_REQUEST, LABEL, TICKET, XLR, JENKINS, GITHUB, JIRA, BITBUCKET, TRAVIS, SERVICENOW, UNRECOGNIZED) |
pluginId |
string |
tenantId |
string |
5.4. ArtefactDedupeUMMetadataResponse
| Name | Schema |
|---|---|
data |
< DedupeMetadata > array |
5.5. BaseProbeMessage
| Name | Schema |
|---|---|
dataSourceDetails |
|
metadata |
|
pluginId |
string |
probeId |
integer (int64) |
tenantId |
string |
5.6. CheckWhichArtefactsAreKnownResponse
| Name | Schema |
|---|---|
artefacts |
< KnownArtefactResponseDescription > array |
5.7. CheckWhichUMArtefactsAreKnownRequestVM
| Name | Schema |
|---|---|
artefacts |
< KnownArtefactRequestDescriptionVM > array |
dataSourceId |
string |
dataSourceType |
enum (FIELDS, BUILD, BUILD_TEMPLATE, COMMIT, PULL_REQUEST, LABEL, TICKET, XLR, JENKINS, GITHUB, JIRA, BITBUCKET, TRAVIS, SERVICENOW, UNRECOGNIZED) |
pluginId |
string |
tenantId |
string |
5.8. CheckWhichUMArtefactsAreKnownResponse
| Name | Schema |
|---|---|
artefacts |
< KnownArtefactResponseDescription > array |
5.10. CrawlerCredentialsPackage
| Name | Schema |
|---|---|
base_ui_url |
string |
credentials |
< CrawlerCredentials > array |
endpoint |
string |
filters |
< string, CrawlerFilters > map |
network_id |
string |
throttle_factor |
number (double) |
5.12. CrawlerDataMessage
| Name | Schema |
|---|---|
dataKey |
string |
dataSourceDetails |
|
dataValue |
|
metadata |
|
pluginId |
string |
tenantId |
string |
5.14. CrawlerDataVM
| Name | Schema |
|---|---|
data |
string |
dedupeMetadata |
|
version |
string |
5.16. CrawlerJobMetadata
| Name | Schema |
|---|---|
delayUntil |
integer (int64) |
errorMessage |
string |
failures |
integer (int32) |
incremental |
boolean |
runTimestamp |
integer (int64) |
5.17. CrawlerMessage
In addition to listed fields "name" field should be either "probeMessage" for BaseProbeMessage or "CrawlerDataMessage" for CrawlerDataMessage
| Name | Schema |
|---|---|
dataSourceDetails |
|
metadata |
|
pluginId |
string |
tenantId |
string |
5.18. CrawlerProbeResult
| Name | Schema |
|---|---|
results |
< ProbeResultMetadata > array |
5.19. CrawlerProbeResultVM
| Name | Schema |
|---|---|
results |
< ProbeResultMetadataVM > array |
5.21. DedupeMetadata
| Name | Schema |
|---|---|
artefact_id |
string |
incremental_run_timestamp |
integer (int64) |
non_incremental_run_timestamp |
integer (int64) |
version |
string |
5.22. DedupeMetadataVM
| Name | Description | Schema |
|---|---|---|
artefactId |
string |
|
incrementalRunTimestamp |
Either incrementalRunTimestamp or nonIncrementalRunTimestamp is required |
integer (int64) |
nonIncrementalRunTimestamp |
Either nonIncrementalRunTimestamp or incrementalRunTimestamp is required |
integer (int64) |
version |
string |
5.23. HalfRelationMatcherConfiguration
| Name | Description | Schema |
|---|---|---|
pluginIds |
list of pluginIds that are allowed for this side of connection. If you only want to allow your plugin itself, use requireMe property |
< string > array |
requireMe |
this means, that this relation should be coming from or to exactly this plugin id (that defines this relation). The only purpose of this property is to avoid repetition of plugin id in plugin manifest JSON |
boolean |
types |
list of types that are allowed for this side of connection |
< enum (FIELDS, BUILD, BUILD_TEMPLATE, COMMIT, PULL_REQUEST, LABEL, TICKET) > array |
5.25. KnownArtefactRequestDescription
| Name | Schema |
|---|---|
artefact_id |
string |
version |
string |
5.26. KnownArtefactRequestDescriptionVM
| Name | Schema |
|---|---|
artefactId |
string |
version |
string |
5.27. KnownArtefactResponseDescription
| Name | Schema |
|---|---|
artefact_id |
string |
is_known |
boolean |
5.28. LoginVM
| Name | Description | Schema |
|---|---|---|
password |
string |
|
rememberMe |
boolean |
|
username |
Length : |
string |
5.29. PatternRelationMatcherConfiguration
see RelationMatcherConfiguration. An example regular expression for 40 char sha hash is "(|[0-9a-f])([0-9a-f]{40})($|[^0-9a-f])"
| Name | Description | Schema |
|---|---|---|
afterGroup |
group that stands for following char, by default it is last group, having index = -1 |
integer (int32) |
beforeGroup |
group that stands for preceding char, by default it is first group |
integer (int32) |
from |
filters for outgoing half of relation |
|
meaningfulGroup |
group that stands for regexp payload group, by default it is 2nd group (between preceding char group and following char group) |
integer (int32) |
patternFields |
which fields should be scanned for regular expression. Default value is "*" which means all fields. This field is a list of jsonPath notations |
< string > array |
preciseField |
which field should be used by precise side |
string |
regexp |
regular expression (Java) that will be used to discover relation |
string |
regexpIsOn |
which side is regexp. The other side is precise |
enum (FROM, TO) |
relationType |
one of known types: "buildTemplate", "upstreamBuild", "commitReference", "ticketReference", "parentCommit" |
enum (buildTemplate, upstreamBuild, commitReference, ticketReference, parentCommit) |
requireSameDataSource |
when set to true relation will only be built when both sides of relation come from same data source |
boolean |
requireSameDomain |
when set to true relation will only be built when both sides of relation have same unified model entity domain |
boolean |
requireSamePluginId |
when set to true relation will only be built when both sides of relation come from same plugin |
boolean |
requireSameType |
when set to true relation will only be built when both sides of relation have same unified model entity type |
boolean |
requiredTimeRange |
allows you to limit maximum distance on timeline between two related entities |
|
to |
filters for incoming half of relation |
|
windowSize |
min window size that will be used to scan. This is effectively max length of your match + some extra for both sides to make sure, that it is not truncated. E.g. for sha hash window size is 42, which is 40 for hash itself, and then 1 char each side to make sure, that it is exactly 40 char hash |
integer (int32) |
5.32. PluginExternalUrl
| Name | Description | Schema |
|---|---|---|
url |
template url, that will be exposed to end user when needed, so that he can conveniently navigate to appropriate entity in data source system. For base url use ${dataSource.baseUiUrl} placeholder, for other moving parts of urls use ${entity.urlParts.urlpartname} where urlpartname is a key from urlParts subdoc of an entity UM document. E.g. "${dataSource.baseUiUrl}/${entity.urlParts.organization}/${entity.urlParts.repository}/commit/${entity.urlParts.sha}" |
string |
5.33. PluginFilter
Declares that crawler supports particular filter and defines its type. Crawler filter values are actually populated in credentials, but possible filters are defined in plugin manifest
| Name | Description | Schema |
|---|---|---|
type |
type of filter values that this filter will have. For example if plugin supports filtering by GitHub repository name, type of that is string |
enum (string, double, integer) |
5.34. PluginManifest
See Plugin Manifest section of documentation
| Name | Description | Schema |
|---|---|---|
additionalFields |
see Additional Fields section in the documentation |
< string, PluginCustomField > map |
crawlerFilters |
< string, PluginFilter > map |
|
entityNames |
names of entities by type. For example usually Annotated Tag in Git is Annotated Tag, but GitHub names them Releases |
< string, PluginEntityName > map |
externalUrls |
see UI URLs section in the documentation |
< string, PluginExternalUrl > map |
manifestVersion |
for future extensibility, use 1 for now |
integer (int32) |
plugin |
plugin name, id, version |
|
relations |
< string, RelationMatcherConfiguration > map |
|
validDataTypes |
list of types, that this plugin can give to XLI |
< enum (FIELDS, BUILD, BUILD_TEMPLATE, COMMIT, PULL_REQUEST, LABEL, TICKET) > array |
5.35. PluginMetadata
| Name | Description | Schema |
|---|---|---|
id |
pluginId, the convention is camelcase with first caps, e.g. "BitbucketServerPlugin" |
string |
name |
human readable plugin name |
string |
version |
e.g. "v1". Used as key when pushing data and validated by XLI |
string |
5.36. PollCrawlerUMMessageRequestVM
| Name | Schema |
|---|---|
networkId |
string |
pluginId |
string |
queue |
enum (NORMAL, PRIORITY, RETRIES, DLQ, UNRECOGNIZED) |
tenantId |
string |
5.37. PollCrawlerUMMessageResponse
| Name | Schema |
|---|---|
credentials_package |
|
data_source_id |
string |
message |
|
message_id |
string |
tenant_id |
string |
5.38. PreciseRelationMatcherConfiguration
| Name | Description | Schema |
|---|---|---|
from |
filters for outgoing half of relation |
|
fromField |
which field is used to link outgoing side of relation, e.g. "other.relations.sha" |
string |
relationType |
one of known types: "buildTemplate", "upstreamBuild", "commitReference", "ticketReference", "parentCommit" |
enum (buildTemplate, upstreamBuild, commitReference, ticketReference, parentCommit) |
requireSameDataSource |
when set to true relation will only be built when both sides of relation come from same data source |
boolean |
requireSameDomain |
when set to true relation will only be built when both sides of relation have same unified model entity domain |
boolean |
requireSamePluginId |
when set to true relation will only be built when both sides of relation come from same plugin |
boolean |
requireSameType |
when set to true relation will only be built when both sides of relation have same unified model entity type |
boolean |
requiredTimeRange |
allows you to limit maximum distance on timeline between two related entities |
|
to |
filters for incoming half of relation |
|
toField |
which field is used to link incoming side of relation, e.g. "id" |
string |
5.39. ProbeResultMetadata
| Name | Schema |
|---|---|
endpoint |
string |
is_success |
boolean |
reason |
string |
username |
string |
5.40. ProbeResultMetadataVM
| Name | Schema |
|---|---|
endpoint |
string |
reason |
string |
success |
boolean |
username |
string |
5.43. PushCrawlerUMDataRequestVM
| Name | Schema |
|---|---|
artefact |
|
crawlerStatus |
string |
dataSourceId |
string |
dataSourceType |
enum (FIELDS, BUILD, BUILD_TEMPLATE, COMMIT, PULL_REQUEST, LABEL, TICKET, XLR, JENKINS, GITHUB, JIRA, BITBUCKET, TRAVIS, SERVICENOW, UNRECOGNIZED) |
pluginId |
string |
tenantId |
string |
5.45. PushCrawlerUMMessageRequestVM
| Name | Description | Schema |
|---|---|---|
dataSourceId |
string |
|
message |
JSON of CrawlerMessage as String |
|
networkId |
string |
|
pluginId |
string |
|
queue |
enum (NORMAL, PRIORITY, RETRIES, DLQ, UNRECOGNIZED) |
|
tenantId |
string |
5.47. PushCrawlerUMStatusRequestVM
| Name | Schema |
|---|---|
crawlerStatus |
string |
dataSourceId |
string |
pluginId |
string |
tenantId |
string |
5.49. PushMultipleCrawlerUMDataRequestVM
| Name | Schema |
|---|---|
artefacts |
< CrawlerDataVM > array |
crawlerStatus |
string |
dataSourceId |
string |
dataSourceType |
enum (FIELDS, BUILD, BUILD_TEMPLATE, COMMIT, PULL_REQUEST, LABEL, TICKET, XLR, JENKINS, GITHUB, JIRA, BITBUCKET, TRAVIS, SERVICENOW, UNRECOGNIZED) |
pluginId |
string |
tenantId |
string |
5.50. PushMultipleCrawlerUMDataResponse
| Name | Schema |
|---|---|
known_artefacts |
< KnownArtefactResponseDescription > array |
5.51. PushProbeUMStatusRequestVM
| Name | Schema |
|---|---|
dataSourceId |
string |
pluginId |
string |
probeId |
integer (int64) |
probeStatus |
|
tenantId |
string |
5.53. RelationMatcherConfiguration
See PreciseRelationMatcherConfiguration, PatternRelationMatcherConfiguration and ViaPreciseRelationMatcherConfiguration
| Name | Description | Schema |
|---|---|---|
from |
filters for outgoing half of relation |
|
relationType |
one of known types: "buildTemplate", "upstreamBuild", "commitReference", "ticketReference", "parentCommit" |
enum (buildTemplate, upstreamBuild, commitReference, ticketReference, parentCommit) |
requireSameDataSource |
when set to true relation will only be built when both sides of relation come from same data source |
boolean |
requireSameDomain |
when set to true relation will only be built when both sides of relation have same unified model entity domain |
boolean |
requireSamePluginId |
when set to true relation will only be built when both sides of relation come from same plugin |
boolean |
requireSameType |
when set to true relation will only be built when both sides of relation have same unified model entity type |
boolean |
requiredTimeRange |
allows you to limit maximum distance on timeline between two related entities |
|
to |
filters for incoming half of relation |
5.54. RelationMatcherTimeRangeConfiguration
| Name | Description | Schema |
|---|---|---|
rangeMs |
max time distance between related items, in milliseconds |
integer (int64) |
timestampFieldOnFromSide |
field on the outgoing side of relation to check time, startMs by default |
string |
timestampFieldOnToSide |
field on the incoming side of relation to check time, startMs by default |
string |
5.55. ResponseEntity
| Name | Schema |
|---|---|
body |
object |
statusCode |
enum (100, 101, 102, 103, 200, 201, 202, 203, 204, 205, 206, 207, 208, 226, 300, 301, 302, 303, 304, 305, 307, 308, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 426, 428, 429, 431, 451, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509, 510, 511) |
statusCodeValue |
integer (int32) |
5.58. ViaPreciseRelationMatcherConfiguration
| Name | Description | Schema |
|---|---|---|
from |
filters for outgoing half of relation |
|
fromField |
which field is used to link outgoing side of relation, e.g. "other.relations.sha" |
string |
relationType |
one of known types: "buildTemplate", "upstreamBuild", "commitReference", "ticketReference", "parentCommit" |
enum (buildTemplate, upstreamBuild, commitReference, ticketReference, parentCommit) |
requireSameDataSource |
when set to true relation will only be built when both sides of relation come from same data source |
boolean |
requireSameDomain |
when set to true relation will only be built when both sides of relation have same unified model entity domain |
boolean |
requireSamePluginId |
when set to true relation will only be built when both sides of relation come from same plugin |
boolean |
requireSameType |
when set to true relation will only be built when both sides of relation have same unified model entity type |
boolean |
requiredTimeRange |
allows you to limit maximum distance on timeline between two related entities |
|
to |
filters for incoming half of relation |
|
toField |
which field is used to link incoming side of relation, e.g. "id" |
string |
via |
filter for middle part of relation |
|
viaFieldToMatchFrom |
field of middle part of relation to match outgoing side of relation |
string |
viaFieldToMatchTo |
field of middle part of relation to match incoming side of relation |
string |