Subject: [scim] SCIM Protocol - 3 suggestions for improvement

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Thu, 2 Aug 2012 07:23:31 +1000

(I posted this on the SCIM Google Group, and I was advised to subscribe to
the mailing list and post it here instead, so here goes.)

Hi,

My name is Ganesh Prasad, and my experience in Identity and Access
Management is mainly through a 3-year project at an Australian insurance
company, an experience I have written about as a eBook on InfoQ (
http://www.infoq.com/minibooks/Identity-Management-Shoestring).

I have been following the SCIM spec off and on, and based on my experience
with a loosely-coupled architecture that I found to be successful, I have
the following 3 suggestions to make.

1. The enterprise client and the cloud provider should maintain their own
internal IDs for a resource, which they should not reveal to each other.
Both of them should map their internal IDs to a shared External ID, and
this is the only ID that should be exposed through the API. The current
specification's provision of an id (which is the external ID and the only
one to be transferred through the API) and an "external ID" (which is the
client's internal ID and should be hidden) is diametrically opposite to
this.

2. When dealing with multi-valued attributes of a resource (expressed as
arrays in JSON), they must be converted from an array into a dictionary
with unique keys (UUIDs generated by the cloud provider when the attribute
is created). Without unique keys for every attribute value of a resource,
manipulating it will be clumsy and inelegant.

3. The PATCH command can be improved in 3 significant ways:
3a. Leverage the fact (from 2 above) that every value has a key, to greatly
simplify the API
3b. Use special verbs as nested operations of the PATCH command to add,
modify and delete attributes at any level
3c. Use the WebDAV status code of "207 Multi-Status" instead of "200 OK" as
the response to a PATCH (or BULK) command.

To elaborate,

1. Revealing private IDs externally is a form of tight coupling. A major
requirement with Identity Management is to split (or merge) identities when
false positives (or false negatives) are detected, i.e., when a resource is
discovered to be more than one, or when multiple resources are detected to
be the same. If internal identifiers are revealed to external domains, such
clean-ups become difficult, hence every domain that wants to expose
references to a resource must map its internal ID to and external one
created for this explicit purpose, and only reveal this.

In the SCIM case, when an enterprise client POSTs a resource creation
request, the cloud provider must generate its own internal UUID as well as
an external UUID, map them together, and only return the external UUID in
the "Location:" header. The enterprise client should map this external UUID
to a newly-generated internal ID of its own. In case the resource already
has an identifier within the enterprise client's domain, then this is the
internal ID that must be mapped to the external UUID returned through the
POST response.

2. If a resource is to be created, and one of its attributes is
multi-valued, e.g.,

    "email-addrs" :
    [
        "[email-masked]",
        "[email-masked]",
        "[email-masked]"
    ]

then on successful creation, the server response should include the
representation of the resource, and this attribute should look like this:

    "email-addrs" :
    [
        { "7dfcb444-74d8-4f17-aa66-daf9ea3bd902" : "[email-masked]" },
        { "3bd10085-c474-43b9-9cda-8646c3085bbf" : "[email-masked]" },
        { "581da5c7-c6e1-4cca-9db7-7a6d1de664e1" : "[email-masked]"
}
    ]

The client now knows what each value is labelled. This now provides an
unambiguous way to reference a value to add, modify and delete it:

Add:

POST /Users/2819c223-7f76-453a-919d-413861904646/email-addrs
value="[email-masked]"

Modify:

PUT
/Users/2819c223-7f76-453a-919d-413861904646/email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf
value="[email-masked]"

Delete:
DELETE
/Users/2819c223-7f76-453a-919d-413861904646/email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1

One can even delete all email addresses like this:
DELETE /Users/2819c223-7f76-453a-919d-413861904646/email-addrs

I believe this is more elegant than what the spec recommends.

3. It's possible to think of the operations POST, PUT and DELETE as nested
operations inside a PATCH. PATCH itself need not be nested because its
semantics apply throughout the "tree" of a resource.

However, the semantics of PUT are a little messy. Also, the use of HTTP
verbs at a different level could be confusing. That's why I would recommend
6 separate verbs that are a little more unambiguous in their meaning:

1. INCLUDE (equivalent to POST): Add this resource to a collection and
return a generated URI
2. PLACE (equivalent to one form of PUT): Add this resource at the location
specified by the accompanying URI. (If there’s already a value at that
location, return an error status.)
3. REPLACE (equivalent to another form of PUT): Replace the value at the
location specified by the accompanying URI with this value. (If there’s no
such URI, return an error status.)
4. FORCE (equivalent to a third form of PUT): This means PLACE or REPLACE.
(At the end of this operation, we want the specified URI to hold the
accompanying value whether the URI already existed or not.)
5. RETIRE (equivalent to DELETE): Delete, deactivate or otherwise render
inaccessible the resource at the specified URI.
6. AMEND (equivalent to PATCH): (This verb is just listed for completeness.
We probably don’t need a nested PATCH since PATCH cascades to every level
of the tree.)

A PATCH request could therefore look like this:

PATCH /Users/2819c223-7f76-453a-919d-413861904646 HTTP/1.1
Host: example.com
Accept: application/json
Authorization: Bearer h480djs93hd8
Content-length: ...

{
    REPLACE: {
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {
        "key" : "address.unit-number",
        "value" : "12"
    },
    PLACE : {
        "key" : "address.state",
        "value" : "SA"
    },
    FORCE : {
        "key" : "address.country",
        "value" : "Australia"
    },
    INCLUDE : {
        "key" : "email-addrs",
        "value" : "[email-masked]"
    },
    REPLACE : {
        "key" : "email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf",
        "value" : "[email-masked]"
    },
    RETIRE : {
        "key" : "email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1"
    }
}

The PATCH response should utilise the status code "207 Multi-Status"
because the nested operations could have varying status codes. A sample
response is below:

HTTP/1.1 207 Multi-Status
Content-Type: application/json
ETag: W/"b431af54f0671a2"
Location:"https://example.com/v1/Users/2819c223-7f76-453a-919d-413861904646"

{
    "schemas":["urn:scim:schemas:core:1.0"],
    "external-id":"2819c223-7f76-453a-919d-413861904646",
    REPLACE: {
        "status" : "200 OK",
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "status" : "200 OK",
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "status" : "200 OK",
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {
        "status" : "200 OK",
        "key" : "address.unit-number",
        "value" : "12"
    },
    PLACE : {
        "status" : "200 OK",
        "key" : "address.state",
        "value" : "SA"
    },
    FORCE : {
        "status" : "200 OK",
        "key" : "address.country",
        "value" : "Australia"
    },
    INCLUDE : {
        "status" : "201 Created",
        "key" : "email-addrs/11f664ec-898b-4f6f-8948-ecfda74deff0",
        "value" : "[email-masked]"
    },
    REPLACE : {
        "status" : "200 OK",
        "key" : "email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf",
        "value" : "[email-masked]"
    },
    RETIRE : {
        "status" : "200 OK",
        "key" : "email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1"
    }
    "meta": {
        "created":"2011-08-08T04:56:22Z",
        "lastModified":"2011-08-08T08:00:12Z",
        "location":"
https://example.com/v1/Users/2819c223-7f76-453a-919d-413861904646",
        "version":"W\/\"b431af54f0671a2\""
    }
}

If there are errors, they will take the place of the "200 OK" or "201
Created" status codes in the above successful case. But the outer status
will remain "207 Multi-Status".

The same scheme can be used to deal with operations on members of a group,
and for bulk operations.

I hope you find these suggestions useful.

I read the SCIM spec afresh last week and these ideas came flooding into my
head because I have been working at another organisation (a telco) for the
last 5 months, also in Identity and Access Management, and my thoughts have
moved further along the direction of evolving a specialised data model
based on specific principles, especially for IAM.

I am planning to write about this and also the data-related principles soon
and am in negotiations with InfoQ regarding publication.

Regards,
Ganesh Prasad

---

From: Kelly Grizzle <[email-masked]>
Date: Wed, 8 Aug 2012 18:55:34 +0000

Thanks for the feedback, Ganesh.  I read through this and your InfoQ article (http://www.infoq.com/articles/scim-data-model-limitations) and have some thoughts.

> The rest of the protocol does not meaningfully use the enterprise client's identifier, the "external ID"
> at all, even though it was ostensibly introduced to make things friendlier for the client.

The usage pattern for an external ID would be to search for a user by externalId and use the ID of the returned user in any desired operation.  For example:

GET /Users?filter=externalId eq "bjensen"&attributes=id

{
  "totalResults": 1,
  "Resources": [
    {
      "id": "2819c223-7f76-453a-919d-413861904646"
    }
  ]
}

Retrieve the ID from the response and use it.

DELETE /Users/2819c223-7f76-453a-919d-413861904646

This does introduce an additional HTTP request if the client chooses not to store the server's id.  An issue was created to consider allowing operations to use the externalId (http://code.google.com/p/scim/issues/detail?id=35), but I believe the general consensus has been to not include this in the spec.  One main point of contention is that much of the rest of the spec (eg - group membership references, manager references, etc...) require knowledge of the server's identifier.  Continuing this discussion on the IETF list would be a good thing, though.

> the cloud provider's ID and the enterprise client's ID are both "Internal IDs" with respect to their domains

I think this comes down to a nomenclature problem.  The server's ID does not necessarily have to be the unique identifier that the underlying identity store uses, it just has to be stable and unique.  In many cases, the underlying identity store will provide identifiers with these properties already (eg - a uuid) and it can be used by the SCIM interface.  The "externalId" is referring to the fact that the id is maintained external to the SCIM server.  As long as the server's identifiers are stable and unique (which is mandated by the spec), I don't see a problem.

> The secret is that every value needs a key, and multi-valued attributes lack that. So our solution is quite
> simple - turn every list or array (of values) into a dictionary (of key-value pairs) by providing each value
> with a unique and meaning-free identifier.

I agree that this would be useful, especially in the PATCH operation.  One reason that this wasn't included in the spec originally is that it can put undue burden on the service provider.  Many service providers are putting SCIM interfaces in front of their existing identity stores (eg - directory servers, SaaS application databases, etc...).  Many of these do not have a unique identifier for multi-valued attributes.  By requiring this, a majority of the server providers would have to start maintaining a unique key for each multi-valued attribute.  I believe this would be a roadblock for many implementers.

> When the SCIM protocol uses PATCH, there are areas where it seems a bit clumsy.

I like the thoughts here.  Your example reminds me of unified diffs (http://en.wikipedia.org/wiki/Diff#Unified_format), which are commonly used with a patch program (pretty much the equivalent of the PATCH verb).  However, the three proposals seem to largely hinge on being able to uniquely address each element within an object.  Without these it is not so easy to address each patch sub-operation (REPLACE, INCLUDE, etc...) or provide a multi-status.

The 207 response would be interesting to consider for the bulk endpoint (http://www.simplecloud.info/specs/draft-scim-api-00.html#bulk-resources), however.

> There are other, non-data aspects of SCIM which may require review, such as its synchronous request-response
> interaction model, which is a form of tight coupling and could prove to be a source of brittleness.

I agree that we should explore optional asynchronous requests in 2.0.

Thanks again for your thoughts.  I hope you stay involved in the discussion as work on SCIM 2.0 goes forward.

--Kelly

From: [email-masked] [mailto:[email-masked]] On Behalf Of Ganesh and Sashi Prasad
Sent: Wednesday, August 01, 2012 4:24 PM
To: [email-masked]
Subject: [scim] SCIM Protocol - 3 suggestions for improvement

(I posted this on the SCIM Google Group, and I was advised to subscribe to the mailing list and post it here instead, so here goes.)

Hi,

My name is Ganesh Prasad, and my experience in Identity and Access Management is mainly through a 3-year project at an Australian insurance company, an experience I have written about as a eBook on InfoQ (http://www.infoq.com/minibooks/Identity-Management-Shoestring).

I have been following the SCIM spec off and on, and based on my experience with a loosely-coupled architecture that I found to be successful, I have the following 3 suggestions to make.

1. The enterprise client and the cloud provider should maintain their own internal IDs for a resource, which they should not reveal to each other. Both of them should map their internal IDs to a shared External ID, and this is the only ID that should be exposed through the API. The current specification's provision of an id (which is the external ID and the only one to be transferred through the API) and an "external ID" (which is the client's internal ID and should be hidden) is diametrically opposite to this.

2. When dealing with multi-valued attributes of a resource (expressed as arrays in JSON), they must be converted from an array into a dictionary with unique keys (UUIDs generated by the cloud provider when the attribute is created). Without unique keys for every attribute value of a resource, manipulating it will be clumsy and inelegant.

3. The PATCH command can be improved in 3 significant ways:
3a. Leverage the fact (from 2 above) that every value has a key, to greatly simplify the API
3b. Use special verbs as nested operations of the PATCH command to add, modify and delete attributes at any level
3c. Use the WebDAV status code of "207 Multi-Status" instead of "200 OK" as the response to a PATCH (or BULK) command.

To elaborate,

1. Revealing private IDs externally is a form of tight coupling. A major requirement with Identity Management is to split (or merge) identities when false positives (or false negatives) are detected, i.e., when a resource is discovered to be more than one, or when multiple resources are detected to be the same. If internal identifiers are revealed to external domains, such clean-ups become difficult, hence every domain that wants to expose references to a resource must map its internal ID to and external one created for this explicit purpose, and only reveal this.

In the SCIM case, when an enterprise client POSTs a resource creation request, the cloud provider must generate its own internal UUID as well as an external UUID, map them together, and only return the external UUID in the "Location:" header. The enterprise client should map this external UUID to a newly-generated internal ID of its own. In case the resource already has an identifier within the enterprise client's domain, then this is the internal ID that must be mapped to the external UUID returned through the POST response.

2. If a resource is to be created, and one of its attributes is multi-valued, e.g.,

    "email-addrs" :
    [
        "[email-masked]<mailto:[email-masked]>",
        "[email-masked]<mailto:[email-masked]>",
        "[email-masked]<mailto:[email-masked]>"
    ]

then on successful creation, the server response should include the representation of the resource, and this attribute should look like this:

    "email-addrs" :
    [
        { "7dfcb444-74d8-4f17-aa66-daf9ea3bd902" : "[email-masked]<mailto:[email-masked]>" },
        { "3bd10085-c474-43b9-9cda-8646c3085bbf" : "[email-masked]<mailto:[email-masked]>" },
        { "581da5c7-c6e1-4cca-9db7-7a6d1de664e1" : "[email-masked]<mailto:[email-masked]>" }
    ]

The client now knows what each value is labelled. This now provides an unambiguous way to reference a value to add, modify and delete it:

Add:

POST /Users/2819c223-7f76-453a-919d-413861904646/email-addrs
value="[email-masked]<mailto:[email-masked]>"

Modify:

PUT /Users/2819c223-7f76-453a-919d-413861904646/email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf
value="[email-masked]<mailto:[email-masked]>"

Delete:
DELETE /Users/2819c223-7f76-453a-919d-413861904646/email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1

3. It's possible to think of the operations POST, PUT and DELETE as nested operations inside a PATCH. PATCH itself need not be nested because its semantics apply throughout the "tree" of a resource.

However, the semantics of PUT are a little messy. Also, the use of HTTP verbs at a different level could be confusing. That's why I would recommend 6 separate verbs that are a little more unambiguous in their meaning:

1. INCLUDE (equivalent to POST): Add this resource to a collection and return a generated URI
2. PLACE (equivalent to one form of PUT): Add this resource at the location specified by the accompanying URI. (If there's already a value at that location, return an error status.)
3. REPLACE (equivalent to another form of PUT): Replace the value at the location specified by the accompanying URI with this value. (If there's no such URI, return an error status.)
4. FORCE (equivalent to a third form of PUT): This means PLACE or REPLACE. (At the end of this operation, we want the specified URI to hold the accompanying value whether the URI already existed or not.)
5. RETIRE (equivalent to DELETE): Delete, deactivate or otherwise render inaccessible the resource at the specified URI.
6. AMEND (equivalent to PATCH): (This verb is just listed for completeness. We probably don't need a nested PATCH since PATCH cascades to every level of the tree.)

PATCH /Users/2819c223-7f76-453a-919d-413861904646 HTTP/1.1
Host: example.com<http://example.com>
Accept: application/json
Authorization: Bearer h480djs93hd8
Content-length: ...

{
    REPLACE: {
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {
        "key" : "address.unit-number",
        "value" : "12"
    },
    PLACE : {
        "key" : "address.state",
        "value" : "SA"
    },
    FORCE : {
        "key" : "address.country",
        "value" : "Australia"
    },
    INCLUDE : {
        "key" : "email-addrs",
        "value" : "[email-masked]<mailto:[email-masked]>"
    },
    REPLACE : {
        "key" : "email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf",
        "value" : "[email-masked]<mailto:[email-masked]>"
    },
    RETIRE : {
        "key" : "email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1"
    }
}

The PATCH response should utilise the status code "207 Multi-Status" because the nested operations could have varying status codes. A sample response is below:

{
    "schemas":["urn:scim:schemas:core:1.0"],
    "external-id":"2819c223-7f76-453a-919d-413861904646",
    REPLACE: {
        "status" : "200 OK",
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "status" : "200 OK",
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "status" : "200 OK",
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {
        "status" : "200 OK",
        "key" : "address.unit-number",
        "value" : "12"
    },
    PLACE : {
        "status" : "200 OK",
        "key" : "address.state",
        "value" : "SA"
    },
    FORCE : {
        "status" : "200 OK",
        "key" : "address.country",
        "value" : "Australia"
    },
    INCLUDE : {
        "status" : "201 Created",
        "key" : "email-addrs/11f664ec-898b-4f6f-8948-ecfda74deff0",
        "value" : "[email-masked]<mailto:[email-masked]>"
    },
    REPLACE : {
        "status" : "200 OK",
        "key" : "email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf",
        "value" : "[email-masked]<mailto:[email-masked]>"
    },
    RETIRE : {
        "status" : "200 OK",
        "key" : "email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1"
    }
    "meta": {
        "created":"2011-08-08T04:56:22Z",
        "lastModified":"2011-08-08T08:00:12Z",
        "location":"https://example.com/v1/Users/2819c223-7f76-453a-919d-413861904646",
        "version":"W\/\"b431af54f0671a2\""
    }
}

If there are errors, they will take the place of the "200 OK" or "201 Created" status codes in the above successful case. But the outer status will remain "207 Multi-Status".

The same scheme can be used to deal with operations on members of a group, and for bulk operations.

I read the SCIM spec afresh last week and these ideas came flooding into my head because I have been working at another organisation (a telco) for the last 5 months, also in Identity and Access Management, and my thoughts have moved further along the direction of evolving a specialised data model based on specific principles, especially for IAM.

I am planning to write about this and also the data-related principles soon and am in negotiations with InfoQ regarding publication.

Regards,
Ganesh Prasad

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Thu, 9 Aug 2012 11:34:54 +1000

Hi Kelly,

Thanks for your response. Let me first respond in brief to the two main
points you have made, and then elaborate on the first.

   1.

   Why should domains not expose their internal identifiers to other
   domains?

a.

We are designing a protocol for a federated system of domains, where all
domains are co-equal peers. (In physics too, N-body problems are much
harder than 2-body problems. :-) Therefore, assuming that there are only
two players in the interaction makes this tightly coupled in a number of
ways. We should rely on messaging and notification, with encapsulation of
domain-specific data.

b.

In any non-trivial data store, there will always be the ongoing need to
merge and split identities as and when “false negatives” and “false
positives” are discovered. A domain should be able to handle this internal
housekeeping freely, only notifying other domains when convenient. Mapping
of internal identifiers to external ones and maintaining this mapping
internally allows this loosely-coupled housekeeping to take place. Sharing
internal identifiers (or otherwise outsourcing the mapping of internal to
external identifiers) forces housekeeping activities to be done in
lock-step across domains.

c.

Asynchronous interaction is not just a matter of a suitable wire protocol
which can be designed later. The data model plays a crucial role in
enabling or constraining such interaction. A tightly-coupled data model
will force the use of synchronous interactions, and the exposure of
internal identifiers is a key part of this tight coupling.

2. The difficulty of assigning unique identifiers to the individual values
of multi-valued attributes:

a.

I'm not belittling the effort involved in migrating legacy data stores to
such a model. However, in the larger historical context of cross-domain
identity management, we are really at the very early stages. If a
relatively new discipline and a brand new spec are held captive to legacy
considerations, we are losing an opportunity to provide a clean and elegant
model to subsequent users of the spec, and this will have repercussions
over many years or even decades.

b.

If incumbent cloud providers find it hard to immediately adopt the
dictionary model for existing multi-valued attributes, they can transition
to this model by offering both “SCIM-compliant” and “non-SCIM-compliant”
APIs to their customers and encouraging new customers to adopt the
“SCIM-compliant” API. Legacy customers can be supported using a
“non-SCIM-compliant” API for an arbitrarily long period and gradually
migrated to the SCIM-compliant API. The logistics are not insurmountable,
and shouldn't prevent the adoption of a dictionary model for multi-valued
attributes.

Elaboration of Point 1:

When we consider federated identity across more than one domain, we have to
assume that domains are not necessarily master-slave in their interaction.
The most generic interaction model is peer-to-peer, where entity lifecycle
events within a domain are notified to other domains (when necessary) in an
asynchronous manner (i.e., through messaging) and the other domains are
free to respond to these events in an appropriate manner and at a time of
their convenience.

A key set of lifecycle events for an entity is the merging and splitting of
identity that is often required.

The question “Is this one entity?” can be answered either yes (positive) or
no (negative). But sometimes, we can discover false positives and false
negatives in our data stores.

Consider a case where customers sign up online, and two customers who are
privacy-conscious enter fake IDs such as “John Smith”, and also use the
same date of birth (say, 1 Jan 1970) or similar attributes. The front-end
application may make an intelligent (but incorrect) guess that these two
persons are the same, and re-assign the same identifier to the second
person. This is a false positive. They appear to be the same entity, but
they're actually different. When the error is discovered, the identities
will need to be split, with a new identifier generated for one of them.

Consider the opposite case where a customer signs up through two different
portals or in two different sessions, using the names “JSmith” and “JohnS”.
It is very likely that they will be treated as two different customers and
assigned two unique identifiers. This is a false negative. They appear to
be two entities, but are actually the same. At a later stage, when the
error is discovered, the identities will have to be merged, and one of the
identifiers will have to be dropped.

These are not theoretical use cases. They form a significant proportion of
the user base in most large Web-facing applications. Let's see how these
can be managed in a federated way by mapping internal identifiers to
external ones and only exposing external identifiers to other domains.

a. False positives:

Domain 1 has the following information about a customer in its data store:

Internal ID: 9caf78aac3d6

Attributes: {name: “John Smith”, dob: “01-Jan-1970”}

When requesting the provisioning of this entity in Domain 2, the following
ID is returned by Domain 2: ff487230b3a0.

Domain 1 then maintains the following in a mapping table and uses it for
translation when talking to Domain 2, taking care never to expose its
internal identifier:

| Internal Entity ID | External Domain ID | External Entity ID | Primary
flag |

| 9caf78aac3d6 | D2 | ff487230b3a0 | true |

When the false positive is discovered and the entity is split, Domain 1
creates a new internal identifier and now has the following entity
information.

Internal ID: 9caf78aac3d6

Attributes: {name: “John Smith”, dob: “01-Jan-1970”}

Internal ID: a99a5feba839

Attributes: {name: “John Smith”, dob: “01-Jan-1970”}

This second entity with its own internal identifier is invisible to Domain
2, and this is by design. Communication about the original entity takes
place as before by mapping “9caf78aac3d6” to “ff487230b3a0” and vice-versa.
At some convenient time (importantly, this doesn't have to be at the time
the split happens), Domain 2 can be requested to provision a second entity,
and when it responds with an identifier of “7a87f27c1dd8”, this can go into
the mapping table as a new record associated with the second entity's
internal identifier.

The mapping table now contains the following entries:

| Internal Entity ID | External Domain ID | External Entity ID | Primary
flag |

| 9caf78aac3d6 | D2 | ff487230b3a0 | true |

| a99a5feba839 | D2 | 7a87f27c1dd8 | true |

Domain 2 is not even aware that a split has happened, and the provisioning
that it does is not in lockstep with the split in identity that occurred in
Domain 1.

(What is the “Primary flag” used for? We'll see when we cover the treatment
of false negatives.)

b. False negatives:

Domain 1 has the following information about what it thinks are two
distinct customers in its data store:

Internal ID: 9caf78aac3d6

Attributes: {name: “JSmith”, dob: “01-Jan-1970”}

Internal ID: 273d36e30d09

Attributes: {name: “JohnS”, dob: “01-Jan-1970”}

When requesting the provisioning of these entities in Domain 2, the
following IDs are returned by Domain 2: ff487230b3a0 and 41206cc97c8b.

Domain 1 then maintains the following in a mapping table and uses it for
translation when talking to Domain 2, taking care never to expose its
internal identifiers:

| Internal Entity ID | External Domain ID | External Entity ID | Primary
flag |

| 9caf78aac3d6 | D2 | ff487230b3a0 | true |

| 273d36e30d09 | D2 | 41206cc97c8b | true |

When the false negative is discovered and the two entities are merged,
Domain 1 drops one of the internal identifiers and rationalises the name of
the customer (say, to “John Smith”). Let's say it retains the first ID
“9caf78aac3d6” and drops the second “273d36e30d09”.

The mapping table now looks like this:

| Internal Entity ID | External Domain ID | External Entity ID | Primary
flag |

| 9caf78aac3d6 | D2 | ff487230b3a0 | true |

| 9caf78aac3d6 | D2 | 41206cc97c8b | false |

Now two external identifiers map to the same internal one, so inbound
communication from Domain 2 can be unambiguously translated to the same
entity internally. However, when going outwards, Domain 1 will have to look
up the translation table to determine the “primary” external ID for this
entity in Domain 2, which was decided to be “ff487230b3a0”. That's where
the “Primary flag” comes in. The second external ID “41206cc97c8b” is never
used thereafter in outbound communication.

At some stage (importantly, not in lockstep with the identity merge),
Domain 2 can be requested to delete the customer record identified by “
41206cc97c8b”, and the second entry in the mapping table can be removed
once this is acknowledged.

This scheme will scale up to multiple domains, because the “External Domain
ID” column helps to keep track of which external ID is shared with which
Domain. (Why don't we use just one external ID for an entity and share it
with all external domains? Tight coupling again. Just as OAuth allows an
access token given to a third party to be invalidated without affecting the
access of other third parties, the use of separate external identifiers for
different domains allows fine-grained control of identity federation.)

The scheme also allows the splitting of an entity into more than two
entities, and the merging of more than two entities into a single one. (Any
organisation with a web-facing application will tell you how many John
Smiths there are who were born on 1 Jan 1970!)

This is a fairly long-winded explanation, but this is why we need to hide
internal identifiers from other domains, and why mappings need to be
managed internally in each domain. Such a data model also allows us to
choose asynchronous protocols for propagation of identity events, since
there is no consistency requirement to update multiple domains concurrently.

Regards,

Ganesh Prasad

On 9 August 2012 04:55, Kelly Grizzle <[email-masked]> wrote:

>  Thanks for the feedback, Ganesh.  I read through this and your InfoQ
> article (http://www.infoq.com/articles/scim-data-model-limitations) and
> have some thoughts.****
>
> ** **
>
> > The rest of the protocol does not meaningfully use the enterprise
> client’s identifier, the "external ID"****
>
> > at all, even though it was ostensibly introduced to make things
> friendlier for the client.****
>
> ** **
>
> The usage pattern for an external ID would be to search for a user by
> externalId and use the ID of the returned user in any desired operation.
> For example:****
>
> ** **
>
> GET /Users?filter=externalId eq “bjensen”&attributes=id****
>
> ** **
>
> {****
>
>   “totalResults”: 1,****
>
>   “Resources”: [****
>
>     {****
>
>       “id”: “2819c223-7f76-453a-919d-413861904646”****
>
>     }****
>
>   ]****
>
> }****
>
> ** **
>
> Retrieve the ID from the response and use it.****
>
> ** **
>
> DELETE /Users/2819c223-7f76-453a-919d-413861904646****
>
> ** **
>
> This does introduce an additional HTTP request if the client chooses not
> to store the server’s id.  An issue was created to consider allowing
> operations to use the externalId (
> http://code.google.com/p/scim/issues/detail?id=35), but I believe the
> general consensus has been to not include this in the spec.  One main point
> of contention is that much of the rest of the spec (eg – group membership
> references, manager references, etc…) require knowledge of the server’s
> identifier.  Continuing this discussion on the IETF list would be a good
> thing, though.****
>
> ** **
>
> ** **
>
> > the cloud provider's ID and the enterprise client's ID are both
> "Internal IDs" with respect to their domains****
>
> ** **
>
> I think this comes down to a nomenclature problem.  The server’s ID does
> not necessarily have to be the unique identifier that the underlying
> identity store uses, it just has to be stable and unique.  In many cases,
> the underlying identity store will provide identifiers with these
> properties already (eg – a uuid) and it can be used by the SCIM interface.
> The “externalId” is referring to the fact that the id is maintained
> external to the SCIM server.  As long as the server’s identifiers are
> stable and unique (which is mandated by the spec), I don’t see a problem.*
> ***
>
> ** **
>
> ** **
>
> > The secret is that *every value needs a key*, and multi-valued
> attributes lack that. So our solution is quite****
>
> > simple - turn every list or array (of values) into a dictionary (of
> key-value pairs) by providing each value****
>
> > with a unique and meaning-free identifier.****
>
> ** **
>
> I agree that this would be useful, especially in the PATCH operation.  One
> reason that this wasn’t included in the spec originally is that it can put
> undue burden on the service provider.  Many service providers are putting
> SCIM interfaces in front of their existing identity stores (eg – directory
> servers, SaaS application databases, etc…).  Many of these do not have a
> unique identifier for multi-valued attributes.  By requiring this, a
> majority of the server providers would have to start maintaining a unique
> key for each multi-valued attribute.  I believe this would be a roadblock
> for many implementers.****
>
> ** **
>
> ** **
>
> > When the SCIM protocol uses PATCH, there are areas where it seems a bit
> clumsy.****
>
> ** **
>
> I like the thoughts here.  Your example reminds me of unified diffs (
> http://en.wikipedia.org/wiki/Diff#Unified_format), which are commonly
> used with a patch program (pretty much the equivalent of the PATCH verb).
>  However, the three proposals seem to largely hinge on being able to
> uniquely address each element within an object.  Without these it is not so
> easy to address each patch sub-operation (REPLACE, INCLUDE, etc…) or
> provide a multi-status.****
>
>
> The 207 response would be interesting to consider for the bulk endpoint (
> http://www.simplecloud.info/specs/draft-scim-api-00.html#bulk-resources),
> however.****
>
> ** **
>
> ** **
>
> > There are other, non-data aspects of SCIM which may require review,
> such as its synchronous request-response****
>
> > interaction model, which is a form of tight coupling and could prove to
> be a source of brittleness.****
>
> ** **
>
> I agree that we should explore optional asynchronous requests in 2.0.****
>
> ** **
>
> Thanks again for your thoughts.  I hope you stay involved in the
> discussion as work on SCIM 2.0 goes forward.****
>
> ** **
>
> --Kelly****
>
> ** **
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Ganesh and Sashi Prasad
> *Sent:* Wednesday, August 01, 2012 4:24 PM
> *To:* [email-masked]
> *Subject:* [scim] SCIM Protocol - 3 suggestions for improvement****
>
> ** **
>
> the mailing list and post it here instead, so here goes.)****
>
> ** **
>
> Hi,****
>
> ** **
>
> http://www.infoq.com/minibooks/Identity-Management-Shoestring).****
>
> ** **
>
> the following 3 suggestions to make.****
>
> ** **
>
> this.****
> ** **
> manipulating it will be clumsy and inelegant.****
>
> ** **
>
> 3. The PATCH command can be improved in 3 significant ways:****
>
> 3a. Leverage the fact (from 2 above) that every value has a key, to
> greatly simplify the API****
>
> modify and delete attributes at any level****
>
> 3c. Use the WebDAV status code of "207 Multi-Status" instead of "200 OK"
> as the response to a PATCH (or BULK) command.****
>
> ** **
>
> To elaborate,****
>
> ** **
>
> created for this explicit purpose, and only reveal this.****
>
> ** **
>
> POST response.****
> ** **
> multi-valued, e.g.,****
> ** **
>     "email-addrs" : ****
>     [****
>         "[email-masked]",****
>
>         "[email-masked]",****
>
>         "[email-masked]"****
>
>     ]****
>
> ** **
>
> representation of the resource, and this attribute should look like this:*
> ***
>
> ** **
>
>     "email-addrs" : ****
>
>     [****
>
>         { "7dfcb444-74d8-4f17-aa66-daf9ea3bd902" : "[email-masked]"
> },****
>
>         { "3bd10085-c474-43b9-9cda-8646c3085bbf" : "[email-masked]"
> },****
>
> }****
>     ]****
> ** **
> unambiguous way to reference a value to add, modify and delete it:****
>
> ** **
>
> Add:****
>
> ** **
>
> POST /Users/2819c223-7f76-453a-919d-413861904646/email-addrs****
>
> value="[email-masked]"****
>
> ** **
>
> Modify:****
>
> ** **
>
> PUT
> ****
> value="[email-masked]"****
>
> ** **
>
> Delete:****
>
> DELETE
> ****
> ** **
> One can even delete all email addresses like this:****
>
> DELETE /Users/2819c223-7f76-453a-919d-413861904646/email-addrs****
>
> ** **
>
> I believe this is more elegant than what the spec recommends.****
>
> ** **
>
> semantics apply throughout the "tree" of a resource.****
>
> ** **
>
> 6 separate verbs that are a little more unambiguous in their meaning:****
>
> ** **
>
> return a generated URI****
>
> 2. PLACE (equivalent to one form of PUT): Add this resource at the
> location specified by the accompanying URI. (If there’s already a value at
> that location, return an error status.)****
>
> such URI, return an error status.)****
>
> accompanying value whether the URI already existed or not.)****
>
> inaccessible the resource at the specified URI.****
>
> 6. AMEND (equivalent to PATCH): (This verb is just listed for
> completeness. We probably don’t need a nested PATCH since PATCH cascades to
> every level of the tree.)****
>
> ** **
>
> A PATCH request could therefore look like this:****
>
> ** **
>
> PATCH /Users/2819c223-7f76-453a-919d-413861904646 HTTP/1.1****
>
> Host: example.com****
>
> Accept: application/json****
>
> Authorization: Bearer h480djs93hd8****
>
> Content-length: ...****
>
> ** **
>
> {****
>
>     REPLACE: {****
>
>         "key" : "first-name",****
>
>         "value" : "Jack"****
>
>     },****
>
>     PLACE : {****
>
>         "key" : "middle-name",****
>
>         "value" : "Richard"****
>
>     },****
>
>     FORCE : {****
>
>         "key" : "dob",****
>
>         "value" : "01-Jan-1971"****
>
>     },****
>
>     REPLACE : {****
>
>         "key" : "address.unit-number",****
>
>         "value" : "12"****
>
>     },****
>
>     PLACE : {****
>
>         "key" : "address.state",****
>
>         "value" : "SA"****
>
>     },****
>
>     FORCE : {****
>
>         "key" : "address.country",****
>
>         "value" : "Australia"****
>
>     },****
>
>     INCLUDE : {****
>
>         "key" : "email-addrs",****
>
>         "value" : "[email-masked]"****
>
>     },****
>
>     REPLACE : {****
>
>         "key" : "email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf",****
>
>         "value" : "[email-masked]"****
>
>     },****
>
>     RETIRE : {****
>
>         "key" : "email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1"****
>
>     }****
>
> }****
>
> ** **
>
> response is below:****
> ** **
> HTTP/1.1 207 Multi-Status****
>
> Content-Type: application/json****
>
> ETag: W/"b431af54f0671a2"****
>
> Location:"
> https://example.com/v1/Users/2819c223-7f76-453a-919d-413861904646"****
>
> ** **
>
> {****
>
>     "schemas":["urn:scim:schemas:core:1.0"],****
>
>     "external-id":"2819c223-7f76-453a-919d-413861904646",****
>
>     REPLACE: {****
>
>         "status" : "200 OK",****
>
>         "key" : "first-name",****
>
>         "value" : "Jack"****
>
>     },****
>
>     PLACE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "middle-name",****
>
>         "value" : "Richard"****
>
>     },****
>
>     FORCE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "dob",****
>
>         "value" : "01-Jan-1971"****
>
>     },****
>
>     REPLACE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "address.unit-number",****
>
>         "value" : "12"****
>
>     },****
>
>     PLACE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "address.state",****
>
>         "value" : "SA"****
>
>     },****
>
>     FORCE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "address.country",****
>
>         "value" : "Australia"****
>
>     },****
>
>     INCLUDE : {****
>
>         "status" : "201 Created",****
>
>         "key" : "email-addrs/11f664ec-898b-4f6f-8948-ecfda74deff0",****
>
>         "value" : "[email-masked]"****
>
>     },****
>
>     REPLACE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "email-addrs/3bd10085-c474-43b9-9cda-8646c3085bbf",****
>
>         "value" : "[email-masked]"****
>
>     },****
>
>     RETIRE : {****
>
>         "status" : "200 OK",****
>
>         "key" : "email-addrs/581da5c7-c6e1-4cca-9db7-7a6d1de664e1"****
>
>     }****
>
>     "meta": {****
>
>         "created":"2011-08-08T04:56:22Z",****
>
>         "lastModified":"2011-08-08T08:00:12Z",****
>
>         "location":"
> https://example.com/v1/Users/2819c223-7f76-453a-919d-413861904646",****
>
>         "version":"W\/\"b431af54f0671a2\""****
>
>     }****
>
> }****
>
> ** **
>
> will remain "207 Multi-Status".****
>
> ** **
>
> and for bulk operations.****
>
> ** **
>
> I hope you find these suggestions useful.****
>
> ** **
>
> I read the SCIM spec afresh last week and these ideas came flooding into
> my head because I have been working at another organisation (a telco) for
> the last 5 months, also in Identity and Access Management, and my thoughts
> have moved further along the direction of evolving a specialised data model
> based on specific principles, especially for IAM.****
>
> ** **
>
> I am planning to write about this and also the data-related principles
> soon and am in negotiations with InfoQ regarding publication.****
>
> ** **
>
> Regards,****
>
> Ganesh Prasad****
>

---

From: Emmanuel Dreux <[email-masked]>
Date: Thu, 9 Aug 2012 08:18:08 +0000

Hi Ganesh,

Nothing prevents you in your SCIM implementation (client or server) to generate a unique identifier for each synchronized object and maintain an internal mapping table ( you would have to map group membership as well).

This is what we are doing with Active Directory sources or targets:
As we didn't find an immutable uniqueID in AD systems (DN,samAccountName, UPN) are subject to change (even objectGuid can change if an AD domain is migrated), we decided to generate and maintain an internal table of ids. This fits your requirements as it hides internal ids.

This was written in dotnet and we have started a project to rewrite our SCIM stack in PHP and will give it to the Open Source community. This implementation will have a parameter : AllocateIds versus UseExistingIDs.
This will give the choice of "hiding" internalIDs or use them as unique ID.

You can also implement such feature without violating the SCIM specs, or without asking to include it in the specs.

--
Regards,
Emmanuel Dreux
http://www.cloudiway.com
Tel: +33 4 26 78 17 58
Mobile: +33 6 47 81 26 70
skype: Emmanuel.Dreux

De : Ganesh and Sashi Prasad [mailto:[email-masked]]
Envoyé : jeudi 9 août 2012 03:35
À : Kelly Grizzle
Cc : [email-masked]
Objet : Re: [scim] SCIM Protocol - 3 suggestions for improvement

Hi Kelly,

Thanks for your response. Let me first respond in brief to the two main points you have made, and then elaborate on the first.

1.      Why should domains not expose their internal identifiers to other domains?

a.

We are designing a protocol for a federated system of domains, where all domains are co-equal peers. (In physics too, N-body problems are much harder than 2-body problems. :-) Therefore, assuming that there are only two players in the interaction makes this tightly coupled in a number of ways. We should rely on messaging and notification, with encapsulation of domain-specific data.

b.

In any non-trivial data store, there will always be the ongoing need to merge and split identities as and when "false negatives" and "false positives" are discovered. A domain should be able to handle this internal housekeeping freely, only notifying other domains when convenient. Mapping of internal identifiers to external ones and maintaining this mapping internally allows this loosely-coupled housekeeping to take place. Sharing internal identifiers (or otherwise outsourcing the mapping of internal to external identifiers) forces housekeeping activities to be done in lock-step across domains.

c.

Asynchronous interaction is not just a matter of a suitable wire protocol which can be designed later. The data model plays a crucial role in enabling or constraining such interaction. A tightly-coupled data model will force the use of synchronous interactions, and the exposure of internal identifiers is a key part of this tight coupling.

2. The difficulty of assigning unique identifiers to the individual values of multi-valued attributes:

a.

I'm not belittling the effort involved in migrating legacy data stores to such a model. However, in the larger historical context of cross-domain identity management, we are really at the very early stages. If a relatively new discipline and a brand new spec are held captive to legacy considerations, we are losing an opportunity to provide a clean and elegant model to subsequent users of the spec, and this will have repercussions over many years or even decades.

b.

If incumbent cloud providers find it hard to immediately adopt the dictionary model for existing multi-valued attributes, they can transition to this model by offering both "SCIM-compliant" and "non-SCIM-compliant" APIs to their customers and encouraging new customers to adopt the "SCIM-compliant" API. Legacy customers can be supported using a "non-SCIM-compliant" API for an arbitrarily long period and gradually migrated to the SCIM-compliant API. The logistics are not insurmountable, and shouldn't prevent the adoption of a dictionary model for multi-valued attributes.

Elaboration of Point 1:

When we consider federated identity across more than one domain, we have to assume that domains are not necessarily master-slave in their interaction. The most generic interaction model is peer-to-peer, where entity lifecycle events within a domain are notified to other domains (when necessary) in an asynchronous manner (i.e., through messaging) and the other domains are free to respond to these events in an appropriate manner and at a time of their convenience.

A key set of lifecycle events for an entity is the merging and splitting of identity that is often required.

The question "Is this one entity?" can be answered either yes (positive) or no (negative). But sometimes, we can discover false positives and false negatives in our data stores.

Consider a case where customers sign up online, and two customers who are privacy-conscious enter fake IDs such as "John Smith", and also use the same date of birth (say, 1 Jan 1970) or similar attributes. The front-end application may make an intelligent (but incorrect) guess that these two persons are the same, and re-assign the same identifier to the second person. This is a false positive. They appear to be the same entity, but they're actually different. When the error is discovered, the identities will need to be split, with a new identifier generated for one of them.

Consider the opposite case where a customer signs up through two different portals or in two different sessions, using the names "JSmith" and "JohnS". It is very likely that they will be treated as two different customers and assigned two unique identifiers. This is a false negative. They appear to be two entities, but are actually the same. At a later stage, when the error is discovered, the identities will have to be merged, and one of the identifiers will have to be dropped.

These are not theoretical use cases. They form a significant proportion of the user base in most large Web-facing applications. Let's see how these can be managed in a federated way by mapping internal identifiers to external ones and only exposing external identifiers to other domains.

a. False positives:

Attributes: {name: "John Smith", dob: "01-Jan-1970"}

When requesting the provisioning of this entity in Domain 2, the following ID is returned by Domain 2: ff487230b3a0.

Domain 1 then maintains the following in a mapping table and uses it for translation when talking to Domain 2, taking care never to expose its internal identifier:

| Internal Entity ID | External Domain ID | External Entity ID | Primary flag |

When the false positive is discovered and the entity is split, Domain 1 creates a new internal identifier and now has the following entity information.

Attributes: {name: "John Smith", dob: "01-Jan-1970"}

Attributes: {name: "John Smith", dob: "01-Jan-1970"}

This second entity with its own internal identifier is invisible to Domain 2, and this is by design. Communication about the original entity takes place as before by mapping "9caf78aac3d6" to "ff487230b3a0" and vice-versa. At some convenient time (importantly, this doesn't have to be at the time the split happens), Domain 2 can be requested to provision a second entity, and when it responds with an identifier of "7a87f27c1dd8", this can go into the mapping table as a new record associated with the second entity's internal identifier.

| Internal Entity ID | External Domain ID | External Entity ID | Primary flag |

Domain 2 is not even aware that a split has happened, and the provisioning that it does is not in lockstep with the split in identity that occurred in Domain 1.

(What is the "Primary flag" used for? We'll see when we cover the treatment of false negatives.)

b. False negatives:

Domain 1 has the following information about what it thinks are two distinct customers in its data store:

Attributes: {name: "JSmith", dob: "01-Jan-1970"}

Attributes: {name: "JohnS", dob: "01-Jan-1970"}

When requesting the provisioning of these entities in Domain 2, the following IDs are returned by Domain 2: ff487230b3a0 and 41206cc97c8b.

Domain 1 then maintains the following in a mapping table and uses it for translation when talking to Domain 2, taking care never to expose its internal identifiers:

| Internal Entity ID | External Domain ID | External Entity ID | Primary flag |

When the false negative is discovered and the two entities are merged, Domain 1 drops one of the internal identifiers and rationalises the name of the customer (say, to "John Smith"). Let's say it retains the first ID "9caf78aac3d6" and drops the second "273d36e30d09".

| Internal Entity ID | External Domain ID | External Entity ID | Primary flag |

Now two external identifiers map to the same internal one, so inbound communication from Domain 2 can be unambiguously translated to the same entity internally. However, when going outwards, Domain 1 will have to look up the translation table to determine the "primary" external ID for this entity in Domain 2, which was decided to be "ff487230b3a0". That's where the "Primary flag" comes in. The second external ID "41206cc97c8b" is never used thereafter in outbound communication.

At some stage (importantly, not in lockstep with the identity merge), Domain 2 can be requested to delete the customer record identified by "41206cc97c8b", and the second entry in the mapping table can be removed once this is acknowledged.

This scheme will scale up to multiple domains, because the "External Domain ID" column helps to keep track of which external ID is shared with which Domain. (Why don't we use just one external ID for an entity and share it with all external domains? Tight coupling again. Just as OAuth allows an access token given to a third party to be invalidated without affecting the access of other third parties, the use of separate external identifiers for different domains allows fine-grained control of identity federation.)

The scheme also allows the splitting of an entity into more than two entities, and the merging of more than two entities into a single one. (Any organisation with a web-facing application will tell you how many John Smiths there are who were born on 1 Jan 1970!)

This is a fairly long-winded explanation, but this is why we need to hide internal identifiers from other domains, and why mappings need to be managed internally in each domain. Such a data model also allows us to choose asynchronous protocols for propagation of identity events, since there is no consistency requirement to update multiple domains concurrently.

Regards,

Ganesh Prasad

On 9 August 2012 04:55, Kelly Grizzle <[email-masked]<mailto:[email-masked]>> wrote:
Thanks for the feedback, Ganesh.  I read through this and your InfoQ article (http://www.infoq.com/articles/scim-data-model-limitations) and have some thoughts.

{
  "totalResults": 1,
  "Resources": [
    {

--Kelly

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Ganesh and Sashi Prasad
Sent: Wednesday, August 01, 2012 4:24 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [scim] SCIM Protocol - 3 suggestions for improvement

Hi,

To elaborate,

    "email-addrs" :
    [

    "email-addrs" :
    [

Add:

Modify:

Delete:

{
    REPLACE: {
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {

{

Regards,
Ganesh Prasad

---

From: Kelly Grizzle <[email-masked]>
Date: Thu, 9 Aug 2012 14:28:11 +0000

Thanks Emmanuel.  I had started writing up a similar response.  As you suggest, storing this information in a mapping table outside of the SCIM spec is a great way to enable this solution.  Part of the key here is that SCIM is just a piece of the architecture for this solution, and is only responsible for the transport layer between domains.

You could also model these ID mappings in the SCIM user as an extension but would probably not want to expose these externally.  Here is an example of how to model the end state of the false positive scenario (splitting a user):

This could be represented as two SCIM users that contain information about the entities on other domains.

{
  "schemas": ["urn:scim:schemas:core:1.0", "urn:scim:schemas:extension:federation:1.0"],
  "id": "9caf78aac3d6",
  "userName": "John Smith",
  "urn:scim:schemas:extension:federation:1.0": {
    "linkedUsers": [
      {
        "domain": "D2",
        "externalEntityId": "ff487230b3a0"
      }
    ]
  }
}

{
  "schemas": ["urn:scim:schemas:core:1.0", "urn:scim:schemas:extension:federation:1.0"],
  "id": "a99a5feba839",
  "userName": "John Smith",
  "urn:scim:schemas:extension:federation:1.0": {
    "linkedUsers": [
      {
        "domain": "D2",
        "externalEntityId": "7a87f27c1dd8"
      }
    ]
  }
}

In the second user, the linkedUsers attribute would be empty until the split user was synced to domain 2.

Similarly, the false negative use case (merging two users) looked like this at the end:

This could be represented with the following SCIM user:

{
  "schemas": ["urn:scim:schemas:core:1.0", "urn:scim:schemas:extension:federation:1.0"],
  "id": "9caf78aac3d6",
  "userName": "John Smith",
  "urn:scim:schemas:extension:federation:1.0": {
    "linkedUsers": [
      {
        "domain": "D2",
        "externalEntityId": "ff487230b3a0"
      },
      {
        "domain": "D2",
        "externalEntityId": "41206cc97c8b",
        "deletionRequired": true
      }
    ]
  }
}

Regarding unique identifiers for multi-valued attributes there is a trade-off involved.  On one hand this makes PATCH semantics easier.  On the other hand it puts extra burden on service providers.  Since the inception of SCIM, a key goal has been to foster adoption by service providers by making things fit easily onto existing systems.  IMO the value gained by unique identifiers for multi-valued attributes is not worth the demands put on a service provider.  I also think that vendors that have a non-SCIM-compliant API will choose to keep things that way if the spec is too hard for them to implement.  In a green field environment we do have the luxury of mandating a model to make certain operations more elegant.  However, we can't ignore legacy systems.

--Kelly

From: [email-masked] [mailto:[email-masked]] On Behalf Of Emmanuel Dreux
Sent: Thursday, August 09, 2012 3:18 AM
To: Ganesh and Sashi Prasad; Kelly Grizzle
Cc: [email-masked]
Subject: Re: [scim] SCIM Protocol - 3 suggestions for improvement

Hi Ganesh,

De : Ganesh and Sashi Prasad [mailto:[email-masked]]
Envoyé : jeudi 9 août 2012 03:35
À : Kelly Grizzle
Cc : [email-masked]<mailto:[email-masked]>
Objet : Re: [scim] SCIM Protocol - 3 suggestions for improvement

Hi Kelly,

a.

b.

c.

a.

b.

Elaboration of Point 1:

a. False positives:

b. False negatives:

Regards,

Ganesh Prasad

{
  "totalResults": 1,
  "Resources": [
    {

--Kelly

Hi,

To elaborate,

    "email-addrs" :
    [

    "email-addrs" :
    [

Add:

Modify:

Delete:

{
    REPLACE: {
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {

{

Regards,
Ganesh Prasad

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Fri, 10 Aug 2012 07:14:54 +1000

>  storing this information in a mapping table outside of the SCIM spec is
a great way to enable this solution.  Part of the key here is that SCIM is
just a piece of the architecture for this solution, and is only responsible
for the transport layer between domains.

I wasn't suggesting that the mapping table be part of the SCIM spec. I
provided that example to illustrate that splitting and merging identities
is a common requirement, and that decoupling local identifiers within a
domain from shared identifiers between domains was the best way to
facilitate it.

I'm suggesting that the spec do less, not more.

What the SCIM spec needs to do there is just refrain from introducing tight
coupling. I would like to see a single identifier exposed through the API,
with the implication (and perhaps the recommendation) that it be the shared
one. Allowing one domain to expose its internal identifier to the other
creates tight coupling and ensures that both domains need simultaneously
split or merge identities, which is not desirable. So I recommend _taking
out_ the "external id" field from the API. The spec shouldn't encourage
tight coupling. If clients want to pass in their internal ids as part of
the resource body, no one can stop them, and they can always do a search on
that attribute to retrieve the URI exactly as you visualise they will with
the "external id", but let's not elevate an anti-pattern to a
recommendation by enshrining the "external id" as an acceptable attribute.

Am I making sense?

>  Regarding unique identifiers for multi-valued attributes there is a
trade-off involved.  On one hand this makes PATCH semantics easier.  On the
other hand it puts extra burden on service providers.

Precisely. The spec has to strike the right balance. It would be
interesting to hear from the other members of the spec mailing list. You
know where I stand on this. It would be good to hear the spectrum of
opinions.

Regards,
Ganesh

On 10 August 2012 00:28, Kelly Grizzle <[email-masked]> wrote:

>  Thanks Emmanuel.  I had started writing up a similar response.  As you
> suggest, storing this information in a mapping table outside of the SCIM
> spec is a great way to enable this solution.  Part of the key here is that
> SCIM is just a piece of the architecture for this solution, and is only
> responsible for the transport layer between domains.****
>
> ** **
>
> You could also model these ID mappings in the SCIM user as an extension
> but would probably not want to expose these externally.  Here is an example
> of how to model the end state of the false positive scenario (splitting a
> user):****
>
> ** **
>
> flag |****
> | 9caf78aac3d6       | D2                 | ff487230b3a0       |
> true         |****
>
> | a99a5feba839       | D2                 | 7a87f27c1dd8       |
> true         |****
>
> ** **
>
> This could be represented as two SCIM users that contain information about
> the entities on other domains.****
>
> ** **
>
> {****
>
>   "schemas": ["urn:scim:schemas:core:1.0",
> "urn:scim:schemas:extension:federation:1.0"],****
>
>   "id": "9caf78aac3d6",****
>
>   "userName": "John Smith",****
>
>   "urn:scim:schemas:extension:federation:1.0": {****
>
>     "linkedUsers": [****
>
>       {****
>
>         "domain": "D2",****
>
>         "externalEntityId": "ff487230b3a0"****
>
>       }****
>
>     ]****
>
>   }****
>
> }****
>
> ** **
>
> {****
>
>   "schemas": ["urn:scim:schemas:core:1.0",
> "urn:scim:schemas:extension:federation:1.0"],****
>
>   "id": "a99a5feba839",****
>
>   "userName": "John Smith",****
>
>   "urn:scim:schemas:extension:federation:1.0": {****
>
>     "linkedUsers": [****
>
>       {****
>
>         "domain": "D2",****
>
>         "externalEntityId": "7a87f27c1dd8"****
>
>       }****
>
>     ]****
>
>   }****
>
> }****
>
> ** **
>
> In the second user, the linkedUsers attribute would be empty until the
> split user was synced to domain 2.****
>
> ** **
>
> ** **
>
> Similarly, the false negative use case (merging two users) looked like
> this at the end:****
>
> ** **
>
> flag |****
> | 9caf78aac3d6       | D2                 | ff487230b3a0       |
> true         |****
>
> | 9caf78aac3d6       | D2                 | 41206cc97c8b       |
> false        |****
>
> ** **
>
> This could be represented with the following SCIM user:****
>
> ** **
>
> {****
>
>   "schemas": ["urn:scim:schemas:core:1.0",
> "urn:scim:schemas:extension:federation:1.0"],****
>
>   "id": "9caf78aac3d6",****
>
>   "userName": "John Smith",****
>
>   "urn:scim:schemas:extension:federation:1.0": {****
>
>     "linkedUsers": [****
>
>       {****
>
>         "domain": "D2",****
>
>         "externalEntityId": "ff487230b3a0"****
>
>       },****
>
>       {****
>
>         "domain": "D2",****
>
>         "externalEntityId": "41206cc97c8b",****
>
>         "deletionRequired": true****
>
>       }****
>
>     ]****
>
>   }****
>
> }****
>
> ** **
>
> ** **
>
> Regarding unique identifiers for multi-valued attributes there is a
> trade-off involved.  On one hand this makes PATCH semantics easier.  On the
> other hand it puts extra burden on service providers.  Since the inception
> of SCIM, a key goal has been to foster adoption by service providers by
> making things fit easily onto existing systems.  IMO the value gained by
> unique identifiers for multi-valued attributes is not worth the demands put
> on a service provider.  I also think that vendors that have a
> non-SCIM-compliant API will choose to keep things that way if the spec is
> too hard for them to implement.  In a green field environment we do have
> the luxury of mandating a model to make certain operations more elegant.
> However, we can’t ignore legacy systems. ****
>
> ** **
>
> --Kelly****
>
> ** **
>
> Of *Emmanuel Dreux
> *Sent:* Thursday, August 09, 2012 3:18 AM
> *To:* Ganesh and Sashi Prasad; Kelly Grizzle
> *Cc:* [email-masked]
> *Subject:* Re: [scim] SCIM Protocol - 3 suggestions for improvement****
>
> ** **
>
> Hi Ganesh,****
>
> ** **
>
> Nothing prevents you in your SCIM implementation (client or server) to
> generate a unique identifier for each synchronized object and maintain an
> internal mapping table ( you would have to map group membership as well).*
> ***
>
> ** **
>
> This is what we are doing with Active Directory sources or targets:****
>
> As we didn’t find an immutable uniqueID in AD systems (DN,samAccountName,
> UPN) are subject to change (even objectGuid can change if an AD domain is
> migrated), we decided to generate and maintain an internal table of ids.
> This fits your requirements as it hides internal ids.****
>
> ** **
>
> This was written in dotnet and we have started a project to rewrite our
> SCIM stack in PHP and will give it to the Open Source community. This
> implementation will have a parameter : AllocateIds versus UseExistingIDs.*
> ***
>
> This will give the choice of “hiding” internalIDs or use them as unique ID.
> ****
>
> ** **
>
> You can also implement such feature without violating the SCIM specs, or
> without asking to include it in the specs.****
>
> ** **
>
> --****
>
> Regards,****
>
> Emmanuel Dreux****
>
> http://www.cloudiway.com****
>
> Tel: +33 4 26 78 17 58****
>
> Mobile: +33 6 47 81 26 70****
>
> skype: Emmanuel.Dreux****
>
> ** **
>
> *De :* Ganesh and Sashi Prasad [mailto:[email-masked]<[email-masked]>]
>
> *Envoyé :* jeudi 9 août 2012 03:35
> *À :* Kelly Grizzle
> *Cc :* [email-masked]
> *Objet :* Re: [scim] SCIM Protocol - 3 suggestions for improvement****
>
> ** **
>
> Hi Kelly,****
>
> points you have made, and then elaborate on the first.****
>
> **1.      **Why should domains not expose their internal identifiers to
> other domains?****
>
> a.****
>
> domain-specific data.****
>
> b. ****
>
> lock-step across domains.****
>
> c.****
>
> internal identifiers is a key part of this tight coupling.****
>
> of multi-valued attributes:****
>
> a. ****
>
> over many years or even decades.****
>
> b. ****
>
> attributes.****
> Elaboration of Point 1:****
>
> When we consider federated identity across more than one domain, we have
> to assume that domains are not necessarily master-slave in their
> interaction. The most generic interaction model is peer-to-peer, where
> entity lifecycle events within a domain are notified to other domains (when
> necessary) in an asynchronous manner (i.e., through messaging) and the
> other domains are free to respond to these events in an appropriate manner
> and at a time of their convenience.****
>
> A key set of lifecycle events for an entity is the merging and splitting
> of identity that is often required.****
>
> The question “Is this one entity?” can be answered either yes (positive)
> or no (negative). But sometimes, we can discover false positives and false
> negatives in our data stores.****
>
> will need to be split, with a new identifier generated for one of them.***
> *
>
> identifiers will have to be dropped.****
>
> external ones and only exposing external identifiers to other domains.****
>
> a. False positives:****
>
> Internal ID: 9caf78aac3d6****
>
> Attributes: {name: “John Smith”, dob: “01-Jan-1970”}****
>
> ID is returned by Domain 2: ff487230b3a0.****
>
> internal identifier:****
> flag |****
> | 9caf78aac3d6 | D2 | ff487230b3a0 | true |****
>
> information.****
> Internal ID: 9caf78aac3d6****
>
> Attributes: {name: “John Smith”, dob: “01-Jan-1970”}****
>
> Internal ID: a99a5feba839****
>
> Attributes: {name: “John Smith”, dob: “01-Jan-1970”}****
>
> internal identifier.****
> The mapping table now contains the following entries:****
>
> flag |****
> | 9caf78aac3d6 | D2 | ff487230b3a0 | true |****
>
> | a99a5feba839 | D2 | 7a87f27c1dd8 | true |****
>
> Domain 1.****
> (What is the “Primary flag” used for? We'll see when we cover the
> treatment of false negatives.)****
>
> b. False negatives:****
>
> distinct customers in its data store:****
>
> Internal ID: 9caf78aac3d6****
>
> Attributes: {name: “JSmith”, dob: “01-Jan-1970”}****
>
> Internal ID: 273d36e30d09****
>
> Attributes: {name: “JohnS”, dob: “01-Jan-1970”}****
>
> following IDs are returned by Domain 2: ff487230b3a0 and 41206cc97c8b.****
>
> internal identifiers:****
>
> flag |****
> | 9caf78aac3d6 | D2 | ff487230b3a0 | true |****
>
> | 273d36e30d09 | D2 | 41206cc97c8b | true |****
>
> “9caf78aac3d6” and drops the second “273d36e30d09”.****
>
> The mapping table now looks like this:****
>
> flag |****
> | 9caf78aac3d6 | D2 | ff487230b3a0 | true |****
>
> | 9caf78aac3d6 | D2 | 41206cc97c8b | false |****
>
> used thereafter in outbound communication.****
>
> Domain 2 can be requested to delete the customer record identified by
> “41206cc97c8b”, and the second entry in the mapping table can be removed
> once this is acknowledged.****
>
> This scheme will scale up to multiple domains, because the “External
> Domain ID” column helps to keep track of which external ID is shared with
> which Domain. (Why don't we use just one external ID for an entity and
> share it with all external domains? Tight coupling again. Just as OAuth
> allows an access token given to a third party to be invalidated without
> affecting the access of other third parties, the use of separate external
> identifiers for different domains allows fine-grained control of identity
> federation.)****
>
> Smiths there are who were born on 1 Jan 1970!)****
>
> Regards, ****

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Fri, 10 Aug 2012 07:25:08 +1000

>  It would be interesting to hear from the other members of the spec
mailing list.

It would be good if members also state their affiliations when they post
their opinions (e.g., cloud service provider, ISV, enterprise customer,
etc.) For the record, I'm an architect currently contracting with a telco
that is an enterprise customer of cloud services.

Regards,
Ganesh

On 10 August 2012 07:14, Ganesh and Sashi Prasad <[email-masked]>wrote:

> >  storing this information in a mapping table outside of the SCIM spec
> is a great way to enable this solution.  Part of the key here is that SCIM
> is just a piece of the architecture for this solution, and is only
> responsible for the transport layer between domains.
>
> What the SCIM spec needs to do there is just refrain from introducing
> tight coupling. I would like to see a single identifier exposed through the
> API, with the implication (and perhaps the recommendation) that it be the
> shared one. Allowing one domain to expose its internal identifier to the
> other creates tight coupling and ensures that both domains need
> simultaneously split or merge identities, which is not desirable. So I
> recommend _taking out_ the "external id" field from the API. The spec
> shouldn't encourage tight coupling. If clients want to pass in their
> internal ids as part of the resource body, no one can stop them, and they
> can always do a search on that attribute to retrieve the URI exactly as you
> visualise they will with the "external id", but let's not elevate an
> anti-pattern to a recommendation by enshrining the "external id" as an
> acceptable attribute.
>
> Am I making sense?
>
> On 10 August 2012 00:28, Kelly Grizzle <[email-masked]>wrote:
>
>> This could be represented as two SCIM users that contain information
>> about the entities on other domains.****
>>
>> ** **
>>
>> {****
>>
>> internal mapping table ( you would have to map group membership as well).
>> ****
>>
>> ** **
>>
>> implementation will have a parameter : AllocateIds versus UseExistingIDs.
>> ****
>>
>> This will give the choice of “hiding” internalIDs or use them as unique
>> ID.****
>>
>> ** **
>>
>> 2. The difficulty of assigning unique identifiers to the individual
>> values of multi-valued attributes:****
>>
>> a. ****
>>
>> will need to be split, with a new identifier generated for one of them.**
>> **
>>
>> Consider the opposite case where a customer signs up through two
>> different portals or in two different sessions, using the names “JSmith”
>> and “JohnS”. It is very likely that they will be treated as two different
>> customers and assigned two unique identifiers. This is a false negative.
>> They appear to be two entities, but are actually the same. At a later
>> stage, when the error is discovered, the identities will have to be merged,
>> and one of the identifiers will have to be dropped.****
>>
>> These are not theoretical use cases. They form a significant proportion
>> of the user base in most large Web-facing applications. Let's see how these
>> When requesting the provisioning of this entity in Domain 2, the
>> following ID is returned by Domain 2: ff487230b3a0.****
>>
>> This second entity with its own internal identifier is invisible to
>> Domain 2, and this is by design. Communication about the original entity
>> takes place as before by mapping “9caf78aac3d6” to “ff487230b3a0” and
>> vice-versa. At some convenient time (importantly, this doesn't have to be
>> at the time the split happens), Domain 2 can be requested to provision a
>> second entity, and when it responds with an identifier of “7a87f27c1dd8”,
>> this can go into the mapping table as a new record associated with the
>> second entity's internal identifier.****
>>
>> Domain 2 is not even aware that a split has happened, and the
>> provisioning that it does is not in lockstep with the split in identity
>> that occurred in Domain 1.****
>>
>> On 9 August 2012 04:55, Kelly Grizzle <[email-masked]>
>> wrote:****
>>
>> stable and unique (which is mandated by the spec), I don’t see a problem.
>> ****
>>
>>  ****
>>
>>  ****
>>
>> I agree that this would be useful, especially in the PATCH operation.
>> One reason that this wasn’t included in the spec originally is that it can
>> put undue burden on the service provider.  Many service providers are
>> putting SCIM interfaces in front of their existing identity stores (eg –
>> directory servers, SaaS application databases, etc…).  Many of these do not
>> have a unique identifier for multi-valued attributes.  By requiring this, a
>> > When the SCIM protocol uses PATCH, there are areas where it seems a
>> bit clumsy.****
>>
>>  ****
>>
>> (I posted this on the SCIM Google Group, and I was advised to subscribe
>> to the mailing list and post it here instead, so here goes.)****
>>
>>  ****
>>
>> Hi,****
>>
>>  ****
>>
>> I have been following the SCIM spec off and on, and based on my
>> experience with a loosely-coupled architecture that I found to be
>> successful, I have the following 3 suggestions to make.****
>>
>>  ****
>>
>>         { "581da5c7-c6e1-4cca-9db7-7a6d1de664e1" : "
>> [email-masked]" }****
>>
>>     ]****
>>
>>  ****
>>
>> 3. It's possible to think of the operations POST, PUT and DELETE as
>> nested operations inside a PATCH. PATCH itself need not be nested because
>> its semantics apply throughout the "tree" of a resource.****
>>
>>  ****
>>
>> 4. FORCE (equivalent to a third form of PUT): This means PLACE or
>> REPLACE. (At the end of this operation, we want the specified URI to hold
>> the accompanying value whether the URI already existed or not.)****
>>
>> The same scheme can be used to deal with operations on members of a
>> group, and for bulk operations.****
>>
>>  ****
>>

---

From: "Diodati,Mark" <[email-masked]>
Date: Thu, 9 Aug 2012 21:25:44 +0000

Hi Ganesh,

--It would be good to hear the spectrum of opinions ...

My assessment is that your proposal adds complexity, strays from the initial simplicity goals of SCIM, and hinders service provider adoption (which is crucial to the success of SCIM). I speak only for myself, but in my conversations with others in the WG, no one supported your proposal.

Mark

Mark Diodati
Research Vice President | Gartner
phone: +1 312.238.9877
blog: http://blogs.gartner.com/mark-diodati
twitter: mark_diodati

From: Ganesh and Sashi Prasad [mailto:[email-masked]]
Sent: Thursday, August 09, 2012 4:15 PM
To: Kelly Grizzle
Cc: [email-masked]; Emmanuel Dreux
Subject: Re: [scim] SCIM Protocol - 3 suggestions for improvement

>  storing this information in a mapping table outside of the SCIM spec is a great way to enable this solution.  Part of the key here is that SCIM is just a piece of the architecture for this solution, and is only responsible for the transport layer between domains.

I wasn't suggesting that the mapping table be part of the SCIM spec. I provided that example to illustrate that splitting and merging identities is a common requirement, and that decoupling local identifiers within a domain from shared identifiers between domains was the best way to facilitate it.

What the SCIM spec needs to do there is just refrain from introducing tight coupling. I would like to see a single identifier exposed through the API, with the implication (and perhaps the recommendation) that it be the shared one. Allowing one domain to expose its internal identifier to the other creates tight coupling and ensures that both domains need simultaneously split or merge identities, which is not desirable. So I recommend _taking out_ the "external id" field from the API. The spec shouldn't encourage tight coupling. If clients want to pass in their internal ids as part of the resource body, no one can stop them, and they can always do a search on that attribute to retrieve the URI exactly as you visualise they will with the "external id", but let's not elevate an anti-pattern to a recommendation by enshrining the "external id" as an acceptable attribute.

Am I making sense?

>  Regarding unique identifiers for multi-valued attributes there is a trade-off involved.  On one hand this makes PATCH semantics easier.  On the other hand it puts extra burden on service providers.

Precisely. The spec has to strike the right balance. It would be interesting to hear from the other members of the spec mailing list. You know where I stand on this. It would be good to hear the spectrum of opinions.

Regards,
Ganesh
On 10 August 2012 00:28, Kelly Grizzle <[email-masked]<mailto:[email-masked]>> wrote:
Thanks Emmanuel.  I had started writing up a similar response.  As you suggest, storing this information in a mapping table outside of the SCIM spec is a great way to enable this solution.  Part of the key here is that SCIM is just a piece of the architecture for this solution, and is only responsible for the transport layer between domains.

{

{

{

--Kelly

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Emmanuel Dreux
Sent: Thursday, August 09, 2012 3:18 AM
To: Ganesh and Sashi Prasad; Kelly Grizzle
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [scim] SCIM Protocol - 3 suggestions for improvement

Hi Ganesh,

--
Regards,
Emmanuel Dreux
http://www.cloudiway.com
Tel: +33 4 26 78 17 58<tel:%2B33%204%2026%2078%2017%2058>
Mobile: +33 6 47 81 26 70<tel:%2B33%206%2047%2081%2026%2070>
skype: Emmanuel.Dreux

Hi Kelly,

a.

b.

c.

a.

b.

Elaboration of Point 1:

a. False positives:

b. False negatives:

Regards,

Ganesh Prasad

{
  "totalResults": 1,
  "Resources": [
    {

--Kelly

Hi,

To elaborate,

    "email-addrs" :
    [

    "email-addrs" :
    [

Add:

Modify:

Delete:

{
    REPLACE: {
        "key" : "first-name",
        "value" : "Jack"
    },
    PLACE : {
        "key" : "middle-name",
        "value" : "Richard"
    },
    FORCE : {
        "key" : "dob",
        "value" : "01-Jan-1971"
    },
    REPLACE : {

{

Regards,
Ganesh Prasad

________________________________

This e-mail message, including any attachments, is for the sole use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Gartner makes no warranty that this e-mail is error or virus free.

---

From: prateek mishra <[email-masked]>
Date: Thu, 09 Aug 2012 17:40:43 -0400

Mark -

what is the basis of these strong opinions? I dont see any evidence or 
support for your statements in your message below.

I am a member of this working group, and I dont agree with you.

This is an IETF mailing list meant for discussion of an emerging 
technology, its possible you are not familiar with this type of discussion.
In that case, I encourage you to educate yourself in this area.

Thanks,
prateek
> My assessment is that your proposal adds complexity, strays from the 
> initial simplicity goals of SCIM, and hinders service provider 
> adoption (which is crucial to the success of SCIM). I speak only for 
> myself, but in my conversations with others in the WG, no one 
> supported your proposal.
>
> Mark
>
> Mark Diodati
>
> *From:*Ganesh and Sashi Prasad [mailto:[email-masked]]
> *Sent:* Thursday, August 09, 2012 4:15 PM
> *To:* Kelly Grizzle
> *Cc:* [email-masked]; Emmanuel Dreux
> *Subject:* Re: [scim] SCIM Protocol - 3 suggestions for improvement
>
> is a great way to enable this solution.  Part of the key here is that 
> SCIM is just a piece of the architecture for this solution, and is 
> only responsible for the transport layer between domains.
>
> provided that example to illustrate that splitting and merging 
> identities is a common requirement, and that decoupling local 
> identifiers within a domain from shared identifiers between domains 
> was the best way to facilitate it.
>
> tight coupling. I would like to see a single identifier exposed 
> through the API, with the implication (and perhaps the recommendation) 
> that it be the shared one. Allowing one domain to expose its internal 
> identifier to the other creates tight coupling and ensures that both 
> domains need simultaneously split or merge identities, which is not 
> desirable. So I recommend _taking out_ the "external id" field from 
> the API. The spec shouldn't encourage tight coupling. If clients want 
> to pass in their internal ids as part of the resource body, no one can 
> stop them, and they can always do a search on that attribute to 
> retrieve the URI exactly as you visualise they will with the "external 
> id", but let's not elevate an anti-pattern to a recommendation by 
> enshrining the "external id" as an acceptable attribute.
>
> Am I making sense?
>
> trade-off involved.  On one hand this makes PATCH semantics easier.  
> On the other hand it puts extra burden on service providers.
>
>
> interesting to hear from the other members of the spec mailing list. 
> You know where I stand on this. It would be good to hear the spectrum 
> of opinions.
>
> Regards,
>
> Ganesh
>
> On 10 August 2012 00:28, Kelly Grizzle <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> suggest, storing this information in a mapping table outside of the 
> SCIM spec is a great way to enable this solution.  Part of the key 
> here is that SCIM is just a piece of the architecture for this 
> solution, and is only responsible for the transport layer between domains.
>
> You could also model these ID mappings in the SCIM user as an 
> extension but would probably not want to expose these externally.  
> Here is an example of how to model the end state of the false positive 
> scenario (splitting a user):
>
> true         |
> true         |
> about the entities on other domains.
>
> {
>
> "urn:scim:schemas:extension:federation:1.0"],
>
>   "id": "9caf78aac3d6",
>
> "urn:scim:schemas:extension:federation:1.0"],
>
>   "id": "a99a5feba839",
>
> split user was synced to domain 2.
>
> this at the end:
> true         |
> false        |
> "urn:scim:schemas:extension:federation:1.0"],
>
>   "id": "9caf78aac3d6",
>
> trade-off involved.  On one hand this makes PATCH semantics easier.  
> On the other hand it puts extra burden on service providers.  Since 
> the inception of SCIM, a key goal has been to foster adoption by 
> service providers by making things fit easily onto existing systems.  
> IMO the value gained by unique identifiers for multi-valued attributes 
> is not worth the demands put on a service provider.  I also think that 
> vendors that have a non-SCIM-compliant API will choose to keep things 
> that way if the spec is too hard for them to implement.  In a green 
> field environment we do have the luxury of mandating a model to make 
> certain operations more elegant. However, we can't ignore legacy systems.
>
> --Kelly
>
> *From:[email-masked] <mailto:[email-masked]> 
> [mailto:[email-masked] <mailto:[email-masked]>] *On 
> Behalf Of *Emmanuel Dreux
> *Cc:* [email-masked] <mailto:[email-masked]>
> *Subject:* Re: [scim] SCIM Protocol - 3 suggestions for improvement
>
> Hi Ganesh,
>
> generate a unique identifier for each synchronized object and maintain 
> an internal mapping table ( you would have to map group membership as 
> well).
>
> As we didn't find an immutable uniqueID in AD systems 
> (DN,samAccountName, UPN) are subject to change (even objectGuid can 
> change if an AD domain is migrated), we decided to generate and 
> maintain an internal table of ids. This fits your requirements as it 
> hides internal ids.
>
> This was written in dotnet and we have started a project to rewrite 
> our SCIM stack in PHP and will give it to the Open Source community. 
> This implementation will have a parameter : AllocateIds versus 
> UseExistingIDs.
>
> You can also implement such feature without violating the SCIM specs, 
> or without asking to include it in the specs.
>
> --
>
> Regards,
>
> Emmanuel Dreux
>
> http://www.cloudiway.com
>
> Tel: +33 4 26 78 17 58 <tel:%2B33%204%2026%2078%2017%2058>
>
> Mobile: +33 6 47 81 26 70 <tel:%2B33%206%2047%2081%2026%2070>
>
> skype: Emmanuel.Dreux
>
> *De :*Ganesh and Sashi Prasad [mailto:[email-masked]]
> *Cc :* [email-masked] <mailto:[email-masked]>
> *Objet :* Re: [scim] SCIM Protocol - 3 suggestions for improvement
>
> Hi Kelly,
>
> Thanks for your response. Let me first respond in brief to the two 
> main points you have made, and then elaborate on the first.
>
> 1.Why should domains not expose their internal identifiers to other 
> domains?
>
> a.
>
> We are designing a protocol for a federated system of domains, where 
> all domains are co-equal peers. (In physics too, N-body problems are 
> much harder than 2-body problems. :-) Therefore, assuming that there 
> are only two players in the interaction makes this tightly coupled in 
> a number of ways. We should rely on messaging and notification, with 
> encapsulation of domain-specific data.
>
> b.
>
> In any non-trivial data store, there will always be the ongoing need 
> to merge and split identities as and when "false negatives" and "false 
> positives" are discovered. A domain should be able to handle this 
> internal housekeeping freely, only notifying other domains when 
> convenient. Mapping of internal identifiers to external ones and 
> maintaining this mapping internally allows this loosely-coupled 
> housekeeping to take place. Sharing internal identifiers (or otherwise 
> outsourcing the mapping of internal to external identifiers) forces 
> housekeeping activities to be done in lock-step across domains.
>
> c.
>
> Asynchronous interaction is not just a matter of a suitable wire 
> protocol which can be designed later. The data model plays a crucial 
> role in enabling or constraining such interaction. A tightly-coupled 
> data model will force the use of synchronous interactions, and the 
> exposure of internal identifiers is a key part of this tight coupling.
>
> values of multi-valued attributes:
>
> a.
>
> I'm not belittling the effort involved in migrating legacy data stores 
> to such a model. However, in the larger historical context of 
> cross-domain identity management, we are really at the very early 
> stages. If a relatively new discipline and a brand new spec are held 
> captive to legacy considerations, we are losing an opportunity to 
> provide a clean and elegant model to subsequent users of the spec, and 
> this will have repercussions over many years or even decades.
>
> b.
>
> dictionary model for existing multi-valued attributes, they can 
> transition to this model by offering both "SCIM-compliant" and 
> "non-SCIM-compliant" APIs to their customers and encouraging new 
> customers to adopt the "SCIM-compliant" API. Legacy customers can be 
> supported using a "non-SCIM-compliant" API for an arbitrarily long 
> period and gradually migrated to the SCIM-compliant API. The logistics 
> are not insurmountable, and shouldn't prevent the adoption of a 
> dictionary model for multi-valued attributes.
>
> Elaboration of Point 1:
>
> When we consider federated identity across more than one domain, we 
> have to assume that domains are not necessarily master-slave in their 
> entity lifecycle events within a domain are notified to other domains 
> (when necessary) in an asynchronous manner (i.e., through messaging) 
> and the other domains are free to respond to these events in an 
> appropriate manner and at a time of their convenience.
>
> A key set of lifecycle events for an entity is the merging and 
> splitting of identity that is often required.
>
> The question "Is this one entity?" can be answered either yes 
> (positive) or no (negative). But sometimes, we can discover false 
> positives and false negatives in our data stores.
>
> Consider a case where customers sign up online, and two customers who 
> are privacy-conscious enter fake IDs such as "John Smith", and also 
> use the same date of birth (say, 1 Jan 1970) or similar attributes. 
> The front-end application may make an intelligent (but incorrect) 
> guess that these two persons are the same, and re-assign the same 
> identifier to the second person. This is a false positive. They appear 
> to be the same entity, but they're actually different. When the error 
> is discovered, the identities will need to be split, with a new 
> identifier generated for one of them.
>
> different portals or in two different sessions, using the names 
> "JSmith" and "JohnS". It is very likely that they will be treated as 
> two different customers and assigned two unique identifiers. This is a 
> false negative. They appear to be two entities, but are actually the 
> same. At a later stage, when the error is discovered, the identities 
> will have to be merged, and one of the identifiers will have to be 
> dropped.
>
> These are not theoretical use cases. They form a significant 
> proportion of the user base in most large Web-facing applications. 
> Let's see how these can be managed in a federated way by mapping 
> internal identifiers to external ones and only exposing external 
> identifiers to other domains.
>
> a. False positives:
>
> following ID is returned by Domain 2: ff487230b3a0.
>
> Domain 1 then maintains the following in a mapping table and uses it 
> for translation when talking to Domain 2, taking care never to expose 
> its internal identifier:
>
> When the false positive is discovered and the entity is split, Domain 
> 1 creates a new internal identifier and now has the following entity 
> information.
>
> Domain 2, and this is by design. Communication about the original 
> entity takes place as before by mapping "9caf78aac3d6" to 
> "ff487230b3a0" and vice-versa. At some convenient time (importantly, 
> this doesn't have to be at the time the split happens), Domain 2 can 
> be requested to provision a second entity, and when it responds with 
> an identifier of "7a87f27c1dd8", this can go into the mapping table as 
> a new record associated with the second entity's internal identifier.
>
> provisioning that it does is not in lockstep with the split in 
> identity that occurred in Domain 1.
>
> Domain 1 then maintains the following in a mapping table and uses it 
> for translation when talking to Domain 2, taking care never to expose 
> its internal identifiers:
>
> Domain 1 drops one of the internal identifiers and rationalises the 
> name of the customer (say, to "John Smith"). Let's say it retains the 
> first ID "9caf78aac3d6" and drops the second "273d36e30d09".
>
> communication from Domain 2 can be unambiguously translated to the 
> same entity internally. However, when going outwards, Domain 1 will 
> have to look up the translation table to determine the "primary" 
> external ID for this entity in Domain 2, which was decided to be 
> "ff487230b3a0". That's where the "Primary flag" comes in. The second 
> external ID "41206cc97c8b" is never used thereafter in outbound 
> communication.
>
> "41206cc97c8b", and the second entry in the mapping table can be 
> removed once this is acknowledged.
>
> This scheme will scale up to multiple domains, because the "External 
> Domain ID" column helps to keep track of which external ID is shared 
> with which Domain. (Why don't we use just one external ID for an 
> entity and share it with all external domains? Tight coupling again. 
> Just as OAuth allows an access token given to a third party to be 
> invalidated without affecting the access of other third parties, the 
> use of separate external identifiers for different domains allows 
> fine-grained control of identity federation.)
>
> entities, and the merging of more than two entities into a single one. 
> (Any organisation with a web-facing application will tell you how many 
> John Smiths there are who were born on 1 Jan 1970!)
>
> This is a fairly long-winded explanation, but this is why we need to 
> hide internal identifiers from other domains, and why mappings need to 
> be managed internally in each domain. Such a data model also allows us 
> to choose asynchronous protocols for propagation of identity events, 
> since there is no consistency requirement to update multiple domains 
> concurrently.
>
> Regards,
>
> Ganesh Prasad
>
> On 9 August 2012 04:55, Kelly Grizzle <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
> article (http://www.infoq.com/articles/scim-data-model-limitations) 
> and have some thoughts.
>
> client's identifier, the "external ID"
>
> externalId and use the ID of the returned user in any desired 
> operation. For example:
>
> This does introduce an additional HTTP request if the client chooses 
> not to store the server's id.  An issue was created to consider 
> allowing operations to use the externalId 
> (http://code.google.com/p/scim/issues/detail?id=35), but I believe the 
> general consensus has been to not include this in the spec.  One main 
> point of contention is that much of the rest of the spec (eg -- group 
> membership references, manager references, etc...) require knowledge 
> of the server's identifier. Continuing this discussion on the IETF 
> list would be a good thing, though.
>
> "Internal IDs" with respect to their domains
>
> I think this comes down to a nomenclature problem.  The server's ID 
> does not necessarily have to be the unique identifier that the 
> underlying identity store uses, it just has to be stable and unique.  
> In many cases, the underlying identity store will provide identifiers 
> with these properties already (eg -- a uuid) and it can be used by the 
> SCIM interface.  The "externalId" is referring to the fact that the id 
> is maintained external to the SCIM server.  As long as the server's 
> identifiers are stable and unique (which is mandated by the spec), I 
> don't see a problem.
>
> > The secret is that /every value needs a key/, and multi-valued 
> attributes lack that. So our solution is quite
>
> One reason that this wasn't included in the spec originally is that it 
> can put undue burden on the service provider.  Many service providers 
> are putting SCIM interfaces in front of their existing identity stores 
> (eg -- directory servers, SaaS application databases, etc...).  Many 
> of these do not have a unique identifier for multi-valued attributes. 
> By requiring this, a majority of the server providers would have to 
> start maintaining a unique key for each multi-valued attribute.  I 
> believe this would be a roadblock for many implementers.
>
> bit clumsy.
> I like the thoughts here. Your example reminds me of unified diffs 
> (http://en.wikipedia.org/wiki/Diff#Unified_format), which are commonly 
> used with a patch program (pretty much the equivalent of the PATCH 
> verb).  However, the three proposals seem to largely hinge on being 
> able to uniquely address each element within an object.  Without these 
> it is not so easy to address each patch sub-operation (REPLACE, 
> INCLUDE, etc...) or provide a multi-status.
>
>
> The 207 response would be interesting to consider for the bulk 
> endpoint 
> (http://www.simplecloud.info/specs/draft-scim-api-00.html#bulk-resources), 
> however.
>
> such as its synchronous request-response
>
> discussion as work on SCIM 2.0 goes forward.
>
> --Kelly
>
> *From:[email-masked] <mailto:[email-masked]> 
> [mailto:[email-masked] <mailto:[email-masked]>] *On 
> Behalf Of *Ganesh and Sashi Prasad
> *To:* [email-masked] <mailto:[email-masked]>
> *Subject:* [scim] SCIM Protocol - 3 suggestions for improvement
>
> (I posted this on the SCIM Google Group, and I was advised to 
> subscribe to the mailing list and post it here instead, so here goes.)
>
> Hi,
>
> Management is mainly through a 3-year project at an Australian 
> insurance company, an experience I have written about as a eBook on 
> InfoQ (http://www.infoq.com/minibooks/Identity-Management-Shoestring).
>
> successful, I have the following 3 suggestions to make.
>
> 1. The enterprise client and the cloud provider should maintain their 
> own internal IDs for a resource, which they should not reveal to each 
> other. Both of them should map their internal IDs to a shared External 
> ID, and this is the only ID that should be exposed through the API. 
> The current specification's provision of an id (which is the external 
> ID and the only one to be transferred through the API) and an 
> "external ID" (which is the client's internal ID and should be hidden) 
> is diametrically opposite to this.
>
> 2. When dealing with multi-valued attributes of a resource (expressed 
> as arrays in JSON), they must be converted from an array into a 
> dictionary with unique keys (UUIDs generated by the cloud provider 
> when the attribute is created). Without unique keys for every 
> attribute value of a resource, manipulating it will be clumsy and 
> inelegant.
>
> greatly simplify the API
> 3b. Use special verbs as nested operations of the PATCH command to 
> add, modify and delete attributes at any level
>
> 3c. Use the WebDAV status code of "207 Multi-Status" instead of "200 
> OK" as the response to a PATCH (or BULK) command.
>
> To elaborate,
>
> 1. Revealing private IDs externally is a form of tight coupling. A 
> major requirement with Identity Management is to split (or merge) 
> identities when false positives (or false negatives) are detected, 
> i.e., when a resource is discovered to be more than one, or when 
> multiple resources are detected to be the same. If internal 
> identifiers are revealed to external domains, such clean-ups become 
> difficult, hence every domain that wants to expose references to a 
> resource must map its internal ID to and external one created for this 
> explicit purpose, and only reveal this.
>
> request, the cloud provider must generate its own internal UUID as 
> well as an external UUID, map them together, and only return the 
> external UUID in the "Location:" header. The enterprise client should 
> map this external UUID to a newly-generated internal ID of its own. In 
> case the resource already has an identifier within the enterprise 
> client's domain, then this is the internal ID that must be mapped to 
> the external UUID returned through the POST response.
>
>         "[email-masked] <mailto:[email-masked]>",
>
>         "[email-masked] <mailto:[email-masked]>",
>
>         "[email-masked] <mailto:[email-masked]>"
>
>     ]
>
>         { "7dfcb444-74d8-4f17-aa66-daf9ea3bd902" : 
> "[email-masked] <mailto:[email-masked]>" },
>
>         { "3bd10085-c474-43b9-9cda-8646c3085bbf" : 
> "[email-masked] <mailto:[email-masked]>" },
>
>         { "581da5c7-c6e1-4cca-9db7-7a6d1de664e1" : 
> "[email-masked] <mailto:[email-masked]>" }
>
>     ]
>
> value="[email-masked] <mailto:[email-masked]>"
>
> Modify:
>
> PUT 
> value="[email-masked] <mailto:[email-masked]>"
>
> Delete:
>
> DELETE 
> nested operations inside a PATCH. PATCH itself need not be nested 
> because its semantics apply throughout the "tree" of a resource.
>
> However, the semantics of PUT are a little messy. Also, the use of 
> HTTP verbs at a different level could be confusing. That's why I would 
> recommend 6 separate verbs that are a little more unambiguous in their 
> meaning:
>
> location specified by the accompanying URI. (If there's already a 
> value at that location, return an error status.)
>
> 3. REPLACE (equivalent to another form of PUT): Replace the value at 
> the location specified by the accompanying URI with this value. (If 
> there's no such URI, return an error status.)
>
> REPLACE. (At the end of this operation, we want the specified URI to 
> hold the accompanying value whether the URI already existed or not.)
>
> 5. RETIRE (equivalent to DELETE): Delete, deactivate or otherwise 
> render inaccessible the resource at the specified URI.
>
> completeness. We probably don't need a nested PATCH since PATCH 
> cascades to every level of the tree.)
>
> Host: example.com <http://example.com>
>
> Accept: application/json
>
> "value" : "[email-masked] <mailto:[email-masked]>"
>
>     },
>
> REPLACE : {
>
> "value" : "[email-masked] <mailto:[email-masked]>"
>
>     },
>
>     RETIRE : {
>
> because the nested operations could have varying status codes. A 
> sample response is below:
>
> "value" : "[email-masked] <mailto:[email-masked]>"
>
>     },
>
> REPLACE : {
>
> "status" : "200 OK",
>
> "value" : "[email-masked] <mailto:[email-masked]>"
>
>     },
>
>     RETIRE : {
>
> "status" : "200 OK",
>
> Created" status codes in the above successful case. But the outer 
> status will remain "207 Multi-Status".
>
> group, and for bulk operations.
>
> I read the SCIM spec afresh last week and these ideas came flooding 
> into my head because I have been working at another organisation (a 
> telco) for the last 5 months, also in Identity and Access Management, 
> and my thoughts have moved further along the direction of evolving a 
> specialised data model based on specific principles, especially for IAM.
>
> soon and am in negotiations with InfoQ regarding publication.
>
> Regards,
>
> Ganesh Prasad
>
>
> ------------------------------------------------------------------------
>
> This e-mail message, including any attachments, is for the sole use of 
> the person to whom it has been sent, and may contain information that 
> is confidential or legally protected. If you are not the intended 
> recipient or have received this message in error, you are not 
> authorized to copy, distribute, or otherwise use this message or its 
> attachments. Please notify the sender immediately by return e-mail and 
> permanently delete this message and any attachments. Gartner makes no 
> warranty that this e-mail is error or virus free.
>
>
> _______________________________________________
> scim mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/scim

---

From: Phil Hunt <[email-masked]>
Date: Thu, 9 Aug 2012 14:53:12 -0700

Phil

> What the SCIM spec needs to do there is just refrain from introducing tight coupling. I would like to see a single identifier exposed through the API, with the implication (and perhaps the recommen…

I see what you are saying. I think scim gets its current simplicity from its single owner hub spoke model implementing tight coupling. 

IMHO loose coupling is a much more complex solution. The reality is that each end-point has value to contribute and thus the single-owner model will eventually need to become multi-owner or multi-hub. 

That said i think the current model provides a practical starting point. 
> Regarding unique identifiers for multi-valued attributes there is a trade-off involved.  On one hand this makes PATCH semantics easier.  On the other hand it puts extra burden on service providers.  Since the inception of SCIM, a key goal has been to foster adoption by service providers by making things fit easily onto existing systems.  IMO the value gained by unique identifiers for multi-valued attributes is not worth the demands put on a service provider.  I also think that vendors that have a non-SCIM-compliant API will choose to keep things that way if the spec is too hard for them to implement.  In a green field environment we do have the luxury of mandating a model to make certain operations more elegant.  However, we can’t ignore legacy systems.
> 
>  
> 
> --Kelly
> 
>  
> 
> As we didn’t find an immutable uniqueID in AD systems (DN,samAccountName, UPN) are subject to change (even objectGuid can change if an AD domain is migrated), we decided to generate and maintain an internal table of ids. This fits your requirements as it hides internal ids.
> 
>  
> 
> In any non-trivial data store, there will always be the ongoing need to merge and split identities as and when “false negatives” and “false positives” are discovered. A domain should be able to handle this internal housekeeping freely, only notifying other domains when convenient. Mapping of internal identifiers to external ones and maintaining this mapping internally allows this loosely-coupled housekeeping to take place. Sharing internal identifiers (or otherwise outsourcing the mapping of internal to external identifiers) forces housekeeping activities to be done in lock-step across domains.
> c.
> If incumbent cloud providers find it hard to immediately adopt the dictionary model for existing multi-valued attributes, they can transition to this model by offering both “SCIM-compliant” and “non-SCIM-compliant” APIs to their customers and encouraging new customers to adopt the “SCIM-compliant” API. Legacy customers can be supported using a “non-SCIM-compliant” API for an arbitrarily long period and gradually migrated to the SCIM-compliant API. The logistics are not insurmountable, and shouldn't prevent the adoption of a dictionary model for multi-valued attributes.
> Elaboration of Point 1:
> The question “Is this one entity?” can be answered either yes (positive) or no (negative). But sometimes, we can discover false positives and false negatives in our data stores.
> Consider a case where customers sign up online, and two customers who are privacy-conscious enter fake IDs such as “John Smith”, and also use the same date of birth (say, 1 Jan 1970) or similar attributes. The front-end application may make an intelligent (but incorrect) guess that these two persons are the same, and re-assign the same identifier to the second person. This is a false positive. They appear to be the same entity, but they're actually different. When the error is discovered, the identities will need to be split, with a new identifier generated for one of them.
> Consider the opposite case where a customer signs up through two different portals or in two different sessions, using the names “JSmith” and “JohnS”. It is very likely that they will be treated as two different customers and assigned two unique identifiers. This is a false negative. They appear to be two entities, but are actually the same. At a later stage, when the error is discovered, the identities will have to be merged, and one of the identifiers will have to be dropped.
> This second entity with its own internal identifier is invisible to Domain 2, and this is by design. Communication about the original entity takes place as before by mapping “9caf78aac3d6” to “ff487230b3a0” and vice-versa. At some convenient time (importantly, this doesn't have to be at the time the split happens), Domain 2 can be requested to provision a second entity, and when it responds with an identifier of “7a87f27c1dd8”, this can go into the mapping table as a new record associated with the second entity's internal identifier.
> (What is the “Primary flag” used for? We'll see when we cover the treatment of false negatives.)
> b. False negatives:
> When the false negative is discovered and the two entities are merged, Domain 1 drops one of the internal identifiers and rationalises the name of the customer (say, to “John Smith”). Let's say it retains the first ID “9caf78aac3d6” and drops the second “273d36e30d09”.
> Now two external identifiers map to the same internal one, so inbound communication from Domain 2 can be unambiguously translated to the same entity internally. However, when going outwards, Domain 1 will have to look up the translation table to determine the “primary” external ID for this entity in Domain 2, which was decided to be “ff487230b3a0”. That's where the “Primary flag” comes in. The second external ID “41206cc97c8b” is never used thereafter in outbound communication.
> At some stage (importantly, not in lockstep with the identity merge), Domain 2 can be requested to delete the customer record identified by “41206cc97c8b”, and the second entry in the mapping table can be removed once this is acknowledged.
> This scheme will scale up to multiple domains, because the “External Domain ID” column helps to keep track of which external ID is shared with which Domain. (Why don't we use just one external ID for an entity and share it with all external domains? Tight coupling again. Just as OAuth allows an access token given to a third party to be invalidated without affecting the access of other third parties, the use of separate external identifiers for different domains allows fine-grained control of identity federation.)
> > The rest of the protocol does not meaningfully use the enterprise client’s identifier, the "external ID"
> 
> GET /Users?filter=externalId eq “bjensen”&attributes=id
> 
>  
> 
> {
> 
>   “totalResults”: 1,
> 
>   “Resources”: [
> 
>     {
> 
>       “id”: “2819c223-7f76-453a-919d-413861904646”
> 
>     }
> 
>   ]
> 
> }
> 
>  
> 
> This does introduce an additional HTTP request if the client chooses not to store the server’s id.  An issue was created to consider allowing operations to use the externalId (http://code.google.com/p/scim/issues/detail?id=35), but I believe the general consensus has been to not include this in the spec.  One main point of contention is that much of the rest of the spec (eg – group membership references, manager references, etc…) require knowledge of the server’s identifier.  Continuing this discussion on the IETF list would be a good thing, though.
> 
>  
> 
>  
> 
> I think this comes down to a nomenclature problem.  The server’s ID does not necessarily have to be the unique identifier that the underlying identity store uses, it just has to be stable and unique.  In many cases, the underlying identity store will provide identifiers with these properties already (eg – a uuid) and it can be used by the SCIM interface.  The “externalId” is referring to the fact that the id is maintained external to the SCIM server.  As long as the server’s identifiers are stable and unique (which is mandated by the spec), I don’t see a problem.
> 
>  
> 
>  
> 
> I agree that this would be useful, especially in the PATCH operation.  One reason that this wasn’t included in the spec originally is that it can put undue burden on the service provider.  Many service providers are putting SCIM interfaces in front of their existing identity stores (eg – directory servers, SaaS application databases, etc…).  Many of these do not have a unique identifier for multi-valued attributes.  By requiring this, a majority of the server providers would have to start maintaining a unique key for each multi-valued attribute.  I believe this would be a roadblock for many implementers.
> 
>  
> 
>  
> 
> I like the thoughts here.  Your example reminds me of unified diffs (http://en.wikipedia.org/wiki/Diff#Unified_format), which are commonly used with a patch program (pretty much the equivalent of the PATCH verb).  However, the three proposals seem to largely hinge on being able to uniquely address each element within an object.  Without these it is not so easy to address each patch sub-operation (REPLACE, INCLUDE, etc…) or provide a multi-status.
> 
> 
> <

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Fri, 10 Aug 2012 20:49:27 +1000

>  I think scim gets its current simplicity from its single owner hub spoke
model implementing tight coupling. [...] IMHO loose coupling is a much more
complex solution.

Phil,

I'm a bit surprised that you're implying "tight coupling == simple" and
"loose coupling == complex". That's contrary to my experience.

When I say "loose coupling", I mean "no unnecessary dependencies".
Invariably, a reduction in dependencies leads to greater simplicity.

Let's not confuse reduction of dependencies in the data model with a
hub-and-spokes architecture. They're entirely orthogonal aspects of the
solution.

All that my suggestion involves is,

1. Take 'external ID' out of the protocol.
2. Allow generic search using 'GET /Users' and arbitrary URI parameters

No planned functionality is lost by this.

1. The client enterprise can still send its internal ID as part of the
resource body, inside some attribute defined by them (but not defined by
the protocol). Let's say they call it 'myID'.
2. The client enterprise can search for resource URIs using any attribute,
including this internal ID
'GET /Users?myID=bjensen'
Since myID is a candidate key, the server will return exactly one URI,
which is the canonical URI for the resource

https://example.com/v1/Users/2819c223-7f76-453a-919d-413861904646

3. The client can use this URI to perform all other operations as usual.

So taking 'externalID' out of the protocol spec only does this:

1. It avoids enshrining tight coupling in the protocol (If clients
want to tightly couple themselves to the cloud provider by sending
their internal IDs, they can do so. Suicide is OK, but the protocol
should not be guilty of assisted suicide. ;-)

2. It encourages loose coupling by nudging clients towards maintaining
their own internal-to-external identifier mappings.

That's what I'd like to see. I don't believe this complicates the
protocol. It simplifies it and it also lends itself to a
loosely-coupled approach.

I'll address the multi-valued attribute suggestion separately.

Regards,

Ganesh

On 10 August 2012 07:53, Phil Hunt <[email-masked]> wrote:

> I see what you are saying. I think scim gets its current simplicity from
> its single owner hub spoke model implementing tight coupling.
>
> IMHO loose coupling is a much more complex solution. The reality is that
> each end-point has value to contribute and thus the single-owner model will
> eventually need to become multi-owner or multi-hub.
>

---

From: Trey Drake <[email-masked]>
Date: Fri, 10 Aug 2012 09:50:59 -0500

Ganesh,

I'll base my comments on your latest reply (below).

The externalID is not part of the protocol.  It is an *optional* attribute
within the *schema* specification.  As for #2, the protocol spec works as
you describe if "arbitrary URI parameters" equate to resource attributes
(Allow generic search using 'GET /Users' and arbitrary URI parameters).
 Please clarify your suggestion.

I'm not tracking your coupling concern.  The client can search and hence
retrieve resources on any attribute it chooses, externalId or otherwise.
 Nothing mandates use of externalId.

What do you mean by "candidate key"?  Given

On Fri, Aug 10, 2012 at 5:49 AM, Ganesh and Sashi Prasad <
[email-masked]> wrote:

> >  I think scim gets its current simplicity from its single owner hub
> spoke model implementing tight coupling. [...] IMHO loose coupling is a
> much more complex solution.
>
> Phil,
>
> 1. It avoids enshrining tight coupling in the protocol (If clients want to tightly couple themselves to the cloud provider by sending their internal IDs, they can do so. Suicide is OK, but the protocol should not be guilty of assisted suicide. ;-)
>
> 2. It encourages loose coupling by nudging clients towards maintaining their own internal-to-external identifier mappings.
>
>
> That's what I'd like to see. I don't believe this complicates the protocol. It simplifies it and it also lends itself to a loosely-coupled approach.
>
>
>>> As we didn’t find an immutable uniqueID in AD systems
>>> (DN,samAccountName, UPN) are subject to change (even objectGuid can change
>>> if an AD domain is migrated), we decided to generate and maintain an
>>> internal table of ids. This fits your requirements as it hides internal ids.
>>> ****
>>>
>>> ** **
>>>
>>> protocol which can be designed later. The data model plays a crucial role
>>> in enabling or constraining such interaction. A tightly-coupled data model
>>> to such a model. However, in the larger historical context of cross-domain
>>> are privacy-conscious enter fake IDs such as “John Smith”, and also use the
>>> will need to be split, with a new identifier generated for one of them.*
>>> ***
>>>
>>> Domain 1 has the following information about a customer in its data
>>> store:****
>>>
>>> hide internal identifiers from other domains, and why mappings need to be
>>> > interaction model, which is a form of tight coupling and could prove
>>> to be a source of brittleness.****
>>>
>>>  ****
>>>
>>> other. Both of them should map their internal IDs to a shared External ID,
>>> and this is the only ID that should be exposed through the API. The current

---

From: "Diodati,Mark" <[email-masked]>
Date: Fri, 10 Aug 2012 15:35:18 +0000

Hi Ganesh,

I am concerned about your second suggestion:
"2. When dealing with multi-valued attributes of a resource (expressed as arrays in JSON), they must be converted from an array into a dictionary with unique keys (UUIDs generated by the cloud provider when the attribute is created). Without unique keys for every attribute value of a resource, manipulating it will be clumsy and inelegant."

One of the primary reasons that SPML failed was lack of adoption by service providers due to its complexity. Very few target applications implemented SPML. Most of the commercial provisioning systems had an SPML interface (either v1 or v2), but not one of them was conformant to the SPML standard because of complexity. If you are interested, I will forward you the research documents that discuss these problems in detail. For SCIM to be successful, it must be adopted by commercial target applications (i.e., service providers). I am confident that a requirement for unique identifiers with multi-valued attributes will preclude its adoption, because it requires major changes to the service provider's existing identity storage mechanisms.
Mark

From: Trey Drake [mailto:[email-masked]]
Sent: Friday, August 10, 2012 9:51 AM
To: Ganesh and Sashi Prasad
Cc: [email-masked]; Emmanuel Dreux; Kelly Grizzle; Phil Hunt
Subject: Re: [scim] SCIM Protocol - 3 suggestions for improvement

Ganesh,

The externalID is not part of the protocol.  It is an *optional* attribute within the *schema* specification.  As for #2, the protocol spec works as you describe if "arbitrary URI parameters" equate to resource attributes (Allow generic search using 'GET /Users' and arbitrary URI parameters).  Please clarify your suggestion.

I'm not tracking your coupling concern.  The client can search and hence retrieve resources on any attribute it chooses, externalId or otherwise.  Nothing mandates use of externalId.

On Fri, Aug 10, 2012 at 5:49 AM, Ganesh and Sashi Prasad <[email-masked]<mailto:[email-masked]>> wrote:
>  I think scim gets its current simplicity from its single owner hub spoke model implementing tight coupling. [...] IMHO loose coupling is a much more complex solution.

Phil,

I'm a bit surprised that you're implying "tight coupling == simple" and "loose coupling == complex". That's contrary to my experience.

When I say "loose coupling", I mean "no unnecessary dependencies". Invariably, a reduction in dependencies leads to greater simplicity.

Let's not confuse reduction of dependencies in the data model with a hub-and-spokes architecture. They're entirely orthogonal aspects of the solution.

1. The client enterprise can still send its internal ID as part of the resource body, inside some attribute defined by them (but not defined by the protocol). Let's say they call it 'myID'.
2. The client enterprise can search for resource URIs using any attribute, including this internal ID
'GET /Users?myID=bjensen'
Since myID is a candidate key, the server will return exactly one URI, which is the canonical URI for the resource

Regards,

Ganesh

On 10 August 2012 07:53, Phil Hunt <[email-masked]<mailto:[email-masked]>> wrote:

Phil

Am I making sense?

Regards,
Ganesh

{

{

{

--Kelly

Hi Ganesh,

Hi Kelly,

a.

b.

c.

a.

b.

Elaboration of Point 1:

a. False positives:

b. False negatives:

Regards,

Ganesh Prasad

{
  "totalResults": 1,
  "Resources": [
    {

--Kelly

Hi,

To elaborate,

    "email-addrs" :
    [
[email-masked]<mailto:[email-masked]>

_______________________________________________
scim mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/scim

---

From: Phil Hunt <[email-masked]>
Date: Fri, 10 Aug 2012 09:48:26 -0700

Ganesh,

See below...

Phil

@independentid
www.independentid.com
[email-masked]

> I'm a bit surprised that you're implying "tight coupling == simple" and "loose coupling == complex". That's contrary to my experience.

I think we actually agree. What I was saying is that for a sufficiently narrow scope model, tight coupling is "simple". But in the long term it *may* turn out to be "simplistic".  SCIM current seems to assume that an enterprise can claim ownership of the entirety of the object being provisioned in the cloud.  The enterprise has full knowledge of the resource in the cloud.  Under that highly restricted definition, it is simple.

The model is fine on a point-to-point basis, but the higher-level provisioning "system" is more complex.

I agree with Trey, externalID is optional.  I also do not equate "externalID" with tight or loose coupling. Yes, you can cause tight coupling, but that doesn't necessarily follow. 

I look at it this way: ObjectID is simply the ID asserted by the holder of the resource.  ExternalID is an optional ID that is meaningful to a particular client.   SAML has a similar concept.

---

From: Trey Drake <[email-masked]>
Date: Fri, 10 Aug 2012 13:31:51 -0500

Maybe.  I suppose that depends on the meaning of "owns".   

- A SP may augment the resource and not expose the augmented info with the consumer.  
- Schema is defined by the SP not the consumer
- A consumer may delete a resource though the SP may simply "hide" it from future requests

Thanks,
Trey

---

From: Phil Hunt <[email-masked]>
Date: Fri, 10 Aug 2012 12:11:46 -0700

By "own", I was implying a possible scenario where a client assumes it knows everything it needs to know about the state of the resource within a SP (because for example only that client can change the resource).  Thus an owning client could execute a PATCH operation on an attribute without a lot of errors (missing values, non-existent attributes and other errors) because it knows of the full state of the resource in the SP.

I think your example nicely points out the complexity that is likely to occur for service providers. At some point, full knowledge of "state" for the provisioning clients isn't realistic. 

Phil

@independentid
www.independentid.com
[email-masked]

---

From: Trey Drake <[email-masked]>
Date: Fri, 10 Aug 2012 14:15:40 -0500

Thanks for the clarification Phil - makes sense to me.

---

From: prateek mishra <[email-masked]>
Date: Fri, 10 Aug 2012 15:46:05 -0400

I could only find Phil Hunt's draft document at -

http://datatracker.ietf.org/wg/scim/

---

From: Trey Drake <[email-masked]>
Date: Fri, 10 Aug 2012 14:51:35 -0500

Core schema is here: https://datatracker.ietf.org/doc/draft-scim-core-schema/

I suppose the API is still making its way through the manual submission process.  It was submitted.

---

From: Igor Faynberg <[email-masked]>
Date: Fri, 10 Aug 2012 18:17:41 -0400

+1

Igor

On 8/10/2012 11:35 AM, Diodati,Mark wrote:
> "2. When dealing with multi-valued attributes of a resource (expressed 
> inelegant."
> One of the primary reasons that SPML failed was lack of adoption by 
> service providers due to its complexity. Very few target applications 
> implemented SPML. Most of the commercial provisioning systems had an 
> SPML interface (either v1 or v2), but not one of them was conformant 
> to the SPML standard because of complexity. If you are interested, I 
> will forward you the research documents that discuss these problems in 
> detail. For SCIM to be successful, it must be adopted by commercial 
> target applications (i.e., service providers). I am confident that a 
> requirement for unique identifiers with multi-valued attributes will 
> preclude its adoption, because it requires major changes to the 
> service provider's existing identity storage mechanisms.
>
> Mark
>
> *From:*Trey Drake [mailto:[email-masked]]
> *Sent:* Friday, August 10, 2012 9:51 AM
> *To:* Ganesh and Sashi Prasad
> *Cc:* [email-masked]; Emmanuel Dreux; Kelly Grizzle; Phil Hunt
> The externalID is not part of the protocol.  It is an *optional* 
> attribute within the *schema* specification.  As for #2, the protocol 
> spec works as you describe if "arbitrary URI parameters" equate to 
> resource attributes (Allow generic search using 'GET /Users' and 
> arbitrary URI parameters).  Please clarify your suggestion.
>
> I'm not tracking your coupling concern.  The client can search and 
> hence retrieve resources on any attribute it chooses, externalId or 
> otherwise.  Nothing mandates use of externalId.
>
> On Fri, Aug 10, 2012 at 5:49 AM, Ganesh and Sashi Prasad 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
> spoke model implementing tight coupling. [...] IMHO loose coupling is 
> a much more complex solution.
>
> Phil,
>
> I'm a bit surprised that you're implying "tight coupling == simple" 
> and "loose coupling == complex". That's contrary to my experience.
>
> hub-and-spokes architecture. They're entirely orthogonal aspects of 
> the solution.
>
> resource body, inside some attribute defined by them (but not defined 
> by the protocol). Let's say they call it 'myID'.
>
>     > storing this information in a mapping table outside of the SCIM
>     spec is a great way to enable this solution.  Part of the key here
>     is that SCIM is just a piece of the architecture for this
>     I wasn't suggesting that the mapping table be part of the SCIM
>     spec. I provided that example to illustrate that splitting and
>     merging identities is a common requirement, and that decoupling
>     local identifiers within a domain from shared identifiers between
>     domains was the best way to facilitate it.
>
>     What the SCIM spec needs to do there is just refrain from
>     introducing tight coupling. I would like to see a single
>     identifier exposed through the API, with the implication (and
>     perhaps the recommendation) that it be the shared one. Allowing
>     one domain to expose its internal identifier to the other creates
>     tight coupling and ensures that both domains need simultaneously
>     split or merge identities, which is not desirable. So I recommend
>     _taking out_ the "external id" field from the API. The spec
>     shouldn't encourage tight coupling. If clients want to pass in
>     their internal ids as part of the resource body, no one can stop
>     them, and they can always do a search on that attribute to
>     retrieve the URI exactly as you visualise they will with the
>     "external id", but let's not elevate an anti-pattern to a
> I see what you are saying. I think scim gets its current simplicity 
> from its single owner hub spoke model implementing tight coupling.
>
> IMHO loose coupling is a much more complex solution. The reality is 
> that each end-point has value to contribute and thus the single-owner 
> model will eventually need to become multi-owner or multi-hub.
>
> friendlier for the client.
>
> key-value pairs) by providing each value
>
> to be a source of brittleness.
>
> [email-masked] <mailto:[email-masked]>
> [email-masked] <mailto:[email-masked]>

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Sat, 11 Aug 2012 08:38:36 +1000

Trey,

> The externalID is not part of the protocol.  It is an *optional*
attribute within the *schema* specification.

I didn't realise SCIM was also specifying a User schema. Does this mean the
User resource can only hold certain attributes as defined by this schema?
If so, that's going to be a severe constraint, because client organisations
are likely to require many specialised User attributes depending on the
nature of their business, and they will expect to be able to store all of
them, not just the subset defined by the SCIM spec. I hope the SCIM User
schema is not trying to be just a representation of inetOrgPerson (the
standard LDAP schema), because that could be severely limiting. Most
organisations with their own LDAP end up extending inetOrgPerson, and they
will expect that they can do the same in the cloud.

> As for #2, the protocol spec works as you describe if "arbitrary URI
parameters" equate to resource attributes

Yes, that's what I meant, but in implementation, it may work out to be
looser than that. The client should be able to specify any arbitrary
attribute in the search parameters, even those that aren't attributes of
the resource. If an attribute is not defined, or if the attribute exists
but the value doesn't match any stored record, the SP will return no
results. In the former case, it's a "400 Bad Request" and in the latter,
it's a "404 Not Found".

By "candidate key" (a data modelling term), I meant another attribute that
also uniquely identifies a single record, but is not the official "primary
key", i.e., the main ID. In this case, a client's internal identifier also
uniquely identifies a record, but is not the ID used in the URI of RESTful
operations.

Look, this may seem to be splitting hairs since a determined client may be
able to store and search by their own internal ID in either case (the
current SCIM spec or my suggestion). The difference is philosophical. If
the spec is showing how clients can store their internal IDs in the cloud
by explicitly providing for such an attribute, that constitutes bad advice,
IMO. If it turns a blind eye to what they store and they store internal IDs
anyway, they're constraining themselves but the spec comes out smelling of
roses because that's not "recommended practice".

Ganesh

On 11 August 2012 00:50, Trey Drake <[email-masked]> wrote:

>>>>  Thanks Emmanuel.  I had started writing up a similar response.  As
>>>> you suggest, storing this information in a mapping table outside of the
>>>> SCIM spec is a great way to enable this solution.  Part of the key here is
>>>> that SCIM is just a piece of the architecture for this solution, and is
>>>> only responsible for the transport layer between domains.****
>>>>
>>>> ** **
>>>>
>>>> | Internal Entity ID | External Domain ID | External Entity ID |
>>>> Primary flag |****
>>>>
>>>> | Internal Entity ID | External Domain ID | External Entity ID |
>>>> Primary flag |****
>>>>
>>>> *From:* [email-masked] [mailto:[email-masked]] *On
>>>> or without asking to include it in the specs.****
>>>>
>>>> ** **
>>>>
>>>> --****
>>>>
>>>> Regards,****
>>>>
>>>> Emmanuel Dreux****
>>>>
>>>> **1.      **Why should domains not expose their internal identifiers
>>>> to other domains?****
>>>>
>>>> a.****
>>>>
>>>> all domains are co-equal peers. (In physics too, N-body problems are much
>>>> splitting of identity that is often required.****
>>>>
>>>> The question “Is this one entity?” can be answered either yes
>>>> (positive) or no (negative). But sometimes, we can discover false positives
>>>> and false negatives in our data stores.****
>>>>
>>>> for translation when talking to Domain 2, taking care never to expose its
>>>> internal identifier:****
>>>>
>>>> | Internal Entity ID | External Domain ID | External Entity ID |
>>>> Primary flag |****
>>>>
>>>> | Internal Entity ID | External Domain ID | External Entity ID |
>>>> Primary flag |****
>>>>
>>>> for translation when talking to Domain 2, taking care never to expose its
>>>> | Internal Entity ID | External Domain ID | External Entity ID |
>>>> Primary flag |****
>>>>
>>>> | Internal Entity ID | External Domain ID | External Entity ID |
>>>> Primary flag |****
>>>>
>>>> and have some thoughts.****
>>>>
>>>>  ****
>>>>
>>>> not to store the server’s id.  An issue was created to consider allowing
>>>> I think this comes down to a nomenclature problem.  The server’s ID
>>>> does not necessarily have to be the unique identifier that the underlying
>>>> *From:* [email-masked] [mailto:[email-masked]] *On
>>>> as arrays in JSON), they must be converted from an array into a dictionary
>>>> OK" as the response to a PATCH (or BULK) command.****
>>>>
>>>>  ****
>>>>
>>>> To elaborate,****
>>>>
>>>>  ****
>>>>
>>>> identities when false positives (or false negatives) are detected, i.e.,
>>>> when a resource is discovered to be more than one, or when multiple
>>>> resources are detected to be the same. If internal identifiers are revealed
>>>> to external domains, such clean-ups become difficult, hence every domain
>>>> that wants to expose references to a resource must map its internal ID to
>>>> and external one created for this explicit purpose, and only reveal this.
>>>> ****
>>>>
>>>>  ****
>>>>

---

From: "Diodati,Mark" <[email-masked]>
Date: Fri, 10 Aug 2012 22:47:51 +0000

Hi Ganesh,

There is a user schema, but it is optional. Still, many of our clients are looking for a common user schema. This was one of the criticisms of the prior specification and an initial requirement for SCIM.

Best,

Mark

http://www.simplecloud.info/specs/draft-scim-core-schema-00.html
http://blogs.gartner.com/mark-diodati/2011/05/06/scim-and-the-future-of-standards-based-provisioning/
http://blogs.gartner.com/mark-diodati/2010/08/20/consensus-on-the-future-of-standards-based-provisioning-and-spml/

From: Ganesh and Sashi Prasad [mailto:[email-masked]]
Sent: Friday, August 10, 2012 5:39 PM
To: Trey Drake
Cc: [email-masked]; Emmanuel Dreux; Kelly Grizzle; Phil Hunt
Subject: Re: [scim] SCIM Protocol - 3 suggestions for improvement

Trey,

> The externalID is not part of the protocol.  It is an *optional* attribute within the *schema* specification.

I didn't realise SCIM was also specifying a User schema. Does this mean the User resource can only hold certain attributes as defined by this schema? If so, that's going to be a severe constraint, because client organisations are likely to require many specialised User attributes depending on the nature of their business, and they will expect to be able to store all of them, not just the subset defined by the SCIM spec. I hope the SCIM User schema is not trying to be just a representation of inetOrgPerson (the standard LDAP schema), because that could be severely limiting. Most organisations with their own LDAP end up extending inetOrgPerson, and they will expect that they can do the same in the cloud.

> As for #2, the protocol spec works as you describe if "arbitrary URI parameters" equate to resource attributes

Yes, that's what I meant, but in implementation, it may work out to be looser than that. The client should be able to specify any arbitrary attribute in the search parameters, even those that aren't attributes of the resource. If an attribute is not defined, or if the attribute exists but the value doesn't match any stored record, the SP will return no results. In the former case, it's a "400 Bad Request" and in the latter, it's a "404 Not Found".

By "candidate key" (a data modelling term), I meant another attribute that also uniquely identifies a single record, but is not the official "primary key", i.e., the main ID. In this case, a client's internal identifier also uniquely identifies a record, but is not the ID used in the URI of RESTful operations.

Look, this may seem to be splitting hairs since a determined client may be able to store and search by their own internal ID in either case (the current SCIM spec or my suggestion). The difference is philosophical. If the spec is showing how clients can store their internal IDs in the cloud by explicitly providing for such an attribute, that constitutes bad advice, IMO. If it turns a blind eye to what they store and they store internal IDs anyway, they're constraining themselves but the spec comes out smelling of roses because that's not "recommended practice".

Ganesh
On 11 August 2012 00:50, Trey Drake <[email-masked]<mailto:[email-masked]>> wrote:
Ganesh,

Phil,

Regards,

Ganesh

Phil

Am I making sense?

Regards,
Ganesh

{

{

{

--Kelly

Hi Ganesh,

Hi Kelly,

a.

b.

c.

a.

b.

Elaboration of Point 1:

a. False positives:

b. False negatives:

Regards,

Ganesh Prasad

{
  "totalResults": 1,
  "Resources": [
    {

--Kelly

Hi,

To elaborate,

    "email-addrs" :
    [

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Sat, 11 Aug 2012 09:19:05 +1000

>  I am concerned about your second suggestion:

Let's discuss that now.

The trade-offs are very clear here.

Pros:

Pro 1. The API to manipulate resources becomes so much cleaner, consistent
and intuitive when every individual attribute value gets its own ID.

Here's how to delete a single member from a Group, as per the current spec:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "schemas": ["urn:scim:schemas:core:1.0"],
     "members": [
       {
         "display": "Babs Jensen",
         "value": "2819c223-7f76-453a-919d-413861904646"
         "operation": "delete"
       }
     ]
   }

Here's how to delete ALL members from a group according to the current spec:

   PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce
   Host: example.com
   Accept: application/json
   Authorization: Bearer h480djs93hd8
   ETag: W/"a330bc54f0671c9"

   {
     "schemas": ["urn:scim:schemas:core:1.0"],
     "meta": {
       "attributes": [
         "members"
       ]
     }
   }

The two operations differ significantly, and it's not very intuitive.
With my suggestion, here's how to delete a single member from a group:

PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce Host:
example.comAccept: application/json Authorization: Bearer h480djs93hd8
ETag:
W/"a330bc54f0671c9" {
"operations" : [
{
 "RETIRE" : {
"key" : "members.2819c223-7f76-453a-919d-413861904646"
 }
}
] }
Here's how I suggest deleting ALL members from a group:

PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce Host:
example.comAccept: application/json Authorization: Bearer h480djs93hd8
ETag:
W/"a330bc54f0671c9" {
"operations" : [
{
 "RETIRE" : {
"key" : "members"
 }
}
] }

I'm sure you'll agree that this is simpler, more consistent and more
intuitive to a reader.

Pro 2: We can apply this mechanism consistently to three areas:
(a) Manipulating multi-valued attributes of a resource
(b) Manipulating members of a group
(c) Performing bulk operations, where we simply use HTTP verbs instead of
the specialised (and semantically less ambiguous) verbs I suggested for
attributes, the "key" becomes the URI, and the "value" becomes the
corresponding JSON object.

All of them return "207 Multi-Status" with the "results" array holding
individual status codes.

In the current spec, (a) and (b) are done similarly but (c) is very
different.

Pro 3: Adoption of the standard by clients is likely to be higher because
it's simpler for them.

Pro 4: New (not incumbent) cloud providers will probably find this easier
to implement because they have no legacy. They will probably use some form
of NoSQL database and won't be constrained by the limitations of LDAP
directories.

Cons:

Con 1: Incumbent cloud providers with existing data stores in a directory
format (where multi-valued attributes are stored as comma-separated values
under a single attribute node) will find it difficult to migrate to this
model and store each attribute value as a sub-node with its own key. This
will "hinder adoption of the spec", which is what you fear.

Have I summed up the Pros and Cons correctly? I'm biased of course, so I
could have missed a Con or hyped a Pro :-).

In other words, we're debating interface complexity (current spec) versus
implementation complexity (my suggestion). Both can hinder adoption of the
spec by different parties.

Here's what we need to discuss - Do the Pros make the suggestion worth
adopting in spite of the Cons, or are the Cons so great that it's best to
leave the spec as it is?

Keep in mind that a complex spec that only favours incumbent cloud
providers can cut both ways. It opens the door to a simpler interface
offered by a new generation of nimbler SPs that don't have the same legacy
issues, and there could be an exodus of clients to these new SPs. SCIM
could end up being obsoleted very soon, because the API interface is very
complex and clumsy, as any new reader can attest. I was taken aback by the
complexity when I saw it, which is why I was prompted to suggest something
simpler.

This is an issue where we need the opinions of many people, and they need
to state their affiliations. If most people weighing in belong to incumbent
SPs and they vote in favour of interface complexity to avoid implementation
complexity, then it means the spec is not doing a good job of balancing the
interests of various groups. I think we should also poll client
organisations to see what they would want.

(Gartner is trusted by enterprise clients to evaluate the capabilities of
vendors (SPs). I believe Gartner should take the lead in representing
client interests in this working group rather than those of incumbent
vendors, which is what it seems like to me. Correct me if I'm being unfair.)

Regards,
Ganesh

On 11 August 2012 01:35, Diodati,Mark <[email-masked]> wrote:

> “2. When dealing with multi-valued attributes of a resource (expressed as
> manipulating it will be clumsy and inelegant.”
>
>
>
> implemented SPML. Most of the commercial provisioning systems had an SPML
> interface (either v1 or v2), but not one of them was conformant to the SPML
> standard because of complexity. If you are interested, I will forward you
> the research documents that discuss these problems in detail. For SCIM to
> be successful, it must be adopted by commercial target applications (i.e.,
> service providers). I am confident that a requirement for unique
> identifiers with multi-valued attributes will preclude its adoption,
> because it requires major changes to the service provider’s existing
> identity storage mechanisms.
>
> Mark
>
>
>
>
>
>
>
> *From:* Trey Drake [mailto:[email-masked]]
> user):
> the entities on other domains.
>
>
>
> {
>
> However, we can’t ignore legacy systems.
>
>
>
> --Kelly
>
>
>
> This fits your requirements as it hides internal ids.
>
>
>
> without asking to include it in the specs.
>
>
>
> --
>
> Regards,
>
> Emmanuel Dreux
>
> http://www.cloudiway.com
>
> Tel: +33 4 26 78 17 58
>
> and at a time of their convenience.
>
> of identity that is often required.
>
> treatment of false negatives.)
>
> b. False negatives:
>
> federation.)
> have some thoughts.
> client’s identifier, the "external ID"
>
> For example:
> thing, though.
> for many implementers.
> clumsy.
> provide a multi-status.
> be a source of brittleness.
>
>
>
> as the response to a PATCH (or BULK) command.
>
>
>
> To elaborate,
>
>
>
> ------------------------------
>
> This e-mail message, including any attachments, is for the sole use of the
> person to whom it has been sent, and may contain information that is
> confidential or legally protected. If you are not the intended recipient or
> have received this message in error, you are not authorized to copy,
> distribute, or otherwise use this message or its attachments. Please notify
> the sender immediately by return e-mail and permanently delete this message
> and any attachments. Gartner makes no warranty that this e-mail is error or
> virus free.
>

---

From: Phil Hunt <[email-masked]>
Date: Fri, 10 Aug 2012 17:36:54 -0700

Ganesh,

In your example you are conflating value with an attribute id. I find that confusing. 

I agree though that operations in patch could be a lot more explicit. 

Eg explicitly deleting a value by saying delete or retire. 

Phil

On 2012-08-10, at 16:19, Ganesh and Sashi Prasad <[email-masked]> wrote:

> Pro 1. The API to manipulate resources becomes so much cleaner, consistent and intuitive when every individual attribute value gets its own ID.
> 
> I'm sure you'll agree that this is simpler, more consistent and more intuitive to a reader.
> 
> (c) Performing bulk operations, where we simply use HTTP verbs instead of the specialised (and semantically less ambiguous) verbs I suggested for attributes, the "key" becomes the URI, and the "value" becomes the corresponding JSON object.
> 
> All of them return "207 Multi-Status" with the "results" array holding individual status codes.
> 
> In the current spec, (a) and (b) are done similarly but (c) is very different.
> 
> Pro 3: Adoption of the standard by clients is likely to be higher because it's simpler for them.
> 
> Pro 4: New (not incumbent) cloud providers will probably find this easier to implement because they have no legacy. They will probably use some form of NoSQL database and won't be constrained by the limitations of LDAP directories.
> 
> Cons:
> 
> Con 1: Incumbent cloud providers with existing data stores in a directory format (where multi-valued attributes are stored as comma-separated values under a single attribute node) will find it difficult to migrate to this model and store each attribute value as a sub-node with its own key. This will "hinder adoption of the spec", which is what you fear.
> 
> Have I summed up the Pros and Cons correctly? I'm biased of course, so I could have missed a Con or hyped a Pro :-).
> 
> In other words, we're debating interface complexity (current spec) versus implementation complexity (my suggestion). Both can hinder adoption of the spec by different parties.
> 
> Here's what we need to discuss - Do the Pros make the suggestion worth adopting in spite of the Cons, or are the Cons so great that it's best to leave the spec as it is?
> 
> Keep in mind that a complex spec that only favours incumbent cloud providers can cut both ways. It opens the door to a simpler interface offered by a new generation of nimbler SPs that don't have the same legacy issues, and there could be an exodus of clients to these new SPs. SCIM could end up being obsoleted very soon, because the API interface is very complex and clumsy, as any new reader can attest. I was taken aback by the complexity when I saw it, which is why I was prompted to suggest something simpler.
> 
> This is an issue where we need the opinions of many people, and they need to state their affiliations. If most people weighing in belong to incumbent SPs and they vote in favour of interface complexity to avoid implementation complexity, then it means the spec is not doing a good job of balancing the interests of various groups. I think we should also poll client organisations to see what they would want.
> 
> (Gartner is trusted by enterprise clients to evaluate the capabilities of vendors (SPs). I believe Gartner should take the lead in representing client interests in this working group rather than those of incumbent vendors, which is what it seems like to me. Correct me if I'm being unfair.)
> 
> Regards,
> Ganesh
> 
> 
> 
> “2. When dealing with multi-valued attributes of a resource (expressed as arrays in JSON), they must be converted from an array into a dictionary with unique keys (UUIDs generated by the cloud provider when the attribute is created). Without unique keys for every attribute value of a resource, manipulating it will be clumsy and inelegant.”
> 
>  
> 
> One of the primary reasons that SPML failed was lack of adoption by service providers due to its complexity. Very few target applications implemented SPML. Most of the commercial provisioning systems had an SPML interface (either v1 or v2), but not one of them was conformant to the SPML standard because of complexity. If you are interested, I will forward you the research documents that discuss these problems in detail. For SCIM to be successful, it must be adopted by commercial target applications (i.e., service providers). I am confident that a requirement for unique identifiers with multi-valued attributes will preclude its adoption, because it requires major changes to the service provider’s existing identity storage mechanisms.
> 
> Mark
> 
>  
> 
>  
> 
>  
> 
> On Fri, Aug 10, 2012 at 5:49 AM, Ganesh and Sashi Prasad <[email-masked]> wrote:
>

---

From: Phil Hunt <[email-masked]>
Date: Fri, 10 Aug 2012 17:34:29 -0700

See below. 

Phil

> (Gartner is trusted by enterprise clients to evaluate the capabilities of vendors (SPs). I believe Gartner should take the lead in representing client interests in this working group rather than th…
Big -1

While I very much respect analysts they are not protocol designers or implementors. 

Many of us are also speaking for customers of many types and it isn't true that those needs line up with the analysts opinion. 

I doubt very much IETF would be prepared to redefine consensus giving any group higher weight.

---

From: prateek mishra <[email-masked]>
Date: Fri, 10 Aug 2012 21:12:23 -0400

Ganesh - we are all participating as individuals in IETF. Maybe you 
should consult the rules for working groups as explained in:

http://tools.ietf.org/pdf/rfc2418.pdf

It is irrelevant whether you belong to a very large technology company, 
or are a technology analyst or whether you are just a great guy/gal.

All the discussion has to be in the open and all sources (no 
confidential research reports!) have to be publicly made available.

- prateek

On 8/10/2012 7:19 PM, Ganesh and Sashi Prasad wrote:
> Pro 1. The API to manipulate resources becomes so much cleaner, 
> consistent and intuitive when every individual attribute value gets 
> its own ID.
>
>     Host:example.com  <http://example.com/>
>     Accept: application/json
>     Host:example.com  <http://example.com/>
>     Accept: application/json
> PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce Host: example.com 
> <http://example.com/> Accept: application/json Authorization: Bearer 
> h480djs93hd8 ETag: W/"a330bc54f0671c9" {
> "operations" : [
> {
> "RETIRE" : {
> PATCH /Groups/acbf3ae7-8463-4692-b4fd-9b4da3f908ce Host: example.com 
> <http://example.com/> Accept: application/json Authorization: Bearer 
> h480djs93hd8 ETag: W/"a330bc54f0671c9" {
> "operations" : [
> {
> "RETIRE" : {
> "key" : "members"
> }
> }
> ] }
>
> (c) Performing bulk operations, where we simply use HTTP verbs instead 
> of the specialised (and semantically less ambiguous) verbs I suggested 
> for attributes, the "key" becomes the URI, and the "value" becomes the 
> Pro 4: New (not incumbent) cloud providers will probably find this 
> easier to implement because they have no legacy. They will probably 
> use some form of NoSQL database and won't be constrained by the 
> limitations of LDAP directories.
>
> Cons:
>
> Con 1: Incumbent cloud providers with existing data stores in a 
> directory format (where multi-valued attributes are stored as 
> comma-separated values under a single attribute node) will find it 
> difficult to migrate to this model and store each attribute value as a 
> sub-node with its own key. This will "hinder adoption of the spec", 
> which is what you fear.
>
> Have I summed up the Pros and Cons correctly? I'm biased of course, so 
> I could have missed a Con or hyped a Pro :-).
>
> In other words, we're debating interface complexity (current spec) 
> versus implementation complexity (my suggestion). Both can hinder 
> adoption of the spec by different parties.
>
> adopting in spite of the Cons, or are the Cons so great that it's best 
> to leave the spec as it is?
>
> offered by a new generation of nimbler SPs that don't have the same 
> legacy issues, and there could be an exodus of clients to these new 
> SPs. SCIM could end up being obsoleted very soon, because the API 
> interface is very complex and clumsy, as any new reader can attest. I 
> was taken aback by the complexity when I saw it, which is why I was 
> prompted to suggest something simpler.
>
> This is an issue where we need the opinions of many people, and they 
> need to state their affiliations. If most people weighing in belong to 
> incumbent SPs and they vote in favour of interface complexity to avoid 
> implementation complexity, then it means the spec is not doing a good 
> job of balancing the interests of various groups. I think we should 
> also poll client organisations to see what they would want.
>
> (Gartner is trusted by enterprise clients to evaluate the capabilities 
> of vendors (SPs). I believe Gartner should take the lead in 
> representing client interests in this working group rather than those 
> of incumbent vendors, which is what it seems like to me. Correct me if 
> I'm being unfair.)
>
> Regards,
> Ganesh
>
>
>
> On 11 August 2012 01:35, Diodati,Mark <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Ganesh,
>
>     "2. When dealing with multi-valued attributes of a resource
>     (expressed as arrays in JSON), they must be converted from an
>     array into a dictionary with unique keys (UUIDs generated by the
>     cloud provider when the attribute is created). Without unique keys
>     for every attribute value of a resource, manipulating it will be
>     clumsy and inelegant."
>
>     One of the primary reasons that SPML failed was lack of adoption
>     by service providers due to its complexity. Very few target
>     applications implemented SPML. Most of the commercial provisioning
>     systems had an SPML interface (either v1 or v2), but not one of
>     them was conformant to the SPML standard because of complexity. If
>     you are interested, I will forward you the research documents that
>     discuss these problems in detail. For SCIM to be successful, it
>     must be adopted by commercial target applications (i.e., service
>     providers). I am confident that a requirement for unique
>     identifiers with multi-valued attributes will preclude its
>     adoption, because it requires major changes to the service
>     provider's existing identity storage mechanisms.
>
>     Mark
>
>     *From:*Trey Drake [mailto:[email-masked]
>     <mailto:[email-masked]>]
>     *Cc:* [email-masked] <mailto:[email-masked]>; Emmanuel Dreux; Kelly
>     Grizzle; Phil Hunt
>
>     attribute within the *schema* specification.  As for #2, the
>     protocol spec works as you describe if "arbitrary URI parameters"
>     equate to resource attributes (Allow generic search using 'GET
>     /Users' and arbitrary URI parameters).  Please clarify your
>     suggestion.
>
>     hence retrieve resources on any attribute it chooses, externalId
>     or otherwise.  Nothing mandates use of externalId.
>
>     >  I think scim gets its current simplicity from its single owner
>     hub spoke model implementing tight coupling. [...] IMHO loose
>     coupling is a much more complex solution.
>
>     Phil,
>
>     I'm a bit surprised that you're implying "tight coupling ==
>     simple" and "loose coupling == complex". That's contrary to my
>     experience.
>
>     Let's not confuse reduction of dependencies in the data model with
>     a hub-and-spokes architecture. They're entirely orthogonal aspects
>     of the solution.
>
>     1. The client enterprise can still send its internal ID as part of
>     the resource body, inside some attribute defined by them (but not
>     defined by the protocol). Let's say they call it 'myID'.
>
>     Since myID is a candidate key, the server will return exactly one
>     URI, which is the canonical URI for the resource
>
>     On 2012-08-09, at 14:14, Ganesh and Sashi Prasad
>         > storing this information in a mapping table outside of the
>         SCIM spec is a great way to enable this solution.  Part of the
>         key here is that SCIM is just a piece of the architecture for
>         this solution, and is only responsible for the transport layer
>         between domains.
>
>         merging identities is a common requirement, and that
>         decoupling local identifiers within a domain from shared
>         identifiers between domains was the best way to facilitate it.
>
>         perhaps the recommendation) that it be the shared one.
>         Allowing one domain to expose its internal identifier to the
>         other creates tight coupling and ensures that both domains
>         need simultaneously split or merge identities, which is not
>         desirable. So I recommend _taking out_ the "external id" field
>         from the API. The spec shouldn't encourage tight coupling. If
>         clients want to pass in their internal ids as part of the
>         resource body, no one can stop them, and they can always do a
>         search on that attribute to retrieve the URI exactly as you
>         visualise they will with the "external id", but let's not
>         elevate an anti-pattern to a recommendation by enshrining the
>         "external id" as an acceptable attribute.
>
>         Am I making sense?
>
>     I see what you are saying. I think scim gets its current
>     simplicity from its single owner hub spoke model implementing
>     tight coupling.
>
>     IMHO loose coupling is a much more complex solution. The reality
>     is that each end-point has value to contribute and thus the
>     single-owner model will eventually need to become multi-owner or
>     multi-hub.
>
>     > Regarding unique identifiers for multi-valued attributes there
>     is a trade-off involved.  On one hand this makes PATCH semantics
>     easier.  On the other hand it puts extra burden on service providers.
>
>
>     interesting to hear from the other members of the spec mailing
>     list. You know where I stand on this. It would be good to hear the
>     spectrum of opinions.
>
>     Regards,
>
>     Ganesh
>
>     On 10 August 2012 00:28, Kelly Grizzle
>     <[email-masked] <mailto:[email-masked]>>
>     wrote:
>
>     you suggest, storing this information in a mapping table outside
>     of the SCIM spec is a great way to enable this solution. Part of
>     the key here is that SCIM is just a piece of the architecture for
>     this solution, and is only responsible for the transport layer
>     between domains.
>
>     Here is an example of how to model the end state of the false
>     positive scenario (splitting a user):
>
>     Primary flag |
>     This could be represented as two SCIM users that contain
>     information about the entities on other domains.
>
>     {
>
>     Primary flag |
>     Regarding unique identifiers for multi-valued attributes there is
>     a trade-off involved.  On one hand this makes PATCH semantics
>     easier.  On the other hand it puts extra burden on service
>     providers. Since the inception of SCIM, a key goal has been to
>     foster adoption by service providers by making things fit easily
>     onto existing systems.  IMO the value gained by unique identifiers
>     for multi-valued attributes is not worth the demands put on a
>     service provider.  I also think that vendors that have a
>     non-SCIM-compliant API will choose to keep things that way if the
>     spec is too hard for them to implement. In a green field
>     environment we do have the luxury of mandating a model to make
>     Nothing prevents you in your SCIM implementation (client or
>     server) to generate a unique identifier for each synchronized
>     object and maintain an internal mapping table ( you would have to
>     map group membership as well).
>
>     (DN,samAccountName, UPN) are subject to change (even objectGuid
>     can change if an AD domain is migrated), we decided to generate
>     and maintain an internal table of ids. This fits your requirements
>     as it hides internal ids.
>
>     This was written in dotnet and we have started a project to
>     rewrite our SCIM stack in PHP and will give it to the Open Source
>     community. This implementation will have a parameter : AllocateIds
>     versus UseExistingIDs.
>
>     You can also implement such feature without violating the SCIM
>     specs, or without asking to include it in the specs.
>
>     --
>
>     Regards,
>
>     Emmanuel Dreux
>
>     http://www.cloudiway.com
>
>     1.Why should domains not expose their internal identifiers to
>     other domains?
>
>     a.
>
>     We are designing a protocol for a federated system of domains,
>     where all domains are co-equal peers. (In physics too, N-body
>     problems are much harder than 2-body problems. :-) Therefore,
>     assuming that there are only two players in the interaction makes
>     this tightly coupled in a number of ways. We should rely on
>     messaging and notification, with encapsulation of domain-specific
>     data.
>
>     b.
>
>     In any non-trivial data store, there will always be the ongoing
>     need to merge and split identities as and when "false negatives"
>     and "false positives" are discovered. A domain should be able to
>     handle this internal housekeeping freely, only notifying other
>     domains when convenient. Mapping of internal identifiers to
>     external ones and maintaining this mapping internally allows this
>     loosely-coupled housekeeping to take place. Sharing internal
>     identifiers (or otherwise outsourcing the mapping of internal to
>     protocol which can be designed later. The data model plays a
>     crucial role in enabling or constraining such interaction. A
>     tightly-coupled data model will force the use of synchronous
>     interactions, and the exposure of internal identifiers is a key
>     part of this tight coupling.
>
>     2. The difficulty of assigning unique identifiers to the
>     individual values of multi-valued attributes:
>
>     a.
>
>     I'm not belittling the effort involved in migrating legacy data
>     stores to such a model. However, in the larger historical context
>     of cross-domain identity management, we are really at the very
>     early stages. If a relatively new discipline and a brand new spec
>     are held captive to legacy considerations, we are losing an
>     opportunity to provide a clean and elegant model to subsequent
>     users of the spec, and this will have repercussions over many
>     years or even decades.
>
>     b.
>
>     customers to adopt the "SCIM-compliant" API. Legacy customers can
>     be supported using a "non-SCIM-compliant" API for an arbitrarily
>     long period and gradually migrated to the SCIM-compliant API. The
>     logistics are not insurmountable, and shouldn't prevent the
>     adoption of a dictionary model for multi-valued attributes.
>
>     Elaboration of Point 1:
>
>     When we consider federated identity across more than one domain,
>     we have to assume that domains are not necessarily master-slave in
>     their interaction. The most generic interaction model is
>     peer-to-peer, where entity lifecycle events within a domain are
>     notified to other domains (when necessary) in an asynchronous
>     manner (i.e., through messaging) and the other domains are free to
>     respond to these events in an appropriate manner and at a time of
>     their convenience.
>
>     Consider a case where customers sign up online, and two customers
>     who are privacy-conscious enter fake IDs such as "John Smith", and
>     also use the same date of birth (say, 1 Jan 1970) or similar
>     attributes. The front-end application may make an intelligent (but
>     incorrect) guess that these two persons are the same, and
>     re-assign the same identifier to the second person. This is a
>     false positive. They appear to be the same entity, but they're
>     actually different. When the error is discovered, the identities
>     "JSmith" and "JohnS". It is very likely that they will be treated
>     as two different customers and assigned two unique identifiers.
>     This is a false negative. They appear to be two entities, but are
>     actually the same. At a later stage, when the error is discovered,
>     the identities will have to be merged, and one of the identifiers
>     will have to be dropped.
>
>     Domain 1 then maintains the following in a mapping table and uses
>     it for translation when talking to Domain 2, taking care never to
>     expose its internal identifier:
>
>     Primary flag |
>     When the false positive is discovered and the entity is split,
>     Domain 1 creates a new internal identifier and now has the
>     following entity information.
>
>     This second entity with its own internal identifier is invisible
>     to Domain 2, and this is by design. Communication about the
>     original entity takes place as before by mapping "9caf78aac3d6" to
>     "ff487230b3a0" and vice-versa. At some convenient time
>     (importantly, this doesn't have to be at the time the split
>     happens), Domain 2 can be requested to provision a second entity,
>     and when it responds with an identifier of "7a87f27c1dd8", this
>     can go into the mapping table as a new record associated with the
>     second entity's internal identifier.
>
>     Primary flag |
>     Domain 1 has the following information about what it thinks are
>     two distinct customers in its data store:
>
>     When requesting the provisioning of these entities in Domain 2,
>     the following IDs are returned by Domain 2: ff487230b3a0 and
>     41206cc97c8b.
>
>     Domain 1 then maintains the following in a mapping table and uses
>     it for translation when talking to Domain 2, taking care never to
>     expose its internal identifiers:
>
>     Primary flag |
>     When the false negative is discovered and the two entities are
>     merged, Domain 1 drops one of the internal identifiers and
>     rationalises the name of the customer (say, to "John Smith").
>     Let's say it retains the first ID "9caf78aac3d6" and drops the
>     second "273d36e30d09".
>
>     Primary flag |
>     Now two external identifiers map to the same internal one, so
>     inbound communication from Domain 2 can be unambiguously
>     translated to the same entity internally. However, when going
>     outwards, Domain 1 will have to look up the translation table to
>     determine the "primary" external ID for this entity in Domain 2,
>     which was decided to be "ff487230b3a0". That's where the "Primary
>     flag" comes in. The second external ID "41206cc97c8b" is never
>     At some stage (importantly, not in lockstep with the identity
>     merge), Domain 2 can be requested to delete the customer record
>     identified by "41206cc97c8b", and the second entry in the mapping
>     table can be removed once this is acknowledged.
>
>     This scheme will scale up to multiple domains, because the
>     "External Domain ID" column helps to keep track of which external
>     ID is shared with which Domain. (Why don't we use just one
>     external ID for an entity and share it with all external domains?
>     Tight coupling again. Just as OAuth allows an access token given
>     to a third party to be invalidated without affecting the access of
>     other third parties, the use of separate external identifiers for
>     The scheme also allows the splitting of an entity into more than
>     two entities, and the merging of more than two entities into a
>     single one. (Any organisation with a web-facing application will
>     tell you how many John Smiths there are who were born on 1 Jan 1970!)
>
>     This is a fairly long-winded explanation, but this is why we need
>     to hide internal identifiers from other domains, and why mappings
>     need to be managed internally in each domain. Such a data model
>     also allows us to choose asynchronous protocols for propagation of
>     identity events, since there is no consistency requirement to
>     update multiple domains concurrently.
>
>     Regards,
>
>     Ganesh Prasad
>
>     Thanks for the feedback, Ganesh.  I read through this and your
>     InfoQ article
>     (http://www.infoq.com/articles/scim-data-model-limitations) and
>     have some thoughts.
>
>     The usage pattern for an external ID would be to search for a user
>     by externalId and use the ID of the returned user in any desired
>     operation. For example:
>
>     This does introduce an additional HTTP request if the client
>     chooses not to store the server's id. An issue was created to
>     consider allowing operations to use the externalId
>     (http://code.google.com/p/scim/issues/detail?id=35), but I believe
>     the general consensus has been to not include this in the spec.
>     One main point of contention is that much of the rest of the spec
>     (eg -- group membership references, manager references, etc...)
>     require knowledge of the server's identifier. Continuing this
>     discussion on the IETF list would be a good thing, though.
>
>     I think this comes down to a nomenclature problem.  The server's
>     ID does not necessarily have to be the unique identifier that the
>     underlying identity store uses, it just has to be stable and
>     unique.  In many cases, the underlying identity store will provide
>     identifiers with these properties already (eg -- a uuid) and it
>     can be used by the SCIM interface. The "externalId" is referring
>     to the fact that the id is maintained external to the SCIM
>     server.  As long as the server's identifiers are stable and unique
>     (which is mandated by the spec), I don't see a problem.
>
>     > simple - turn every list or array (of values) into a dictionary
>     (of key-value pairs) by providing each value
>
>     I agree that this would be useful, especially in the PATCH
>     operation. One reason that this wasn't included in the spec
>     originally is that it can put undue burden on the service
>     provider. Many service providers are putting SCIM interfaces in
>     front of their existing identity stores (eg -- directory servers,
>     SaaS application databases, etc...).  Many of these do not have a
>     unique identifier for multi-valued attributes. By requiring this,
>     a majority of the server providers would have to start maintaining
>     a unique key for each multi-valued attribute.  I believe this
>     would be a roadblock for many implementers.
>
>     I like the thoughts here.  Your example reminds me of unified
>     diffs (http://en.wikipedia.org/wiki/Diff#Unified_format), which
>     are commonly used with a patch program (pretty much the equivalent
>     of the PATCH verb).  However, the three proposals seem to largely
>     hinge on being able to uniquely address each element within an
>     object. Without these it is not so easy to address each patch
>     sub-operation (REPLACE, INCLUDE, etc...) or provide a multi-status.
>
>
>     > There are other, non-data aspects of SCIM which may require
>     review, such as its synchronous request-response
>
>     insurance company, an experience I have written about as a eBook
>     on InfoQ
>     (http://www.infoq.com/minibooks/Identity-Management-Shoestring).
>
>     1. The enterprise client and the cloud provider should maintain
>     their own internal IDs for a resource, which they should not
>     reveal to each other. Both of them should map their internal IDs
>     to a shared External ID, and this is the only ID that should be
>     exposed through the API. The current specification's provision of
>     an id (which is the external ID and the only one to be transferred
>     through the API) and an "external ID" (which is the client's
>     internal ID and should be hidden) is diametrically opposite to this.
>
>     2. When dealing with multi-valued attributes of a resource
>     (expressed as arrays in JSON), they must be converted from an
>     array into a dictionary with unique keys (UUIDs generated by the
>     cloud provider when the attribute is created). Without unique keys
>     for every attribute value of a resource, manipulating it will be
>     clumsy and inelegant.
>
>     3c. Use the WebDAV status code of "207 Multi-Status" instead of
>     "200 OK" as the response to a PATCH (or BULK) command.
>
>     To elaborate,
>
>     identifiers are revealed to external domains, such clean-ups
>     become difficult, hence every domain that wants to expose
>     references to a resource must map its internal ID to and external
>     one created for this explicit purpose, and only reveal this.
>
>     In the SCIM case, when an enterprise client POSTs a resource
>     creation request, the cloud provider must generate its own
>     internal UUID as well as an external UUID, map them together, and
>     only return the external UUID in the "Location:" header. The
>     enterprise client should map this external UUID to a
>     newly-generated internal ID of its own. In case the resource
>     already has an identifier within the enterprise client's domain,
>     then this is the internal ID that must be mapped to the external
>     UUID returned through the POST response.
>
>     This e-mail message, including any attachments, is for the sole
>     use of the person to whom it has been sent, and may contain
>     information that is confidential or legally protected. If you are
>     not the intended recipient or have received this message in error,
>     you are not authorized to copy, distribute, or otherwise use this
>     message or its attachments. Please notify the sender immediately
>     by return e-mail and permanently delete this message and any
>     attachments. Gartner makes no warranty that this e-mail is error
>     or virus free.
>
>
>
>

---

From: Trey Drake <[email-masked]>
Date: Sat, 11 Aug 2012 10:59:23 -0500

> I didn't realise SCIM was also specifying a User schema. Does this mean the User resource can only hold certain attributes as defined by this schema? If so, that's going to be a severe constraint,…
I recommend you read the relevant specifications. The working group has accepted both a schema and protocol draft as input to the effort. 

> Yes, that's what I meant, but in implementation, it may work out to be looser than that. The client should be able to specify any arbitrary attribute in the search parameters, even those that aren'…
And that's because?  I don't see any value to the behavior (enabling attribute searches on non-existent schema) you suggest. 

> By "candidate key" (a data modelling term), I meant another attribute that also uniquely identifies a single record, but is not the official "primary key", i.e., the main ID. In this case, a client…

I know what a candidate key is.  My question is why do you believe an SP ought to care about "candidate keys"?   Resources are uniquely identified exactly 1 way by the SP and that is by the SP minted id.  The Consumer can retrieve a resource by any attribute though only the id is guaranteed to be unique from the POV of the SP.  The SCIM model is a *mapping* between the consumer and SP hence it is not a prescription for how the Resource data is actually stored on either side.
  
> Look, this may seem to be splitting hairs since a determined client may be able to store and search by their own internal ID in either case (the current SCIM spec or my suggestion). The difference…

If externalId were dropped it wouldn't bother me a bit.

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Sun, 12 Aug 2012 02:18:46 +1000

> And that's because?  I don't see any value to the behavior (enabling
attribute searches on non-existent schema) you suggest.

I wasn't suggesting there was value in it. I was covering the possibility
that the client could erroneously send URI parameters that are not valid
attributes of the resource, and the only response to such requests is a
"400 Bad Request".

>  My question is why do you believe an SP ought to care about "candidate
keys"?   Resources are uniquely identified exactly 1 way by the SP and that
is by the SP minted id.  The Consumer can retrieve a resource by any
attribute though only the id is guaranteed to be unique from the POV of the
SP.  The SCIM model is a *mapping* between the consumer and SP hence it is
not a prescription for how the Resource data is actually stored on either
side.

I think we may actually be in violent agreement here! Your description
exactly matches my recommendation of a data model that is loosely-coupled
between client and SP. The SP should not care about "candidate keys", of
course. But if one of the attributes of the resource happens to be a
client-internal key, then the _client_ knows that it is a candidate key,
and that any search based on the candidate key will return at most one URI
with the "SP minted ID".

This is what I'd like to see in the SCIM spec. There is only one key
exposed _as a key_ between client and SP, and that is the "SP minted ID".
There is no _special_ attribute in the API called an "external ID". If the
client wants to embed its internal identifier as an attribute within the
resource, it is free to do so. The SP will treat it as just another
attribute and will not even bother to impose a uniqueness constraint on it.

> If externalId were dropped it wouldn't bother me a bit.

Good. Does anyone have an actual _objection_ to dropping the "externalID"
field? As I said above, it doesn't prevent a client from sending such an
attribute as an ordinary attribute and doing searches on it. Keeping the SP
ignorant of its status as a key within the client decouples the two parties
to some extent, so dropping the specially-named "externalID" is desirable.

Regards,
Ganesh

On 12 August 2012 01:59, Trey Drake <[email-masked]> wrote:

> I didn't realise SCIM was also specifying a User schema. Does this mean
> the User resource can only hold certain attributes as defined by this
> schema? If so, that's going to be a severe constraint, because client
> organisations are likely to require many specialised User attributes
> depending on the nature of their business, and they will expect to be able
> to store all of them, not just the subset defined by the SCIM spec. I hope
> the SCIM User schema is not trying to be just a representation of
> inetOrgPerson (the standard LDAP schema), because that could be severely
> limiting. Most organisations with their own LDAP end up extending
> inetOrgPerson, and they will expect that they can do the same in the cloud.
>
> I recommend you read the relevant specifications. The working group has
> accepted both a schema and protocol draft as input to the effort.
>
> And that's because?  I don't see any value to the behavior (enabling
> attribute searches on non-existent schema) you suggest.
>
>
> I know what a candidate key is.  My question is why do you believe an SP
> ought to care about "candidate keys"?   Resources are uniquely identified
> exactly 1 way by the SP and that is by the SP minted id.  The Consumer can
> retrieve a resource by any attribute though only the id is guaranteed to be
> unique from the POV of the SP.  The SCIM model is a *mapping* between the
> consumer and SP hence it is not a prescription for how the Resource data is
> actually stored on either side.
>
>
>
>> attribute within the *schema* specification.  As for #2, the protocol spec
>> works as you describe if "arbitrary URI parameters" equate to resource
>> attributes (Allow generic search using 'GET /Users' and arbitrary URI
>> parameters).  Please clarify your suggestion.
>>
>>>> that each end-point has value to contribute and thus the single-owner model
>>>> will eventually need to become multi-owner or multi-hub.
>>>>
>>>>> extension but would probably not want to expose these externally.  Here is
>>>>> an example of how to model the end state of the false positive scenario
>>>>> (splitting a user):****
>>>>>
>>>>> ** **
>>>>>
>>>>> our SCIM stack in PHP and will give it to the Open Source community. This
>>>>> This will give the choice of “hiding” internalIDs or use them as
>>>>> unique ID.****
>>>>>
>>>>> ** **
>>>>>
>>>>> main points you have made, and then elaborate on the first.****
>>>>>
>>>>> to merge and split identities as and when “false negatives” and “false
>>>>> proportion of the user base in most large Web-facing applications. Let's
>>>>> see how these can be managed in a federated way by mapping internal
>>>>> identifiers to external ones and only exposing external identifiers to
>>>>> other domains.****
>>>>>
>>>>> a. False positives:****
>>>>>
>>>>> endpoint (
>>>>> add, modify and delete attributes at any level****
>>>>>

---

From: Ganesh and Sashi Prasad <[email-masked]>
Date: Sun, 12 Aug 2012 02:52:15 +1000

There's another aspect to this discussion that I've been meaning to bring
up for a while, but I've been getting side-tracked by the (necessary)
discussions on the external ID and multi-valued attribute treatment.

This is about asynchronous behaviour.

I think someone mentioned early on in the thread that we would like to get
a synchronous spec out the door first and then think about doing an async
version.

IMO, this "synchronous first, asynchronous later" approach could entail a
lot of rework at a later stage, which may then not be undertaken because it
breaks backwards-compatibility. We should plan ahead for asynchronous
support.

Adding synchronous protocol support to an asynchronous protocol is trivial.
We just need to define what happens when the initiator blocks on a response
instead of going off to do other things and relying on notification.

However, the reverse is not true. When we design synchronous protocols, we
may often slip aspects of tight coupling (at the data layer) into the
protocol, and this will not be visible because the protocol is synchronous
(tightly coupled) anyway. When we then attempt to make this protocol
asynchronous, the data layer tight coupling will emerge as a major
constraint preventing this.

An example is an attribute that has to be kept in sync across two data
stores. This implies some form of two-phase commit protocol which doesn't
lend itself well to asynchronous interactions. The external ID is a prime
example of this kind of data coupling. If one domain has to do housekeeping
internally to clean up its data, and this involves splitting or merging
identities, then having these identifiers stored within another domain
(e.g., externalID) means those domains also have to be changed at the same
time. I had earlier shown how to do it asynchronously with mapping tables.

This is another reason why I suggest using non-HTTP verbs inside a PATCH.
When we change from synchronous to asynchronous interaction, we _may_ use a
wire protocol other than HTTP (e.g., WebSockets) and then the outer HTTP
verbs will have to be replaced with something else. If we use the suggested
reserved words INCLUDE, PLACE, REPLACE, FORCE and RETIRE, the inner verbs
will not have to change at all.

We should also think about idempotence and models like "eventual
consistency". INCLUDE is not idempotent, and this is where the client may
have to be aware of dictionary keys for array elements and generate its own
suggested UUIDs. Using a PLACE instead of an INCLUDE will make the request
idempotent. (This is not tight coupling because the UUID is the only shared
identifier between client and SP.)

Regards,
Ganesh

---

From: Phil Hunt <[email-masked]>
Date: Sat, 11 Aug 2012 11:06:38 -0600

Yes. I object. ExternalId is optional. There is no need for this major breaking change. IMHO. 

Phil

On 2012-08-11, at 10:18, Ganesh and Sashi Prasad <[email-masked]> wrote:

> I wasn't suggesting there was value in it. I was covering the possibility that the client could erroneously send URI parameters that are not valid attributes of the resource, and the only response to such requests is a "400 Bad Request".
> 
> >  My question is why do you believe an SP ought to care about "candidate keys"?   Resources are uniquely identified exactly 1 way by the SP and that is by the SP minted id.  The Consumer can retrieve a resource by any attribute though only the id is guaranteed to be unique from the POV of the SP.  The SCIM model is a *mapping* between the consumer and SP hence it is not a prescription for how the Resource data is actually stored on either side. 
> 
> I think we may actually be in violent agreement here! Your description exactly matches my recommendation of a data model that is loosely-coupled between client and SP. The SP should not care about "candidate keys", of course. But if one of the attributes of the resource happens to be a client-internal key, then the _client_ knows that it is a candidate key, and that any search based on the candidate key will return at most one URI with the "SP minted ID".
> 
> This is what I'd like to see in the SCIM spec. There is only one key exposed _as a key_ between client and SP, and that is the "SP minted ID". There is no _special_ attribute in the API called an "external ID". If the client wants to embed its internal identifier as an attribute within the resource, it is free to do so. The SP will treat it as just another attribute and will not even bother to impose a uniqueness constraint on it.
> 
> Good. Does anyone have an actual _objection_ to dropping the "externalID" field? As I said above, it doesn't prevent a client from sending such an attribute as an ordinary attribute and doing searches on it. Keeping the SP ignorant of its status as a key within the client decouples the two parties to some extent, so dropping the specially-named "externalID" is desirable.
> 
> Regards,
> Ganesh
>

---

From: Leif Johansson <[email-masked]>
Date: Sun, 12 Aug 2012 15:18:49 +0200

> (Gartner is trusted by enterprise clients to evaluate the
> capabilities of vendors (SPs). I believe Gartner should take the
> lead in representing client interests in this working group rather
> than those of incumbent vendors, which is what it seems like to me.
> Correct me if I'm being unfair.)

Ganesh,

If you have contacts at Gartner who are interested in participating
please encourage them to do so. However, participants in IETF WGs
are not selected to "represent" any one constituency as somebody
already pointed out.

As co-chair of this WG , may I suggest that you present your proposal
for an alternative SCIM protocol in the form of an Internet Draft.

That way the WG can comment on your proposals as a whole.

	Best R,
	Leif

---

From: Leif Johansson <[email-masked]>
Date: Sun, 12 Aug 2012 15:22:41 +0200

On 08/10/2012 09:51 PM, Trey Drake wrote:
> Core schema is here:
> https://datatracker.ietf.org/doc/draft-scim-core-schema/
> 

I was wondering about that - why not cancel the submission and re-submit
with corrected metadata using the automatic submissions process?

The reason these drafts don't show up in the wg page on datatracker
is that they were named draft-scim-* instead of draft-<author
moniker>-scim-* which is the norm for individual submissions to the
IETF.

However I don't propose changing that now since we'll have a consensus
call in the WG to adopt (or not) these drafts as WG documents in which
case they will change names anyway.

	Cheers Leif