Subject: Re: [decade] -req and -arch documents

From: "Zhang Peng" <[email-masked]>
Date: Mon, 2 Jul 2012 01:01:34 -0500

Hi Richard,

Thanks for your effort. Here are some suggestion about the revision of -req documents.

First, regarding Q1, I think there are indeed some overlaps on term definitions in -req and -arch documents. In addition, they are even not consistent. I think we can just put them in the -req document, and give a pointer in -arch document. 
Regarding Q2, I agree that in -arch, there are some paragraphs that also do the job of define requirements, and may better be put in -req. For naming, the reqiin -requirements given in -req are rather simplistic, and we can move the move the detailed requirements to -req. But for "immutable object", I think it is more like a design option for the architecture (so that the design would be much simplified), rather than a system requirements for DECADE to function.  Thus, I suggest we don't move the immutable data objects in -arch to -req, and we can even delete them in -req. Finally, I would say that there is no clear cut for requirement and architecture, since when we specify the requirements, we have some design rationale in mind; and when we design the architecture, we would find more clear requirements for the specific design. It is not easy to fully decouple them.
Regarding Q3, I think the new outline of the -req document seems better, but I think it is better to organize it into three parts: access/authorization, resource control, data storage/transfer, failure handling/gaming prevention. The rationale is that the first part is about how client access the server, the second part corresponds to DRP, the third part corresponds to SDT, and the last two parts are about reliability/security. The following is the outline in my mind. I would like to discuss with you about the organization further. And I would like to help revise the -req document.  Thanks.

Best,

Peng.

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 5
2. Terminology and Concepts . . . . . . . . . . . . . . . . . . . 6
3. System/Functional Components
4. Requirements Structure . . . . . . . . . . . . . . . . . . . . 7

5. Data Object Requirements
4.5.1. Data Naming . . . . . . . . . . . . . . . . . . . . . .14
Move part of Arch 4.4 to -req
4.2.1. Data Object Size . . . . . . . . . . . . . . . . . 11
4.4.2. System Data Object Attributes . . . . . . . . . . . . .13
4.4.3. Time-to-live for Written Data Objects . . . . . . 13
4.4.4. Application-defined Properties and Metadata . . . . . 13

6. Authentication and Access Requirements
4.7.1. Per-Remote-Client, Per-Data Read Access . . . . . . . 16
4.7.2. Per-User Write Access . . . . . . . . . . . . . . . . 17
4.7.3. Default Access Permissions . . . . . . . . . . . . . . 17
4.7.4. Authorization Checks . . . . . . . . . . . . . . . . . 17
4.7.5. Cryptographic Credentials . . . . . . . . . . . . . . 17
4.7.6. Server Involvement . . . . . . . . . . . . . . . . . . 18
4.7.7. Protocol Reuse . . . . . . . . . . . . . . . . . . . . 18
4.3.1. Read/Write Own Storage . . . . . . . . . . . . . . .. 11
4.3.2. Read/Write by Remote Clients . . . . . . . . . . . . . 11
4.4.5. Offline Access . . . . . . . . . . . . . . . . . . . 14

7. Data Transfer Requirements 
4.1.1.1. NATs and Firewalls TRaversal . . .. . . . . . . . . 8
Connections to Clients . . . . . . . . . . . . . . . . . . . 8
6.1.1. Support for Clients Behind NATs and Firewalls . . . 21
4.3.3. Negotiable Data Transport Protocol . . . . . . . . . . 12
4.1.2.1. Secure Transport . . . . . . . . . . . . . . . . . . 8

8. Resource Control Requirements. . . . . . . . . . . . . . . . . 15
4.6.1. Multiple Applications . . . . . . . . . . . . . . . . 15
4.6.2. Per-Remote-Client, Per-Data Control . . . . . . . . . 15
4.6.3. Resource Control Set . . . . . . . . . . . . . . . . . 16
4.6.4. Server Involvement . . . . . . . . . . . . . . . . . . 16
8. Authorization Requirements

9. Error and Failure Handling Requirements. . . . . . . . . . . . 9
4.1.3.2. Insufficient Resources . . . . . . . . . . . . . . . 9
4.1.3.1. Overload Condition . . . . . . . . . . . . . . . 9
4.1.3.3. Unavailable and Deleted Data . . . . . . . . . . . . 10
4.1.3.4. Insufficient Permissions . . . . . . . . . . . . . . 10
4.1.3.5. Redirection . . . . . . . . . . . . . . . . . . . . 10
4.1.2.2. Gaming Prevention . . . . . . . . . . . . . . . . . .8
4.4.1. Server reliability (Agnostic of reliability) . . . . . 12

10. Server Status Query Requirements . . . . . . . . . . . . . . . 18
5.7. Storage Status . . . . . . . . . . . . . . . . . . . . . 20
....

11. Security Considerations . . . . . . . . . . . . . . . . . . . 21
7.1. Authentication and Authorization . . . . . . . . . . . . 21
7.2. Encrypted Data . . . . . . . . . . . . . . . . . . . . . 22
7.3. Protection against Gaming . . . . . . . . . . . . . . . 22

Zhang Peng

From: Y. Richard Yang
Date: 2012-06-30 06:40
To: [email-masked]
Subject: [decade] -req and -arch documents
Dear DECADE participants,

Both -req and -arch documents have received extensive, excellent 
reviews. In the process of revising the documents and looking at the two 
documents, we have some questions to seek input from the participants:

Q1. Both -req and -arch define some terms, including DECADE-compatible 
client, DECADE-compatible server, etc. This appears to be redundant. 
Should such terms be defined in -req or -arch? The current thinking is 
-req. Any comments?

Q2. There are multiple paragraphs defining requirements in -arch, e.g., 
Sec. 4.4 on naming. Should these be moved to -req? A more general 
question is on the dependency between -req and -arch. On one hand, it is 
ideal to define -req first without a particular architecture to allow 
competing/alternative architectures satisfying the requirements. On the 
other hand, it appears that the -arch document contains content, in 
particular, systems/function components, that can be helpful to better 
understand the -req document. The current thinking is still to make -req 
independent of -arch, and move some general requirements in -arch to 
-req. Any comments?

Q3: There are some good comments on reorganizing the -req documents. 
Below is a new structure (second level section #'s and page numbers are 
from the current -req so that others can see how we re-organize the 
document). Any comments/feedback before we start the revision will be 
greatly appreciated.

Also, anyone who can help with direct revisions of the -req document 
will be more than welcome!

Thanks.

-- 
Richard

1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  5
2.  Terminology and Concepts . . . . . . . . . . . . . . . . . . .  6
3.  System/Functional Components
4.  Requirements Structure . . . . . . . . . . . . . . . . . . . .  7

5.  Data Object Requirements
     4.5.1.  Data Naming . . . . . . . . . . . . . . . . . . . . . .14
             Move part of Arch 4.4 to -req
     5.1.  --Immutable Data . . . . . . . . . . . . . . . . . . . . 18
     4.4.2.  System Data Object Attributes . . . . . . . . . . . . .13
         4.2.1.  Data Object Size . . . . . . . . . . . . . . . . . 11
         4.4.3.  Time-to-live for Written Data Objects  . . . . . . 13
     4.4.4.  Application-defined Properties and Metadata  . . . . . 13

6.  Access to Server Requirements
     4.3.1.  Read/Write Own Storage  . . . . . . . . . . . . . . .. 11
     4.3.2.  Read/Write by Remote Clients . . . . . . . . . . . . . 11
     4.4.5.  Offline Access  . . . . . . . . . . . . . . . .  . . . 14
     4.1.1.1.  NATs and Firewalls TRaversal . . .. . .  . . . . . .  8
       Connections to Clients . . . . . . . . . . . . . . . . . . .  8
       6.1.1.  Support for Clients Behind NATs and Firewalls  . . . 21
     4.3.3.  Negotiable Data Transport Protocol . . . . . . . . . . 12
     5.2.  Explicit Deletion of Data  . . . . . . . . . . . . . . . 19
     5.3.  Concurrent Write (Multiple writing). . . . . . . . . . . 19
     5.4.  Concurrent Read (Multiple reading) . . . . . . . . . . . 19
     5.5.  Read While Write . . . . . . . .   . . . . . . . . . . . 19
     5.6.  Partial Write (Writing model) . . . . . . . . . . . . . 20
     4.1.2.1.  Secure Transport . . . . . . . . . . . . . . . . . .  8

7.  Resource Control Requirements. . . . . . . . . . . . . . . . . 15
     4.6.1.  Multiple Applications  . . . . . . . . . . . . . . . . 15
     4.6.2.  Per-Remote-Client, Per-Data Control  . . . . . . . . . 15
     4.6.3.  Resource Control Set . . . . . . . . . . . . . . . . . 16
     4.6.4.  Server Involvement . . . . . . . . . . . . . . . . . . 16

8.  Authorization Requirements
     4.7.1.  Per-Remote-Client, Per-Data Read Access  . . . . . . . 16
     4.7.2.  Per-User Write Access  . . . . . . . . . . . . . . . . 17
     4.7.3.  Default Access Permissions . . . . . . . . . . . . . . 17
     4.7.4.  Authorization Checks . . . . . . . . . . . . . . . . . 17
     4.7.5.  Cryptographic Credentials  . . . . . . . . . . . . . . 17
     4.7.6.  Server Involvement . . . . . . . . . . . . . . . . . . 18
     4.7.7.  Protocol Reuse . . . . . . . . . . . . . . . . . . . . 18

9.  Error and Failure Handling Requirements. . . . . . . . . . . .  9
     4.1.3.2.  Insufficient Resources . . . . . . . . . . . . . . .  9
         4.1.3.1.  Overload Condition . . . . . . . . . . . . . . .  9
     4.1.3.3.  Unavailable and Deleted Data . . . . . . . . . . . . 10
     4.1.3.4.  Insufficient Permissions . . . . . . . . . . . . . . 10
     4.1.3.5.  Redirection  . . . . . . . . . . . . . . . . . . . . 10
     4.1.2.2.  Gaming Prevention  . . . . . . . . . . . . . . . . . .8
     4.4.1.  Server reliability (Agnostic of reliability) . . . . . 12

10. Server Status Query Requirements . . . . . . . . . . . . . . . 18
     5.7.  Storage Status . . . . . . . . . . . . . . . . . . . . . 20
     ....

11. Security Considerations  . . . . . . . . . . . . . . . . . . . 21
     7.1.  Authentication and Authorization . . . . . . . . . . . . 21
     7.2.  Encrypted Data . . . . . . . . . . . . . . . . . . . . . 22
     7.3.  Protection against Gaming  . . . . . . . . . . . . . . . 22

---

From: "C. Alexandre Tian" <[email-masked]>
Date: Mon, 2 Jul 2012 14:27:37 +0800 (CST)

Regards Q2:
 
To me, all subsections in -arch "4.  Architectural Principles" can be considered
as candidates, which can be carefully re-organized and moved to -req.
 
For example, similar content in "4.1.  Decoupled Control/Metadata and Data Planes
- Resource Scheduling Algorithms" has already appeared in problem statement doc
"3.3 Ineffective integration of P2P applications". If we want to repeat/emphasize
this information, the best place is -req.

At 2012-06-30 18:40:26,"Y. Richard Yang" <[email-masked]> wrote: >Dear DECADE participants, > >Both -req and -arch documents have received extensive, excellent  >reviews. In the process of revising the documents and looking at the two  >documents, we have some questions to seek input from the participants: > >Q1. Both -req and -arch define some terms, including DECADE-compatible  >client, DECADE-compatible server, etc. This appears to be redundant.  >Should such terms be defined in -req or -arch? The current thinking is  >-req. Any comments? > >Q2. There are multiple paragraphs defining requirements in -arch, e.g.,  >Sec. 4.4 on naming. Should these be moved to -req? A more general  >question is on the dependency between -req and -arch. On one hand, it is  >ideal to define -req first without a particular architecture to allow  >competing/alternative architectures satisfying the requirements. On the  >other hand, it appears that the -arch document contains content, in  >parti
 cular, systems/function components, that can be helpful to better  >understand the -req document. The current thinking is still to make -req  >independent of -arch, and move some general requirements in -arch to  >-req. Any comments? > >Q3: There are some good comments on reorganizing the -req documents.  >Below is a new structure (second level section #'s and page numbers are  >from the current -req so that others can see how we re-organize the  >document). Any comments/feedback before we start the revision will be  >greatly appreciated. > >Also, anyone who can help with direct revisions of the -req document  >will be more than welcome! > >Thanks. > >--  >Richard > >1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  5 >2.  Terminology and Concepts . . . . . . . . . . . . . . . . . . .  6 >3.  System/Functional Components >4.  Requirements Structure . . . . . . . . . . . . . . . . . . . .  7 > >5.  Data Object Requirements >     4.5.1.  Data Naming . . . . .
  . . . . . . . . . . . . . . . . .14 >             Move part of Arch 4.4 to -req >     5.1.  --Immutable Data . . . . . . . . . . . . . . . . . . . . 18 >     4.4.2.  System Data Object Attributes . . . . . . . . . . . . .13 >         4.2.1.  Data Object Size . . . . . . . . . . . . . . . . . 11 >         4.4.3.  Time-to-live for Written Data Objects  . . . . . . 13 >     4.4.4.  Application-defined Properties and Metadata  . . . . . 13 > >6.  Access to Server Requirements >     4.3.1.  Read/Write Own Storage  . . . . . . . . . . . . . . .. 11 >     4.3.2.  Read/Write by Remote Clients . . . . . . . . . . . . . 11 >     4.4.5.  Offline Access  . . . . . . . . . . . . . . . .  . . . 14 >     4.1.1.1.  NATs and Firewalls TRaversal . . .. . .  . . . . . .  8 >       Connections to Clients . . . . . . . . . . . . . . . . . . .  8 >       6.1.1.  Support for Clients Behind NATs and Firewalls  . . . 21 >     4.3.3.  Negotiable Data Transport Protocol . . . . . . . . . . 12 >     5
 .2.  Explicit Deletion of Data  . . . . . . . . . . . . . . . 19 >     5.3.  Concurrent Write (Multiple writing). . . . . . . . . . . 19 >     5.4.  Concurrent Read (Multiple reading) . . . . . . . . . . . 19 >     5.5.  Read While Write . . . . . . . .   . . . . . . . . . . . 19 >     5.6.  Partial Write (Writing model) . . . . . . . . . . . . . 20 >     4.1.2.1.  Secure Transport . . . . . . . . . . . . . . . . . .  8 > >7.  Resource Control Requirements. . . . . . . . . . . . . . . . . 15 >     4.6.1.  Multiple Applications  . . . . . . . . . . . . . . . . 15 >     4.6.2.  Per-Remote-Client, Per-Data Control  . . . . . . . . . 15 >     4.6.3.  Resource Control Set . . . . . . . . . . . . . . . . . 16 >     4.6.4.  Server Involvement . . . . . . . . . . . . . . . . . . 16 > >8.  Authorization Requirements >     4.7.1.  Per-Remote-Client, Per-Data Read Access  . . . . . . . 16 >     4.7.2.  Per-User Write Access  . . . . . . . . . . . . . . . . 17 >     4.7.3.  Default Acce
 ss Permissions . . . . . . . . . . . . . . 17 >     4.7.4.  Authorization Checks . . . . . . . . . . . . . . . . . 17 >     4.7.5.  Cryptographic Credentials  . . . . . . . . . . . . . . 17 >     4.7.6.  Server Involvement . . . . . . . . . . . . . . . . . . 18 >     4.7.7.  Protocol Reuse . . . . . . . . . . . . . . . . . . . . 18 > >9.  Error and Failure Handling Requirements. . . . . . . . . . . .  9 >     4.1.3.2.  Insufficient Resources . . . . . . . . . . . . . . .  9 >         4.1.3.1.  Overload Condition . . . . . . . . . . . . . . .  9 >     4.1.3.3.  Unavailable and Deleted Data . . . . . . . . . . . . 10 >     4.1.3.4.  Insufficient Permissions . . . . . . . . . . . . . . 10 >     4.1.3.5.  Redirection  . . . . . . . . . . . . . . . . . . . . 10 >     4.1.2.2.  Gaming Prevention  . . . . . . . . . . . . . . . . . .8 >     4.4.1.  Server reliability (Agnostic of reliability) . . . . . 12 > >10. Server Status Query Requirements . . . . . . . . . . . . . . . 18 >    
  5.7.  Storage Status . . . . . . . . . . . . . . . . . . . . . 20 >     .... > >11. Security Considerations  . . . . . . . . . . . . . . . . . . . 21 >     7.1.  Authentication and Authorization . . . . . . . . . . . . 21 >     7.2.  Encrypted Data . . . . . . . . . . . . . . . . . . . . . 22 >     7.3.  Protection against Gaming  . . . . . . . . . . . . . . . 22 > >

---

From: Haiwei Xue <[email-masked]>
Date: Mon, 2 Jul 2012 09:42:55 -0400

Hi Richard,

Here are some of my comments.

Thanks.

Haiwei

On Sat, Jun 30, 2012 at 6:40 AM, Y. Richard Yang <[email-masked]> wrote:

> Both -req and -arch documents have received extensive, excellent reviews.
> In the process of revising the documents and looking at the two documents,
> we have some questions to seek input from the participants:
>
> client, DECADE-compatible server, etc. This appears to be redundant. Should
> such terms be defined in -req or -arch? The current thinking is -req. Any
> comments?
>

[Haiwei] 1. I think the current definition is good if we add some
explanation in -arch, e.g. we can add a sentence in Sec. 2 (-arch) at the
begin of definitions: some terms are already defined in '-req'.

[Haiwei] 2. If you think it is still redundant, then you can remove the
definition from -arch, and then reference the terms from -req.
                I agree with you to leave the def. in -req, because if let
me choose the documents which I want to read first, I will choose -req. So
-req should keep dependent.

BTW, I like the style of terminology definition in -arch, because it is
very clear in the "Table of Contents".

> Sec. 4.4 on naming. Should these be moved to -req? A more general question
> is on the dependency between -req and -arch. On one hand, it is ideal to
> define -req first without a particular architecture to allow
> independent of -arch, and move some general requirements in -arch to -req.
> Any comments?
>

[Haiwei] Point 1: If we leave the  requirements on naming in -arch, would
that broke the dependency of -req? I don't think so.

[Haiwei] Point 2: In the -req you proposed the naming question, they are
high level consideration, and in -arch based on
the data-object oriented architecture and the design details you proposed
some new requirements on naming. This sounds reasonable. And if you move
those  requirements  from -arch to -req, it will let the readers confused:
which documents should they read first.

So I think the requirements on details should keep int -arch.

>
> Q3: There are some good comments on reorganizing the -req documents. Below
> is a new structure (second level section #'s and page numbers are from the
> current -req so that others can see how we re-organize the document). Any
> comments/feedback before we start the revision will be greatly appreciated.
>

Also, anyone who can help with direct revisions of the -req document will
> be more than welcome!

---

From: "Y. Richard Yang" <[email-masked]>
Date: Mon, 2 Jul 2012 22:11:17 +0800

Hi Peng,

Thanks a lot for the feedback! Please see below.

On Monday, July 2, 2012, Zhang Peng wrote:

> **
> First, regarding Q1, I think there are indeed some overlaps on term
> definitions in -req and -arch documents. In addition, they are even not
> consistent. I think we can just put them in the -req document, and give a
> pointer in -arch document.
>

Good to see your point. I am wondering what the other authors think?

> Regarding Q2, I agree that in -arch, there are some paragraphs that also
> do the job of define requirements, and may better be put in -req. For
> naming, the reqiin -requirements given in -req are rather simplistic, and
> we can move the move the detailed requirements to -req. But for "immutable
> object", I think it is more like a design option for the architecture (so
> that the design would be much simplified), rather than a system
> requirements for DECADE to function.  Thus, I suggest we don't move the
> immutable data objects in -arch to -req, and we can even delete them in
> -req. Finally, I would say that there is no clear cut for requirement and
> architecture, since when we specify the requirements, we have some design
> rationale in mind; and when we design the architecture, we would find more
> clear requirements for the specific design. It is not easy to fully
> decouple them.
>

I tend to agree on your proposed split: naming reqs in -arch to -req;
immutable keeps in -arch. It appears that you think immutable PBS should
not at all be in -req.  Introducing immutable objects was an assumption to
simplify potential design. So it is not a req but a design point. I tend to
agree but want to see if others have a different opinion.

Regarding Q3, I think the new outline of the -req document seems better,
> but I think it is better to organize it into three parts:
> access/authorization, resource control, data storage/transfer, failure
> handling/gaming prevention. The rationale is that the first part is about
> how client access the server, the second part corresponds to DRP, the third
> part corresponds to SDT, and the last two parts are about
> reliability/security. The following is the outline in my mind. I would like
> to discuss with you about the organization further. And I would like to
> help revise the -req document.  Thanks.
>
>

Sounds interesting. Could you please give a more detailed list on the
outline?

Thanks a lot!

Richard

> ------------------------------
> Zhang Peng
>
>  *From:* Y. Richard Yang <javascript:_e({}, 'cvml', [email-masked]');>
> *Date:* 2012-06-30 06:40
> *To:* [email-masked] <javascript:_e({}, 'cvml', [email-masked]');>
> *Subject:* [decade] -req and -arch documents
>      4.7.2.  Per-User Write Access  . . . . . . . . . . . . . .
>

---

From: "Y. Richard Yang" <[email-masked]>
Date: Mon, 2 Jul 2012 22:16:21 +0800

Ho Alexander,

> as candidates, which can be carefully re-organized and moved to -req.

I think your proposal is a bit extreme. I agree with the example topics
blow, but they are not
fully developed, e.g., what properties of resource sharing to satisfy.
Also, for example, the
preceding discussion was to to leave  immutable objects to arch. What do
you think?

Thanks!

Richard

>
>
>
>
> At 2012-06-30 18:40:26,"Y. Richard Yang" <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>> wrote:
> >Dear DECADE participants, >
> >documents, we have some questions to seek input from the participants: >
> >-req. Any comments? >
> >-req. Any comments? >
> >greatly appreciated. >
> >will be more than welcome! > >Thanks. > >--  >Richard >
> >4.  Requirements Structure . . . . . . . . . . . . . . . . . . . .  7 >
> >     4.4.4.  Application-defined Properties and Metadata  . . . . . 13 >
> >     4.1.2.1.  Secure Transport . . . . . . . . . . . . . . . . . .  8 >
> >     4.6.4.  Server Involvement . . . . . . . . . . . . . . . . . . 16 >
> >     4.7.7.  Protocol Reuse . . . . . . . . . . . . . . . . . . . . 18 >
> >     4.4.1.  Server reliability (Agnostic of reliability) . . . . . 12 >
> >     .... >
> >     7.3.  Protection against Gaming  . . . . . . . . . . . . . . . 22 > >
>

---

From: "Y. Richard Yang" <[email-masked]>
Date: Mon, 2 Jul 2012 22:21:57 +0800

On Monday, July 2, 2012, Haiwei Xue wrote:

> On Sat, Jun 30, 2012 at 6:40 AM, Y. Richard Yang <[email-masked]<javascript:_e({}, 'cvml', [email-masked]');>
> > wrote:
>
> Ok.

> very clear in the "Table of Contents".

Thanks for the feedback. We will do so for -req.

> So what you want is some general req in -req, and specific reqs in -arch.
I tend to agree with this split, but will need a detailed look to see this
such a split is possible. Any volunteer to propose a split related on
naming?

Thanks!

Richard

---

From: Songhaibin <[email-masked]>
Date: Fri, 6 Jul 2012 09:12:13 +0000

Hi Peng,
I tend to agree with all your proposals.
Thanks,
-Haibin
From: [email-masked] [mailto:[email-masked]] On Behalf Of Zhang Peng
Sent: Monday, July 02, 2012 2:02 PM
To: yry; [email-masked]
Subject: Re: [decade] -req and -arch documents

Hi Richard,

Best,

Peng.

________________________________
Zhang Peng

From: Y. Richard Yang<mailto:[email-masked]>
Date: 2012-06-30 06:40
To: [email-masked]<mailto:[email-masked]>
Subject: [decade] -req and -arch documents
Dear DECADE participants,

Thanks.

--
Richard