Subject: [Idnet] IDN dedicated session call for case

From: yanshen <[email-masked]>
Date: Wed, 2 Aug 2017 10:12:24 +0000

Dear all,

Since we plan to organize a dedicated session in NMRG, IETF100, for applying AI into network management (NM), I’d try to list some Use Cases and propose a roadmap and ToC before Nov. 

These might be rough. You are welcome to refine them and propose your focused use cases or ideas.

Use case 1: Traffic Prediction
	Description: Collect the history traffic data and external data which may influence the traffic. Predict the traffic in short/long/specific term. Avoid the congestion or risk in previously.
	Process: 1. Data collection (e.g. traffic sample of physical/logical port ); 2. Training Model; 3. Real-time data capture and input; 4. Predication output; 5. Fix error and go back to 3.
	Data Format: 	Time : [Start, End, Unit, Number of Value, Sampling Period] 
				Position: [Device ID, Port ID]
				Direction: IN / OUT
				Route : [R1, R2, ..., RN]  (might be useful for some scenarios)
				Service : [Service ID, Priority, ...]  (Not clear how to use it but seems useful)
				Traffic: [T0, T1, T2, ..., TN]
	Message : 	Request: ask for the data 
				Reply: Data
				Notice: For notification or others 
				Policy: Control policy
				
Use case 2: QoS Management
	Description: Use multiple paths to distribute the traffic flows. Adjust the percentages. Avoid congestion and ensure QoS. 
	Process: 1. Data capture (e.g. traffic sample of physical/logical port ); 2. Training Model; 3. Real-time data capture and input; 4. Output percentages; 5. Fix error and go back to 3.
	Data Format: 	Time : [Timestamp, Value type (Delay/Packet Loss/...), Unit, Number of Value, Sampling Period] 
				Position: [Link ID, Device ID]
				Value: [V0, V1, V2, ..., VN]
	Message : 	Request: ask for the data 
				Reply: Data
				Notice: For notification or others 
				Policy: Control policy

Use case N: Waiting for your Ideas

Also I suggest a roadmap before Nov if possible. 

### Roadmap ###
Aug. : Collecting the use cases (related with NM). Rough thoughts and requirements
Sep. : Refining the cases and abstract the common elements
Oct. : Deeply analysis. Especially on Data Format, control flow, or other key points
Nov.: F2F discussions on IETF100
### Roadmap End ###

A rough ToC is listed in following. We may take it as a scope before Nov. Hope that the content could become the draft of draft. 

###Table of Content###
1. Gap and Requirement Analysis
	1.1 Network Management requirement 
	1.2 TBD
2. Use Cases
	2.1 Traffic Prediction
	2.2 QoS Management
	3.3 TBD
3. Data Focus
	3.1 Data attribute
	3.2 Data format
	3.3 TBD
4. Aims
	4.1 Benchmarking Framework
	4.2 TBD
###ToC End###

Yansen

---

From: "Ciavaglia, Laurent (Nokia - FR/Nozay)" <[email-masked]>
Date: Wed, 2 Aug 2017 12:35:50 +0000

Hello,

Hint: it would be nice to contact the NMRG chairs regarding the dedicated session you are mentioning.
Also, participants of the NMRG could/should be interested/aware and could eventually contribute to the effort.

Thanks, Laurent (as NMRG co-chair).
---

-----Original Message-----
From: IDNET [mailto:[email-masked]] On Behalf Of yanshen
Sent: Wednesday, August 02, 2017 12:12 PM
To: [email-masked]
Subject: [Idnet] IDN dedicated session call for case

Dear all,

				

### Roadmap ###
Aug. : Collecting the use cases (related with NM). Rough thoughts and requirements Sep. : Refining the cases and abstract the common elements Oct. : Deeply analysis. Especially on Data Format, control flow, or other key points
Nov.: F2F discussions on IETF100
### Roadmap End ###

###Table of Content###

Yansen

_______________________________________________
IDNET mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idnet

---

From: Aydin Ulas <[email-masked]>
Date: Wed, 2 Aug 2017 16:22:10 +0300

Hi,
Here is my suggestion.

Use case N+1: Application (and/or DDoS) detection
        Description: Detect the application (or attack) from network
packets (HTTPS or plain) Collect the history traffic data and identify a
service or attack (ex: Skype, Viber, DDoS attack etc.)
        Process: 1. Data collection (e.g. traffic sample of application to
be detected or attack dump); 2. Training Model (includes data correlation
from different parts of the network); 3. Real-time data capture and input;
4. Output prediction; 5. Online or periodically updating model parameters
according to changes in the application (or attack style)
        Data Format:   Time: [Timestamp, ...]
                                Data Packet: [Packet Header, optional:
Packet Content or the first few bytes: src/dst ip, src/dst port, frame
length/number, ...]
                                Direction: IN / OUT
                                Route : [R1, R2, ..., RN]
                                Traffic: [T0, T1, T2, ..., TN]

        Message :       Request: ask for the data
                                Reply: Identification of application
                                Notice: For notification or others
                                Policy: Control policy

Best regards,
Aydin

On Wed, Aug 2, 2017 at 3:35 PM, Ciavaglia, Laurent (Nokia - FR/Nozay) <
[email-masked]> wrote:

> Since we plan to organize a dedicated session in NMRG, IETF100, for
> applying AI into network management (NM), I’d try to list some Use Cases
> and propose a roadmap and ToC before Nov.
>
>         Description: Collect the history traffic data and external data
> which may influence the traffic. Predict the traffic in short/long/specific
> term. Avoid the congestion or risk in previously.
>         Process: 1. Data collection (e.g. traffic sample of
> physical/logical port ); 2. Training Model; 3. Real-time data capture and
> input; 4. Predication output; 5. Fix error and go back to 3.
>                                 Service : [Service ID, Priority, ...]
> (Not clear how to use it but seems useful)
>         Description: Use multiple paths to distribute the traffic flows.
> Adjust the percentages. Avoid congestion and ensure QoS.
>         Process: 1. Data capture (e.g. traffic sample of physical/logical
> port ); 2. Training Model; 3. Real-time data capture and input; 4. Output
> percentages; 5. Fix error and go back to 3.
>         Data Format:    Time : [Timestamp, Value type (Delay/Packet
> Loss/...), Unit, Number of Value, Sampling Period]
> Aug. : Collecting the use cases (related with NM). Rough thoughts and
> requirements Sep. : Refining the cases and abstract the common elements
> A rough ToC is listed in following. We may take it as a scope before Nov.
> Hope that the content could become the draft of draft.
>
> ###Table of Content###

---

From: yanshen <[email-masked]>
Date: Wed, 2 Aug 2017 16:20:51 +0000

Hi Aydin,

Got it. Thanks for the first support.

This use case is based on the support of efficient measurement and recognition. Because the attack often happens within a short time. Some of the applications are similar. So that the processing time must be short enough. Therefore, a new use case in regard to measurement may be significant.

And we may have a deeper analysis in next weeks.

Regards,

Yansen

From: [email-masked] [mailto:[email-masked]] On Behalf Of Aydin Ulas
Sent: Wednesday, August 2, 2017 09:22 PM
To: yanshen <[email-masked]>; [email-masked]
Subject: Re: [Idnet] IDN dedicated session call for case

Hi,
Here is my suggestion.

Use case N+1: Application (and/or DDoS) detection
        Description: Detect the application (or attack) from network packets (HTTPS or plain) Collect the history traffic data and identify a service or attack (ex: Skype, Viber, DDoS attack etc.)
        Process: 1. Data collection (e.g. traffic sample of application to be detected or attack dump); 2. Training Model (includes data correlation from different parts of the network); 3. Real-time data capture and input; 4. Output prediction; 5. Online or periodically updating model parameters according to changes in the application (or attack style)
        Data Format:   Time: [Timestamp, ...]
                                Data Packet: [Packet Header, optional: Packet Content or the first few bytes: src/dst ip, src/dst port, frame length/number, ...]
                                Direction: IN / OUT
                                Route : [R1, R2, ..., RN]
                                Traffic: [T0, T1, T2, ..., TN]
        Message :       Request: ask for the data
                                Reply: Identification of application
                                Notice: For notification or others
                                Policy: Control policy

Best regards,
Aydin

On Wed, Aug 2, 2017 at 3:35 PM, Ciavaglia, Laurent (Nokia - FR/Nozay) <[email-masked]<mailto:[email-masked]>> wrote:
Hello,

-----Original Message-----
From: IDNET [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of yanshen
Sent: Wednesday, August 02, 2017 12:12 PM
To: [email-masked]<mailto:[email-masked]>
Subject: [Idnet] IDN dedicated session call for case

Dear all,

### Roadmap ###

###Table of Content###

Yansen

_______________________________________________
IDNET mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idnet
_______________________________________________
IDNET mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idnet

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Thu, 3 Aug 2017 09:35:18 +0900

On Wed, Aug 02, 2017 at 12:35:50PM +0000, Ciavaglia, Laurent (Nokia - FR/Nozay) wrote:
> Also, participants of the NMRG could/should be interested/aware and
> could eventually contribute to the effort.
> 

Dear Laurent,

I think there is a confusion regarding the "dedicated session". It is
not scheduled or planned just it is a "wish". First we were discussing
to do some work, possibly to write a document (as mentioned), and then
try to find a session to fit it, which would potentially be NMRG. There
is no content so we have still not asked for any time-slot but we would
probably ask for it. Would you please intercede in our name to ask the
chairs (yourself included) if NMRG can host this work?

Regards,
Pedro

-- 
Pedro Martinez-Julia
Network Science and Convergence Device Technology Laboratory
Network System Research Institute
National Institute of Information and Communications Technology (NICT)
4-2-1, Nukui-Kitamachi, Koganei, Tokyo 184-8795, Japan
Email: [email-masked]
---------------------------------------------------------
*** Entia non sunt multiplicanda praeter necessitatem ***

---

From: Sheng Jiang <[email-masked]>
Date: Tue, 8 Aug 2017 01:23:03 +0000

Hi IDneter, especially the contributors, 

(copied to NMRG mail list for coordination between two groups)

It is our intention to organize a dedicated IDNet session for applying AI into network management in NMRG in IETF100, Singapore. It will depend on whether we could converge and organize good content/presentations. So far, we are actually in the stage of calling interests, use cases and contributors. We need to keep discussing and converge to specific use cases soon so that we could form a good proposal for the potential session.

Shen's work has given a constructive thought that may lead to converged content. It is welcome to everybody contribute more valuable user cases. After refining and abstracting the general element from various use cases, the potential standard points will become more and more clear. And maybe, after several round of discussion and convergence, we could understand the essential of applying AI into network management more so that we could have a good foundation to form an IETF working group for standardization work.

Regards,

Sheng

> Sent: Wednesday, August 02, 2017 6:12 PM
> To: [email-masked]
> 	Description: Collect the history traffic data and external data which
> may influence the traffic. Predict the traffic in short/long/specific term.
> Avoid the congestion or risk in previously.
> 	Process: 1. Data collection (e.g. traffic sample of physical/logical port );
> 2. Training Model; 3. Real-time data capture and input; 4. Predication
> output; 5. Fix error and go back to 3.
> 	Description: Use multiple paths to distribute the traffic flows. Adjust
> the percentages. Avoid congestion and ensure QoS.
> 	Process: 1. Data capture (e.g. traffic sample of physical/logical port ); 2.
> Training Model; 3. Real-time data capture and input; 4. Output

---

From: Albert Cabellos <[email-masked]>
Date: Tue, 8 Aug 2017 13:52:23 +0900

Hi all

Here´s another use-case:

Use case N+2: QoE
        Description: Collect low-level metrics (SNR, latency, jitter,
losses, etc) and measure QoE. Then use ML to understand what is the
relation between satisfactory QoE and the low-level metrics. As an example
learn that when delay>N then QoE is degraded, but when M<delay<N then QoE
is satisfactory for the customers (please note that QoE cannot be measured
directly over your network). This is useful to understand how the network
must be operated to provide satisfactory QoE.
        Process: 1. Low-level data collection and QoE measurement ; 2.
Training Model (input low-level metrics, output QoE); 3. Real-time data
capture and input; 4. Predict QoE; 5. Operate network to meet target QoE
requirement, go to 3.
        Data Format:    Time : [Start, End, Unit, Number of Value, Sampling
Period]
                                Position: [Device ID, Port ID]
                                Direction: IN / OUT
                                Low-level metric : SNR, Delay, Jitter,
queue-size, etc

Kind regards

Albert

On Wed, Aug 2, 2017 at 7:12 PM, yanshen <[email-masked]> wrote:

> requirements

---

From: Özgü Alay <[email-masked]>
Date: Tue, 8 Aug 2017 08:01:32 +0200

Dear Albert,
We are interested in this use case and will support the activities in this
area.
Best Regards,
Özgü

---

From: yanshen <[email-masked]>
Date: Tue, 8 Aug 2017 10:02:06 +0000

Dear Albert,

At least two supporters you have : )

I think that the QoS and QoE is just similar with my opinion mentioned before that is the data can be divided into subjective and objective.  This will be related with the data format and the way of obtaining. And your case build up a bridge between the subjective and objective.

Yansen

From: Özgü Alay [mailto:[email-masked]]
Sent: Tuesday, August 08, 2017 2:02 PM
To: Albert Cabellos <[email-masked]>
Cc: yanshen <[email-masked]>; [email-masked]
Subject: Re: [Idnet] IDN dedicated session call for case

Dear Albert,
We are interested in this use case and will support the activities in this area.
Best Regards,
Özgü

On 8 August 2017 at 06:52, Albert Cabellos <[email-masked]<mailto:[email-masked]>> wrote:
Hi all

Here´s another use-case:

Use case N+2: QoE
        Description: Collect low-level metrics (SNR, latency, jitter, losses, etc) and measure QoE. Then use ML to understand what is the relation between satisfactory QoE and the low-level metrics. As an example learn that when delay>N then QoE is degraded, but when M<delay<N then QoE is satisfactory for the customers (please note that QoE cannot be measured directly over your network). This is useful to understand how the network must be operated to provide satisfactory QoE.
        Process: 1. Low-level data collection and QoE measurement ; 2. Training Model (input low-level metrics, output QoE); 3. Real-time data capture and input; 4. Predict QoE; 5. Operate network to meet target QoE requirement, go to 3.
        Data Format:    Time : [Start, End, Unit, Number of Value, Sampling Period]
                                Position: [Device ID, Port ID]
                                Direction: IN / OUT
                                Low-level metric : SNR, Delay, Jitter, queue-size, etc

Kind regards

Albert

On Wed, Aug 2, 2017 at 7:12 PM, yanshen <[email-masked]<mailto:[email-masked]>> wrote:
Dear all,

### Roadmap ###

###Table of Content###

Yansen

---

From: Jérôme François <[email-masked]>
Date: Tue, 8 Aug 2017 16:40:08 +0200

Hi,

I support this use case and suggest to also include DDoS mitigation as
part of the process, e.g. re-routing to DPI middleboxes / blocking
traffic. You can thus evaluate the result of your mitigation decision
and readjust your model accordingly with reinforcement learning.
We have some on-going work in this area.

Best regards,
jerome

Le 02/08/2017 à 15:22, Aydin Ulas a écrit :
> packets (HTTPS or plain) Collect the history traffic data and identify
> a service or attack (ex: Skype, Viber, DDoS attack etc.)
> application to be detected or attack dump); 2. Training Model
> (includes data correlation from different parts of the network); 3.
> Real-time data capture and input; 4. Output prediction; 5. Online or
> periodically updating model parameters according to changes in the
> application (or attack style) 
> On Wed, Aug 2, 2017 at 3:35 PM, Ciavaglia, Laurent (Nokia - FR/Nozay)
> <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hello,
>
>     Also, participants of the NMRG could/should be interested/aware
>     and could eventually contribute to the effort.
>
>     From: IDNET [mailto:[email-masked]
>     <mailto:[email-masked]>] On Behalf Of yanshen
>     To: [email-masked] <mailto:[email-masked]>
>     Since we plan to organize a dedicated session in NMRG, IETF100,
>     for applying AI into network management (NM), I’d try to list some
>     Use Cases and propose a roadmap and ToC before Nov.
>
>             Description: Collect the history traffic data and external
>     data which may influence the traffic. Predict the traffic in
>     short/long/specific term. Avoid the congestion or risk in previously.
>     physical/logical port ); 2. Training Model; 3. Real-time data
>     capture and input; 4. Predication output; 5. Fix error and go back
>     to 3.
>                                     Service : [Service ID, Priority,
>     ...]  (Not clear how to use it but seems useful)
>             Description: Use multiple paths to distribute the traffic
>     flows. Adjust the percentages. Avoid congestion and ensure QoS.
>             Process: 1. Data capture (e.g. traffic sample of
>     physical/logical port ); 2. Training Model; 3. Real-time data
>     capture and input; 4. Output percentages; 5. Fix error and go back
>     to 3.
>             Data Format:    Time : [Timestamp, Value type
>     (Delay/Packet Loss/...), Unit, Number of Value, Sampling Period]
>     Aug. : Collecting the use cases (related with NM). Rough thoughts
>     and requirements Sep. : Refining the cases and abstract the common
>     elements Oct. : Deeply analysis. Especially on Data Format,
>     control flow, or other key points
>     A rough ToC is listed in following. We may take it as a scope
>     before Nov. Hope that the content could become the draft of draft.
>
>     ###Table of Content###
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/idnet>
>     [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/idnet>
>
>
>
>

---

From: Jérôme François <[email-masked]>
Date: Tue, 8 Aug 2017 16:49:32 +0200

Hi all,

Here is another use case about traffic classification.

Use case N+3: (encrypted) traffic classification

    Description: collect flow-level traffic metrics such as protocol
information but also meta metrics such as distribution of packet sizes,
inter-arrival times... Then use such information to label the trafic
with the underlying application assuming that the granularity of
classification may vary (type of application, exact application name,
version...)
    Process: 1. collect packet information 2. flow reassembly (using
directly flow format such as IPFIX might be possible but depends on the
type of traffic, e.g. extracting the TLS application data is useful for
encrypted traffic) 3. Collect application specific information (useful
when targeting a single type of application) = out of network
information 4. train the model 5. Online or offline testing 4. Apply
application level policies.
    Data Format:    Time : [Start, End, Unit, Number of Value, Sampling
Period]
                                Position: [Device ID, Port ID]
                                Direction: IN / OUT
                                Flow level metric: packet size
distributions, number of packets, inter-arrival time distribution,
                                 (+ application specific knowledge :
payload parsing)

Best regards,
jerome
 
Le 08/08/2017 à 06:52, Albert Cabellos a écrit :
> relation between satisfactory QoE and the low-level metrics. As an
> example learn that when delay>N then QoE is degraded, but when
> M<delay<N then QoE is satisfactory for the customers (please note that
> QoE cannot be measured directly over your network). This is useful to
> understand how the network must be operated to provide satisfactory QoE.
> Training Model (input low-level metrics, output QoE); 3. Real-time
> data capture and input; 4. Predict QoE; 5. Operate network to meet
> target QoE requirement, go to 3.
>     and requirements

---

From: "Diego R. Lopez" <[email-masked]>
Date: Tue, 8 Aug 2017 14:56:58 +0000

Hi Jerome,

Agreed. This is a use case we are very much interested in, and actually working in it now. Just let me say we are trying to evaluate which are the significant features of the flow to perform a proper classification, depending on the flow nature (TLS, DTLS, QUIC, IPsec…), and that would define the concrete data to be exchanged or stored.

Be goode,

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D
http://people.tid.es/diego.lopez/

e-mail: [email-masked]
Tel:        +34 913 129 041
Mobile: +34 682 051 091
-----------------------------------

On 8/8/2017, 16:49 , "IDNET on behalf of Jérôme François" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

Hi all,

    Description: collect flow-level traffic metrics such as protocol information but also meta metrics such as distribution of packet sizes, inter-arrival times... Then use such information to label the trafic with the underlying application assuming that the granularity of classification may vary (type of application, exact application name, version...)
    Process: 1. collect packet information 2. flow reassembly (using directly flow format such as IPFIX might be possible but depends on the type of traffic, e.g. extracting the TLS application data is useful for encrypted traffic) 3. Collect application specific information (useful when targeting a single type of application) = out of network information 4. train the model 5. Online or offline testing 4. Apply application level policies.
    Data Format:    Time : [Start, End, Unit, Number of Value, Sampling Period]
                                Position: [Device ID, Port ID]
                                Direction: IN / OUT
                                Flow level metric: packet size distributions, number of packets, inter-arrival time distribution,
                                 (+ application specific knowledge : payload parsing)

Best regards,
jerome

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

### Roadmap ###

###Table of Content###

Yansen

IDNET mailing list

________________________________

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: Jérôme François <[email-masked]>
Date: Tue, 8 Aug 2017 16:59:19 +0200

100% agree with you. I was far from being exhaustive as traffic features
may depend on types of traffic (kin of sub use cases)

jerome
Le 08/08/2017 à 16:56, Diego R. Lopez a écrit :
>
> Hi Jerome,
>
>  
>
> Agreed. This is a use case we are very much interested in, and
> actually working in it now. Just let me say we are trying to evaluate
> which are the significant features of the flow to perform a proper
> classification, depending on the flow nature (TLS, DTLS, QUIC,
> IPsec…), and that would define the concrete data to be exchanged or
> stored.
>
>  
>
> Be goode,
>
>  
>
> --
>
> On 8/8/2017, 16:49 , "IDNET on behalf of Jérôme François"
> <[email-masked] <mailto:[email-masked]> on behalf of
> [email-masked] <mailto:[email-masked]>> wrote:
>
>  
>
> Hi all,
>
> information but also meta metrics such as distribution of packet
> sizes, inter-arrival times... Then use such information to label the
> trafic with the underlying application assuming that the granularity
> of classification may vary (type of application, exact application
> name, version...)
> directly flow format such as IPFIX might be possible but depends on
> the type of traffic, e.g. extracting the TLS application data is
> useful for encrypted traffic) 3. Collect application specific
> information (useful when targeting a single type of application) = out
> of network information 4. train the model 5. Online or offline testing
> 4. Apply application level policies.
>             Description: Collect low-level metrics (SNR, latency,
>     jitter, losses, etc) and measure QoE. Then use ML to understand
>     what is the relation between satisfactory QoE and the low-level
>     metrics. As an example learn that when delay>N then QoE is
>     degraded, but when M<delay<N then QoE is satisfactory for the
>     customers (please note that QoE cannot be measured directly over
>     your network). This is useful to understand how the network must
>     be operated to provide satisfactory QoE.
>             Process: 1. Low-level data collection and QoE measurement
>     ; 2. Training Model (input low-level metrics, output QoE); 3.
>     Real-time data capture and input; 4. Predict QoE; 5. Operate
>     network to meet target QoE requirement, go to 3.
>         Since we plan to organize a dedicated session in NMRG,
>         IETF100, for applying AI into network management (NM), I’d try
>         to list some Use Cases and propose a roadmap and ToC before Nov.
>
>                 Description: Collect the history traffic data and
>         external data which may influence the traffic. Predict the
>         traffic in short/long/specific term. Avoid the congestion or
>         risk in previously.
>         capture and input; 4. Predication output; 5. Fix error and go
>         back to 3.
>         (might be useful for some scenarios)
>                                         Service : [Service ID,
>         Priority, ...]  (Not clear how to use it but seems useful)
>                 Description: Use multiple paths to distribute the
>         traffic flows. Adjust the percentages. Avoid congestion and
>         ensure QoS.
>         capture and input; 4. Output percentages; 5. Fix error and go
>         back to 3.
> ------------------------------------------------------------------------
>
> Este mensaje y sus adjuntos se dirigen exclusivamente a su
> destinatario, puede contener información privilegiada o confidencial y
> es para uso exclusivo de la persona o entidad de destino. Si no es
> usted. el destinatario indicado, queda notificado de que la lectura,
> utilización, divulgación y/o copia sin autorización puede estar
> prohibida en virtud de la legislación vigente. Si ha recibido este
> mensaje por error, le rogamos que nos lo comunique inmediatamente por
> esta misma vía y proceda a su destrucción.
>
> The information contained in this transmission is privileged and
> confidential information intended only for the use of the individual
> or entity named above. If the reader of this message is not the
> intended recipient, you are hereby notified that any dissemination,
> distribution or copying of this communication is strictly prohibited.
> If you have received this transmission in error, do not read it.
> Please immediately reply to the sender that you have received this
> communication in error and then delete it.
>
> Esta mensagem e seus anexos se dirigem exclusivamente ao seu
> destinatário, pode conter informação privilegiada ou confidencial e é
> para uso exclusivo da pessoa ou entidade de destino. Se não é vossa
> senhoria o destinatário indicado, fica notificado de que a leitura,
> utilização, divulgação e/ou cópia sem autorização pode estar proibida
> em virtude da legislação vigente. Se recebeu esta mensagem por erro,
> rogamos-lhe que nos o comunique imediatamente por esta mesma via e
> proceda a sua destruição

---

From: "Diego R. Lopez" <[email-masked]>
Date: Tue, 8 Aug 2017 15:38:13 +0000

Hi Stenio,

Please do! Any reference would be extremely appreciated.

Be goode,

Dr Diego R. Lopez
Telefonica I+D

On 8/8/2017, 17:20 , "[email-masked] on behalf of Stenio Fernandes" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Jerome, Diego, et al,

    Those are excellent use cases. I have some published work on applied
    machine learning to computer networking problems, including flow-based
    traffic classification. I think another use case would be applying
    unsupervised learning techniques for anomaly detection. I can
    elaborate further on this.

    Stenio

    On Tue, Aug 8, 2017 at 10:59 AM, Jérôme François
    <[email-masked]> wrote:
    > 100% agree with you. I was far from being exhaustive as traffic features may
    > depend on types of traffic (kin of sub use cases)
    >
    > jerome
    >
    > Agreed. This is a use case we are very much interested in, and actually
    > working in it now. Just let me say we are trying to evaluate which are the
    > significant features of the flow to perform a proper classification,
    > depending on the flow nature (TLS, DTLS, QUIC, IPsec…), and that would
    > define the concrete data to be exchanged or stored.
    >
    >
    >
    > Be goode,
    >
    >
    >
    > --
    >
    > <[email-masked] on behalf of [email-masked]> wrote:
    >
    >
    >
    > Hi all,
    >
    > inter-arrival times... Then use such information to label the trafic with
    > the underlying application assuming that the granularity of classification
    > may vary (type of application, exact application name, version...)
    > directly flow format such as IPFIX might be possible but depends on the type
    > of traffic, e.g. extracting the TLS application data is useful for encrypted
    > traffic) 3. Collect application specific information (useful when targeting
    > a single type of application) = out of network information 4. train the
    > model 5. Online or offline testing 4. Apply application level policies.
    > losses, etc) and measure QoE. Then use ML to understand what is the relation
    > between satisfactory QoE and the low-level metrics. As an example learn that
    > when delay>N then QoE is degraded, but when M<delay<N then QoE is
    > satisfactory for the customers (please note that QoE cannot be measured
    > Since we plan to organize a dedicated session in NMRG, IETF100, for applying
    > AI into network management (NM), I’d try to list some Use Cases and propose
    > a roadmap and ToC before Nov.
    >
    >         Process: 1. Data collection (e.g. traffic sample of physical/logical
    > port ); 2. Training Model; 3. Real-time data capture and input; 4.
    > Predication output; 5. Fix error and go back to 3.
    > Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario,
    > puede contener información privilegiada o confidencial y es para uso
    > exclusivo de la persona o entidad de destino. Si no es usted. el
    > destinatario indicado, queda notificado de que la lectura, utilización,
    > divulgación y/o copia sin autorización puede estar prohibida en virtud de la
    > legislación vigente. Si ha recibido este mensaje por error, le rogamos que
    > nos lo comunique inmediatamente por esta misma vía y proceda a su
    > destrucción.
    >
    > confidential information intended only for the use of the individual or
    > entity named above. If the reader of this message is not the intended
    > recipient, you are hereby notified that any dissemination, distribution or
    > copying of this communication is strictly prohibited. If you have received
    > this transmission in error, do not read it. Please immediately reply to the
    > sender that you have received this communication in error and then delete
    > it.
    >
    > Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário,
    > pode conter informação privilegiada ou confidencial e é para uso exclusivo
    > da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário
    > indicado, fica notificado de que a leitura, utilização, divulgação e/ou
    > cópia sem autorização pode estar proibida em virtude da legislação vigente.
    > Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique
    > imediatamente por esta mesma via e proceda a sua destruição
    >
    >
    >

    --
    Prof. Stenio Fernandes
    CIn/UFPE
    http://www.steniofernandes.com

---

From: Stenio Fernandes <[email-masked]>
Date: Tue, 8 Aug 2017 11:20:37 -0400

Hi Jerome, Diego, et al,

Stenio

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Wed, 9 Aug 2017 09:00:40 +0900

Dear colleagues,

I want to declare my interest and add my rough/initial contribution
about the benchmarking framework.

A proper benchmarking framework comprises a set of reference procedures,
methods, and models that can (or better *must*) be followed to assess
the quality of an AI mechanism proposed to be applied to the network
management/control area. It is therefore essential to standardize such
framework but it is even more important for it to have demonstrated
widely accepted effectiveness (not just assumed), so it must be derived
or adapted from a framework with demonstrated usefulness that already
exists in other area with similar objectives and overall methodology.

Moreover, and much more specific to the IDNET topics, is the inclusion,
dependency, or just the general relation of a standard format enforced
to the data that is used (input) and produced (output) by the framework,
so a kind of "data market" can arise without requiring to transform the
data. The initial scope of input/output data would be the datasets, but
also the new knowledge items that are stated as a result of applying the
benchmarking procedures defined by the framework, which can be collected
together to build a database of benchmark results, or just contrasted
with other existing entries in the database to know the position of the
solution just evaluated. This increases the usefulness of IDNET.

That is all for now. Let's be in contact and go further in our effort to
get a proper document (and session) in the IETF 100. If it is achieved,
I want to have some time slot to present this idea (~ 10 min), so please
count on me to organize the schedule of the potential session. Thank you
very much.

Regards,
Pedro

---

From: Simone Ferlin <[email-masked]>
Date: Wed, 9 Aug 2017 11:28:40 +0900

Dear Jerome,

Very interesting use-case, +1 support. I have interest in such
activities for traffic classification, anomaly detection in particular
for encrypted traffic.

> On Wed, Aug 9, 2017 at 12:20 AM, Stenio Fernandes <[email-masked]> wrote:
>> Hi Jerome, Diego, et al,
>>

---

From: Albert Cabellos <[email-masked]>
Date: Wed, 9 Aug 2017 11:44:21 +0900

Hi Yanshen

In my view they are not the same use-case

- QoS use-case: As far as I understand here you aim to configure the
routing/switching infrastructure to achieve QoS for flows.

- QoE use-case: Here the idea is that QoE can not typically be measured
directly over the network, it has to be measured either over the
application (e.g., buffering in on-demand video services) or by polling the
human users. Thus, measuring QoE is expensive.

What can be measured are the low-level network metrics (delay, jitter, SNR,
etc). Then the question is, which is the relation between low-level metrics
and QoE metrics? The idea is to create a data-set containing low-level and
QoE metrics. Then and thanks to ML we model the relation between low-level
and QoE metrics, this means that we understand that when delay<N, jitter<M
and SNR>K QoE levels are satisfactory. With this, operators know what
low-level performance they need to target to offer good QoE.

A nice relation is that once you establish the target performance of
low-level metrics to achieve QoE, you can then use the 'QoS use-case' (or
similar) to operate the network.

Albert

On Tue, Aug 8, 2017 at 7:02 PM, yanshen <[email-masked]> wrote:

> I think that the QoS and QoE is just similar with my opinion mentioned
> before that is the data can be divided into subjective and objective.  This
> will be related with the data format and the way of obtaining. And your
> case build up a bridge between the subjective and objective.
>
>
>
> Yansen
>
>
>
>
>
> *From:* Özgü Alay [mailto:[email-masked]]
> *Sent:* Tuesday, August 08, 2017 2:02 PM
> *To:* Albert Cabellos <[email-masked]>
> *Cc:* yanshen <[email-masked]>; [email-masked]
> *Subject:* Re: [Idnet] IDN dedicated session call for case
>
>
>
> Dear Albert,
>

---

From: yanshen <[email-masked]>
Date: Wed, 9 Aug 2017 04:07:22 +0000

Hi Stenio,

Recently I also think about that use un/supervised ML method to implement the routing process. 

May I suggest you to have a simple refining and push it into mail list ? Just roughly refer to the format.

It will be helpful for both the discussion and your future work. 

Yansen

> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Stenio Fernandes
> Sent: Tuesday, August 08, 2017 11:21 PM
> To: Jérôme François <[email-masked]>
> Cc: Diego R. Lopez <[email-masked]>; Albert Cabellos
> <[email-masked]>; yanshen <[email-masked]>; [email-masked]
> Those are excellent use cases. I have some published work on applied machine
> learning to computer networking problems, including flow-based traffic
> classification. I think another use case would be applying unsupervised learning
> techniques for anomaly detection. I can elaborate further on this.
> 
> Stenio
> 
> On Tue, Aug 8, 2017 at 10:59 AM, Jérôme François <[email-masked]>
> wrote:
> > 100% agree with you. I was far from being exhaustive as traffic
> > features may depend on types of traffic (kin of sub use cases)
> >
> > jerome
> >
> > IPsec…), and that would define the concrete data to be exchanged or stored.
> >
> >
> >
> > Be goode,
> >
> >
> >
> > --
> >
> > of classification may vary (type of application, exact application name,
> version...)
> > useful for encrypted
> > traffic) 3. Collect application specific information (useful when
> > targeting a single type of application) = out of network information
> > 4. train the model 5. Online or offline testing 4. Apply application level policies.
> > payload
> > parsing)
> > applying AI into network management (NM), I’d try to list some Use
> > Cases and propose a roadmap and ToC before Nov.
> >
> > physical/logical port ); 2. Training Model; 3. Real-time data capture and input; 4.
> > physical/logical port ); 2. Training Model; 3. Real-time data capture
> > and input; 4. Output percentages; 5. Fix error and go back to 3.
> > requirements Sep. : Refining the cases and abstract the common
> > elements Oct. : Deeply analysis. Especially on Data Format, control
> > flow, or other key points
> > utilização, divulgação e/ou cópia sem autorização pode estar proibida em
> virtude da legislação vigente.

---

From: Yassine HADJADJ-AOUL <[email-masked]>
Date: Wed, 9 Aug 2017 09:04:02 +0200

Dear Albert, all,

Please find some comments below.

Yassine
_______________________________________
Yassine HADJADJ AOUL

Associate Professor, University of Rennes1
Dionysos Team / IRISA Lab
Room: F334
Phone: +33 (0) 2 99 84 71 35

> Le 9 août 2017 à 04:44, Albert Cabellos <[email-masked]> a écrit :
> 
> Hi Yanshen
> 
> - QoS use-case: As far as I understand here you aim to configure the routing/switching infrastructure to achieve QoS for flows.
> 
> - QoE use-case: Here the idea is that QoE can not typically be measured directly over the network, it has to be measured either over the application (e.g., buffering in on-demand video services) or by polling the human users. Thus, measuring QoE is expensive.

That is true that to have a very good accuracy it is generally necessary to have application-level measures (like the quantization parameter in a video, or the playout interruptions). However, there is many papers in the literature, some from our team, which demonstrate a good accuracy for voice and video (UDP or RTP) using only network level parameters.

We considered using Random Neural Networks in our developed tools … and we have ongoing work on that ...

Concerning the cost, it is indeed expensive generally. However, there are some techniques in the literature, which consist in doing the learning with objective techniques (like VQM) … this eliminate the need of users' panel … 

A former PhD student of our team had some contributions on that:
No-reference Quality of Experience estimation of H264/SVC stream
http://ieeexplore.ieee.org/document/6477778/ <http://ieeexplore.ieee.org/document/6477778/>

> 
> What can be measured are the low-level network metrics (delay, jitter, SNR, etc). Then the question is, which is the relation between low-level metrics and QoE metrics? The idea is to create a data-set containing low-level and QoE metrics. Then and thanks to ML we model the relation between low-level and QoE metrics, this means that we understand that when delay<N, jitter<M and SNR>K QoE levels are satisfactory. With this, operators know what low-level performance they need to target to offer good QoE. 

The relation between QoS and QoE is generally non linear, so I believe that in spite of finding a precise target we will have a function, which may predict the QoE … and thus, we may decline some actions accordingly.

Yassine

> 
> A nice relation is that once you establish the target performance of low-level metrics to achieve QoE, you can then use the 'QoS use-case' (or similar) to operate the network.
> 
> Albert 
> 
> On Tue, Aug 8, 2017 at 7:02 PM, yanshen <[email-masked] <mailto:[email-masked]>> wrote:
> Dear Albert,
> 
>  
> 
> From: Özgü Alay [mailto:[email-masked] <mailto:[email-masked]>] 
> To: Albert Cabellos <[email-masked] <mailto:[email-masked]>>
> Cc: yanshen <[email-masked] <mailto:[email-masked]>>; [email-masked] <mailto:[email-masked]>
> On 8 August 2017 at 06:52, Albert Cabellos <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Hi all
> 
>  
> 
> Here´s another use-case:
> 
>  
> 
> Use case N+2: QoE
> On Wed, Aug 2, 2017 at 7:12 PM, yanshen <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Dear all,
> 
> https://www.ietf.org/mailman/listinfo/idnet <https://www.ietf.org/mailman/listinfo/idnet>
>  
> 
> 
> https://www.ietf.org/mailman/listinfo/idnet <https://www.ietf.org/mailman/listinfo/idnet>
>  
> 
>

---

From: Jérôme François <[email-masked]>
Date: Wed, 9 Aug 2017 10:25:27 +0200

Hi,

There are many potential applications of trafic classification depending
on how you want to classify, i.e. accroding to what criteria, e.g. types
of applications, types of devices /OS (fingerprinting), anomalous/normal
traffic, users profles...

I presented in NMLRG  last year some wokr related to HTTPS traffic
classification (https://datatracker.ietf.org/doc/slides-95-nmlrg-7/).

jerome

---

From: yanshen <[email-masked]>
Date: Wed, 9 Aug 2017 09:37:02 +0000

Hi Albert,

My fault. I make it confused. I agree with you that they are different cases. And both of them are valuable.

Actually I means that the data involved in QoS use-case mostly can be captured, formatted and processed periodically and regularly. Or say most of them are objective (Pedro would like to call them controllable). As you mentioned that the some of the QoE data may be measured by polling the human users. I divide this into subjective class (or say uncontrollable). These two types of data may need different methods to process.

This consideration may be related to the data structure design and context design in future standardization work.

This is my two cents.

Regards,

Yansen

From: Albert Cabellos [mailto:[email-masked]]
Sent: Wednesday, August 09, 2017 10:44 AM
To: yanshen <[email-masked]>
Cc: [email-masked]; Özgü Alay <[email-masked]>
Subject: Re: [Idnet] IDN dedicated session call for case

Hi Yanshen

Albert

On Tue, Aug 8, 2017 at 7:02 PM, yanshen <[email-masked]<mailto:[email-masked]>> wrote:
Dear Albert,

Yansen

From: Özgü Alay [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, August 08, 2017 2:02 PM
To: Albert Cabellos <[email-masked]<mailto:[email-masked]>>
Cc: yanshen <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Idnet] IDN dedicated session call for case

Dear Albert,

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

### Roadmap ###

###Table of Content###

Yansen

---

From: 김민석 <[email-masked]>
Date: Thu, 10 Aug 2017 02:55:09 +0000

HI,

We have an use-case for this:

Use case n+4: Real time traffic classfication using deep learning
Description: continuously collect packet data, then applying learning process for traffic classification with generating application using deep learning models such as CNN (convolutional neural network) and RNN (recurrent neural network). Data-set to apply into the models are generated by propecessing with features of information from flow in packet data.

process: 1. collect packet data in real-time, 2. preprocessing data-set for deep learning models, 3. Training model using deep learning (CNN & RNN), 4. On-line data learning & classifying 5. Monitoring and analyzing traffic in the web

Data Format: Time : [Start, End, Unit, Number of Value, Sampling Period]
                            Position: [Device ID, Port ID]
                            Direction: IN / OUT
                            Flow level metric: packet & flow size, number of packet(RNN), payload parsing

 Message: Request: ask for the data
                          Reply: Data
                          Notice: For notification or others
                          Policy: Control policy

Regards,

Min-Suk Kim

Senior Researcher / Ph.D.

________________________________
보낸 사람 : "Jérôme François" <[email-masked]>
보낸 날짜 : 2017-08-08 23:49:47 ( +09:00 )
받는 사람 : Albert Cabellos <[email-masked]>, yanshen <[email-masked]>
참조 : [email-masked] <[email-masked]>
제목 : Re: [Idnet] IDN dedicated session call for case

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

style="font-size: 12px;">        Description: Collect low-level metrics (SNR, latency, jitter, losses, etc) and measure QoE. Then use ML to understand what is the relation between satisfactory QoE and the low-level metrics. As an example learn that when delay>N then QoE is degraded, but when M<delay<N then QoE is satisfactory for the customers (please note that QoE cannot be measured directly over your network). This is useful to understand how the network must be operated to provide satisfactory QoE.

style="font-size: 12px;">        Process: 1. Low-level data collection and QoE measurement ; 2. Training Model (input low-level metrics, output QoE); 3. Real-time data capture and input; 4. Predict QoE; 5. Operate network to meet target QoE requirement, go to 3.

style="font-size: 12px;">        Data Format:    Time : [Start, End, Unit, Number of Value, Sampling Period]

style="font-size: 12px;">                                Position: [Device ID, Port ID]

style="font-size: 12px;">                                Direction: IN / OUT

style="font-size: 12px;">                                Low-level metric : SNR, Delay, Jitter, queue-size, etc

style="font-size: 12px;">        Message :       Request: ask for the data

style="font-size: 12px;">                                Reply: Data

style="font-size: 12px;">                                Notice: For notification or others

style="font-size: 12px;">                                Policy: Control policy

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

---

From: "dingxiaojian (A)" <[email-masked]>
Date: Thu, 10 Aug 2017 03:38:16 +0000

Hi Albert and yanshen,
      Good use case. Another supporter is come. ^_^
      Based on QoS metric, OoE can be easily inferred/predicted. However, I think the process in implicit, you don’t know which metric or what value of metric is useful to improve QoE. And also you have no idea about the relationship between QoS and QoE.
      In some case like network planning, if there exists some QoE demand and some QoS metrics need to be deployed onto network devices, the mapping relation of QoE and QoS (explicit mapping) is definitely need to guide how to select the metric and the corresponding value.

Kind regards
Xiaojian

发件人: IDNET [mailto:[email-masked]] 代表 yanshen
发送时间: 2017年8月8日 18:02
收件人: Albert Cabellos <[email-masked]>
抄送: [email-masked]; Özgü Alay <[email-masked]>
主题: Re: [Idnet] IDN dedicated session call for case

Dear Albert,

Yansen

Dear Albert,

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

### Roadmap ###

###Table of Content###

Yansen

---

From: yanshen <[email-masked]>
Date: Thu, 10 Aug 2017 12:40:04 +0000

Hi Kim,

Thanks for your case in advance.

BTW, have you ever check the one that Jerome mentioned on Tuesday? It is also a traffic classification case.

Apologized that I have no more insight in this area. What is the difference between these two?

At least, whatever, this topic is high focused in current.

Yansen

From: 김민석 [mailto:[email-masked]]
Sent: Thursday, August 10, 2017 10:55 AM
To: Jérôme François <[email-masked]>; Albert Cabellos <[email-masked]>; yanshen <[email-masked]>
Cc: [email-masked]
Subject: RE: [Idnet] IDN dedicated session call for case

HI,

Regards,

Min-Suk Kim

________________________________
보낸 사람 : "Jérôme François" <[email-masked]<mailto:[email-masked]>>
보낸 날짜 : 2017-08-08 23:49:47 ( +09:00 )
받는 사람 : Albert Cabellos <[email-masked]<mailto:[email-masked]>>, yanshen <[email-masked]<mailto:[email-masked]>>
참조 : [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
제목 : Re: [Idnet] IDN dedicated session call for case

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

---

From: Haoyu song <[email-masked]>
Date: Thu, 10 Aug 2017 19:16:42 +0000

Hi Min-Suk,

Sorry I’m confused by this use-case. What do you want to classify exactly? Or in other words, what kind of classification needs (or has to use) ML to solve? Without clear answer to these questions, this is hardly to be seen as a use-case.

Best,
Haoyu
From: IDNET [mailto:[email-masked]] On Behalf Of ???
Sent: Wednesday, August 09, 2017 7:55 PM
To: Jérôme François <[email-masked]>; Albert Cabellos <[email-masked]>; yanshen <[email-masked]>
Cc: [email-masked]
Subject: Re: [Idnet] IDN dedicated session call for case

HI,

Regards,

Min-Suk Kim

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

---

From: 김민석 <[email-masked]>
Date: Fri, 11 Aug 2017 02:32:39 +0000

Hi Yansen,

Thank you for check my usecase.

I know that the usecase is similar topic witht Jerome's one.

However, I'm focusing on creative dataset for ML-based model. We already discussed dataset applying of learning process for a network architecture in last IETF side meeting, but we lost some of points that pre-processing data to apply ML-based learning model is needed with much more efforts. Especially, in trendy deep learning models such as CNN & RNN, cretive dataset is a significant part for efficiently deciding and making system performance. As many guys knows, traffic classification using classical ML algorithms such as anomaly detection or random decision forest had discussed in last NMLRG so that we need more hot trendy issues in aspect of new network machine learning.

Acually, our team is developing real time deep learning model for traffic classification and makes an effort of pre-processing to create ml dataset to apply a couple of deep models. In case of CNN, we collect features for information of applications in payload, then transfer it as like an image[MxN] of dataset. We have another approach of pre-processing of RNN that we are collecting specific patterns from # of packets per application. We also consider a few different methods of ml-based pre-processing for deep learning models in a network achitecture.

If possible, we should set of a new usecase that how ml-based dataset for deep learning models are created by pre-processing in a network architecture.

Best,

Min-Suk Kim

________________________________
보낸 사람 : "yanshen" <[email-masked]>
보낸 날짜 : 2017-08-10 21:40:15 ( +09:00 )
받는 사람 : 김민석 <[email-masked]>
참조 : [email-masked] <[email-masked]>, Jérôme François <[email-masked]>
제목 : RE: [Idnet] IDN dedicated session call for case

Hi Kim,

Yansen

Cc: [email-masked]

HI,

                            Direction: IN / OUT

                          Reply: Data

                          Policy: Control policy

Regards,

Min-Suk Kim

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

---

From: 김민석 <[email-masked]>
Date: Fri, 11 Aug 2017 02:41:25 +0000

Hi Haoyu,

Sorry for giving you confused.

Our purpose is to classify applications using deep learning models. Please read the last email that I sent to Yansen!!

Regards,

Min-Suk Kim

________________________________
보낸 사람 : "Haoyu song" <[email-masked]>
보낸 날짜 : 2017-08-11 04:17:16 ( +09:00 )
받는 사람 : 김민석 <[email-masked]>, Jérôme François <[email-masked]>, Albert Cabellos <[email-masked]>, yanshen <[email-masked]>
참조 : [email-masked] <[email-masked]>
제목 : RE: [Idnet] IDN dedicated session call for case

Hi Min-Suk,

Best,
Haoyu

Cc: [email-masked]

HI,

                            Direction: IN / OUT

                          Reply: Data

                          Policy: Control policy

Regards,

Min-Suk Kim

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

---

From: Jérôme François <[email-masked]>
Date: Fri, 11 Aug 2017 08:54:48 +0200

Hi,

As you said, there might be several algorithms or techniques to be used
in ML problems.

However, I understand from the first use case description that use case
description should be independent of the ML algorithm as much as possible.
Otherwise, we will mutliply the number of use cases.

jerome

Le 11/08/2017 à 04:32, 김민석 a écrit :
> However, I'm focusing on creative dataset for ML-based model.
> We already discussed dataset applying of learning process for a
> network architecture in last IETF side meeting, but we lost some
> of points that pre-processing data to apply ML-based learning model is
> needed with much more efforts. Especially, in trendy deep
> learning models such as CNN & RNN, cretive dataset is a significant
> part for efficiently deciding and making system performance. As many
> guys knows, traffic classification using classical ML algorithms such
> as anomaly detection or random decision forest had discussed in last
> NMLRG so that we need more hot trendy issues in aspect of new network
> machine learning.
>
>
> Acually, our team is developing real time deep learning model for
> traffic classification and makes an effort of pre-processing to create
> ml dataset to apply a couple of deep models. In case of CNN, we
> collect features for information of applications in payload, then
> transfer it as like an image[MxN] of dataset. We have another approach
> of pre-processing of RNN that we are collecting specific patterns from
> # of packets per application. We also consider a few different methods
> of ml-based pre-processing for deep learning models in a network
> achitecture.
>
>
> If possible, we should set of a new usecase that how ml-based dataset
> for deep learning models are created by pre-processing in a network
> architecture.
>
>
> Best, 
>
>  
>
> Min-Suk Kim
>  
> *보낸 사람 : *"yanshen" <[email-masked]>
> *보낸 날짜 : *2017-08-10 21:40:15 ( +09:00 )
> *받는 사람 : *김민석 <[email-masked]>
> *참조 : [email-masked] <[email-masked]>, Jérôme François
> <[email-masked]>
> *제목 : *RE: [Idnet] IDN dedicated session call for case
>
>  
>
> Hi Kim,
>
>  
>
> *From:*김민석[mailto:[email-masked]]
>
>  
>
> *Sent:* Thursday, August 10, 2017 10:55 AM
>
>  
>
> *To:* Jérôme François <[email-masked]>; Albert Cabellos
> <[email-masked]>; yanshen <[email-masked]>
>
>  
>
> *Cc:* [email-masked]
>
>  
>
> *Subject:* RE: [Idnet] IDN dedicated session call for case
>
>  
>
> HI,
>
>  
>
> Description: continuously collect packet data, then applying learning
> process for traffic classification with generating application using
> deep learning models such as CNN (convolutional neural network) and
> RNN (recurrent neural network). Data-set to apply into the models are
> generated by propecessing with features of information from flow in
> packet data.
>
>  
>
> process: 1. collect packet data in real-time, 2. preprocessing
> data-set for deep learning models, 3. Training model using deep
> learning (CNN & RNN), 4. On-line data learning & classifying 5.
> Monitoring and analyzing traffic in the web 
>
>  
>
> *보낸 사람: *"Jérôme François" <[email-masked]
> <mailto:[email-masked]>>
>
> *보낸 날짜: *2017-08-08 23:49:47 ( +09:00 )
>
> *받는 사람: *Albert Cabellos <[email-masked]
> <mailto:[email-masked]>>, yanshen <[email-masked]
> <mailto:[email-masked]>>
>
> *참조: [email-masked] <mailto:[email-masked]><[email-masked]
> <mailto:[email-masked]>>
>
> *제목: *Re: [Idnet] IDN dedicated session call for case
>
>  
>
> Hi all,
>
>  
>
>  
>
> Le 08/08/2017 à06:52, Albert Cabellos a écrit :
>
>  
>
>     Hi all
>
>      
>
>     Here´s another use-case:
>
>      
>
>     Use case N+2: QoE
>
>      
>
>     style="font-size: 12px;">        Description: Collect low-level
>     metrics (SNR, latency, jitter, losses, etc) and measure QoE. Then
>     use ML to understand what is the relation between satisfactory QoE
>     and the low-level metrics. As an example learn that when delay>N
>     then QoE is degraded, but when M<delay<N then QoE is satisfactory
>     for the customers (please note that QoE cannot be measured
>     directly over your network). This is useful to understand how the
>     network must be operated to provide satisfactory QoE.
>
>      
>
>     style="font-size: 12px;">        Process: 1. Low-level data
>     collection and QoE measurement ; 2. Training Model (input
>     low-level metrics, output QoE); 3. Real-time data capture and
>     input; 4. Predict QoE; 5. Operate network to meet target QoE
>     requirement, go to 3.
>
>      
>
>     style="font-size: 12px;">        Data Format:    Time : [Start,
>     End, Unit, Number of Value, Sampling Period]
>
>      
>
>     style="font-size: 12px;">                                Low-level
>     metric : SNR, Delay, Jitter, queue-size, etc
>
>      
>

---

From: "dingxiaojian (A)" <[email-masked]>
Date: Fri, 11 Aug 2017 09:38:28 +0000

Agree, this is exactly what I’m thought.
If you use deep learning method, the network problem (use case n+4) should be very fit with this method.

发件人: IDNET [mailto:[email-masked]] 代表 Jér?me Fran?ois
发送时间: 2017年8月11日 14:55
收件人: [email-masked]
主题: Re: [Idnet] IDN dedicated session call for case

Hi,

As you said, there might be several algorithms or techniques to be used in ML problems.

However, I understand from the first use case description that use case description should be independent of the ML algorithm as much as possible.
Otherwise, we will mutliply the number of use cases.

jerome

Hi Yansen,

Best,

Min-Suk Kim

________________________________
보낸 사람 : "yanshen" <[email-masked]><mailto:[email-masked]>
보낸 날짜 : 2017-08-10 21:40:15 ( +09:00 )
받는 사람 : 김민석 <[email-masked]><mailto:[email-masked]>
참조 : [email-masked]<mailto:[email-masked]> <[email-masked]><mailto:[email-masked]>, Jérôme François <[email-masked]><mailto:[email-masked]>
제목 : RE: [Idnet] IDN dedicated session call for case

Hi Kim,

Yansen

To: Jérôme François <[email-masked]><mailto:[email-masked]>; Albert Cabellos <[email-masked]><mailto:[email-masked]>; yanshen <[email-masked]><mailto:[email-masked]>

Cc: [email-masked]<mailto:[email-masked]>

HI,

                            Direction: IN / OUT

                          Reply: Data

                          Policy: Control policy

Regards,

Min-Suk Kim

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

IDNET mailing list

---

From: yanshen <[email-masked]>
Date: Mon, 14 Aug 2017 03:19:01 +0000

Hi Min-Suk Kim,

Personally, I agree with Jerome and Ding. The reason has been described that the use case should be method/algorithm independent. It should face to the process, the functional entity, the problem and the benefit after solving the problem. Please let me know if I miss some key differences surely.

BTW, as you said that you are currently focusing on the dataset. May I invite you to give some suggestions about the data organizing and data format? As you known that this part has been discussed in our mail list for quite a long time.  Especially the format and aspects that I proposed currently, your research is related with the practical data processing, I think you must have lot of experience to help improving.

Many thanks,

Yansen

From: IDNET [mailto:[email-masked]] On Behalf Of dingxiaojian (A)
Sent: Friday, August 11, 2017 5:38 PM
To: Jérôme François <[email-masked]>; [email-masked]
Subject: [Idnet] 答复: IDN dedicated session call for case

发件人: IDNET [mailto:[email-masked]] 代表 Jér?me Fran?ois
发送时间: 2017年8月11日 14:55
收件人: [email-masked]<mailto:[email-masked]>
主题: Re: [Idnet] IDN dedicated session call for case

Hi,

jerome

Hi Yansen,

Best,

Min-Suk Kim

Hi Kim,

Yansen

HI,

                            Direction: IN / OUT

                          Reply: Data

                          Policy: Control policy

Regards,

Min-Suk Kim

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

IDNET mailing list

---

From: yanshen <[email-masked]>
Date: Mon, 14 Aug 2017 03:34:03 +0000

Hi Stenio, 

I'm looking forward to them!

Yansen

---

From: 김민석 <[email-masked]>
Date: Wed, 16 Aug 2017 05:41:51 +0000

Hi Yansen,

Sorry about I did reply your email a little late due to my vacation until yesterday.

You can invite me or our team for some of suggestion. Nowadays, we are focuing on ml-based network dataset for deep learning algorithm so that we may give you researching tips of dataset based on ml algorithm.

Best,

Min-Suk Kim

________________________________
보낸 사람 : "yanshen" <[email-masked]>
보낸 날짜 : 2017-08-14 12:19:23 ( +09:00 )
받는 사람 : 김민석 <[email-masked]>, [email-masked] <[email-masked]>
참조 : dingxiaojian (A) <[email-masked]>, Jérôme François <[email-masked]>
제목 : RE: [Idnet] 答复: IDN dedicated session call for case

Hi Min-Suk Kim,

Many thanks,

Yansen

发送时间: 2017年8月11日 14:55

Hi,

jerome

Hi Yansen,

Best,

Min-Suk Kim

Hi Kim,

Yansen

HI,

                            Direction: IN / OUT

                          Reply: Data

                          Policy: Control policy

Regards,

Min-Suk Kim

Hi all,

                                Direction: IN / OUT

                           Reply: Data

                           Policy: Control policy

Best regards,

jerome

Hi all

Here´s another use-case:

Use case N+2: QoE

Kind regards

Albert

Dear all,

                                Direction: IN / OUT

                                Reply: Data

                                Policy: Control policy

                                Reply: Data

                                Policy: Control policy

### Roadmap ###

### Roadmap End ###

###Table of Content###

        1.2 TBD

2. Use Cases

        2.1 Traffic Prediction

        2.2 QoS Management

        3.3 TBD

3. Data Focus

        3.1 Data attribute

        3.2 Data format

        3.3 TBD

4. Aims

        4.2 TBD

###ToC End###

Yansen

IDNET mailing list

IDNET mailing list

IDNET mailing list