Subject: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

From: Sheng Jiang <[email-masked]>
Date: Tue, 28 Mar 2017 16:29:50 +0000

Hi, all,

Although there are many understanding for Intelligence-Defined Network, we are actually using this IDN as a term reference to the SDN-beyond architecture that we presented in IETF97, see the below link. A reference model is presented in page 3, while potential standardization works is presented in page 9. 

https://www.ietf.org/proceedings/97/slides/slides-97-nmlrg-intelligence-defined-network-01.pdf

Although it might be a little bit too early for AI/ML in network giving the recent story of the concluded proposed NMLRG, we still would like to call for interests in IDN. Anybody (on site in Chicago this week) are interested in this or even wider topics regarding to AI/ML in network, please contact me on [email-masked] . Then we may have an informal meeting to discuss some common interests and potential future activities (not any activities in IETF, but also other STO or experimental trails, etc.)  on Thursday morning.

FYI, we have already working on a Work Item, called IDN in the ETSI NGP (Next Generation Protocol) ISG, links below.

https://portal.etsi.org/tb.aspx?tbid=844&SubTB=844
https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=51011

Meanwhile, please do use this mail list as a forum to discuss any topics that may applying AI/ML into network area.

Best regards,

Sheng

---

From: Sheng Jiang <[email-masked]>
Date: Tue, 28 Mar 2017 16:44:42 +0000

Oops... An important typo. Try again:

we may have an informal meeting to discuss some common interests and potential future activities (not "only" activities in IETF, but also other STO or experimental trails, etc.)  on Thursday morning.

Sheng
________________________________________
From: IDNET [[email-masked]] on behalf of Sheng Jiang [[email-masked]]
Sent: 29 March 2017 0:29
To: [email-masked]
Subject: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Hi, all,

Best regards,

Sheng
_______________________________________________
IDNET mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idnet

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Mar 2017 09:48:52 -0700

Understand that it is an informal meeting, but any chance we can get a way
to enable remote participation?

Thanks,

Dave

On Tue, Mar 28, 2017 at 9:44 AM, Sheng Jiang <[email-masked]> wrote:

> we may have an informal meeting to discuss some common interests and
> potential future activities (not "only" activities in IETF, but also other
> STO or experimental trails, etc.)  on Thursday morning.
>
> Sheng
> Although there are many understanding for Intelligence-Defined Network, we
> are actually using this IDN as a term reference to the SDN-beyond
> architecture that we presented in IETF97, see the below link. A reference
> model is presented in page 3, while potential standardization works is
> presented in page 9.
>
> https://www.ietf.org/proceedings/97/slides/slides-
> 97-nmlrg-intelligence-defined-network-01.pdf
>
> Although it might be a little bit too early for AI/ML in network giving
> the recent story of the concluded proposed NMLRG, we still would like to
> call for interests in IDN. Anybody (on site in Chicago this week) are
> interested in this or even wider topics regarding to AI/ML in network,
> please contact me on [email-masked] . Then we may have an informal
> meeting to discuss some common interests and potential future activities
> (not any activities in IETF, but also other STO or experimental trails,
> etc.)  on Thursday morning.
>
> FYI, we have already working on a Work Item, called IDN in the ETSI NGP
> (Next Generation Protocol) ISG, links below.
>
> Meanwhile, please do use this mail list as a forum to discuss any topics
> that may applying AI/ML into network area.
>
> Best regards,
>
> Sheng

---

From: Shamik Mishra <[email-masked]>
Date: Tue, 28 Mar 2017 16:52:41 +0000

Hi Sheng,
Is it possible to participate in this meeting remotely?

Regards
Shamik

-----Original Message-----
From: IDNET [mailto:[email-masked]] On Behalf Of Sheng Jiang
Sent: Tuesday, March 28, 2017 10:00 PM
To: [email-masked]
Subject: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Hi, all,

Best regards,

Sheng
_______________________________________________
IDNET mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idnet
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."

---

From: Oscar Mauricio Caicedo Rendon <[email-masked]>
Date: Tue, 28 Mar 2017 12:02:08 -0500

Hi, all,

If there is a meeting, I would like to participate remotely.

Best regards,

Oscar

-- 
*Oscar Mauricio Caicedo Rendón*
*PhD Computer Science - Federal University of Rio Grande do Sul*
*Full Profesor - University of Cauca*

-- 

------------------------------
*Universidad del Cauca: Comprometidos con la calidad.*

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Mar 2017 10:59:38 -0700

Hey Sheng,

I just wanted to revive my key concern on [0] (same one I made at the
NMRL): The hard parts of getting Machine Learning intelligence into
Networking is the Machine Learning part. In addition, successful deployment
of ML requires knowledge of ML combined with domain knowledge. We
definitely have the domain knowledge; the problem is that we don't have the
ML knowledge, and this is one of the big factors holding us back; see e.g.
Andrew's discussion of talent in [1].  Slides such as [0] seem to imply
that *someone else* (in particular, not us)  will handle the ML part of all
of this. I'll just note that in general successful deployments of ML don't
work this way; the domain experts will have to learn ML (and vice versa)
for us to be successful (again, see [1] and many others).

Perhaps a useful exercise would be to write an ID that makes your
assumptions explicit?

Thanks,

Dave

[0]
https://www.ietf.org/proceedings/97/slides/slides-97-nmlrg-intelligence-defined-network-01.pdf
[1]
https://hbr.org/2016/11/what-artificial-intelligence-can-and-cant-do-right-now

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Mar 2017 11:01:46 -0700

s/NMRL/NMLRG/   (sorry about that). Dave

On Tue, Mar 28, 2017 at 10:59 AM, David Meyer <[email-masked]> wrote:

> [0] https://www.ietf.org/proceedings/97/slides/slides-
> [1] https://hbr.org/2016/11/what-artificial-intelligence-can-
> and-cant-do-right-now
>
>
>> Although there are many understanding for Intelligence-Defined Network,
>> we are actually using this IDN as a term reference to the SDN-beyond
>> please contact me on [email-masked] . Then we may have an
>> informal meeting to discuss some common interests and potential future
>> activities (not any activities in IETF, but also other STO or experimental
>> trails, etc.)  on Thursday morning.
>>

---

From: Brian Njenga <[email-masked]>
Date: Tue, 28 Mar 2017 18:06:15 +0000

I'm glad that the discussions​ on how to make AI/ML useful in the design of
the internet's architecture are continuing, albeit under a different name.
I'd like to participate remotely as well.

I have a question that someone more thoughtful​ than I could answer: Are
there research questions on how to anticipate network-threatening DDOS
attacks, such as those coming from the Mirai botnet family, using ML? Even
going as far as designing and standardising an efficient, secure network
protocol for IoT devices. This is a complicated issue, which involves
emerging markets, so I'm interested in useful ideas from any angle.

Thanks, and I'm glad to meet you all.

Best regards,
Brian Muhia.

-- 
Some say he really tries to learn efficiently.

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Mar 2017 11:17:18 -0700

Hey Brian,

We are working on anomaly detection technologies to detect things like DDOS
attacks; we're about to publish some results and will let you know when
that is ready. In the mean time, you might want to look at attacks against
ML itself, see [0], [1], and [2].  The bottom line here is that even simple
linear models are susceptible to adversarial attacks, so things like the
autoencoders we used to do binary classification (e.g., anomaly detection)
are susceptible; see slides 16+ of [3].

Dave

[0] https://arxiv.org/pdf/1312.6199.pdf
[1] https://arxiv.org/pdf/1412.6572.pdf
[2] https://arxiv.org/pdf/1602.02697.pdf
[3] http://www.1-4-5.net/~dmm/ml/talks/2016/cor_ml4networking.pdf

On Tue, Mar 28, 2017 at 11:06 AM, Brian Njenga <[email-masked]>
wrote:

> I'm glad that the discussions​ on how to make AI/ML useful in the design
> of the internet's architecture are continuing, albeit under a different
> name. I'd like to participate remotely as well.
>

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Wed, 29 Mar 2017 03:25:30 +0900

> for us to be successful (again, see [1] and many others).

Dear Dave,

You are true in that ML/domain knowledge is necessary but, however it is
worth to take into account that it is not strictly required and it will
even be counterproductive in some (or maybe most) situations. At the end
of the day, encouraging (or forcing) a network expert to learn ML is
quite difficult, the results will be delayed until the learning phase
ends, and (most probably) s/he will never get a better solution than a
person that has been an expert in ML from a long time ago. Therefore, it
is better to make separate experts (in ML and the domain itself) to
collaborate in a common solution. Therefore, and I think it has been
mentioned before, we have to (try to) enroll experts in ML to the IDNET
group and see what can we do together...

> Perhaps a useful exercise would be to write an ID that makes your

Regards,
Pedro

-- 
Pedro Martinez-Julia
Network Science and Convergence Device Technology Laboratory
Network System Research Institute
National Institute of Information and Communications Technology (NICT)
4-2-1, Nukui-Kitamachi, Koganei, Tokyo 184-8795, Japan
Email: [email-masked]
---------------------------------------------------------
*** Entia non sunt multiplicanda praeter necessitatem ***

---

From: Brian Njenga <[email-masked]>
Date: Tue, 28 Mar 2017 18:26:33 +0000

Thanks, Dave. I'll check them out.

Something I currently think is very cool is the Distill journal, at
https://distill.pub, edited by Shan Carter and Chris Olah.

I would love to see more ML tutorials​ and lessons written in that style.
They're​accepting submissions for tutorials, I think, so if any ML expert
here would like to explain any technique in an explorable manner, please
reach out to them.

On Tue, 28 Mar 2017, 21:17 David Meyer, <[email-masked]> wrote:

> Hey Brian,
>
> We are working on anomaly detection technologies to detect things like
> DDOS attacks; we're about to publish some results and will let you know
> when that is ready. In the mean time, you might want to look at attacks
> against ML itself, see [0], [1], and [2].  The bottom line here is that
> even simple linear models are susceptible to adversarial attacks, so things
> like the autoencoders we used to do binary classification (e.g., anomaly
> detection) are susceptible; see slides 16+ of [3].
>
> Dave
>

---

From: Jérôme François <[email-masked]>
Date: Tue, 28 Mar 2017 20:26:36 +0200

Hi,

Le 28/03/2017 à 20:06, Brian Njenga a écrit :
>
> I'm glad that the discussions​ on how to make AI/ML useful in the
> design of the internet's architecture are continuing, albeit under a
> different name. I'd like to participate remotely as well.
>
> I have a question that someone more thoughtful​ than I could answer:
> Are there research questions on how to anticipate network-threatening
> DDOS attacks, such as those coming from the Mirai botnet family, using
> ML? Even going as far as designing and standardising an efficient,
> secure network protocol for IoT devices. This is a complicated issue,
> which involves emerging markets, so I'm interested in useful ideas
> from any angle.
>
Regarding DDoS, I think we have to consider two angles even when
considering protecting the network. The flooding-like attacks (1) and
the most sophisticated attack that can highly degrade service
performance even with few packets (2) (and of course this also concerns
network service, even more than before with network softwarization which
puts network functions into VM, being thus more exposed than before in
my opinion).

For type 1, there are usually observable signs of a future attack
(without ML) in the hours or days before with an increasing load.
However even if you observe it, it is very hard to fully characterize it
and so prevent it to really happen. For type 2, you basically need to
predict what should be the load induced by a packet or flow when
considering the targeted service. Assuming you have no complete
knwoledge of the service, you can try to use some learning and
regression techniques to do it but assuming also that the trafic is
encrypted, the problem becomes more difficult.

jerome

> On Tue, 28 Mar 2017, 20:02 Oscar Mauricio Caicedo Rendon,
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hi, all,
>
>     On Tue, Mar 28, 2017 at 11:29 AM, Sheng Jiang
>     <[email-masked] <mailto:[email-masked]>> wrote:
>
>         Hi, all,
>
>         Although there are many understanding for Intelligence-Defined
>         Network, we are actually using this IDN as a term reference to
>         the SDN-beyond architecture that we presented in IETF97, see
>         the below link. A reference model is presented in page 3,
>         while potential standardization works is presented in page 9.
>
>         Although it might be a little bit too early for AI/ML in
>         network giving the recent story of the concluded proposed
>         NMLRG, we still would like to call for interests in IDN.
>         Anybody (on site in Chicago this week) are interested in this
>         or even wider topics regarding to AI/ML in network, please
>         contact me on [email-masked]
>         <mailto:[email-masked]> . Then we may have an informal
>         meeting to discuss some common interests and potential future
>         activities (not any activities in IETF, but also other STO or
>         experimental trails, etc.)  on Thursday morning.
>
>         FYI, we have already working on a Work Item, called IDN in the
>         ETSI NGP (Next Generation Protocol) ISG, links below.
>
>         Meanwhile, please do use this mail list as a forum to discuss
>         any topics that may applying AI/ML into network area.
>
>         Best regards,
>
>         Sheng
>         [email-masked] <mailto:[email-masked]>
>     ------------------------------------------------------------------------
>     *
>     */**/Universidad del Cauca: Comprometidos con la calidad/**/.*
>     *_______________________________________________
>     IDNET mailing list
>     [email-masked] <mailto:[email-masked]>

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Mar 2017 11:28:46 -0700

Brian,

Fortunately for us, there is tons of tutorial material (and code) around
the network. Distill is a cool idea, but there are many more.

Dave

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Wed, 29 Mar 2017 03:29:21 +0900

Dear David,

You did it again: You wrote a quite interesting paragraph and well
supported by references. Would you give me permission to copy the text,
and possibly the references, to the Git repository?

Regards,
Pedro

---

From: David Meyer <[email-masked]>
Date: Tue, 28 Mar 2017 11:32:48 -0700

Sure, be my guest. Dave

---

From: Brian Njenga <[email-masked]>
Date: Tue, 28 Mar 2017 18:38:20 +0000

Got it, thanks! Great response. What other research questions do people
have?

On Tue, 28 Mar 2017, 21:26 Jérôme François, <[email-masked]>
wrote:

> considering protecting the network. The flooding-like attacks (1) and the
> most sophisticated attack that can highly degrade service performance even
> with few packets (2) (and of course this also concerns network service,
> even more than before with network softwarization which puts network
> functions into VM, being thus more exposed than before in my opinion).
>
> For type 1, there are usually observable signs of a future attack (without
> ML) in the hours or days before with an increasing load. However even if
> you observe it, it is very hard to fully characterize it and so prevent it
> to really happen. For type 2, you basically need to predict what should be
> the load induced by a packet or flow when considering the targeted service.
> Assuming you have no complete knwoledge of the service, you can try to use
> some learning and regression techniques to do it but assuming also that the
> trafic is encrypted, the problem becomes more difficult.
>
> jerome
>
>
> * Universidad del Cauca: Comprometidos con la calidad. *
> IDNET mailing [email-masked]://www.ietf.org/mailman/listinfo/idnet
>
>
> --

---

From: Jérôme François <[email-masked]>
Date: Tue, 28 Mar 2017 20:46:44 +0200

Hi

> group and see what can we do together...
This is a general trend that only a single person cannot be expert in
everything. Actually, a good network expert may require good ML but also
good software skills (including software formal verification knowledge).
So, in my opinion the problem is larger.

Enhancing collaboration between network and ML expert is a path that
starts in many company and insitutes I think. Discussing with ML
experts, they are usually open and happy to discover new "use cases" 
but their first question will be "do you have some labelled datasets
that we can work with" which relates to the problem raised in previous
emails about open datasets.

In my opinion, if we wan to attract ML experts in our dicussions, we
should identify few scenrios, defined them precisely and provide an open
dataset. By defining them, I mean we have to give them all background
they need to understand (that can be built incrementally through
discussion) in a well-documented format.

jerome

---

From: 김민석 <[email-masked]>
Date: Tue, 28 Mar 2017 23:04:29 +0000

Hi Brian,

As you mentioned by the prior email, anticipating network DDos attacks is really trendy issue to solve by ML techniques.

We also make some efforts how to avoid fagile nodes by a trustworthy communication, that means quantifying trustworthiness of node with normalization of various requirements such as security function, bandwidth and etc.

We are freshly approaching in routing layer with confidence using our own requirements, TPD(Trust Policy Distribution) and TD(Trust Degree). These requirements are considered to be solved by Reinforcement Learning (RL) that is one of the ML algorithms. RL is useful to control some of network policy about specific actions and states with reinforced and purnished rewards (+/-), but the problem is too slow to acquire satisified performance. Other ways to say it, anormaly dectection and regression analysis might be both efficient approaching methods to solve the issues Dave mentioned.

Best Regards,

Min-Suk Kim

Senior Researcher / Ph.D.
Intelligent IoE Network Research Section,
Electronics and Telecommunications Research Institute (ETRI)
e-mail          :  [email-masked]<mailto:[email-masked]>
http://www.etri.re.kr/

________________________________
보낸 사람 : "Brian Njenga" <[email-masked]>
보낸 날짜 : 2017-03-29 03:38:49 ( +09:00 )
받는 사람 : Jérôme François <[email-masked]>, Oscar Mauricio Caicedo Rendon <[email-masked]>, Sheng Jiang <[email-masked]>
참조 : [email-masked] <[email-masked]>
제목 : Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Got it, thanks! Great response. What other research questions do people have?

On Tue, 28 Mar 2017, 21:26 Jérôme François, <[email-masked]<mailto:[email-masked]>> wrote:

Hi,

class="gmail_msg">

class="gmail_msg">

class="gmail_msg">

I'm glad that the discussions​ on how to make AI/ML useful in the design of the internet's architecture are continuing, albeit under a different name. I'd like to participate remotely as well.

I have a question that someone more thoughtful​ than I could answer: Are there research questions on how to anticipate network-threatening DDOS attacks, such as those coming from the Mirai botnet family, using ML? Even going as far as designing and standardising an efficient, secure network protocol for IoT devices. This is a complicated issue, which involves emerging markets, so I'm interested in useful ideas from any angle.

Regarding DDoS, I think we have to consider two angles even when considering protecting the network. The flooding-like attacks (1) and the most sophisticated attack that can highly degrade service performance even with few packets (2) (and of course this also concerns network service, even more than before with network softwarization which puts network functions into VM, being thus more exposed than before in my opinion).

class="gmail_msg">

class="gmail_msg">For type 1, there are usually observable signs of a future attack (without ML) in the hours or days before with an increasing load. However even if you observe it, it is very hard to fully characterize it and so prevent it to really happen. For type 2, you basically need to predict what should be the load induced by a packet or flow when considering the targeted service. Assuming you have no complete knwoledge of the service, you can try to use some learning and regression techniques to do it but assuming also that the trafic is encrypted, the problem becomes more difficult.

class="gmail_msg">

class="gmail_msg">jerome

class="gmail_msg">

class="gmail_msg">

Best regards,

class="gmail_msg">Brian Muhia.

class="gmail_msg">
On Tue, 28 Mar 2017, 20:02 Oscar Mauricio Caicedo Rendon, <[email-masked]<mailto:[email-masked]>> wrote:

class="gmail_msg">
Hi, all,

class="gmail_msg">

class="gmail_msg">

class="gmail_msg">

class="gmail_msg">
Best regards,

class="gmail_msg">

class="gmail_msg">
Oscar

class="gmail_msg">

class="gmail_msg">
On Tue, Mar 28, 2017 at 11:29 AM, Sheng Jiang <[email-masked]<mailto:[email-masked]>> wrote:

class="gmail_msg">
Hi, all,

class="gmail_msg">

class="gmail_msg">Although there are many understanding for Intelligence-Defined Network, we are actually using this IDN as a term reference to the SDN-beyond architecture that we presented in IETF97, see the below link. A reference model is presented in page 3, while potential standardization works is presented in page 9.

class="gmail_msg">

class="gmail_msg">https://www.ietf.org/proceedings/97/slides/slides-97-nmlrg-intelligence-defined-network-01.pdf

class="gmail_msg">

class="gmail_msg">Although it might be a little bit too early for AI/ML in network giving the recent story of the concluded proposed NMLRG, we still would like to call for interests in IDN. Anybody (on site in Chicago this week) are interested in this or even wider topics regarding to AI/ML in network, please contact me on [email-masked]<mailto:[email-masked]> . Then we may have an informal meeting to discuss some common interests and potential future activities (not any activities in IETF, but also other STO or experimental trails, etc.)  on Thursday morning.

class="gmail_msg">

class="gmail_msg">FYI, we have already working on a Work Item, called IDN in the ETSI NGP (Next Generation Protocol) ISG, links below.

class="gmail_msg">

class="gmail_msg">https://portal.etsi.org/tb.aspx?tbid=844&SubTB=844

class="gmail_msg">https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=51011

class="gmail_msg">

class="gmail_msg">Meanwhile, please do use this mail list as a forum to discuss any topics that may applying AI/ML into network area.

class="gmail_msg">

class="gmail_msg">Best regards,

class="gmail_msg">

class="gmail_msg">Sheng

class="gmail_msg">_______________________________________________

class="gmail_msg">IDNET mailing list

class="gmail_msg">[email-masked]<mailto:[email-masked]>

class="gmail_msg">https://www.ietf.org/mailman/listinfo/idnet

class="gmail_msg">

class="gmail_msg">

class="gmail_msg" clear="all">

class="gmail_msg">--

class="gmail_msg">
Oscar Mauricio Caicedo Rendón
PhD Computer Science - Federal University of Rio Grande do Sul
Full Profesor - University of Cauca

class="gmail_msg">
________________________________
Universidad del Cauca: Comprometidos con la calidad.
_______________________________________________

class="gmail_msg">IDNET mailing list

class="gmail_msg">[email-masked]<mailto:[email-masked]>

class="gmail_msg">https://www.ietf.org/mailman/listinfo/idnet

class="gmail_msg">
--

class="gmail_msg">

class="gmail_msg">

class="gmail_msg">

_______________________________________________
IDNET mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/idnet

class="gmail_msg">
--

---

From: "dingxiaojian (A)" <[email-masked]>
Date: Wed, 29 Mar 2017 01:59:51 +0000

Hi Sheng,
    I want to join the meeting remotely as well.

Regards
 XiaoJian

-----邮件原件-----
发件人: IDNET [mailto:[email-masked]] 代表 Shamik Mishra
发送时间: 2017年3月29日 0:53
收件人: Sheng Jiang <[email-masked]>; [email-masked]
主题: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Hi Sheng,

Regards
Shamik

-----Original Message-----
From: IDNET [mailto:[email-masked]] On Behalf Of Sheng Jiang
Sent: Tuesday, March 28, 2017 10:00 PM
To: [email-masked]
Subject: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Hi, all,

Best regards,

Sheng

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Mar 2017 07:17:08 -0700

Hey Min-Suk,

Totally agree we need to learn from our environment, and RL is a natural
approach. After all, the network is always changing, has adversaries, etc.
All of this means. among other things,  that we can't make simplifying
assumptions like stationary distributions,  iid data, .... So RL is one way
to attack these problems, and the classic algorithms you mention below are
certainly a reasonable approach (I've been working with policy gradients
[0], trying to model/adapt the two-player game approach of AlphaGo to
networking; the problem there is that we don't have a source of labeled
expert data like the KGS Go server (https://www.gokgs.com/) to build the
supervised policy network....).

You might also want to check out the recent "boot" of evolution strategies
as a black-box approach to RL (in particular no gradients). See [1],  [2],
 [3]. There is also a ton of code around if you want to try some of this
out (see e.g.,https://github.com/dennybritz/reinforcement-learning; this
one is in tensorflow). Finally, I've attached a few summary slides with
some of my musings on this topic from past talks.

Thanks,

Dave

[BTW, two player minimax games seem to be popping up everywhere: AlphaGo,
variational autoencoders [4], GANs [5], and many others; something to thing
about for our domain]

[0]
https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf
[1] https://blog.openai.com/evolution-strategies/
[2] https://arxiv.org/pdf/1703.03864.pdf
[3] http://jmlr.csail.mit.edu/papers/volume15/wierstra14a/wierstra14a.pdf
[4] http://www.1-4-5.net/~dmm/ml/vae.pdf
[5] https://arxiv.org/pdf/1406.2661.pdf

On Tue, Mar 28, 2017 at 4:04 PM, 김민석 <[email-masked]> wrote:

> Hi Brian,
>
>
> As you mentioned by the prior email, anticipating network DDos
> attacks is really trendy issue to solve by ML techniques.
>
> We also make some efforts how to avoid fagile nodes by a trustworthy
> communication, that means quantifying trustworthiness of node with
> normalization of various requirements such as security function, bandwidth
> and etc.
>
> We are freshly approaching in routing layer with confidence using our own
> requirements, TPD(Trust Policy Distribution) and TD(Trust Degree). These
> requirements are considered to be solved by Reinforcement Learning
> (RL) that is one of the ML algorithms. RL is useful to control some of
> network policy about specific actions and states with reinforced and
> purnished rewards (+/-), but the problem is too slow to acquire satisified
> performance. Other ways to say it, anormaly dectection and regression
> analysis might be both efficient approaching methods to solve the issues
> Dave mentioned.
>
>
> Best Regards,
>
>
> Min-Suk Kim
>
> *E*lectronics and *T*elecommunications *R*esearch *I*nstitute (*ETRI)*
> e-mail          :  [email-masked] <[email-masked]>
> http://www.etri.re.kr/
>
>
>
>
>
>
>

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Mar 2017 07:25:40 -0700

Apparently you can't attach a .pptx. The attachment is here (pptx and pdf):

http://www.1-4-5.net/~dmm/ml/misc/musings.pptx
http://www.1-4-5.net/~dmm/ml/misc/musings.pdf

Thx,

Dave

On Wed, Mar 29, 2017 at 7:17 AM, David Meyer <[email-masked]> wrote:

> [0] https://papers.nips.cc/paper/1713-policy-gradient-
> methods-for-reinforcement-learning-with-function-approximation.pdf

---

From: Alex Galis <[email-masked]>
Date: Wed, 29 Mar 2017 09:33:19 -0500

Dear Sheng

Would you be so kind as to provide the IDET meeting time and venue  for Thursday

Thank you in advance
Best Regards
Alex Galis

> On 28 Mar 2017, at 11:44, Sheng Jiang <[email-masked]> wrote:
>

---

From: Chris Hammerschmidt <[email-masked]>
Date: Wed, 29 Mar 2017 16:33:36 +0200

Hi,

equally important to having datasets* is proposing benchmarks with the
datasets.* Right now, everyone has different expectations and from my
personal experience, many reviewers at networking conferences won't accept
a "typical" machine learning paper with applications in networking
consisting of data+model+evaluation. Expectations range from something like
a large-scale practical application case to addressing worries about
adversarial influence on the learned model, or systems that take not only
care of machine-learning malicious/anomalous/interesting behavior but want
to have guidelines how to apply it, e.g. filtering rules and
parameter/threshold setting guidelines. Often, these values have to be set
with domain knowledge, adjusted to the specific application case (as
networks can be very different).

The lack of benchmarks makes it incredibly hard to compare different
solutions to each other, or even to just apply a new method to an old
problem; and indeed, reviewers have pointed out to me that the problem of
machine learning from network traffic has been solved already and there is
no point in further papers on this topic. A *shared benchmark* solves this
problem by delegating the argument for the relevance ONCE while setting up
the benchmark, rather than distributing the task to each author.

Cheers
Christian

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Mar 2017 07:58:00 -0700

Chris,

Totally agree. This is part of the reason we need standardized data sets,
namely, so we can compare results and understand if we're making progress.
See e.g., http://image-net.org/challenges/LSVRC/2016/ for an example.  The
result has been the steady ratcheting down of error rates to super-human
levels. We have nothing like this for networking, and as I have been
saying, that is one of our biggest challenges.

Dave

On Wed, Mar 29, 2017 at 7:33 AM, Chris Hammerschmidt <[email-masked]>
wrote:

> care of machine-learning malicious/anomalous/interesting behavior but
> want to have guidelines how to apply it, e.g. filtering rules and
> no point in further papers on this topic. A *shared benchmark* solves
> this problem by delegating the argument for the relevance ONCE while
> setting up the benchmark, rather than distributing the task to each author.
>
> Cheers
> Christian
>

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Thu, 30 Mar 2017 00:01:17 +0900

Hi,

In the long term, having some good-quality and widely-accepted datasets
is essential for the demonstration of qualities (benchmarking) as well
as continuous assessment (Is my solution doing right? How good is it?).

That said, I think that building and provisioning such datasets is a key
role that can only be played by operators, since they have tons of data
from which obtaining them. The only issues I see are the anonymization
of such data, to protect both the privacy of users and the potentially
confidential information from the operator itself, and the selection of
the *good quality* extracts.

Therefore, I hereby call to anyone from the list that is part of (or
involved with) an operator to please try to work with us on this issue
and evaluate the possible alternatives to get/build such datasets. Thank
you very much.

Regards,
Pedro

---

From: 김민석 <[email-masked]>
Date: Wed, 29 Mar 2017 15:29:30 +0000

Hi Dave,

Thank you for giving me the great information.

I absolutely agree your opinion that we need real ML data applied by data pre-processing so that we have been already trying to make available ML data on many ways such as clustering and classification. (using datasets of contents and URL)

It's so challenge-able steps before using adaptive ML algorithm to network field.

As you mentioned, RL is classical ML algorithm, but it is rapidly going develpment and make great results with tensorflow in many fields, unfortunately not network.

For our tutorial, I attach some of practical examples with tensorflow as below,

https://github.com/tensorflow/models

Additionally, I submitted a personal draft to NMLRG even if it was closed from last meeting. It's about collaborative distributed multi-agent using re-inforcement learning and we trying to apply it to network real architecture.

The attachment is on the email. I really appreciate giving me a small piece of your feedback and comment if you have a chance.

Sincerely,

Min-Suk Kim

ETRI

________________________________
보낸 사람 : "David Meyer" <[email-masked]>
보낸 날짜 : 2017-03-29 23:25:48 ( +09:00 )
받는 사람 : 김민석 <[email-masked]>
참조 : Brian Njenga <[email-masked]>, Jérôme François <[email-masked]>, Oscar Mauricio Caicedo Rendon <[email-masked]>, Sheng Jiang <[email-masked]>, [email-masked] <[email-masked]>
제목 : Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Thx,

Dave

On Wed, Mar 29, 2017 at 7:17 AM, David Meyer <[email-masked]<mailto:[email-masked]>> wrote:

Hey Min-Suk,

Totally agree we need to learn from our environment, and RL is a natural approach. After all, the network is always changing, has adversaries, etc. All of this means. among other things,  that we can't make simplifying assumptions like stationary distributions,  iid data, .... So RL is one way to attack these problems, and the classic algorithms you mention below are certainly a reasonable approach (I've been working with policy gradients [0], trying to model/adapt the two-player game approach of AlphaGo to networking; the problem there is that we don't have a source of labeled expert data like the KGS Go server (https://www.gokgs.com/) to build the supervised policy network....).

You might also want to check out the recent "boot" of evolution strategies as a black-box approach to RL (in particular no gradients). See [1],  [2],  [3]. There is also a ton of code around if you want to try some of this out (see e.g.,https://github.com/dennybritz/reinforcement-learning; this one is in tensorflow). Finally, I've attached a few summary slides with some of my musings on this topic from past talks.

Thanks,

Dave

[BTW, two player minimax games seem to be popping up everywhere: AlphaGo, variational autoencoders [4], GANs [5], and many others; something to thing about for our domain]

[0] https://papers.nips.cc/paper/1713-policy-gradient-methods-for-reinforcement-learning-with-function-approximation.pdf

On Tue, Mar 28, 2017 at 4:04 PM, 김민석 <[email-masked]<mailto:[email-masked]>> wrote:

Hi Brian,

Best Regards,

Min-Suk Kim

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Mar 2017 10:12:06 -0700

Hey Min-Suk,

On Wed, Mar 29, 2017 at 8:29 AM, 김민석 <[email-masked]> wrote:

> I absolutely agree your opinion that we need real ML data applied by data
> pre-processing so that we have been already trying to make available ML
> data on many ways such as clustering and classification. (using datasets of
> contents and URL)
>
> As you mentioned, RL is classical ML algorithm, but it is rapidly going
> develpment and make great results with tensorflow in many fields,
> unfortunately not network.
>
> Additionally, I submitted a personal draft to NMLRG even if it was closed
> from last meeting. It's about collaborative distributed multi-agent using
> re-inforcement learning and we trying to apply it to network real
> architecture.
>
> The attachment is on the email. I really appreciate giving me a small
> piece of your feedback and comment if you have a chance.
>

Thanks. I will try to read/comment later today.

Thanks again,

Dave

> *보낸 사람 : *"David Meyer" <[email-masked]>
> *보낸 날짜 : *2017-03-29 23:25:48 ( +09:00 )
> *받는 사람 : *김민석 <[email-masked]>
> *참조 : *Brian Njenga <[email-masked]>, Jérôme François <
> [email-masked]>, Oscar Mauricio Caicedo Rendon <
> [email-masked]>, Sheng Jiang <[email-masked]>,
> [email-masked] <[email-masked]>
> *제목 : *Re: [Idnet] Intelligence-Defined Network Architecture and Call for
> Interests
>
>
>> You might also want to check out the recent "boot" of evolution
>> strategies as a black-box approach to RL (in particular no gradients). See
>> [1],  [2],  [3]. There is also a ton of code around if you want to try some
>> of this out (see e.g.,https://github.com/dennyb
>> ritz/reinforcement-learning; this one is in tensorflow). Finally, I've
>> attached a few summary slides with some of my musings on this topic from
>> past talks.
>>
>>
>> Thanks,
>>
>>
>> Dave
>>
>>
>> [0] https://papers.nips.cc/paper/1713-policy-gradient-method
>> s-for-reinforcement-learning-with-function-approximation.pdf
>>> We are freshly approaching in routing layer with confidence using our
>>> own requirements, TPD(Trust Policy Distribution) and TD(Trust Degree).
>>> These requirements are considered to be solved by Reinforcement Learning
>>> *E*lectronics and *T*elecommunications *R*esearch *I*nstitute (*ET**R*
>>> *I)*

---

From: 김민석 <[email-masked]>
Date: Wed, 29 Mar 2017 17:37:59 +0000

Thank you so much =:)

-Minsuk Kim

Sent from my iPhone

On 29 Mar 2017, at 12:12 PM, David Meyer <[email-masked]<mailto:[email-masked]>> wrote:

Hey Min-Suk,

On Wed, Mar 29, 2017 at 8:29 AM, 김민석 <[email-masked]<mailto:[email-masked]>> wrote:

Hi Dave,

Thanks again,

Dave

Sincerely,

Min-Suk Kim

________________________________
보낸 사람 : "David Meyer" <[email-masked]<mailto:[email-masked]>>
보낸 날짜 : 2017-03-29 23:25:48 ( +09:00 )
받는 사람 : 김민석 <[email-masked]<mailto:[email-masked]>>
참조 : Brian Njenga <[email-masked]<mailto:[email-masked]>>, Jérôme François <[email-masked]<mailto:[email-masked]>>, Oscar Mauricio Caicedo Rendon <[email-masked]<mailto:[email-masked]>>, Sheng Jiang <[email-masked]<mailto:[email-masked]>>, [email-masked]<mailto:[email-masked]> <[email-masked]<mailto:[email-masked]>>
제목 : Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Thx,

Dave

Hey Min-Suk,

Thanks,

Dave

Hi Brian,

Best Regards,

Min-Suk Kim

---

From: David Meyer <[email-masked]>
Date: Wed, 29 Mar 2017 11:18:33 -0700

Minsuk,

Attached are a few quick comments. I'll read more carefully this afternoon.
I also have to read the references.

Thanks,

Dave

On Wed, Mar 29, 2017 at 10:37 AM, 김민석 <[email-masked]> wrote:

> Thank you so much =:)
>
> -Minsuk Kim
>
> Sent from my iPhone
>
> On 29 Mar 2017, at 12:12 PM, David Meyer <[email-masked]> wrote:
>
>
> Hey Min-Suk,
>
>
>> *제목 : *Re: [Idnet] Intelligence-Defined Network Architecture and Call
>> for Interests
>>
>>
>>> expert data like the KGS Go server (https://www.gokgs.com/) to build
>>> the supervised policy network....).
>>>
>>>
>>> [BTW, two player minimax games seem to be popping up everywhere:
>>> AlphaGo, variational autoencoders [4], GANs [5], and many others; something
>>> to thing about for our domain]
>>>
>>>

---

From: Sheng Jiang <[email-masked]>
Date: Wed, 29 Mar 2017 18:31:54 +0000

I am trying set up a meeting place for tomorrow morning 8am~9am, will send the information later to on-site participants directly. Giving that the meetecho is only for formal WG/RG meetings, I am afraid it is almost impossible for remote participant. In other private meeting I was in on Tuesday, we did use webex. But the quality is not good without professional meeting equipment. It was bad to bridge four people on site with one remote participant.

Regards,

Sheng
________________________________________
From: IDNET [[email-masked]] on behalf of Sheng Jiang [[email-masked]]
Sent: 29 March 2017 0:44
To: [email-masked]
Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Sheng

Hi, all,

Best regards,

Sheng

---

From: Michele Zorzi <[email-masked]>
Date: Thu, 30 Mar 2017 00:24:19 +0200 (CEST)

Hi all,

as I think I mentioned some time back, there is an interesting venue for 
the type of work this community would like to promote, please check 
www.comsoc.org/tccn

Please note that, although "cognitive communications and networking" may 
immediately ring the "cognitive radio" bell and thereby raise negative 
feelings in people not interested in PHY and radio, our interpretation 
tries to be much broader, and in fact we are trying to promote a vision in 
which cognition is applied at all layers of the protocol stack and across 
the network, with machine learning techniques and software defined 
networking as two key ingredients.

As the EiC of this journal, I see it as a great opportunity if this 
community wants to start promoting good research in this area and I am 
willing to support it, e.g., through publishing (after rigorous 
peer-review of course) a good position paper/research roadmap coming from 
networking people.

Best regards and good luck with your meeting,

Michele

---

From: Manav Bhatia <[email-masked]>
Date: Wed, 29 Mar 2017 07:43:20 +0530

Hi,

I am looking at machine learning for predictive analytics. We are building
an SDN controller that has access to large volumes of network data and
events. We can look at traffic shifts over a period of time and look for
underlying patterns in that and use that to predict how the network
services could be provisioned.

Do we have any use case document that describes how we could apply machine
learning/advanced analytics principles in the networking domain?

Cheers, Manav

---

From: Sheng Jiang <[email-masked]>
Date: Thu, 30 Mar 2017 14:41:40 +0000

Hi, David,

I think I agree with you, but in slight different  expression. Yes, the hard parts of getting ML into Network lies on machine learning. But, it is not that we need to develop any new ML technical/algorithms for networking in particular. It is that we MUST re-set up our network domain knowledge from the perspective of applying ML. My slides [0] does not suggest that *someone else* will handle the ML part. Actually, oppositely, it suggests some experts who have knowledge of both ML and network (probably we) would develop tools/algorithms/systems to handle the ML part for other network experts (more than 98 percent of current network administrators). So that, these network experts would be allowed to manage their network easily with intelligence association, but no need to become ML experts themselves. Here, we would like to treat the network administrators like the users in other successful ML application. We are the domian experts to do the dirty AI work for them.

I believe we have common understanding in the above description. But certainly my slides needs further refine to clarify my viewpoint.

Best regards,

Sheng
________________________________
From: IDNET [[email-masked]] on behalf of David Meyer [[email-masked]]
Sent: 29 March 2017 2:01
To: Sheng Jiang
Cc: [email-masked]
Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

On Tue, Mar 28, 2017 at 10:59 AM, David Meyer <[email-masked]<mailto:[email-masked]>> wrote:
Hey Sheng,

I just wanted to revive my key concern on [0] (same one I made at the NMRL): The hard parts of getting Machine Learning intelligence into Networking is the Machine Learning part. In addition, successful deployment of ML requires knowledge of ML combined with domain knowledge. We definitely have the domain knowledge; the problem is that we don't have the ML knowledge, and this is one of the big factors holding us back; see e.g. Andrew's discussion of talent in [1].  Slides such as [0] seem to imply that *someone else* (in particular, not us)  will handle the ML part of all of this. I'll just note that in general successful deployments of ML don't work this way; the domain experts will have to learn ML (and vice versa) for us to be successful (again, see [1] and many others).

Perhaps a useful exercise would be to write an ID that makes your assumptions explicit?

Thanks,

Dave

[0] https://www.ietf.org/proceedings/97/slides/slides-97-nmlrg-intelligence-defined-network-01.pdf
[1] https://hbr.org/2016/11/what-artificial-intelligence-can-and-cant-do-right-now

On Tue, Mar 28, 2017 at 9:29 AM, Sheng Jiang <[email-masked]<mailto:[email-masked]>> wrote:
Hi, all,

Although it might be a little bit too early for AI/ML in network giving the recent story of the concluded proposed NMLRG, we still would like to call for interests in IDN. Anybody (on site in Chicago this week) are interested in this or even wider topics regarding to AI/ML in network, please contact me on [email-masked]<mailto:[email-masked]> . Then we may have an informal meeting to discuss some common interests and potential future activities (not any activities in IETF, but also other STO or experimental trails, etc.)  on Thursday morning.

Best regards,

Sheng

---

From: "Wei Jiang" <[email-masked]>
Date: Thu, 30 Mar 2017 16:43:00 +0200

Hi Manav,

I am working on SELFNET project (one of the nineteen H2020 5G-PPP Phase 1 projects: https://5g-ppp.eu/5g-ppp-phase-1-projects/ ), which was driven by the motivation of applying Machine Learning to autonomically manage the SDN/NFV-based 5G networks (https://selfnet-5g.eu/ ).

Three Use Cases, i.e., Self-Healing, Self-Protection, and Self-Optimization, are exploring, and have been specified in D2.2 (https://bscw.selfnet-5g.eu/pub/bscw.cgi/d18751-4/*/*/*/*/DOI-D2.1.html ).

Don’t know whether we can find some synergies.

My 2 cents.

Best regards,

Wei Jiang

From: IDNET [mailto:[email-masked]] On Behalf Of Manav Bhatia
Sent: Wednesday, March 29, 2017 4:13 AM
To: Sheng Jiang
Cc: [email-masked]
Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Hi,

I am looking at machine learning for predictive analytics. We are building an SDN controller that has access to large volumes of network data and events. We can look at traffic shifts over a period of time and look for underlying patterns in that and use that to predict how the network services could be provisioned.

Do we have any use case document that describes how we could apply machine learning/advanced analytics principles in the networking domain?

Cheers, Manav

On Tue, Mar 28, 2017 at 9:59 PM, Sheng Jiang <[email-masked] <mailto:[email-masked]> > wrote:

Hi, all,

Although it might be a little bit too early for AI/ML in network giving the recent story of the concluded proposed NMLRG, we still would like to call for interests in IDN. Anybody (on site in Chicago this week) are interested in this or even wider topics regarding to AI/ML in network, please contact me on [email-masked] <mailto:[email-masked]>  . Then we may have an informal meeting to discuss some common interests and potential future activities (not any activities in IETF, but also other STO or experimental trails, etc.)  on Thursday morning.

https://portal.etsi.org/tb.aspx?tbid=844 <https://portal.etsi.org/tb.aspx?tbid=844&SubTB=844> &SubTB=844
https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=51011

Best regards,

Sheng

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 31 Mar 2017 04:37:00 +1300

Agreed, and there are (still) two key points:

1. What is our underlying model (what Dave called a "theory of networking")? With no such model, it's very hard to tell the ML system what to do.

2. And as others have said: get hold of large datasets that can processed by ML according to that model. For developing open solutions, a corpus of open data sets seems essential. As anybody from the network measurement community will tell you, getting hold of large data sets from operators is extremely difficult for both privacy and commercial reasons.

   Brian

---

From: Aydin Ulas <[email-masked]>
Date: Thu, 30 Mar 2017 19:22:18 +0300

Dear all,
As a researcher having almost 20 years of experience in machine learning
and working as a solution architect on SDN and NFV for the past four years,
I would like to offer my two cents.
I believe that I have the unique perspective to look at the problem from
the other angle and this mailing list is the perfectly suited ground for
it. The problem (or the solution as David suggested) is the lack of
standardized data sets and also the problems defined over those data sets.
In the machine learning area, we had this luxury almost since the beginning
(see ML repository: https://archive.ics.uci.edu/ml/about.html); it makes
the job relatively easy and also comparable since researchers are working
on the same problem, trying to do better, and also exchanging information.
Of course, with the rise of the Internet, smart phone revolution and the
huge amount of data produced in the last decade, combined with privacy
concerns, things have not been like they were before (Netflix reverse
engineering scandal did not help either) but people are still donating
their data for researchers to use because the community is *more
research,* *less
industry* oriented. Apart from medical data (where doctors are more
conservative to give you and privacy has a stronger impact), the trend
continues to this day and makes it easier for the likes of us. Instead, in
the networking community, as far as I have seen in the past five years,
usually the operators own the data and they are reluctant (also privacy is
an important aspect) to release this data (even an obfuscated version of
it) even for their own needs. I have been trying to detect a protocol using
packets for the past two weeks and the operator which requires this feature
does not provide us the data to work on.

To draw researchers from Machine Learning area what should be done in my
opinion is to provide well-defined problems with data supporting them and
the rest will definitely come in rapid succession. For the data, of course
supervised learning is the most established and easier to apply technique
(so labels are important) but you do not always need labels (at least all
of them) to apply semi-supervised and non-supervised learning algorithms.
Most algorithms are fault tolerant, fault being missing features, missing
labels, etc. There are techniques which support multiple instances,
dissimilarities between objects that I have yet to see in the networking
domain, which could bring unique perspectives to problems. I am trying to
persuade my colleagues who have ML experience to work on networking
problems but it is really hard to provide them the problem and the data
which could get the ball rolling.

Best regards,
Aydin Ulas, PhD,
Argela A.S., Bogazici University, MEF University
https://www.cmpe.boun.edu.tr/~ulas/

On Thu, Mar 30, 2017 at 6:37 PM, Brian E Carpenter <
[email-masked]> wrote:

> 1. What is our underlying model (what Dave called a "theory of
> networking")? With no such model, it's very hard to tell the ML system what
> to do.
>
> 2. And as others have said: get hold of large datasets that can processed
> by ML according to that model. For developing open solutions, a corpus of
> open data sets seems essential. As anybody from the network measurement
> community will tell you, getting hold of large data sets from operators is
> extremely difficult for both privacy and commercial reasons.
>
>    Brian
>
>
> > I think I agree with you, but in slight different  expression. Yes, the
> hard parts of getting ML into Network lies on machine learning. But, it is
> not that we need to develop any new ML technical/algorithms for networking
> in particular. It is that we MUST re-set up our network domain knowledge
> from the perspective of applying ML. My slides [0] does not suggest that
> *someone else* will handle the ML part. Actually, oppositely, it suggests
> some experts who have knowledge of both ML and network (probably we) would
> develop tools/algorithms/systems to handle the ML part for other network
> experts (more than 98 percent of current network administrators). So that,
> these network experts would be allowed to manage their network easily with
> intelligence association, but no need to become ML experts themselves.
> Here, we would like to treat the network administrators like the users in
> other successful ML application. We are the domian experts to do the dirty
> AI work for them.
> >
> > I believe we have common understanding in the above description. But
> certainly my slides needs further refine to clarify my viewpoint.
> >
> > Best regards,
> >
> > Sheng
> please contact me on [email-masked]<mailto:[email-masked]>
> . Then we may have an informal meeting to discuss some common interests and
> potential future activities (not any activities in IETF, but also other STO
> or experimental trails, etc.)  on Thursday morning.
> >