Subject: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

From: "dingxiaojian (A)" <[email-masked]>
Date: Fri, 31 Mar 2017 01:04:17 +0000

Hi Brian,
    You are right.
     I have worked in an institute more than five years. The main work is  use ML to solve the domain problem.  However, for privacy reason, it's very hard to get some real domain data sets. So the results learned by any ML models is not reliable.
    So I think the important/first thing we do is to construct real and reliable data sets of network domain. Just like UCI repository (https://archive.ics.uci.edu/ml/datasets.html) or images datasets (https://www.cs.utah.edu/~lifeifei/datasets.html) . Only the common and real data sets are agreed with all we, different ML models can be applied to validate and predict. 
   So my point is data set, not ML model.  We can select some existing models (SVM, ELM, bayes, etc) to learn different tasks. We do not research the principle of ML algorithm, but use them to slove network problems.

 Best regards,
 
Xiaojian

-----邮件原件-----
发件人: IDNET [mailto:[email-masked]] 代表 Brian E Carpenter
发送时间: 2017年3月30日 23:37
收件人: Sheng Jiang <[email-masked]>; David Meyer <[email-masked]>
抄送: [email-masked]
主题: Re: [Idnet] Intelligence-Defined Network Architecture and Call for Interests

Agreed, and there are (still) two key points:

1. What is our underlying model (what Dave called a "theory of networking")? With no such model, it's very hard to tell the ML system what to do.

2. And as others have said: get hold of large datasets that can processed by ML according to that model. For developing open solutions, a corpus of open data sets seems essential. As anybody from the network measurement community will tell you, getting hold of large data sets from operators is extremely difficult for both privacy and commercial reasons.

   Brian

On 31/03/2017 03:41, Sheng Jiang wrote:
> Hi, David,
> 
> I think I agree with you, but in slight different  expression. Yes, the hard parts of getting ML into Network lies on machine learning. But, it is not that we need to develop any new ML technical/algorithms for networking in particular. It is that we MUST re-set up our network domain knowledge from the perspective of applying ML. My slides [0] does not suggest that *someone else* will handle the ML part. Actually, oppositely, it suggests some experts who have knowledge of both ML and network (probably we) would develop tools/algorithms/systems to handle the ML part for other network experts (more than 98 percent of current network administrators). So that, these network experts would be allowed to manage their network easily with intelligence association, but no need to become ML experts themselves. Here, we would like to treat the network administrators like the users in other successful ML application. We are the domian experts to do the dirty AI work for them.
> 
> I believe we have common understanding in the above description. But certainly my slides needs further refine to clarify my viewpoint.
> 
> Best regards,
> 
> Sheng
> ________________________________
> From: IDNET [[email-masked]] on behalf of David Meyer 
> [[email-masked]]
> Sent: 29 March 2017 2:01
> To: Sheng Jiang
> Cc: [email-masked]
> Subject: Re: [Idnet] Intelligence-Defined Network Architecture and 
> Call for Interests
> 
> s/NMRL/NMLRG/   (sorry about that). Dave
> 
> On Tue, Mar 28, 2017 at 10:59 AM, David Meyer <[email-masked]<mailto:[email-masked]>> wrote:
> Hey Sheng,
> 
> I just wanted to revive my key concern on [0] (same one I made at the NMRL): The hard parts of getting Machine Learning intelligence into Networking is the Machine Learning part. In addition, successful deployment of ML requires knowledge of ML combined with domain knowledge. We definitely have the domain knowledge; the problem is that we don't have the ML knowledge, and this is one of the big factors holding us back; see e.g. Andrew's discussion of talent in [1].  Slides such as [0] seem to imply that *someone else* (in particular, not us)  will handle the ML part of all of this. I'll just note that in general successful deployments of ML don't work this way; the domain experts will have to learn ML (and vice versa) for us to be successful (again, see [1] and many others).
> 
> Perhaps a useful exercise would be to write an ID that makes your assumptions explicit?
> 
> Thanks,
> 
> Dave
> 
> 
> [0] 
> https://www.ietf.org/proceedings/97/slides/slides-97-nmlrg-intelligenc
> e-defined-network-01.pdf [1] 
> https://hbr.org/2016/11/what-artificial-intelligence-can-and-cant-do-r
> ight-now
> 
> 
> On Tue, Mar 28, 2017 at 9:29 AM, Sheng Jiang <[email-masked]<mailto:[email-masked]>> wrote:
> Hi, all,
> 
> Although there are many understanding for Intelligence-Defined Network, we are actually using this IDN as a term reference to the SDN-beyond architecture that we presented in IETF97, see the below link. A reference model is presented in page 3, while potential standardization works is presented in page 9.
> 
> https://www.ietf.org/proceedings/97/slides/slides-97-nmlrg-intelligenc
> e-defined-network-01.pdf
> 
> Although it might be a little bit too early for AI/ML in network giving the recent story of the concluded proposed NMLRG, we still would like to call for interests in IDN. Anybody (on site in Chicago this week) are interested in this or even wider topics regarding to AI/ML in network, please contact me on [email-masked]<mailto:[email-masked]> . Then we may have an informal meeting to discuss some common interests and potential future activities (not any activities in IETF, but also other STO or experimental trails, etc.)  on Thursday morning.
> 
> FYI, we have already working on a Work Item, called IDN in the ETSI NGP (Next Generation Protocol) ISG, links below.
> 
> https://portal.etsi.org/tb.aspx?tbid=844&SubTB=844
> https://portal.etsi.org/webapp/WorkProgram/Report_WorkItem.asp?WKI_ID=
> 51011
> 
> Meanwhile, please do use this mail list as a forum to discuss any topics that may applying AI/ML into network area.
> 
> Best regards,
> 
> Sheng
> _______________________________________________
> IDNET mailing list
> [email-masked]<mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/idnet
> 
> 
> 
> 
> 
> _______________________________________________
> IDNET mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/idnet
> 

_______________________________________________
IDNET mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/idnet

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Fri, 31 Mar 2017 12:07:20 +0900

Hi all,

That's it and I insist that, if anybody in the mailing list is bound to
an operator or a similar organization that manages huge amounts of data,
just the data we need, please try to provide it. We will be able to help
you to build some anonymization solution, if needed. Thank you.

Regards,
Pedro

-- 
Pedro Martinez-Julia
Network Science and Convergence Device Technology Laboratory
Network System Research Institute
National Institute of Information and Communications Technology (NICT)
4-2-1, Nukui-Kitamachi, Koganei, Tokyo 184-8795, Japan
Email: [email-masked]
---------------------------------------------------------
*** Entia non sunt multiplicanda praeter necessitatem ***

---

From: Laurent Ciavaglia <[email-masked]>
Date: Fri, 31 Mar 2017 06:12:19 +0200

Hello,

A couple of suggestions:
     -Why not raise this point / request in various WGs (e.g. in the OPS 
Area) and RGs (e.g. MAPRG, NMRG).
     -Why not work with the IETF meeting NOC to collect data sets?

BR, Laurent.

-- 

Laurent Ciavaglia

Nokia, Bell Labs

+33 160 402 636

route de Villejust - Nozay, France

linkedin.com/in/laurent.ciavaglia

---

From: Laurent Ciavaglia <[email-masked]>
Date: Fri, 31 Mar 2017 06:15:55 +0200

Also CAIDA has multiple measurement tools to collect data.
Some IETFers are specialists of these tools and report findings on data 
collected via these tools.

Might be something to look at.

Also, and with this I'm finished for today, documenting the 
characteristics of the requested data set(s) might help.

BR, Laurent.

On 31/03/2017 06:12, Laurent Ciavaglia wrote:
> Hello,
>
> A couple of suggestions:
>     -Why not raise this point / request in various WGs (e.g. in the 
> OPS Area) and RGs (e.g. MAPRG, NMRG).

-- 

Laurent Ciavaglia

Nokia, Bell Labs

+33 160 402 636

---

From: Jérôme François <[email-masked]>
Date: Fri, 31 Mar 2017 06:57:29 +0200

Hi,

Data is primordial but sharing them is very difficult (I'm not an
operator) due to privacy but also legislative issues.
Here are few comments:
1) on anonymization: this far from being simple and I do not believe
that anybody can guarentee that nothing will be leaked out as sensitive
data (and at the end this will the responsability of the operator who
have shared the data).
-> That is why sharing but not in public manner (with a NDA) is still
the most viable solution (in my opinion)

2) charaterizing the data we would like to access is clearly a first
step (even if many of us like to have raw data)

3) Another possibility is to run the algorithms you want to test on a
remote platform without accessing the data and get back the results
(also here there might be some safguards to avoid that program tries to
extract sensitive data as results !)

jerome

Le 31/03/2017 à 06:15, Laurent Ciavaglia a écrit :
> Some IETFers are specialists of these tools and report findings on
> data collected via these tools.
>

---

From: Lori Jakab <[email-masked]>
Date: Fri, 31 Mar 2017 11:12:39 +0300

On 3/31/17 7:57 AM, Jérôme François wrote:
> that anybody can guarentee that nothing will be leaked out as
> sensitive data (and at the end this will the responsability of the
> operator who have shared the data).
> (also here there might be some safguards to avoid that program tries
> to extract sensitive data as results !)

I think 3) is probably the most viable way forward for now. Lack of
public data sets is not a new challenge for networking research, and I
doubt saying that we're going to use traffic traces for ML applications
is going to change that. Creating an open source platform, either pure
software or software + hardware (like RIPE Atlas probes) that runs the
algorithms on the data sending back only the results may have a better
chance of success.

-Lori

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Apr 2017 01:40:36 +1300

I discussed this briefly yesterday with Al Morton (IETF BMWG co-chair, and very expert on traffic measurement). He confirms the problem. And of course the available datasets are all pcap format. Does that contain what we need?

Regards
   Brian

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 1 Apr 2017 01:47:21 +1300

> 1) on anonymization: this far from being simple...

Yes, and another issue is that it may even make the data useless. I hit
that problem a few years ago when evaluating stateless hash algorithms for
use in the IPv6 flow label. The problem was that anonymization would change
the addresses in a pcap trace in such a way that some regularity in the original
data would be replaced by pseudo-random bits, which would completely falsify
the results from a statistical viewpoint. I think that in the same way, ML
results from anonymized data might be meaningless - and there is no way to
know that.

Regards
   Brian

---

From: Oscar Mauricio Caicedo Rendon <[email-masked]>
Date: Fri, 31 Mar 2017 08:48:14 -0500

Hi everyone.

I think third option us most viable.

On the other hand, in
https://www.cl.cam.ac.uk/research/srg/netos/projects/brasil/data/index.html
is the BRASIl data set that has been used for IP traffic classification.
Furthermore, in next links there are interesting papers using ML for IP
traffic classification:

- *Lightweight Application Classification for Network Management. *
https://www.cl.cam.ac.uk/research/srg/netos/projects/brasil/publications/2007-jiang2007lightweight.pdf

- *A Machine Learning Approach for Efficient Traffic Classification. *
https://www.cl.cam.ac.uk/research/srg/netos/projects/brasil/publications/2007-li2007machine.pdf

- *Experience with High-Speed Automated Application-Identification for
Network-Management. *
https://www.cl.cam.ac.uk/research/srg/netos/projects/brasil/publications/atoz.ancs09.pdf

Regards,

-- 
*Oscar Mauricio Caicedo Rendón*
*PhD Computer Science - Federal University of Rio Grande do Sul*
*Full Profesor - University of Cauca*

-- 

------------------------------
*Universidad del Cauca: Comprometidos con la calidad.*

---

From: Pedro Martinez-Julia <[email-masked]>
Date: Sat, 1 Apr 2017 00:06:46 +0900

>     -Why not work with the IETF meeting NOC to collect data sets?

Dear Laurent and others,

The main reason I did not consider to request it within/through the IETF
is that I do not know how it would be addressed, so I cannot help in any
aspect. Since I somehow know (have experience with) the process required
to get access to information from operators, I encouraged it.

That said, and considering that operators did not demonstrate interest
on this topic, I request the list that if somebody knows how to address
this request within/through the IETF, please let us know so we can work
together to get it done as best as possible. It would mean a lot for the
network research and engineering community, inside and outside IETF, so
it is worth to try.

Also, please, consider that any kind of data would be worth having. Even
if anonymization breaks something or most of it, since it would be much
more useful than synthetic data. Remember that this is about AI, so the
proposed models and algorithms should also work with adulterated data to
some extent. They would be later validated and tuned using better data
will less (or none) adulteration.

To sum up, any real information is worth to have, regardless of it being
adulterated, filtered, or anything. If anybody knows some way of getting
it or has some examples, please point us to them, as others have already
done, or give us some "easy" guide of how to get the access to it. Thank
you very much.

Regards,
Pedro

---

From: David Meyer <[email-masked]>
Date: Fri, 31 Mar 2017 08:08:18 -0700

I don't think there is any question that there is a problem with data sets;
as many of you know this has been one of my main points regarding what is
holding back ML for networking over the last 4 or so years. Of course, how
to solve this problem is a different question.

However, as least I don't necessarily agree with this statement: " So my
point is data set, not ML model.  We can select some existing models (SVM,
ELM, bayes, etc) to learn different tasks. We do not research the principle
of ML algorithm, but use them to slove [sic] network problems",  we might
check out what Andrew says while discussing what the scare resources
required to make progress in ML for any domain ([0], near the bottom):

o Data. Among leading AI teams, many can likely replicate others’ software
in, at most, 1–2 years. But it is exceedingly difficult to get access to
someone else’s data. Thus data, rather than software, is the defensible
barrier for many businesses.

o Talent. Simply downloading and “applying” open-source software to your
data won’t work. AI needs to be customized to your business context and
data. This is why there is currently a war for the scarce AI talent that
can do this work.

There is a ton of experience (and literature) reinforcing these points, but
suffice it to say that Andrew knows what he's talking about.

In any event, it seems clear that we are all on the same page with respect
to the problems with data sets (again, exactly how to solve this problem is
open). However,  we seem to have a difference in our understanding of both
the field is today and how we make progress. In particular, Andrew seems to
be saying the opposite of what you say above (quoted text). My experience
tracks more with what Andrew is saying.

Summary:  We need to attack both problems.

Dave

[0]
https://hbr.org/2016/11/what-artificial-intelligence-can-and-cant-do-right-now

On Thu, Mar 30, 2017 at 6:04 PM, dingxiaojian (A) <[email-masked]>
wrote:

> Hi Brian,
>     You are right.
>      I have worked in an institute more than five years. The main work is
> use ML to solve the domain problem.  However, for privacy reason, it's very
> hard to get some real domain data sets. So the results learned by any ML
> models is not reliable.
>     So I think the important/first thing we do is to construct real and
> reliable data sets of network domain. Just like UCI repository (
> https://archive.ics.uci.edu/ml/datasets.html) or images datasets (
> https://www.cs.utah.edu/~lifeifei/datasets.html) . Only the common and
> real data sets are agreed with all we, different ML models can be applied
> to validate and predict.
>    So my point is data set, not ML model.  We can select some existing
> models (SVM, ELM, bayes, etc) to learn different tasks. We do not research
> the principle of ML algorithm, but use them to slove network problems.
>
>  Best regards,
>
> Xiaojian
>
>
> -----邮件原件-----
> 1. What is our underlying model (what Dave called a "theory of
> networking")? With no such model, it's very hard to tell the ML system what
> to do.
>
> 2. And as others have said: get hold of large datasets that can processed
> by ML according to that model. For developing open solutions, a corpus of
> open data sets seems essential. As anybody from the network measurement
> community will tell you, getting hold of large data sets from operators is
> extremely difficult for both privacy and commercial reasons.
>
>    Brian
>
>
> > I think I agree with you, but in slight different  expression. Yes, the
> hard parts of getting ML into Network lies on machine learning. But, it is
> not that we need to develop any new ML technical/algorithms for networking
> in particular. It is that we MUST re-set up our network domain knowledge
> from the perspective of applying ML. My slides [0] does not suggest that
> *someone else* will handle the ML part. Actually, oppositely, it suggests
> some experts who have knowledge of both ML and network (probably we) would
> develop tools/algorithms/systems to handle the ML part for other network
> experts (more than 98 percent of current network administrators). So that,
> these network experts would be allowed to manage their network easily with
> intelligence association, but no need to become ML experts themselves.
> Here, we would like to treat the network administrators like the users in
> other successful ML application. We are the domian experts to do the dirty
> AI work for them.
> >
> > I believe we have common understanding in the above description. But
> certainly my slides needs further refine to clarify my viewpoint.
> >
> > Best regards,
> >
> > Sheng
> > I just wanted to revive my key concern on [0] (same one I made at the
> NMRL): The hard parts of getting Machine Learning intelligence into
> Networking is the Machine Learning part. In addition, successful deployment
> of ML requires knowledge of ML combined with domain knowledge. We
> definitely have the domain knowledge; the problem is that we don't have the
> ML knowledge, and this is one of the big factors holding us back; see e.g.
> Andrew's discussion of talent in [1].  Slides such as [0] seem to imply
> that *someone else* (in particular, not us)  will handle the ML part of all
> of this. I'll just note that in general successful deployments of ML don't
> work this way; the domain experts will have to learn ML (and vice versa)
> for us to be successful (again, see [1] and many others).
> >
> > Although there are many understanding for Intelligence-Defined Network,
> we are actually using this IDN as a term reference to the SDN-beyond
> architecture that we presented in IETF97, see the below link. A reference
> model is presented in page 3, while potential standardization works is
> presented in page 9.
> >
> > Although it might be a little bit too early for AI/ML in network giving
> the recent story of the concluded proposed NMLRG, we still would like to
> call for interests in IDN. Anybody (on site in Chicago this week) are
> interested in this or even wider topics regarding to AI/ML in network,
> please contact me on [email-masked]<mailto:[email-masked]>
> . Then we may have an informal meeting to discuss some common interests and
> potential future activities (not any activities in IETF, but also other STO
> or experimental trails, etc.)  on Thursday morning.
> >
> > FYI, we have already working on a Work Item, called IDN in the ETSI NGP
> (Next Generation Protocol) ISG, links below.
> >
> > Meanwhile, please do use this mail list as a forum to discuss any topics
> that may applying AI/ML into network area.
> >
> > Best regards,
> >
> > Sheng