Subject: [Maprg] follow-up "Recursives in the Wild" presentation @ietf100

From: "Giovane C. M. Moura" <[email-masked]>
Date: Tue, 14 Nov 2017 10:48:28 +0800

Hi,

So yesterday I presented our "Recursives in the Wild" paper [1,2] at the
MAPRG session (IETF100). Our main finding was that, in the wild,
recursives resolvers will query all available authoritative servers
available for a given DNS zone, even if there's a long delay between the
recursive resolver and authoritative. Recursive resolvers distribute
their queeries roughly inversely proportional to the RTT between
recursives and authoritatives.

>From this findings, our main recommendation is to use *anycast* in every
single authoritative server to reduce latency. Unicast cannot deliver
good latency to all resolvers globally simply because some clients will
be geographically too far from a unicast authoritative (implying long
delays).

After the presentation, there was a question about our recommendation,
that anycast for all authoritatives may not be good idea because of ICMP
IPv6 fragmentation issues issues [3].  In short: on IPv6 only
environments, some resolvers may not be able to receive fragmented
packets on IPv6 only environments.

This is a problem which is orthogonal to ours. Our goal was to
understand how recursives in the wild choose authoritative servers, so
we can better engineer them. Our findings hold for both IPv4 and IPv6
stacks: to deliver lower latency, you have to use anycast, in all your
authoritatives.

In fact, this is already used in  many production environments:
 * 12 of the 13 root server letters[4]
 * DNS providers (e.g: Amazon Route 53[5], Dyn[6], Netnod[7]),
 * many ccTLDs (as well as the authoritative name servers (ns5) we run
at our .nl ccTLD).

So in short: anycast is a robust technology that has been used in
production environments for quite some years. And it is the best way to
deliver reduced latency.

The question of fragmentation of IPv6 only environments is important,
and maybe there's an opportunity for an *open datasets* measurement
paper in here, so other people can reproduce and analyze the results,
and determine if this is a problem in the wild (actually DNS-OARC DITL
datasets[8] is a good starting point for anyone wanting to drill down on
this -- it has pcap data from the roots servers). Thanks Geoff for
bringing this up.

However, this does not change any of our findings in the paper. Anycast
is the way forward to reduce latency, for both IPv4 and IPv6, and we
(.nl) are moving towards that too.

/giovane

[1]
https://datatracker.ietf.org/meeting/100/materials/slides-100-maprg-recursives-in-the-wild-engineering-authoritative-dns-servers-giovane-c-m-moura/
[2] https://conferences.sigcomm.org/imc/2017/papers/imc17-final12.pdf
[3]
https://pc.nanog.org/static/published/meetings/NANOG71/1465/20171003_Huston_Ipv6_The_Dns_v1.pdf
[4] http://root-servers.org/
[5]
https://aws.amazon.com/about-aws/whats-new/2016/10/amazon-route-53-now-supports-dns-queries-over-ipv6-networks
[6] https://dyn.com/dns/managed-dns/
[7] https://www.netnod.se/dns/netnod-dns-services
[8] https://www.dns-oarc.net/oarc/data/ditl

---

From: Geoff Huston <[email-masked]>
Date: Wed, 15 Nov 2017 00:26:33 +1100

> On 14 Nov 2017, at 1:48 pm, Giovane C. M. Moura <[email-masked]> wrote:
> 
> Hi,
> 

It appears from this explanation that you may not have understood the question.

The problem here is that the ICMPv6 PTB messages that are sent in response
to a PTB issue on the path are sent may not necessarily arrive at the anycast instance
that sent the original packet - the return path of the response is not necessarily the same
as the forwarding path of the query and the “closest” anycast instance
of the routers on the return path may be different from the sending anycast
instance.

What this means it that anycast should not be used with impunity - you need to
be careful about path MTU considerations as you may not receive the
conventional ICMP signalling back to the same anycast instance.

Fragmentation in IPv6 is its own special problem, irrespective of anycast.

Geoff

---

From: Geoff Huston <[email-masked]>
Date: Wed, 15 Nov 2017 00:27:20 +1100

Geoff

---

From: Tobias Fiebig <[email-masked]>
Date: Tue, 14 Nov 2017 14:42:07 +0100

Heho,

> > be geographically too far from a unicast authoritative (implying long

To pic up on this, because i have been wondering about it for some time now: Does that not defeat
the purpose of having multiple authoritative servers set for DNS zones? Given that one deploys
anycast for all auth. NS, queries from a client, in my understanding, would mostly arrive at the
(geographically) same node. Hence an issue at that one node, e.g., a faulty Bind refusing to start
or a dDoS to the anycast zone, will make the whole zone unavailable to clients of a specific region,
as all IP traffic for any auth NS will be directed to a potentially unresponsive AS, requiring semi-
manual failover (retracting the network used for anycast)?

This would not be the case when using the DNS "redundancy measure", which, while introducing latency
due to (potential timeouts) provides better resilience?

With best regards,
Tobias

-- 
Tobias Fiebig                      building MAR, 4th floor, room 4.003
Fachgebiet INET - Sekr. MAR 4-4        phone: +49 162 243 205 2
Technische Universität Berlin         gpg-fp: 21A9 91EB 4971 ED48 001E
Marchstrasse 23 - 10587 Berlin                FBBC B6AF 3CD8 59A2 795B

---

From: Moritz Muller <[email-masked]>
Date: Tue, 14 Nov 2017 19:32:22 +0000

Hi,

I’m one of the authors of the paper Giovane presented at the maprg.

You’re raising a good point Tobias.
I think it does not make sense to have different NSes all sharing the same sites.
We recommend having different anycast services for the NSes, equally well peered, but not peered equally.

In the paper we mention a couple of things you should consider when deploying anycast on all your NSes but maybe we should have been more explicit.
And overall a DNS operator should always find a balance between performance and stability.

Cheers,
Moritz

> On 14 Nov 2017, at 14:42, Tobias Fiebig <[email-masked]> wrote:
> 
> Heho,
> 
> _______________________________________________
> Maprg mailing list
> [email-masked]
> https://www.irtf.org/mailman/listinfo/maprg

---

From: Geoff Huston <[email-masked]>
Date: Wed, 15 Nov 2017 08:35:12 +1100

> On 15 Nov 2017, at 12:42 am, Tobias Fiebig <[email-masked]> wrote:
> 
> Heho,
> 

It would be cynical to observe that current fashion is to address an issue at every level of the
protocol stack - simultaneously. So while there is an application level response in the DNS,
obviously the IP folk felt left out, hence anycast. After all too much is always better, right? :-)

As I recall, when anycast was first considered, much of the motivation was to partition DOS attacks,
so that an attack might take out one or two instances of the authoritative servers but it would be
harder to take out all of them. We seem to have changed the motivational story behind anycast
from damage limitation to some mythical concept of improved performance. I don’t quite understand 
the performance argument, given that caching is meant to be the major performance lever in
name resolution, and in the wonderful world of caching the only queries (other than cache 
miss/refresh) that head directly to the authoritative servers are for non-existent names. So is
the more precise statement of the performance story one of “saying no faster”? If thats the case,
then whether you measure the speed of “no” or not, it seems to be rather a fanciful performance
objective!

Geoff

---

From: Tobias Fiebig <[email-masked]>
Date: Tue, 14 Nov 2017 23:27:31 +0100

Heho,
> obviously the IP folk felt left out, hence anycast. After all too much is always better, right? :-
Well... i think i may have to plead guilty here... i will try to limit myself on that front... 

> I don’t quite understand the performance argument, given that caching is meant to be the major
> performance lever in name resolution, and in the wonderful world of caching the only queries
> (other than cache miss/refresh) that head directly to the authoritative servers are for non-
> existent names. So is the more precise statement of the performance story one of “saying no
> faster”? If thats the case, then whether you measure the speed of “no” or not, it seems to be
> rather a fanciful performance objective!
The underlying problem is, in my understanding, that the modern(tm) web evolved to a rather request-
heavy and latency driven place. This increases the importance of DNS in the overall latency of a
websession.

When thinking about a small, standard site, we commonly have some (or all) of the following
resources involved:
- The frontend.example.org host, mostly pushing a small php application written a decade ago which
has grown to a javascript delivery platform
- api(v1|v2).example.org to feed the frontend with data via some json-rest-stateless-plain-http-api
- (eu|na|sa|af|ap)-example-org.somecdn.org for the static content larger than 4kb
- around 50 different hosts supplying various forms of ads, tracking, and, ad-tracking in minified-
javascript spaghetti code

Sadly, in many websites, these also build iteratively upon each other, so you are not resolving 50
names at once, but more-or-less in sequence.

The problem here now is, assuming made up numbers for illustrative purposes, that if all
authoritative servers for a zone are (potentially) the first to be contacted, or, during a
subsequent request for a name the one to be contacted, this my introduce a cascading latency spike.
For example, if you run the authoritative servers for .org, and the site in question has a lot of
different slds under .org, and your six authoritatives are evenly spread around the globe, the
recursor on a customers CPE will only query a geographically close server in 1/6th of the cases---
given that the CPE does simple round-robbin---which may lead to a significantly higher site load
time than it _would have to be_ if all those queries would go to a geographically close server.
After all, 50*70ms is quiet some time, if you could have had 50*15ms.

So, to reach this optimal 50*15ms loadtime, the authors now propose (and demonstrate that it reduces
load-time) to have, instead of six authoritative servers spread around the globe, six anycast zones,
all with members spread around the globe. That way, every user will always have six authoritative
servers in her own region, and always enjoy a 15ms latency to the next authoritative server.

So, in my opinion this proposal aims at solving the same problems that brought us syn-with-data:
Cascading latencies in the overly fragmented web eccosystem that created use-cases that were not
anticipated during the initial design of the protocols running the Internet.

With best regards,
Tobias

---

From: Geoff Huston <[email-masked]>
Date: Wed, 15 Nov 2017 10:01:43 +1100

> On 15 Nov 2017, at 9:27 am, Tobias Fiebig <[email-masked]> wrote:
> 
> Heho,

So your response is that you are engineering for the cache miss. Seems a bit silly when you could be
engineering to figure out how to optimise the cache. If the app is so concerned about this then why not 
perform the DNS queries well before the point when you need to actually get the named object?

i.e. if the app is having a hard time why are you making this a problem that the DNS authoritative servers
feel that have to solve when the app itself could solve it alone?

>> protocol stack - simultaneously.

qed 

:-)

Geoff

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 15 Nov 2017 00:28:09 +0100 (CET)

On Wed, 15 Nov 2017, Geoff Huston wrote:

> It would be cynical to observe that current fashion is to address an 
> issue at every level of the protocol stack - simultaneously. So while 
> there is an application level response in the DNS, obviously the IP folk 
> felt left out, hence anycast. After all too much is always better, 
> right? :-)

To make something work best, yes, you need redundancy on all levels. Often 
this redundancy solves different problems, and to solve the most problems 
that are solvable, you need multiple redundancy methods.

Anycast mitigates the DDoS problem, having multiple anycast cluster 
operators mitigates some of the logical/configuration errors that might 
occur within one entity, and having non-anycasted servers mitigates the 
shared resources problem (you being taken down by someone DDoS:ing someone 
else that might sit on the same anycast clusters).

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Wed, 15 Nov 2017 07:38:15 +0800

> of the routers on the return path may be different from the sending anycast

An interesting hypothesis. Thanks for pointing it. However, how
frequently does this happen? Maybe something worth measuring in another
open datasets measurement paper.

Even if it does, we recommend using dual-stack anycast multiple
authoritative name servers. If one router in the path cannot forward a
packet too big, too bad, the resolver with either move to the next
authoritative or use IPv4, ultimately delivering the answer requested.

> conventional ICMP signalling back to the same anycast instance.

As any technology, anycast requires good engineering and planning, and
testing. And measuring it.  But it is robust and already widely
deployed. Corner cases should be covered by dual-stack multi NS
deployments.

/giovane

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Wed, 15 Nov 2017 07:55:15 +0800

> due to (potential timeouts) provides better resilience?

This is not what we recommend. We recommend using anycast on every NS,
not using the same infrastructure for every NS. The scenario your
describe is, in practice a single NS scenario! See our IEPG presentation
on that[1].

Let's say you run a large website with four NSes. You don't want to have
all your NS on the same infrastructure  -- the scenario you described.
In the case of a DDoS that is bigger than what your infra can handle,
all of your NS will go down. That happened before, leaving many clients
unreachable[2].

The key here is *diversity*. You don't want to run your four NS on the
same infra[1], in fact, you should diversify. If you run a large
website, maybe you could run 4 NSes: 2 on two different anycast networks
you maintain (to avoid collateral damage -- see about it on [3]) and the
two others on two different DNS providers.

In this way, you have a good mix, which it makes it more difficult to
bring your NSes down.

/giovane

[1] http://iepg.org/2017-11-12-ietf100/02-moura-iepg-ietf100-overlap.pdf
[2]
https://www.nytimes.com/2016/10/22/business/internet-problems-attack.html
[3] https://www.isi.edu/~johnh/PAPERS/Moura16b.html

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Wed, 15 Nov 2017 08:21:31 +0800

> harder to take out all of them.
We have already taken a look on the behavior of anycast and DDoS
attacks, and indeed it fulfills its goal [1].

> We seem to have changed the motivational story behind anycast
> miss/refresh) that head directly to the authoritative servers are for
non-existent names.

Actually there's a different reasoning. At .nl, we noticed that 23% of
our traffic to 4 NSes geo-located in the Netherlands was coming from the
US, experiencing RTT > 100ms. We are talking hundreds of millions of
daily valid queries.

On the other hand, the same NSes were delivering RTTs < 10ms to nl-based
resolvers.

Given we have a globally diverse query distribution, how can we deliver
to all our global clients short response times? First, we needed to
understand how resolvers choose authoritatives, and that is what we did
with our IMC2017 paper.

We learned that recursives will use all available NSes. Therefore, to
improve the performance of resovlers such as the US-based ones we see,
the only way to do it efficiently is to use anycast, and place sites
nearby your biggest clients.

And as bonus for using anycast, you are likely to have better DDoS
resilience.

/giovane

---

From: Tobias Fiebig <[email-masked]>
Date: Wed, 15 Nov 2017 09:46:06 +0100

Heho,
> So your response is that you are engineering for the cache miss. 
Just... for the record... I am not an author of the paper we are discussing, and I am honestly not
convinced that the proposed solution solves the root issues in how the web evolved. Instead, it
tries to adjust existing practices without changing protocols to fix a design issue somewhere else:
On the web-application's part.

I agree with the authors, that the problem they are trying to solve is real, and that their solution
solves/reduces it. I simply do not think that the solution is the right way to solve things in the
context of DNS.

> Seems a bit silly when you could be engineering to figure out how to optimise the cache. If the
> app is so concerned about this then why not perform the DNS queries well before the point when you
> need to actually get the named object?
Because, due to the (bad) design of many webapps, this is not possible. You first have to have the
client request all the JS you crammed into a site, so the client can then process the JS, only then
can it determine the names needed in that JS and so forth.

Sadly, you can not pre-determine the DNS names as well, as some are dynamically generated by the
client, based on, e.g., local properties, and because several pieces of JS are included from remote
places you do not even have control over as a site operators. So, for example a blob with all names
sent with the root page, while it would reduce the problem (100ms would not be an issue for UX
anymore, as it would only be 100ms once, in parallel, best-case), is not a feasible solution.

> i.e. if the app is having a hard time why are you making this a problem that the DNS authoritative
> servers feel that have to solve when the app itself could solve it alone?
Probably because the app developers do not necessarily care, while the DNS operators want to improve
UX for all clients, and hence used a lever they do actually have control about.

With best regards,
Tobias

---

From: Tobias Fiebig <[email-masked]>
Date: Wed, 15 Nov 2017 09:51:46 +0100

Heho,
> In this way, you have a good mix, which it makes it more difficult to
I did not suggest using the same anycast infrastructure. Only that it will probably lead to six
nodes from six anycast pools sitting in one geographic location. At the moment, my anecdotal
perception is "all .nl auth servers are directly behind amsix."

I think that this should actually be a matter for further work:

Traceroute the auth-NS from all atlas nodes, check how often the paths converge, and how many SPOFs
there could be (large IX directly in front of all nodes, one big DC operator in a (small) region
hosting all nodes etc.)

I know that ideally this should not be the case. But well... things tend to go wrong.

With best regards,
Tobias

---

From: "Giovane C. M. Moura" <[email-masked]>
Date: Thu, 16 Nov 2017 11:07:47 +0800

> nodes from six anycast pools sitting in one geographic location. 

Not a problem per se; the roots have 13 letters, but some letters have
sites in the same city. See[1,2].

It all depends on how you design your anycast network. you can have 100s
of sites if you want, on very diff locations. And on the internet, what
matters is your peering, not the geo location. You can have 6 sites in
AMS, on 6 diff datacenters/ASes, totally isolated from each other.

>At the moment, my anecdotal

Not true. We have 8 NSes, some anycast with more than 50 sites, some
third parties, some global dns providers...
And I dunno any DNS provider that would config such a dangerous SPOF
setup. If you have evidence of any, let us know.

> I think that this should actually be a matter for further work:
thats the conclusion on [3]

> there could be (large IX directly in front of all nodes, one big DC operator in a (small) region

That's a new neat trick that does something similar that was presented
at IMC2017 [4]  :)

/giovane

[1] http://root-servers.org/
[2] https://www.isi.edu/~johnh/PAPERS/Moura16b.html
[3]  http://iepg.org/2017-11-12-ietf100/02-moura-iepg-ietf100-overlap.pdf
[4] https://conferences.sigcomm.org/imc/2017/papers/imc17-final106.pdf