Re: NIST NTP servers

[Steven Miano]

NTP has vulnerabilities, so using an external source opens your networks
and infrastructure to disruptions.

Going with an internal GPS/GLONASS/RADIO based S1 allows you to restrict
incoming traffic and not rely on volunteers or external entities (which may
undergo maintenance or budget issues).

My preference is more so something akin to the GLN180PEX (I am not
affiliated or paid to endorse this product). It allows you to use commodity
hardware (like a decommissioned 1U or several preferably) and creation of
ones own reliable internal time source(s). Introducing black boxes into a
production (revenue generation or expected services by paying customers)
environment is undesirable.

>From there setting up NTPd, Chronyd, and PTPd is up to you.

Relying on satellites may seem like just another external reliance, but the
next life is proposing a design life of 12 years.

On Mon, May 9, 2016 at 11:12 PM, Majdi S. Abbas  wrote:

> On Tue, May 10, 2016 at 03:08:16AM +, Mel Beckman wrote:
> > NTP has vulnerabilities that make it generally unsuitable for
> > provider networks. I strongly recommend getting a GPS-based
> > time server. These are as cheap as $300. Here is one I use quite a bit:
>
> So how does this stop from distributing time to their
> customers via NTP?
>
> GPS doesn't save the protocol, in particular where the S1
> clocks involved are embedded devices with rather coarse clocks and
> timestamping.
>
> --msa
>



-- 
Miano, Steven M.
http://stevenmiano.com

RE: NIST NTP servers

[Chuck Church]

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Majdi S. Abbas

>   So how does this stop from distributing time to their customers via
NTP?
>   GPS doesn't save the protocol, in particular where the S1 clocks
involved are embedded devices with rather coarse clocks and timestamping.
>   --msa

It doesn't really.  Granted there are a lot of CVEs coming out for NTP the
last year or so.  But I just don't think there are that many attacks on it.
It's just not worth the effort.  Changing time on devices is more an
annoyance than anything, and doesn't necessarily get you into a device.
Sure you can hide your tracks a little by altering time in logs and altering
it back, but that's more of an in-depth nation-state kind of attack, not
going to be a script kiddie kind of thing.  Just follow the best practices
for verifying packet sources and NTP security itself, and you should be ok.

Chuck

Re: NIST NTP servers

[Stephane Bortzmeyer]

On Tue, May 10, 2016 at 06:48:52AM -0400,
 Steven Miano  wrote 
 a message of 41 lines which said:

> Going with an internal GPS/GLONASS/RADIO based S1 allows you to
> restrict incoming traffic and not rely on volunteers or external
> entities (which may undergo maintenance or budget issues).

You mean the GPS network is not managed by an external entity? With
budget issues?

http://www.schriever.af.mil/GPS

Re: NIST NTP servers

[Valdis . Kletnieks]

On Tue, 10 May 2016 16:39:54 +0200, Stephane Bortzmeyer said:

> You mean the GPS network is not managed by an external entity? With
> budget issues?
>
> http://www.schriever.af.mil/GPS

Note that they *do* have motivation to keep it working, simply because
so much of their *own* gear (from gear for individual soldiers all the
way to strategic bombers and aircraft carriers) wants a working GPS signal...


pgpNB7bU9HRAq.pgp
Description: PGP signature

Re: NIST NTP servers

[David Hubbard]

Ed, and anyone else reading this thread, I’m curious if you’ve looked at their 
authenticated NTP offering which uses different servers:

http://www.nist.gov/pml/div688/grp40/auth-ntp.cfm


We’re considering that but haven’t tried yet.

David




On 5/9/16, 11:01 PM, "NANOG on behalf of b f"  wrote:

>Hello List,
>
>
>In search of stable, disparate stratum 1 NTP sources.
>
>Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s
>NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi
>
>We tried using “time.nist.gov” which returns varying round-robin addresses
>(as the link says), but Cisco IOS resolved the FQDN and embedded the
>numeric address in the “ntp server” config statement.
>
>
>
>After letting the new server config go through a few days of update cycles,
>the drift, offset and reachability stats are not anywhere as good as what
>the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.
>
>
>I would greatly appreciate and feedback / advice, etc.
>
>
>Thanks!!!
>
>
>Ed

RE: NIST NTP servers

[Chuck Church]

True, but I did mention verifying packet sources.  That needs to happen 
everywhere, and it's not hard to do.  Just getting everyone to do it is tough.

Chuck

-Original Message-
From: Allan Liska [mailto:al...@allan.org] 
Sent: Tuesday, May 10, 2016 10:40 AM
To: Chuck Church ; 'Majdi S. Abbas' ; 
nanog@nanog.org
Subject: RE: NIST NTP servers



On 5/10/2016 at 10:30 AM, "Chuck Church"  wrote:

>
>It doesn't really.  Granted there are a lot of CVEs coming out for NTP 
>the last year or so.  But I just don't think there are that many 
>attacks on it.
>It's just not worth the effort.  Changing time on devices is more an 
>annoyance than anything, and doesn't necessarily get you into a device.
>Sure you can hide your tracks a little by altering time in logs and 
>altering it back, but that's more of an in-depth nation-state kind of 
>attack, not going to be a script kiddie kind of thing.  Just follow the 
>best practices for verifying packet sources and NTP security itself, 
>and you should be ok.
>
>Chuck

I would argue that the fact the NTP can, and has been, be used in DDoS 
amplification attacks is a serious concern for using protocol going forward.



allan

Re: NIST NTP servers

[Stephane Bortzmeyer]

On Tue, May 10, 2016 at 10:52:28AM -0400,
 valdis.kletni...@vt.edu  wrote 
 a message of 37 lines which said:

> Note that they *do* have motivation to keep it working, simply
> because so much of their *own* gear (from gear for individual
> soldiers all the way to strategic bombers and aircraft carriers)
> wants a working GPS signal...

Yes, but they may switch it off for civilian use (by going encrypted,
for instance) at any time, if it is better for *their* operations.

Re: NIST NTP servers

[Leo Bicknell]

In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:
> In search of stable, disparate stratum 1 NTP sources.

http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm

> We tried using “time.nist.gov” which returns varying round-robin addresses
> (as the link says), but Cisco IOS resolved the FQDN and embedded the
> numeric address in the “ntp server” config statement.

Depending on your hardware platform your Cisco Router is likely not
a great NTP server.  IOS is not designed for hyper-accuracy.

> After letting the new server config go through a few days of update cycles,
> the drift, offset and reachability stats are not anywhere as good as what
> the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.

The correct answer here is to run multiple NTP servers in your
network.  And by servers I mean real servers, with good quality
oscellators on the motherboard.  Then configure them to talk to
_many_ sources.  You need 4 sources of time minimum to redundantly
detect false tickers.  If you're serious about it then find ~10
Stratum 1 sources (ideally authenticated and from trusted entities),
one of which could be GPS as several have suggested.  You'll then
have high quality false ticker rejection.

Configure all of your devices to get NTP from the servers you run
using authentication.

-- 
Leo Bicknell - bickn...@ufp.org
PGP keys at http://www.ufp.org/~bicknell/


pgpRuzcNumYGj.pgp
Description: PGP signature

Re: NIST NTP servers

[Josh Reynolds]

That would be a very poor idea, since a lot of the circuits the DoD
still uses to communicate with are ATM lines :)

On Tue, May 10, 2016 at 9:59 AM, Stephane Bortzmeyer  wrote:
> On Tue, May 10, 2016 at 10:52:28AM -0400,
>  valdis.kletni...@vt.edu  wrote
>  a message of 37 lines which said:
>
>> Note that they *do* have motivation to keep it working, simply
>> because so much of their *own* gear (from gear for individual
>> soldiers all the way to strategic bombers and aircraft carriers)
>> wants a working GPS signal...
>
> Yes, but they may switch it off for civilian use (by going encrypted,
> for instance) at any time, if it is better for *their* operations.
>
>

Re: NIST NTP servers

[Valdis . Kletnieks]

On Tue, 10 May 2016 08:07:15 -0700, Brandon Vincent said:
> On May 10, 2016 7:59 AM, "Stephane Bortzmeyer"  wrote:
> > Yes, but they may switch it off for civilian use (by going encrypted,
> > for instance) at any time, if it is better for *their* operations.
>
> I think you are referring to selective availability which degraded the
> positional accuracy for civilians.

I seem to recall that the positional accuracy is degraded to "on the order
of 200 meters" when selective availability is enabled (and yes, they *can*
do it by geographic area by careful turning on/off on each orbit of each
satellite - devices are told the signal is degraded and can downvote that
signal.  So unless you're in the war zone, you'll just notice your device
reporting a lock on 2-3 fewer signals).

The upshot is that Grace Hopper told us about electricity moving a foot per
nanosecond - which means that the time-domain jitter introduced is all of
600 nanoseconds or so.  In other words, anybody using it to feed an NTP
server will hardly notice on the millisecond level.  If you're trying to
use NTP to get microsecond stability, you'll notice - but you have enough
*other* things to do correctly to do this that it shouldn't be an actual 
issue...


pgpCZR1NfsvBU.pgp
Description: PGP signature

Re: NIST NTP servers

[Mike]

On 5/10/2016 11:22 AM, Leo Bicknell wrote:
> In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:
>> In search of stable, disparate stratum 1 NTP sources.
> 
> http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm
> 
>> We tried using “time.nist.gov” which returns varying round-robin addresses
>> (as the link says), but Cisco IOS resolved the FQDN and embedded the
>> numeric address in the “ntp server” config statement.
> 
> Depending on your hardware platform your Cisco Router is likely not
> a great NTP server.  IOS is not designed for hyper-accuracy.
> 
>> After letting the new server config go through a few days of update cycles,
>> the drift, offset and reachability stats are not anywhere as good as what
>> the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.
> 
> The correct answer here is to run multiple NTP servers in your
> network.  ...
>[snip]


I think the correct answer here starts with a question --- what level of
time accuracy is required for the local NTP server(s)? Which then begs
the question, what level of accuracy is needed for the clients?

A shop with a client need for nanosecond accuracy begs for an entirely
different solution set than a shop where a millisecond of accuracy is
needed on the clients, and still a different solution set that a shop
where "a few milliseconds either way" is quite OK.

Re: NIST NTP servers

[Laszlo Hanyecz]

On 2016-05-10 15:36, Mike wrote:

On 5/10/2016 11:22 AM, Leo Bicknell wrote:

In a message written on Mon, May 09, 2016 at 11:01:23PM -0400, b f wrote:

In search of stable, disparate stratum 1 NTP sources.

http://wpollock.com/AUnix2/NTPstratum1PublicServers.htm


We tried using “time.nist.gov” which returns varying round-robin addresses
(as the link says), but Cisco IOS resolved the FQDN and embedded the
numeric address in the “ntp server” config statement.

Depending on your hardware platform your Cisco Router is likely not
a great NTP server.  IOS is not designed for hyper-accuracy.


After letting the new server config go through a few days of update cycles,
the drift, offset and reachability stats are not anywhere as good as what
the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.

The correct answer here is to run multiple NTP servers in your
network.  ...
[snip]


I think the correct answer here starts with a question --- what level of
time accuracy is required for the local NTP server(s)? Which then begs
the question, what level of accuracy is needed for the clients?

A shop with a client need for nanosecond accuracy begs for an entirely
different solution set than a shop where a millisecond of accuracy is
needed on the clients, and still a different solution set that a shop
where "a few milliseconds either way" is quite OK.




You can get pretty nutty with this, and it's fun to do, but regular NTP 
over the internet is good enough for millisecond accuracy.  A default 
install with pool servers is pretty good.  A custom config with a local 
NTP server (with less, possibly more symmetric network latency) is a 
little better, but for common sysadmin needs like cron jobs and log 
correlation, you likely won't notice a difference.  I would worry more 
about having that config distributed correctly and monitoring all your 
servers to make sure their NTP is healthy, rather than worrying about 
the source of your NTP sync.  The pool servers are fine, and NTP is good 
at deciding when they're acting up.  The computer running NTP doesn't 
have to be anything special, but beware of VMs - depending on the 
virtualization type and the guest OS, you may not even be able to get 
NTP to engage because of the clock instability.  Chrony might work 
better for VMs.  For a linux NTP server, I prefer modern Intel CPUs with 
invariant tsc - linux will use it as a clocksource (cat 
/sys/devices/system/clocksource/clocksource0/current_clocksource
) .  A Raspberry Pi or something in between also works equally well if 
you're going to be syncing this over a jittery shared network anyway.  I 
would suggest having more than one server, in different locations if you 
can, and if you're able to supplement with pool servers, add those too.  
The most likely failure mode you're going to have is that it doesn't 
work at all because it's not running, it can't correct the local clock 
because of excess instability, or you lost all network connections.  
Worrying about whether you have 4 or 8 servers is kind of moot in any of 
those (much more likely) faults.


-Laszlo

TeamNANOG youtube video seeding

[james machado]

First I am thrilled to see older Nanog meetings making it to youtube.

Having said that can the people putting up the files put the Nanog
meeting number in the title of the videos to make it easier to search
and determine relevance?

Thanks,
james

Re: CALEA

[Matt Hoppes]

Perhaps the silence is an indication no one is doing CALEA or knows 
anything about it?


Personally, I can't say I've heard anything about CALEA, seen people 
trying to sell CALEA appliances, or received a CALEA request in maybe 8 
years?


On 5/10/16 12:34 AM, Josh Reynolds wrote:

Hrm?
On May 9, 2016 11:04 PM, "shawn wilson"  wrote:


The OP is also asking someone to register a throwaway email, subscribe, and
respond "yes" so that the owner can't be tracked to their employer. That's
kind of a steep ask for something that's almost moot.
On May 9, 2016 23:16, "Greg Sowell"  wrote:

I haven't had a request in ages...back then all of the links worked.
On May 9, 2016 3:02 PM, "Jeremy Austin"  wrote:


On Thu, May 5, 2016 at 4:43 PM, Justin Wilson  wrote:


 What is the community hearing about CALEA?



Crickets?


--
Jeremy Austin

(907) 895-2311
(907) 803-5422
jhaus...@gmail.com

Heritage NetWorks
Whitestone Power & Communications
Vertical Broadband, LLC

Schedule a meeting: http://doodle.com/jermudgeon

Re: NIST NTP servers

[Gary E. Miller]

Yo Chuck!

On Tue, 10 May 2016 10:29:35 -0400
"Chuck Church"  wrote:

> Changing time on
> devices is more an annoyance than anything, and doesn't necessarily
> get you into a device.

So, you are not worried about getting DoS'ed?

How about you set the time on your server ahead by 5 years.  Got any
idea what would happen?

Most of your passwords would expire.

All your SSL certs would expire.

All your TOTPs, like Google Authenticator would fail.

All your IPSEC tunnels would drop, and refuse to restart.

Many of your cron jobs would got nuts, possibly deleting all your logs.

Much of your DNSSEC would expire.

Many of your backups would be deleted since they 'expired'.

Until recently, setting your iPhone to 1 Jan 1970 would brick it.

I'm sure there are many more examples, but likely you can no longer log
in, via SSH or HTTPS, and your iPhone is dead.  I think any of those
would qualify as more than an annoyance.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpNcDMQ1pC0W.pgp
Description: OpenPGP digital signature

Re: CALEA

[Josh Reynolds]

This is a large list that includes many Tier 1 network operators,
government agencies,  and Fortune 500 network operators.

The silence should be telling.
On May 10, 2016 2:52 PM, "Matt Hoppes" 
wrote:

> Perhaps the silence is an indication no one is doing CALEA or knows
> anything about it?
>
> Personally, I can't say I've heard anything about CALEA, seen people
> trying to sell CALEA appliances, or received a CALEA request in maybe 8
> years?
>
> On 5/10/16 12:34 AM, Josh Reynolds wrote:
>
>> Hrm?
>> On May 9, 2016 11:04 PM, "shawn wilson"  wrote:
>>
>> The OP is also asking someone to register a throwaway email, subscribe,
>>> and
>>> respond "yes" so that the owner can't be tracked to their employer.
>>> That's
>>> kind of a steep ask for something that's almost moot.
>>> On May 9, 2016 23:16, "Greg Sowell"  wrote:
>>>
>>> I haven't had a request in ages...back then all of the links worked.
>>> On May 9, 2016 3:02 PM, "Jeremy Austin"  wrote:
>>>
>>> On Thu, May 5, 2016 at 4:43 PM, Justin Wilson  wrote:

  What is the community hearing about CALEA?
>
>
 Crickets?


 --
 Jeremy Austin

 (907) 895-2311
 (907) 803-5422
 jhaus...@gmail.com

 Heritage NetWorks
 Whitestone Power & Communications
 Vertical Broadband, LLC

 Schedule a meeting: http://doodle.com/jermudgeon


>>>

Re: CALEA

[Christopher Morrow]

On Tue, May 10, 2016 at 4:00 PM, Josh Reynolds  wrote:

> This is a large list that includes many Tier 1 network operators,
> government agencies,  and Fortune 500 network operators
>

​no one gets calea requests because prism gets all requests?​

Re: CALEA

[Josh Reynolds]

The first rule of prism is...


*silence*


:)

On Tue, May 10, 2016 at 3:04 PM, Christopher Morrow
 wrote:
>
>
> On Tue, May 10, 2016 at 4:00 PM, Josh Reynolds  wrote:
>>
>> This is a large list that includes many Tier 1 network operators,
>> government agencies,  and Fortune 500 network operators
>
>
> no one gets calea requests because prism gets all requests?
>

Re: NIST NTP servers

[Jared Mauch]

> On May 10, 2016, at 3:58 PM, Gary E. Miller  wrote:
> 
> I'm sure there are many more examples, but likely you can no longer log
> in, via SSH or HTTPS, and your iPhone is dead.  I think any of those
> would qualify as more than an annoyance.

An unnamed vendor has code where if the clock on their router is not
set SSH won’t work as the crypto package signature says the
package isn’t valid.

Many of the not-before and not-after certificate systems have some fairly
serious issues.

https://www.cs.bu.edu/~goldbe/pub-index.html#NTP

is one place to start when it comes to on-path and off-path
NTP attacks that can skew clocks.

- jared

RE: NIST NTP servers

[Chuck Church]

-Original Message-
From: Gary E. Miller [mailto:g...@rellim.com] 
Sent: Tuesday, May 10, 2016 3:58 PM
To: Chuck Church 
Cc: 'Majdi S. Abbas' ; nanog@nanog.org
Subject: Re: NIST NTP servers

Yo Chuck!

On Tue, 10 May 2016 10:29:35 -0400
"Chuck Church"  wrote:

> Changing time on
> devices is more an annoyance than anything, and doesn't necessarily 
> get you into a device.

So, you are not worried about getting DoS'ed?

How about you set the time on your server ahead by 5 years.  Got any idea
what would happen?

Most of your passwords would expire.

All your SSL certs would expire.

All your TOTPs, like Google Authenticator would fail.

All your IPSEC tunnels would drop, and refuse to restart.

Many of your cron jobs would got nuts, possibly deleting all your logs.

Much of your DNSSEC would expire.

Many of your backups would be deleted since they 'expired'.

Until recently, setting your iPhone to 1 Jan 1970 would brick it.

I'm sure there are many more examples, but likely you can no longer log in,
via SSH or HTTPS, and your iPhone is dead.  I think any of those would
qualify as more than an annoyance.

RGDS
GARY



Ok, annoyance might have been a little light on the severity wording.
Still, modifying all your incoming NTP packets from all your sources to
actually get your NTP servers to agree on a bad time is tricky.  That is
assuming you've got multiple links, multiple sources from multiple
organizations (more than 4), they're all authenticated, etc.  Even if a
criminal was to do all that damage you listed, it still probably doesn't
result in obtaining sensitive data or money that would be the main
motivators for such extreme hacking.   If I had an iPhone, perhaps I'd worry
about that as well.  But fortunately, not an issue ;)

Chuck

Re: NIST NTP servers

[Harlan Stenn]

Leo Bicknell writes:
> ...
> 
> The correct answer here is to run multiple NTP servers in your
> network.  And by servers I mean real servers, with good quality
> oscellators on the motherboard.  Then configure them to talk to
> _many_ sources.  You need 4 sources of time minimum to redundantly
> detect false tickers.  If you're serious about it then find ~10
> Stratum 1 sources (ideally authenticated and from trusted entities),

Byzantine General's problem.

With 3 sources you can detect *1* false ticker.

But if one of those becomes unreachable you only have 2 time sources.
Dilemma.

With 4 sources you run the risk of 2 going one way, and 2 going another
way.  This happened to several folks recently, when some sites put NTP
servers on the 'net that offered leap-smeared time.  That's really a
different problem where one of the effects is that it causes "time
islands".

> one of which could be GPS as several have suggested.  You'll then
> have high quality false ticker rejection.

For extra points, use GPS receivers from different manufacturers, using
whatever "variety" you can get for all of the components involved.

Are you mounting each GPS receiver inside a coffee can to prevent
drive-by jamming?

Are the cables properly grounded?  Using gas discharge tubes?
Periodically tested/inspected?

How much fun do you want to have thinking about all of these cases?

> Configure all of your devices to get NTP from the servers you run
> using authentication.

Yes, and properly monitor your ntpd instances.

-- 
Harlan Stenn 
http://networktimefoundation.org - be a member!

Re: NIST NTP servers

[Mel Beckman]

Accurate time to the millisecond is pretty much essential for any network 
troubleshooting. Say you want to diagnose a SIP problem. You collect 
transaction logs from both phones, the VoIP gateway, and the PBX. Now you try 
to merge them to derive the sequence of events. You NEED millisecond accuracy.

But more importantly, Gary is right about the risks. I’ve had several customers 
receive major NTP DoS attacks using forged source addresses. In today’s 
Internet, there is very little source address verification (despite several 
mechanisms being proposed). Everyone relies on the originating network 
preventing spoofing, but thousands of ISPs — particularly overseas — do not do 
spoof checks. 

And the issues of NTP pollution are even more dangerous. As Gary notes, 
changing dates is a risk. A big enough change (say 30 days) would be 
catastrophic to most accounting systems. A big leap — a year or more — could 
expire software license and disable all kinds of encryption. We haven’t even 
discussed multi-stage attacks, where NTP is used to disrupt systems at multiple 
points, and then the attacker storms in and takes over unnoticed during the 
confusion.

All because of misplaced trust in a tiny UDP packet that can worm its way into 
your network from anywhere on the Internet.

I say you’re crazy if you don’t run a GPS-based NTP server, especially given 
that they cost as little as $300 for very solid gear. Heck, get two or three!

 -mel

> On May 10, 2016, at 12:58 PM, Gary E. Miller  wrote:
> 
> Yo Chuck!
> 
> On Tue, 10 May 2016 10:29:35 -0400
> "Chuck Church"  wrote:
> 
>> Changing time on
>> devices is more an annoyance than anything, and doesn't necessarily
>> get you into a device.
> 
> So, you are not worried about getting DoS'ed?
> 
> How about you set the time on your server ahead by 5 years.  Got any
> idea what would happen?
> 
> Most of your passwords would expire.
> 
> All your SSL certs would expire.
> 
> All your TOTPs, like Google Authenticator would fail.
> 
> All your IPSEC tunnels would drop, and refuse to restart.
> 
> Many of your cron jobs would got nuts, possibly deleting all your logs.
> 
> Much of your DNSSEC would expire.
> 
> Many of your backups would be deleted since they 'expired'.
> 
> Until recently, setting your iPhone to 1 Jan 1970 would brick it.
> 
> I'm sure there are many more examples, but likely you can no longer log
> in, via SSH or HTTPS, and your iPhone is dead.  I think any of those
> would qualify as more than an annoyance.
> 
> RGDS
> GARY
> ---
> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>   g...@rellim.com  Tel:+1 541 382 8588

Re: NIST NTP servers

[Gary E. Miller]

Yo Chuck!

On Tue, 10 May 2016 16:18:41 -0400
"Chuck Church"  wrote:

> Ok, annoyance might have been a little light on the severity wording.

Yup.

> Still, modifying all your incoming NTP packets from all your sources
> to actually get your NTP servers to agree on a bad time is tricky.
> That is assuming you've got multiple links, multiple sources from
> multiple organizations (more than 4), they're all authenticated,
> etc.

NTP Authentication (autokey) has been broken, and no one used it anyway.  

If I have a copy of your ntp.conf I can spoof all your chimers.  Not
hard at all.  This is UDP after all.

> Even if a criminal was to do all that damage you listed, it
> still probably doesn't result in obtaining sensitive data or money
> that would be the main motivators for such extreme hacking.

Correct, it would just get me fired due to the extended downtime.

Or maybe my company just decided to pay the ransom to get un-DoS'ed.
I still get fired.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgpGSd6Se1CbY.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Jared Mauch]

> On May 10, 2016, at 4:21 PM, Harlan Stenn  wrote:
> 
>> Configure all of your devices to get NTP from the servers you run
>> using authentication.
> 
> Yes, and properly monitor your ntpd instances.

And upgrade them.

Some software distributors don’t ship modern software.  if you
are using a distribution packaged ntpd it’s likely old and
difficult to determine its lineage due to how it’s packaged.

If you’re using Redhat based systems consider using chrony 
instead, even the new beta fedora 24 uses 4.2.6 derived code
vs 4.2.8

- Jared

Re: NIST NTP servers

[Gary E. Miller]

Yo Jared!

On Tue, 10 May 2016 16:29:26 -0400
Jared Mauch  wrote:

> If you’re using Redhat based systems consider using chrony 
> instead, even the new beta fedora 24 uses 4.2.6 derived code
> vs 4.2.8

Or, new but under heavy development: NTPsec : https://www.ntpsec.org/

It is a fork of classic NTPD, but was not vulnerable to most of the 
recent NTPD CVEs.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588


pgp5kLhcr45Li.pgp
Description: OpenPGP digital signature

Re: NIST NTP servers

[Jared Mauch]

> On May 10, 2016, at 4:40 PM, Gary E. Miller  wrote:
> 
> Yo Jared!
> 

Yo, Gary!

> On Tue, 10 May 2016 16:29:26 -0400
> Jared Mauch  wrote:
> 
>> If you’re using Redhat based systems consider using chrony 
>> instead, even the new beta fedora 24 uses 4.2.6 derived code
>> vs 4.2.8
> 
> Or, new but under heavy development: NTPsec : https://www.ntpsec.org/
> 
> It is a fork of classic NTPD, but was not vulnerable to most of the 
> recent NTPD CVEs.


Yeah, there are some issues here in how the NTP community has implemented
solutions without discussing with each other through the community splits.

The NTPWG at IETF has been in a bit of stasis for years now because the
various aspects of how it works, and those who present sometimes don’t
output in the most organized fashion requiring a lot of effort on the
receiver.

There’s also a very narrow universe of people who actually care about the
implementations and details, with people like Majdi, Harlan and Miroslav
understanding the needs more than I’ve seen anyone from the ntpsec/cisco
funded side grasp the nuances of.

As a general statement, we are well served by having diverse and robust
implementations, but as we’ve seen in the (mostly) router space that NANOG
community cares about.. there are far more BGP implementations than NTP.

This isn’t good if the community wants to move to a model of certificate based
routing and the dependent infrastructure is weak.

I would suggest moving parts of this discussion to either the NTP Pool or the
NTPWG mailing lists.

- jared

Re: NIST NTP servers

[Mel Beckman]

Boss: So how did a hacker get in and crash our accounting server, break our 
VPNs, and kill our network performance?

IT guy: He changed our clocks.

Boss: How did he do that?

IT guy: We have an opening in our firewall that permits time clock packets to 
come from anywhere in the world, under certain conditions.

Boss: Why didn’t you block that?

IT guy: Well, we filtered to only accept clock settings from a trusted source, 
but the hacker lied and pretended to be that protected source.

Boss: I thought encryption was supposed to prevent that.

IT guy: Time clock packets aren’t encrypted. There is no standard for that.

Boss: Not even a password?

IT guy: Yes, there is a sophisticated authentication mechanism, but it doesn’t 
work.

Boss: So how could we have prevented this?

IT guy: We could have purchased our own time server synchronized to the U.S. 
Department of Standards atomic clock via Global Positioning System satellites 
using a special antenna. Then we wouldn’t need time from the Internet.

Boss: That sounds expensive. How much are we talking?

IT guy: $300

Boss: You’re fired.


On May 10, 2016, at 1:51 PM, Jared Mauch 
mailto:ja...@puck.nether.net>> wrote:


On May 10, 2016, at 4:40 PM, Gary E. Miller 
mailto:g...@rellim.com>> wrote:

Yo Jared!


Yo, Gary!

On Tue, 10 May 2016 16:29:26 -0400
Jared Mauch mailto:ja...@puck.nether.net>> wrote:

If you’re using Redhat based systems consider using chrony
instead, even the new beta fedora 24 uses 4.2.6 derived code
vs 4.2.8

Or, new but under heavy development: NTPsec : https://www.ntpsec.org/

It is a fork of classic NTPD, but was not vulnerable to most of the
recent NTPD CVEs.


Yeah, there are some issues here in how the NTP community has implemented
solutions without discussing with each other through the community splits.

The NTPWG at IETF has been in a bit of stasis for years now because the
various aspects of how it works, and those who present sometimes don’t
output in the most organized fashion requiring a lot of effort on the
receiver.

There’s also a very narrow universe of people who actually care about the
implementations and details, with people like Majdi, Harlan and Miroslav
understanding the needs more than I’ve seen anyone from the ntpsec/cisco
funded side grasp the nuances of.

As a general statement, we are well served by having diverse and robust
implementations, but as we’ve seen in the (mostly) router space that NANOG
community cares about.. there are far more BGP implementations than NTP.

This isn’t good if the community wants to move to a model of certificate based
routing and the dependent infrastructure is weak.

I would suggest moving parts of this discussion to either the NTP Pool or the
NTPWG mailing lists.

- jared

Re: NIST NTP servers

[Chris Adams]

Once upon a time, Mel Beckman  said:
> Boss: So how did a hacker get in and crash our accounting server, break our 
> VPNs, and kill our network performance?
> 
> IT guy: He changed our clocks.

So, this has been repeated several times (with how bad things will go if
your clocks get changed by years).  It isn't that easy.

First, out of the box, if you use the public pool servers (default
config), you'll typically get 4 random (more or less) servers from the
pool.  There are a bunch, so Joe Random Hacker isn't going to have a
high chance of guessing the servers your system is using.

Second, he'd have to guess at least three to "win".

Third, at best, he'd only be able to change your clocks a little; the
common software won't step the clock more than IIRC 15 minutes.  Yes,
that can cause problems, but not the catastrophes of years in the future
or Jan 1, 1970 mentioned in this thread.

Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
not so sure, and I haven't seen proof to the contrary.
-- 
Chris Adams 

Re: NIST NTP servers

[Mel Beckman]

I don't pretend to know all the ways a hacker can find out what nap servers a 
company uses, but I can envision a virus that could do that once behind a 
firewall. Every ntp response lists the current reference ntp server in the next 
higher stratum. There are many ways that process could harvest all ntp servers 
over time, and then pass the public IP back to a mother ship controller. It 
could be going on right now.

My point is, when the fix is so cheap, why put up with this risk at all? 

 -mel beckman

> On May 10, 2016, at 5:18 PM, Chris Adams  wrote:
> 
> Once upon a time, Mel Beckman  said:
>> Boss: So how did a hacker get in and crash our accounting server, break our 
>> VPNs, and kill our network performance?
>> 
>> IT guy: He changed our clocks.
> 
> So, this has been repeated several times (with how bad things will go if
> your clocks get changed by years).  It isn't that easy.
> 
> First, out of the box, if you use the public pool servers (default
> config), you'll typically get 4 random (more or less) servers from the
> pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> high chance of guessing the servers your system is using.
> 
> Second, he'd have to guess at least three to "win".
> 
> Third, at best, he'd only be able to change your clocks a little; the
> common software won't step the clock more than IIRC 15 minutes.  Yes,
> that can cause problems, but not the catastrophes of years in the future
> or Jan 1, 1970 mentioned in this thread.
> 
> Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
> not so sure, and I haven't seen proof to the contrary.
> -- 
> Chris Adams 

Re: NIST NTP servers

[Roland Dobbins]

On 11 May 2016, at 8:59, Mel Beckman wrote:

My point is, when the fix is so cheap, why put up with this risk at 
all?


Time and Position Spoofing With Open Source Projects.

[.pdf link]



Just keep in mind, *nothing* is perfect.

---
Roland Dobbins 

Re: NIST NTP servers

[Joe Klein]

Is this group aware of the incident with tock.usno.navy.mil &
tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
years for the period of one hour, then return?

The reasons were not fully explained, but the impact was global. Routers,
switches, power grids, phone systems, certificates, encryption, Kerberos,
logging and any tightly coupled transaction systems were impacted.

So I began doing 'security research' on the topic (don't confuse me with
joe hacker), and discovered both interesting and terrifying issues, which I
will not disclose on an open forum.

Needless to say, my suggestions are:
1. Configure a trusted time source and good time stratum architecture for
your organization.
2. When identifying your source of time, the majority of the technologies
can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
authentication.
3. For distribution of time information inside your organization, ensure
your critical systems (Encryption, PKI, transactions, etc) are using your
redundant sources and authentication.
4. Operating systems, programming languages, libraries, and applications
are sensitive to time changes and can fail in unexpected ways. Test them
before it's too late.
5. Disallow internal system to seek NTP from other sources beyond your edge
routers.
6. All core time systems should be monitored by your security team or SOC.

One question, is this a topic anyone would find interested at a future
NANOG? Something like "Hacking and Defending time?".


Joe Klein
"Inveniam viam aut faciam"

PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  wrote:

> I don't pretend to know all the ways a hacker can find out what nap
> servers a company uses, but I can envision a virus that could do that once
> behind a firewall. Every ntp response lists the current reference ntp
> server in the next higher stratum. There are many ways that process could
> harvest all ntp servers over time, and then pass the public IP back to a
> mother ship controller. It could be going on right now.
>
> My point is, when the fix is so cheap, why put up with this risk at all?
>
>  -mel beckman
>
> > On May 10, 2016, at 5:18 PM, Chris Adams  wrote:
> >
> > Once upon a time, Mel Beckman  said:
> >> Boss: So how did a hacker get in and crash our accounting server, break
> our VPNs, and kill our network performance?
> >>
> >> IT guy: He changed our clocks.
> >
> > So, this has been repeated several times (with how bad things will go if
> > your clocks get changed by years).  It isn't that easy.
> >
> > First, out of the box, if you use the public pool servers (default
> > config), you'll typically get 4 random (more or less) servers from the
> > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > high chance of guessing the servers your system is using.
> >
> > Second, he'd have to guess at least three to "win".
> >
> > Third, at best, he'd only be able to change your clocks a little; the
> > common software won't step the clock more than IIRC 15 minutes.  Yes,
> > that can cause problems, but not the catastrophes of years in the future
> > or Jan 1, 1970 mentioned in this thread.
> >
> > Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
> > not so sure, and I haven't seen proof to the contrary.
> > --
> > Chris Adams 
>

Re: NIST NTP servers

[Eric Kuhnke]

For quite some time, in debian the default configuration for the ntpd.conf
that ships with the package for the ntpd is to poll from four different,
semi-randomly assigned DNS pool based sources. I believe the same is true
for redhat/centos.

In the event that one out of four sources is wildly wrong the ntpd will
ignore it.

If people have routers/networking equipment inside their network that only
supports retrieving ntp from one IP address (or hostname) and have manually
configured it to request time from a single external source, not their own
internal ntpd that is <10ms away, bad things could definitely happen.

It is worthwhile to have both polling from external sources via IP as well
as GPS sync. Many locations in a network have no hope of getting a GPS
signal or putting an antenna with a clear view to the sky, but may be on a
network segment that is <4ms away from many other nodes where you can
colocate a 1U box and GPS antenna.

On Tue, May 10, 2016 at 9:05 PM, Joe Klein  wrote:

> Is this group aware of the incident with tock.usno.navy.mil &
> tick.usno.navy.mil on November 19. 2012 2107 UTC, when the systems lost 12
> years for the period of one hour, then return?
>
> The reasons were not fully explained, but the impact was global. Routers,
> switches, power grids, phone systems, certificates, encryption, Kerberos,
> logging and any tightly coupled transaction systems were impacted.
>
> So I began doing 'security research' on the topic (don't confuse me with
> joe hacker), and discovered both interesting and terrifying issues, which I
> will not disclose on an open forum.
>
> Needless to say, my suggestions are:
> 1. Configure a trusted time source and good time stratum architecture for
> your organization.
> 2. When identifying your source of time, the majority of the technologies
> can be DDOS'ed, spoofed or MITM, so consider using redundant sources and
> authentication.
> 3. For distribution of time information inside your organization, ensure
> your critical systems (Encryption, PKI, transactions, etc) are using your
> redundant sources and authentication.
> 4. Operating systems, programming languages, libraries, and applications
> are sensitive to time changes and can fail in unexpected ways. Test them
> before it's too late.
> 5. Disallow internal system to seek NTP from other sources beyond your edge
> routers.
> 6. All core time systems should be monitored by your security team or SOC.
>
> One question, is this a topic anyone would find interested at a future
> NANOG? Something like "Hacking and Defending time?".
>
>
> Joe Klein
> "Inveniam viam aut faciam"
>
> PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
>
> On Tue, May 10, 2016 at 9:59 PM, Mel Beckman  wrote:
>
> > I don't pretend to know all the ways a hacker can find out what nap
> > servers a company uses, but I can envision a virus that could do that
> once
> > behind a firewall. Every ntp response lists the current reference ntp
> > server in the next higher stratum. There are many ways that process could
> > harvest all ntp servers over time, and then pass the public IP back to a
> > mother ship controller. It could be going on right now.
> >
> > My point is, when the fix is so cheap, why put up with this risk at all?
> >
> >  -mel beckman
> >
> > > On May 10, 2016, at 5:18 PM, Chris Adams  wrote:
> > >
> > > Once upon a time, Mel Beckman  said:
> > >> Boss: So how did a hacker get in and crash our accounting server,
> break
> > our VPNs, and kill our network performance?
> > >>
> > >> IT guy: He changed our clocks.
> > >
> > > So, this has been repeated several times (with how bad things will go
> if
> > > your clocks get changed by years).  It isn't that easy.
> > >
> > > First, out of the box, if you use the public pool servers (default
> > > config), you'll typically get 4 random (more or less) servers from the
> > > pool.  There are a bunch, so Joe Random Hacker isn't going to have a
> > > high chance of guessing the servers your system is using.
> > >
> > > Second, he'd have to guess at least three to "win".
> > >
> > > Third, at best, he'd only be able to change your clocks a little; the
> > > common software won't step the clock more than IIRC 15 minutes.  Yes,
> > > that can cause problems, but not the catastrophes of years in the
> future
> > > or Jan 1, 1970 mentioned in this thread.
> > >
> > > Is it possible to cause problems?  Yes.  Is it a practical attack?  I'm
> > > not so sure, and I haven't seen proof to the contrary.
> > > --
> > > Chris Adams 
> >
>