In addition to the tremendous amount of history around how we deal with dates and times, there are a lot of things you need to understand about how dates and times are handled by computers, as these can complicate calculations that are already complicated.
Motherboard has a battery (CMOS battery) that is used to power internal clocks, even while the system is off. The batteries do have to be changed every five years or so, but they are otherwise resistant to power loss and the like. Network Time Protocol (NTP) is used to synchronize time over packet-switched, variable-latency data networks. The goal is to synchronize all participating computers to within a few milliseconds of Coordinated Universal Time (UTC).
“You don’t wear a foil hat, you just have a foil wrapped router.”
Time and date management are complicated and nasty on computers, because they are complicated and nasty in real life. While we often think of being able to tell time as being simple, historically it hasn’t been. We’re continually having to work to keep our timekeeping devices accurate so that we can use their results for everything from manufacturing to navigation, encryption, and missile guidance.
Coordinated Universal Time (UTC)
“We do need to share certain delusions for things to work.”
14:20 What is Coordinated Universal Time?
It’s an agreed upon universal time that is within about 1 second of solar time at zero degrees longitude. It’s roughly equivalent to Greenwich Mean Time (GMT), but GMT isn’t precisely defined any more.
“It does not observe daylight savings time and there’s a lot of reasons for this.”
UTC is currently defined by the International Telecommunications Union and is the time standard used in aviation, weather forecasts, and is used on the international space station. UTC timezones are specified as positive or negative offsets, which theoretically go from -12 UTC to + 12 UTC. However, in practice, there is always the chance that some nation will adjust. The island nation of Kiribati moved some of its atolls into UTC +14 from UTC -10 so that the whole country would be on the same day.
12:14 How UTC time works.
Days are divided into hours, minutes, and seconds. Each day 24 hours and each hour has 60 minutes. However, minutes are of variable length mostly 60 seconds but occasionally 61 or 59 seconds due to leap seconds. This is because the mean solar day is slightly more than 86400 seconds.
“They were thinking circularly whereas it’s more of an ellipsoid.”
The International Rotation and Reference Systems Service announces leap seconds at least six months in advance, but the need for them cannot be predicted over longer timespans because the earth’s rate of rotation is unpredictable. For instance, the Sumatran earthquake of 2004 sped up the earth’s rotation by 2.68 microseconds. Tidal friction continually slows the rotation of the earth as well and would lengthen the day by about 2.3 milliseconds per century on its own. Essentially, what’s happening here is related to angular momentum. This means that seconds and timespans of shorter duration are of fixed length, but everything above that is of variable length. Leap seconds occur on average once every 19 months.
28:05 Why UTC is painful in computing.
We would prefer for minutes, hours, days, etc. to be of a uniform length to simplify calculations, but that’s not the way the world works. Unpredictable leap seconds can also play havoc with systems that don’t readily accept updates or systems that people simply don’t update.
“For instance, software running in orbit isn’t something you update willy nilly.”
There are tons of examples of software having issues with leap seconds. Not everyone handles leap seconds by inserting an extra second. Google, for instance, used a leap smear by slightly lengthening seconds over a time window prior to the occurrence of a leap second. Amazon announced that they would do something similar, but different.
The current implementation tends to lead to a proliferation of timescales in computer. This occors either because they need to use something other than UTC, because they used UTC and didn’t quite use it in the same way, or because they aren’t up to date.
Network Time Protocol (NTP)
“You know what, I read all this stuff and I’m off loading this.”
Obviously, you don’t want to be trying to update your system time to match UTC time all the time, even though you need to for many things to work. Thus, most systems that can use NTP do so in order to offload the work. This doesn’t work over the internet if you have to be precise at a level less than 10 milliseconds or so. Network congestion can also make time readings incorrect by 100ms or more.
“The only time 10ms counts is when you are running unit tests…I’m joking.”
39:33 Implementation of NTP.
“We’re going to show how we’re not very good nerds.”
This is done via exchange of timestamps over port 123 using UDP. It can be broadcast or multicast, with clients listening passively after initially connecting. NTP will supply warnings of leap second adjustments.
However, it does not include information about time zones or daylight savings time. Your system is expected to convert the incoming time to local time where required. NTP uses a hierarchical, semi-layered system of time sources. Each level is called a stratum and is assigned a number to indicate its distance from the top. Stratum level is not necessarily an indication of accuracy, but often gets treated that way.
42:30 Different strata.
“Because this is not the 1950’s.”
Stratum zero are high precision timekeeping devices, like cesium and rubidium clocks. These are known as reference clocks. Stratum 1 clocks are synchronized to within microseconds of Stratum 0 systems to which they are attached. They peer with other stratum 1 devices for sanity checking. These are your primary time servers. Stratum 2 systems are synchronized over network with stratum 1 devices and peer with other stratum 2 devices. It continues on down the chain the same way.
44:40 NTP timestamps.
64 bits total. 32 for seconds and 32 for fractions of a second. This means it rolls over every 2^32 seconds or 136 years and has a theoretical resolution of 2^-32 seconds, or 233 picoseconds. This level of resolution was built for future-proofing.
“It’s like the Y2K problem.”
Future versions of NTP may adjust to 128 bits, at which point the fractional second portion (64 bits) will have a theoretical time resolution of less than the time required for a photon to pass an electron, with the second portion providing enough room for unambiguous time representation until the universe dies.
48:00 How NTP puts it altogether.
“On average it’s going to be right.”
Collects network times from three or more servers on different networks. A time offset is then calculated based on the timestamps from these servers and the calculated roundtrip delay as determined by the system clock on the current system. These are then subjected to statistical analysis, with outliers being discarded and an estimate of actual time offset being calculated from the best three remaining candidates.
“They really shouldn’t be standing under a thing of molten metal but they might be, because there’s always that one guy.”
The clock frequency is then adjusted to gradually reduce the offset. You should note that this is vulnerable to roundtrip delay between different servers as well as network topology, for instance if incoming and outgoing have widely different latency.
52:30 How you deal with it.
Deal with this mostly by using whatever your operating system has built in. This is difficult and complicated coding that you really don’t want to do on your own. Use built-in libraries.
“Like this is stuff that people spend their entire careers on.”
Understand the difference between being able to represent a date or time and being able to make calculations on the date and time. Don’t try to roll your own.
HarperDB is a single model, fully indexed, schema-less database. The idea behind it is to simplify the database landscape. It can use both ANSI SQL and NoSQL natively and in real-time. It’s designed for IoT app developers and data scientists. The idea is for it to be a one stop database for all needs be they data warehouse, document store, or an app database. The interface is a REST API written in Node.js. HarperDB is built to be run via a small executable and deployed on devices like a Raspberry Pi. It even comes with a lightweight studio for managing the database. There is a free and enterprise version so I’ll be checking it out.
Tricks of the Trade
Computing uses a mix of real, pure math that assumes no imprecision and dirty, real-world math that assumes imprecision. You have to be very careful about which one you are using, because the results can vary wildly. BJ recently got burned by this, but he was a lucky one, because he realized he got burned – many others don’t.