Jul 25, 2010

route-views.wide maintenance

route-views.wide has experienced a hardware failure. The machine is in the process of being replace. This will be an extended outage, but we do expect to have a replacement installed and running in the near future.

Jul 6, 2010

route-views2 reset

The route-views2 router required a reset. Network Maintenance at approximately 17:25 GMT on July 6 caused the router to lose connectivity and enter a state where sessions dropped. This exercised a bug in some our reconnect startup code. We are currently looking at possible fixes for this bug.

Jun 6, 2010

archive2 aspath/asn query updated

The dns query service for aspath and asn is now duplicated on archive2.routeviews.org. The processing of the route-views2 RIB is roughly every 3 hours and is staggered from one machine to the other in order to reduce load and increase availability.

dig +short @archive.routeviews.org soa aspath.routeviews.org.
dig +short @archive.routeviews.org soa asn.routeviews.org.
dig +short @archive2.routeviews.org soa aspath.routeviews.org.
dig +short @archive2.routeviews.org soa asn.routeviews.org.

This makes it possible to query either archive, or archive2 during the reload of the zone (which is a fairly resource intensive process,) or in case one the machines is down. Heavy queries who script these queries should be aware of both hostnames, or should consider running using the files locally.

Previously, archive2 served up an different source location RIB view, but that is no longer the case. Both are now derived from the route-views2 RIB.

See the May 12 entry below for more information on this service.

Jun 3, 2010

archive/ftp/www rebooted

The archive/ftp/www machine was rebooted. This was in order to deal with a kernel oops related to a fiber channel storage area network issue. No data loss or other problems occurred related to this issue.

May 12, 2010

route-views.eqix bgpd restart

The new daemon on the route-views.eqix collector was restarted. This was to remove unused socket options error reporting. Time of restart was: May 14 17:25:17 GMT.

May 12, 2010

ROUTE-VIEWS.EQIX VERSION AND FORMAT UPDATE

The route-views.eqix collector has been updated from bgpd 0.95a to bgpd 99.16. It is important for researchers to be aware that the data will now include the current formatting.

Support for 32-bit ASNs is now current. And use of the MRT TABLE DUMPV2 format is now current. These features will appear in the files including updates.20100512.2100 and rib.20100512.2059 and any beyond that point.

May 12, 2010

route-views.paix restarts

The bgpd process on route-views.paix was restarted a few times around 12-May-2010 21:58 GMT. This was an attempt to perform upgrades, which was not successful. There is now a plan in place to upgrade the machine at a later date.

May 4, 2010

bgpd restarts

On May 4 2010 between 05:13 and 05:35, the bgpd processes on route-views2, route-views4, and route-views6 were restarted. This was done to repair a resource depletion problem.

May 3, 2010

RouteViews backbone switch maintenance completed

Between 15:40 and 15:45 GMT, and again between 16:50 and 16:55, maintenance was performed on the RouteViews primary layer 2 device. The work was completed in to move to more permanent rack infrastructure.

April 30, 2010

route-views.linx synchronized to archive

The LINX RIBS and UPDATES are now caught up. An intermittent problem with the machine is causing some issues. We are planning on upgrading the hardware as soon as possible. Until then, expect the LINX files to take some time to come across.

April 30, 2010

archive reboot

We saw a kernel panic on a filesystem error. There was data no loss, but the system was unreachable during this outage.

April 26, 2010

archive data re-sync completed

All data on archive is now up to date. The most recent files did not lose sync, however earlier weeks in April did sync over during this 3-day period.

The new primary archive is built on new hardware and has a faster and more direct interface to the storage. This should provide better performance for data storage and more responsive access.

April 23, 2010

archive data archives re-sync in progress

We are in the process of re-syncing the data archives. This should be completed by the end of business day on Friday, April 23rd, PDT. Until this is complete, the recent data is not available.

April 22, 2010

route-views / route-views3 router maintenance

route-views.routeviews.org and route-views3.routeviews.org maintenance will be performed around 10AM PDT (17:00 UTC) on April 23. Expected downtime is about 15 minutes.

April 22, 2010

IPV6 multi-hop EBPG peering outage

The IPV6 peerings to our route-views and route-views2,3,4,6 routers were down between 11:07GMT - 12:44GMT. This outage was caused by the loss of our Internet2 transit connection during maintenance on that link.

NOTE: the goal of this work had been the start of preparation for another leg to Internet2. So in the very near future, we expect that this particular failure mode will be eliminated.

April 14, 2010

route-views.routeviews.org router rebooted

The router appears to have been dropping sessions since Apr 14 05:30 GMT. The router was rebooted at approximately Apr 14 18:00 GMT. We continue to monitor the situation.

April 13, 2010

route-views2/route-views4/route-views6 bgpd updates

The quagga bgpd version on these machines is now 99.16. Note: on the cli, show of specific routes will show a Last update time. This time may vary between invocations, due to the use of an internal monotonic with the bgpd for stability. The route info uptime data itself is fixed, but the relative times shown on the cli may drift a bit due to this variance.

April 3, 2010

bgplay.routeviews.org updated

The bgplay.routeviews.org BGPLAY server/service has been updated. The system should be operating correctly from this point forward. Please send any problem reports to help@routeviews.org.

April 1, 2010

route-views6 new hardware

The route-views6 machine has been upgraded. This was done to remove the dependency between rv6 and other system operations.

The bgpdata files are now updating and synching into the archive. The update took place between the hours of 20100331 18:00GMT and 20100401 04:30GMT

March 31, 2010

archive maintenance finished.

archive/www required a bit more work than anticipated. The system is now running back to normal. NOTE: rv6 data is being collected but is not currently syncing. The syncing will be reenabled tomorrow afternoon.

March 15, 2010

route-views feature enhancement

The route-views.routeviews.org router will receive a configuration update on Tuesday, March 16th, at 9:00AM PST [ 16:00 GMT ]. This change will require a reload and all BGP sessions will be reset. Expected downtime is 10 minutes. This work is being done to enable IPV6 features on the router. Once completed, the address: 2001:468:D01:33::80df:3367 will be the available address for ebgp multi-hop peering for IPV6. Peers may then extend their configuration to include any of IPV4, IPV4 MULTICAST, and IPV6, upon request.

March 9, 2010

route-views3 IPV6 enabled

route-views3 router config updated. route-views3 is now configured to support IPV6 neighbors and commands. Any existing or new peers are welcome to contact us to add additional peerings with 2001:468:d01:33::80df:336c .

February 26, 2010

route-views2 RIB data stoppage February 23/24/25

route-views2 saw RIB dump stoppages between the dump of RIBS rib.20100223.0200 and rib.20100226.0800. This is appears to have been caused by a latent bug in the latest bgpd code revision. It should be noted that the UPDATES files are still available for this time period. Location of the bug is under investigation; in the meantime we have rolled back the code to the version just prior to the updated version. Even though we did not see any outages on route-views4 and route-views6 ( which were also running this code rev ) as a safety precaution we have rolled back all three collectors to the most recent pre-update code until we determine the cause of this stoppage.

February 4th/5th/9th, 2010

route-views2, route-views4, route-views6 bgpd updates

route-views2, route-views4, and route-views6 have been bgpd code updated. This is to support data synchronization with bgpmon processes.

February 3, 2010

archive/www/ftp old addresses removed

As noted on January 15, the renumbering of the services for archive.routeviews.org and www.routeviews.org and ftp.routeviews.org is done. Any hard-coded use of the old addresses is unsupported.

January 31, 2010

archive/www/ftp route-views6 system panic

The archive/route-views6 machine was exhibiting kernel panics. We are investigating the cause of these failures. Most services are running normal at this time. FTP is however currently unavailable.

January 15, 2010

archive/www/ftp renumbering

We are in the process of renumbering archive.routeviews.org and www.routeviews.org. Old addresses were: 128.223.61.17 and 128.223.61.18 New addresses are: 128.223.51.20 and 128.223.51.19 "ftp", "asn", and "aspath" are all aliases for archive.

As a courtesy, we are continuing to provide HTTP, DNS, and FTP service on the old addresses. Use of the old addresses will go away on February 1, 2010.

January 14, 2010

archive/route-views6 maintenance

Operating Systems updates and hardware changes require reboots for archive.routeviews.org/route-views6.routeviews.org. Expected down time is 15 minutes, occuring at approximately 1:30PM PST.

January 4, 2010

bgpd neighbor status logs

We had a request to make bgpd transition status logging available. This is now active. Files are created in: http://www.routeviews.org/bgpstatus/ Directories are by month, and files are created per day if and when any messages appear. Messages are updated and stored in realtime.

All of the following are currently configured to forward messages to this location: route-views, route-views2, route-views3, route-views4, route-views6, route-views.eqix, route-views.paix, route-views.linx, route-views.wide, route-views.kixp.

Some names may require interpretation. route-views6 is logged as "archive". route-views.kixp is logged as "80.240.194.138.swiftkenya.com". When these machines are upgraded to current revisions, the names will be updated.

January 4, 2010

route-views.linx bgpd restarted

Between 17:30 - 19:30 UTC the bgpd process on route-views.linx.routeviews.org was restarted a few times. This was the result of software update attempts.

December 17, 2009

route-views and route-views3 IOS upgraded

route-views.routeviews.org and route-views3.routeviews.org have both been upgraded to Cisco IOS Version 12.4.24 to support the use of 4-byte ASNs.

September 10, 2009

route-views4 version update

Testing code updates on route-views4. Hopefully this will result in no more than 2 process restarts.

September 4, 2009

route-views4 restarts

A resource depletion error caused a disk full condition. This condition has now been corrected, and the bgpd processes restarted.

August 9, 2009

aspath asn DNS format reversion

The CIDR notation from the TXT records in the second entities in the aspath.routeviews.org and asn.routeviews.org DNS files has now been removed. Apologies for the mistaken formatting, and especially to those who had to adjust code, and now have to adjust back.

We also have updated the NS records for those domains to point to just archive.routeviews.org for the moment. Until we can verify that the rest of the machines in our architecture can support the memory requirements for loading those files, we will just be supporting them in the one machine.

Please also note that the files are updated and downloadable for anyone who wants to utilize them locally in their own DNS server. Files complete usually no later than 23:55, 5:55, 11:55, 17:55 UTC and are available at: ftp://ftp.routeviews.org/dnszones/ Please see the May 12 reference below for additional details.

July 22, 2009

LINX Maintenance

The box hosted at LINX, route-views.linx.routeviews.org, will be down from approximately July 22, 2009 15:00 GMT - 17:00 GMT. This outage is to support infrastructure upgrades within the LINX racks.

July 9, 2009

ipv6 routing to uonet

An IPV6 routing issue at the University of Oregon caused intermittent IPV6 data collection for the collectors route-views4 and route-views6. This event only affected IPV6 peerings and data. The time of the event was: July 8th, 03:48 GMT - 16:15 GMT

July 6, 2009

route-views router crash

The eldest of the Cisco routers, route-views.routeviews.org (128.223.51.103), crashed at approximately Jul 6 13:02:01 UTC. We are investigating the cause of the crash and will post here if we come up with an answer. Router was rebooted at approximately Jul 6 15:01:28 UTC.

June 22, 2009

ipv6 RIB table dump v2 hints

We added a pointer to the Marco d'Itri zebra-dump-parser in our tools section. This perl code is capable of parsing most of the RouteViews files, and in particularly, the IPV6 MRT RIB files from route-views6.

The script can be found at the following URL. http://www.linux.it/~md/software/zebra-dump-parser.tgz At the top of the script, set the format to 1, and the ignore_v6_routes to 0.

June 2, 2009

eqix bgpd reset

The interface on the EQIX box is throwing some unusual errors. We reset the interface, and restarted bgpd. We will continue to examine this issue.

May 15, 2009

rv2/rv4/rv6 bgpd updates

Latest rev of bgpd code put in place on all three machines. In addition, telnet on IPV6 now supported for machines that are doing V6 peering.

May 12, 2009

Now responding for aspath.routeviews.org. asn.routeviews.org.

The dnszones directory is now receiving updates of the aspath, aspath.zone, originas, originas.zone files. We are answering for these domains as well. 
Example usage, lookup for 128.223/16:
dig +short @routeviews.org -t TXT 223.128.aspath.routeviews.org.
dig +short @routeviews.org -t TXT 223.128.asn.routeviews.org.
Files are complete by 0:00 6:00 12:00 18:00 daily GMT. Other sites can retrieve the files via the URL's. The The YOURDOMAIN for contact and NS should be replaced if you utilize the .zone files. Once setup locally, you may just want to update serial number.

http://archive.routeviews.org/dnszones/aspath.bz2
http://archive.routeviews.org/dnszones/aspath.zone
http://archive.routeviews.org/dnszones/originas.bz2
http://archive.routeviews.org/dnszones/originas.zone
Feedback on the data is welcome. As with the snapshot, these files are derived from a route-views2 RIB. The entries are the 1st match found in the RIB dump for a route, or the 1st match in the RIB dump for a more specific route.

May 4, 2009

Resume oix-full-snapshot-latest.dat.bz2 dumps

We are about to restart the convenience RIB dump ASCII files. Recall, these are really outputs derived from route-views2.routeviews.org MRT TABLE DUMP V2 RIB dumps, which take place every 2 hours.

All attempts were made to make this format look identical to the previous format. Please send comments if you see any problems with the new converter output. An example of the new conversion can be found in ftp://www.routeviews.org/oix-route-views/2009.05/

The new files should look remarkably similar to the old files. We will start to backfill the previous missing files this week.

Please note: the IGP/EGP/Incomplete tags will now periodically show the Incomplete indicator, i.e. a question mark. Previously, I believe that this field always showed "i".

April 26, 2009

UPS Maintenance

UPS maintenance work is being performed in one of our data centers between 4AM and 8AM Pacific Time, Sunday April 26. We do not expect any outages. Nevertheless, reachability to the machines route-views2, route-views4, archive/www/ftp, and bgplay, may be affected by this outage.

April 19, 2009

DNS aspath zones and ASCII Maintenance

We continue to work to recover the aspath.zone DNS feature set, and the ASCII dump of the oix-full-snapshot-latest.dat.bz2 file. This work is related to the transition to the MRT RIB Table Dump V2 format (see April 15 notice below). We expect the work to be completed by the end of this week.

route-views2 File Size Notice

The MRT RIB Table Dump V2 files on route-views2 look smaller, but in fact aren't. This is an artifact of the new dump format (see April 15 notice below). The data is roughly the same, but the format turns out to compress much better, and store data a bit more efficiently. For example, we are seeing the same amount of data in the March files (before) and the April files (after), 10million+ entries in each of the files. Seems to be a rougly 20% difference in size of uncompressed new format v. uncompressed old format, but roughly same size in data. Here's the random counts for reference, 2 old, and 1 new: 
	command used: bgpdump -M file | wc -l

	before: rib.20090307.2257, 782177804 bytes, 10784680 lines
	before: rib.20090331.0824, 752643656 bytes, 10301668 lines
	after:  rib.20090418.1359, 605085754 bytes, 10640877 lines

Faster Update Syncing

On the collectors route-views2/4/6, the data syncing to push the data fromt the collectors to the archive is now on a tighter schedule. Compression for the UPDATES is performed at :01,:16,:31,:46. Syncing is performed at :05,:20,:35:,50. Similarly, the 2-hour interval RIB files should appear every 2 hours at :05 after, plus transfer time. This change does not affect collection times. We are simply trying to make all data available as soon as it is ready.

April 15, 2009

DATA FORMAT CHANGE NOTIFICATION: RIB DUMP FILE FORMAT

We have been migrating to MRT TABLE DUMP V2 format for RIB files. This has been occuring as we have been updating hardware and software on the collectors. Data handlers may need to update their tools to accomodate the use of this format. The RIS libbgpdump tools should be sufficient in most cases. The IPV6 RIB files require Marco d'Itri zebra-dump-parser tool.

The following collectors now utilize this format:

Ultimately, all of the RouteViews collectors will be transitioned to this format. The newer tools are capable of reading both old and new format files. Collectors still operating with the older format are: One file of particular interest has been the historical route-views ASCII file. This was a "show ip bgp" screen dump from the original route-views router up until April of 2008. Between then and now it has been derived from converting the route-views2 collector RIB files. As of April 2009, this file will exhibit information derived from the RIB DUMP V2 format files that are now on the route-views2 collector. Typically, the location of this file looks like: ftp://archive.routeviews.org/oix-route-views/2009.04/oix-full-snapshot-latest.dat.bz2

It should be noted that the recommended procedures for handling RIB files is to download them and process them using the various MRT tools. The ASCII conversions are provided merely as a convenience for historical continuity, and are likely to be discontinued at some point. For more information regarding this format, see the following references:

For example, the "bgpdump" tool in the libbgpdump library can output an ASCII dump of the RIB files in mode 0 (default), mode 1 (-m), and mode 2 (-M). One of these modes should be sufficient for matching ASCII formats of earlier data if required. Setting format to 1 in the zebra-dump-parser script, and setting ignore_v6_routes to 0, will produce a full dump of any MRT RIB file.

April 10, 2009

route-views.isc.routeviews.org (a.k.a. paix) became unreachable as of April 9, 04:05 PDT. The system became reachable at 16:23 PDT. We believe this issue related to the vandalism fiber cut detailed in the San Francisco Chronicle on that date. The system is now performing normally.

April 7, 2009

The bgplay machine hardware continues to exhibit problems. A replacement piece of hardware has been identified, and should be in place no later than April 15th.

April 3, 2009

The rib dump interval on route-views4 has been changed from 5m to 2h to match the other collectors.

New link added for the libbgpdump toolset. Source location is http://www.ris.ripe.net/source/libbgpdump-1.4.99.11.tar.gz The "bgpdump" tool in this distribution works well.

The new installation for route-views2 now appears to be stable. The updates routing data for the machine remained fairly consistent. The ribs data however shows some gaps between April 1st and 3rd. The rib data returned to normal starting at April 3rd at 20:00 GMT.

April 2, 2009

We are continuing to work on the route-views2 issues. A bug is causing us to have to reset at RIB table dump intervals, currently 2 hours. We will continue to investigate this issue and note here when it is resolved.

April 1, 2009

New hardware has been installed for route-views2 to replace a failed disk subsystem. Unexpected outage time was on 04-01-2009 from 11:20:11 to 13:29:41, PST. Additional work is still being performed to sync the new data being gathered to the comprehensive data archive. This work should be finished no later than the end of the business day, April 2.

March 31, 2009

route-views2.routeviews.org (a.k.a. the bgpdata primary directory) is being worked on. We are cleaning up some issues related to the RAID controller and the host filesystem. There have been no data loses at this point, but some file transfer and compression operations are still pending. Some reboots of route-views2 will occur during the morning of April 1st.

March 16, 2009

bgplay.routeviews.org is under maintenance. We expect the box to return to operation sometime this week or next. We will add a note here once the box is online.

March 13, 2009

route-views.eqix.routeviews.org is now operating normally as of 10:20AM PST.

March 12, 2009

route-views.eqix.routeviews.org is undergoing some unanticipated maintenance. We expect the box to be returned to service within the next 24 hours.

March 10, 2009

bgplay.routeviews.org is under maintenance. We expect it will be returned to service within a week or so.

March 4, 2009

route-views2 experienced a kernel oops. Down at: Wed Mar 4 04:32:25 PST, Up at: Wed Mar 4 13:52:25 PST This system has been added to the schedule for update/replacement.

route-views4 is receiving updates and performance tuning. The system will be unavailable briefly this evening. Expected downtime should be less than 15 minutes.

Feb 1, 2009

The vty interface for route-views6.routeviews.org is currently only reachable using: telnet 128.223.51.112 We will continue to work on this problem so that we can make the shell bind to the v6 address as well.

January 9, 2009

The vty interface for route-views6.routeviews.org is available again. A typical command to check status would be: "show bgp ipv6 summary".

January 5, 2009

The mrt2cisco ascii dumps in the archive.routeviews.org/oix-route-views/ directory have been verified and are continuing to run. Please note that these files, starting in April 2008, are being derived from the route-views2.routeviews.org RIBS. The files are provided as a convenience for historical reference to users who are interested in viewing ascii data.

Remaining work from the upgrade of archive.routeviews.org is to reestablish the vty connections for route-views6.routeviews.org. Data continues to be collected for the route-views6 machine, but the vty is not yet available. We will attempt to correct this problem as soon as possible.

Tuesday, December 23, 2008

The machine archive.routeviews.org is currently being upgraded. We expect this work to be completed by December 29, 2008.

We do not expect to lose any data during this time. But we do expect that we will delay the data sync update timing until the machine upgrade is in a determinate state.

Services affected by the upgrade include the following:

We will attempt to restore each of these services as quickly as possible. Your patience during this upgrade is greately appreciated.