Service Availability Notice For Linksys Router Owners
November 30, 2003
For the past several months, DynDNS.org has been monitoring a significant increase in the amount of resources consumed by automated updates originating from Linksys WRT54G routers. We have worked with Linksys to resolve this issue, and the conclusions section at the end of this page provides information on how to help us correct this problem. All Linksys WRT54G owners should read that information and take the actions described as soon as possible.
A detailed explanation of the problem, causes, and solutions, can be found below. The important message for anyone using a Linksys WRT54G router can be found in the conclusions section.
Background
Anyone wishing to automatically update a hostname in any of the services provided by DynDNS.org to point to their current IP address needs what is known as a “client”. This is a small software application, which runs on the customer’s server, or, is built into a hardware router that is providing Internet access. These clients detect changes of IP addresses and send them to our servers using a special HTTP request, the details of which, along with a considerable amount of supporting documentation, are published in the developers section of our site. These specifications have been published and fundamentally unchanged for over two years.
A multitude of software clients have been written and provided for our customers' use by third parties. These clients are generally of high quality, providing excellent features and user interfaces to customers, and avoiding excessive updates to our systems. We are grateful to the authors of these clients for providing an important service to our customers.
As our services became more popular, especially within the home/small business market, manufacturers of home broadband routers, such as Netgear, D-Link, and Linksys, among others, began implementing clients into the firmware of their products. This, in and of itself, was actually a great thing for us – the existence of a client to update hostnames in our services in these products amounts to free advertising. However, it quickly became clear that many of these products had been written without consulting our published specification. Instead, it appears that their developers designed their software by examining existing clients' interactions with our system and mimicking them. They quite possibly never visited our site at all, and thus missed or simply ignored all of the important warnings and notices about client development.
This disregard for our published specifications resulted in a whole range of problems, from routers that wouldn't update at all, because they had incorrectly hard-coded our IP addresses instead of using DNS hostnames, to routers that would update every minute or even more frequently, from the same IP address, causing our system to quickly block, and eventually delete, the hostname. All of these problems generated additional load on our technical support staff, as customers wondered why their router, which was advertised as working with DynDNS.org, wasn't working. The damage to our reputation and number of potential sales lost due to these problems is impossible to guess.
As bad as all of these problems were, though, they were all manageable. There were very few instances under which most of these products would go into a mode that involved their slamming our servers with requests. Much more frequently they simply didn't work, and shut down. The Linksys WRT54G, on the other hand, was another can of worms entirely.
The Linksys WRT54G
 |
Analysis of Linksys WRT54G Traffic to DynDNS.org Website, October 26 – November 9, 2003 |
The Linksys WRT54G is based on an embedded version of the Linux operating system. Under the GPL (the license under which Linux is distributed), the source code must be open, and as such, Linksys has published the source code, along with all of their build tools, for the WRT54G (along with other products they make which use Linux). In the course of our investigation, we downloaded the source code for the WRT54G, and we discovered that Linksys hadn't decompiled or packet dumped an existing client – they had actually taken the client, in its entirety, and stuck it into the router. Now, this client was also released under the GPL, so it was perfectly legal for them to do this. The problem is with the way they are interfacing to the client.
Rather than letting the existing client do its job and operate as a “daemon”, or background process, Linksys configured it such that it would simply start up, perform an update, and shut back down. They then built their own wrapper around the client to check every minute to see if changes were needed, and call it as necessary. Unfortunately, their wrapper did not work properly, and under a multitude of circumstances, it can go into a mode where it repeatedly hits our servers every minute, despite the fact that the error it is receiving is not one which will simply go away. In fact, our specification explicitly states that the errors they are ignoring should result in an immediate cessation of all updates, with user input required before further updates can be performed. These updates every minute, multiplied across hundreds, even thousands, of routers, are generating millions of invalid updates to our servers every day.
From October 26th, 2003, through November 9th, 2003, we have found that over 31% of all requests to DynDNS.org systems (including our main website and our update system that these routers are hitting) have originated from Linksys WRT54G routers. A shocking 99.66% of those WRT54G updates have been invalid, resulting in return codes from our system which would indicate to a properly written client that it should stop updating immediately. The chart at right graphically demonstrates those percentages.
It is unfortunate that the problem had to expand to this scale to get Linksys' attention, and we can only hope that they and other router manufacturers will work more closely with us in the future to ensure that we don't see these problems again.
Conclusions
Linksys has largely fixed these problems in the most recent firmware release for the WRT54G. We ask that all owners of the WRT54G, whether they are actively using it to update a DynDNS hostname or not, upgrade to the latest firmware as soon as possible. You can get the latest firmware at the Linksys firmware download page. You must be running at least firmware version 1.42.3 to avoid causing serious problems for our systems.
On or after Monday, December 15th, 2003, we intend to block all access to our site by WRT54G routers that have not been upgraded. This block will also impact a small number of customers running Angus Mackay’s ez-ipupdate client, version 3.0.11b7 – if you are running that client, you should download and install the latest version (3.0.11b8) from the author’s site, as that version will not be impacted. Users of the Netgear DG834 and DG834G should upgrade to firmware version 1.03.00 or higher, available on the Netgear firmware download page to avoid being impacted by this block.
Please note that no versions of the Linksys WRT54G software currently support our Custom DNS service. You will need to use a software client to update Custom DNS domains. Software clients for all of our services can be found on the clients page. Any of the clients available for your operating system and the service you are using should be able to update your host without a problem.
We strongly urge all vendors who wish to include access to our service in their hardware products to read our hardware client development information. We have compiled a significant body of information from our experience with this type of product which will help you include a more effective client in your product. We also strongly urge you to participate in the DynDNS.org Certified Program, to allow us to evaluate your products and provide you with feedback before they are released to the world.
In The News
The following are links to news stories about this release. If you got here from a news story we don't link to yet, e-mail media-relations@dyndns.org with a URL we can link to.