What's not new at lucketts.net? This is the news archive page for 2005.
31 Dec 2005– Happy New Year... be safe. Lucketts.net is ready to hire a new employee to help with installations and technical support to customers. Details on the post in the Forum. AT&T gave a 2 week estimate for the latest delay on turning on the new T1. *sigh*
24 Dec 2005– The new router from AT&T has been installed and the T1 lines have been re-routed through the AT&T network so that we can add additional T1 lines now. A new T1 (#3) is physically installed and ready to add in, so we should get the new bandwidth in the next few business days. I'll keep prodding them. When AT&T did their change, we were also able to transition the network over to a routed configuration so that we can get rid of the bridging. This will make the network much tougher. The occasional bad customer router will not cause the same kind of trouble they have been causing. Another T1 is physically installed (#4), so we will be integrating that one in as soon as #3 is on-line. With 4 active T1s, our bandwidth capability will double. Today, one user can actually draw the entire bandwidth, and the addition of 2 T1s will help alot. Sorry it's taken so long.
12 Dec 2005– Flight canceled, so Steve will be around on Thursday after-all. Gone Fri-Sat instead. Rather long-winded post about network downtimes on the Lucketts.net forum. Some of the more important announcements will be shifted to the forums so that customers can post their feedback directly.
15 Dec 2005 Part 2– Flight canceled, so Steve will be around on Thursday after-all. Gone Fri-Sat instead. Rather long-winded post about network downtimes on the Lucketts.net forum. Some of the more important announcements will be shifted to the forums so that customers can post their feedback directly.
14 Dec 2005 Part 2– Thursday and Friday will have limited support as I'll be unavoidably out of town for 2 days. Unfortunately, it looks like we will be having a nice ice storm at the same time. I'll be available for phone support, but will not be able to respond to hardware outages until Saturday. Any network outages on Friday would likely extend until Saturday AM. We will try to toughen up the network as much as possible in preparation. While we're not predicting problems, it is best to give everyone a heads up that repairs, if needed, may be delayed.
9 Dec 2005 Part 2– Another unstable linksys router was removed from the network Friday afternoon and everything stabilized nicely. Do not use the new Linksys WRT54G v5 router on our network please. Three in a row have caused significant network problems. We are tweaking the network to deal with this new Linksys "feature" since people will be buying these units regardless of my warnings. AT&T tech support is working with us to adjust routing to limit any linksys problems to only the router owner so that the rest of the network is not effected.
2 new T1s were activated by AT&T at close of business Friday and I was given the go ahead to use them (5:30PM). I plugged them in hoping to have lots of extra bandwidth for this weekend, but the entire network lost internet access for 20 minutes until I unplugged them. Sorry about that. Turns out that the AT&T router for the new T1s duplicates the address of the router for the existing T1s. AT&T Tech support had assured me that the router had been set up with a new range of addresses so that there would not be a conflict. Looks like they were wrong. I'll work with them when they are back in the office on Monday to correct this so that we can get that expanded bandwidth on line ASAP.
9 Dec 2005– AP #8 at Taylorstown was knocked offline this morning at 3:30 AM. That appears to be when the main storm moved through with sleet, so icing could have been part of the problem. But there appears to be another transmitter on channel 11 blocking the access point from working. When I change the AP to an alternate channel, service is restored, but there are still occasional hits. Did anyone make a change last night that would affect the 2.4GHz spectrum on channel 11? This is a serious impact if that channel is being blocked.
7 Dec 2005– We took some odd hit today at 3:30PM that slowed everything down for 20 minutes. I believe it was a new client router. I started getting paged just minutes after it was plugged in, and everything started recovering right after I unplugged it. I've coordinated with AT&T on adjusting the internal routing to make our network tougher to that sort of problem. I'll try to get the modification in place ASAP. The modification itself will cause a similar impact, but then everything should be more stable. I'll try to implement after midnight so that it doesn't cause any service interruptions.
Bandwidth has been a problem lately. The new T1 was scheduled for installation the first week of November, but AT&T and Verizon are pointing fingers at each other as to who's fault the delay is. I'm escalating regularly to get this resolved so we can get the additional bandwidth. Until then, I would be nice if everyone could avoid downloading DVDs during the day and early evening until the additional bandwidth gets online. The additional bandwidth we're getting through a wireless connection to Leesburg is delayed and the first set of equipment they sent me will not establish a link. I know the feeling..... Either a new T1 or the wireless link will be installed ASAP. I'm working on both.
27 Nov 2005– Wow... some power outage. Much of our coverage area lost power from about 630AM to 11AM today. Network impacts: AP10 (Stumptown)& AP9(Montressor) where down for awhile. Their batteries were not up to handling the load. The AP10 site actually went down early because one of the components onsite was not plugged into the UPS. When I plugged it intothe right place, that site ran for another hour before the battery gave out. The network center ran on UPS until the generator was cranked up. Two small but serial componets there were plugged into the surge side of an UPS instead of the UPS side, so they were offline for a couple of minutes. The primary server for taylorstown drained its UPS in the 20 minutes it took me to get the generator going, so it shut down and had to be restarted. The mail server didn't restart and I didn't notice that for a couple of hours. Sorry for delays in getting some emails out today. All in all, it was a good, but unexpected test of the entire backup power system. We now know where some of the weak spots are and can augment the backup power there.
25 Nov 2005– The Lucketts.net firewall decided to prohibit most outbound traffic this morning at 2AM. Downloads were slowed a little, but uploads were at a crawl. Firewall has been taken offline for treatment. Probably frostbite. In the meantime, please make sure you have a firewall between you and the internet. If you have a router, that's fine. If you run Windows XP, you should re-enable the built in firewall if you have turned it off. If you still are running XP Service pack 1 instead of the new version, you will have trouble. I've been blocking the attacks on XP SP1 with my firewall, but that is gone now. If you haven't already upgraded, you should do so.
24 Nov 2005– As I've mentioned before, it's important to let me know if your connection goes down and rebooting doesn't fix it. Today the 30ft tower on Stumptown Rd blew over. In this case I didn't find out until a customer let me know awhile later. I just had time to do a quick rebuilt before it got too dark and cold. If I had found out any later, the repair would probably have waited until Friday. Usually I get paged if an access point goes down, but my pager is hiding from me today. Still working on the new T1, the new link to Leesburg for additional bandwidth, and continued upgrading of the access points.
21 Nov 2005– It's been a rough 4 days, worst performance since the the storms this last spring. On Thursday, the Verizon MUX that controls all the phones and T1 up RT 15 started to die early afternoon. Our T1s went first, then much of the Lucketts phone service followed. Verizon replaced the equipment by about 7:00PM and we were back up. It failed again Friday morning as they were working on it again for about an hour. Previously, I would have posted that there is nothing I can do to prevent Verizon failures from taking our network down, but not this time. With the help of Chris and his crew from Skynet Wireless in Leesburg, I have the hardware in my hands today to install a backup high speed link to the internet. The Skynet link will also be able to provide significant bandwidth so that we do not have to rely on the 90 day T1 installation cycle for new T1s. The new link will be complicated to set up, as we have to coordinate between Lucketts.net, Skynet, AT&T, and ARIN (the people that run the internet) to ensure that everything works. I'll post more info as the installation proceeds.
On Sunday, we installed a new Access Point to the Lucketts/GlynnTarra area. With this installation, there are now 2 new APs serving that location. I'll be adjusting the setup to determine if 2 running APs or 1 running and 1 hot spare is the best. Probably end up with the main AP running and hold the other as a hot spare that can pick up the network immediately in case of failure. Running both APs at the same time would take up 1/2 of the radio spectrum, and it would cause a minor problem with your radios switching back and forth between the 2 APs, causing an occasional 2 second break in service entirely at the whim of the radio doing the switch. The radios will switch on their own when they think the other AP has a better signal, perhaps caused by a bird sitting on the antenna. Then they switch back when the bird flys away. This would not affect normal web browsing, but it could impact some VPNs. We're running in this mode today to see how it works out. Please let me know if you're having any problems in Glynn Tarra.
Sunday between 4PM and 6PM we were installing a new backbone link to Taylorstown. The old link should have been unaffected, but around 6PM it got confused and decided to shut off. As the backup wasn't fully in place yet, I had to drive out and manually reboot the equipment. Sorry, this was an unplanned shutdown. I didn't believe that the work we were doing would impact the operational network, as we were not actually connecting to any operational devices. But it did (impact), so now I'll sit back and try to figure out why. In the meantime, I will defer additional work on that backbone link until off-peak time (weekday nights after midnight and/or weekend afternoons with advance notice.)
The new backbone link to Stumptown was also part of that install on Sunday, but got held up when the Taylorstown link dropped. I'll be working on it early this week along with establishing the backup link to the internet. These should be non-interference to the existing network, sort of like Sundays work was. I'll be carefull, but this has to get done.
17 Nov 2005–
That's what I get for pestering Verizon to finish the installation of the new T1. Everything was down today for 5 hours,
but it's back up and running now. Kudos to Chris Chamberlin from Skynet Wireless in Leesburg who offered to work Thursday
evening and early Friday to help us get a backup circuit installed if we needed it. As this is the 3rd time that the
carrier class T1s from Verizon have gone down this year, I believe I will work with Chris at a more leasurely pace to
get an alternate circuit intalled. It will cost alot so don't forget to mention this service to your friends
15 Nov 2005– The new T1 has been installed physically for 2 weeks, but AT&T cannot get the routing set up so we are waiting for the additional bandwidth. It would be nice to have right now. Upgrades coming in the next few days: high speed backbone to Stumptown; new antenna to chapel; replace main AP in Lucketts to make sure it is not the source of or contributor to the every two week shutdown of the network; Add backup AP in Lucketts; increase backup power to taylorstown; improve grounding system in taylorstown; Add higher power directional antennas in taylorstown to improve signal and reduce interference from/to other access points; Upgrade APs to include integrated routers will occur about one every two weeks.
10 Nov 2005– Lucketts.net uses a program called SPAMAssassin to catch SPAM email going through our servers. When a SPAM is detected, the email is marked and sent on to the customer for disposition. Sometimes SPAMAssassin will mark good email as SPAM, and sometimes it will not detect an obvious SPAM and let it through unmarked. You can help me train SPAMAssassin by sending me copies of any SPAM email that makes it through unmarked. Send the entire email to spam@lucketts.net. The easiest way to do that from Outlook express is to create an email to spam@lucketts.net then drag and drop the SPAM email onto it. That way I get an exact copy of the SPAM, including all of the headers. If on the otherhand, some real email is getting marked as SPAM, let me know about that and I can set the program to allow it to pass. thanks... Steve
29 Oct 2005– The new T1 was physically installed and tested on Friday. Verizon was 11 days early... good for them. Unfortunately, the new router AT&T is sending has not arrived yet, so we cannot take advantage of the new bandwidth until it gets here and is installed. The extra 1.5 Mbps to the internet will be sweet.
I'm going to try to do the change-over from bridged to routed network on Sunday afternoon and evening. Any interruptions should be of short duration.
I've practiced this proceedure offline to make sure 1) it actually works with the systems we use 2) it actually works with the technicians we use
26 Oct 2005– For a bad day it was a good day as I did learn a few new things. It started poorly last night when most of the network decided to sort of slow down, or more precisely, to stop. Even the new very high speed link to Taylorstown I was so proud of was cut down, with ping times around 400 ms. It took until Wed at 3 PM to find the set of problems that interacted to shut us down. Quick summary: two (of 18) of my router/access points had spanning tree protocol implemented, and 2 customer home routers had incorrect subnet masks. This combination caused all of the routers to in effect shut down every 2 seconds. The exact reason these problems interacted this way is mostly techno-babble, but the result was easy to express: it was real slow.
I learned a few things about spanning tree that I didn't know, particulary that it could impact other devices not using it. I turn Spanning Tree off by
default on new machines, but 2 slipped through. This would not have caused a noticable issue unless a bridging loop were to be established between these two
machines on opposite ends of lucketts.net and I'm much to good at design to allow that.
Lesson to customers from this: If you change the settings on your router, please be very careful to use the numbers I gave you. Those 240's in the subnet mask actually are very important. I've learned a lesson about not leaving processes running on a machine if they aren't needed. That just opens up the possibility for something to break or for an unforeseen interaction to mess things up. Third lesson: if I had the fully routed network up this would not have happened. Interesting that the activity I had scheduled for Tuesday night and Wed night was to change the network over to routed vs bridged. That will slip a few more days as I'm tired and going to sleep now.
22 Oct 2005– I've been consulting with Skynet Wireless in Leesburg for tech support. They advise me to shut off users who use p2p file sharing because of the number of connections that software establishes to the internet. Normal browsers run 1 to 10 connections at any one time, sometimes more. The p2p programs can open over 100 per second. The large number of connections, particularly the inbound ones, cause problems for RF access points. I've tried bandwidth limiting on the ports used by p2p, but then the software (or the user) adjusts the ports to evade the limits. When the software gets tricky like that (or the user) I can put a bandwidth limit on everything that client sends out to the internet, not restricted only to p2p traffic.
Some of the recent problems we've experienced have been because of the access points, dns servers and routers shuting down under the p2p load. Lucketts.net will implement limits on the number of p2p connections allowed by a single user to keep the network running for the majority of users. This may impact a few users that use p2p extensively. The user will have enough connections that they will still be able to download p2p at full speed as long as they do not allow the internet to upload from them. If they instead set up their p2p to allow the outside world to "borrow" the files on their PC, then the outside world will quickly use up all of their allocated connections, bringing their downloading to a halt. If you run your PC as a server of p2p files you will have very slow download of files yourself, but your web traffic should be unaffected. If a user circumvents the connection limit by shifting their p2p to hidden ports or ports normally used for other services, such as web or mail, then their total connections for all services will be capped instead.
Running a p2p file server is a violation of the terms of service. We looked the other way (alot) to try to provide everyone with the service they want, but I can no longer allow p2p traffic to impact the entire customer base. If you are not a p2p user but the connection limits impact non-p2p traffic on your system let me know and we can adjust the limits so that you are not affected. If there are special circumstances where you require unrestricted p2p to update a program or for work, let me know and we can work out a solution with you.
18 Oct 2005– The long awaited and promised high speed backbone link to Taylorstown and Lovettsville has finally been installed! Ping times will be improved, downloads will be faster, interference will be ignored, and the existing access points can now be linked to provide backup to each other. The new link will be tougher and help prevent some of the recent routing problems, but it's real value is diagnostics. There will still be some problems, and the new equipment will allow me to identify the source of any problems easier.
15 Oct 2005– The same Rt 15 interference is back. It shut down the West Lucketts network completely. When I discovered it (A few customers finaly called me) I had to switch to the backup channel again. I ran RF sweeps to detect it but came up empty. It was either gone or is of a type that I cannot tune to with my RF gear. Was anyone around Lucketts road/Rt 15 running some RF generating equipment between 1 and 3 today?
6 Oct 2005– I need customer help locating a problem. The affected area is Lucketts RD, RT 15 north of Lucketts Rd, and GlynnTarra. Sometime on Wed evening, the network in that area came to a near halt. All radios were connected with good signal strength, but speeds were in the 1 to 50 KBps range. Uploads appeared to be much faster, but still slower than normal. Sometime around 9AM on Thursday morning the trouble went away. I tried lots of debugging, rebooting, re-routing, etc but nothing I did fixed it.
The likely causes would be bad power supplies, damaged or misconfigured routers, or RF interference. I need help identifying something that coincides with the slow times. Did anyone plug in some new equipment Wed night and turn it off Thursday monring? Anyone running a special software package during that time? Anyone playing with radio emitting equipment during that time? Possible equipment conflicts could come from wireless speakers, video links, baby monitors, etc. If anyone has a candidate suspect, we can test to see if it causes problems or not.
2 Oct 2005– Wow..... that was infintely harder than it was supposed to be. When the new link finally was established to taylorstown, the ping times on the network went wild, climbing up to 6000+ ms. I eventually had to turn the new link off and restore the network to original configuration. That took a couple of hours to get settled down. Best bet is that the new radio has a messed up power supply causing it to fail under load. I'll take a new radio up on Monday and get it running. Once it works, I'll switch the Taylorstown network over to the new link one small piece at a time to prevent a repeat of today's crash.
1 Oct 2005– Work for the next week will focus on network upgrades. We've grown alot in the past few weeks and it's time to slow down the growth and work on infrastructure. The first item is to upgrade the backbone connection to taylorstown. Next in line will be the Stumptown AP backbone upgrade. As soon as the new router arrives, the change over to a fully routed network will occur. The 2nd class C IP block is installed and routing OK, so the new router will run the existing class C set of addresses. When we change over, there will be a 2 hour interruption of service. We'll announce the downtine a few days in advance here, on the main home page, and via email.
23 Sept 2005– Ongoing tracking of interference on AP0 serving Glenn Tarra. The interference followed us to the backup channel, so I'm switching back to the primary. So far looks like everyone is full speed again. Looks like we may have a bad radio in our bunch that is messing up the AP, but it doesn't happen right away. Everything works well for days, then interference starts again. Assuming nobody is changing settings on their home routers, I'll have to figure out how to isolate the faulty radio. For now, if you are connected to AP0 (Glynn Tarra and other homes directly to the west of me) move your home wireless routers to ch 1 or ch 7. Please make sure you are not using spread spectrum, freq hopping, or multiple channels as each of these will impact the signal of you and your neighbors.
The Stumptown Rd access point has been acting up. We have two links to it from the network center, so on Friday we switched over to the 2nd link which appears to be working well. That link is more sensitive to rain, so we'll have to watch the weather. Tom and I are installing a new 5.8 GHz link to the site that will be put on top of a new 50 ft pole. This link will shoot above the trees so should work much better. Customer on this site will be pleasantly surprised by the performance when the work is completed. All the parts are here, it's just an installation job now.
The Taylorstown site has 2 2.4 GHz access points and 2 900 MHz APs. It currently has 2 data links to the network center to that if one fails we can use the other. A third link on 5.8 GHz will be added here as well to speed things up.
A new T1 was just ordered and should be active in 4 to 6 weeks. This will increase our bandwith to the internet to 4.5 Mbps. Sometime in Dec we will be adding a 4th T1 for a total of 6 Mbps. We have a 2nd class C set of IP addresses to handle expansion and new routers have been tested on it to demostrate the new routing setup. Once tested this week, I'll be transitioning the rntire network over to the fully routed solution next weekend. We'll post more details on schedule, but the down time should be minimal.
If you have a problem accessing the internet that is not cured by rebooting your machines, please let me know ASAP. Specifically, if you ever see a packet loss rate of more than 10 percent let me know. Try calling our phone line to report problems, and if that doesn't work page me. Current numbers are on our contact page. Please don't call my cell.
11 Sept 2005– The primary Access Point took a hit today and slowed a bit. Ok, it screached to a halt. After an hour of rebooting, debugging and checking logs I found that whan I changed its channel to the backup channel everything worked again. A-ha.... This appears to be a repeat of the 18 Aug problem. I did a site survey for new transmitters and found a powerful one sitting right on our main channel. I drove around and located the house it is coming from. Hate mail should be sent to:________ No, don't do that. The email address is.... {oh nevermind. We turned his radio off and the interference is still there on channel 11- steve 6:15]
Lesson learned here: if you get an 802.11 b or g wireless router please be careful. Be kind to others and pick a channel to transmit on that does not shut down your neighbor's internet access. In Glynn Tarra, please use channel 7 or 1. Do not use channel 11 or 4. In between channels are worse as they just take two of the channels just listed. Also, if you turn on the 108MBps feature of your routers, you will take up multiple channels. There are only 4 usable channels for 802.11 b & g radios, and the 108 Mbps feature is a hog. Lucketts.net provides the fasted internet access available in Lucketts, and it tops out right now at 3 Mbps to the internet (soon to be 4.5 Mbps). While the 108 MBps feature is cool, it offers no benefit to internet access in this environment
I'm pretty confident that this radio was the source of the problem. There is a slight chance that there is another radio that I did not detect yet. We'll find out when I get the owner to change their channel. If the interference leaves channel 11 then we'll know.
6 Sept 2005– Starting software upgrades on servers. I hope to get a newer version of SPAMAssassin running on the lucketts mailserver. The other domains have the new version running and it just works better. I tried a mass upgrade on the secondary server, but that just broke things. So after a backup, I'll upgrade the main server one application at a time.
In related news, the bullettin board is offline until I can get the SQL server restarted.
24 Aug 2005– The webmail server for lucketts.net has stopped sending mail. The webmail server for Taylorstown, Glynn Tarra, and BVWISP is working fine. I'm looking for the cause now. Hope to fix soon. You can still read mail, you just cannot send, forward or reply. - Update: I was able to get the webmail server to work by reverting back to an older version. Apparently, the new version upgrade a couple of days ago killed it. I had to go back three versions to get it working again. I'll work on it later tonight to see if I can get it back up to the newest version minus 1.
18 Aug 2005– There is a new RF noise source that is impactinf the main access point at my property on Lucketts Rd. The noise is jamming channel 11 so that it is almost unusable. Chris, it's not your phones. I cut everyone off at 1:15 this morning and the noise was still there. My other transmitters were also shut down and the interference remained. When I hopped to another channel, the jamming decreased as I got farther away from channel 11. I've set the AP to channel 1 now. Unfortunately, many client have set their home wireless routers to channel 1 as well, so if you are connected to the main access point then you may want to consider adjusting your settings. Try channel 7.
I guess it's possible my kids might have picked up some wireless headphones or new gizmo causing the problem, so I'll do a power down of my home Thurday to ensure that the RF source is not local. If I'm clear, I'll start visiting neighbors. Hopefully I can switch back to channel 11 soon. All of my access points on the hills also use channel 1, so making this a final solution will require a large amount of changing channels for everyone that has routers. I hope to avoid that.
I was really hoping this was Chris' cordless phones.
11 Aug 2005– I've slowed the growth of the customer base the past few weeks so I could catch up on repairs , troubleshooting and infrastructure upgrades. There are still some nagging problems hiding out there. The worst is still the RF interference around Chapel Lane. It is proving very hard to find.
[WARNING: Geek material] I've tracked down a routing problem that randomly affects Taylorstown customers. It was simply the size of an ARP lookup table on the relay link up to the mountaintop. I've mitigated this for the short term by blocking a few subnets that I use for maintainance. This limits the ARP tables to the critical IP addresses (the customers). The longer term fix is to replace that relay link with equipment that has greater capability. That equipment is in place, but it takes an hour of network downtime to do the change over. Everytime I've scheduled that switch to date, another issue has come up requiring immediate attention. The day of the posthole digger was perfect - everyone was already down so I could hurt anything. I discovered that routing this new equipment held some surprises, and after 90 minutes I had to undo the changes because they just were not working yet. I did learn what I had to though, and the next attempt has a much greater chance of success. I'll try again this weekend.
The replacement parts for another 900 MHz AP just came in today, so I'll try to get the radio installed on Friday on Lucketts Rd. This new AP will provide better service to some of the customers that have been with us for a long time but have weak signals. They will be moved off of the 2.4 GHz radios onto the new 900 MHz solutions. I'll proceed with those upgrades as fast as budget permits. (They cost alot)
2 Aug 2005– I'm beginning to seriously dislike post-hole diggers. The same crew took out our data lines at the same location as they did this spring. Verizon hustled and rebuilt the 350 pair cable in a few hours and Lucketts now has phone and internet again.
1 Aug 2005– - Update - Focused on looking for a noise source at Chapel that is impacting the access point. As a backup plan, I just ordered a new 900 MHz access point that will avoid the interference if I can't fix it. The high speed link to the Taylorstown relay didn't get turned on this weekend as planned (again). I need 3 hours dedicated to this during a time when users will not be impacted by the repeated service breaks. Weekdays and early evenings are out because of the impact to users. I'll schedule again for another late evening this week or over the weekend and give at least 48 hours notice.
29 July 2005– Chapel access point is experiencing difficulty that is severly limiting access to the internet. I'll be replacing the entire assembly this morning (Friday). There will be a few service interruptions while switching out the equipment. I'll do as much testing as possible offline to minimize the down time.
The Lovettsville/Taylorstown APs are working at high speed, but some customers are still experiencing occasional dropouts, lasting from seconds to minutes. An upgrade/replace/reconfigure plan of individual client equipment will be the best bet for correcting that problem. We will continue to monitor for outside interference as well. There will be some down time this Saturday afternoon or Sunday morning when a new link is activated and tested. There many be a few service breaks lasting no more than 5 minutes. If you're offline longer than that please page me.
Home routers, especially wireless routers, continue to be a problem during stormy weather. They tend to be very sensitive to power spikes and nearby lightning. Please check your router status before calling me to tell me the network is down. A reboot of your router and PC will usually fix most router problems. Sometimes, a reboot of the Lucketts.net radio/antenna, your router and your PC will be needed. Please try this first.
It is fairly easy to determine if your router has a problem and is causing your service break. You most likely have a router problem if any of these steps fail: 1) make sure your PC has a valid IP address (something like 192.168.1.100). Anything starting with 0, 169 or 167 is a failure. 2) make sure you can ping the gateway address (usually 192.168.1.1). If you get no reply you have a failure. 3) make sure you can ping your static IP address (usually 12.145.55.x where x is the number assigned to you by Lucketts.net. No reply is a failure.
If the three steps above all work, and a reboot of the Lucketts radio, your router AND your PC did not bring back your access, give me a call or page me. We can do some more quick diagnosis via phone to isolate the problem or I can tell you right away if your connection on Lucketts.net is down. Fixing any outage caused by Lucketts.net is of course a priority, so please do not hesitate to call after you've done the initial troubleshooting on your side. If you cannot get your router back online, we can schedule a service call usually same day, sometimes immediately. There will be no charge to repair any problem caused by Lucketts.net. If your internal network is broken, then standard labor rates will apply with a minimum 1/2 hour charge.
We're considering offering service plans to support, maintain and fix your internal networks. This would include some amount of detailed phone technical support and x number of site visits to fix equipment. Pricing and what number "x" is are TBD. A few customers have suggested and requested this additional "PC Doctor" service. I'd love to hear feedback on this from more customers.
26 July 2005– Just discovered on Monday a Linksys router that was failng when it got hot in an attic. Bring it down to cool off and it works again. It did not appear to affect the network, but it did affect the customer's access. Since this is pretty much the hottest time of year, I expect a few other routers in a few other attics to have similar problems.
The Chapel access point has a new backbone link that runs much faster than the old, but it is on a seperate circuit so that the Chapel users do not compete for radio bandwidth with users from other APs. When I upgraded the backbone and changed the routing equipment, the alwas slow speed problem disappeared and peek speeds above 300 KBps are being seen again. There is still an intermittent problem that can bring that subnet to a crawl. Finding the source of the problem may be difficult. There are a few clues now, but if we don't stumble across it by this weekend, I have a plan of attack. I will install the new antenna with an additional radio. Existing users will be transfered over one every few days. When the problem disappears from 1 network and switchs to the other, we'll have a real good clue as to the source of the problem.
The 900 MHz AP has been pressed into service in Taylorstown a little before I had fully checked it out. This AP should help me find the source of the intermittent dropout issue there, like the 2nd AP at Chapel will do. If something affects the 900 MHz at the same time as the 2.4 GHz, then it's clearly not just RF interference.
Oh ya... sorry for the 5 minute dropout on AP0 this evening at 9:30. I was moving equipment racks and kicked a power supply loose. Didn't notice tha access point down for 5 minutes.
21 July 2005– I stand corrected. I was reminded that there are still some unresolved network problems in Taylorstown/Lovettsville. I should have said on 19 July "News Flash: it's not Taylorstown/Lovettsville that are dead in the water! (again)". So now that the large blocks of users all have connectivity, I can start knocking off those individual performance problems. Chapel gets a new antenna ASAP to boost their signal, and I have to find a source of RF interference in Taylorstown. There are 5 customers waiting for the 900 MHz service, but I just can't get back to that until I get the current customers performance issues resolved.
20 July 2005– There is now a brand new access point on top of the tower and the high speed backbone is again active. Clients off Streamfarm and Spinks Ferry were impacted for an hour while I switched radios, but to makeup for it they should be working much faster now as the new backbone directly upgrades their link. I will be upgrading the AP at other sites as soon as cash flow permits (and lightning holds off). The new AP also has additional lightning protection at the top of the tower, so hopefully this one will survive longer than 3 months. The should be few to none minor perturbations tonight as I shift the network back to its baseline configuration.
19 July 2005– News flash: it's not Taylorstown/Lovettsville that is having network problems! Access Point #1 in the main tower at my place is acting up. Starting Tuesday about 6:07 AM it decided to start dropping 30% of the traffic, and occasionally all of the traffic. I have been looking for causes all day, and have eliminated many, but demonstrated none. Customers browsing small files are not badly affected, but anyone transferring large files or emails through AP1 is getting bumped after 10 to 20 seconds of file transfer. The plan now is to climb the tower Wed AM and replace the entire radio. While I'm there I'll upgrade the grounding and put in a better backbone link for the remote access points. This will delay the activation of the 900 MHz links and probably preclude any site surveys for new customers until the problem is fixed. Sorry for those of you waiting to get service installed, I'll get back to growing after I fix the broken network.
I will put up a temporary AP1 tonight to serve as many of the local clients as possible, so that in the AM you have a usable signal while I'm working on the tower. [This was done at 8PM - Steve]
For the computer geeks: AP1 runs off of 18V 3A POE over 100 ft of cat5 to the top of the 70ft tower. The radio needs 15 watts delivered between 10 and 18 volts to operate properly. From the POE injector, there is another 150ft of cat 5 to the main switch bank that connects to the T1 router. The loss of packets occurs with routed and unrouted IP addresses. The loss did decrease when I added a switch next to the POE injector to boost the signal to the switch next to the T1 router. I did this because the signal indicator for the cat5 plugged into the router switch was dark. It has never had a distance problem before. The new switch resolved the dark link. This one symptom gives me reason to believe that the ethernet port on the AP is almost toast. It no longer can drive the signal 250 ft. I believe it is likely that the remaining dropped packets are because of the weak ethernet port.
Also, sorry about turning the Taylorstown mail server off for an hour around 6 PM. It needed to be reset because of the other problems so I rebooted it. About an hour later I walked by and noticed that it had hung and failed to complete the reboot cycle.
16 July 2005– I thought I'd take a few minutes to update the news as long as I'm sitting here waiting for the next lightning strike. The storm tonight took out one switch so far, but it was a cheap one. I figure it's a small cost to pay to find out where I have another surge leak into the core network. I'll do some more grounding and surge surpression on Sunday to toughen it up a little more.
The 900 MHz AP is up but not performing yet as expected. I will go back and correctly mount the antenna to improve coverage. I'm hoping to cover most of the trouble spots in the network from there.
There will be a new AP at Point of Rocks next week. I finally have all the parts in and assembled, so customer schedule permitting, it will go up as soon as possible.
I've got to do some spring cleaning, so I'm going to start posting free stuff on the bulletin board. Feel free to add anything you'd like to get rid of. There also a place to post for sale and want ads. If we can get more people using the bulletin board, it will be much more useful.
6 July 2005– Our main Internet router is unhappy and is adding 500 ms to packets crossing between our subnets. That slows things down alot. If I take the router offline everything speeds right up, but then no Internet. I'm working with AT&T tech support tonight to get everything back up to speed. (02:00 AM 6 July)
29 Jun 2005– The lightning hit again and found a few weak links to knock out. Between 6:30 and 8:30 I've rebuilt the network firewall, replaced the main switch, replaced the main relay link to the mountain that carries all the Lovettsville/Taylorstown data, remotely reset a stuck AP on the mountain, replaced a burned out lightning suppressor that gave it's life for the main Lucketts Access Point, and restored two other stuck access points. I had spares on hand and was able to debug and identify the problems in real time as they happened. The eastern side equipment was restored before the western equipment went down as the storm moved through.
PS on the Trojan stuff. Turns out the Trojan should be named Mike Meyers. I kept killing it and it kept getting back up when I turned my back. At least there were no screaming coeds. If your anti-spyware program finds anything named either "Aurora" or "ABetterInternet" or anything like that, I suggest you give me a call if you want it really removed. I can work with you over the phone to see if it's still there, but to fix it you will need a house call from a PC doctor.
25 Jun 2005– There is a new Trojan program out there that is very annoying. It will take over your browser, force pop-up ads through, change your settings, and Microsoft and Spybot cannot remove it. Simple way to check if you have this one is to look for this file: "C:\WINDOWS\Nail.exe". If this file is on your PC, you have problems. My daughter browsed a music site from my business-only machine, and it was immediately infected. I'm running Spybot, Microsoft, McAfee, and another special app but it got through them all. It took me 3 hours last night and 6 hours this morning to kill it. I missed two appointments already today (sorry) and was quite irritated, but my machines are now clear.
Here's the point of today's posting: if you get this Trojan, you will have to call someone to come fix it. This one must have been written by a PC repair
person, because there is no way the normal user will be able to get rid of it. ( I didn't do it...
Whom to thank for this and where to find more info? Check here: NetRN.net If I find a commercial package that can get rid of this, I'll post the information here. If you come across something, please share so we can minimize pain.
22 Jun 2005 part 2– Anti-SPAM program called SPAMAssassin is now running on the Lucketts.net mail server. There was a pretty serious threadt with the newest trojan emails recently, and I had to stop them. Emailis that looked like they were from Lucketts.net admin were being sent out with vituses attached. They were produced by an infected machine out there somewhere (I have the IP adresses). Thje trojan would look through the infected machines email contact list and send emails to everyone on it pretending to be the admin from the target's domain. Of course the trojan was included in the attached zip file and would have infected anyone's machine that opened it. This trojan didn't stop with just mailing everyone on the list. It would guess names and send mails to the domains it discovered with those random names. I caught on when the mail server started complaining about emails to jimmyjoe-bob@lucketts.net. SPAMAssassin was set up and ready to go on Lucketts.net so I just turned it back on. The Taylorstown mail server does not have SPAMAssassin yet, as it is tricky to install without impacting current mail service. I'll find a nice quiet time on the weekend to install it. {[5:30 update: SPAMAssassin is now installed on Taylorstown.net. It was easier than I remembered.]
22 Jun 2005 part 2– Anti-SPAM program called SPAMAssassin is now running on the Lucketts.net mail server. There was a pretty serious threadt with the newest trojan emails recently, and I had to stop them. Emailis that looked like they were from Lucketts.net admin were being sent out with vituses attached. They were produced by an infected machine out there somewhere (I have the IP adresses). Thje trojan would look through the infected machines email contact list and send emails to everyone on it pretending to be the admin from the target's domain. Of course the trojan was included in the attached zip file and would have infected anyone's machine that opened it. This trojan didn't stop with just mailing everyone on the list. It would guess names and send mails to the domains it discovered with those random names. I caught on when the mail server started complaining about emails to jimmyjoe-bob@lucketts.net. SPAMAssassin was set up and ready to go on Lucketts.net so I just turned it back on. The Taylorstown mail server does not have SPAMAssassin yet, as it is tricky to install without impacting current mail service. I'll find a nice quiet time on the weekend to install it. {[5:30 update: SPAMAssassin is now installed on Taylorstown.net. It was easier than I remembered.]
AP4 on Chapel Lane started turning itself off recently. It would shut down once a week, then daily, then on Monday it shut itself down 3 or 4 times. I replaced it with a new AP Monday night, but that model did not like the etherant 3 radios that are already deployed to a few homes. I had to go back out Tuesday and replace it with one of the older models that does work with all the radios. Let me know if you have any more problems connecting. The relay to Chapel will be upgraded later this week, and that should also improve speeds to the customer.
AP8 to Taylorstown and Lovettsville disconnected itself Tuesday morning at 430 AM. I didn't discover it until 7:30 when I turned on email and found that there were a bunch of early-birds already trying to work. After a quick trip up the mountain it was up and running again at 8:00AM. The redundant circuit is installed, but I haven't set it up to automatically switch over yet. Automatic switching is nice, but tricky. It tends to switch when you do not want it to and drop everyone for a minute while the connection re-establishes. I have to sit up there some afternoon when the problems have calmed down and properly balance it. Probably over a weekend when the business users are not on-line. I'll send out an email warning at least a day in advance.
13 Jun 2005 part 2– Caught up now with mountian access point to Lovettsville. There are now 2 working access points on the mountain and 2 backbone links getting data to the access points. The masts are secured tightly with guy wires and a grounding system is installed to help preclude further lightning damage. If everything stays stable for a couple of days, I'll try turning on the high speed data link to the mountaintop.
AP10 was toasted by lightning today. Nobody is using that AP now, so no impact other than to the host of the site who has no access. Replacements will be here Tuesday PM.
AP4 upgrade that was placed on hold is back on the schedule. Will install high speed backbone to that site this week. Should perk up performance there considerably.
Did I mention that grounding is a good thing? If your radio is on a pole you should consider grounding it to help prevent dropouts. I'm doing that now for all access points, and may start doing it as an routine part of the normal installation.
10 Jun 2005 part 2– I removed the old Access Point from the roof of my home and replaced it with a nice, new, more compatible one. It will carry the load around Lucketts Rd until I get the new sector antenna and radio installed that will serve everyone to the West. The tower will serve everyone to the East. Assuming no asteroids strike (or other unexpected event) I'll finish tuning the current set of servers then start upgrading the remote access points to add high speed links and redundancy.
If you have a problem, please let me know. Don't just assume it's you. You might be the first to notice a system wide problem, and early notification can greatly improve repair time. One thing I ask is that you first try rebooting your machines before calling. Turn them all off, and then back on, with the closest to lucketts.net turned on first. For example, turn on your Etherant radio, turn on your wireless router, turn on your RF bridge, turn on your PC. If you still have the problem after that, give me a call or page me if urgent.
10 Jun 2005 – I picked a fine week to give up drinking... Here's a recap of highlights: Power to AP8 knocked out by construction accident. Large thunderstorm bends over AP8 30ft mast; Surges knock radios off air and everyone forgets to reboot before calling; T1 & phone lines dug up by neighbor's posthole digger; New radio backbone installed, but it refuses to work with any existing client radios; 30 ft Mast can be lowered and repaired by one person, but takes 2 to raise back up (oops); New high speed link installed to AP8, immune to rain!!!; AP8 high speed link not immune to lightning, $800 radio toasted by near direct lightning strike before I could put it in service; assorted other emergencies.
I'm sorry if I've been short with anyone or haven't resolved a problem to your satisfaction. It's been an unexpected struggle keeping the network online this past week. I am working the big problems full time and there are resolution plans in place for each. Please hang in a there and you'll be impressed. And to the Lovettsville/Taylorstown folks... this is not normal. You should expect to receive, and I expect to provide uninterrupted service. It will be so.
3 Jun 2005 – Network status: All Lucketts Access Points are operating at full speed. A few of the customers near my place are flipping between two accesspoints, and that causes occasional disconnects while the radio negotiates the new connection. I'm looking at ways to keep the redundency of two APs while preventing the flipping between APs.
Taylorstown and Lovettsville are still not running at full speed. The rain hasn't helped. Jim cleared some trees out of the signal path before the rain, but there were still drop-outs today. There are two APs serving this area, and one of them appears to be dropping connections on its own regardless of the rain. The other, lower, AP is working like a champ but it's not as high. I also installed a new 802.11a high speed link last week but so far that link has not worked.
Here's the plan to fix the drop-outs, both weather related and otherwise. We'll continue to clear leaves from the path of the existing link in an attempt to improve its reliability. It will stay in place as a backup link. The non-functional 802.11a link will come down and I'll replace it with a dedicated 802.11b/g link. I just recieved a pair of very high powered, long-range radios that work with the new routing AP on the tallest pole on the mountaintop. I'll increase the size of the power supply to this AP and install the high power radio. A very high gain antenna will be used to provide a very narrow beam to a matching radio/antenna at my property. This dedicated link will shoot above the trees, avoiding weather problems, and giving a high speed routed connection that will fix some of the more minor problems that you are not even noticing because of the weather problems. While that is in place, I'll work with shorter range 802.11a links in he valley to get more experience with them, with the eventual goal of getting the 11a link re-established to the mountaintop. Esimated implementation day: Sunday.
29 May 2005 – AP0 (the primary access Point on the roof of my house) is on it's last legs. It still downloads fine, but the upload just doesn't work anymore. I believe the storm from a few weeks ago had damaged it as well, but it's just getting around to failing. Later today, I'll be putting it down. The replacement is here and being tested with installation this afternoon. Between now and then, browsing the internet works, but uploading anything (including email) will be painfully slow. Looks like all three access points on my property (on three different poles) were damaged by the storm and have been replaced (or will be shortly).
21 May 2005 – Last spring was easy when we only had 20 customers. New customers please kick me if I suggest putting up an antenna in an "easy" location for now and coming back to move it later. With all of the leaves coming out now, it's "later" now. The worst problem was the new Taylorsville Access Point. Signal started dropping there on Monday and by Friday it was unusable. Physics to the rescue, and with a big radar dish installed, Taylorsville and Lovettsville are back on line. Jim and I also tested a 20ft extension for the antenna pole, and that greatly improved the reception and will also improve the signal to the customers west of Furnace Mountain. We'll install an additional AP and relay on the taller pole Monday night.
Some have asked about backup power supplies for Lucketts.net. We have UPS on all of our equipment and a generator on the property to run everything if power goes out for more than a few minutes. I was planning on upgrading to an auto-start generator this spring, but that has been delayed. Now that I'm managing the network full time, I can start the generator by hand for awhile longer. The money can be better spent on upgrades to the data backbone to improve reliability. Occasionally a power hit that lasts longer than 30 minutes might slip through, but the increase in reliability by providing redundant data backbone will more than make up for the occasional power hit.
The damage to the network during the storm last week was not due to power surges on the AC power grid. Lightning storms can induce electrical surges in the equipment itself, over wires leading to and from the equipment, and even in large metal objects such as towers that connect to the equipment. An UPS or even a power line surge supressor cannot protect you against induced surges. I mentioned last year that everyone should consider using a surge arrestor for the cables leading from their antennas to the interior of their homes. If I had taken my own advise sooner, I wouldn't have lost $2200 worth of equipment last week. I now have a good grounding system and surge arrestors on all long wire runs. Anyone with an outdoor antenna should consider doing the same.
Steve now works for Tammy rather than his old company, and Tammy expects Steve to actually work for a living. There will be new services offered by Lucketts.net in the very near future. PC repair, virus cleaning, spyware cleaning, network installation, house calls, etc will be offered soon. Pricing is being considered and a new list of services will be published soon. Thanks everyone for helping us build the business to the point where we can make it our full time job, and thank you for your patience while we work through the growing pains.
14 May 2005 – Just as the new antenna came online (sorry for the hour drop to 4 of you on that access point) a surge from the thunderstorm took out the tower access point. I scrambled to find what was blown out this time, and it appears that a main switch and the access point itself are toast. Needless to say, on Sunday I'm going to revisit my grounding strategy. Until then, the new AP with the cool new antenna is taking the operational load from everyone to the west that was using the tower, and AP0 (the original house Access Point) is picking up the tower refugees from the east in addition to its normal load.
I'll be climbing the tower to replace AP1 either Sunday or Monday. Until then, we are a little more fragile than normal. If anyone is still offline, give me a call up to 10PM Sat night or anytime after 8AM on Sunday and we can get you back online.
13 May 2005 – Trees on the mountain think it springtime or something. Signal has dropped due to leaf growth. The new, very high gain antenna arrived Friday and will be installed 1st thing Sat morning to correct the problem. A custom high-power transmitter has also been ordered and will be here next week to help a little more. This will correct the occasional dropouts experienced west of Furnace Mountain.
2 April 2005 – Transition to the new servers and T1s has started for a few customers. They will get to play with it for awhile to make sure nothing is broken. If all is well, then I'll send out instructions to everyone else for the migration. Note the change of webmail servers! If you are a Taylorstown account you should use the "NEW" webmail link. Everyone else use the "OLD" webmail link until told otherwise.
25 March 2005 – We've been busy. Service is now extended to Taylorstown and Lovettsville. The new Access Point at Montressor is running. The Stumptown Access Point we were testing will be ready to cover Rt 15 next week. The first of the new backbone links should be active in a few days, increasing oerformance of remote Access Points. New email accounts are being prepared so that the new server can take over next weekend. Instructions on how to use the new server are forthcoming. Expect some emails this week.
10 March 2005 – New T1s are installed and the new mail/domain server is running. I'd like to test things for a week then start migrating everyone. If anyone wants to be a trailblazer, email me and I'll transition you first.
7 March 2005 – Quick update on the upgrade plan. The main Access Point was acting up again, so replacing it cannot wait. I'll change it out this afternoon. Most users can be shifted to the backup AP for the 15 minutes needed to do the switch-over, but there may be a brief interruption. Email me if you have a critical connection today and I can coordinate the break with you. Once the new AP is installed, I'll start connecting the new T1 lines. After a couple days of test, I'll start reassigning static IP numbers to customers by email and phone. Later this week or next weekend, I'll climb the tower to replace its AP so that our redundancy is restored. It will be a busy week.
28 Feb 2005 – 11 AM update: Looks like AOL mail is getting through now. I convinced the upstream provider to modify their reverse DNS records, and coincidently the problem stopped. Maybe AOL turned off their new protection schema, or maybe the DNS change solved the problem. Please let me know if you get any more email rejected by AOL.
28 Feb 2005 – There's always something. Looks like AOL has been rejecting large amounts of our email. According to the web, they have just added new security procedures for their email system to reduce SPAM. This has resulted in lots of small ISPs having trouble getting mail through to AOL addresses. If anyone notices lots of rejections from other providers, please let me know. I'm working with DSL.net to fix this. They control our reverse DNS lookup, and that's what AOL is complaining about.
If you have to get a message thru, use a free email account from hotmail or yahoo. I'll get this fixed as soon as possible. You could have your intended email recipient complain to AOL about the abrupt change in AOL policy that has stopped their email.
20 Feb 2005 – Everything is now restored to pre-storm status (or a little better). You'll be glad to know I now have four spare Access points and 6 spare Access Point POE power supplies. I'm shifting my focus back to improving the weak RF links a few customers have so that everyone consistently gets full speed service. The new backbone architecture will take care of many of those problems, especially increasing reliability for those at the end of many relays. The basic design work for the new backbone is done and a vendor has been selected for the equipment. Once the new T1's are activated by AT&T, (1 March) we'll be ready to start the transition.
The new account/domain server is running. When we do the transition, I'll have to re-create everyone's account on the new server. Passwords do not transfer across machines, so I'll have to coordinate with each customer to give you new passwords. These will be emailed to your old account with instructions on how to do the transition. Once that email is send, all subsequent email will be going to the new accounts. I'll send out more detail after I've run a few tests with the new system.
18 Feb 2005 –
The new T1 wires were installed today. Should be another week or two to get the circuits up. I have some time now to get the new servers
up as well. Given the expenses of the past month, now would be a real good time to refer that neighbor that has been complaining about
dialup or satellite.
14 Feb 2005 – Oops, lost an hour of uptime between 630 and 730PM. The new Access Point software installed last week appears to have the same problem we experienced a couple of times in Dec, every day or so it just turns off everyone's connection. I'll replace it tonight with the fixed software we used in Jan. If you notice the network is down, please give me a call. Tonight at 10PM I'll modify the system to add a backup Access Point, so that if we lose one, the other can pick up the load. That should double the reliability for most users.
[work complete, everything should be back up and running]
13 Feb 2005 – Looks like most of the problems impacting service have been fixed. Turns out that a few customer radios were also damaged by the storm and had to be swapped out. Looks like a broken radio can significantly impact everyone else on the same access point. I've done a scan to make sure everyone gets a good signal again, but there is no test better than what you experience when using the network. Please let me know if you believe your service speed is slow and I can track down the problem.
To check download speeds yourself, I suggest the following simple test. Use your browser and go to www.lucketts.net/downloads. Right click on one of the larger files in the list, select "Save Target as" or "Save Link Target as", and choose a location to copy the download to on your hard drive. If you are asked about overwriting or replacing the file, do that. Watch the download speed or transfer rate, it should average above 125KBps and may be as high as 500 KBps. If the speed occasionally drops below 125KBps, that's OK. Other users are sharing the bandwidth and sometimes the network does get busy. They may even be running the same test. But if your speed is usually or always less than 125 KBps, please let me know as that is below my standard of service.
While there are still a few nagging problems to close out, I've turned my attention back to network improvements. The new Access Point on Stumptown Road at the top of the hill is online, but performance is still shaky. Once it has a solid link to the rest of the network, it will provide service as far south as Montresor. When that access point is in place, we will be able to potentially provide service to the entire area bounded by the mountains and the river as far south as Montresor Rd. There will still be a few spots in forests and behind hills that we cannot get to, but I someone is willing to work with me I will get service to them eventually.
Last summer we offered a referral bonus for any customer that points new business our way. I've continued to honor that through the fall and winter, and a number of customers have received $50 checks in the mail for their referrals. Our application for service now has a line for the new customer to identify who referred them to us. If a current customer's name is on that line, we will send them $50 when the new customer makes their first payment. There are some rules for eligibility and lengthy legal disclaimers, so if you would like to know more, please send me an email requesting details.
New server and additional T1 bandwidth are still in work. The server is ready, but I have to transfer accounts, passwords, web space, and other stuff for each user. Since the T1 install is getting closer, I'll bring the new server up to cover that transition period from the old network to the new. The new T1's will be installed no later than 1 March (per AT&T), and we will get a new range of IP addresses. I'll have to assign new addresses to each customer and work with each of you to transition from old to new. The old server will provide Lucketts.net service to old IP addresses and the new server will be used by the users that have transitioned to the new addresses. If the new T1s are really in by 1 March, I'd like to retire the old T1 line and its IP addresses (and its bill) by 20 March. We should have at lest three weeks to do the transition. A more detailed plan is forthcoming.
31 Jan 2005 – This has been a tough week. My apologies to those that I couldn't respond to immediately. I'm still finding little problems here and there from the storm, plus a few that would have happened anyway. At least everyone is up now and running again.
The electrical snow storm last week took out a lot of equipment. The most vulnerable appears to have been the power supplies. Looks like we lost 4 of those. I've ordered additional spares to have on hand for future storms.
While working on the tower, I discovered that a bad mouse can bring down the network. Corrective action has been taken.
Current Status: All Access points are running. The tower and the test site up the hill on Stumptown Rd. have degraded performance due to power supply problems. Replacements and additional spares are on order and should be here by the weekend.
New T1s are still on track for Feb installation. The new backbone will be delayed a month while I pay for the stuff broken by the storm.
On a good note, congratulations to those that have successfully referred new customers to Lucketts.net. The bounty program worked pretty well over the slow summer, so we decided to continue it. If an existing customer refers a new customer and the new customer signs up for the high-speed service, we'll credit $50 to the existing customer account. If you prefer, we can cut you a check. On the application form the new customer can indicate who referenced them. Only one $50 bounty per new customer.
Last note: if you haven't noticed, we accept VISA and Mastercard now. You can pay the invoice online or we can setup automatic monthly billing.
23 Jan 2005 – Saturday was a tough day for the network. Significant portions of the network went down for no apparent cause between 1130 AM and 1230 PM. I was out doing installs at the time, and my first thought was of course I broke something, again. After reviewing logs and examining equipment all Saturday, I've come to the conclusion that the electrical storm that hit around noon took out parts of the network. The damage was too severe and extensive to be caused by a single problem. Widely separated pieces of equipment had hard failures, and there were soft failures on isolated subnets.
Here's the damage:
Network Center: The main switch at lucketts.net had one port completely burn out, with 2 or 3 others damaged and probably too unreliable to use again. I'm still troubleshooting it. The tower Access Point(AP) had its memory and settings wiped. I've rebuilt the AP and it appears to be running fine now.
AP2 (Brian's): The main switch at this AP had two of its five ports burn out. The remaining ports are carrying the traffic, but it will have to be replaced ASAP. The power transformers for the local receiver and the AP were both burned out and have been replaced. Brian's router also was damaged and is now unusable.
AP4 (Steve S): The AP here had its memory and settings wiped, and its radio/ethernet interface was damaged beyond repair. This AP is still offline and I'll have to order a replacement. I'll try to jury rig something to get Andy R. and Dan W. back online ASAP, but it may take a few days.
If anyone is having trouble accessing the internet, or they can get to some web sites but not others, please let me know. I need that kind of information to discover other damaged or degraded components.
Lesson learned: Long runs of cat 5 cable still need grounding and surge suppression. I have lightning suppressors (see 13 Oct news) suitable for protecting the wires running down from the antennas, but didn't think they would be needed until spring, so I put off installing them here at my place. Looks like I'll be putting them in sooner than I planned. They would not have protected the Access Points, but they would have protected the switches and power transformers.
The planned upgrade of the backbone may be delayed a month so that I can replace the damaged equipment. The new T1(s) are currently scheduled by AT&T to be installed in 3 weeks (about 14 Feb) Instead of waiting for the new T1(s), I'll try to get the new server online sooner to improve mail access and security (per request of many customers).
18 Jan 2005 – Power has been unreliable lately. It went down yesterday and the generator would not start, so we were down for about 30 minutes. When it went down the 2nd time, everything worked as is should and there was no interruption.
If your email stops working, please check your Norton Anti-virus or Firewall program. Recent updates do not play well with email. If you can get to mail through the Lucketts.net homepage, but not from Outlook or Outlook Express, then it's almost certain that your firewall has stopped the mail, even if you haven't changed anything recently.
7 Jan 2005 – You may notice a change with this month's invoices. The new software and services I've been promising have started to arrive. As of Thursday, Lucketts.net accepts VISA & Mastercard for payments. We can accept single payments or recurring payments. New accounting software was also installed so we could handle the cards. Your invoices will look different and now they are stored on a server at the bank. You can even pay your invoices by credit card online!
When you log-in to see or pay the online invoices, you should not use your lucketts.net email password. You should set up a different password. Sorry for this, I cannot set it up so a single password works on my servers and the bank's servers.
Other new stuff in the pipeline: new mail servers, domain hosting, pop3 access to your email from outside of Lucketts.net, domain hosting, and double the T1 bandwidth. Customers hosting repeaters will have access to a new data backbone running at 54 Mbps. We'll probably have to change over to new static IP addresses when the new T1's are installed. I'll try to install the new server at the same time to minimize perturbations.
Old news is archived here: Old News for 2004
