11:30: Our phones had a problem all morning that we just became aware of. Only some of the calls were getting through to us because because most were being forwarded to a dead line. We've fixed that line. I suppose there will be more phone calls now, and we thought we were doing pretty well with only about 1/4 the normal calls on an event like this.

[6:30AM 26 July]: Main network is up and running. Power still out in much of Lucketts. Access points with power deliver internet fine. If you want to know what Access Point you are on, give us a call. Here is the status map of all access points:

10:15 Most customers were offline and our providers were down so we didn't update this page much - we were out replacing generators and batteries. Our primary internet line is now back, and we have brought up some access points so many customers are back on. Everyone else should be one as soon as power is restored, and if power is still out Monday morning we'll deploy some more generators.

4:00 Primary provider back offline. No internet right now... keep checking this site.

Sunday 3:30 Big storm ... reports day 90 MPH sustained winds. We lost part of roof, generators and multiple antennas. Assessing damage now. All three of our providers are offline. As soon as one come back up we'll start restoring service where we can. We will post an estimate as soon as we can figure out how much is broken.

24 July: This upstream provider nonsense is getting old. We have a contract in place and start trenching next week to install a fiber to our network so that we have an independent source of bandwidth.

4:15 PM System restored. We will get a report on why it was so bad. This time the upstream provider disabled BGP for our domains, so the alternate providers could not provide service.

3:30 PM 22 July: Upstream provider just went offline completely. We're rerouting, should have emergency backup link up in minutes, large backup link up in a hour or so. Check here for updates.

[Monday 19 July 11:55 AM] We are having network problems with an upstream provider. They did an emergency router replacement and during the process created a routing loop that knocked us offline. We're working on it now. ETA for a fix is 15 minutes.

[9:50 28 Jun 2010 ] Power outage in Lucketts area - power lines down on Wilt Store Rd. A few sites are already impacted, and more may go offline as the outage continues.

[1:00 PM 23 June] We are back on the primary link with commercial power and do not expect any additional problems.

Our provider lost cell tower power in Maryland, cutting 2 of our links. We went out and provided generators to them to mitigate the problem until they could fix the commercial power. Our primary and main backup run through that site - cell sites are not supposed to go down. But since they do, we are working with 3 different providers too get an alternate connection to provide complete redundancy.

[7:45 AM Wed 23 Jun] System wide Network problem: upstream provider went offline at 4AM. We're setting up bypasses now to get traffic going again. We'll post status here.

[Sunday, 6 June 2010) All Access points are operational. We are currently backlogged with customers reporting individual outages. Most are because equipment, usually the home router, needs a reboot after the recent storms. We're working through the backlog of individuals to help them get back online. If we haven't gotten to you yet, try shutting down the PC, turning off power to the router and our radio. Then turn the radio back on, wait 60 seconds, turn the router back, wait 60 seconds, then turn the PC back on, wait 60 seconds. If this doesn't get you back online then it is likely there is some damage to equipment, usually your router but sometimes the radio. Check that the internet light on your router is on to verify that the router and radio are still connected.

[5 may 2010] Would you like to see a sample of our new network status board? Link here for static version. We'll make a live version viewable by customers ASAP.

[10:00 25 April ] Main link restored. Lightning caused an upset that had to be manually restarted. Backup systems back to standby. We are getting a 2nd major link provider so that manual backup routes will no longer be needed. We'll announce details as soon as contract signed.

[9:30 25 April] Lightning knocked out our upstream provider. On the phone with them now to get an estimate. Web pages working again through backup proxy servers.

[ 2PM 9 April 2010 ] Detailed update on network status: We still only have 2 of our 4 links up. To make matters worse this morning, #3 went down for a few hours and we had to scramble to get it back up. Here's the status:

DOWN - Link 1 - Primary internet connection - Waiting on Trango to fedex Saturday delivery the replacement part in from LA, install ASAP after that.

DOWN - Link 2 - Fully functional backup link, less bandwidth than #1 - weather damage on cell tower, tried to fix Thursday, trying again on Friday - climbing at 4PM.

UP - Link 3 - Supports Proxy connections only - Converting this link late afternoon to be fully functional like #2, just a little slower.

UP - Link 4 - Supports VoIP, VPN, games, and everything not running across links 1-3 but with very limited bandwidth - emergency lifeline.

We are contracting now for a new backup that will be equal to link #1, effectively doubling our speed with fully functional bandwidth - gives complete redundancy.

Here are a few pictures of the cell to give you some perspective: pic1 pic2 pic3

[ 9:20 PM 7 April ] We're on backup internet connections now - one of our service providers blew a fuse on a cell tower. Web pages are being routed through a proxy server, so some sites may not work quite right or at all. All other traffic is routing through T1s and is very limited. We'll balancing the T1s to try to keep Voip functional, but games, video, P2P download and other non-web page stuff will be impacted.

[ Wed 8:15 pm 7 April ] MAIN LINK PROBLEM! We're working on fixing it and switching to backup at the same time. We'll update status ASAP... but it's a little busy here right now.....

[9:30 13 Feb] We have staff in today so let us know if you are offline. Most problems are because of antennas buried by snow, but normal problems still happen. We'll have staff here both Sat and Sun to provide support.

[ 10:29 AM 12 Feb ] It appears we have fixed the email problem, if you are still having difficulty accessing Lucketts.net Email, Let us know!

[ 9:25 AM 12 Feb ] Nortons Anti Virus Protection is blocking Lucketts.net, as a result if you have Nortons you are probably not receiving Lucketts.net email.

[ 9:23 AM 12 Feb] Lucketts Rd is still closed from Firestation to St Clair. Thru traffic is not possible.

[Tues 11PM ] main link back online carrying traffic. There will be some more testing and tuning over then next few days, but full speeds should be available again.

[Tuesday 6:15 PM] Equipment installed and doing final testing. Should be able to turn on main link by 8PM.

[Tues 9 Feb 10:45 AM] We are at 50% capacity, and are adjusting speeds in real time. Current speeds are at at least 750 Kbps download and 500 Kbps upload - Everything still works at 750K, after all it is about the same as our competitions normal offering. Speeds will increase during the day as we balance the system. Our priority today will be tuning for stability, not tuning for most speed. That way VoIP and VPN will work reliably for emergency support and preparing for the storm. If Fedex delivers today, we should have main link back up in the afternoon.

[Mon 8 Feb 10:00PM] main backup back online. All avg speeds on this backup should now be around 1000Kbps.

[Mon 8 Feb 6:15PM] Primary backup coming online tonight. This will restore VPNs and games.

[Sun 7 Feb 5:45PM] New power backup at Cottege Grove. This battery will only run for about 5 to 6 hours, so use the internet now if you need to. Power is not expected back until Tuesday We will deploy a generator there Monday AM.

[Sun 7 Feb 1:15 PM] Power on at Rt15/ Wilt Store and at PointofRocks Cell tower - service restored there. Still working on Milltown access point. That's the one Bucky uses so he'll want it running before he goes home today.

[Sun 7 Feb 11:15 AM] Storm update: Access points offline due to extended power outage: Milltown/Cottege grove - power company says later this afternoon; PointOfRock Cell tower - no estimate from power company, generator being deployed there as soon as we can plow through the mountain road; RT15/Wilt Store/ - No estimate yet; Hamilton Station - No estimate yet.

[3:30 PM 6 Feb ] When it rains it pours: The problem with our main line came back today. This time, after some confusion, we have a backup in place that will run the network about 50%. To keep things working right for now, port 80 web downloads will be 1 Mbps max, and other downloads. Proxies are turned on, so if you have trouble getting to a specific website, email us the details and we'll bypass the proxy for that site. Non web traffic will be caped significantly so that phones and VPNs will work through the T1s - starting at 250Kbps for now, but I'll adjust it dynamically to keep the latencies low enough that VoIP will work. We have them send a new radio fedex this time rather than sending a tech to repair old unit.

[1:30PM 6 Feb] Storm Update: Only known network issues are power related: Rt15 from Wilt Store to Potomac Overlook and Milltown at Cottege Grove are out of power, no ETA from power company. If you are unable to connect, give us a call (and you should probably try rebooting your router.)

26 Jan 2010: We had a big problem with the wind storm yesterday. A very large antenna mount was twisted and some of our internal links were also broken as a result of the high winds. It was very hectic here in the morning as we were rushing around trying to fix everything without breaking anything else in the process. We were almost successful, a primary router was zapped and we had to reconfigure it and many of its routes. The morning was grim... sorry we didn;t get a good chance to post progress. Problem was, with the messed up router, most people would not have been able to see any posting, so we just worked straight through on that. We had 99% of folks back online in the morning, and it took the rest of the day to fix the special routing cases.

Good news front: we have the new bandwidth filtering out to users. Everyone should have had a faster December, or at least more consistently fast. We're working on improving the internal distribution of bandwidth so we can start raising speeds again across the board.

We have a VoIP system working in the office now. We'll test that for awhile and then start offering it to customers ASAP.
We're working the finishing touches now, adding things like fax support, mailboxes, remote access, and integrating the billing.