[1 Sept 2011] More restaurant stuff: We're now open 7 days per week. We are at the "On a Whim" Antique store for another week or so, then moving to the corner of Spinks Ferry and Rt 15 ASAP. Thank you for the response to our email notifying customers of the new service. Of 780 emails messages sent out to customers notifying them of the restaurant, 105 sent back positive comments, 674 did not respond, and one customer complained about us sending an unsolicited advertizement. For those of you that might have been mildly irritated at receiving a restaurant notice from an internet company, thank you for being neighborly and giving us a pass. We promise not to make a habit of it. As a result of the complaint though, we'll be posting soon an update to our privacy policy.

[10:30 23 July] All access points running well after heat caused power outages in Waterford area yesterday. We only lost 1 UPS due to heat, but power was out to many subdivisions for many hours, eventually running down our batteries. We have generators staged to help us if it happens again today.

[5:30 3 July] Large thunderstorm approaching and power is already out to much of Lucketts. Expect more power outages. We have generators, but can be overwhelmed if too many sites are down.

[1 May 2011] New network status map with additional sites. Your site may have moved.

[0900 11 March] Lightning zapped one of our main routers overnight and knocked half the network offline until 7:15 this A.M. We already had a new improved router in place as a planned upgrade, but we had hoped to schedule the downtime to make the transition rather than waking up to the problem. At least it is done. We're still replacing other infrastructure, and with the added redundancy we're building up there should be fewer outages from single point failures.

[27 Jan 3:30 PM] All access points back up and running. All of our internet providers now fully functional. Phones are working properly now that they no longer forward to a cell with a full mailbox. If you are still having individual connection difficulties, please give us a call so we can track it down for you.

[26 Jan] There will be serious problems tonight with snow & ice buildup on antennas. We will adjust routes as much as possible to keep up, but there will likely be outages.

[10:50 AM Sunday 2 Jan ] We are replacing one of our mail servers between now and 1PM today. Sending and receiving mail for most of our hosted domains will be interrupted during that time. If you have an email than must get in or out, give the office a call and we can set up a workaround.

[31 Oct 2010] Lost the primary router today at 1PM. Don't know why yet, we've been to busy getting the network back online. That router is the only single point of failure that can bring down the entire network. We have a spare, but of course nobody was in the office to bring it online. The network is now back up, and we're now fixing the remaining small outages. Those are mostly from recent configuration changes (i.e. new customers and sites) that were not yet stored in the backup systems. 99% of customers are up now, and we'll be bringing the rest online in minutes. If you are still offline after 4PM, check your connections/cables to make sure nothing got unplugged while you were trying to figure out what was wrong, and then give us a call if needed.

[6 Sept 2010] Fiber is installed on our property, and they are still working on the upstream connections. We should be able to turn on the redundant circuit in a week or so.

[25 Aug 2010] Phone system glitch today. T1s were turned off and they took the phones with them. Took almost an hour to figure it out. Good thing is that it cannot happen again.... Verizon T1s are now retired.

[8:00 PM] Hopefully last entry for the night: Everything online and 100%. Washingtonian has commercial power restored, batteries are recharging, storms have passed through. Fiber optic trench is in place and the due date for the install is 24 Aug!!!!

[6:25] Severe thunderstorm activity knocked out the link at 5:40. It's starting to clear now, so we should be regaining signal soon.

[3:20 PM] I forgot to announce the good news. All circuits back up 100% at 12:50 PM. We are shuttling batteries up and down 26 flights of stairs to keep everything running. Large Pepco power outage in Gaitherburg affecting systems that are not supposed to be affected.

[11:20] Some days it does not pay to get out of bed. The batteries did not last long. New batteries are at the building, just 26 flooors down and being carried up now. Until then only web pages work.

[10:45] All systems back online at full speed. We delivered spare heavy duty UPS to our provider 26 floors up the Washingtonian. Power failed there. We will be relaying fresh batteries up and down the building until power is restored. There may be short interruptions later as we change out batteries, but we'll try to minimize those.

[10:00] OK.. everything should be reasonably online now. VPNs, VoIP, web browsing, mail, etc should work, just slower than usual. Some web pages will fail because they are not compatible with the proxies we are using as a backup. We can fix those web pages individually. If you need a specific web site that fails while others work, email the details to admin@lucketts.net and we can make it work.

[9:40] Got the backup running again. Did not expect it to glitch so when it caught us by surprise when it shut itself off after 15 minutes on.. We'll monitor it more closely while we work on bringing up the main system.

[9:15] . Something went wrong with one of the backups and I'm trying to figure it out. Cannot answer phones at the same time.

[8:15 AM] VPNs should be back online. Everything will be slower than normal until we get the primary provider back online. We'll be restricting gaming in favor of VoIP and VPNs.

[7:50AM] Emergency backup now online. Web browsing and mail work - slowly. WE're adding in additional backup and will get VPNs working again ASAP.

[3:00 AM 7 Aug] We got the second backup link activated a little after midnight to increase browsing speeds and to bring non-browsing activities back online. The primary link was restored at 2:45. The permanent fix is the trenches we dug the Thursday and Fridays for the new fiber. We are working as fast as possible to no longer be reliant on the power problems of our upstream provider. We'll post a fiber ETA as soon as we get a schedule.

[11:PM 6 Aug] We know about it, took awhile to get bodies back in the office to find the problem.. Gaithersburg power outage affecting us of all things. Backup system online now providing a minimal connection. Additional circuits being brought up to get speeds up to a more usable level.

[11:30: 26 July] Our phones had a problem all morning that we just became aware of. Only some of the calls were getting through to us because because most were being forwarded to a dead line. We've fixed that line. I suppose there will be more phone calls now, and we thought we were doing pretty well with only about 1/4 the normal calls on an event like this.

[6:30AM 26 July]: Main network is up and running. Power still out in much of Lucketts. Access points with power deliver internet fine. If you want to know what Access Point you are on, give us a call. Here is the status map of all access points:

10:15 Most customers were offline and our providers were down so we didn't update this page much - we were out replacing generators and batteries. Our primary internet line is now back, and we have brought up some access points so many customers are back on. Everyone else should be one as soon as power is restored, and if power is still out Monday morning we'll deploy some more generators.

4:00 Primary provider back offline. No internet right now... keep checking this site.

Sunday 3:30 Big storm ... reports day 90 MPH sustained winds. We lost part of roof, generators and multiple antennas. Assessing damage now. All three of our providers are offline. As soon as one come back up we'll start restoring service where we can. We will post an estimate as soon as we can figure out how much is broken.

24 July: This upstream provider nonsense is getting old. We have a contract in place and start trenching next week to install a fiber to our network so that we have an independent source of bandwidth.

4:15 PM System restored. We will get a report on why it was so bad. This time the upstream provider disabled BGP for our domains, so the alternate providers could not provide service.

3:30 PM 22 July: Upstream provider just went offline completely. We're rerouting, should have emergency backup link up in minutes, large backup link up in a hour or so. Check here for updates.

[Monday 19 July 11:55 AM] We are having network problems with an upstream provider. They did an emergency router replacement and during the process created a routing loop that knocked us offline. We're working on it now. ETA for a fix is 15 minutes.

[9:50 28 Jun 2010 ] Power outage in Lucketts area - power lines down on Wilt Store Rd. A few sites are already impacted, and more may go offline as the outage continues.

[1:00 PM 23 June] We are back on the primary link with commercial power and do not expect any additional problems.

Our provider lost cell tower power in Maryland, cutting 2 of our links. We went out and provided generators to them to mitigate the problem until they could fix the commercial power. Our primary and main backup run through that site - cell sites are not supposed to go down. But since they do, we are working with 3 different providers too get an alternate connection to provide complete redundancy.

[7:45 AM Wed 23 Jun] System wide Network problem: upstream provider went offline at 4AM. We're setting up bypasses now to get traffic going again. We'll post status here.

[Sunday, 6 June 2010) All Access points are operational. We are currently backlogged with customers reporting individual outages. Most are because equipment, usually the home router, needs a reboot after the recent storms. We're working through the backlog of individuals to help them get back online. If we haven't gotten to you yet, try shutting down the PC, turning off power to the router and our radio. Then turn the radio back on, wait 60 seconds, turn the router back, wait 60 seconds, then turn the PC back on, wait 60 seconds. If this doesn't get you back online then it is likely there is some damage to equipment, usually your router but sometimes the radio. Check that the internet light on your router is on to verify that the router and radio are still connected.

[5 may 2010] Would you like to see a sample of our new network status board? Link here for static version. We'll make a live version viewable by customers ASAP.

[10:00 25 April ] Main link restored. Lightning caused an upset that had to be manually restarted. Backup systems back to standby. We are getting a 2nd major link provider so that manual backup routes will no longer be needed. We'll announce details as soon as contract signed.

[9:30 25 April] Lightning knocked out our upstream provider. On the phone with them now to get an estimate. Web pages working again through backup proxy servers.

[ 2PM 9 April 2010 ] Detailed update on network status: We still only have 2 of our 4 links up. To make matters worse this morning, #3 went down for a few hours and we had to scramble to get it back up. Here's the status:

DOWN - Link 1 - Primary internet connection - Waiting on Trango to fedex Saturday delivery the replacement part in from LA, install ASAP after that.

DOWN - Link 2 - Fully functional backup link, less bandwidth than #1 - weather damage on cell tower, tried to fix Thursday, trying again on Friday - climbing at 4PM.

UP - Link 3 - Supports Proxy connections only - Converting this link late afternoon to be fully functional like #2, just a little slower.

UP - Link 4 - Supports VoIP, VPN, games, and everything not running across links 1-3 but with very limited bandwidth - emergency lifeline.

We are contracting now for a new backup that will be equal to link #1, effectively doubling our speed with fully functional bandwidth - gives complete redundancy.

Here are a few pictures of the cell to give you some perspective: pic1 pic2 pic3

[ 9:20 PM 7 April ] We're on backup internet connections now - one of our service providers blew a fuse on a cell tower. Web pages are being routed through a proxy server, so some sites may not work quite right or at all. All other traffic is routing through T1s and is very limited. We'll balancing the T1s to try to keep Voip functional, but games, video, P2P download and other non-web page stuff will be impacted.

[ Wed 8:15 pm 7 April ] MAIN LINK PROBLEM! We're working on fixing it and switching to backup at the same time. We'll update status ASAP... but it's a little busy here right now.....

[9:30 13 Feb] We have staff in today so let us know if you are offline. Most problems are because of antennas buried by snow, but normal problems still happen. We'll have staff here both Sat and Sun to provide support.

[ 10:29 AM 12 Feb ] It appears we have fixed the email problem, if you are still having difficulty accessing Lucketts.net Email, Let us know!

[ 9:25 AM 12 Feb ] Nortons Anti Virus Protection is blocking Lucketts.net, as a result if you have Nortons you are probably not receiving Lucketts.net email.

[ 9:23 AM 12 Feb] Lucketts Rd is still closed from Firestation to St Clair. Thru traffic is not possible.

[Tues 11PM ] main link back online carrying traffic. There will be some more testing and tuning over then next few days, but full speeds should be available again.

[Tuesday 6:15 PM] Equipment installed and doing final testing. Should be able to turn on main link by 8PM.

[Tues 9 Feb 10:45 AM] We are at 50% capacity, and are adjusting speeds in real time. Current speeds are at at least 750 Kbps download and 500 Kbps upload - Everything still works at 750K, after all it is about the same as our competitions normal offering. Speeds will increase during the day as we balance the system. Our priority today will be tuning for stability, not tuning for most speed. That way VoIP and VPN will work reliably for emergency support and preparing for the storm. If Fedex delivers today, we should have main link back up in the afternoon.

[Mon 8 Feb 10:00PM] main backup back online. All avg speeds on this backup should now be around 1000Kbps.

[Mon 8 Feb 6:15PM] Primary backup coming online tonight. This will restore VPNs and games.

[Sun 7 Feb 5:45PM] New power backup at Cottege Grove. This battery will only run for about 5 to 6 hours, so use the internet now if you need to. Power is not expected back until Tuesday We will deploy a generator there Monday AM.

[Sun 7 Feb 1:15 PM] Power on at Rt15/ Wilt Store and at PointofRocks Cell tower - service restored there. Still working on Milltown access point. That's the one Bucky uses so he'll want it running before he goes home today.

[Sun 7 Feb 11:15 AM] Storm update: Access points offline due to extended power outage: Milltown/Cottege grove - power company says later this afternoon; PointOfRock Cell tower - no estimate from power company, generator being deployed there as soon as we can plow through the mountain road; RT15/Wilt Store/ - No estimate yet; Hamilton Station - No estimate yet.

[3:30 PM 6 Feb ] When it rains it pours: The problem with our main line came back today. This time, after some confusion, we have a backup in place that will run the network about 50%. To keep things working right for now, port 80 web downloads will be 1 Mbps max, and other downloads. Proxies are turned on, so if you have trouble getting to a specific website, email us the details and we'll bypass the proxy for that site. Non web traffic will be caped significantly so that phones and VPNs will work through the T1s - starting at 250Kbps for now, but I'll adjust it dynamically to keep the latencies low enough that VoIP will work. We have them send a new radio fedex this time rather than sending a tech to repair old unit.

[1:30PM 6 Feb] Storm Update: Only known network issues are power related: Rt15 from Wilt Store to Potomac Overlook and Milltown at Cottege Grove are out of power, no ETA from power company. If you are unable to connect, give us a call (and you should probably try rebooting your router.)

26 Jan 2010: We had a big problem with the wind storm yesterday. A very large antenna mount was twisted and some of our internal links were also broken as a result of the high winds. It was very hectic here in the morning as we were rushing around trying to fix everything without breaking anything else in the process. We were almost successful, a primary router was zapped and we had to reconfigure it and many of its routes. The morning was grim... sorry we didn;t get a good chance to post progress. Problem was, with the messed up router, most people would not have been able to see any posting, so we just worked straight through on that. We had 99% of folks back online in the morning, and it took the rest of the day to fix the special routing cases.

Good news front: we have the new bandwidth filtering out to users. Everyone should have had a faster December, or at least more consistently fast. We're working on improving the internal distribution of bandwidth so we can start raising speeds again across the board.

We have a VoIP system working in the office now. We'll test that for awhile and then start offering it to customers ASAP.
We're working the finishing touches now, adding things like fax support, mailboxes, remote access, and integrating the billing.