Wednesday, March 2, 2011

memcached + repcached a great Gear6 replacement

I recently started a project looking for a good replacement for our Gear6 appliance. I started testing memcached + repcached as a solution since my needs were purely none persistant memory storage.

* I found memcached + repcached to replicate well and consistantly.

* When I failed either instance of the master-master setup and then brought it back up, the replication was restored.

* I did notice that during the restoration phase- the remaining active memcached instance would saturate the CPU and cause a bit of a "brown-out" due to elevated response times to my active cluster.

All in all- this is a simple, easy to use solution for a master-master memcached.

Other known limitation:

* You can only run 2 servers in this sync mode. For a more extensive solution which is more scalable you may need to look at Cassandra.

Monday, November 15, 2010

How to get more power out of your eeepc and Fedora 14 with the help of Jupiter

Have you noticed that despite your best efforts and best practices you are not able to quite get that promised battery time? Well, I sure have. When I first got my laptop and installed Fedora 13 I was able to get maybe 7-8 real hours while in complete idle more or less (I have the 6 cell 10.5 hour rated battery). This in itself is not bad and I had many setting helping me achieve that including all the tips that "powertop" recommended. Bu the way, if you haven't already installed "powertop" it is a must tool for laptop users to find what is taking your battery power.

After installing Jupiter, I was able to achieve much improved performance.

Jupiter optimizes your power usage by doing several things and giving you additional several features:

  1. It gives uses a CPU governer setting to force your CPU while in battery mode to go to its greatest power saving mode.
  2. It optimizes many of the recommendations given by powertop to save power.
  3. You have an easy quick menu that allows you to quickly toggle bluetooth/wifi/cpu mode on demand.

After installing this, with my wifi working and ~30 screen brightness I had an average power consumption of 6.0-6.2W of power while actively browsing- that is not bad!

A couple of notes/problems:

  • the notify script does not work well with the KDE environment so if you are using KDE, simply delete /usr/lib/jupiter/scripts/notify.
  • Do not install the additional eee package for Jupiter, it is supposed to add a sort of compatible Asus hybrid app but it does not seem to work well at all with this Fedora and Asus eee 1001P.

Sunday, November 14, 2010

Fedora 14 on Asus eee 1001P

It's been quite some time since I've written to this blog so I wanted to post an update with some important information.

Same as Fedora 13, Fedora 14 continues to work well on the Asus platform. For a successful installation, start with these instructions here:

Then follow my instructions here for the Fedora 13 specifics on the Asus eee:

Finally, I have found some issues with the kernel 2.6.34 and higher (Fedora 14 comes with 2.6.35 even). The problems seem to be that the function keys do not respond properly:

1. The volume has a lag and often has a mind of its own.

2. The brightness function is not set properly and even at the brightest setting does not product the brightest option.

I have played a lot with the kernel parameters and could not find a more correct ACPI setting. As such, I recommend using kernel If needed, e-mail me and I will gladly give you links for the RPM's.

Using yum localinstall for the RPM files will place the kernel with no problem. After that, your laptop will work perfectly.

Good luck

Saturday, July 24, 2010

Thoughts on Velocity 2010

I really enjoyed the Velocity 2010 conference. Before I start, here are links to some photos:
Here is a summary of day 1 most interesting notes:
Metrics 101 by Sean Power:
Sean gave me some important insight about metrics for monitoring/application performance, I'll just share my notes "as is" more or less, so forgive me if it's a little raw:
Why collect metrics:
* to detect problems
* user experience
* Needed for a scalable system based on automation
* Mesurement for optimization
Create a baseline.
It's not enough to just collect analytics.
Metrics are used to tie them in to the aspects of your business.
Page load time:
Lots of objects creates latency
3rd party dependency can cause your site to break without your knowledge
Importance of monitoring your external 3rd party.
Look for an external monitoring service with detailed info for performance and what part of the process could be impacting your website.
Synthetic isn't enough-
testing with a browser based test along with scripted tests.
Real user monitoring- Each part of your servers can report performance to a centralized place, then you can parse and get true metrics.
(truesite, tealie)
RUM Pros and Cons
corolates with Analytics you have.
Watches everything
Can be used to re-produce problems
Measures traffic as well as Performance.
May require physical installation
Can be a privacy risk
Summary- shows you that your site is working
Getting the math right:
* Forget about Averages- they miss represent data
* 80th percentile is much more meaningful.
* You can use histograms to build buckets of performance values and see how much fell in to each.
* By using the percentage you can measure 80-95% or the "knee curve" to get a hint that your site can't handle increased load.
* Reporting can't be complicated, when you have complicated data, a lot gets lost.
Consider 3 histogram's for as an example:
Checkout Page
If you had all aggregated, everything looks bad, but per histogram you can find the useful data.
Add context such as:
74% of tranactions took less than 4s
Trends- you can use it to predict and corolate.
This can help with planning- but don't trust it.
Target your metrics for your audiance-
Define it first
How technical
How will they use it?
To fix something
To escalate to others To plan for the future.
What words do they use?
Marching orders:
Collect data and increase a baseline
Evaluate what is causing the most slowdown
DNS etc
Set target thresholds
Train your audiance:
Get them used to the information
All in the same format
At the same time
From the same place.
Some tools to checkout:
Ajaz Gomez Coradiant truesight
Agent based (Aternity)
Coradiant tealleaf Beatbox Atomica labs moniforce
IBM Coremetrics
Open Source
Firebug- can be used to test load time - great external tool to test from several sources
Planning for the future:
Mobile devices are the future
HTML 5 is awesome
Onload time from many placed to the top landing page
Server time from one place, often to your core business process
Server time from one place, often to each tier web/io/cpu)
TopN worst pages by error rate server latencey network latency
I also went to a cassandra workshop. Here were my notes from that:
10 things you should answer when designing your cluster
1. What is the avg record size?
2. What is the service level obkective (latency)
3. What do I know about my traffic patterns
4. How fast do we need to scale this thing?
5. What percentage of my usage is read VS. write?
6. How much of my data is "hot"?
7. How big will this data set be?
8. What is my tolerace for data loss?
9. how many 9s do my {boss users sanity) expect?
While computing is becoming cheaper, managing your data record sizes is important, keep them efficient.
Users expect instant response
Broadband speed exceeds DB latency
Adding a Cassandra node as an "Instant Provision" can cause a slow down because it needs to transfer all your data.
REad vs write Am I picking the right tool?
Writing is 10X faster than reading
But reading is generally what the user "feels".
Write your data as you want to read it.
Try to detect if you have any "Hot spot" of data, is your data evenly divided in your cluster? load?
8 nodes per cluster is the recommended amount so you can scale and perform maintenance easily with minimal impact to your production environment, helps distribute the data even more.
Do we need to backup?
Yes. (Snapshot?)
jconsole- is your friend
Especially vm Summary
Backup strategy?

Saturday, June 19, 2010

Preparing for O'rielly Velocity conf. 2010

This Tuesday (the 22nd) I'll be going to the Velocity conference in California. Some of the events look really interesting

Here are a few of the ones that strike me as interesting for the first day:

(taken from the O'rielly Velocity website)

Metrics 101: What to Measure on Your Website

Web Performance

Sean Power (Watching Websites)

This session will help you build a complete web monitoring strategy. We'll cover the many different metrics you can collect, from latency and uptime to usability and navigation - and show you how to tie them to the goals of your web business. Read more.

Scalable Internet Architectures

Internet traffic spikes aren't what they used to be. It is now evident that even the smallest sites can suffer the attention of the global audience. This presentation dives into techniques to avoid collapse under dire circumstances. Looking at some real traffic spikes, we'll pinpoint what part of the architecture is crumbling under the load; then, walk though stop-gaps and

Cassandra Workshop

This workshop will cover Cassandra design, deployment, and operations in both a theoretical best case target for those building their own facilities as well as an evaluation of how it can be deployed on various public cloud platforms. Read more.

I'll post a summary and thoughts after I attend.

Wednesday, June 16, 2010

New Kernel is more eee laptop friendly!

Recently Fedora released kernel, I noticed immediately that on my Asus eee 1001P the power consumption got much better, extending my battery life ~15-20%. Looking through the changelog I saw some extra support for chipset's, but I'm not sure if this is what did it.
Here is a small example of power top.
Settings during this test:
Screen- 30%
KDE desktop (no "compiz" type effects)
Firefox open with 3-4 tabs (Flash is disabled with "flashblock")
Thunderbird (with davmail)
Wireless network with medium reception (Bluetooth disabled)

After running for an hour, here is a screen shot of my "powertop" read out:

Friday, June 4, 2010

Asus eee 1001P and Linux Fedora 13

I wanted to share with everyone some my Fedora 13 installation experience.
It seems that everyone is installing the netbook version of Ubuntu but I have been a long time user of the Redhat type Linux. For a good number of years I used Mandriva (known then as "Mandrake") and was really pleased but since Fedora 10 there has been a gradual improvement. Fedora seems to have the latest and greatest out there and the system is well tailored together. Fedora 13 definitely creates a new step up and I'm rather pleased.
The first day I received my Asus 1001P I started off with preparing it for a dual boot OS. I am not anti Windows, I think at times it could be useful and I believe it still may have a certain edge with battery life over Linux. In addition- the starter edition loads up FAST and seems to be pretty snappy, so I recommend keeping it around.

Preparing for dual-boot
I used under Windows a program called "Easus", the home edition is free for private use and seems to only limit you on creating new partitions (which I did from Windows or Linux itself). I recommend the following:
  • Shrink Windows to 40GB partition.
  • Boot in to Linux rescue and create an extended volume.
  • Create a 40GB Linux partition (Could be less, in reality I'm using maybe 6GB-10GB).
  • Create a 512MB partition (for SWAP). 
Go back in to Windows, load up Easus- grow the "data" volume to Maximum, should be around 140GB.
Use Easus to format the 140GB "data" partition to Fat32 (This is used as a shared partition between the two OS's). This strategy will allow you to format Linux and re-install without affecting the data partition and also allow you to to a restore operation at boot time to restore your Windows partition from the hidden partition on disk and not lose data.

You are now ready to install Fedora 13.
Installation process should be pretty streight forward. I installed from a USBkey image (8GB) containing the entire DVD image. I did this "livecd-iso-to-disk" utility on the ISO image of Fedora (livecd-iso-to-disk --reset-mbr).

In the section for partition installation method I chose custom and made sure to select the partition I reserved for Linux (format as well) and the small 512MB partition for SWAP.

In the boot menu section I renamed "other" to "Windows 7" (which came with my laptop) in order to make it clear.
I chose to "choose now" on the package install section and made sure to install both gnome and KDE desktops. I am a big fan of KDE, especially with the recent improvements and find KDE 4.4 to be nothing like earlier versions as far as stability and resources.Once the installation is complete, pretty much everything should be working from networking down to the video camera. However, there are a number of adjustments required:

Fix for skype:
If you installed a 64bit OS (which is what I did) you will need to install several packages of 32 bit before you do a full yum update, I learned that later on some strange dependencies happen due to the 64bit version being newer the the 32bit and failing to install 32bit versions. I actually re-installed at this point due to understanding that.
Run the following commands:

yum install libXv.i686*
yum install libXScrnSaver.i686*
yum install qt.i686
yum install qt-x11*.i686
Now install the Fedora 10+ RPM available from Skype (2.1 beta).

Run update!
There are quite a number of GUI little bugs that got fixed after running a full update. It will take most likely a couple hours to complete since when I did it we were talking about 500MB of updates.
To make this process faster run:
yum install yum-plugin-fastestmirror
yum update

Screen brightness control:
As reported by other people with eeepc's, the brightness does kind of act random when using the fn-brightup/brightdown. The fix for this is to apply to your kernel parameters under grub.conf the following line:
"acpi_osi=Linux acpi_backlight=vendor"
Reboot and you should have normal controls.

Sound card:
The sound card works out of the box but does have a couple bugs:
1. When plugging in headphones the sound continues to work on both the external speakers and headset. This seems to be due to the jack sense not being recognized. I found that this entry did the trick:
edit the file /etc/modprobe.d/dist-alsa.conf with your favorite editor and make sure it has this:
install snd-pcm /sbin/modprobe --ignore-install snd-pcm && /sbin/modprobe snd-seq
options snd-hda-intel model=auto position_fix=1
(Basically I added "position_fix=1").
Once again, reboot or reload the modules if you know how and it should be solved.
2. Microphone under skype not working.
This one was a little bit of a chalance since skype on a 64bit platform to begin with was a chalange. It did work OK with hearing audio and video conferencing even, just the microphone didn't pick up. The KDE mixer was not much help since it only showed 1 channel for capture, 1 channel for output and no special controls. I installed padevchooser:
yum install pavucontrol.
I then ran in and balanced my controls as follows:
input devices tab:
40% left
80% right.
This seemed to me optimal after lots of trial and error for skype. Still lots of background noise but it will be good enough to have some medium quality conversation.

Battery performance maximizing:
While the eeepc_laptop module does load, I did find some adjustments required to be more efficient on battery life. I am now able to get close to 8 hours of battery from my 6 cell 48W battery with the following parameters:
Wireless Internet on, 40% screen brightness, Browsing and typing/chatting with quite a few applications running. (skype, Empathy, Thunderbird, Firefox, Openoffice).
Some adjustments to the kernel params:
Start with adding these to your kernel params under /etc/grub.conf:
This will utilize power management functionality with the eeepc_laptop module (which should be loaded).
I also added the following parameters to my /etc/rc.local based on recommandeation from "powertop":
modprobe deadline-iosched
echo 1500 > /proc/sys/vm/dirty_writeback_centisecs
echo min_power > /sys/class/scsi_host/host0/link_power_management_policy
It could be I should have added the module under /etc/modules.d but I guess I was testing first.

Some extra control:
I installed an app called "Kill switch manager" for the KDE desktop (you can find it on google). It seems to be rather "Beta" but does offer the ability to graphically turn on/off your bluetooth or wireless.

Next, edit your services, there are many that can be disabled. I will tell you the only ones I have on, you can research and choose more if you feel you need it but for the most part I don't think you will:
bluetooth (assuming you have a 1001P with bluetooth version, there is one without too. Find out)
This should make your laptop a lean, efficient but powerful machine.

I'm guessing that thanks to the synaptics driver, the mouse pad works pretty well. I was disapointed that for me, at least, the two finger scroll option under the KDE System Settings->Keyboard & Mouse->Touchpad didn't work. Also the tap to click didn't seem to have any effect. I did a lot of playing around trying to manually add it to the X11 configuration but to no evail. The scrolling works out of the box by sliding your finger along the right side of the touchpad.
One thing I was able to at least correct is the pointer going crazy if you accedently touch it with two fingers.
My /usr/share/X11/xorg.conf.d/50-synaptics.conf looks like this:
Section "InputClass"
Identifier "touchpad catchall"
Driver "synaptics"
MatchIsTouchpad "on"
MatchDevicePath "/dev/input/event*"
# Quirks for special touchpads
# Some devices have the buttons as part of the lower edge of the touchpad.
# Pressing a button moves the cursor, causing cursor jumps and erroneous
# clicks.
# Use the synaptics area property to work around this.
Section "InputClass"
# Identifier "touchpad button overlap"
Identifier "tap-by-default"
MatchIsTouchpad "on"
# MatchTag "touchpad_button_overlap"
MatchTag "tap-by-default"
Option "AreaBottomEdge" "4000"
Option "SHMConfig" "on"
Option "ClickFinger2" "right"
Option "TapButton1" "1"
Option "RBCornerButton" "3"

Update 06/12/2010
I have solved the tap-to-click problem I was having in KDE, I simply removed from my ~homedir/.kde/share/config/kcmtouchpadrc, there must have been legacy settings there conflicting, the KDE re-creates it next login with correct settings.

Some KDE adjustments: I recommend avoiding too many plasma type widgets, some of them are quite heavy on CPU.
I also resized my task bar on the bottom to be a little smaller and chose a "Oxygen netbook" theme which seemed a little more optimized. Aside of a few large configuration menu's from firefox plugins, I have not encountered too many applications which had problems with the screen size. 

Here is a small example of my desktop: