Showing posts with label scripts. Show all posts
Showing posts with label scripts. Show all posts

Apr 3, 2013

ab 漸進壓力測試工具

最近在學Go,也用Go寫一個 ab-like 的玩具(Go-HttpBench),為了要比較兩者效能另外寫了 bash script 用來轉換 ab 報表為行列數據,功能跟 autobench 相似,用階段性漸增的壓力來測試 http server,搜集到的數據可再用 gnuplot 或用 excel 繪制圖型

Example:

[email protected]:~/workspaces/go/gb$ ./ab+.sh -k http://localhost/
Start benchmark
#/usr/bin/ab -c CONCURRENCY=(100 to 10000 step 100) -n NUM=(100*CONCURRENCY) -k http://localhost/
#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%
Wed Apr  3 11:52:29 CST 2013,100,10000,0,31272.48,0,3,5.4,0,32,11,16,22
Wed Apr  3 11:52:29 CST 2013,200,20000,0,32050.72,0,6,14.2,0,113,23,40,64
Wed Apr  3 11:52:30 CST 2013,300,30000,0,33527.94,0,5,11.6,0,207,16,28,53
Wed Apr  3 11:52:31 CST 2013,400,40000,0,32396.87,0,12,25.0,0,189,48,70,105
Wed Apr  3 11:52:32 CST 2013,500,50000,0,32492.75,0,15,35.9,0,391,51,109,166
Wed Apr  3 11:52:34 CST 2013,600,60000,0,33188.85,0,18,38.8,0,448,73,110,166
Wed Apr  3 11:52:36 CST 2013,700,70000,0,33436.38,0,21,46.9,0,483,71,134,221
Wed Apr  3 11:52:38 CST 2013,800,80000,0,33370.64,0,24,54.5,0,1015,74,145,264
Wed Apr  3 11:52:40 CST 2013,900,90000,0,34093.27,0,26,66.4,0,1150,78,133,260
Wed Apr  3 11:52:43 CST 2013,1000,100000,0,34448.17,0,27,68.4,2,1251,85,136,266
Wed Apr  3 11:52:46 CST 2013,1100,110000,0,34072.87,0,32,78.1,0,1129,101,170,368
Wed Apr  3 11:52:49 CST 2013,1200,120000,0,33332.79,0,36,86.2,0,1032,144,228,386
Wed Apr  3 11:52:53 CST 2013,1300,130000,0,34580.66,0,37,96.0,5,3349,116,201,346
Wed Apr  3 11:52:57 CST 2013,1400,140000,0,33776.24,0,41,96.3,0,689,153,262,475
Wed Apr  3 11:53:01 CST 2013,1500,150000,0,31418.26,0,47,114.7,0,1031,164,295,533
Wed Apr  3 11:53:06 CST 2013,1600,160000,0,33285.65,0,48,111.8,0,629,217,340,492
Wed Apr  3 11:53:11 CST 2013,1700,170000,0,33441.14,0,50,121.5,0,825,211,356,559
Wed Apr  3 11:53:16 CST 2013,1800,180000,0,31452.71,0,52,166.0,0,4253,163,294,496
Wed Apr  3 11:53:22 CST 2013,1900,190000,0,33781.89,0,56,139.8,0,1018,234,425,627
Wed Apr  3 11:53:28 CST 2013,2000,200000,0,33766.12,0,59,137.8,1,1335,265,398,624
...


ab+.sh 使用方式跟 ab 一樣,除了 -c 跟 -n 是由 script 動態來調整外,其它 ab 參數可以從命令列尾端代入,另外測試前要修改壓力參數來給定測試壓力範圍
  • LOW_CONCURRENCY 初始的 concurreny 數
  • HIGH_CONCURRENCY 最大的 concurreny 數
  • RATE_CONCURRENCY 每回測試 concurreny 的累加值
  • NUM_CALL 單個 concurrent user 要發送的 request 次數
執行結果會顯示在畫面上並輸出至當前目錄下的 csv 檔 e.g. localhost_20130403_1152.csv

ab+.sh
#!/bin/bash
# ab+
#
# os and network tuning
# ulimit -n 100000
# sudo echo "2048 64512" > /proc/sys/net/ipv4/ip_local_port_range
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle
# sudo echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse
# sudo echo "10" > /proc/sys/net/ipv4/tcp_fin_timeout
# sudo echo "65535" > /proc/sys/net/core/somaxconn
# sudo echo "65535" > /proc/sys/net/ipv4/tcp_max_syn_backlog  

AB_PATH=/usr/bin/ab
SS_PATH=/bin/ss

LOW_CONCURRENCY=100
HIGH_CONCURRENCY=1000
CONCURRENCY_STEP=100
NUM_CALL=100

HOST=$(echo "$*" | perl -ne '/http.*?:\/\/([^\/?:#]*)/; print $1')
PREFIX=$HOST$(date +"_%Y%m%d_%H%M")
OUTPUT_FILE=$PREFIX.csv
MAX_OPEN_FILES=$(ulimit -n)

trap "exit;" SIGINT SIGTERM

if [ "$MAX_OPEN_FILES" -le $HIGH_CONCURRENCY ]; then
  echo "Warning: open file limit < HIGH_CONCURRENCY" 
  exit 1
fi

echo "Starting benchmark"

mkdir -p temp/"$PREFIX"

echo "#$AB_PATH -c CONCURRENCY=($LOW_CONCURRENCY to $HIGH_CONCURRENCY step $CONCURRENCY_STEP) -n NUM=($NUM_CALL*CONCURRENCY) $*" | tee "$OUTPUT_FILE"
echo "#start time,concurrency,complete requests,failed requests,tps,min,mean,stddev,median,max,90%,95%,99%" | tee -a "$OUTPUT_FILE"
for (( concurrency=LOW_CONCURRENCY; concurrency<=HIGH_CONCURRENCY; concurrency+=CONCURRENCY_STEP ))
do
  tempFile="temp/$PREFIX/c$concurrency-n$((NUM_CALL*concurrency)).log"
  echo "$AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) $*" > "$tempFile"
  
  startTime=$(date)
  $AB_PATH -c $concurrency -n $((NUM_CALL*concurrency)) "[email protected]" >> "$tempFile" 2>&1

  if [ $? -ne 0 ]; then
    echo "Error: please check $tempFile"
    exit 1
  fi

  stats=$(egrep "Complete requests:|Failed requests:|Requests per second:|Total:" "$tempFile" | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')
  histogram=$(egrep "90%|95%|99%" "$tempFile" | cut -c6- | perl -pe 's/[^\d\.]+/ /g; s/^\s+//g;' | perl -pe 's/\s+$//g; s/\s+/,/g')

  echo "$startTime,$concurrency,$stats,$histogram" | tee -a "$OUTPUT_FILE"

  tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  while [ $((tcpConnections+concurrency+CONCURRENCY_STEP)) -ge "$MAX_OPEN_FILES" ]
  do
    sleep 1
    tcpConnections=$($SS_PATH -s | egrep 'TCP:' | perl -ne '/(\d+)/; print $1')
  done
done

rm -rf temp

echo "Finished benchmark, see $OUTPUT_FILE"

Feb 4, 2009

ubuntu configuration memo

Install gcin and liu

sudo apt-get install gcin
tar -zxf liu_gcin120.tar.gz
sudo ./liu_gcin120/install.sh
gedit ~/.gnomerc

#gcin environment

export GTK_IM_MODULE=gcin
export XMODIFIERS="@im=gcin"

gcin &

clip_image002

Launch System > Preferences > Sound > gcin Setup
click Setting for gtab input methods then unselect Auto-send when keycodes are filled
relogin x-window
Disable alert sound
System > Preferences > Sound > Soulds Tab
unselect “play alert sound”
Install stardic
sudo apt-get install stardict

Add firefox search plugin(/usr/lib/firefox-3.0.5/searchplugins)
Install windows applications
sudo apt-get install wine

Download the wine-door package from the link after that follow below commands to install
sudo dpkg -i wine-doors_0.1.2_all.deb

Install pdf-XChange right click on PDFXVwer.exe and select Open With Wine Windows Program Loader (This tool is useful to highlight and comment in pdf file)
Install IE6 just access link and fellow the instructions
Configuration Pidgin
right click on pidgin icon and select Blink on New Message. Again, click plugins and select below plugins.
  • History
  • Offline Mesasge Emulation
  • Message Notification
    Notify For
    ■ IM Windows
    ■ Chat windows
Accouns > select account which you are using > Modify
Enter Local alise and select New mail notifications
clip_image004


Install chm reader
sudo apt-get install gnochm

Issue: the default font size is small
Download the font-patch from the link after that follow below commands to apply.
sudo patch -b /usr/bin/gnochm < gnochm-fontsize.patch

Install Google gadget
Download the package from the link after that follow below commands to install.
sudo dpkg -i google-gadgets_0.9.3-0~getdeb1_i386.deb
Launch Applications > Accessories > Google Gadget (GTK)
Install Notes.
Download IBM Lotus Notes 8.5 Client for Linux (Debian Install) from IBM after that follow below commands to install.

mkdir notes
tar -xf '/home/brandon/Desktop/notes85_notes_linux_deb_beta2_prod.tar' -C notes
sudo dpkg -i notes/ibm_lotus_notes-8.5.i586.deb
/opt/ibm/lotus/notes/notes
Install JDK with Netbean / Eclipse
Download JDK from Sun after that follow below commands to install.
chmod u+x jdk-6u11-linux-i586.bin
./jdk-6u11-linux-i586.bin
gedit /home/brandon/.bashrc
export $JAVA_HOME=/home/brandon/jdk1.6.0_11
export PATH=$PATH:$JAVA_HOME/bin
clip_image006


Download Netbean from link after that follow below commands to install.
sudo '/media/Brandon'\''s HDD/untitled folder/netbeans-6.5-ml-java-linux.sh' --javahome $JAVA_HOME
Download Eclipse from link after that follow below commands to install.
sudo tar -zxvf eclipse-jee-ganymede-SR1-linux-gtk.tar.gz -C /opt/
gedit ~/.gnomerc

#JAVA Environment
export JAVA_HOME=/home/brandon/jdk1.6.0_11
export PATH=$PATH:$JAVA_HOME/bin 


clip_image008

Add new Launcher in deskop
clip_image010
Install Mail-Notification and enable SSL. (package)
(Reference : http://glyphobet.net/blog/essay/286)
sudo dpkg -i mail-notification_5.4.dfsg.1-1build1.deb

Google Notebook Gadget Resizer

The default height of notebook gadget is 200px , it’s too short that need often to scroll down/up , so that write a script to change the height.
image
Install Greasemonkey
https://addons.mozilla.org/en-US/firefox/addon/748
image
User Script:Google Notebook Gadget Resizer
http://userscripts.org/scripts/show/41679
image
Result:
image

Code:

// ==UserScript==
// @name          Google Notebook Gadget Resizer
// @description   To change height of gadgets in iGoogle
// @include       http://www.google.com/ig*
// @include       http://www.google.com.tw/ig*

// @author        [email protected]
// @version       1.0
// ==/UserScript==

// This script are working on Google notebook gadget and some similar gadgets which content type is url and not using dynamic-height feature.
//
// you can fellow below instruction to modify default height of Google notebook gadget or add more gadget to apply new height.
//
// 1.input "about:config" in url box
// 2.search "greasemonkey.scriptvals.userscripts.org/Google Notebook Gadget Resizer.oGadgets"
// 3.The properly is a array of url pattern with height that can modify of you want to.
//
// default:
// ({'http://www.google.com/notebook/ig':"600px", 'url pattern':"400px"})
//
// The url of gadget you could look at page source that place in iframe tag.(ex.http ://87.gmodules.com/ig/ifr?view=home&url=http://itszero.googlepages.com/iGoogleTVSchedule.xml&nocache=0&up_timeRange=0&up_showPrograms=6&up_fontSize=0.75&up_favoriteChannels=&up_defaultChannel=74&lang=en&country=us&.lang=en&.country=us&synd=ig&mid=87&ifpctok=-5100993727886241321&parent=http://www.google.com&extern_js=/extern_js/f/CgJlbhICdXMrMAo4ACwrMBA4ACwrMBI4ACwrMBM4ACwrMBU4ACw/GgLq_VyeDu0.js)
//
// The URL parameter is good to become a URL pattern.
// http://itszero.googlepages.com/iGoogleTVSchedule.xml

var oGadgets = {'http://www.google.com/notebook/ig':'600px','url pattern':'400px'};
var oGadgetsStr = GM_getValue('oGadgets');
if(oGadgetsStr){
 oGadgets = eval(oGadgetsStr);
}else{
GM_setValue('oGadgets',uneval(oGadgets));
}
var oFrames = document.getElementsByTagName('iframe');
for(var i=0 ; i < oFrames.length ; i++){ 
 for(var key in oGadgets){
   var re = new RegExp(key); 
   if(re.test(oFrames[i].src)){
     oFrames[i].style.height = oGadgets[key];
   }
 }
}

Resources:

Greasemonkey Manual:API
http://wiki.greasespot.net/API_reference

Managing Gadget Height
http://code.google.com/apis/gadgets/docs/ui.html#Dyn_Height

Jan 23, 2008

自訂副檔名搜尋

SearchFileByExt.vbs

start = Now
WScript.Echo("Start:" & start)
strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
& "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colFiles = objWMIService.ExecQuery _
("Select * from CIM_DataFile where Extension = 'mp3' OR Extension = 'wma'")
For Each objFile in colFiles
Wscript.echo(objFile.Caption)
'objFile
'objFile.Delete
Next
finish = Now
WScript.Echo("Finish:" & finish)
WScript.Echo("Cost:" & DateDiff("n",start,finish))

vbscript 版 wget

PingList.vbs

<package>
<job>
<runtime>
<named
name="SourceUrl"
helpstring="HTTP檔案位址"
type="string"
required="true"
/>
<named
name="DestinationFile"
helpstring="儲存檔案位址"
type="string"
required="true"
/>
<example>
Example : wget /SourceUrl:"http://xxx/xxx.exe" /DestinationFile:"C:\Documents and Settings\All Users\桌面\xxx.exe"
</example>
<description>
用途:下載Web檔案
版本:1.0
作者:Parkghost</description>
</runtime>
<SCRIPT language="VBScript">
If WScript.Arguments.Named.Exists("SourceUrl") And WScript.Arguments.Named.Exists("DestinationFile") Then
strUrl = WScript.Arguments.Named.Item("SourceUrl")
strFile = WScript.Arguments.Named.Item("DestinationFile")
Const adTypeBinary = 1
Const adSaveCreateOverWrite = 2
Const ForWriting = 2
Dim web, varByteArray, strData, strBuffer, lngCounter, ado
Err.Clear
Set web = Nothing
Set web = CreateObject("WinHttp.WinHttpRequest.5.1")
If web Is Nothing Then Set web = CreateObject("WinHttp.WinHttpRequest")
If web Is Nothing Then Set web = CreateObject("MSXML2.ServerXMLHTTP")
If web Is Nothing Then Set web = CreateObject("Microsoft.XMLHTTP")
web.Open "GET", strURL, False
web.Send
If Err.Number <> 0 Then
SaveWebBinary = False
Set web = Nothing
WScript.Quit
End If
If web.Status <> "200" Then
SaveWebBinary = False
Set web = Nothing
WScript.Quit
End If
varByteArray = web.ResponseBody
Set web = Nothing

'Save the file
'On Error Resume Next
Set ado = Nothing
Set ado = CreateObject("ADODB.Stream")
If ado Is Nothing Then
Set fs = CreateObject("Scripting.FileSystemObject")
Set ts = fs.OpenTextFile(strFile, ForWriting, True)
strData = ""
strBuffer = ""
For lngCounter = 0 to UBound(varByteArray)
ts.Write Chr(255 And Ascb(Midb(varByteArray,lngCounter + 1, 1)))
Next
ts.Close
Else
ado.Type = adTypeBinary
ado.Open
ado.Write varByteArray
ado.SaveToFile strFile, adSaveCreateOverWrite
ado.Close
End If
SaveWebBinary = True
Else
WScript.Arguments.ShowUsage()
End if
</SCRIPT>
</job>
</package>

解決vbscript len function 無法判斷 double byte 字元


strText = ("中文123")
WScript.Echo(strText & ":" & Len(strText))
WScript.Echo(strText & ":" & newLen(strText))

Function newLen(text)
oldlen = Len(text)
newLen = 0
For i = 1 To oldlen
If Asc(Mid(text,i,1)) > 0 Then
newLen = newLen + 1
Else
newLen = newLen + 2
End If
Next
End Function

網路連線測式(Ping by List)

PingList.vbs
Msg = "站點" & vbTab & paddingStr("位址",27) & vbTab & paddingStr("IP",15) & vbTab & "回應時間" & vbNewLine
Set objWMIService = GetObject("winmgmts:\\.\root\cimv2")
Set objXML = CreateObject("Microsoft.XMLDOM")
objXML.async = False
objXML.Load "ipList.xml"
If objXML.parseError.errorCode <> 0 Then
 WScript.Echo("設定檔載入錯誤")
End If

Set objList = objXML.getElementsByTagName("*")
Set nodes = objList.item(i).selectSingleNode("//site").childNodes

For Each node In nodes
 Dim ProtocolAddress
 Dim ResponseTime
 Set objNamedNodeMap = node.attributes  
 Set objPing = objWMIService.ExecQuery("select * from Win32_PingStatus where address = '" & node.text & "'")
 For Each objStatus in objPing
  ProtocolAddress = objStatus.ProtocolAddress
  ResponseTime = objStatus.ResponseTime
 Next 
 Msg = Msg & objNamedNodeMap.item(0).text & vbTab & paddingStr(node.text,25) & vbTab & paddingStr(ProtocolAddress,15) & vbTab & ResponseTime & vbNewLine 
Next
WScript.Echo(Msg)

Function paddingStr(str,lenght)
 If Len(str) > lenght then
  paddingStr = str
 Else
  'todo:double byte analysis
  paddingStr = str &  Space(lenght - Len(str))
 End If
End Function


ipList.xml
<?xml version="1.0" encoding="big5"?>
<site>
<item name="HiNet">www.hinet.net</item>
<item name="Yahoo">tw.yahoo.com</item>
<item name="Pchome">www.pchome.com</item>
</site>