Sqlite error -file is encrypted or is not a database while analyzing firefox browser history

While writing my own script for Firefox browsing history, I came across a peculiar error:

SQLite version 3.6.22
Enter “.help” for instructions
Enter SQL statements terminated with a “;”
sqlite> SELECT moz_places.url,datetime(moz_historyvisits.visit_date/1000000,’unixepoch’,’localtime’) from moz_historyvisits, moz_places WHERE  moz_historyvisits.place_id=moz_places.id and datetime(moz_historyvisits.visit_date/1000000,’unixepoch’,’localtime’)>datetime(‘now’,’-1 day’,’localtime’) order by datetime(moz_historyvisits.visit_date/1000000,’unixepoch’,’localtime’) desc limit 100;
Error: file is encrypted or is not a database

It turns out that the remedy is simple after going through Stackoverflow  thread that you have to upgrade your existing sqlite3 version!

So, if you encounter this error, please check your platform and download appropriate binary on this page: http://www.sqlite.org/download.html

 

Find file owner and file permissions using python

Here is a handy python function you can use if you wish to find owner of file.

def find_owner(filename):
import os
import pwd
if filename:
st = os.stat(filename)
uid = st.st_uid
print(uid)
userinfo = pwd.getpwuid(st.st_uid)
print(userinfo)
# for file: /etc/passwd –
# output: pwd.struct_passwd(pw_name=’root’, pw_passwd=’x’, pw_uid=0,
#pw_gid=0, pw_gecos=’root’, pw_dir=’/root’, pw_shell=’/bin/bash’)
return pwd.getpwuid(st.st_uid).pw_name
else:
return None

Also, if you are interested in file permissions, here is one liner you can use:

print oct(stat.S_IMODE(os.stat(‘/etc/passwd’).st_mode))

Thanks for the great tip from – https://stomp.colorado.edu/blog/blog/2010/10/22/on-python-stat-octal-and-file-system-permissions/

Python/Django tips

Datetime to unix timestamp
import datetime
import time
dt = datetime.datetime(2010, 2, 25, 23, 23)
time.mktime(dt.timetuple())

Sort a list of dictionaries by values
newlist = sorted(list_to_be_sorted, key=lambda k: k[‘name’])

or

from operator import itemgetter
newlist = sorted(list_to_be_sorted, key=itemgetter(‘name’))

Django database queries
from django.db import connection
print connection.queries

or

Querysets also have a query attribute containing the query to be executed.

print MyModel.objects.filter(name=”my name”).query
# to see the query which will be executed for a given statement
print str(MyModel.objects.filter(name=”my name”).query)

django aggregation to lower resolution using grouping by a date range
from django.db.models import Avg

Viewers.objects.filter(date__range=(start_time, end_time)).aggregate(average=Avg(‘value’))

That will get you the average of all the values between start_time and end_time, returned as a dictionary in the form of { ‘average’: <the average> }.

start_time and end_time need to be Python datetime objects. So if you have a timestamp, or something, you’ll need to convert it first. You can also use datetime.timedelta to calculate the end_time based on the start_time. For a five minute resolution, something like this:

from datetime import timedelta

end_time = start_time + timedelta(minutes=5)

Or you can use range filter in Django – https://docs.djangoproject.com/en/dev/ref/models/querysets/#range

Ref –
http://stackoverflow.com/questions/6266397/django-aggregation-to-lower-resolution-using-grouping-by-a-date-range?lq=1
http://stackoverflow.com/questions/9950573/django-queryset-aggregate-by-time-interval

Show all python packages
pip freeze | grep Django

Retrieving python module path
import a_module
print a_module.__file__

Install a local python package using pip
pip install -e /path/to/package

(/path/to/package – where setup.py is located)

Run a sample Django project – on Ubuntu
sudo apt-get install python-virtualenv git 
cd /opt
virtualenv myenv
source myenv/bin/activate
git clone https://github.com/shimon/djangotutorial.git
cd djangotutorial
pip install django
python manage.py syncdb
python manage.py runserver

Running multiple bash commands using subprocess in python
import subprocess
process = subprocess.Popen(command,stdout=subprocess.PIPE, shell=True)
proc_stdout = process.communicate()[0].strip()
print proc_stdout

-> subprocess_cmd(‘echo a; echo b’)
this will return
a
b

 

 

Some Linux tips

Some of the commands I usually use in Linux – for my reference:

Check partition type
$df -T -h
or $sudo fdisk -l

Add user
$useradd -m -p -s

du and df usage
Show summary for each *
$du -hsck

Show in human readable format with total
$df -ah –total

Clean swap space
$sudo swapoff -a
$sudo swapon -a

Network statistics
$ip -s link

Changing default route
$sudo ip route add default via 192.168.1.1

Routing table entry
$ip route show

Ref- http://linoxide.com/linux-command/use-ip-command-linux/

Change file or directory permissions
For file:
#find /var/lib -type f -exec chmod 644 {} \;
For directory:
#find /var/lib -type d -exec chmod 755 {} \;

 

 

 

 

Setting up Graphite on CentOS 6.5 or Scientific Linux 6.5

Graphite is an awesome tool for collecting,viewing various metrics on Linux/Unix system.

Graphie consists of three components:

1) Carbon- This daemon listens on TCP/2003 for clients to deliver metrics to it. Metrics received are stored in Whisper database.
2) Diamond – Client side daemon can be used to collect system statistics and deliver them to Carbon. It is possible to write your own script for collection of data and deliver it to Carbon over TCP/2003.
3) Graphite – This is a web front-end for viewing the metric information and is based on Django framework.

This post details the step required for installation of graphite on CentOS equivalient machines.

Add and enable EPEL repository:
# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm

On the machine, install dependencies:
# yum install -y gcc zlib-devel curl curl-devel openssl rpm-build gcc-c++ rpm-build python python-ldap python-memcached python-sqlite2 pycairo pycairo-devel cairo cairo-devel python-twisted Django django-tagging bitmap bitmap-fonts python-devel glibc-devel gcc-c++ openssl-devel python-zope-interface httpd memcached mod_wsgi

Install carbon, whisper, graphite and other packages using pip
# pip install whisper
# pip install carbon
# pip install graphite-web

If you want to download tar.gz files, they are available here:
https://github.com/graphite-project/graphite-web/archive/0.9.12.tar.gz
https://github.com/graphite-project/carbon/archive/0.9.12.tar.gz
https://github.com/graphite-project/whisper/archive/0.9.12.tar.gz

# pip install -Iv ‘https://www.djangoproject.com/download/1.5.5/tarball/&#8217;
# pip install daemonize
# pip install ‘Twisted<12.0’
# pip install tagging
# pip install django-tagging
# pip install pyparsing
# pip install simplejson
# pip install python-memcached
# pip install python-ldap
# pip install txAMQP

Now, check if any dependencies are missing or not using: check-dependencies.py provided as a part of graphite-web tar.gz package.

Setup the carbon and graphite-web configuration files:

# cd /opt/graphite/conf/
# cp graphite.wsgi.example graphite.wsgi
# cp storage-schemas.conf.example storage-schemas.conf
# cp carbon.conf.example carbon.conf
# cd ../webapp/graphite
# cp local_settings.py.example local_settings.py
# cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite-vhost.conf

Update local_settings.py above with correct Timezone, memcache location, ldap database for authentication. For a basic setup this is the only config file you need to modify. Change also “SECRET KEY” to your liking – with random number!

# cd /opt/graphite/
# python ./webapp/graphite/manage.py syncdb

Change WSGISocketPrefix in /etc/httpd/conf.d/graphite.conf to
WSGISocketPrefix /var/run/httpd/graphite

find DJANGO_ROOT using
# find / -name django -type d -print
# DJANGO_ROOT=/usr/lib/python2.6/site-packages/django
# echo #DJANGO_ROOT

Change permission to apache:apache for storage folder so that apache can read!
# chown -R apache:apache /opt/graphite/storage/

Start Carbon and Graphite
# service memcached start
# service carbon-cache start
Or
# opt/graphite/bin/carbon-cache.py start
# service httpd start

Check for any Apache errors using
# tail -f /var/log/httpd/error_log
# tail -f /opt/graphite/storage/log/webapp/error.log

The following links are quite useful during installation of graphite:
http://www.ericshalov.com/2014/01/03/how-do-i-install-graphite-on-centos/
http://chasemp.github.io/2013/09/12/graphite-on-centos6-lessons-learned/
http://graphite.readthedocs.org/en/latest/install.html
https://gist.github.com/cesar-carrasco/5813859
http://www.rampmeupscotty.com/blog/2012/08/07/installing-graphite-on-centos-6-dot-2
http://www.alrix.com/2012/04/installing-graphite-on-centos-part-2.html
https://wiki.icinga.org/display/howtos/graphite

Compiling Snort – daq_static library not found error

I downloaded latest DAQ and Snort-2.9.x from http://www.snort.org/snort-downloads and I was having issues during compilation of Snort with on Cent OS /Scientific Linux.

 

Specifically, the error was:

 

checking for daq_load_modules in -ldaq_static… no

 

   ERROR!  daq_static library not found, go get it from

   http://www.snort.org/.

 

Prior to snort installation, I have compiled DAQ library and I was expecting that I can compile snort in the next few minutes. But, daq_static library gave me lot of headaches. Finally, the solution to this problem was found.

To get rid, check daq-modules-config is in your path.

 

# which daq-modules-config

which: no daq-modules-config in (/sbin:/bin:/usr/sbin:/usr/bin)

 

Then, include daq-modules-config in the path.

# export PATH=$PATH:/usr/local/bin

 

and try ./configure again. Hola!!

 

If that fails please see your config.log for more clues!!

Disable IPv6 on CentOS/Scientific Linux 6.x

If you wish to disable IPv6 on the machine, please do  the following:

Check if IPv6 is enabled or not.

[root@psj ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

Disable IPv6

Create a file /etc/modprobe.d/disable-ipv6.conf and add line “install ipv6 /bin/true

[root@psj ~]# vi /etc/modprobe.d/disable-ipv6.conf
install ipv6 /bin/true

[root@psj ~]# vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no

[root@psj ~]# echo “net.ipv6.conf.all.disable_ipv6 = 1″ >> /etc/sysctl.conf

[root@psj ~]# reboot

Check after reboot, IPv6 is disabled or not by using

[root@psj ~]# ip addr

That’s all.

Setting up Network flow monitoring using Nfsen on CentOS

Nfsen(http://nfsen.sourceforge.net/) is amazing project that supports various netflow collectors and it allows you to investigate the netflows to your imagination!

The installation procedure for nfsen is described below:

Dependent package installation using yum
——————————————————
The following rpms and perl modules are required for installation and I used yum to install them:
httpd
php
perl
gcc
make
rrdtool
rrdtool-devel
rrdtool-perl
perl-TimeDate
perl-MailTools
perl-Socket6
bison
flex

Make sure that SELinux is disabled.
vi /etc/selinux/config
set SELINUX=disabled
reboot

Start and enable httpd at boot
# service httpd start
# chkconfig httpd on

Install fprobe
——————
Download fprobe from SourceForge Site and install
# tar -xjvf fprobe-1.11.tar.bz2
# cd fprobe-1.11
# ./configure
# make
# make install

Now run fprobe using command – /usr/local/sbin/fprobe -i eth0 -f”ip” -n7 remote:port
e.g.
# /usr/local/sbin/fprobe -i eth0 -f”ip” -n7 127.0.0.1:9995

and make sure that the process is running using ‘ps aux’ command.

Nfdump installation
———————-
Download nfdump from SourceForge Site and install
# tar -zxvf nfdump-1.6.11.tar.gz
# cd nfdump-1.6.11
nfdump-1.6.11]# ./configure –enable-nfprofile –enable-nftrack –enable-nfpcapd
nfdump-1.6.11]# make
nfdump-1.6.11]# make install

### You can check whether nfcapd can collect network data or not using
# nfcapd -w -D -S 2 -B 1024000 -l /opt/nfcapd_test/ -p 9995

Nfsen installation
————————

Download nfsen from SourceForge Site and install
# tar -zxvf nfsen-1.3.6p1
# cd nfsen-1.3.6p1
# cp etc/nfsen-dist.conf etc/nfsen.conf

Make all the necessay changes in nfsen.conf. Just go through the README file of nfsen for installation and configuration instructions. These are really helpful.

Typical options in ‘nfsen.conf’ configuration file look like:

# Example Config file!
$BASEDIR = “/opt/nfsen”;
$BINDIR=”${BASEDIR}/bin”;
$LIBEXECDIR=”${BASEDIR}/libexec”;
$CONFDIR=”${BASEDIR}/etc”;
$HTMLDIR = “/var/www/nfsen/”;
$DOCDIR=”${HTMLDIR}/doc”;
$VARDIR=”${BASEDIR}/var”;
$PROFILESTATDIR=”${BASEDIR}/profiles-stat”;
$PROFILEDATADIR=”${BASEDIR}/profiles-data”;
$BACKEND_PLUGINDIR=”${BASEDIR}/plugins”;
$FRONTEND_PLUGINDIR=”${HTMLDIR}/plugins”;
$PREFIX = ‘/usr/local/bin’; $USER = “apache”;
$WWWUSER = “apache”;
$WWWGROUP = “apache”;
$EXTENSIONS = ‘all’;
$SUBDIRLAYOUT = 1;
$ZIPcollected = 1;
$ZIPprofiles = 1;
$PROFILERS = 2;
$DISKLIMIT = 98;
$PROFILERS = 6;
%sources = (
‘home’ => { ‘port’ => ‘9995’, ‘col’ => ‘#0000ff’, ‘type’ => ‘netflow’ },
);

When you install nfsen, it will automatically configure nfcapd program as a daemon.
So, there is no need to run nfcapd seperately as a daemon.

But, yes, if you would like to test it with tools like fprobe/softflowd, you can make use of nfcapd
program as a standalone and capture the flows from softflowd/fprobe.

nfcapd -w -D -S 2 -B 1024000 -l /opt/flow_base_dir/ -p 9995

Make sure that /opt/data/nfsen directory is apache writable.i.e.
# chown -R apache:apache /opt/data/nfsen

Install nfsen using
# perl install.pm etc/nfsen.conf

# Make nfsen to start at boot. Please refer to reference links for writing a daemon script.
# chmod 755 nfsen && chkconfig –add nfsen && chkconfig nfsen on

Configure apche for nfsen.
———————————-
# vi /etc/httpd/conf.d/nfsen.conf

Alias /nfsen /var/www/nfsen
<Directory /var/www/nfsen/>
DirectoryIndex nfsen.php
Options -Indexes
order allow,deny
allow from all
</Directory>

# service httpd restart

Typical Errors
——————
If you encounter errors like:

Rebuilding profile stats for ‘./live’
Unable to create graph: No such file or directory
Error GenGraph: Profile: live, traffic-day: Legend set but no color: peer2 at libexec/NfSenRRD.pm line 337.
Unable to create graph: No such file or directory

Please ignore them. For some reason, pre-built graphs could not be generated. Make sure that RRD.pm is installed correctly.
After some time, the graphs will be rebuilt.

Another error you might encounter is – “ERROR: nfsend connect() error: Permission denied!” This is a permissions issue, as documented in – https://code.google.com/p/nfsenplugins/wiki/NFSEN_Installation_Gotchas. You need to make sure that the nfsen package can read the nfsen.comm socket file.

Typical checks
———————
Check whether fprobe is working correctly or not by using tcpdump. Fprobe is collecting traffic on interface eth0 and capturing network flows and sending it to port 9995 over udp.

tcpdump -n -i eth0 -v dst port 9995

Check if fprobe,nfcapd processes are running or not
# ps aux | grep fprobe
# ps axu | grep nfcapd

Create a soft link for nfsen
# ln -s /opt/nfsen/bin/nfsen /usr/sbin/nfsen

## You can start and stop nfsen using
# nfsen start
# nfsen stop
### Check status of nfsen daemon
# nfsen status

### If you change the configuration in nfsen.conf, do not forget to reconfigure nfsen by using command:

# nfsen reconfig

#### check if the required ports are listening to the traffic using netstat command:
# netstat -t -u -c

Reference Links:

I found the following blog entries to be useful for nfsen configuration:

• http://www.forwardingplane.net/2014/01/install-nfsen-and-nfdump-on-centos-6-5-for-netflow-and-or-sflow-collection/
• https://www.m00nie.com/2014/02/install-nfsen-centos-6-fedora-20/
• http://manpages.ubuntu.com/manpages/hardy/man8/fprobe.8.html
• http://www.itsprite.com/linuxhow-to-install-nfdump-and-nfsen/
• http://blogfranz.blogspot.in/2010/11/netflow-on-endpoint.html

 

Snort, DAQ and PF_RING installation on CentOS

Though Snort is single threaded, PF_RING has software load-balancing capabilities which will allow you to run it as if it were multi-threaded. The following steps describe how to setup Snort, DAQ and PF_RING on CentOS. 

The following packages are required to be installed with yum:

• kernel-devel
• libtool
• subversion
• automake
• make
• autoconf
• pcre-devel
• libpcap-devel
• libpcap
• flex
• bison
• byacc
• gcc
• gcc-c++
• zlib-devel
• numactl
• numactl-devel

Also, make sure that you kernel header libraries and kernel version is same otherwise build errors will be reported.

Download and install libdnet library from http://code.google.com/p/libdnet/downloads/detail?name=libdnet-1.12.tgz&amp;

build PF_RING inline libraries and kernel module.

Setting up PF_RING
——————
Download the latest version of PF_RING module from NTOP site – http://sourceforge.net/projects/ntop/files/PF_RING/ and extract it
$ tar zxvf PF_RING-5.6.2.tar.gz
$ cd PF_RING-5.6.2
$ make clean
$ cd kernel
$ make clean
$ make
$ sudo make install
$ cd ../userland/lib
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
$ export LIBS=’-L/usr/local/lib’
$./configure
$ make clean
$ make
$ sudo make install
$ cd ../libpcap
# $ LDFLAGS=’-wrt’
$ export LIBS=’-L/usr/local/lib -lpfring -lpthread’
$ ./configure
$ make clean
$ make
$ sudo make install

### load PF_RING kernel module
$ cd ../../kernel
$ sudo insmod pf_ring.ko enable tx_capture=0 transparent_mode=0 min_num_slots=16384

### To remove kernel module
$ sudo rmmod pf_ring.ko

### To check the status of PF_RING:
# modinfo pf_ring
# cat /proc/net/pf_ring/info
# lsmod | grep pf_ring

Setting up DAQ
————–
Download the latest version of DAQ module from snort site – http://www.snort.org/downloads and extract it.
$ tar zxvf daq-2.0.2.tar.gz
$ cd daq-2.0.2
$ ./configure -h
$ export LD_LIBRARY_PATH=/usr/local/lib
$ export LIBS=”
$ sudo ldconfig -v
$ ./configure –disable-nfq-module –disable-ipq-module –with-libpcap-includes=/usr/local/include –with-libpcap-lib
raries=/usr/local/lib –with-libpfring-includes=/usr/local/include –with-libpfring-libraries=/usr/local/lib
$ make
$ sudo make install

You will see messages like this after installation:

Libraries have been installed in:
/usr/local/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR’
flag during linking and do at least one of the following:
– add LIBDIR to the `LD_LIBRARY_PATH’ environment variable
during execution
– add LIBDIR to the `LD_RUN_PATH’ environment variable
during linking
– use the `-Wl,-rpath -Wl,LIBDIR’ linker flag

 

Build PF_RING DAQ Module
————————–
Downlod the latest version of PF_RING (PF_RING-5.6.2.tar.gz) from NTOP site and extract it
$ tar zxvf PF_RING-5.6.2.tar.gz

Goto userland/snort/pfring-daq-module directory. e.g. /home/joshi/PF_RING-5.6.2/userland/snort/pfring-daq-module

$ cd /home/joshi/PF_RING-5.6.2/userland/snort/pfring-daq-module
$ autoreconf -ivf
$ export LD_LIBRARY_PATH=/usr/local/lib
$ export LIBS=’-lrt -L/usr/local/lib’ # ( Do not use export LIBS=’-lrt -L/usr/local/lib -lpcap -lpfring -lpthread’)
$ make clean
$.configure
$ make
$ sudo make install

Download and Build Snort 2.9.x from http://www.snort.org/downloads/
——————-
$ tar -zxvf snort-2.9.6.0.tar.gz
$ cd snort-2.9.6.0
$ export LD_LIBRARY_PATH=/usr/local/lib
$ make clean
$ ./configure –with-libpcap-includes=/usr/local/includes –with-libpcap-libraries=/usr/local/lib –with-libpfring-includes=/usr/local/include/daq –with-libpfring-libraries=/usr/local/lib/daq –enable-sourcefire –enable-zlib –enable-perfprofiling
$ make
$ sudo make install

Run snort
$ sudo /usr/local/bin/snort –daq-dir=/usr/local/lib/daq –daq pfring -v -i eth0

It is possible to run as many instances of snort and take advantage of PF_RING features.

Typical errors that you will encounter are:

1) C Compiler can not create executable
This typically happens when LD_LIBRARY_PATH and LIBS environment variables are not ok.

2) Error while loading shared libraries:libdnet.1
Solution to this issue is as follows:
# cp /usr/local/lib/libdnet.1.0.1 /usr/local/lib/libdnet.so.1.0.1
# /sbin/ldconfig
# updatedb

I found the following links to be useful during the process of installation:
http://www.metaflows.com/solutions2/pf-ring/
https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Installation_with_PF_RING
http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/
http://dukegenius.blogspot.in/2012/03/load-balanced-snort.html
http://www.ntop.org/pf_ring/installation-guide-for-pf_ring/
http://blog.securitymonks.com/2010/08/26/three-little-idsips-engines-build-their-open-source-solutions/
http://blog.gunjanbansal.in/2010/06/installation-guide-for-pfring.html
http://www.libcrack.so/2012/09/21/pf_ring-intel-igb-snort-daq-on-debian/
http://ossectools.blogspot.in/2011/07/running-load-balanced-snort-in-pfring.html
http://www.netsec.pro/dokuwiki/doku.php?id=multi-snortnotes
http://www.marshut.com/tphus/pf-ring-and-dna-with-snort.html
http://www.engardelinux.org/modules/index/list_archives.cgi?list=snort-users&page=0020.html&month=2012-09
http://www.aldeid.com/wiki/Suricata-vs-snort

 

 

 

Show running processes in linux using python

If you want to find the currently running processes,  ‘ps aux’ is the de-facto command used in linux environment. In addition, there are ready to use programs like htop and atop and they do a great job. But, I wanted to write a python-based script which can be re-used later on and here it goes:

import os

import hashlib

def md5Checksum(filePath):

with open(filePath, ‘rb’) as f:

m = hashlib.md5()

while True:

data = f.read(8192)

if not data:

break

m.update(data)

return m.hexdigest()

if __name__==’__main__’:

process_list = []

pids= [pid for pid in os.listdir(‘/proc’) if pid.isdigit()]

for pid in pids:

if os.path.isfile(‘/proc/’ + str(pid) +’/exe’):

process_path = os.path.realpath(‘/proc/’ + str(pid) + ‘/exe’)

filesize=os.path.getsize(‘/proc/’ + str(pid) + ‘/exe’)

md5_result = md5Checksum(process_path)

process_list.append((pid,process_path,filesize,md5_result))

for item in process_list:

print item