Posts

Showing posts with the label Python

[ Level 1 ] Limit memory usage in Python program

We could limit memory usage in Python program. #!/usr/bin/env python3.5 import resource def show_memory_limit(): soft, hard = resource.getrlimit(resource.RLIMIT_DATA) print 'Soft limit changed to :', soft print 'Hard limit changed to :', hard def set_memory_limit(soft, hard): resource.setrlimit(resource.RLIMIT_DATA, (soft, hard)) #limit to one kilobyte if __name__ == '__main__': show_memory_limit() set_memory_limit(1024, 2048) show_memory_limit() Wish this helps. regards, Stanley Huang

[Level 2] Python reverse engineering.

If you want to reverse engineering of python source code, you could use pylint to get class diagram. ref: http://manpages.ubuntu.com/manpages/saucy/man1/pyreverse.1.html http://www.logilab.org/6883 http://planet.logilab.fr/index.php?post_id=74 $ apt-cache search pyreverse pylint - python code static checker and UML diagram generator ... $ sudo apt-get -y install pylint $ pyreverse ./*.py $ dotty ./classes_No_Name.dot $ doggy ./packages_No_Name.dot or $ pyreverse -o png ./*.py $ eog ./classes_No_Name.png $ eog ./packages_No_Name.png related projects: http://floss.zoomquiet.io/data/20041231094746/index.html Wish this helps. regards, Stanley Huang

[Level 1] How to trace the source code more efficient.

If you want to trace python call stack, you could get call sequence by using call_seq module. You could download call_seq from pypi website. https://pypi.python.org/pypi/call_seq/0.0.1 After you download it, you could prepare the python script like following. python from call_seq import CallSeq trail = CallSeq() trail.set_trace() # the code you want to trace. trail.unset_trace() trail.dump_to_file('output.json') After you create the output json file, then you could use browser module of call_seq to get visualization of call sequence. $ python -m call_seq.browser ./output.json Wish this helps. regards, Stanley Huang

[ Level 2 ] Test Singleton Implementation in Python.

Days ago, just search how to use implement "Singleton" design pattern. ( http://stackoverflow.com/questions/6760685/crea(ing-a-singleton-in-python ) And today, I just need to implement for it. In my case, I need to create mutliple loggers and I also want to use "Singleton" to reduce system resource usage. Therefore, I create a Singleton meta class and also use a parameter called "singleton_id" to define the instance category. Source Code: #!/bin/env python class Singleton(object): _singleton_key = 'singleton_id' _instance = {} def __new__(class_, *args, **kwargs): if class_._singleton_key not in kwargs.keys(): kwargs[class_._singleton_key] = '' if class_._singleton_key in kwargs and kwargs[class_._singleton_key] not in class_._instance.keys(): class_._instance[kwargs[class_._singleton_key]] = object.__new__(class_) return class_._instance[kwargs[class_._singleton_key]] class my...

[Level 2] pstree implementation with Python.

Because I couldn't find pstree command in our company's product. Therefore, I just search from internet and found someone implement pstree layout with Python. (http://stackoverflow.com/questions/16395207/create-a-process-tree-like-pstree-command-with-python-in-linux) And I just modify it and create a pstree.py system utility. #!/bin/env python ''' ## Implement pstree in Python ## data structure of tree, cmd_list tree = { 0: [1], 1: [2, 3], 2: [5, 6, 7, 8], ... } cmd_list = { 0: '/sbin/init', 1: '[kthreadd]', ... } ''' import os import sys import re tree = {} cmd_list = {} def printTree(parent, tree, cmd_list, indent=''): print '%s:%s' % (parent, cmd_list[parent]) if parent not in tree: return for child in tree[parent][:-1]: sys.stdout.write(indent + '|-') printTree(child, tree, cmd_list, indent + '| ') child = tree[parent][-1] sys.stdout.write(indent + '`-') ...

[Level 1] Create secure web for iPython notebook.

The default protocol for iPython notebook is http and you didn't passphrase to enter notebook. If you want your notebook be secure, you could follow the steps to enable SSL and passphrase for it. 1. create profile: In [1]: ## create profile for secure web !ipython profile create secureweb 2. create passphrase: In [2]: ## create passphrase from IPython.lib import passwd passwd(passphrase='passphrase') Out[2]: 'sha1:24be7c5ab59a:b8b7d3c691b2db67a5ef855b625cb560e125e5e1' 3. Create SSL certificate $ cd /home/stanley/iPython_notebook/certs $ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem Generating a 1024 bit RSA private key ..............++++++ ..........++++++ writing new private key to 'mycert.pem' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There ar...

[Level 1] Install slideshow support in iPython notebook.

Just found an iPython notebook extension support slideshow and you could install by the following steps. Precondition: Because this introduction would try to clone a github project, you have install git utility first. $ sudo apt-get -y install git 1. Use the following commands to install slideshow support. ## get porfile directory profile_dir = get_ipython().profile_dir.location ## clone extension from github import os tgt = os.path.join( profile_dir, 'static', 'custom') !git clone https://github.com/ipython-contrib/IPython-notebook-extensions.git $tgt %cd $tgt ## create a javascript for supporting slideshow %%writefile custom.js // we want strict javascript that fails // on ambiguous syntax "using strict"; // do not use notebook loaded event as it is re-triggerd on // revert to checkpoint but this allow extesnsion to be loaded // late enough to work. // $([IPython.events]).on('app_initialized.NotebookApp', function(){ /** Use path to...

[Level 1] How to auto restore when start iPython

iPython have a magic command call "alias". This command just command "alias" in unix shell could help you to create alias command in iPython environment. How could we save the alias that we created before, you could use "store" magic command. The alias would be saved in iPython internal db. ex. In [1]: %alias ipython_alias echo 'hello world' In [2]: %store ipython_alias After you exit iPython and restart it again, you need to restore the aliases from internal db. In [1]: %store -r And how could we make iPython auto-restore when we launch it? You could modify ipython_config.py in profile. ex. If you use default profile $> cat /home/stanley/.config/ipython/profile_default/ipython_config.py ... # StoreMagics configuration c.StoreMagics.autorestore = True # uncomment this line and assign autorestore as 'True' ... You could also create a script to add alias automatically. ## Add often use commands. security_commands = 'chmod ch...

[ Level 1 ] Assign external python path to program.

Sometimes, you want to assign external python path to program, then you could use "PYTHONPATH" environment variables for this purpose. ex. $ PYTHONPATH=/tmp/my_python_path python - <<EOF > import sys > print sys.path > EOF ['', '/tmp/my_python_path', '/usr/lib/python2.7', ...] $ Wish this helps. regards, Stanley Huang

[ Level 2 ] Create an egg for Python in Ubuntu.

How to create an egg file for Python. First of all, you must have setuptools module. $ sudo apt-get -y install python-setuptools Now, you could try to create an empty egg now. $ mkdir /tmp/demo $ cd /tmp/demo $ cat &ht; ./setup.py <<EOF #!/bin/env python #-*- coding:utf-8 -*- from setuptools import setup setup() EOF $ python setup.py bdist_egg ## bdist_egg is the option for creating egg. $ ls -ALb build dist setup.py UNKNOWN.egg-info You could find, we have three more directories after you execute setup.py build -> dist -> final egg file UNKNOW.egg-info -> egg info Now, we could give setuptools more information about egg. cat > ./setup.py <<EOF #!/bin/env python #-*- coding:utf-8 -*- from setuptools import setup, find_packages setup( name = "my_first_egg", version="0.0.1", packages = find_packages(), zip_safe = False, description = "my first egg.", long_descriptio...

[ Level 3 ] Test your Python code.

Ref: https://python-guide.readthedocs.org/en/latest/writing/tests.html Mock: http://www.voidspace.org.uk/python/mock/getting-started.html# Wish this helps. regards, Stanley Huang

[ Level 2 ] Tips of unittest for Python.

There are tips for Python: 1. Use mock for replace method. ex. myMock=mox.Mox() myMock.StubOutWithMock(myModule, 'replaceMethod') myModule.replaceMethod(input).AndReturn('Hello World!') myMock.ReplayAll() expected = 'Hello World!' self.assertEqual(module.replaceMethod(input), expected) myMock.VerifyAll() #verify if all mocks be executed. myMock.UnsetStubs() #release all mocks 2. Use -m to test one method only. ex. $ python -m unittest myApp.TestClass.testMethod Wish this helps. regards, Stanley Huang

[ Level 2 ] Coverage in Python.

$ sudo pip install coverage $ coverage run myApp.py arg1 arg2... $ coverage report -m $ coverage html $ coverage help (run) ## or coverage run --help Commands of coverage: run – Run a Python program and collect execution data. report – Report coverage results. html – Produce annotated HTML listings with coverage results. xml – Produce an XML report with coverage results. annotate – Annotate source files with coverage results. erase – Erase previously collected coverage data. combine – Combine together a number of data files. debug – Get diagnostic information. Ref: http://nedbatchelder.com/code/coverage/ http://nedbatchelder.com/code/coverage/cmd.html#cmd Wish this helps. regards, Stanley Huang

[ Level 2 ] Python with vim

Python with vim: http://wiki.python.org/moin/Vim Wish this helps. regards, Stanley Huang

[ Level 1 ] How ansi color in terminal by Python script.

Sample for show ansi color in terminal by Python script. #!/bin/env python import os print os.popen('echo "\033[31m\033[43mHello World!\033[0m"').read() print '\033[31m\033[43m' + 'Hello World!' + '\033[0m' Wish this helps. regards, Stanley Huang

[ Level 3 ] Python MRO.

Python MRO introduction. http://docs.python-guide.org/en/latest/writing/style/ Wish this helps. regards, Stanley Huang

[ Level 3 ] Python coding style.

Python coding style: http://docs.python-guide.org/en/latest/writing/style/ Wish this helps. regards, Stanley Huang

[Level 2] Implement with syntax in Python.

#!/bin/env python class myOpen(object): def __init__(self, filename, readwrite): self.__filename = filename self.__readwrite = readwrite self.__fd = None def __enter__(self): self.__fd = open(self.__filename, self.__readwrite) return self.__fd def __exit__(self, *args): self.__fd.close() with myOpen('/tmp/my.txt', 'r') as f: print f.readlines() with myOpen('/tmp/hello.txt', 'w') as f: f.write('hello world!\n') $ ./test.py ['-rwxr--r-- 1 stanley stanley 846 2012-12-26 13:07 ./c.py\n', '-rwxr--r-- 1 stanley stanley 486 2013-01-08 17:54 ./test.py\n'] $ cat ./my.txt -rwxr--r-- 1 stanley stanley 846 2012-12-26 13:07 ./c.py -rwxr--r-- 1 stanley stanley 486 2013-01-08 17:54 ./test.py $ cat ./hello.txt hello world! Wish this helps. regards, Stanley Huang

[Level 2] Python abstract method testing

#!/bin/env python import os, sys from abc import * # ================================================= class ToolCmd(object): __metaclass__ = ABCMeta def __init__(self): pass @abstractmethod def doLocalAction(self): pass class DoAction(ToolCmd): def __init__(self): pass def doLocalAction(self): print 'doLocalAction()' # ================================================= class Test1(DoAction, ToolCmd): def __init__(self): pass class Test2(ToolCmd): def __init__(self): pass #class Test3(ToolCmd, DoAction): # def __init__(self): # pass # ================================================= t1 = Test1() t1.doLocalAction() t2 = Test2() t2.doLocalAction() #t3 = Test3() #t3.doLocalAction() # ================================================= Test Run: (Run-time error) $ ./c.py doLocalAction() Traceback (most recent call last): File "./c.py", line 42, in t2 =...

[ Level 3 ] Python config factory with __subclasses__.

It's a sample code for use __subclasses__ to validate the type. I write a config factory utility #!/bin/env python import os, sys if './' not in sys.path: sys.path.append('./') if './lib' not in sys.path: sys.path.append('./lib') if '../lib' not in sys.path: sys.path.append('../lib') if os.getcwd()+'/lib' not in sys.path: sys.path.append(os.getcwd()+'/lib') import yaml, json from libproperty import * from StringIO import StringIO from abc import * ## Property Abstract Class ## Subclass naming rule: xxxProperty. (e.g. JavaProperty, YamlProperty) class CommonProperty(object): __metaclass__ = ABCMeta def __init__(self, FILE=None): self._prop_file_output = self._prop_file = FILE self._prop = None pass @staticmethod def isValidType(type): return '%sProperty' % type in [ c.__name__ for c in CommonProperty.__subclasses__() ] @abstractmethod def save(self, FI...