Development How To

How to serve changes from atom with atom-live-server over HTTPS

Install atom-live-server using atoms extensions install mechanism.
Get the live server https module
npm i -g live-server-https
create .atom-live-server.json in your project's folder

{
  "https":"/global/NodeModules/lib/node_modules/live-server-https"
  "port":8443,
  "host":"localhost"
}

How to change location for user's ruby gems, python packages and node modules

Ruby

export GEM_HOME=/path/RubyGems/
export PATH=$PATH:/path/RubyGems/bin/

add to ~/.bashrc
check with
gem env
or
gem env gemdir

An alternative is to


echo "gem: --user-install -n~/Programs/RubyGems/" >> ~/.gemrc

Node

export NPM_CONFIG_PREFIX=/path/NodeModules/
export PATH=$PATH:/path/NodeModules/bin/

to ~/.bashrc
You can also do it by setting prefix=/path/NodeModules/ inside ~/.npmrc
To check
npm list -g
Note that his configures what npm calls "user global modules", ie these that are not specific to the current project you are working on. The project modules are installed into the project's folder.

An alternative is to


echo "prefix = ${HOME}/Programs/NodeModules" >> ~/.npmrc


Python

export PIP_PREFIX=/path/PythonPackages/
export PYTHONPATH=$PYTHONPATH:/media/Stuff/Programs/PythonPackages/lib/python2.7/site-packages
export PATH="$PATH:/path/PythonPackages/bin"

to ~/.bashrc. Like Node this only affects global packages. For local ones use virtualenv
You can also do it by setting

[global]
target=/media/Stuff/Programs/PythonPackages

inside .config/pip/pip.conf
http://pip.readthedocs.io/en/stable/user_guide/#configuration

How to configure pydev to ignore "Undefined variable from import” errors

Window -> Preferences -> PyDev -> Editor -> Code Analysis -> Undefined -> Undefined Variable From Import -> Ignore

How to check password strength with regular expressions

Use look-ahead constructs from the extended version of the regular expressions:

  • Strong (at least eight characters with one of each, an upper alpha, lower alpha, digit and a special character)
^(?=.{8,})(?=.*[A-Z])(?=.*[a-z])(?=.*[0-9])(?=.*\W).*$
  • Medium (at least seven characters with an upper and a lower alpha, an upper and a digit or a lower and a digit)
^(?=.{7,})(((?=.*[A-Z])(?=.*[a-z]))|((?=.*[A-Z])(?=.*[0-9]))|((?=.*[a-z])(?=.*[0-9]))).*$
  • Weak (at least six characters)
^.{6,}$


How to run PyDoc as a windows service

The trick is that PyDoc does not exit after being started Try this
sc create PyDoc binpath= "F:\python\bin\python -c \"import pydoc;pydoc.cli()\" -p 8000"
or this
sc create PyDoc binpath= "start F:\python\bin\python -c \"import pydoc;pydoc.cli()\" -p 8000"
Jython's PyDoc is a bit trickier since the executable started is not python but java One could try to build a clever batch and try setting it up like this:
sc create PyDoc binpath= "start F:\apiscript\bin\jythonscript.bat -c \"import pydoc;pydoc.cli()\" -p 8000"

How to communicate between python processes

The following outlines different methods for communication between two independent python processes running on the same machine, although some, like socket and dbus can be used for across the network communication.

Communicate with signals

Pros

  • Easy and simple
Cons:
  • Just signaling, no data transfers
  • Signals may get delayed if the receiver is stuck on a long atomic python call
  • Need to keep track of the receiver's PID
You could find out PID in the real time by running
ps aux | grep name here | grep -v 'grep' | grep -v 'gdb' | grep -v "

Documentation - pydoc. Here is the example with the PID in a temp file. Receiver:

pidfile_name = "/tmp/myprog.pid"
if os.access(os.path.expanduser(pidfile_name), os.F_OK): #if the lockfile is already there then check the PID number in the lock file
    pidfile = open(os.path.expanduser(pidfile_name), "r")
    pidfile.seek(0)
    pid = pidfile.readline()
    # Now we check the PID from lock file matches to the current process PID
    if os.path.exists("/proc/%s" % pid):
        print "Already running. Stopping."
        sys.exit(1)
    else:
        os.remove(os.path.expanduser(pidfile_name)) # File is there but the program is not running. Removing lock file.

file(pidfile_name, 'w').write(str(os.getpid()))

def flag_handler(sig, frame):
    global flag
    flag = (sig == signal.SIGUSR1)

signal.signal(signal.SIGUSR1,flag_handler) # register the signal
signal.signal(signal.SIGUSR2,flag_handler)

Sender:

pidfile_name = "/tmp/myprog.pid"
def sendsignal(flag):
    if os.access(os.path.expanduser(pidfile_name), os.F_OK):
        #if the lockfile is already there then check the PID number the lock file
        pidfile = open(os.path.expanduser(pidfile_name), "r")
        pidfile.seek(0)
        pid = pidfile.readline()
        if os.path.exists("/proc/%s" % pid): # Now we check the PID from lock file matches to the current process PID
            os.kill(int(pid),signal.SIGUSR1 if flag else signal.SIGUSR2) # send the signal
        else:
            os.remove(os.path.expanduser(pidfile_name)) # File is there but the program is not running. Removing lock file.
    else:
        print "Target is not running"

Communicate with named pipes

Pros

  • Send data between processes
  • In memory communication. File is used just as a pointer
Cons:
  • Blocking reads. Needs extra code to make it non-blocking.

Here is an example using JSON packaging. Shared class:

import os, time, json, errno

def retry_write(*args, **kwargs):
    # Like os.write, but retries until EAGAIN stops appearing
    while True:
        try:
            return os.write(*args, **kwargs)
        except OSError as e:
            if e.errno == errno.EAGAIN:
                time.sleep(0.5)
            else:
                raise
class Pipe(object):
    # FIFO based IPC based on newline-separated JSON
    ENCODING = 'utf-8'

    def __init__(self,sPath):
        self.sPath = sPath
        if not os.path.exists(sPath):
            os.mkfifo(sPath)
        self.fd = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
        self.file_blocking = open(sPath, "r", encoding=self.ENCODING)

    def write(self, dmsg):
        serialised = json.dumps(dmsg) + "\n"
        dat = bytes(serialised.encode(self.ENCODING))
        # This blocks until data can be read by other process.
        # Can just use os.write and ignore EAGAIN if you want
        # to drop the data
        retry_write(self.fd, dat)

    def read(self):
        serialised = self.file_blocking.readline()
        return json.loads(serialised)

Sender:

oP = Pipe(some_path)
while 1:
    oP.write({'a':'foo','b':'bar','c':'erm...','d':'plop!','e':'etc'})

Receiver:

oP = Pipe(same_path_as_before)
while 1:
    print(oP.read())

Communicate with memory mapped files

Pros

  • Persistent data
Cons:
  • Use disk write/read for data transfers, which may wear out Flash storage
  • This is just a flat storage, data structures, like FIFO, rolling queues etc, need to be implemented by hand.
Receiver:
import mmap, os, struct, time

def main():
    fd = os.open('/tmp/mmaptest', os.O_RDONLY)
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ) # Memory map the file
    i = None
    s = None
    while 1:
        new_i, = struct.unpack('i', buf[:4])
        new_s, = struct.unpack('3s', buf[4:7])
        if i != new_i or s != new_s:
            print 'i: %s => %d' % (i, new_i)
            print 's: %s => %s' % (s, new_s)
            print 'Press Ctrl-C to exit'
            i = new_i
            s = new_s
        time.sleep(1)

if __name__ == '__main__':
    main()

Sender:

import ctypes, mmap, os, struct

def main():
    fd = os.open('/tmp/mmaptest', os.O_CREAT | os.O_TRUNC | os.O_RDWR)     # Create new empty file to back memory map on disk
     assert os.write(fd, '\x00' * mmap.PAGESIZE) == mmap.PAGESIZE          # Zero out the file to insure it's the right size
    # Create the mmap instace with the following params:
    # fd: File descriptor which backs the mapping or -1 for anonymous mapping
    # length: Must in multiples of PAGESIZE (usually 4 KB)
    # flags: MAP_SHARED means other processes can share this mmap
    # prot: PROT_WRITE means this process can write to this mmap
    buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
    # Now create an int in the memory mapping
    i = ctypes.c_int.from_buffer(buf)
    i.value = 10
    i.value += 1     # And manipulate it for kicks
    assert i.value == 11
    # Before we create a new value, we need to find the offset of the next free
    # memory address within the mmap
    offset = struct.calcsize(i._type_)
    # The offset should be uninitialized ('\x00')
    assert buf[offset] == '\x00'
    # Now ceate a string containing 'foo' by first creating a c_char array
    s_type = ctypes.c_char * len('foo')
    # Now create the ctypes instance
    s = s_type.from_buffer(buf, offset)
    # And finally set it
    s.raw = 'foo'
    print 'First 10 bytes of memory mapping: %r' % buf[:10]
    raw_input('Now run b.py and press ENTER')
    print 'Changing i'
    i.value *= i.value
    print 'Changing s'
    s.raw = 'bar'
    new_i = raw_input('Enter a new value for i: ')
    i.value = int(new_i)

if __name__ == '__main__':
    main()

Communicate with sockets

Pros

  • Can be local (bind to "localhost") or over the network
Cons:
  • A generic layer 7 library, not tailored for IPC specific functionality
  • A lot of overhead for services running on the same machine
Documentation - library, how to.
class mysocket:
    def __init__(self, sock=None):
        if sock is None:
            self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        else:
            self.sock = sock

    def connect(self, host, port):
        self.sock.connect((host, port))

    def mysend(self, msg):
        totalsent = 0
        while totalsent < MSGLEN:
            sent = self.sock.send(msg[totalsent:])
            if sent == 0:
                raise RuntimeError("socket connection broken")
            totalsent = totalsent + sent

    def myreceive(self):
        chunks = []
        bytes_recd = 0
        while bytes_recd < MSGLEN:
            chunk = self.sock.recv(min(MSGLEN - bytes_recd, 2048))
            if chunk == '':
                raise RuntimeError("socket connection broken")
            chunks.append(chunk)
            bytes_recd = bytes_recd + len(chunk)
        return ''.join(chunks)

Communicate via dbus

Pros

  • Local and over the network communication, tailored for efficient IPC calls and data transfers
Cons:
  • Linux oriented. Other OS's require dbus installed.
Documentation - http://dbus.freedesktop.org/doc/dbus-python/doc/tutorial.html


How to configure or reconfigure CPAN

perm -MCPAN -e shell
o conf make_install_arg UNINGS=1
o conf prerequisites_policy follow
o conf commit
quit

might need to clean the old configs by
o conf init

How to get cpan modules under cygwin

open the cygwin setup program and make sure you have

  • gzip
  • tar
  • unzp
  • make
  • lynx
  • wget
  • ncftpget
  • gnupg
run
perl -MCPAN -eshell
configure
point to /cygdrive/c/Windows/system32/ftp for the ftp
refresh CPAN if needed:
intall Bundle::CPAN
reload cpan

How to install CPAN module

if not configured - configure first via
perl -MCPAN -e shell
then
perl -MCPAN -e 'install Convert/Cyrillic'

How to run a file from a browser

The following technique uses xul

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<window id="xxx" title="Process launcher" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul" xmlns:html="http://www.w3.org/1999/xhtml">
<script type="application/x-javascript">
<![CDATA[
    function getFile() {
        try {
            // 1. Turn off the security. For the local files it should work.
            netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
            // 2. Get the file to be executed via file picker. File picker is not the only way to get the local file object,
            const nsIFilePicker = Components.interfaces.nsIFilePicker;
            var fp = Components.classes["@mozilla.org/filepicker;1"].createInstance(Components.interfaces.nsIFilePicker);
            fp.init(window, "Open Executable file", nsIFilePicker.modeOpen);
            var rv = fp.show();
            if (rv == nsIFilePicker.returnOK) {
                // So far so good...
                var file = fp.file;
                // Now create the process, itit and launch it
                var process = Components.classes["@mozilla.org/process/util;1"].createInstance(Components.interfaces.nsIProcess);
                process.init(file);
                // The process will not block the browser (1st arg), no command line options (2nd and 3rd args)
                process.run(false, {array: {}, size_iz: function() {return 0;}}, 0);
            }
        } catch(e) {
            alert('!!! Bad news - "Exception, Sir!" ["+e+"]');
        }
        return;
    }
]]>
</script>
<groupbox>
<description>Make sure that the browser's preference <html:code><html:b>signed.applets.codebase_principal_support</html:b></html:code> is set to <html:b>true</html:b>.
(Complete set of FF preferences is accessable via <html:u>about:config</html:u> in the browser's URL bar).
This approach to bypass FF's security is pretty fine for the development, especially when you run the developed code in the dedicated isolated FF profile.
Just don't forget to set it back to <html:b>false</html:b> when you are done with this sample.
Otherwise the page must be signed to have access to the browser's internals and then to the local resources.<html:br/><html:br/></description>
<button id="btnFile" label="OpenFile" oncommand="getFile();" />
</groupbox>
</window>

How to correct C code indentation, remove blank lines and line breaks

sudo apt-get install indent
indent -nbc -br -brs -c33 -cd33 -ncdb -ce -ci4 -cli0 -cp33 -cs -d0 -di1 -nfc1 -nfca -hnl -i4 -ip0 -lp -npcs -nprs -npsl -saf -sai -saw -nsc -nbap -nbad -cdw -nss -npcs -nbc -brs -sob -l800 File.in -o File.out

How to correct HTML/XML indentations and syntax

tidy < infile > outfile
For XML do this:
tidy -mi -xml file-name

  • -m - to modify the file
  • -i - ti indent the file
  • -xml - to indicate that the input file is an XML file


How to read a file from jar

Read a text file from a jar

import java.applet.*;
import java.io.*;

public class ReadFromJar extends Applet{

public void init(){
  readTextFromJar("datafile1.txt");
  readTextFromJar("test/datafile2.txt");
  }

  public void readTextFromJar(String s) {
    String thisLine;
    try {
      InputStream is = getClass().getResourceAsStream(s);
      BufferedReader br = new BufferedReader
         (new InputStreamReader(is));
      while ((thisLine = br.readLine()) != null) {
         System.out.println(thisLine);
         }
      }
    catch (Exception e) {
      e.printStackTrace();
      }
  }
}

Create 2 datafiles. datafile1.txt in the same directory as ReadfromJar.class and datafile2.txt in a subdirectory called test

[datafile1.txt]
datafile1  line 1
datafile1  line 2
datafile1  line 3
datafile1  line 4
datafile1  line 5
[test\datafile2.txt]
datafile2  line 1
datafile2  line 2
datafile2  line 3
datafile2  line 4
datafile2  line 5

Create the jar with
jar cf readjar.jar ReadFromJar.class datafile1.txt test\datafile2.txt
Try with the follwing HTML page

<HTML><HEAD></HEAD>
  <BODY>
    <APPLET CODE=ReadFromJar.class width=1 height=1 archive=readjar.jar></APPLET>
    See java console for output
  </BODY>
</HTML>

NOTE: In Applet, there is some restrictions about the file extension that can be used

How to log what files are being accessed by a program

Use strace

strace -t -e trace=file -o filetrace.log /path/program

If it feels like drinking from a fire hose, limit it further with the -e switch. You can also tie into an already running process:

strace -e trace=open,read -p PID -o tracefile.log 

How to get the PID of an X window

Run the following verbatim
xprop _NET_WM_PID
then click on the window you're interested in. To list all processes with windows do this:
wmctrl -pl | awk '{print $3}' | xargs ps -p
To list all windows on the screen
xlsclients -al
To get info on a window id (0x12345)
xprop -id 0x12345

@HowTo @Development