Development How To
How to serve changes from atom with atom-live-server over HTTPS
Install atom-live-server using atoms extensions install mechanism.
Get the live server https module
npm i -g live-server-https
create .atom-live-server.json in your project's folder
{ "https":"/global/NodeModules/lib/node_modules/live-server-https" "port":8443, "host":"localhost" }
How to change location for user's ruby gems, python packages and node modules
Ruby
export GEM_HOME=/path/RubyGems/ export PATH=$PATH:/path/RubyGems/bin/
add to ~/.bashrc
check with
gem env
or
gem env gemdir
An alternative is to
echo "gem: --user-install -n~/Programs/RubyGems/" >> ~/.gemrc
Node
export NPM_CONFIG_PREFIX=/path/NodeModules/ export PATH=$PATH:/path/NodeModules/bin/
to ~/.bashrc
You can also do it by setting prefix=/path/NodeModules/ inside ~/.npmrc
To check
npm list -g
Note that his configures what npm calls "user global modules", ie these that are not specific to the current project you are working on. The project modules are installed into the project's folder.
An alternative is to
echo "prefix = ${HOME}/Programs/NodeModules" >> ~/.npmrc
Python
export PIP_PREFIX=/path/PythonPackages/ export PYTHONPATH=$PYTHONPATH:/media/Stuff/Programs/PythonPackages/lib/python2.7/site-packages export PATH="$PATH:/path/PythonPackages/bin"
to ~/.bashrc. Like Node this only affects global packages. For local ones use virtualenv
You can also do it by setting
[global] target=/media/Stuff/Programs/PythonPackages
inside .config/pip/pip.conf
http://pip.readthedocs.io/en/stable/user_guide/#configuration
How to configure pydev to ignore "Undefined variable from import” errors
Window -> Preferences -> PyDev -> Editor -> Code Analysis -> Undefined -> Undefined Variable From Import -> Ignore
How to check password strength with regular expressions
Use look-ahead constructs from the extended version of the regular expressions:
- Strong (at least eight characters with one of each, an upper alpha, lower alpha, digit and a special character)
- Medium (at least seven characters with an upper and a lower alpha, an upper and a digit or a lower and a digit)
- Weak (at least six characters)
How to run PyDoc as a windows service
The trick is that PyDoc does not exit after being started Try this
sc create PyDoc binpath= "F:\python\bin\python -c \"import pydoc;pydoc.cli()\" -p 8000"
or this
sc create PyDoc binpath= "start F:\python\bin\python -c \"import pydoc;pydoc.cli()\" -p 8000"
Jython's PyDoc is a bit trickier since the executable started is not python but java One could try to build a clever batch and try setting it up like this:
sc create PyDoc binpath= "start F:\apiscript\bin\jythonscript.bat -c \"import pydoc;pydoc.cli()\" -p 8000"
How to communicate between python processes
The following outlines different methods for communication between two independent python processes running on the same machine, although some, like socket and dbus can be used for across the network communication.
Communicate with signals
Pros
- Easy and simple
- Just signaling, no data transfers
- Signals may get delayed if the receiver is stuck on a long atomic python call
- Need to keep track of the receiver's PID
ps aux | grep name here | grep -v 'grep' | grep -v 'gdb' | grep -v "
Documentation - pydoc. Here is the example with the PID in a temp file. Receiver:
pidfile_name = "/tmp/myprog.pid"
if os.access(os.path.expanduser(pidfile_name), os.F_OK): #if the lockfile is already there then check the PID number in the lock file
pidfile = open(os.path.expanduser(pidfile_name), "r")
pidfile.seek(0)
pid = pidfile.readline()
# Now we check the PID from lock file matches to the current process PID
if os.path.exists("/proc/%s" % pid):
print "Already running. Stopping."
sys.exit(1)
else:
os.remove(os.path.expanduser(pidfile_name)) # File is there but the program is not running. Removing lock file.
file(pidfile_name, 'w').write(str(os.getpid()))
def flag_handler(sig, frame):
global flag
flag = (sig == signal.SIGUSR1)
signal.signal(signal.SIGUSR1,flag_handler) # register the signal
signal.signal(signal.SIGUSR2,flag_handler)
Sender:
pidfile_name = "/tmp/myprog.pid"
def sendsignal(flag):
if os.access(os.path.expanduser(pidfile_name), os.F_OK):
#if the lockfile is already there then check the PID number the lock file
pidfile = open(os.path.expanduser(pidfile_name), "r")
pidfile.seek(0)
pid = pidfile.readline()
if os.path.exists("/proc/%s" % pid): # Now we check the PID from lock file matches to the current process PID
os.kill(int(pid),signal.SIGUSR1 if flag else signal.SIGUSR2) # send the signal
else:
os.remove(os.path.expanduser(pidfile_name)) # File is there but the program is not running. Removing lock file.
else:
print "Target is not running"
Communicate with named pipes
Pros
- Send data between processes
- In memory communication. File is used just as a pointer
- Blocking reads. Needs extra code to make it non-blocking.
Here is an example using JSON packaging. Shared class:
import os, time, json, errno
def retry_write(*args, **kwargs):
# Like os.write, but retries until EAGAIN stops appearing
while True:
try:
return os.write(*args, **kwargs)
except OSError as e:
if e.errno == errno.EAGAIN:
time.sleep(0.5)
else:
raise
class Pipe(object):
# FIFO based IPC based on newline-separated JSON
ENCODING = 'utf-8'
def __init__(self,sPath):
self.sPath = sPath
if not os.path.exists(sPath):
os.mkfifo(sPath)
self.fd = os.open(sPath,os.O_RDWR | os.O_NONBLOCK)
self.file_blocking = open(sPath, "r", encoding=self.ENCODING)
def write(self, dmsg):
serialised = json.dumps(dmsg) + "\n"
dat = bytes(serialised.encode(self.ENCODING))
# This blocks until data can be read by other process.
# Can just use os.write and ignore EAGAIN if you want
# to drop the data
retry_write(self.fd, dat)
def read(self):
serialised = self.file_blocking.readline()
return json.loads(serialised)
Sender:
oP = Pipe(some_path)
while 1:
oP.write({'a':'foo','b':'bar','c':'erm...','d':'plop!','e':'etc'})
Receiver:
oP = Pipe(same_path_as_before)
while 1:
print(oP.read())
Communicate with memory mapped files
Pros
- Persistent data
- Use disk write/read for data transfers, which may wear out Flash storage
- This is just a flat storage, data structures, like FIFO, rolling queues etc, need to be implemented by hand.
import mmap, os, struct, time
def main():
fd = os.open('/tmp/mmaptest', os.O_RDONLY)
buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ) # Memory map the file
i = None
s = None
while 1:
new_i, = struct.unpack('i', buf[:4])
new_s, = struct.unpack('3s', buf[4:7])
if i != new_i or s != new_s:
print 'i: %s => %d' % (i, new_i)
print 's: %s => %s' % (s, new_s)
print 'Press Ctrl-C to exit'
i = new_i
s = new_s
time.sleep(1)
if __name__ == '__main__':
main()
Sender:
import ctypes, mmap, os, struct
def main():
fd = os.open('/tmp/mmaptest', os.O_CREAT | os.O_TRUNC | os.O_RDWR) # Create new empty file to back memory map on disk
assert os.write(fd, '\x00' * mmap.PAGESIZE) == mmap.PAGESIZE # Zero out the file to insure it's the right size
# Create the mmap instace with the following params:
# fd: File descriptor which backs the mapping or -1 for anonymous mapping
# length: Must in multiples of PAGESIZE (usually 4 KB)
# flags: MAP_SHARED means other processes can share this mmap
# prot: PROT_WRITE means this process can write to this mmap
buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
# Now create an int in the memory mapping
i = ctypes.c_int.from_buffer(buf)
i.value = 10
i.value += 1 # And manipulate it for kicks
assert i.value == 11
# Before we create a new value, we need to find the offset of the next free
# memory address within the mmap
offset = struct.calcsize(i._type_)
# The offset should be uninitialized ('\x00')
assert buf[offset] == '\x00'
# Now ceate a string containing 'foo' by first creating a c_char array
s_type = ctypes.c_char * len('foo')
# Now create the ctypes instance
s = s_type.from_buffer(buf, offset)
# And finally set it
s.raw = 'foo'
print 'First 10 bytes of memory mapping: %r' % buf[:10]
raw_input('Now run b.py and press ENTER')
print 'Changing i'
i.value *= i.value
print 'Changing s'
s.raw = 'bar'
new_i = raw_input('Enter a new value for i: ')
i.value = int(new_i)
if __name__ == '__main__':
main()
Communicate with sockets
Pros
- Can be local (bind to "localhost") or over the network
- A generic layer 7 library, not tailored for IPC specific functionality
- A lot of overhead for services running on the same machine
class mysocket:
def __init__(self, sock=None):
if sock is None:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
else:
self.sock = sock
def connect(self, host, port):
self.sock.connect((host, port))
def mysend(self, msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError("socket connection broken")
totalsent = totalsent + sent
def myreceive(self):
chunks = []
bytes_recd = 0
while bytes_recd < MSGLEN:
chunk = self.sock.recv(min(MSGLEN - bytes_recd, 2048))
if chunk == '':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_recd = bytes_recd + len(chunk)
return ''.join(chunks)
Communicate via dbus
Pros
- Local and over the network communication, tailored for efficient IPC calls and data transfers
- Linux oriented. Other OS's require dbus installed.
How to configure or reconfigure CPAN
perm -MCPAN -e shell o conf make_install_arg UNINGS=1 o conf prerequisites_policy follow o conf commit quit
might need to clean the old configs by
o conf init
How to get cpan modules under cygwin
open the cygwin setup program and make sure you have
- gzip
- tar
- unzp
- make
- lynx
- wget
- ncftpget
- gnupg
perl -MCPAN -eshell
configure
point to /cygdrive/c/Windows/system32/ftp for the ftp
refresh CPAN if needed:
intall Bundle::CPAN
reload cpan
How to install CPAN module
if not configured - configure first via
perl -MCPAN -e shell
then
perl -MCPAN -e 'install Convert/Cyrillic'
How to run a file from a browser
The following technique uses xul
<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/" type="text/css"?>
<window id="xxx" title="Process launcher" xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul" xmlns:html="http://www.w3.org/1999/xhtml">
<script type="application/x-javascript">
<![CDATA[
function getFile() {
try {
// 1. Turn off the security. For the local files it should work.
netscape.security.PrivilegeManager.enablePrivilege('UniversalXPConnect');
// 2. Get the file to be executed via file picker. File picker is not the only way to get the local file object,
const nsIFilePicker = Components.interfaces.nsIFilePicker;
var fp = Components.classes["@mozilla.org/filepicker;1"].createInstance(Components.interfaces.nsIFilePicker);
fp.init(window, "Open Executable file", nsIFilePicker.modeOpen);
var rv = fp.show();
if (rv == nsIFilePicker.returnOK) {
// So far so good...
var file = fp.file;
// Now create the process, itit and launch it
var process = Components.classes["@mozilla.org/process/util;1"].createInstance(Components.interfaces.nsIProcess);
process.init(file);
// The process will not block the browser (1st arg), no command line options (2nd and 3rd args)
process.run(false, {array: {}, size_iz: function() {return 0;}}, 0);
}
} catch(e) {
alert('!!! Bad news - "Exception, Sir!" ["+e+"]');
}
return;
}
]]>
</script>
<groupbox>
<description>Make sure that the browser's preference <html:code><html:b>signed.applets.codebase_principal_support</html:b></html:code> is set to <html:b>true</html:b>.
(Complete set of FF preferences is accessable via <html:u>about:config</html:u> in the browser's URL bar).
This approach to bypass FF's security is pretty fine for the development, especially when you run the developed code in the dedicated isolated FF profile.
Just don't forget to set it back to <html:b>false</html:b> when you are done with this sample.
Otherwise the page must be signed to have access to the browser's internals and then to the local resources.<html:br/><html:br/></description>
<button id="btnFile" label="OpenFile" oncommand="getFile();" />
</groupbox>
</window>
How to correct C code indentation, remove blank lines and line breaks
sudo apt-get install indent indent -nbc -br -brs -c33 -cd33 -ncdb -ce -ci4 -cli0 -cp33 -cs -d0 -di1 -nfc1 -nfca -hnl -i4 -ip0 -lp -npcs -nprs -npsl -saf -sai -saw -nsc -nbap -nbad -cdw -nss -npcs -nbc -brs -sob -l800 File.in -o File.out
How to correct HTML/XML indentations and syntax
tidy < infile > outfile
For XML do this:
tidy -mi -xml file-name
- -m - to modify the file
- -i - ti indent the file
- -xml - to indicate that the input file is an XML file
How to read a file from jar
Read a text file from a jar
import java.applet.*;
import java.io.*;
public class ReadFromJar extends Applet{
public void init(){
readTextFromJar("datafile1.txt");
readTextFromJar("test/datafile2.txt");
}
public void readTextFromJar(String s) {
String thisLine;
try {
InputStream is = getClass().getResourceAsStream(s);
BufferedReader br = new BufferedReader
(new InputStreamReader(is));
while ((thisLine = br.readLine()) != null) {
System.out.println(thisLine);
}
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Create 2 datafiles. datafile1.txt in the same directory as ReadfromJar.class and datafile2.txt in a subdirectory called test
[datafile1.txt] datafile1 line 1 datafile1 line 2 datafile1 line 3 datafile1 line 4 datafile1 line 5 [test\datafile2.txt] datafile2 line 1 datafile2 line 2 datafile2 line 3 datafile2 line 4 datafile2 line 5
Create the jar with
jar cf readjar.jar ReadFromJar.class datafile1.txt test\datafile2.txt
Try with the follwing HTML page
<HTML><HEAD></HEAD>
<BODY>
<APPLET CODE=ReadFromJar.class width=1 height=1 archive=readjar.jar></APPLET>
See java console for output
</BODY>
</HTML>
NOTE: In Applet, there is some restrictions about the file extension that can be used
How to log what files are being accessed by a program
Use strace
strace -t -e trace=file -o filetrace.log /path/program
If it feels like drinking from a fire hose, limit it further with the -e switch. You can also tie into an already running process:
strace -e trace=open,read -p PID -o tracefile.log
How to get the PID of an X window
Run the following verbatim
xprop _NET_WM_PID
then click on the window you're interested in. To list all processes with windows do this:
wmctrl -pl | awk '{print $3}' | xargs ps -p
To list all windows on the screen
xlsclients -al
To get info on a window id (0x12345)
xprop -id 0x12345
@HowTo @Development