using gmail smtp with redmine 2.4.1

It took me forever to get redmine emails working with gmail's smtp server.  After perusing the ruby action mailer website, I set up my redmine configuration.yml to look like this:

# default configuration options for all environments

default:
  # Outgoing emails configuration (see examples above)
  email_delivery:
    delivery_method: :smtp
    smtp_settings:
      address: 'smtp.gmail.com'
      port: 587
      domain: 'pickemupproductions.com'
      user_name: 'redmine@pickemupproductions.com'
      password: '****'
      authentication: 'plain'
      enable_starttls_auto: true

obviously replace password with your actual password.  Also, I'm using google apps with my wife's business domain.  So my username and domain are a bit differnet than what you would use for regular gmail (gmail.com).

Installing new CA certs on ubuntu for chrome or firefox

I recently needed to install a new CA cert into my ubuntu VM.  There are lots of tutorials out there that show how to use

sudo dpkg-reconfigure ca-certificates

This is all well and good if you want to install the CA into the global openssl store for things like git, wget, and apt.  However, if you want Chrome or Firefox to be able to trust the new CA, you need to install the CA cert into the NSS database.  (?!).

This SO article saved me about a day of frustration and sadly I don't have enough rep to upvote Johann's answer:

http://superuser.com/questions/437330/how-do-you-add-a-certificate-authority-ca-to-ubuntu/657177#657177?newreg=ad85846ee69249969b698b48bda371c3

Basically, you need to download the NSS client tools and use them to install the CA (the same one you put in openssl's store) into the backend NSS.  Chromium has the process documented here:

https://code.google.com/p/chromium/wiki/LinuxCertManagement

To install a new cert using the NSS client tools, do this:

certutil -d sql:$HOME/.pki/nssdb -A -t "C,," -n <certificate nickname> -i <certificate filename>

<certificate nickname> can be anything.  <certificate filename> is the full path to your PEM file (*.crt or *.cer).

Stitching together geoTiff files with gdal

Recently I was asked to import some geotiffs into a geoserver instance. I also needed to stitch the geotiffs together so I didn't have to create 100 some odd layers in geoserver.

I found a helpful python script here:
https://gist.github.com/philipn/1148693

But it was producing files that were way too big for my geoserver to handle. Inspired, I wrote the following to group a bunch of tiffs together in several composite geotiffs:

 
import os
import shutil
import sys
import time
import subprocess
from collections import OrderedDict


DIRS = ['/mnt/hgfs/f/imagery/va', '/mnt/hgfs/f/imagery/md']
GROUP_SIZE = 8

def runCmd( cmd ):
 #print cmd
 return subprocess.call( cmd, shell=True )

def tif(l) : return l.endswith( '.tif' ) and not l.startswith('group')

def sizeof_fmt(num):
 for x in ['bytes', 'kb', 'mb', 'gb', 'tb']:
  if num < 1024.0:
   return "%3.2f %s" % (num, x)
  num /= 1024.0
 return "%3.2f%s" % (num, ' tb')

def human_time( secs ):
 interval = OrderedDict( [("Y", 365*86400),
                          ("M", 30*86400),
                          ("W", 7*86400),
                          ("D", 86400),
                          ("h", 3600),
                          ("m", 60),
                          ("s", 1)])
 secs = int(secs)
 s = ""
 for unit, value in interval.items():
  subres = secs / value
  if subres > 0:
   secs = secs % subres
   s += str(subres) + unit
 return s



totalFilesProcessed = 0
totalFileCount = sum([len( filter( tif, sorted( os.listdir( dir ) ) ) ) for dir in DIRS])
pStart = time.time()
secsPerFile = 0;
for dir in DIRS:
 dStart = time.time()
 files = sorted( os.listdir( dir ) );
 files = filter( tif, files );
 i = 0
 rel_i = i * GROUP_SIZE
 dirFilesProcessed = 0
 dirFileCount = len(files)
 while( rel_i < dirFileCount ):
  gStart = time.time()
  tifsToProcess = [os.path.join(dir,x) for x in files[rel_i:rel_i + GROUP_SIZE]]
  finalTif = os.path.join(dir, 'group-%i.tif' % i)

  if os.path.exists( finalTif ):
   print 'WARNING, %s exists, removing...' % finalTif
   os.remove( finalTif )

  print 'processing %i files in group %i inside %s:\n\t%s\n' % (len(tifsToProcess), i, dir, '\n\t'.join(tifsToProcess))
 
  print 'uncompressing and nearblack\'ing...'
  for t in tifsToProcess:
   uc = '%s.uncompressed' % t
   if not os.path.exists( uc ):
    cmd = '/usr/local/bin/gdal_translate %s %s' % (t, uc)
    if runCmd( cmd ) != 0:
     sys.exit(1)
    cmd = '/usr/local/bin/nearblack -near 20 %s' %uc 
    if runCmd( cmd ) != 0:
     sys.exit(1)

  print '\nstitching...'
  cmd = '/usr/local/bin/gdalwarp -dstalpha -srcnodata 0 -wo "SKIP_NOSOURCE" --config "GDAL_CACHEMAX=500" -wm=500 %s.uncompressed %s' % ('.uncompressed '.join(tifsToProcess), finalTif) 
  if runCmd( cmd ) != 0:
   sys.exit(1)

  print '\nremoving temporary files...'
  for t in tifsToProcess:
   #print 'removing: %s.uncompressed' % t
   os.remove( '%s.uncompressed' % t )

  print '\nadding overviews...'
  cmd = '/usr/local/bin/gdaladdo %s 2 4 8 16 32' % finalTif
  if runCmd( cmd ) != 0:
   sys.exit(1)

  gStop = time.time()
  dirFilesProcessed += len(tifsToProcess)
  totalFilesProcessed += len(tifsToProcess)

  print '\nproduced %s file in %s: %s' % (sizeof_fmt(os.path.getsize(finalTif)), human_time(gStop - gStart), finalTif)
  print '%3.1f%% complete with directory %s. %i / %i files remain' % ((dirFilesProcessed * 100) / float(dirFileCount), dir, dirFileCount - dirFilesProcessed, dirFileCount )
  print '%3.1f%% complete with total. %i / %i files remain' % ((totalFilesProcessed * 100) / float(totalFileCount), totalFileCount - totalFilesProcessed, totalFileCount)

  secsPerFile = int((gStop - pStart) / (totalFilesProcessed))
  print 'estimated time remaining for directory: %s' % (human_time( (dirFileCount - dirFilesProcessed) * secsPerFile ))
  print 'estimated time remaining for all dirs:  %s' % (human_time( (totalFileCount - totalFilesProcessed) * secsPerFile))
  print '\n\n'

  i += 1
  rel_i = i * GROUP_SIZE

 dStop = time.time()
 print 'processed dir %s in %s' % (dir, human_time(dStop - dStart))

pStop = time.time()
print 'total runtime: %s' % (human_time(pStop - pStart))

Basically this script uncompresses and runs nearblack on each tiff in a group and then stitches them together using gdalwarp into a single image.  You can define a set of directories to operate over and it will look through and create grouped tiffs in each.

KNOWN ISSUES

One issue with this script is it will blindly take the first 8 (in its posted state) files and stitch them together.  That may be what you want.  But if your tiff filenames are such that the neighbor images may not be next to each other in a directory listing, the script has the potential to generate some really big resulting tiffs.

For example, say you define a GROUP_SIZE of 2 and you have two geoTiffs tile1.tiff and tile2.tiff.  If those two imagery tiles are separated by 100 miles, the resulting tiff will include those two tiles plus whitespace in between them.  The resulting tiff file size is going to be much larger than the sum of the file sizes of tile1 and tile2.

Even one outlier tile has the potential to really drive up filesize if it is far enough away from the rest of the group.

It would be better to somehow parse the geographic information in the group and only stitch together adjacent tiles.  Perhaps you define a max limit to the number of adjacent tiles in a single group.  But it would be nice to avoid generating groups with outliers.

adding an icon to a C++ project with Visual Studio Express 2012

I wrote a simple application to perform a left mouse click after a certain amount of time elapsed.  Its pretty simple.  Took me maybe a half hour to code it up while wading though MSDN.

It took me at least 3 times as long to add an icon to my project.  Here's how I did it.

First, I created a .png icon.  This is no good, we need an .ico.  Fortunately, there is an awesome website that converts images to .icos:

http://www.icoconverter.com/

I had the best luck with All Sizes and 256 colors.

Next I needed to manually create a resource script for my project.  Real (non-free) versions of Visual Studio do this for you automatically.  Simply use your favorite text editor (mine is gvim) and create a text file that looks something like the following:

programIcon ICON "icon.ico"

I named my script "resources.script" and placed it in the same directory as my icon.ico file.

Next you need to manually run rc.exe.  This should get installed with Visual Studio, mine was located here:

C:\Program Files (x86)\Windows Kits\8.0\bin\x86

I added this location to my path and then opened a shell to the location of my resources.script and ran:

rc.exe resources.script

This produced a "resources.res" file.  I imported this into my Visual Studio project by right clicking the "resources" folder under my project under my solution and "Add Existing".  Then I rebuilt my solution and voila!  icon!

mp4 video in chrome

I was having a really hard time getting mp4 video to render in Chrome with the html 5 <video> tag.  After screwing around with ffmpeg for a day I think I found the correct settings.  I used the following command line to transcode a wmv to the proper mp4:

$ ffmpeg -i movie.wmv -pix_fmt yuv420p -vf setsar='1/1' vid.mp4

The key difference from the default .wmv -> .mp4 conversion is specifying the pixel format as yuv420p and setting the storage aspect ratio (SAR) to 1:1 (or 1/1). The default using a rgb pixel format and inherites the storage aspect ratio from the source... which may or may not be one to one. The resulting video looks like this:



$ ffprobe -show_format vid.mp4 
ffprobe version 1.1.git Copyright (c) 2007-2013 the FFmpeg developers
  built on Feb  5 2013 07:56:01 with gcc 4.6 (Ubuntu/Linaro 4.6.3-1ubuntu5)
  configuration: --enable-gpl --enable-libass --enable-libfaac --enable-libfdk-aac --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-librtmp --enable-libtheora --enable-libvorbis --enable-x11grab --enable-libx264 --enable-nonfree --enable-version3
  libavutil      52. 17.101 / 52. 17.101
  libavcodec     54. 91.100 / 54. 91.100
  libavformat    54. 61.104 / 54. 61.104
  libavdevice    54.  3.103 / 54.  3.103
  libavfilter     3. 35.101 /  3. 35.101
  libswscale      2.  2.100 /  2.  2.100
  libswresample   0. 17.102 /  0. 17.102
  libpostproc    52.  2.100 / 52.  2.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'vid.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf54.61.104
  Duration: 00:00:17.07, start: 0.000000, bitrate: 1817 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 880x646 [SAR 1:1 DAR 440:323], 1814 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc
    Metadata:
      handler_name    : VideoHandler
[FORMAT]
filename=vid.mp4
nb_streams=1
format_name=mov,mp4,m4a,3gp,3g2,mj2
format_long_name=QuickTime / MOV
start_time=0.000000
duration=17.067000
size=3877456
bit_rate=1817522
TAG:major_brand=isom
TAG:minor_version=512
TAG:compatible_brands=isomiso2avc1mp41
TAG:encoder=Lavf54.61.104
[/FORMAT]



Chrome's embedded flash broken by Microsoft's Enhanced Mitigation Experience Toolkit (EMET)

Microsoft's Enhanced Mitigation Experience Toolkit breaks recent versions of Chrome. I had a nagging issue with Chrome in that it wouldn't load the newly sandboxed flash player. This would cause minor irritations like not being able to view youtube videos in Chrome as well as major problems like Google Calendar completely crashing.

After spending a day and a half troubleshooting drivers, services, and startup programs between safe mode and normal mode (Chrome worked fine in safe mode with networking), I uninstalled EMET on a whim and low and behold, Chrome works.

EMET is designed to automatically employ ASLR and structured exception handling protections on executables that weren't necessarily compiled with these features in mind. It also breaks stuff. Lots of stuff by the looks of it. Including Chrome.

I'm not sure which mitigation was breaking Chrome, but uninstalling EMET took care of it :) I wish I knew more about the flash sandboxing implementation in Chrome in order to identify exactly what went wrong. Reversing Chrome's sandboxing sounds like a project for another day. I'm sure PinkiePie knows exactly what happened.

EDIT: Looks like SEHOP is the offending mitigation. Disabling this for chrome.exe may be less hammer like than uninstalling EMET.

git differencing

i can never remember these, and unfortunately i don't use git often enough to adequately drill it into my head.

To get a name and status difference of what changed between two branches, use this:

git diff --name-status master..branch

To compare two files from different branches, use this:

git difftool branch1:file branch2:file