Creating a debian package from binaries

Create directory following the naming $PACKAGE-$VERSION:

mkdir private-tomcat-7.0.42
cd private-tomcat-7.0.42

Copy to this directory the files that you want to be installed.


Create configuration:

#dh_make --createorig

Type of package: single binary, indep binary, multiple binary, library, kernel module, kernel patch?
 [s/i/m/l/k/n] s

Maintainer name  : root
Email-Address    : root@unknown
Date             : Wed, 21 Aug 2013 01:36:59 -0700
Package Name     : private-tomcat
Version          : 7.0.42
License          : blank
Type of Package  : Single
Hit  to confirm: 
Currently there is no top level Makefile. This may require additional tuning.
Done. Please edit the files in the debian/ subdirectory now. You should also
check that the private-tomcat Makefiles install into $DESTDIR and not in / .

Edit the created debian/control file:

Source: private-tomcat
Section: main
Priority: extra
Maintainer: root 
Build-Depends: debhelper (>= 8.0.0)
Standards-Version: 3.9.2
#Vcs-Git: git://git.debian.org/collab-maint/private-tomcat.git
#Vcs-Browser: http://git.debian.org/?p=collab-maint/private-tomcat.git;a=summary

Package: private-tomcat
Replaces: 7.0.35
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: Apache tomcat for internal use in private Project

Edit debian/changelog and remove references to ITP (is not needed)

Rename debian/postinst.ex to postinst

mv postinst.ex postinst

Create the file debian/packagename.install with the following structure:

path/to/file/relative/to/source/root path/to/install/relative/to/system/root

i.e.

apache-tomcat-7.0.42/* opt/

change debian/compat to level 7

echo "7" > debian/compat

Copy the init.d or upstart script for the installed package name it as
debian/$PACKAGENAME.init

then run:

dh_installinit

IF you want to install the init.d package under a different name, please rename the init file to debian/$NAME.init and edit debian/rules and add the following

override_dh_installinit
 dh_installinit --name=tomcat (this is the desired name)

please remember to run dh_prep between runs of dh_installinit, Otherwise, it may cause multiple instances of the same text to be added to maintainer scripts.


from the directory above:

dpkg-buildpackage -b -us -uc

This will build the package and avoid signing errors

Puppet: certificates signed with multiple names

The default creation of SSL certs for the puppet master create a certicate base on the hostname of the server. If this server is reached with two different names (zB internalname.domain.com & external.domain.com) this creates problems such as: Server hostname 'internalname.domain.com' did not match server certificate; expected observer1 You can do the following:

rm -fr /var/lib/puppet/ssl #could delete individual .pem files 
puppet cert generate external.domain.com --dns_alt_names=internalname.domain,puppet

If executed as above (deleting all certs, all puppet nodes need to request certificate and the master to sign it)

Managing SIP extensions with puppet

I decided to take the next step into the world of Puppet and actually created my own Provider and Type. So I would like to introduce "puppet-asterisk" which creates a custom type and provider for managing Asterisk's SIP configuration. The provider allows for simple SIP resource management, by managing /etc/asterisk/sip.conf and treating each SIP extension defined as an individual resource, so the following extension:

[100]
type=friend
host=dynamic
secret=MyPass123
context=internal
mailbox=100@default
callgroup=1
pickupgroup=1
dtmfmode=rfc2833
canreinvite=no
permit=10.34.0.1/32
deny=10.34.0.2/32

Can be expressed as:

sip {'100':
    ensure  => present,
    type    => 'friend',
    host    => 'dynamic',
    secret  => 'MyPass123',
    context => 'internal'
    mailbox => '100@default'
    permit  => '10.34.0.1/32',
    deny    => '10.34.0.2/32',
}

This allow to actually manage a SIP extension and it's properties individually and even grouping them as Virtual Resources. I have scouted Puppet Forge and most solutions focus completely on uploading the entire sip.conf as a template, which is pretty limiting. We must aim to keep as much as the infrastructure managed by Puppet as possible. Installing At the time of this writing I have not yet submitted it to puppet forge, so you must clone the repository from Github https://github.com/AlexRRR/puppet-asterisk and copy it to your /etc/puppet/modules Testing Please make sure you have rake, rspec installed

gem install rake
gem install rspec

Run the tests from root directory of project

>
rake spec

Creating a private signed repository with Ubuntu

To have a streamlined deployment and management strategy via Puppet it is necessary to be able to create signed Debian Packages and have a private signed repository that can contain such packages, this makes the Puppet manifest simpler and the updates will be done in a controlled manner.

Webserver configuration

Installing the webserver

apt-get install apache

<Directory /var/www/repos/ >
        Options Indexes FollowSymLinks Multiviews
        Order allow,deny
        Allow from all
</Directory>
<Directory "/var/www/repos/apt/*/db/">
        Order allow,deny
        Deny from all
</Directory>
<Directory "/var/www/repos/apt/*/conf/">
        Order allow,deny
        Deny from all
</Directory>
<Directory "/var/www/repos/apt/*/incoming/">
        Order allow,deny
        Deny from all
<?/Directory>

RNG Tools

The rngd daemon acts as a bridge between a Hardware TRNG (true random number generator) such as the ones in some Intel/AMD/VIA chipsets, and the kernel's PRNG (pseudo-random number generator). Virtual Machines are generally really bad at generating enough entropy that the gpg key generation need. RNG Tools aids that process.< /br>

NOTE: This step is optional but highly recommended if you are using a virtual machine

apt-get install rng-tools
vi /etc/default/rng-tools

Edit and Add the following

HRNGDEVICE=/dev/urandom

Finally start the service

/etc/init.d/rng-tools start

GPG key file generation

Next we will generate a GPG key to sign our repository and all of our packages with.

root@agent1:/home/alex# gpg --gen-key
gpg (GnuPG) 1.4.11; Copyright (C) 2010 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) "
Real name: DevOps stuff
Email address: dev@gmx.de
Comment: DevOps stuff
You selected this USER-ID:
    "devops (DevOps stuff) "
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.
gpg: gpg-agent is not available in this session
passphrase not correctly repeated; try again.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
...+++++
+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.................+++++
.......+++++
gpg: key F15542E1 marked as ultimately trusted
public and secret key created and signed.
gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/F15542E1 2012-11-30
      Key fingerprint = FF1E A313 F3DA 256B A40E  FB3E 0749 613D F155 42E1
uid                  devops (DevOps stuff) 
sub   2048R/024EC2A2 2012-11-30

Create public GPG key

Now the key is already created and in the keychain. You may list it with

root@agent1:/home/alex# gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub   2048R/F15542E1 2012-11-30
uid                  devops (DevOps stuff) 
sub   2048R/024EC2A2 2012-11-30

Create the gpg key file that will be shared publicly via the webserver.

root@agent1:/home/alex# gpg --armor --output secure.gpg.key --export 024EC2A2

please note 024EC2A2 comes from the gpg --list-keys above!

Configure the Repository

Create the directory structure

mkdir -p /var/www/repos/apt/debian/conf
cd /var/www/repos/apt/debian/conf

Create the file "distributions" in the cwd and make sure the gpg key is used to sign the repository, add the following to the file

Origin: Alex Debian repo
Label: alex
Codename: precise
Architectures: amd64 i386 source
Components: main
Description: alex apt repository
DebOverride: override.precise
DscOverride: override.precise
SignWith: 024EC2A2

Create a file called "options" with the following content:

verbose
basedir .
ask-passphrase

Now create an empty file called "override.precise"

touch override.precise

Signing existing packages

dpkg-sig --sign builder /home/alex/demo-package-2.0.6_1.0-1_amd64.deb

Managing the repository

For this task we will use the tool "reprepro" which makes it very straightforward:
Install reprepro:

sudo apt-get install reprepro

Add a package:

reprepro -Vb . includedeb precise /home/alex/demo-package-2.0.6_1.0-1_amd64.deb

Remove a package:

reprepro -b . remove precise demo-package

Please note that when removing we are using the package name as it is described in the package manifest and the the filename itself

Encryped private keys

Protecting your private ssh-keys

I've noticed that most people are content just generating a keypair via ssh-keygen and using it as it is, but it is extremely important that the private key itself is encrypted. (I am not talking about the passhphrase... I am taking about encryption here). This can be achieved this way

openssl genrsa -aes256 -out raal.pem 2048

This will generate an encrypted .pem file, protected with the password you entered. This pemfile contains both your public and private keys. To extract your public key you need to do this:

chmod 600 raal.pem

ssh-keygen -y -f raal.pem > raal.pub

Counting occurences of subclasses in Django

Counting occurences of subclasses in Django

As a Django learning project I have began CONTMAN, a simple mobile content distribution system, and as expected I have ran into many different kinds of problems that I have thought it would be interesting to keep a log of... and this is it.

This might be the classic case of "When all you have is a hammer everything looks like a nail" as when I was thinking on how to design the project I came across Django inheritance and polymorphism features. "Aha! this is EXACTLY what I need" I said to myself. Now I am not so sure, but I am perhaps one of the most stubborn people in the world, so I kept going in that direction.

So I thought. I have two, maybe three different kinds of content I want to be able to distribute with this platform, for now I have "Ringtones" and "Wallpapers" (NB: I know it is not bleeding edge stuff, but keep in mind this was for learning, and I could not think of anything else at the time), and both of them are Content.

So we have a parent Content Class

class Content(models.Model):
    name = models.CharField(max_length=100)
    created_on = models.DateTimeField('created', auto_now_add = True )
    keyword = models.CharField(max_length=100)

And to children subclasses

class Ringtone(Content):
    artist = models.ManyToManyField(Artist)
    data_file = models.FileField(upload_to="uploads")

class Wallpaper(Content):
    category = models.ForeignKey(Category)
    data_file = models.ImageField(upload_to="uploads")

Ok, so there is a whole bunch of other stuff there but this is enough to get the point of this post across.

To me this was going to be very helpful in a lot of ways to mention some:

• Delivering content would be as easy as pointing to Content.data_file (which you can't is it is clearly stated at the very end of this doc entry.)
• The admin app would let me easily create a manager for uploading the file depending on the type (see comment above)
• Creating reports would be very easy, just create a query set from the Content model.

This last point is the one we will be discussing. 

So it turns out that

a = Content.objects.get(pk=1)

returns a Content object, so you cannot count or fetch content using  "a.data_file".  So how can I do a simple report showing the count of each type of content available? The easy (lame) way could be to simple do two queries of Ringtone and Wallpapers as we only have those and leave Content alone. But this is not nice, in fact it reeks of bad code and self-pity. What if I later add support for "applications" and "games" Content types?

So after a nice session of Django docs + stack overflow + google I think I came across a very simple but effective solution here in stack overflow

The idea is to take advantage of the Contenttypes framework to keep track of what kind of model each object represents. This way we can aggregate easily on our query a create this type of report. So we modify the base (parent) class like this:

class Content(models.Model):
    name = models.CharField(max_length=100)
    created_on = models.DateTimeField('created', auto_now_add = True )
    keyword = models.CharField(max_length=100)
    content_type = models.ForeignKey("contenttypes.ContentType")

With this then we can easily do this:

class Content(models.Model):
>>>Content.objects.values('content_type__name').annotate(hits=Count("content_type"))
[{'hits': 1, 'content_type__name': u'ringtone'}, {'hits': 2, 'content_type__name': u'wallpaper'}]

Mysql and group_concat!

I'm determined to write a little more often on my blog (I've skipped 2008 altogether) because it also works as my personal notepad for the world to see. So let's get started.

Today I've discovered a wonderful new MySQL trick: group_concat and it saved me from doing tedious/slow scripting and also from trying to understand how Stored Procedures work in MySQL (which is something like arameic to me right now).

I've been working with Eventum which is a lovely open source bug tracker from the great thinkers at MySQL AB (now.. Sun). Eventum lets you add N custom fields to the issue creation and listing page, and it adds all of them to a table called eventum_issue_custom_fields. very much like this:


Each custom field has an Id, but they are all entered in different "rows" like this:
(lets say custon field "1" is the tv show name and field "2" is a short description)

IssueId	CustomFieldId	CustomFieldData
50	1	90210
50	2	Lame TV show
51	1	Family Guy
51	2	Funny Stuff

what if i wanted to select from this table the tv show name and the description in just one row?

Thinking about loops in your scripts?
Thinking about stored procedures?
Thinking it cannot be done at all?

Check this out:

SELECT IssueId,GROUP_CONCAT(CustomFieldData ORDER BY CustomFieldData SEPARATOR ',') AS value_list FROM tv_show_table GROUP BY IssueId ORDER BY IssueId

And you will get some nice stuff like:

IssueId	value_list
50	90210, Lame TV Show
51	Family Guy, Funny Stuff

How cool is that?!

the actual eventum query would be:

SELECT icf_iss_id,group_concat(icf_value order by icf_value separator ',') as values_list from eventum_issue_custom_field where icf_fld_id=2 OR icf_fld_id=1 group by icf_iss_id order by icf_iss_id;

Because i only want fields with id 1 or 2.


Bye!