Archive

Archive for the ‘devel’ Category

log-malloc2 v0.2 released

February 9th, 2012 No comments

I’ve actually released log-malloc2 library for linux, that logs calls to memory allocation functions and should be very helpful when trying to locate memory leaks. It can be used without recompiling application, simply by preloading it using LD_PRELOAD .

Logged functions:

  • malloc
  • realloc
  • calloc
  • free
  • memalign

Every function call  is logged with their parameters, amount of allocated/deallocated memory, total amount of allocated memory, copy of /proc/self/statm content and backtrace (call stack) if possible. Additionally function call counts is logged and printed out when application exits.

Typical usage:

LD_PRELOAD=../liblog-malloc2.so PROGRAM ARGUMENTS 1022>/tmp/malloc.log

More complete description of library, its usage, logging format and internals can be found in README file.

Library can be downloaded from project homepage. Actually there is no script helping with malloc log file analysis included, but logfile format is very simple and there should be no big problem write your own doing exactly what you want.

You should note that using this library harms application performance, is intensively uses allocate/deallocate functions. Consider logging to file located on tmpfs (in memory) filesystem, to improve logging IO throughput ;)

Categories: devel, log- malloc2 Tags:

perl-XML::LibXML + global external entity loader

May 15th, 2011 No comments

Just created a quick patch against perl XML::LibXML module, that adds global external entity loader support. Till now it was only possible to have per instance entity loader, but this is not enough if you want i.e., XML::LibXSLT to also use yours entity loader for imports, and input callbacks doesn’t suit all your needs.

Usage is simple:

XML::LibXML::externalEntityLoader(\&_entity_handler);

where _entity_handler is subroutine like by option ext_ent_handler described in (http://search.cpan.org/dist/XML-LibXML/lib/XML/LibXML/Parser.pod#PARSER_OPTIONS).

Note: when you define global entity loader, per instance entity loader is simply ignored.

You can download this patch  from http://devel.dob.sk/patches/perl-XML::LibXML+global_entity_loader-0.1.diff. Just download XML::LibXML from cpan, patch it and install. I’ll try to push it to CPAN if possible ;-)

Categories: devel, perl, xml Tags: ,

Oracle VARNUM/NUMBER encoding in C

April 28th, 2011 No comments

Just a short example C implementation of Oracle VARNUM/NUMBER type encoding, in case you could not use library functions. I didn’t found it anywhere when needed (and afterwards I found it doesn’t helps me at all due to little ‘problem’ in ora :( ) and Oracle documentation is pretty unclear about how-to do it.

 

typedef	struct
{
	uint8_t len;
	uint8_t exp;
	uint8_t man[20];
} VARNUM_t;
static void setVARNUM(VARNUM_t *varnum, int64_t value)
{
	int8_t digits = 0;
	uint64_t u_value = 0;
	/* init */
	memset(varnum, 0, sizeof(*varnum));
	varnum->len	= 1;	//start len
	/* check sign */
	if(value >= 0)
	{
		u_value		= value;
		varnum->exp	= trunc(log(u_value) / log(100)) + 128 + 65;
	}
	else
	{
		u_value		= value * - 1;
		varnum->exp	= trunc(log(u_value) / log(100)) + 128 + 65;
		varnum->exp	= ~varnum->exp;
	}
	/* count value digits */
	digits = trunc(log(u_value) / log(10));
	digits = ((digits / 2) + 1);
	/* mantisa */
	for(; digits >= 0 && varnum->len <= 20; digits--)
        {
                uint64_t v_tmp = 0;
                /* prevent INT overflow for too much digits */
                if(digits > 0)
		{
			uint64_t v_pow = powl(100, digits - 1);
			v_tmp = ((u_value / v_pow) / 100) % 100;
		}
		else
			v_tmp = u_value % 100;
		if(!v_tmp)
			continue;
		/* +1 for positive, subtract 101 for negative */
		v_tmp = (value < 0 ? (101 - v_tmp) : (v_tmp + 1));
                varnum->man[varnum->len - 1] = v_tmp;
		varnum->len++;
	}
	/* terminator byte for negative value */
	if(value < 0 && varnum->len <= 20) 	{
                varnum->man[varnum->len - 1] = 102;
		varnum->len++;
	}
	return;
}

NUMBER is just the same as VARNUM but without len member, thus is one byte shorter.

Note: this is just a quick implementation that works, there are some easy optimizations possible…

Categories: devel, rdbms Tags: ,

perl utf8 and using Digest functions

April 27th, 2011 No comments

I’ve implemented new neat feature (to store unique content only once in cache) to my perl based etl tool and suddenly it started to print sometimes ‘Wide character in subroutine entry perl warning in sha1_hex call. As if this was not enough processed content after being stored in cache  started to be utf-8 corrupted in comparition to the one stored in the cache.

It took few funny hours with of playing with perl, till I’ve found that sha1_hex function somehow destroyed parameter content but in such a beautiful way, that it was almost impossible to detect it. What the best is the cached content was output of LibXML toString() function, but the XML tree itself (or what) has been also corrupted. Well it must one of that  ‘perl secrets’.

Afterwards Google given me some explanation of this activity – I’ve found similar problem reported for md5_sum function from digest::md5 package.

So finally to fix that problem, one must call encode_utf8() on sha1_hex parameter to let sha1_hex work on this copied content.

 

Categories: devel Tags: ,

Solr Pager 0.2.2 released

December 20th, 2009 No comments

Released solr_pager 0.2.2. This release fixes little bug that prevented solr_pager to work in some configurations (ie. with standard search handler). Well, Solr documentation did not mentioned that it can pass empty (null) result set to my component.

Also added some sanity checks to prevent similar situation again.

New version can be found on http://devel.dob.sk/solr_pager, enjoy.

Categories: devel, pager, solr, xml, xsl Tags: