Well, it turns out, someone else already did, and result is called Viking. It can do openstreetmap/openaerialmap (SuSE version), and it can do transparency, so results look pretty well.
On a related note, I hacked tangogps a bit, and found out that POIs are indeed pretty easy to import with some sql scripting.
#!/bin/bash gpsbabel -i gpx -o csv $1 /tmp/delme.csv ( while IFS=, read A B C; do echo -n 'INSERT INTO "poi" VALUES (' echo "'628450940673161107', $A, $B, 1.0, 0.0, 0.0, '$C', '', 3.0, 0.0, NULL,NULL,NULL,NULL,NULL);" done ) < /tmp/delme.csv | sqlite3 ~/.tangogps/poi.db