Talend Data Preparation – Tutorial – Part 1

Talend offers a software for cleaning and formatting data. The software is very intuitive. I’m using free Desktop version, so I can’t get wild with data sources. There are only two options: Excel and CSV. 

Ausländer in Deutschland

Ich habe Daten über in Deutschland ansässige Ausländer gefunden. Solche Daten gibt es schon seit langer Zeit, denn Deutschland hat mindestens seit dem Mittelalter das „Meldewesen“:  alle müssen sich in der Stadt, in der sie wohnen, melden, auch die Deutschen – einfach so vor sich hin leben geht nicht!

Aus diesen Daten habe ich eine Visualisierung gebastelt. Vieles daran kann man sich durch anklicken genauer anschauen.

Was ihr jetzt sehen könnt:

  - Migration früher und heute. Ihr müsst dabei aber vorsichtig sein, denn das alte Deutsche Reich, die Bundesrepublik bis 1990 und die Bundesrepublik nach 1990 sind  hinsichtlich ihres Staatsgebiets und hinsichtlich ihrer Bevölkerung sehr unterschiedlich. Ich mag aber alte Daten und wollte euch die vom Deutschen Reich nicht vorenthalten.

  -  Ausländeranteil je 1000 Einwohner.

  - Anteil der (ausländischen) Frauen je 1000 männliche Ausländer. Das zeigt die Genderstruktur der Einwanderung.

  - Ausländer nach Staatsangehörigkeit und Bundesländern. Italiener zum Beispiel werden im Norden Deutschlands immer seltener. Und: Berlin ist klar eine  „Ausländerstadt“, das benachbarte Brandenburg hingegen ….

  - Ausländer nach Staatsangehörigkeit und Geschlecht.

Energy from renewable sources in the EU

Eurostat has published statistics on renewable energy sources in the European Union.

Below my data visualization showing how countries are dealing with their energy targets.

German food mapped

You don't need any geo software to produce a map, a cupboard is perfectly enough.
Some time ago I've found in museum of Haus der Geschichte in Bonn this map of German local food specialties. The map is fully interactive, you can open each part to check what is inside. No computer, no programming involved and a nice interactive map was created.

Free data journalism tools - data extracting and cleaning

Extracting data from PDF

There are plenty of people who strongly believe that the best place for their data is PDF, everyone does it, statistical offices, international organizations. There is only one reason behind it, their pleasure, yes they find it pleasurable to imagine people who try to extract their data from PDF.

Free data journalism tools - time lines

This post starts series of posts with free tools for data journalism. To make everything clear, I have nothing against paid apps but I just wanted to show that having strong willingness to work with data and no budget is enough to build a working place for a data journalist / yourself / data blogger / student / random person who likes data. In more romantic words, love for data is everything, you don't need more :)

Creating and saving path as KML file in Google Earth

You can use KML files to store routes of your trips, bird migration, geographical discoveries or whatever you want. Later you can use KML files to present created geo data in Google Fusion Tables.

How to create KML file to store your geo data

You can use KML files to create maps in Google Fusion Tables. 

Comments to the tutorial: I keep everything in folders, it's not a must but when you have a few polygons or lines or points it's easier to keep order. I segregate everything by type, so for example polygons and points are never in the same folder, I do this just in case if I wanted to export KML to shape file.

This tutorial is useful when you want to create geographic files of places that are hard to find in ready geo files. It's recommended for smaller projects, as everything is done by hand. Easy to imagine that mapping all buildings in a selected country isn't this case :)

Shape file directory

It's a list from my perspective, I work mainly with geo data for Europe and Asian post Soviet Union countries. I wanted to keep all my sources in one place so I started this list, which I am planning to update when I find something interesting. I focus on open source shape files.

If you came across a website with shape files and you want to share it with me, it will be the happiest day of my life.

Велодорожки Москвы / Cycle lanes of Moscow

Портал открытых данных правительства Москвы опубликовал географические координаты московских велодорожек.
Общая длина велодорожек состоит 111 км. Самая длинная велодорожка находится в парке «Лосиный Остров».

Я хотела сделать карту поинтереснее и добавить другие данные к велосипедным геоданным, но не нашла ничего подходящего.

На моей карте можно включить Street View чтобы увидеть фотографии.
========================================================

English version :)

Reshaping data in Data Wrangler

Data Wrangler is a hero of this tutorial. It's free online tool to transform data.

There are three reasons why it's worth to use Data Wrangler:

• it's very easy to use
• although it's simple it offers many useful functions
• the app's done by nice people

The app has limitations, Wrangler only supports up to 40 columns and 1000 rows. If you are planning to work with something bigger than that, then you have to divide your data. This limitation is the most painful part of this tutorial so you can relax and keep reading, now there will be only positive things.

Something old, something new

I'm totally addicted to archaeological documentaries. My favourite archaeological site is Göbekli Tepe, google it, it's just WOW. In July 2014 Klaus Schmidt, who headed the excavations at the site, passed away. I started to thinking about doing something in this field. I wanted to analyse some data devoted to Göbekli Tepe, but I couldn't find anything. Archaeological data published in the internet are rarity.

Simplifying geometries of polygon shape files in QGIS's SimpliPy

First you have to install SimpliPy

Note that the plug-in says: You must have installed Shapely 1.2.18 or you will get an error installing or using this plug-in. Shapely is open source.

==============================================
Here you can learn what simplifying geometries of shape files is.
==============================================

How the world has changed from 1960, population comparator

I was playing with population data but I wanted to do something different so I took into account data for a long period.

I prepared population comparator, a tool to compare how the growth of population proceed in countries. Don't get wild and don't select too many countries as it will be impossible to see anything.

Cycling in London

I've done a cycling map of London.
Not everything was as I planned. I was searching for a line shape file of the London's cycle lanes, as logical would be adding length of cycle lanes per ward or any data set saying about an average commuting distance to work. I asked Transport for London and a few cycling organizations based in London for the shape file but nobody was able to help me.

Simplifying geometries of polygon shape files

If your map works very slowly and you have to wait for a respond for a loooooooooong time, it's a clear sign that you need to simplify the geometries of the shape files.

How to export a point shape file to csv in QGIS

A point shape file in QGIS looks like this:

Maps in Tableau 8.2 - street level

There're new maps in Tableau 8.2. Now you can choose between Tableau (the new maps) and Tableau Classic (the old ones, not medieval maps but just from 8.1 version).

Tableau Mapping Template for London Boroughs

Feel free to use this template in your Tableau projects, just remember about placing the information about the owner of the original shape file.

You can check Tableau template for London wards too.

Tableau Mapping Template for London Wards

If you've searched for a Tableau mapping template for London, you can stop googling :) If you need other templates for the UK, you should check http://tableaumapping.bi

Tableau Mapping is a website where Tableau users share their maps created from open shape files, feel free to join them. Situation with shape files is complex as only in some countries the Statistics Offices publish them for free, so if you spot such file it's worth to give the link to Tableau Mapping.